selftest/ftrace: Add test results to "-h" help text to ftracetest

selftest/ftrace: Add test results to "-h" help text to ftracetest

[Steven Rostedt]

Peter Zijlstra recently had to debug jump labels that were failing in
the ftracetest script. He became frustrated as there was no easily
available documentation on what the meanings of UNSUPPORTED and
UNRESOLVED were. The only place they are documented is in the code.

At the very least, add the results to the help text, as I find that I
also sometimes need to figure out what those terms mean.

Now "./ftracetest -h" produces:

Usage: ftracetest [options] [testcase(s)] [testcase-directory(s)]
 Options:
		-h|--help  Show help message
		-k|--keep  Keep passed test logs
		-v|--verbose Increase verbosity of test messages
		-vv        Alias of -v -v (Show all results in stdout)
		-vvv       Alias of -v -v -v (Show all commands immediately)
		--fail-unsupported Treat UNSUPPORTED as a failure
		--fail-unresolved Treat UNRESOLVED as a failure
		-d|--debug Debug mode (trace all shell commands)
		-l|--logdir <dir> Save logs on the <dir>
		            If <dir> is -, all logs output in console only

 Test results:
	PASS		The test succeeded
	FAIL		The test failed, but was expected to succeed.
	UNRESOLVED	The test produced indeterminate results. (e.g. interrupted)
	UNTESTED	The test was not run, currently just a placeholder.
	UNSUPPORTED	The test failed because of lack of feature.
	XFAIL		The test failed, and was expected to fail.

Link: https://lore.kernel.org/lkml/YNmflj%2FWpFG5Ivmb@hirez.programming.kicks-ass.net/

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
index 8ec1922e974e..f34580c70650 100755
--- a/tools/testing/selftests/ftrace/ftracetest
+++ b/tools/testing/selftests/ftrace/ftracetest
@@ -21,6 +21,14 @@ echo "		--fail-unresolved Treat UNRESOLVED as a failure"
 echo "		-d|--debug Debug mode (trace all shell commands)"
 echo "		-l|--logdir <dir> Save logs on the <dir>"
 echo "		            If <dir> is -, all logs output in console only"
+echo ""
+echo " Test results:"
+echo "	PASS		The test succeeded"
+echo "	FAIL		The test failed, but was expected to succeed."
+echo "	UNRESOLVED	The test produced indeterminate results. (e.g. interrupted)"
+echo "	UNTESTED	The test was not run, currently just a placeholder."
+echo "	UNSUPPORTED	The test failed because of lack of feature."
+echo "	XFAIL		The test failed, and was expected to fail."
 exit $1
 }
 

Re: selftest/ftrace: Add test results to "-h" help text to ftracetest

[Masami Hiramatsu]

On Mon, 28 Jun 2021 09:50:15 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> Peter Zijlstra recently had to debug jump labels that were failing in
> the ftracetest script. He became frustrated as there was no easily
> available documentation on what the meanings of UNSUPPORTED and
> UNRESOLVED were. The only place they are documented is in the code.

oops, thanks for adding the help message!

> 
> At the very least, add the results to the help text, as I find that I
> also sometimes need to figure out what those terms mean.
> 
> Now "./ftracetest -h" produces:
> 
> Usage: ftracetest [options] [testcase(s)] [testcase-directory(s)]
>  Options:
> 		-h|--help  Show help message
> 		-k|--keep  Keep passed test logs
> 		-v|--verbose Increase verbosity of test messages
> 		-vv        Alias of -v -v (Show all results in stdout)
> 		-vvv       Alias of -v -v -v (Show all commands immediately)
> 		--fail-unsupported Treat UNSUPPORTED as a failure
> 		--fail-unresolved Treat UNRESOLVED as a failure
> 		-d|--debug Debug mode (trace all shell commands)
> 		-l|--logdir <dir> Save logs on the <dir>
> 		            If <dir> is -, all logs output in console only
> 
>  Test results:
> 	PASS		The test succeeded
> 	FAIL		The test failed, but was expected to succeed.
> 	UNRESOLVED	The test produced indeterminate results. (e.g. interrupted)
> 	UNTESTED	The test was not run, currently just a placeholder.
> 	UNSUPPORTED	The test failed because of lack of feature.
> 	XFAIL		The test failed, and was expected to fail.

Looks good to me.

Acked-by: Masami Hiramatsu <mhiramat@kernel.org>

Thank you,

> 
> Link: https://lore.kernel.org/lkml/YNmflj%2FWpFG5Ivmb@hirez.programming.kicks-ass.net/
> 
> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
> ---
> diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
> index 8ec1922e974e..f34580c70650 100755
> --- a/tools/testing/selftests/ftrace/ftracetest
> +++ b/tools/testing/selftests/ftrace/ftracetest
> @@ -21,6 +21,14 @@ echo "		--fail-unresolved Treat UNRESOLVED as a failure"
>  echo "		-d|--debug Debug mode (trace all shell commands)"
>  echo "		-l|--logdir <dir> Save logs on the <dir>"
>  echo "		            If <dir> is -, all logs output in console only"
> +echo ""
> +echo " Test results:"
> +echo "	PASS		The test succeeded"
> +echo "	FAIL		The test failed, but was expected to succeed."
> +echo "	UNRESOLVED	The test produced indeterminate results. (e.g. interrupted)"
> +echo "	UNTESTED	The test was not run, currently just a placeholder."
> +echo "	UNSUPPORTED	The test failed because of lack of feature."
> +echo "	XFAIL		The test failed, and was expected to fail."
>  exit $1
>  }
>  


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: selftest/ftrace: Add test results to "-h" help text to ftracetest

[Peter Zijlstra]

On Mon, Jun 28, 2021 at 09:50:15AM -0400, Steven Rostedt wrote:
> +echo "	UNRESOLVED	The test produced indeterminate results. (e.g. interrupted)"
> +echo "	UNSUPPORTED	The test failed because of lack of feature."

So in my case I've seen both these as a result of .config fails.

Specifically test.d/kprobe/kprobe_module.tc returns exit_unresolved when
it can't load the module (because it doesn't exist).

Also, would it be at all possible to, for both these options, always
print the missing CONFIG_symbol ? The above test actually has an echo
for it, but I've never seen the result of that appear.

Re: selftest/ftrace: Add test results to "-h" help text to ftracetest

[Masami Hiramatsu]

On Mon, 28 Jun 2021 17:08:09 +0200
Peter Zijlstra <peterz@infradead.org> wrote:

> On Mon, Jun 28, 2021 at 09:50:15AM -0400, Steven Rostedt wrote:
> > +echo "	UNRESOLVED	The test produced indeterminate results. (e.g. interrupted)"
> > +echo "	UNSUPPORTED	The test failed because of lack of feature."
> 
> So in my case I've seen both these as a result of .config fails.
> 
> Specifically test.d/kprobe/kprobe_module.tc returns exit_unresolved when
> it can't load the module (because it doesn't exist).

Since I defined these result code from DejaGnu (based on POSIX 1003.3) result,
I made such error (failed to load kernel module, which just gives a target
but is not an implementation of the tested feature) UNRESOLVED.

https://www.gnu.org/software/dejagnu/manual/A-POSIX-Conforming-Test-Framework.html
---
Here are some of the ways a test may wind up UNRESOLVED:
...
- The test was set up incorrectly.
---
My logic is that you are not correctly (compile and) install the target module,
so this is a setting issue.


> Also, would it be at all possible to, for both these options, always
> print the missing CONFIG_symbol ? The above test actually has an echo
> for it, but I've never seen the result of that appear.

That's a good idea :) Let me try to add depending CONFIG_ options for
each test cases.

Thank you, 

-- 
Masami Hiramatsu <mhiramat@kernel.org>