[PATCH v2 0/2] ftrace/selftest: clean up failure cases

[PATCH v2 0/2] ftrace/selftest: clean up failure cases

[Alan Maguire]

When running the ftrace selftests, 2 failures and 6 unresolved
cases were observed.  The failures can be avoided by setting
a sysctl prior to test execution (fixed in patch 1) and by
having unresolved cases not return 0 from ftracetest by default
since they indicate an absence of testing modules/programs
rather than ftrace issues (patch 2).

The latter are classified as "unresolved" tests, which operate
differently from "unsupported" tests.  For unsupported tests,
we note the unsupported count but do not consider the tests
as having failed, whereas with unresolved the test run is
considered to have failed so returns "not ok" when run via
kselftest ("make -C tools/testing/selftest/ftrace run_tests").

Patch 2 aligns the unresolved behaviour with the unsupported;
by default, unresolved outcomes do not trigger overall failure,
but they can if --fail-unresolved is specified.

Changes since v1:

- updated patch 1 to use /proc path instead of sysctl (Masami)
- updated patch 2 to modify unresolved handling in ftracetest
  rather than change individual unresolved -> unsupported (Masami)

Alan Maguire (2):
  ftrace/selftests: workaround cgroup RT scheduling issues
  ftrace/selftest: make unresolved cases cause failure if
    --fail-unresolved set

 tools/testing/selftests/ftrace/ftracetest | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)

-- 
1.8.3.1

[PATCH v2 1/2] ftrace/selftests: workaround cgroup RT scheduling issues

[Alan Maguire]

wakeup_rt.tc and wakeup.tc tests in tracers/ subdirectory
fail due to the chrt command returning:

 chrt: failed to set pid 0's policy: Operation not permitted.

To work around this, temporarily disable grout RT scheduling
during ftracetest execution.  Restore original value on
test run completion.  With these changes in place, both
tests consistently pass.

Fixes: c575dea2c1a5 ("selftests/ftrace: Add wakeup_rt tracer testcase")
Fixes: c1edd060b413 ("selftests/ftrace: Add wakeup tracer testcase")
Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 tools/testing/selftests/ftrace/ftracetest | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
index 063ecb2..144308a 100755
--- a/tools/testing/selftests/ftrace/ftracetest
+++ b/tools/testing/selftests/ftrace/ftracetest
@@ -29,8 +29,25 @@ err_ret=1
 # kselftest skip code is 4
 err_skip=4
 
+# cgroup RT scheduling prevents chrt commands from succeeding, which
+# induces failures in test wakeup tests.  Disable for the duration of
+# the tests.
+
+readonly sched_rt_runtime=/proc/sys/kernel/sched_rt_runtime_us
+
+sched_rt_runtime_orig=$(cat $sched_rt_runtime)
+
+setup() {
+  echo -1 > $sched_rt_runtime
+}
+
+cleanup() {
+  echo $sched_rt_runtime_orig > $sched_rt_runtime
+}
+
 errexit() { # message
   echo "Error: $1" 1>&2
+  cleanup
   exit $err_ret
 }
 
@@ -39,6 +56,8 @@ if [ `id -u` -ne 0 ]; then
   errexit "this must be run by root user"
 fi
 
+setup
+
 # Utilities
 absdir() { # file_path
   (cd `dirname $1`; pwd)
@@ -235,6 +254,7 @@ TOTAL_RESULT=0
 
 INSTANCE=
 CASENO=0
+
 testcase() { # testfile
   CASENO=$((CASENO+1))
   desc=`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
@@ -406,5 +426,7 @@ prlog "# of unsupported: " `echo $UNSUPPORTED_CASES | wc -w`
 prlog "# of xfailed: " `echo $XFAILED_CASES | wc -w`
 prlog "# of undefined(test bug): " `echo $UNDEFINED_CASES | wc -w`
 
+cleanup
+
 # if no error, return 0
 exit $TOTAL_RESULT
-- 
1.8.3.1

[PATCH v2 2/2] ftrace/selftest: make unresolved cases cause failure if --fail-unresolved set

[Alan Maguire]

Currently, ftracetest will return 1 (failure) if any unresolved cases
are encountered.  The unresolved status results from modules and
programs not being available, and as such does not indicate any
issues with ftrace itself.  As such, change the behaviour of
ftracetest in line with unsupported cases; if unsupported cases
happen, ftracetest still returns 0 unless --fail-unsupported.  Here
--fail-unresolved is added and the default is to return 0 if
unresolved results occur.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 tools/testing/selftests/ftrace/ftracetest | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
index 144308a..19e9236 100755
--- a/tools/testing/selftests/ftrace/ftracetest
+++ b/tools/testing/selftests/ftrace/ftracetest
@@ -17,6 +17,7 @@ echo "		-v|--verbose Increase verbosity of test messages"
 echo "		-vv        Alias of -v -v (Show all results in stdout)"
 echo "		-vvv       Alias of -v -v -v (Show all commands immediately)"
 echo "		--fail-unsupported Treat UNSUPPORTED as a failure"
+echo "		--fail-unresolved Treat UNRESOLVED as a failure"
 echo "		-d|--debug Debug mode (trace all shell commands)"
 echo "		-l|--logdir <dir> Save logs on the <dir>"
 echo "		            If <dir> is -, all logs output in console only"
@@ -112,6 +113,10 @@ parse_opts() { # opts
       UNSUPPORTED_RESULT=1
       shift 1
     ;;
+    --fail-unresolved)
+      UNRESOLVED_RESULT=1
+      shift 1
+    ;;
     --logdir|-l)
       LOG_DIR=$2
       shift 2
@@ -176,6 +181,7 @@ KEEP_LOG=0
 DEBUG=0
 VERBOSE=0
 UNSUPPORTED_RESULT=0
+UNRESOLVED_RESULT=0
 STOP_FAILURE=0
 # Parse command-line options
 parse_opts $*
@@ -280,7 +286,7 @@ eval_result() { # sigval
     $UNRESOLVED)
       prlog "	[${color_blue}UNRESOLVED${color_reset}]"
       UNRESOLVED_CASES="$UNRESOLVED_CASES $CASENO"
-      return 1 # this is a kind of bug.. something happened.
+      return $UNRESOLVED_RESULT # depends on use case
     ;;
     $UNTESTED)
       prlog "	[${color_blue}UNTESTED${color_reset}]"
-- 
1.8.3.1

Re: [PATCH v2 2/2] ftrace/selftest: make unresolved cases cause failure if --fail-unresolved set

[Masami Hiramatsu]

On Wed, 19 Feb 2020 09:33:30 +0000
Alan Maguire <alan.maguire@oracle.com> wrote:

> Currently, ftracetest will return 1 (failure) if any unresolved cases
> are encountered.  The unresolved status results from modules and
> programs not being available, and as such does not indicate any
> issues with ftrace itself.  As such, change the behaviour of
> ftracetest in line with unsupported cases; if unsupported cases
> happen, ftracetest still returns 0 unless --fail-unsupported.  Here
> --fail-unresolved is added and the default is to return 0 if
> unresolved results occur.
> 

OK, this looks good to me. One note, with this change, ftracetest doesn't
fail even if your test environment is not well prepared anymore.

Acked-by: Masami Hiramatsu <mhiramat@kernel.org>

Thank you,

> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> ---
>  tools/testing/selftests/ftrace/ftracetest | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
> index 144308a..19e9236 100755
> --- a/tools/testing/selftests/ftrace/ftracetest
> +++ b/tools/testing/selftests/ftrace/ftracetest
> @@ -17,6 +17,7 @@ echo "		-v|--verbose Increase verbosity of test messages"
>  echo "		-vv        Alias of -v -v (Show all results in stdout)"
>  echo "		-vvv       Alias of -v -v -v (Show all commands immediately)"
>  echo "		--fail-unsupported Treat UNSUPPORTED as a failure"
> +echo "		--fail-unresolved Treat UNRESOLVED as a failure"
>  echo "		-d|--debug Debug mode (trace all shell commands)"
>  echo "		-l|--logdir <dir> Save logs on the <dir>"
>  echo "		            If <dir> is -, all logs output in console only"
> @@ -112,6 +113,10 @@ parse_opts() { # opts
>        UNSUPPORTED_RESULT=1
>        shift 1
>      ;;
> +    --fail-unresolved)
> +      UNRESOLVED_RESULT=1
> +      shift 1
> +    ;;
>      --logdir|-l)
>        LOG_DIR=$2
>        shift 2
> @@ -176,6 +181,7 @@ KEEP_LOG=0
>  DEBUG=0
>  VERBOSE=0
>  UNSUPPORTED_RESULT=0
> +UNRESOLVED_RESULT=0
>  STOP_FAILURE=0
>  # Parse command-line options
>  parse_opts $*
> @@ -280,7 +286,7 @@ eval_result() { # sigval
>      $UNRESOLVED)
>        prlog "	[${color_blue}UNRESOLVED${color_reset}]"
>        UNRESOLVED_CASES="$UNRESOLVED_CASES $CASENO"
> -      return 1 # this is a kind of bug.. something happened.
> +      return $UNRESOLVED_RESULT # depends on use case
>      ;;
>      $UNTESTED)
>        prlog "	[${color_blue}UNTESTED${color_reset}]"
> -- 
> 1.8.3.1
> 


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH v2 2/2] ftrace/selftest: make unresolved cases cause failure if --fail-unresolved set

[Masami Hiramatsu]

On Wed, 19 Feb 2020 20:39:41 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:

> On Wed, 19 Feb 2020 09:33:30 +0000
> Alan Maguire <alan.maguire@oracle.com> wrote:
> 
> > Currently, ftracetest will return 1 (failure) if any unresolved cases
> > are encountered.  The unresolved status results from modules and
> > programs not being available, and as such does not indicate any
> > issues with ftrace itself.  As such, change the behaviour of
> > ftracetest in line with unsupported cases; if unsupported cases
> > happen, ftracetest still returns 0 unless --fail-unsupported.  Here
> > --fail-unresolved is added and the default is to return 0 if
> > unresolved results occur.
> > 
> 
> OK, this looks good to me. One note, with this change, ftracetest doesn't
> fail even if your test environment is not well prepared anymore.
> 
> Acked-by: Masami Hiramatsu <mhiramat@kernel.org>

Hi Shuah,
Could you pick this up?

Po-Hsu Lin seemed to face same problem recently. If this applied, it will be solved.

Thank you,

> 
> Thank you,
> 
> > Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> > ---
> >  tools/testing/selftests/ftrace/ftracetest | 8 +++++++-
> >  1 file changed, 7 insertions(+), 1 deletion(-)
> > 
> > diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
> > index 144308a..19e9236 100755
> > --- a/tools/testing/selftests/ftrace/ftracetest
> > +++ b/tools/testing/selftests/ftrace/ftracetest
> > @@ -17,6 +17,7 @@ echo "		-v|--verbose Increase verbosity of test messages"
> >  echo "		-vv        Alias of -v -v (Show all results in stdout)"
> >  echo "		-vvv       Alias of -v -v -v (Show all commands immediately)"
> >  echo "		--fail-unsupported Treat UNSUPPORTED as a failure"
> > +echo "		--fail-unresolved Treat UNRESOLVED as a failure"
> >  echo "		-d|--debug Debug mode (trace all shell commands)"
> >  echo "		-l|--logdir <dir> Save logs on the <dir>"
> >  echo "		            If <dir> is -, all logs output in console only"
> > @@ -112,6 +113,10 @@ parse_opts() { # opts
> >        UNSUPPORTED_RESULT=1
> >        shift 1
> >      ;;
> > +    --fail-unresolved)
> > +      UNRESOLVED_RESULT=1
> > +      shift 1
> > +    ;;
> >      --logdir|-l)
> >        LOG_DIR=$2
> >        shift 2
> > @@ -176,6 +181,7 @@ KEEP_LOG=0
> >  DEBUG=0
> >  VERBOSE=0
> >  UNSUPPORTED_RESULT=0
> > +UNRESOLVED_RESULT=0
> >  STOP_FAILURE=0
> >  # Parse command-line options
> >  parse_opts $*
> > @@ -280,7 +286,7 @@ eval_result() { # sigval
> >      $UNRESOLVED)
> >        prlog "	[${color_blue}UNRESOLVED${color_reset}]"
> >        UNRESOLVED_CASES="$UNRESOLVED_CASES $CASENO"
> > -      return 1 # this is a kind of bug.. something happened.
> > +      return $UNRESOLVED_RESULT # depends on use case
> >      ;;
> >      $UNTESTED)
> >        prlog "	[${color_blue}UNTESTED${color_reset}]"
> > -- 
> > 1.8.3.1
> > 
> 
> 
> -- 
> Masami Hiramatsu <mhiramat@kernel.org>


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH v2 2/2] ftrace/selftest: make unresolved cases cause failure if --fail-unresolved set

[shuah]

On 5/1/20 3:42 AM, Masami Hiramatsu wrote:
> On Wed, 19 Feb 2020 20:39:41 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
>> On Wed, 19 Feb 2020 09:33:30 +0000
>> Alan Maguire <alan.maguire@oracle.com> wrote:
>>
>>> Currently, ftracetest will return 1 (failure) if any unresolved cases
>>> are encountered.  The unresolved status results from modules and
>>> programs not being available, and as such does not indicate any
>>> issues with ftrace itself.  As such, change the behaviour of
>>> ftracetest in line with unsupported cases; if unsupported cases
>>> happen, ftracetest still returns 0 unless --fail-unsupported.  Here
>>> --fail-unresolved is added and the default is to return 0 if
>>> unresolved results occur.
>>>
>>
>> OK, this looks good to me. One note, with this change, ftracetest doesn't
>> fail even if your test environment is not well prepared anymore.
>>
>> Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
> 
> Hi Shuah,
> Could you pick this up?
> 
> Po-Hsu Lin seemed to face same problem recently. If this applied, it will be solved.
> 

Sorry about this. I will get these in

thanks,
-- Shuah

Re: [PATCH v2 2/2] ftrace/selftest: make unresolved cases cause failure if --fail-unresolved set

[Masami Hiramatsu]

On Fri, 1 May 2020 08:25:50 -0600
shuah <shuah@kernel.org> wrote:

> On 5/1/20 3:42 AM, Masami Hiramatsu wrote:
> > On Wed, 19 Feb 2020 20:39:41 +0900
> > Masami Hiramatsu <mhiramat@kernel.org> wrote:
> > 
> >> On Wed, 19 Feb 2020 09:33:30 +0000
> >> Alan Maguire <alan.maguire@oracle.com> wrote:
> >>
> >>> Currently, ftracetest will return 1 (failure) if any unresolved cases
> >>> are encountered.  The unresolved status results from modules and
> >>> programs not being available, and as such does not indicate any
> >>> issues with ftrace itself.  As such, change the behaviour of
> >>> ftracetest in line with unsupported cases; if unsupported cases
> >>> happen, ftracetest still returns 0 unless --fail-unsupported.  Here
> >>> --fail-unresolved is added and the default is to return 0 if
> >>> unresolved results occur.
> >>>
> >>
> >> OK, this looks good to me. One note, with this change, ftracetest doesn't
> >> fail even if your test environment is not well prepared anymore.
> >>
> >> Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
> > 
> > Hi Shuah,
> > Could you pick this up?
> > 
> > Po-Hsu Lin seemed to face same problem recently. If this applied, it will be solved.
> > 
> 
> Sorry about this. I will get these in

Thanks Shuah!

> 
> thanks,
> -- Shuah


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH v2 0/2] ftrace/selftest: clean up failure cases

[Steven Rostedt]

Shuah,

Can you take these two patches through your tree?

Acked-by: Steven Rostedt (VMware) <rostedt@goodmis.org>

-- Steve


On Wed, 19 Feb 2020 09:33:28 +0000
Alan Maguire <alan.maguire@oracle.com> wrote:

> When running the ftrace selftests, 2 failures and 6 unresolved
> cases were observed.  The failures can be avoided by setting
> a sysctl prior to test execution (fixed in patch 1) and by
> having unresolved cases not return 0 from ftracetest by default
> since they indicate an absence of testing modules/programs
> rather than ftrace issues (patch 2).
> 
> The latter are classified as "unresolved" tests, which operate
> differently from "unsupported" tests.  For unsupported tests,
> we note the unsupported count but do not consider the tests
> as having failed, whereas with unresolved the test run is
> considered to have failed so returns "not ok" when run via
> kselftest ("make -C tools/testing/selftest/ftrace run_tests").
> 
> Patch 2 aligns the unresolved behaviour with the unsupported;
> by default, unresolved outcomes do not trigger overall failure,
> but they can if --fail-unresolved is specified.
> 
> Changes since v1:
> 
> - updated patch 1 to use /proc path instead of sysctl (Masami)
> - updated patch 2 to modify unresolved handling in ftracetest
>   rather than change individual unresolved -> unsupported (Masami)
> 
> Alan Maguire (2):
>   ftrace/selftests: workaround cgroup RT scheduling issues
>   ftrace/selftest: make unresolved cases cause failure if
>     --fail-unresolved set
> 
>  tools/testing/selftests/ftrace/ftracetest | 30 +++++++++++++++++++++++++++++-
>  1 file changed, 29 insertions(+), 1 deletion(-)
>