[RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Steven Rostedt]

[
  Masami, I took two of my test scripts and added some basic comments
  to them and copied them pretty much unchanged into a ftrace directory
  under test.d. Is this fine, or is there more massaging I need to do
  to them?

  I know the echos don't show up, but I kept them anyway. What should
  happen with them?
]

Added two test cases to get the feel of adding tests to ftracetest.
The two cases are:

  function profiling test, to make sure function profiling still works
   with function tracing (was a regression)

  function graph filter test to make sure that the function graph filter
   does filter and also continues to filter when another function tracer
   is running (like the stack tracer)

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 .../ftrace/test.d/ftrace/fgraph-filter.tc          | 111 +++++++++++++++++++++
 .../ftrace/test.d/ftrace/func_profiler.tc          |  78 +++++++++++++++
 2 files changed, 189 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc

diff --git a/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc b/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
new file mode 100644
index 000000000000..42f764d1f8d2
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
@@ -0,0 +1,111 @@
+#!/bin/sh
+# description: ftrace - function graph filters
+
+# Make sure that function graph filtering works, and is not
+# affected by other tracers enabled (like stack tracer)
+
+if ! grep -q function_graph available_tracers; then
+    echo "no function graph tracer configured"
+    exit 0;
+fi
+
+if [ ! -f set_ftrace_filter ]; then
+    echo "set_ftrace_filter not found? Is dynamic ftrace not set?"
+    exit 0
+fi
+
+do_reset() {
+    echo nop > current_tracer
+    echo 0 > /proc/sys/kernel/stack_tracer_enabled
+    echo 1 > tracing_on
+    echo > trace
+    echo > set_ftrace_filter
+}
+
+echo 0 > tracing_on
+# clear trace
+echo > trace
+# filter something, schedule is always good
+if ! echo "schedule" > set_ftrace_filter; then
+    # test for powerpc 64
+    if ! echo ".schedule" > set_ftrace_filter; then
+	echo "can not enable schedule filter"
+	exit -1
+    fi
+fi
+
+echo function_graph > current_tracer
+echo 1 > tracing_on
+sleep 1
+# search for functions (has "()" on the line), and make sure
+# that only the schedule function was found
+count=`cat trace | grep '()' | grep -v schedule | wc -l`
+if [ $count -ne 0 ]; then
+    echo "Graph filtering not working by itself?"
+    exit -1;
+fi
+
+# Make sure we did find something
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
+    exit -1
+fi
+
+echo "Graph filtering works by itself"
+
+if [ ! -f stack_trace ]; then
+    echo "Stack tracer not configured, can't continue test"
+    # stack tracer not configured in this kernel? pass test anyway
+    do_reset
+    exit 0;
+fi
+
+echo "Now testing with stack tracer"
+
+echo 1 > /proc/sys/kernel/stack_tracer_enabled
+
+echo 0 > tracing_on
+echo > trace
+echo 1 > tracing_on
+sleep 1
+
+count=`cat trace | grep '()' | grep -v schedule | wc -l`
+
+if [ $count -ne 0 ]; then
+    echo "Graph filtering not working with stack tracer?"
+    exit -1
+fi
+
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
+    exit -1
+fi
+
+echo "Graph filtering works with stack tracer"
+
+echo "Now testing if filtering still works without stack tracer"
+
+echo 0 > /proc/sys/kernel/stack_tracer_enabled
+echo > trace
+sleep 1
+
+
+count=`cat trace | grep '()' | grep -v schedule | wc -l`
+
+if [ $count -ne 0 ]; then
+    echo "Graph filtering not working after stack tracer disabled?"
+    exit -1
+fi
+
+count=`cat trace | grep 'schedule()' | wc -l` 
+if [ $count -eq 0 ]; then
+    echo "No schedule traces found?"
+    exit -1
+fi
+
+do_reset
+
+echo "SUCCESS!"
+exit 0
diff --git a/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc b/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc
new file mode 100644
index 000000000000..f35c58968d09
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc
@@ -0,0 +1,78 @@
+#!/bin/sh
+# description: ftrace - function profiler with function tracing
+
+# There was a bug after a rewrite of the ftrace infrastructure that
+# caused the function_profiler not to be able to run with the function
+# tracer, because the function_profiler used the function_graph tracer
+# and it was assumed the two could not run simultaneously.
+#
+# There was another related bug where the solution to the first bug
+# broke the way filtering of the function tracer worked.
+#
+# This test triggers those bugs on those kernels.
+#
+# We need function_graph and profiling to to run this test
+if ! grep -q function_graph available_tracers; then
+    echo "no function graph tracer configured"
+    # allow it to pass, as the kernel doesn't support it
+    exit 0;
+fi
+
+if [ ! -f set_ftrace_filter ]; then
+    echo "set_ftrace_filter not found? Is dynamic ftrace not set?"
+    exit 0
+fi
+
+if [ ! -f function_profile_enabled ]; then
+    echo "function_profile_enabled not found, function profiling enabled?"
+    exit 0
+fi
+
+echo "Testing function tracer with profiler:"
+echo "enable function tracer"
+echo function > current_tracer || exit 1
+echo "enable profiler"
+echo 1 > function_profile_enabled || exit 1
+
+sleep 1
+
+echo "Now filter on just schedule"
+echo '*schedule' > set_ftrace_filter
+> trace
+
+echo "Now disable function profiler"
+echo 0 > function_profile_enabled || exit 1
+
+sleep 1
+
+# make sure only schedule functions exist
+
+echo "testing if only schedule is being traced"
+if grep -v -e '^#' -e 'schedule' trace; then
+	echo "more than schedule was found"
+	exit 1
+fi
+
+echo "Make sure schedule was traced"
+if ! grep -e 'schedule' trace > /dev/null; then
+	cat trace
+	echo "can not find schedule in trace"
+	exit 1
+fi
+
+echo "clear filter"
+echo > set_ftrace_filter || exit 1
+> trace
+
+sleep 1
+
+echo "make sure something other than scheduler is being traced"
+if ! grep -v -e '^#' -e 'schedule' trace > /dev/null; then
+	cat trace
+	echo "no other functions besides schedule was found"
+	exit 1
+fi
+
+echo nop > current_tracer || exit 1
+
+exit 0
-- 
1.8.1.4

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Masami Hiramatsu]

(2014/09/24 6:38), Steven Rostedt wrote:
> 
> [
>   Masami, I took two of my test scripts and added some basic comments

Thanks!

>   to them and copied them pretty much unchanged into a ftrace directory
>   under test.d. Is this fine, or is there more massaging I need to do
>   to them?

Yeah, ftrace has a double meaning :), so test.d/ftrace is fine to me.

> 
>   I know the echos don't show up, but I kept them anyway. What should
>   happen with them?

I think you'd better use exit_unsupported/exit_xfail to notify
that the test target is not configured, or expected to fail.
Then the user can reconfigure that. Maybe we should keep the
detailed log of such results. (you can do it with --keep option)

> ]
> 
> Added two test cases to get the feel of adding tests to ftracetest.
> The two cases are:
> 
>   function profiling test, to make sure function profiling still works
>    with function tracing (was a regression)
> 
>   function graph filter test to make sure that the function graph filter
>    does filter and also continues to filter when another function tracer
>    is running (like the stack tracer)
> 
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
> ---
>  .../ftrace/test.d/ftrace/fgraph-filter.tc          | 111 +++++++++++++++++++++
>  .../ftrace/test.d/ftrace/func_profiler.tc          |  78 +++++++++++++++
>  2 files changed, 189 insertions(+)
>  create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
>  create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc
> 
> diff --git a/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc b/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
> new file mode 100644
> index 000000000000..42f764d1f8d2
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
> @@ -0,0 +1,111 @@
> +#!/bin/sh
> +# description: ftrace - function graph filters
> +
> +# Make sure that function graph filtering works, and is not
> +# affected by other tracers enabled (like stack tracer)
> +
> +if ! grep -q function_graph available_tracers; then
> +    echo "no function graph tracer configured"
> +    exit 0;

This should call exit_unsupported, because the test is not passed.

> +fi
> +
> +if [ ! -f set_ftrace_filter ]; then
> +    echo "set_ftrace_filter not found? Is dynamic ftrace not set?"
> +    exit 0

Ditto.

> +fi
> +
> +do_reset() {
> +    echo nop > current_tracer
> +    echo 0 > /proc/sys/kernel/stack_tracer_enabled
> +    echo 1 > tracing_on
> +    echo > trace
> +    echo > set_ftrace_filter
> +}
> +
> +echo 0 > tracing_on
> +# clear trace
> +echo > trace
> +# filter something, schedule is always good
> +if ! echo "schedule" > set_ftrace_filter; then
> +    # test for powerpc 64
> +    if ! echo ".schedule" > set_ftrace_filter; then
> +	echo "can not enable schedule filter"
> +	exit -1
> +    fi
> +fi
> +
> +echo function_graph > current_tracer
> +echo 1 > tracing_on
> +sleep 1
> +# search for functions (has "()" on the line), and make sure
> +# that only the schedule function was found
> +count=`cat trace | grep '()' | grep -v schedule | wc -l`
> +if [ $count -ne 0 ]; then
> +    echo "Graph filtering not working by itself?"
> +    exit -1;
> +fi
> +
> +# Make sure we did find something
> +count=`cat trace | grep 'schedule()' | wc -l` 
> +if [ $count -eq 0 ]; then
> +    echo "No schedule traces found?"
> +    exit -1
> +fi
> +
> +echo "Graph filtering works by itself"
> +

I think the following part should be a separated test for
stack trace.

> +if [ ! -f stack_trace ]; then
> +    echo "Stack tracer not configured, can't continue test"
> +    # stack tracer not configured in this kernel? pass test anyway
> +    do_reset
> +    exit 0;

Here should call exit_unsupported.

> +fi
> +
> +echo "Now testing with stack tracer"
> +
> +echo 1 > /proc/sys/kernel/stack_tracer_enabled
> +
> +echo 0 > tracing_on
> +echo > trace
> +echo 1 > tracing_on
> +sleep 1
> +
> +count=`cat trace | grep '()' | grep -v schedule | wc -l`
> +
> +if [ $count -ne 0 ]; then
> +    echo "Graph filtering not working with stack tracer?"
> +    exit -1
> +fi
> +
> +count=`cat trace | grep 'schedule()' | wc -l` 
> +if [ $count -eq 0 ]; then
> +    echo "No schedule traces found?"
> +    exit -1
> +fi
> +
> +echo "Graph filtering works with stack tracer"
> +
> +echo "Now testing if filtering still works without stack tracer"
> +
> +echo 0 > /proc/sys/kernel/stack_tracer_enabled
> +echo > trace
> +sleep 1
> +
> +
> +count=`cat trace | grep '()' | grep -v schedule | wc -l`
> +
> +if [ $count -ne 0 ]; then
> +    echo "Graph filtering not working after stack tracer disabled?"
> +    exit -1
> +fi
> +
> +count=`cat trace | grep 'schedule()' | wc -l` 
> +if [ $count -eq 0 ]; then
> +    echo "No schedule traces found?"
> +    exit -1
> +fi
> +
> +do_reset
> +
> +echo "SUCCESS!"
> +exit 0
> diff --git a/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc b/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc
> new file mode 100644
> index 000000000000..f35c58968d09
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc
> @@ -0,0 +1,78 @@
> +#!/bin/sh
> +# description: ftrace - function profiler with function tracing
> +
> +# There was a bug after a rewrite of the ftrace infrastructure that
> +# caused the function_profiler not to be able to run with the function
> +# tracer, because the function_profiler used the function_graph tracer
> +# and it was assumed the two could not run simultaneously.
> +#
> +# There was another related bug where the solution to the first bug
> +# broke the way filtering of the function tracer worked.
> +#
> +# This test triggers those bugs on those kernels.
> +#
> +# We need function_graph and profiling to to run this test
> +if ! grep -q function_graph available_tracers; then
> +    echo "no function graph tracer configured"
> +    # allow it to pass, as the kernel doesn't support it
> +    exit 0;
> +fi
> +
> +if [ ! -f set_ftrace_filter ]; then
> +    echo "set_ftrace_filter not found? Is dynamic ftrace not set?"
> +    exit 0
> +fi
> +
> +if [ ! -f function_profile_enabled ]; then
> +    echo "function_profile_enabled not found, function profiling enabled?"
> +    exit 0
> +fi

These checks should call exit_unsupported if failed.

> +
> +echo "Testing function tracer with profiler:"
> +echo "enable function tracer"
> +echo function > current_tracer || exit 1
> +echo "enable profiler"
> +echo 1 > function_profile_enabled || exit 1

Now you don't need "|| exit 1", since if the command fails,
the shell exits with error code.

> +
> +sleep 1
> +
> +echo "Now filter on just schedule"
> +echo '*schedule' > set_ftrace_filter
> +> trace

echo > trace?

Thank you,

> +
> +echo "Now disable function profiler"
> +echo 0 > function_profile_enabled || exit 1
> +
> +sleep 1
> +
> +# make sure only schedule functions exist
> +
> +echo "testing if only schedule is being traced"
> +if grep -v -e '^#' -e 'schedule' trace; then
> +	echo "more than schedule was found"
> +	exit 1
> +fi
> +
> +echo "Make sure schedule was traced"
> +if ! grep -e 'schedule' trace > /dev/null; then
> +	cat trace
> +	echo "can not find schedule in trace"
> +	exit 1
> +fi
> +
> +echo "clear filter"
> +echo > set_ftrace_filter || exit 1
> +> trace
> +
> +sleep 1
> +
> +echo "make sure something other than scheduler is being traced"
> +if ! grep -v -e '^#' -e 'schedule' trace > /dev/null; then
> +	cat trace
> +	echo "no other functions besides schedule was found"
> +	exit 1
> +fi
> +
> +echo nop > current_tracer || exit 1
> +
> +exit 0
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Steven Rostedt]

On Wed, 24 Sep 2014 11:58:50 +0900
Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:

> (2014/09/24 6:38), Steven Rostedt wrote:
 
> >   to them and copied them pretty much unchanged into a ftrace directory
> >   under test.d. Is this fine, or is there more massaging I need to do
> >   to them?
> 
> Yeah, ftrace has a double meaning :), so test.d/ftrace is fine to me.

I was thinking that more complex tests can go into ftrace, and simple
tests can go into basic.

> 
> > 
> >   I know the echos don't show up, but I kept them anyway. What should
> >   happen with them?
> 
> I think you'd better use exit_unsupported/exit_xfail to notify
> that the test target is not configured, or expected to fail.

OK, that answers the exit codes (as you also stated below), but what
about the echos themselves?

> Then the user can reconfigure that. Maybe we should keep the
> detailed log of such results. (you can do it with --keep option)

You mean keep the echos, as they are ignored anyway, but if we add
--keep, then the echos will show?  Maybe that option should be -v, like
other tools use for "verbose".


> > --- /dev/null
> > +++ b/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
> > @@ -0,0 +1,111 @@
> > +#!/bin/sh
> > +# description: ftrace - function graph filters
> > +
> > +# Make sure that function graph filtering works, and is not
> > +# affected by other tracers enabled (like stack tracer)
> > +
> > +if ! grep -q function_graph available_tracers; then
> > +    echo "no function graph tracer configured"
> > +    exit 0;
> 
> This should call exit_unsupported, because the test is not passed.

Will update this and others.


> > +# Make sure we did find something
> > +count=`cat trace | grep 'schedule()' | wc -l` 
> > +if [ $count -eq 0 ]; then
> > +    echo "No schedule traces found?"
> > +    exit -1
> > +fi
> > +
> > +echo "Graph filtering works by itself"
> > +
> 
> I think the following part should be a separated test for
> stack trace.

Good point. A lot of my test scripts do multiple tests. I think for
putting them into tools/testing/selftests/ftrace, I'll break them up
and make them separate tests. For example, the simple graph filtering
above is an example of something that can go into the basic directory,
but the test against stack tracer should be in ftrace. What do you
think?


> > +
> > +echo "Now filter on just schedule"
> > +echo '*schedule' > set_ftrace_filter
> > +> trace
> 
> echo > trace?

OK. Hmm, I wonder if we should make a bunch of functions that the tests
can use. Like a "clear_trace" call that does this. Can we export
functions that this shell will be able to use? When we see lots of
duplicate code we may want to have something like that. Well, for this,
it may not be that big of a deal, because "echo > trace" is rather
simple. But I do have other operations that are a bit more intrusive.

-- Steve

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Steven Rostedt]

On Wed, 24 Sep 2014 11:42:35 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> > I think you'd better use exit_unsupported/exit_xfail to notify
> > that the test target is not configured, or expected to fail.
> 

Are these functions defined anywhere? I don't seem them in your v5
patch set, or in selftests anywhere.

-- Steve

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Shuah Khan]

On 09/24/2014 09:58 AM, Steven Rostedt wrote:
> On Wed, 24 Sep 2014 11:42:35 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
>>> I think you'd better use exit_unsupported/exit_xfail to notify
>>> that the test target is not configured, or expected to fail.
>>
> 
> Are these functions defined anywhere? I don't seem them in your v5
> patch set, or in selftests anywhere.
> 
> -- Steve
> 

Steve,

I am working on adding the framework for c tests. We will have
to get this evolved for shell scripts. Please see the following
thread:

https://lkml.org/lkml/2014/9/23/926

thanks,
-- Shuah

-- 
Shuah Khan
Sr. Linux Kernel Developer
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Steven Rostedt]

On Wed, 24 Sep 2014 10:01:38 -0600
Shuah Khan <shuahkh@osg.samsung.com> wrote:

> I am working on adding the framework for c tests. We will have
> to get this evolved for shell scripts. Please see the following
> thread:
> 
> https://lkml.org/lkml/2014/9/23/926
> 

Thanks,

Also, what's the rules for a test that requires other tools. Some of my
tests require trace-cmd to be installed as well as perf. My stress test
runs perf against trace-cmd running hackbench (another required tool),
but I believe that perf has hackbench built in so I could replace
hackbench with the perf version.

-- Steve

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Shuah Khan]

On 09/24/2014 10:08 AM, Steven Rostedt wrote:
> On Wed, 24 Sep 2014 10:01:38 -0600
> Shuah Khan <shuahkh@osg.samsung.com> wrote:
> 
>> I am working on adding the framework for c tests. We will have
>> to get this evolved for shell scripts. Please see the following
>> thread:
>>
>> https://lkml.org/lkml/2014/9/23/926
>>
> 
> Thanks,
> 
> Also, what's the rules for a test that requires other tools. Some of my
> tests require trace-cmd to be installed as well as perf. My stress test
> runs perf against trace-cmd running hackbench (another required tool),
> but I believe that perf has hackbench built in so I could replace
> hackbench with the perf version.
> 

One simple approach is to have the user install the required tools
for the test. Indicate what is needed and fail the test with
appropriate message with xskip. This is what I have in mind for
tests that can't be run because of a missing config. The same can
be extended to external tools that the test depends on.

thanks,
-- Shuah


-- 
Shuah Khan
Sr. Linux Kernel Developer
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Steven Rostedt]

On Wed, 24 Sep 2014 10:22:12 -0600
Shuah Khan <shuahkh@osg.samsung.com> wrote:


> One simple approach is to have the user install the required tools
> for the test. Indicate what is needed and fail the test with
> appropriate message with xskip. This is what I have in mind for
> tests that can't be run because of a missing config. The same can
> be extended to external tools that the test depends on.

Should I hold off adding my tests until we have all this in place. Or
should I just push my tests with simply passing if the config isn't
supported for now and add it later?

I wont add any of the tests that require other tools besides echo, grep
and cat.

-- Steve

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Shuah Khan]

On 09/24/2014 10:30 AM, Steven Rostedt wrote:
> On Wed, 24 Sep 2014 10:22:12 -0600
> Shuah Khan <shuahkh@osg.samsung.com> wrote:
> 
> 
>> One simple approach is to have the user install the required tools
>> for the test. Indicate what is needed and fail the test with
>> appropriate message with xskip. This is what I have in mind for
>> tests that can't be run because of a missing config. The same can
>> be extended to external tools that the test depends on.
> 
> Should I hold off adding my tests until we have all this in place. Or
> should I just push my tests with simply passing if the config isn't
> supported for now and add it later?
> 
> I wont add any of the tests that require other tools besides echo, grep
> and cat.
> 

Please go ahead and get them in. The framework might have to wait
until 3.19, so there is no need to have it gate test development.
We can evolve them as we go. If tests require special, it means that
they aren't suitable for certain envs such as qemu boot with busybox.
There is a plan to add test targets to cover such cases.

I would recommend at least printing a message saying what you are
missing if you find something missing.

thanks,
-- Shuah


-- 
Shuah Khan
Sr. Linux Kernel Developer
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

Re: Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Masami Hiramatsu]

(2014/09/25 0:58), Steven Rostedt wrote:
> On Wed, 24 Sep 2014 11:42:35 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
>>> I think you'd better use exit_unsupported/exit_xfail to notify
>>> that the test target is not configured, or expected to fail.
>>
> 
> Are these functions defined anywhere? I don't seem them in your v5
> patch set, or in selftests anywhere.

Hmm, could you check my 4/4 patch in the series??
http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg730578.html

Thanks,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Steven Rostedt]

On Thu, 25 Sep 2014 09:35:05 +0900
Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:

> (2014/09/25 0:58), Steven Rostedt wrote:
> > On Wed, 24 Sep 2014 11:42:35 -0400
> > Steven Rostedt <rostedt@goodmis.org> wrote:
> > 
> >>> I think you'd better use exit_unsupported/exit_xfail to notify
> >>> that the test target is not configured, or expected to fail.
> >>
> > 
> > Are these functions defined anywhere? I don't seem them in your v5
> > patch set, or in selftests anywhere.
> 
> Hmm, could you check my 4/4 patch in the series??
> http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg730578.html

Hmm, I somehow missed this. Can you check to make sure my for-next
patches are your latest version. I'll pull in 4/4 tomorrow.

Thanks,

-- Steve

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Masami Hiramatsu]

(2014/09/25 0:42), Steven Rostedt wrote:
> On Wed, 24 Sep 2014 11:58:50 +0900
> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
> 
>> (2014/09/24 6:38), Steven Rostedt wrote:
>  
>>>   to them and copied them pretty much unchanged into a ftrace directory
>>>   under test.d. Is this fine, or is there more massaging I need to do
>>>   to them?
>>
>> Yeah, ftrace has a double meaning :), so test.d/ftrace is fine to me.
> 
> I was thinking that more complex tests can go into ftrace, and simple
> tests can go into basic.

Hmm, what kind of "complex" test will be there?
I think unit tests are usually simple, selftests/ftrace/test.d/ftrace/
looks redundant if the first ftrace and the second one has same meaning.

>>>   I know the echos don't show up, but I kept them anyway. What should
>>>   happen with them?
>>
>> I think you'd better use exit_unsupported/exit_xfail to notify
>> that the test target is not configured, or expected to fail.
> 
> OK, that answers the exit codes (as you also stated below), but what
> about the echos themselves?

Again, ftracetest is for unit tests, exit code will return one
concrete result, but we can anything with echo (including typo)

> 
>> Then the user can reconfigure that. Maybe we should keep the
>> detailed log of such results. (you can do it with --keep option)
> 
> You mean keep the echos, as they are ignored anyway, but if we add
> --keep, then the echos will show?

No, --keep keeps all output logs of the test script. Since all scripts
run under set -x, all commands and outputs are logged to a logfile.

>  Maybe that option should be -v, like
> other tools use for "verbose".

OK, it will be easy, just turning "> $LOGFILE" into "| tee $LOGFILE" :)

>>> --- /dev/null
>>> +++ b/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter.tc
>>> @@ -0,0 +1,111 @@
>>> +#!/bin/sh
>>> +# description: ftrace - function graph filters
>>> +
>>> +# Make sure that function graph filtering works, and is not
>>> +# affected by other tracers enabled (like stack tracer)
>>> +
>>> +if ! grep -q function_graph available_tracers; then
>>> +    echo "no function graph tracer configured"
>>> +    exit 0;
>>
>> This should call exit_unsupported, because the test is not passed.
> 
> Will update this and others.
> 
> 
>>> +# Make sure we did find something
>>> +count=`cat trace | grep 'schedule()' | wc -l` 
>>> +if [ $count -eq 0 ]; then
>>> +    echo "No schedule traces found?"
>>> +    exit -1
>>> +fi
>>> +
>>> +echo "Graph filtering works by itself"
>>> +
>>
>> I think the following part should be a separated test for
>> stack trace.
> 
> Good point. A lot of my test scripts do multiple tests. I think for
> putting them into tools/testing/selftests/ftrace, I'll break them up
> and make them separate tests. For example, the simple graph filtering
> above is an example of something that can go into the basic directory,
> but the test against stack tracer should be in ftrace. What do you
> think?

Hm, I think only really basic operation tests go into the basic
directory, like reading and writing (no format checks).
And test.d/ftrace may be for more advanced tests, like checking
output, setting options, etc.

>>> +
>>> +echo "Now filter on just schedule"
>>> +echo '*schedule' > set_ftrace_filter
>>> +> trace
>>
>> echo > trace?
> 
> OK. Hmm, I wonder if we should make a bunch of functions that the tests
> can use. Like a "clear_trace" call that does this. Can we export
> functions that this shell will be able to use?

Yeah, it's a good idea :) We can use "." to include it.
I think test.d/functions will be good.

> When we see lots of
> duplicate code we may want to have something like that. Well, for this,
> it may not be that big of a deal, because "echo > trace" is rather
> simple. But I do have other operations that are a bit more intrusive.

Agreed.

Thank you!

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Masami Hiramatsu]

(2014/09/25 10:14), Steven Rostedt wrote:
> On Thu, 25 Sep 2014 09:35:05 +0900
> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
> 
>> (2014/09/25 0:58), Steven Rostedt wrote:
>>> On Wed, 24 Sep 2014 11:42:35 -0400
>>> Steven Rostedt <rostedt@goodmis.org> wrote:
>>>
>>>>> I think you'd better use exit_unsupported/exit_xfail to notify
>>>>> that the test target is not configured, or expected to fail.
>>>>
>>>
>>> Are these functions defined anywhere? I don't seem them in your v5
>>> patch set, or in selftests anywhere.
>>
>> Hmm, could you check my 4/4 patch in the series??
>> http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg730578.html
> 
> Hmm, I somehow missed this. Can you check to make sure my for-next
> patches are your latest version. I'll pull in 4/4 tomorrow.

Yeah, I ensured that your for-next patches are the v5. You will be
able to cleanly apply 4/4 on it :)

Thank you,

> 
> Thanks,
> 
> -- Steve
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Namhyung Kim]

Hi Steve,

On Wed, 24 Sep 2014 12:08:53 -0400, Steven Rostedt wrote:
> On Wed, 24 Sep 2014 10:01:38 -0600
> Shuah Khan <shuahkh@osg.samsung.com> wrote:
>
>> I am working on adding the framework for c tests. We will have
>> to get this evolved for shell scripts. Please see the following
>> thread:
>> 
>> https://lkml.org/lkml/2014/9/23/926
>> 
>
> Thanks,
>
> Also, what's the rules for a test that requires other tools. Some of my
> tests require trace-cmd to be installed as well as perf. My stress test
> runs perf against trace-cmd running hackbench (another required tool),
> but I believe that perf has hackbench built in so I could replace
> hackbench with the perf version.

Yeah, and I think we can at least build perf (if not found) for test as
we can always have the source.  The trace-cmd is different and it'd be
another reason I need to work on 'perf ftrace' command soon.  ;-)

Thanks,
Namhyung

Re: Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Masami Hiramatsu]

(2014/09/26 15:06), Namhyung Kim wrote:
> Hi Steve,
> 
> On Wed, 24 Sep 2014 12:08:53 -0400, Steven Rostedt wrote:
>> On Wed, 24 Sep 2014 10:01:38 -0600
>> Shuah Khan <shuahkh@osg.samsung.com> wrote:
>>
>>> I am working on adding the framework for c tests. We will have
>>> to get this evolved for shell scripts. Please see the following
>>> thread:
>>>
>>> https://lkml.org/lkml/2014/9/23/926
>>>
>>
>> Thanks,
>>
>> Also, what's the rules for a test that requires other tools. Some of my
>> tests require trace-cmd to be installed as well as perf. My stress test
>> runs perf against trace-cmd running hackbench (another required tool),
>> but I believe that perf has hackbench built in so I could replace
>> hackbench with the perf version.
> 
> Yeah, and I think we can at least build perf (if not found) for test as
> we can always have the source.  The trace-cmd is different and it'd be
> another reason I need to work on 'perf ftrace' command soon.  ;-)

+1 :)

For the kselftest, we can only expect to have tools in kernel tree (which
should be static-linked) and busybox, so that we can use qemu emulator to
run kselftest on other archs. I guess perf still need some shared
objects to run. That should be solved too.

Anyway, at first we'd better add simpler tests which is not relay on
other tools except busybox shell.

Thank you,


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Masami Hiramatsu]

(2014/09/25 1:36), Shuah Khan wrote:
> On 09/24/2014 10:30 AM, Steven Rostedt wrote:
>> On Wed, 24 Sep 2014 10:22:12 -0600
>> Shuah Khan <shuahkh@osg.samsung.com> wrote:
>>
>>
>>> One simple approach is to have the user install the required tools
>>> for the test. Indicate what is needed and fail the test with
>>> appropriate message with xskip. This is what I have in mind for
>>> tests that can't be run because of a missing config. The same can
>>> be extended to external tools that the test depends on.
>>
>> Should I hold off adding my tests until we have all this in place. Or
>> should I just push my tests with simply passing if the config isn't
>> supported for now and add it later?
>>
>> I wont add any of the tests that require other tools besides echo, grep
>> and cat.

They are supported by busybox :)

> Please go ahead and get them in. The framework might have to wait
> until 3.19, so there is no need to have it gate test development.
> We can evolve them as we go. If tests require special, it means that
> they aren't suitable for certain envs such as qemu boot with busybox.
> There is a plan to add test targets to cover such cases.

Agreed, anyway, in some env those tools(commands) may not supported
by busybox because of the memory size etc. In that case, we can just
mark the test unsupported or xfail.

Thank you,

> I would recommend at least printing a message saying what you are
> missing if you find something missing.
> 
> thanks,
> -- Shuah
> 
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Steven Rostedt]

On Fri, 26 Sep 2014 15:06:51 +0900
Namhyung Kim <namhyung@kernel.org> wrote:


> Yeah, and I think we can at least build perf (if not found) for test as
> we can always have the source.  The trace-cmd is different and it'd be
> another reason I need to work on 'perf ftrace' command soon.  ;-)

Well, if a tool is not found. I wouldn't recommend building it. Either
case, I'll hold off test that is required.

Hmm, maybe I'll post all my tests on my website, for others to play
with. That way I'm not limited in tests that take too much time or use
tools not supported by the kernel selftest infrastructure.

-- Steve

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Steven Rostedt]

On Fri, 26 Sep 2014 21:04:26 +0900
Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:


> > Please go ahead and get them in. The framework might have to wait
> > until 3.19, so there is no need to have it gate test development.
> > We can evolve them as we go. If tests require special, it means that
> > they aren't suitable for certain envs such as qemu boot with busybox.
> > There is a plan to add test targets to cover such cases.
> 
> Agreed, anyway, in some env those tools(commands) may not supported
> by busybox because of the memory size etc. In that case, we can just
> mark the test unsupported or xfail.
> 

Note, if memory size is such a big issue it that other tools may not be
there, then most of the ftrace features will probably be disabled due
to memory constraints on the kernel itself.

As we have a way to state "unsupported" we can just return that from
tests that require tools not available, and say why it is the case.

-- Steve

Re: Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Namhyung Kim]

Hi Masami,

2014-09-26 (금), 20:54 +0900, Masami Hiramatsu:
> (2014/09/26 15:06), Namhyung Kim wrote:
> > Hi Steve,
> > 
> > On Wed, 24 Sep 2014 12:08:53 -0400, Steven Rostedt wrote:
> >> On Wed, 24 Sep 2014 10:01:38 -0600
> >> Shuah Khan <shuahkh@osg.samsung.com> wrote:
> >>
> >>> I am working on adding the framework for c tests. We will have
> >>> to get this evolved for shell scripts. Please see the following
> >>> thread:
> >>>
> >>> https://lkml.org/lkml/2014/9/23/926
> >>>
> >>
> >> Thanks,
> >>
> >> Also, what's the rules for a test that requires other tools. Some of my
> >> tests require trace-cmd to be installed as well as perf. My stress test
> >> runs perf against trace-cmd running hackbench (another required tool),
> >> but I believe that perf has hackbench built in so I could replace
> >> hackbench with the perf version.
> > 
> > Yeah, and I think we can at least build perf (if not found) for test as
> > we can always have the source.  The trace-cmd is different and it'd be
> > another reason I need to work on 'perf ftrace' command soon.  ;-)
> 
> +1 :)
> 
> For the kselftest, we can only expect to have tools in kernel tree (which
> should be static-linked) and busybox, so that we can use qemu emulator to
> run kselftest on other archs. I guess perf still need some shared
> objects to run. That should be solved too.

The perf should be built with no extra dependencies (other than libc).
It won't provide rich features but basic functionality (record, report,
stat) should be supported.  And I'm also thinking about implementing a
very simple elf parser for symbol loading in such a minimal build.

Thanks,
Namhyung


> 
> Anyway, at first we'd better add simpler tests which is not relay on
> other tools except busybox shell.
> 
> Thank you,
> 
> 

Re: Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Masami Hiramatsu]

(2014/09/26 23:18), Steven Rostedt wrote:
> On Fri, 26 Sep 2014 21:04:26 +0900
> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
> 
> 
>>> Please go ahead and get them in. The framework might have to wait
>>> until 3.19, so there is no need to have it gate test development.
>>> We can evolve them as we go. If tests require special, it means that
>>> they aren't suitable for certain envs such as qemu boot with busybox.
>>> There is a plan to add test targets to cover such cases.
>>
>> Agreed, anyway, in some env those tools(commands) may not supported
>> by busybox because of the memory size etc. In that case, we can just
>> mark the test unsupported or xfail.
>>
> 
> Note, if memory size is such a big issue it that other tools may not be
> there, then most of the ftrace features will probably be disabled due
> to memory constraints on the kernel itself.

Agreed :)

> 
> As we have a way to state "unsupported" we can just return that from
> tests that require tools not available, and say why it is the case.

OK, if we can count "unsupported" as a test failure (yes, it is
unexpectedly failed), it is easy to fix. I'll update XFAIL and
other error code support patch and send it.

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Re: Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Masami Hiramatsu]

(2014/09/26 23:46), Namhyung Kim wrote:
> Hi Masami,
> 
> 2014-09-26 (금), 20:54 +0900, Masami Hiramatsu:
>> (2014/09/26 15:06), Namhyung Kim wrote:
>>> Hi Steve,
>>>
>>> On Wed, 24 Sep 2014 12:08:53 -0400, Steven Rostedt wrote:
>>>> On Wed, 24 Sep 2014 10:01:38 -0600
>>>> Shuah Khan <shuahkh@osg.samsung.com> wrote:
>>>>
>>>>> I am working on adding the framework for c tests. We will have
>>>>> to get this evolved for shell scripts. Please see the following
>>>>> thread:
>>>>>
>>>>> https://lkml.org/lkml/2014/9/23/926
>>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Also, what's the rules for a test that requires other tools. Some of my
>>>> tests require trace-cmd to be installed as well as perf. My stress test
>>>> runs perf against trace-cmd running hackbench (another required tool),
>>>> but I believe that perf has hackbench built in so I could replace
>>>> hackbench with the perf version.
>>>
>>> Yeah, and I think we can at least build perf (if not found) for test as
>>> we can always have the source.  The trace-cmd is different and it'd be
>>> another reason I need to work on 'perf ftrace' command soon.  ;-)
>>
>> +1 :)
>>
>> For the kselftest, we can only expect to have tools in kernel tree (which
>> should be static-linked) and busybox, so that we can use qemu emulator to
>> run kselftest on other archs. I guess perf still need some shared
>> objects to run. That should be solved too.
> 
> The perf should be built with no extra dependencies (other than libc).
> It won't provide rich features but basic functionality (record, report,
> stat) should be supported.  And I'm also thinking about implementing a
> very simple elf parser for symbol loading in such a minimal build.

Maybe you can install some static libraries (e.g. elfutils-libelf-devel-static
on fedora) for enabling it. :)
Some libraries may not work, but I could enable libelf by installing that.

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [RFC][PATCH] ftracetest: Add a couple of ftrace test cases

[Namhyung Kim]

Hi Masami,

On Mon, 29 Sep 2014 20:40:51 +0900, Masami Hiramatsu wrote:
> (2014/09/26 23:46), Namhyung Kim wrote:
>> The perf should be built with no extra dependencies (other than libc).
>> It won't provide rich features but basic functionality (record, report,
>> stat) should be supported.  And I'm also thinking about implementing a
>> very simple elf parser for symbol loading in such a minimal build.
>
> Maybe you can install some static libraries (e.g. elfutils-libelf-devel-static
> on fedora) for enabling it. :)
> Some libraries may not work, but I could enable libelf by installing that.

Yep, installing static libs should also work.  But I'd like to have
minimal depencency especially for embedded systems.  And AFAIK cross
building elfutils/libelf always has been a PITA.

Thanks,
Namhyung