[RFC PATCH v2 0/4] ftrace: Add a ftrace test collection

[RFC PATCH v2 0/4] ftrace: Add a ftrace test collection

[Masami Hiramatsu]

Hi,

Here is the collection of testcases for ftrace version 2.
In this version, I just updated the test framework according
to some discussions on kernel summit testing unconference.

The major updates are:
 - Using dash (busybox shell) instead of bash, because it
   allows us to test on the minimal environments running on 
   cross-arch emulators.
 - Support XFAIL/XPASS/UNSUPPORTED results for testcase.
   Some of ftrace features depend on kconfig and some of
   them do not work by environmental reason.
 - Support partial testing under given directory.

ftracetest is a small dash script for testing ftrace features.
It will be required to add a unit test for each new feature
after this is merged, because of preventing feature regressions.

This version still be out of the selftest. However, at the kernel
summit, I talked with Shuah about that. And since the kselftest
will be the standard kernel unit test framework, I agreed to
move this under the selftest. But I also would like to ask Steven
and Namhyung to agree with it before moving.

Actually, we've already has similar shell-script base testcase
under kselftest, see tools/testing/selftests/sysctl/Makefile.
I think we can mimic it for ftracetest :)

Thank you,

---

Masami Hiramatsu (4):
      ftracetest: Initial commit for ftracetest
      ftracetest: Add ftrace basic testcases
      ftracetest: Add kprobe basic testcases
      ftracetest: Add XFAIL/XPASS/UNSUPPORTED as result code


 tools/testing/ftrace/README                        |   45 +++++
 tools/testing/ftrace/ftracetest                    |  185 ++++++++++++++++++++
 tools/testing/ftrace/test.d/basic1.tc              |    7 +
 tools/testing/ftrace/test.d/basic2.tc              |    8 +
 tools/testing/ftrace/test.d/basic3.tc              |    9 +
 .../testing/ftrace/test.d/kprobe/add_and_remove.tc |   12 +
 tools/testing/ftrace/test.d/kprobe/busy_check.tc   |   14 ++
 tools/testing/ftrace/test.d/template               |    8 +
 8 files changed, 288 insertions(+)
 create mode 100644 tools/testing/ftrace/README
 create mode 100755 tools/testing/ftrace/ftracetest
 create mode 100644 tools/testing/ftrace/test.d/basic1.tc
 create mode 100644 tools/testing/ftrace/test.d/basic2.tc
 create mode 100644 tools/testing/ftrace/test.d/basic3.tc
 create mode 100644 tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
 create mode 100644 tools/testing/ftrace/test.d/kprobe/busy_check.tc
 create mode 100644 tools/testing/ftrace/test.d/template

--

[RFC PATCH v2 1/4] ftracetest: Initial commit for ftracetest

[Masami Hiramatsu]

ftracetest is a collection of testcase shell-scripts for ftrace.
To avoid regressions of ftrace, these testcases check correct
ftrace behaviors. If someone would like to add any features on
ftrace, the patch series should have at least one testcase for
checking the new behavior.

Changes in v2:
 - Remove unneeded 'function' keyword.
 - Add abspath and find_testcases.
 - Make OPT_TEST_CASES a local var.
 - Ensure given testcase ended with .tc.
 - Accept a directory option which has multiple testcases.
 - Change [PASSED]/[FAILED] to [PASS]/[FAIL]
 - Change the basic shell to sh (dash).

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/testing/ftrace/README          |   45 ++++++++++
 tools/testing/ftrace/ftracetest      |  152 ++++++++++++++++++++++++++++++++++
 tools/testing/ftrace/test.d/template |    4 +
 3 files changed, 201 insertions(+)
 create mode 100644 tools/testing/ftrace/README
 create mode 100755 tools/testing/ftrace/ftracetest
 create mode 100644 tools/testing/ftrace/test.d/template

diff --git a/tools/testing/ftrace/README b/tools/testing/ftrace/README
new file mode 100644
index 0000000..5d5de16
--- /dev/null
+++ b/tools/testing/ftrace/README
@@ -0,0 +1,45 @@
+Linux Ftrace Testcases
+
+This is a collection of testcases for ftrace tracing feature in the Linux
+kernel. Since ftrace exports interfaces via the debugfs, we just need
+shell scripts for testing. Feel free to add new test cases.
+
+Running the ftrace testcases
+============================
+
+At first, you need to be the root user to run this script.
+To run all testcases:
+
+  $ sudo ./ftracetest
+
+To run specific testcases:
+
+  # ./ftracetest test.d/basic3.tc
+
+Or you can also run testcases under given directory:
+
+  # ./ftracetest test.d/kprobe/
+
+Contributing new testcases
+==========================
+
+Copy test.d/template to your testcase (whose filename must has *.tc extension)
+and rewrite test description line.
+
+ * The working directory of the script is <debugfs>/tracing/.
+
+ * Take care about the side effect, because that is run with root privilege.
+
+ * Don't take too long. It's a kind of unit test.
+
+ * You can add a directory for your testcases under test.d/ if needed.
+
+ * The test cases should run on dash (busybox shell) for testing on
+   minimal cross-build environments.
+
+TODO
+====
+
+ * Fancy colored output :)
+
+ * Integrate with selftest?
diff --git a/tools/testing/ftrace/ftracetest b/tools/testing/ftrace/ftracetest
new file mode 100755
index 0000000..0378c8a
--- /dev/null
+++ b/tools/testing/ftrace/ftracetest
@@ -0,0 +1,152 @@
+#!/bin/sh
+# ftracetest - Ftrace test shell scripts
+# Written by Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
+
+usage() { # errno [message]
+[ "$2" ] && echo $2
+echo "Usage: ftracetest [options] [testcase(s)] [testcase-directory(s)]"
+echo " Options:"
+echo "		-h|--help  Show help message"
+echo "		-k|--keep  Keep passed test logs"
+echo "		-d|--debug Debug mode (trace all shell commands)"
+exit $1
+}
+
+errexit() { # message
+  echo "Error: $1" 1>&2
+  exit 1
+}
+
+# Ensuring user privilege
+if [ `id -u` -ne 0 ]; then
+  errexit "this must be run by root user"
+fi
+
+# Utilities
+absdir() { # file_path
+  (cd `dirname $1`; pwd)
+}
+
+abspath() {
+  echo `absdir $1`/`basename $1`
+}
+
+find_testcases() { #directory
+  echo `find $1 -name \*.tc`
+}
+
+parse_opts() { # opts
+  local OPT_TEST_CASES=
+  local OPT_TEST_DIR=
+
+  while [ "$1" ]; do
+    case "$1" in
+    --help|-h)
+      usage 0
+    ;;
+    --keep|-k)
+      KEEP_LOG=1
+      shift 1
+    ;;
+    --debug|-d)
+      DEBUG=1
+      shift 1
+    ;;
+    *.tc)
+      if [ -f "$1" ]; then
+        OPT_TEST_CASES="$OPT_TEST_CASES `abspath $1`"
+        shift 1
+      else
+        usage 1 "$1 is not a testcase"
+      fi
+      ;;
+    *)
+      if [ -d "$1" ]; then
+        OPT_TEST_DIR=`abspath $1`
+        OPT_TEST_CASES="$OPT_TEST_CASES `find_testcases $OPT_TEST_DIR`"
+        shift 1
+      else
+        usage 1 "Invalid option ($1)"
+      fi
+    ;;
+    esac
+  done
+  if [ "$OPT_TEST_CASES" ]; then
+    TEST_CASES=$OPT_TEST_CASES
+  fi
+}
+
+# Parameters
+DEBUGFS_DIR=`grep debugfs /proc/mounts | cut -f2 -d' '`
+TRACING_DIR=$DEBUGFS_DIR/tracing
+TOP_DIR=`absdir $0`
+TEST_DIR=$TOP_DIR/test.d
+TEST_CASES=`find_testcases $TEST_DIR`
+LOG_DIR=$TOP_DIR/logs/`date +%Y%m%d-%H%M%S`/
+KEEP_LOG=0
+DEBUG=0
+# Parse command-line options
+parse_opts $*
+
+[ $DEBUG -ne 0 ] && set -x
+
+# Verify parameters
+if [ -z "$DEBUGFS_DIR" -o ! -d "$TRACING_DIR" ]; then
+  errexit "No ftrace directory found"
+fi
+
+# Preparing logs
+LOG_FILE=$LOG_DIR/ftracetest.log
+mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR"
+date > $LOG_FILE
+prlog() { # messages
+  echo $@ | tee -a $LOG_FILE
+}
+catlog() { #file
+  cat $1 | tee -a $LOG_FILE
+}
+
+# Testcase management
+PASSED_CASES=
+FAILED_CASES=
+CASENO=0
+testcase() { # testfile
+  CASENO=$((CASENO+1))
+  prlog -n "[$CASENO]"`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
+}
+failed() {
+  prlog -e "\t[FAIL]"
+  FAILED_CASES="$FAILED_CASES $CASENO"
+}
+passed() {
+  prlog -e "\t[PASS]"
+  PASSED_CASES="$PASSED_CASES $CASENO"
+}
+
+
+# Run one test case
+run_test() { # testfile
+  local testname=`basename $1`
+  local testlog=`mktemp --tmpdir=$LOG_DIR ${testname}-XXXXXX.log`
+  testcase $1
+  echo "execute: "$1 > $testlog
+  (cd $TRACING_DIR; set -x ; source $t) >> $testlog 2>&1
+  ret=$?
+  if [ $ret -ne 0 ]; then
+    failed
+    catlog $testlog
+  else
+    passed
+    [ $KEEP_LOG -eq 0 ] && rm $testlog
+  fi
+}
+
+# Main loop
+for t in $TEST_CASES; do
+  run_test $t
+done
+prlog ""
+prlog "# of passed: " `echo $PASSED_CASES | wc -w`
+prlog "# of failed: " `echo $FAILED_CASES | wc -w`
+
+test -z "$FAILED_CASES" # if no error, return 0
diff --git a/tools/testing/ftrace/test.d/template b/tools/testing/ftrace/test.d/template
new file mode 100644
index 0000000..ce5f735
--- /dev/null
+++ b/tools/testing/ftrace/test.d/template
@@ -0,0 +1,4 @@
+#!/bin/sh
+# description: %HERE DESCRIBE WHAT THIS DOES%
+# you have to add ".tc" extention for your testcase file
+exit 0 # Return 0 if the test is passed, otherwise return !0

Re: [RFC PATCH v2 1/4] ftracetest: Initial commit for ftracetest

[Luis Henriques]

Hi,

Not really a complete review, but just 2 comments on this script:

On Tue, Aug 26, 2014 at 11:15:18AM +0000, Masami Hiramatsu wrote:
...
> +prlog() { # messages
> +  echo $@ | tee -a $LOG_FILE
> +}
> +catlog() { #file
> +  cat $1 | tee -a $LOG_FILE
> +}
> +
> +# Testcase management
> +PASSED_CASES=
> +FAILED_CASES=
> +CASENO=0
> +testcase() { # testfile
> +  CASENO=$((CASENO+1))
> +  prlog -n "[$CASENO]"`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
> +}
> +failed() {
> +  prlog -e "\t[FAIL]"
> +  FAILED_CASES="$FAILED_CASES $CASENO"
> +}
> +passed() {
> +  prlog -e "\t[PASS]"
> +  PASSED_CASES="$PASSED_CASES $CASENO"
> +}

What I see here is a '-e' being echo'ed and not really a '-e' switch
being used to 'echo'.  (Also, I'm not sure if this is a standard
switch...).

This applies to all the other 'prlog -e'.

> +
> +
> +# Run one test case
> +run_test() { # testfile
> +  local testname=`basename $1`
> +  local testlog=`mktemp --tmpdir=$LOG_DIR ${testname}-XXXXXX.log`
> +  testcase $1
> +  echo "execute: "$1 > $testlog
> +  (cd $TRACING_DIR; set -x ; source $t) >> $testlog 2>&1
> +  ret=$?

I believe the usage of 'source' is a bashism, and '.' should be used
instead.  In my environment, 'source' results in ret=127.  Replacing
it by '.' fixes it.

Cheers,
--
Luís

Re: [RFC PATCH v2 1/4] ftracetest: Initial commit for ftracetest

[Masami Hiramatsu]

(2014/08/28 0:54), Luis Henriques wrote:
> Hi,
> 
> Not really a complete review, but just 2 comments on this script:
> 
> On Tue, Aug 26, 2014 at 11:15:18AM +0000, Masami Hiramatsu wrote:
> ...
>> +prlog() { # messages
>> +  echo $@ | tee -a $LOG_FILE
>> +}
>> +catlog() { #file
>> +  cat $1 | tee -a $LOG_FILE
>> +}
>> +
>> +# Testcase management
>> +PASSED_CASES=
>> +FAILED_CASES=
>> +CASENO=0
>> +testcase() { # testfile
>> +  CASENO=$((CASENO+1))
>> +  prlog -n "[$CASENO]"`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
>> +}
>> +failed() {
>> +  prlog -e "\t[FAIL]"
>> +  FAILED_CASES="$FAILED_CASES $CASENO"
>> +}
>> +passed() {
>> +  prlog -e "\t[PASS]"
>> +  PASSED_CASES="$PASSED_CASES $CASENO"
>> +}
> 
> What I see here is a '-e' being echo'ed and not really a '-e' switch
> being used to 'echo'.  (Also, I'm not sure if this is a standard
> switch...).
> 
> This applies to all the other 'prlog -e'.

Oh, really? what shell did you use?
My target shell is the busybox and I've tested it on fedora20.

e.g. busybox echo command seems accept -e.
  $ busybox echo -e '\tfoo'
  	foo

Of course maybe I'd better not use \t, but "	"...


>> +
>> +
>> +# Run one test case
>> +run_test() { # testfile
>> +  local testname=`basename $1`
>> +  local testlog=`mktemp --tmpdir=$LOG_DIR ${testname}-XXXXXX.log`
>> +  testcase $1
>> +  echo "execute: "$1 > $testlog
>> +  (cd $TRACING_DIR; set -x ; source $t) >> $testlog 2>&1
>> +  ret=$?
> 
> I believe the usage of 'source' is a bashism, and '.' should be used
> instead.  In my environment, 'source' results in ret=127.  Replacing
> it by '.' fixes it.

Ah, right. I missed that, I'll fix that :)

Thank you,


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [RFC PATCH v2 1/4] ftracetest: Initial commit for ftracetest

[Luis Henriques]

On Thu, Aug 28, 2014 at 05:40:22PM +0900, Masami Hiramatsu wrote:
> (2014/08/28 0:54), Luis Henriques wrote:
> > Hi,
> > 
> > Not really a complete review, but just 2 comments on this script:
> > 
> > On Tue, Aug 26, 2014 at 11:15:18AM +0000, Masami Hiramatsu wrote:
> > ...
> >> +prlog() { # messages
> >> +  echo $@ | tee -a $LOG_FILE
> >> +}
> >> +catlog() { #file
> >> +  cat $1 | tee -a $LOG_FILE
> >> +}
> >> +
> >> +# Testcase management
> >> +PASSED_CASES=
> >> +FAILED_CASES=
> >> +CASENO=0
> >> +testcase() { # testfile
> >> +  CASENO=$((CASENO+1))
> >> +  prlog -n "[$CASENO]"`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
> >> +}
> >> +failed() {
> >> +  prlog -e "\t[FAIL]"
> >> +  FAILED_CASES="$FAILED_CASES $CASENO"
> >> +}
> >> +passed() {
> >> +  prlog -e "\t[PASS]"
> >> +  PASSED_CASES="$PASSED_CASES $CASENO"
> >> +}
> > 
> > What I see here is a '-e' being echo'ed and not really a '-e' switch
> > being used to 'echo'.  (Also, I'm not sure if this is a standard
> > switch...).
> > 
> > This applies to all the other 'prlog -e'.
> 
> Oh, really? what shell did you use?
> My target shell is the busybox and I've tested it on fedora20.
> 
> e.g. busybox echo command seems accept -e.
>   $ busybox echo -e '\tfoo'
>   	foo
> 
> Of course maybe I'd better not use \t, but "	"...
> 
> 

I've tested it with dash (version 0.5.7).

Anyway, I didn't investigated this any further but I'm sure we'll
enter the usual POSIX-compliance discussion. :-)

Cheers,
--
Luís

> >> +
> >> +
> >> +# Run one test case
> >> +run_test() { # testfile
> >> +  local testname=`basename $1`
> >> +  local testlog=`mktemp --tmpdir=$LOG_DIR ${testname}-XXXXXX.log`
> >> +  testcase $1
> >> +  echo "execute: "$1 > $testlog
> >> +  (cd $TRACING_DIR; set -x ; source $t) >> $testlog 2>&1
> >> +  ret=$?
> > 
> > I believe the usage of 'source' is a bashism, and '.' should be used
> > instead.  In my environment, 'source' results in ret=127.  Replacing
> > it by '.' fixes it.
> 
> Ah, right. I missed that, I'll fix that :)
> 
> Thank you,
> 
> 
> -- 
> Masami HIRAMATSU
> Software Platform Research Dept. Linux Technology Research Center
> Hitachi, Ltd., Yokohama Research Laboratory
> E-mail: masami.hiramatsu.pt@hitachi.com
> 
> 

[RFC PATCH v2 2/4] ftracetest: Add ftrace basic testcases

[Masami Hiramatsu]

Add ftrace basic testcases. This just checks ftrace debugfs
interface works as it is designed.

Changes in v2:
  - Change shell to sh instead of bash.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/testing/ftrace/test.d/basic1.tc |    3 +++
 tools/testing/ftrace/test.d/basic2.tc |    6 ++++++
 tools/testing/ftrace/test.d/basic3.tc |    8 ++++++++
 3 files changed, 17 insertions(+)
 create mode 100644 tools/testing/ftrace/test.d/basic1.tc
 create mode 100644 tools/testing/ftrace/test.d/basic2.tc
 create mode 100644 tools/testing/ftrace/test.d/basic3.tc

diff --git a/tools/testing/ftrace/test.d/basic1.tc b/tools/testing/ftrace/test.d/basic1.tc
new file mode 100644
index 0000000..9980ff1
--- /dev/null
+++ b/tools/testing/ftrace/test.d/basic1.tc
@@ -0,0 +1,3 @@
+#!/bin/sh
+# description: Basic trace file check
+test -f README -a -f trace -a -f tracing_on -a -f trace_pipe
diff --git a/tools/testing/ftrace/test.d/basic2.tc b/tools/testing/ftrace/test.d/basic2.tc
new file mode 100644
index 0000000..b04f30d
--- /dev/null
+++ b/tools/testing/ftrace/test.d/basic2.tc
@@ -0,0 +1,6 @@
+#!/bin/sh
+# description: Basic test for tracers
+for t in `cat available_tracers`; do
+  echo $t > current_tracer || exit 1
+done
+echo nop > current_tracer
diff --git a/tools/testing/ftrace/test.d/basic3.tc b/tools/testing/ftrace/test.d/basic3.tc
new file mode 100644
index 0000000..0c1a3a2
--- /dev/null
+++ b/tools/testing/ftrace/test.d/basic3.tc
@@ -0,0 +1,8 @@
+#!/bin/sh
+# description: Basic trace clock test
+[ -f trace_clock ] || exit 1
+for c in `cat trace_clock | tr  -d \[\]`; do
+  echo $c > trace_clock || exit 1
+  grep '\['$c'\]' trace_clock || exit 1
+done
+echo local > trace_clock

[RFC PATCH v2 3/4] ftracetest: Add kprobe basic testcases

[Masami Hiramatsu]

Add basic testcases for kprobe dynamic events.
This also shows that the ftracetest accepts sub-directory
for new testcases.

Changes in v2:
  - Change shell to sh instead of bash.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 .../testing/ftrace/test.d/kprobe/add_and_remove.tc |   11 +++++++++++
 tools/testing/ftrace/test.d/kprobe/busy_check.tc   |   14 ++++++++++++++
 2 files changed, 25 insertions(+)
 create mode 100644 tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
 create mode 100644 tools/testing/ftrace/test.d/kprobe/busy_check.tc

diff --git a/tools/testing/ftrace/test.d/kprobe/add_and_remove.tc b/tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
new file mode 100644
index 0000000..5ddfb47
--- /dev/null
+++ b/tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
@@ -0,0 +1,11 @@
+#!/bin/sh
+# description: Kprobe dynamic event - adding and removing
+
+[ -f kprobe_events ] || exit 1
+
+echo 0 > events/enable || exit 1
+echo > kprobe_events || exit 1
+echo p:myevent do_fork > kprobe_events || exit 1
+grep myevent kprobe_events || exit 1
+[ -d events/kprobes/myevent ] || exit 1
+echo > kprobe_events
diff --git a/tools/testing/ftrace/test.d/kprobe/busy_check.tc b/tools/testing/ftrace/test.d/kprobe/busy_check.tc
new file mode 100644
index 0000000..588fde97
--- /dev/null
+++ b/tools/testing/ftrace/test.d/kprobe/busy_check.tc
@@ -0,0 +1,14 @@
+#!/bin/sh
+# description: Kprobe dynamic event - busy event check
+
+[ -f kprobe_events ] || exit 1
+
+echo 0 > events/enable || exit 1
+echo > kprobe_events || exit 1
+echo p:myevent do_fork > kprobe_events || exit 1
+[ -d events/kprobes/myevent ] || exit 1
+echo 1 > events/kprobes/myevent/enable || exit 1
+echo > kprobe_events && exit 1 # this must fail
+echo 0 > events/kprobes/myevent/enable || exit 1
+echo > kprobe_events # this must succeed
+

[RFC PATCH v2 4/4] ftracetest: Add XFAIL/XPASS/UNSUPPORTED as result code

[Masami Hiramatsu]

Add XFAIL, XPASS and UNSUPPORTED as a result code. These are
used for the results that test case is expected to fail or
unsupported feature (by config).
This also introduces PASS/FAIL/XFAIL/XPASS/UNSUP result codes
for each testcase. Since the results are not binary, each
testcase must use these code to return the test result.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/testing/ftrace/ftracetest                    |   61 +++++++++++++++-----
 tools/testing/ftrace/test.d/basic1.tc              |    6 ++
 tools/testing/ftrace/test.d/basic2.tc              |    6 +-
 tools/testing/ftrace/test.d/basic3.tc              |    9 ++-
 .../testing/ftrace/test.d/kprobe/add_and_remove.tc |   15 +++--
 tools/testing/ftrace/test.d/kprobe/busy_check.tc   |   20 +++----
 tools/testing/ftrace/test.d/template               |    6 ++
 7 files changed, 84 insertions(+), 39 deletions(-)

diff --git a/tools/testing/ftrace/ftracetest b/tools/testing/ftrace/ftracetest
index 0378c8a..bfcd56a 100755
--- a/tools/testing/ftrace/ftracetest
+++ b/tools/testing/ftrace/ftracetest
@@ -107,22 +107,53 @@ catlog() { #file
 }
 
 # Testcase management
+# Test result codes
+PASS=0	# The test succeeded.
+FAIL=1	# The test failed, but was expected to succeed.
+XFAIL=2	# The test failed, and was expected to fail.
+XPASS=3	# The test succeeded, but was expected to fail.
+UNSUP=4 # The test failed because of lack of feature.
+# Accumulations
 PASSED_CASES=
 FAILED_CASES=
+XFAILED_CASES=
+XPASSED_CASES=
+UNSUPPORTED_CASES=
+
 CASENO=0
 testcase() { # testfile
   CASENO=$((CASENO+1))
   prlog -n "[$CASENO]"`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
 }
-failed() {
-  prlog -e "\t[FAIL]"
-  FAILED_CASES="$FAILED_CASES $CASENO"
-}
-passed() {
-  prlog -e "\t[PASS]"
-  PASSED_CASES="$PASSED_CASES $CASENO"
-}
 
+eval_result() { # retval testlog
+  case $1 in
+    $PASS)
+      prlog -e "\t[PASS]"
+      PASSED_CASES="$PASSED_CASES $CASENO"
+      return 0
+    ;;
+    $FAIL)
+      prlog -e "\t[FAIL]"
+      FAILED_CASES="$FAILED_CASES $CASENO"
+      return 1 # this is a bug
+    ;;
+    $XFAIL)
+      prlog -e "\t[XFAIL]"
+      XFAILED_CASES="$XFAILED_CASES $CASENO"
+      return 0
+    ;;
+    $XPASS)
+      prlog -e "\t[XPASS]"
+      XPASSED_CASES="$XPASSED_CASES $CASENO"
+      return 1 # this can be a bug
+    ;;
+    $UNSUP)
+      prlog -e "\t[UNSUPPORTED]"
+      UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO"
+      return 0 # this is not a bug
+  esac
+}
 
 # Run one test case
 run_test() { # testfile
@@ -132,11 +163,9 @@ run_test() { # testfile
   echo "execute: "$1 > $testlog
   (cd $TRACING_DIR; set -x ; source $t) >> $testlog 2>&1
   ret=$?
-  if [ $ret -ne 0 ]; then
-    failed
-    catlog $testlog
-  else
-    passed
+  eval_result $ret $testlog
+  if [ $? -eq 0 ]; then
+    # Remove test log if the test was done as it was expected.
     [ $KEEP_LOG -eq 0 ] && rm $testlog
   fi
 }
@@ -145,8 +174,12 @@ run_test() { # testfile
 for t in $TEST_CASES; do
   run_test $t
 done
+
 prlog ""
 prlog "# of passed: " `echo $PASSED_CASES | wc -w`
 prlog "# of failed: " `echo $FAILED_CASES | wc -w`
+prlog "# of xfailed: " `echo $XFAILED_CASES | wc -w`
+prlog "# of xpassed: " `echo $XPASSED_CASES | wc -w`
+prlog "# of unsupported: " `echo $UNSUPPORTED_CASES | wc -w`
 
-test -z "$FAILED_CASES" # if no error, return 0
+test -z "$FAILED_CASES$XPASSED_CASES" # if no error, return 0
diff --git a/tools/testing/ftrace/test.d/basic1.tc b/tools/testing/ftrace/test.d/basic1.tc
index 9980ff1..1e95209 100644
--- a/tools/testing/ftrace/test.d/basic1.tc
+++ b/tools/testing/ftrace/test.d/basic1.tc
@@ -1,3 +1,7 @@
 #!/bin/sh
 # description: Basic trace file check
-test -f README -a -f trace -a -f tracing_on -a -f trace_pipe
+if test -f README -a -f trace -a -f tracing_on -a -f trace_pipe; then
+  exit $PASS
+else
+  exit $FAIL
+fi
diff --git a/tools/testing/ftrace/test.d/basic2.tc b/tools/testing/ftrace/test.d/basic2.tc
index b04f30d..44949e9 100644
--- a/tools/testing/ftrace/test.d/basic2.tc
+++ b/tools/testing/ftrace/test.d/basic2.tc
@@ -1,6 +1,8 @@
 #!/bin/sh
 # description: Basic test for tracers
+test -f available_tracers || exit $FAIL # this is basic feature, must be there.
 for t in `cat available_tracers`; do
-  echo $t > current_tracer || exit 1
+  echo $t > current_tracer || exit $FAIL
 done
-echo nop > current_tracer
+echo nop > current_tracer || exit $FAIL
+exit $PASS
diff --git a/tools/testing/ftrace/test.d/basic3.tc b/tools/testing/ftrace/test.d/basic3.tc
index 0c1a3a2..7bc5a53 100644
--- a/tools/testing/ftrace/test.d/basic3.tc
+++ b/tools/testing/ftrace/test.d/basic3.tc
@@ -1,8 +1,9 @@
 #!/bin/sh
 # description: Basic trace clock test
-[ -f trace_clock ] || exit 1
+[ -f trace_clock ] || exit $FAIL # this is basic feature, must be there
 for c in `cat trace_clock | tr  -d \[\]`; do
-  echo $c > trace_clock || exit 1
-  grep '\['$c'\]' trace_clock || exit 1
+  echo $c > trace_clock || exit $FAIL
+  grep '\['$c'\]' trace_clock || exit $FAIL
 done
-echo local > trace_clock
+echo local > trace_clock || exit $FAIL
+exit $PASS
diff --git a/tools/testing/ftrace/test.d/kprobe/add_and_remove.tc b/tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
index 5ddfb47..fa3034b 100644
--- a/tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
+++ b/tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
@@ -1,11 +1,12 @@
 #!/bin/sh
 # description: Kprobe dynamic event - adding and removing
 
-[ -f kprobe_events ] || exit 1
+[ -f kprobe_events ] || exit $UNSUP # this is configurable
 
-echo 0 > events/enable || exit 1
-echo > kprobe_events || exit 1
-echo p:myevent do_fork > kprobe_events || exit 1
-grep myevent kprobe_events || exit 1
-[ -d events/kprobes/myevent ] || exit 1
-echo > kprobe_events
+echo 0 > events/enable || exit $FAIL
+echo > kprobe_events || exit $FAIL
+echo p:myevent do_fork > kprobe_events || exit $FAIL
+grep myevent kprobe_events || exit $FAIL
+[ -d events/kprobes/myevent ] || exit $FAIL
+echo > kprobe_events || exit $FAIL
+exit $PASS
diff --git a/tools/testing/ftrace/test.d/kprobe/busy_check.tc b/tools/testing/ftrace/test.d/kprobe/busy_check.tc
index 588fde97..6f47784 100644
--- a/tools/testing/ftrace/test.d/kprobe/busy_check.tc
+++ b/tools/testing/ftrace/test.d/kprobe/busy_check.tc
@@ -1,14 +1,14 @@
 #!/bin/sh
 # description: Kprobe dynamic event - busy event check
 
-[ -f kprobe_events ] || exit 1
-
-echo 0 > events/enable || exit 1
-echo > kprobe_events || exit 1
-echo p:myevent do_fork > kprobe_events || exit 1
-[ -d events/kprobes/myevent ] || exit 1
-echo 1 > events/kprobes/myevent/enable || exit 1
-echo > kprobe_events && exit 1 # this must fail
-echo 0 > events/kprobes/myevent/enable || exit 1
-echo > kprobe_events # this must succeed
+[ -f kprobe_events ] || exit $UNSUP
 
+echo 0 > events/enable || exit $FAIL
+echo > kprobe_events || exit $FAIL
+echo p:myevent do_fork > kprobe_events || exit $FAIL
+[ -d events/kprobes/myevent ] || exit $FAIL
+echo 1 > events/kprobes/myevent/enable || exit $FAIL
+echo > kprobe_events && exit $FAIL # this must fail
+echo 0 > events/kprobes/myevent/enable || exit $FAIL
+echo > kprobe_events || exit $FAIL # this must succeed
+exit $PASS
diff --git a/tools/testing/ftrace/test.d/template b/tools/testing/ftrace/test.d/template
index ce5f735..881eddd 100644
--- a/tools/testing/ftrace/test.d/template
+++ b/tools/testing/ftrace/test.d/template
@@ -1,4 +1,8 @@
 #!/bin/sh
 # description: %HERE DESCRIBE WHAT THIS DOES%
 # you have to add ".tc" extention for your testcase file
-exit 0 # Return 0 if the test is passed, otherwise return !0
+exit $PASS # Return $PASS if the test is passed.
+# If the test is failed, return $FAIL.
+# If the test is failed as expected, return $XFAIL.
+# If the test is passed, but it was expected to fail, return $XPASS.
+# If the test can not run because of lack of feature, return $UNSUP

Re: [RFC PATCH v2 4/4] ftracetest: Add XFAIL/XPASS/UNSUPPORTED as result code

[Shuah Khan]

On 08/26/2014 05:15 AM, Masami Hiramatsu wrote:
> Add XFAIL, XPASS and UNSUPPORTED as a result code. These are
> used for the results that test case is expected to fail or
> unsupported feature (by config).
> This also introduces PASS/FAIL/XFAIL/XPASS/UNSUP result codes
> for each testcase. Since the results are not binary, each
> testcase must use these code to return the test result.
> 
> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> ---
>  tools/testing/ftrace/ftracetest                    |   61 +++++++++++++++-----
>  tools/testing/ftrace/test.d/basic1.tc              |    6 ++
>  tools/testing/ftrace/test.d/basic2.tc              |    6 +-
>  tools/testing/ftrace/test.d/basic3.tc              |    9 ++-
>  .../testing/ftrace/test.d/kprobe/add_and_remove.tc |   15 +++--
>  tools/testing/ftrace/test.d/kprobe/busy_check.tc   |   20 +++----
>  tools/testing/ftrace/test.d/template               |    6 ++
>  7 files changed, 84 insertions(+), 39 deletions(-)
> 
> diff --git a/tools/testing/ftrace/ftracetest b/tools/testing/ftrace/ftracetest
> index 0378c8a..bfcd56a 100755
> --- a/tools/testing/ftrace/ftracetest
> +++ b/tools/testing/ftrace/ftracetest
> @@ -107,22 +107,53 @@ catlog() { #file
>  }
>  
>  # Testcase management
> +# Test result codes
> +PASS=0	# The test succeeded.
> +FAIL=1	# The test failed, but was expected to succeed.
> +XFAIL=2	# The test failed, and was expected to fail.
> +XPASS=3	# The test succeeded, but was expected to fail.

This one is confusing. It is still a failure. XFAIL case is
expected to fail, using that convention XPASS should be
expected to pass?

Can we use XFAIL0 for XFAIL pass case and XFAIL1 for XFAIL fail
case?

thanks,
-- Shuah


-- 
Shuah Khan
Sr. Linux Kernel Developer
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

Re: [RFC PATCH v2 4/4] ftracetest: Add XFAIL/XPASS/UNSUPPORTED as result code

[Masami Hiramatsu]

(2014/08/29 10:05), Shuah Khan wrote:
> On 08/26/2014 05:15 AM, Masami Hiramatsu wrote:
>> Add XFAIL, XPASS and UNSUPPORTED as a result code. These are
>> used for the results that test case is expected to fail or
>> unsupported feature (by config).
>> This also introduces PASS/FAIL/XFAIL/XPASS/UNSUP result codes
>> for each testcase. Since the results are not binary, each
>> testcase must use these code to return the test result.
>>
>> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
>> ---
>>  tools/testing/ftrace/ftracetest                    |   61 +++++++++++++++-----
>>  tools/testing/ftrace/test.d/basic1.tc              |    6 ++
>>  tools/testing/ftrace/test.d/basic2.tc              |    6 +-
>>  tools/testing/ftrace/test.d/basic3.tc              |    9 ++-
>>  .../testing/ftrace/test.d/kprobe/add_and_remove.tc |   15 +++--
>>  tools/testing/ftrace/test.d/kprobe/busy_check.tc   |   20 +++----
>>  tools/testing/ftrace/test.d/template               |    6 ++
>>  7 files changed, 84 insertions(+), 39 deletions(-)
>>
>> diff --git a/tools/testing/ftrace/ftracetest b/tools/testing/ftrace/ftracetest
>> index 0378c8a..bfcd56a 100755
>> --- a/tools/testing/ftrace/ftracetest
>> +++ b/tools/testing/ftrace/ftracetest
>> @@ -107,22 +107,53 @@ catlog() { #file
>>  }
>>  
>>  # Testcase management
>> +# Test result codes
>> +PASS=0	# The test succeeded.
>> +FAIL=1	# The test failed, but was expected to succeed.
>> +XFAIL=2	# The test failed, and was expected to fail.
>> +XPASS=3	# The test succeeded, but was expected to fail.
> 
> This one is confusing. It is still a failure. XFAIL case is
> expected to fail, using that convention XPASS should be
> expected to pass?

I see, but they have been already used as above by Dejagnu testing framework.
http://www.delorie.com/gnu/docs/dejagnu/dejagnu_6.html

Of course, it might be better to support only POSIX compatible results at
first.

> Can we use XFAIL0 for XFAIL pass case and XFAIL1 for XFAIL fail
> case?

No, new definition can confuse developers.
To avoid confusion, I'd like to use the same definitions as Dejegnu does.

Thank you,


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [RFC PATCH v2 4/4] ftracetest: Add XFAIL/XPASS/UNSUPPORTED as result code

[Masami Hiramatsu]

(2014/08/26 20:15), Masami Hiramatsu wrote:
> @@ -132,11 +163,9 @@ run_test() { # testfile
>    echo "execute: "$1 > $testlog
>    (cd $TRACING_DIR; set -x ; source $t) >> $testlog 2>&1
>    ret=$?
> -  if [ $ret -ne 0 ]; then
> -    failed
> -    catlog $testlog
> -  else
> -    passed
> +  eval_result $ret $testlog
> +  if [ $? -eq 0 ]; then
> +    # Remove test log if the test was done as it was expected.
>      [ $KEEP_LOG -eq 0 ] && rm $testlog
>    fi

Oops, I missed to print out the logs for failed tests here.
It should be output.

Thank you,

-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com