linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v4 0/4] ftrace: Add a ftrace test collection
@ 2014-09-18 12:04 Masami Hiramatsu
  2014-09-18 12:04 ` [PATCH v4 1/4] ftracetest: Initial commit for ftracetest Masami Hiramatsu
                   ` (3 more replies)
  0 siblings, 4 replies; 14+ messages in thread
From: Masami Hiramatsu @ 2014-09-18 12:04 UTC (permalink / raw)
  To: Shuah Khan, Tom Zanussi, Oleg Nesterov, Steven Rostedt,
	Namhyung Kim, Fengguang Wu, Ingo Molnar
  Cc: Linux Kernel Mailing List

Hi,

Here is the collection of testcases for ftrace version 4.
In this version, I moved all scripts under the 
tools/testing/selftest :)

Updates are:
 - Use errexit option to simplify tests.
 - Use signal-based result notify for result codes (except PASS/FAIL).
 - Fix README according to Steven's comments.
 - Move under the tools/testing/selftest.
 - Remove XPASS support (unless someone really needs it)

ftracetest is a shell(dash) script for testing ftrace features.
It will be required to add a unit test for each new feature
after this is merged, because of preventing feature regressions.

Thank you,

---

Masami Hiramatsu (4):
      ftracetest: Initial commit for ftracetest
      ftracetest: Add ftrace basic testcases
      ftracetest: Add kprobe basic testcases
      ftracetest: Add POSIX.3 standard and XFAIL result codes


 MAINTAINERS                                        |    1 
 tools/testing/selftests/Makefile                   |    1 
 tools/testing/selftests/ftrace/Makefile            |    7 +
 tools/testing/selftests/ftrace/README              |   82 +++++++
 tools/testing/selftests/ftrace/ftracetest          |  249 ++++++++++++++++++++
 tools/testing/selftests/ftrace/samples/fail.tc     |    4 
 tools/testing/selftests/ftrace/samples/pass.tc     |    3 
 .../testing/selftests/ftrace/samples/unresolved.tc |    4 
 .../selftests/ftrace/samples/unsupported.tc        |    3 
 tools/testing/selftests/ftrace/samples/untested.tc |    3 
 tools/testing/selftests/ftrace/samples/xfail.tc    |    3 
 .../selftests/ftrace/test.d/00basic/basic1.tc      |    3 
 .../selftests/ftrace/test.d/00basic/basic2.tc      |    7 +
 .../selftests/ftrace/test.d/00basic/basic3.tc      |    8 +
 .../ftrace/test.d/kprobe/add_and_remove.tc         |   11 +
 .../selftests/ftrace/test.d/kprobe/busy_check.tc   |   13 +
 tools/testing/selftests/ftrace/test.d/template     |    9 +
 17 files changed, 411 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/Makefile
 create mode 100644 tools/testing/selftests/ftrace/README
 create mode 100755 tools/testing/selftests/ftrace/ftracetest
 create mode 100644 tools/testing/selftests/ftrace/samples/fail.tc
 create mode 100644 tools/testing/selftests/ftrace/samples/pass.tc
 create mode 100644 tools/testing/selftests/ftrace/samples/unresolved.tc
 create mode 100644 tools/testing/selftests/ftrace/samples/unsupported.tc
 create mode 100644 tools/testing/selftests/ftrace/samples/untested.tc
 create mode 100644 tools/testing/selftests/ftrace/samples/xfail.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/00basic/basic1.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/template

--


^ permalink raw reply	[flat|nested] 14+ messages in thread
* [PATCH v4 1/4] ftracetest: Initial commit for ftracetest
  2014-09-18 12:04 [PATCH v4 0/4] ftrace: Add a ftrace test collection Masami Hiramatsu
@ 2014-09-18 12:04 ` Masami Hiramatsu
  2014-09-18 13:07   ` Shuah Khan
  2014-09-18 12:04 ` [PATCH v4 2/4] ftracetest: Add ftrace basic testcases Masami Hiramatsu
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 14+ messages in thread
From: Masami Hiramatsu @ 2014-09-18 12:04 UTC (permalink / raw)
  To: Shuah Khan, Tom Zanussi, Oleg Nesterov, Steven Rostedt,
	Namhyung Kim, Fengguang Wu, Ingo Molnar
  Cc: Linux Kernel Mailing List

ftracetest is a collection of testcase shell-scripts for ftrace.
To avoid regressions of ftrace, these testcases check correct
ftrace behaviors. If someone would like to add any features on
ftrace, the patch series should have at least one testcase for
checking the new behavior.

Changes in v4:
 - Move this under selftests. :)
 - Add a copyright and note of GPLv2.
 - Fix documents acconding to Steven's comments.
 - Fix a small bug pointed by Namhyng.

Changes in v3:
 - Use "." instead of "source".
 - Don't use -e option for echo since dash doesn't support it.

Changes in v2:
 - Remove unneeded 'function' keyword.
 - Add abspath and find_testcases.
 - Make OPT_TEST_CASES a local var.
 - Ensure given testcase ended with .tc.
 - Accept a directory option which has multiple testcases.
 - Change [PASSED]/[FAILED] to [PASS]/[FAIL]
 - Change the basic shell to sh (dash).

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 MAINTAINERS                                    |    1 
 tools/testing/selftests/Makefile               |    1 
 tools/testing/selftests/ftrace/Makefile        |    7 +
 tools/testing/selftests/ftrace/README          |   46 +++++++
 tools/testing/selftests/ftrace/ftracetest      |  157 ++++++++++++++++++++++++
 tools/testing/selftests/ftrace/test.d/template |    4 +
 6 files changed, 216 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/Makefile
 create mode 100644 tools/testing/selftests/ftrace/README
 create mode 100755 tools/testing/selftests/ftrace/ftracetest
 create mode 100644 tools/testing/selftests/ftrace/test.d/template

diff --git a/MAINTAINERS b/MAINTAINERS
index 5e7866a..29e22db 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9295,6 +9295,7 @@ F:	include/*/ftrace.h
 F:	include/linux/trace*.h
 F:	include/trace/
 F:	kernel/trace/
+F:	tools/testing/selftests/ftrace
 
 TRIVIAL PATCHES
 M:	Jiri Kosina <trivial@kernel.org>
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 36ff2e4..45f145c 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -14,6 +14,7 @@ TARGETS += powerpc
 TARGETS += user
 TARGETS += sysctl
 TARGETS += firmware
+TARGETS += ftrace
 
 TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
diff --git a/tools/testing/selftests/ftrace/Makefile b/tools/testing/selftests/ftrace/Makefile
new file mode 100644
index 0000000..76cc9f1
--- /dev/null
+++ b/tools/testing/selftests/ftrace/Makefile
@@ -0,0 +1,7 @@
+all:
+
+run_tests:
+	@/bin/sh ./ftracetest || echo "ftrace selftests: [FAIL]"
+
+clean:
+	rm -rf logs/*
diff --git a/tools/testing/selftests/ftrace/README b/tools/testing/selftests/ftrace/README
new file mode 100644
index 0000000..ee2cef7
--- /dev/null
+++ b/tools/testing/selftests/ftrace/README
@@ -0,0 +1,46 @@
+Linux Ftrace Testcases
+
+This is a collection of testcases for ftrace tracing feature in the Linux
+kernel. Since ftrace exports interfaces via the debugfs, we just need
+shell scripts for testing. Feel free to add new test cases.
+
+Running the ftrace testcases
+============================
+
+At first, you need to be the root user to run this script.
+To run all testcases:
+
+  $ sudo ./ftracetest
+
+To run specific testcases:
+
+  # ./ftracetest test.d/basic3.tc
+
+Or you can also run testcases under given directory:
+
+  # ./ftracetest test.d/kprobe/
+
+Contributing new testcases
+==========================
+
+Copy test.d/template to your testcase (whose filename must have *.tc
+extension) and rewrite the test description line.
+
+ * The working directory of the script is <debugfs>/tracing/.
+
+ * Take care with side effects as the tests are run with root privilege.
+
+ * The tests should not run for a long period of time (more than 1 min.)
+   These are to be unit tests.
+
+ * You can add a directory for your testcases under test.d/ if needed.
+
+ * The test cases should run on dash (busybox shell) for testing on
+   minimal cross-build environments.
+
+TODO
+====
+
+ * Fancy colored output :)
+
+ * Integrate with selftest?
diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
new file mode 100755
index 0000000..02f4e7a
--- /dev/null
+++ b/tools/testing/selftests/ftrace/ftracetest
@@ -0,0 +1,157 @@
+#!/bin/sh
+
+# ftracetest - Ftrace test shell scripts
+#
+# Copyright (C) Hitachi Ltd., 2014
+#  Written by Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
+#
+# Released under the terms of the GPL v2.
+
+usage() { # errno [message]
+[ "$2" ] && echo $2
+echo "Usage: ftracetest [options] [testcase(s)] [testcase-directory(s)]"
+echo " Options:"
+echo "		-h|--help  Show help message"
+echo "		-k|--keep  Keep passed test logs"
+echo "		-d|--debug Debug mode (trace all shell commands)"
+exit $1
+}
+
+errexit() { # message
+  echo "Error: $1" 1>&2
+  exit 1
+}
+
+# Ensuring user privilege
+if [ `id -u` -ne 0 ]; then
+  errexit "this must be run by root user"
+fi
+
+# Utilities
+absdir() { # file_path
+  (cd `dirname $1`; pwd)
+}
+
+abspath() {
+  echo `absdir $1`/`basename $1`
+}
+
+find_testcases() { #directory
+  echo `find $1 -name \*.tc`
+}
+
+parse_opts() { # opts
+  local OPT_TEST_CASES=
+  local OPT_TEST_DIR=
+
+  while [ "$1" ]; do
+    case "$1" in
+    --help|-h)
+      usage 0
+    ;;
+    --keep|-k)
+      KEEP_LOG=1
+      shift 1
+    ;;
+    --debug|-d)
+      DEBUG=1
+      shift 1
+    ;;
+    *.tc)
+      if [ -f "$1" ]; then
+        OPT_TEST_CASES="$OPT_TEST_CASES `abspath $1`"
+        shift 1
+      else
+        usage 1 "$1 is not a testcase"
+      fi
+      ;;
+    *)
+      if [ -d "$1" ]; then
+        OPT_TEST_DIR=`abspath $1`
+        OPT_TEST_CASES="$OPT_TEST_CASES `find_testcases $OPT_TEST_DIR`"
+        shift 1
+      else
+        usage 1 "Invalid option ($1)"
+      fi
+    ;;
+    esac
+  done
+  if [ "$OPT_TEST_CASES" ]; then
+    TEST_CASES=$OPT_TEST_CASES
+  fi
+}
+
+# Parameters
+DEBUGFS_DIR=`grep debugfs /proc/mounts | cut -f2 -d' '`
+TRACING_DIR=$DEBUGFS_DIR/tracing
+TOP_DIR=`absdir $0`
+TEST_DIR=$TOP_DIR/test.d
+TEST_CASES=`find_testcases $TEST_DIR`
+LOG_DIR=$TOP_DIR/logs/`date +%Y%m%d-%H%M%S`/
+KEEP_LOG=0
+DEBUG=0
+# Parse command-line options
+parse_opts $*
+
+[ $DEBUG -ne 0 ] && set -x
+
+# Verify parameters
+if [ -z "$DEBUGFS_DIR" -o ! -d "$TRACING_DIR" ]; then
+  errexit "No ftrace directory found"
+fi
+
+# Preparing logs
+LOG_FILE=$LOG_DIR/ftracetest.log
+mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR"
+date > $LOG_FILE
+prlog() { # messages
+  echo "$@" | tee -a $LOG_FILE
+}
+catlog() { #file
+  cat $1 | tee -a $LOG_FILE
+}
+
+# Testcase management
+PASSED_CASES=
+FAILED_CASES=
+CASENO=0
+testcase() { # testfile
+  CASENO=$((CASENO+1))
+  prlog -n "[$CASENO]"`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
+}
+failed() {
+  prlog "	[FAIL]"
+  FAILED_CASES="$FAILED_CASES $CASENO"
+}
+passed() {
+  prlog "	[PASS]"
+  PASSED_CASES="$PASSED_CASES $CASENO"
+}
+
+
+# Run one test case
+run_test() { # testfile
+  local testname=`basename $1`
+  local testlog=`mktemp --tmpdir=$LOG_DIR ${testname}-XXXXXX.log`
+  testcase $1
+  echo "execute: "$1 > $testlog
+  (cd $TRACING_DIR; set -x ; . $1) >> $testlog 2>&1
+  ret=$?
+  if [ $ret -ne 0 ]; then
+    failed
+    catlog $testlog
+  else
+    passed
+    [ $KEEP_LOG -eq 0 ] && rm $testlog
+  fi
+}
+
+# Main loop
+for t in $TEST_CASES; do
+  run_test $t
+done
+prlog ""
+prlog "# of passed: " `echo $PASSED_CASES | wc -w`
+prlog "# of failed: " `echo $FAILED_CASES | wc -w`
+
+test -z "$FAILED_CASES" # if no error, return 0
diff --git a/tools/testing/selftests/ftrace/test.d/template b/tools/testing/selftests/ftrace/test.d/template
new file mode 100644
index 0000000..ce5f735
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/template
@@ -0,0 +1,4 @@
+#!/bin/sh
+# description: %HERE DESCRIBE WHAT THIS DOES%
+# you have to add ".tc" extention for your testcase file
+exit 0 # Return 0 if the test is passed, otherwise return !0



^ permalink raw reply related	[flat|nested] 14+ messages in thread
* [PATCH v4 2/4] ftracetest: Add ftrace basic testcases
  2014-09-18 12:04 [PATCH v4 0/4] ftrace: Add a ftrace test collection Masami Hiramatsu
  2014-09-18 12:04 ` [PATCH v4 1/4] ftracetest: Initial commit for ftracetest Masami Hiramatsu
@ 2014-09-18 12:04 ` Masami Hiramatsu
  2014-09-18 12:04 ` [PATCH v4 3/4] ftracetest: Add kprobe " Masami Hiramatsu
  2014-09-18 12:05 ` [PATCH v4 4/4] ftracetest: Add POSIX.3 standard and XFAIL result codes Masami Hiramatsu
  3 siblings, 0 replies; 14+ messages in thread
From: Masami Hiramatsu @ 2014-09-18 12:04 UTC (permalink / raw)
  To: Shuah Khan, Tom Zanussi, Oleg Nesterov, Steven Rostedt,
	Namhyung Kim, Fengguang Wu, Ingo Molnar
  Cc: Linux Kernel Mailing List

Add ftrace basic testcases. This just checks ftrace debugfs
interface works as it is designed.

Changes in v2:
  - Change shell to sh instead of bash.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 .../selftests/ftrace/test.d/00basic/basic1.tc      |    3 +++
 .../selftests/ftrace/test.d/00basic/basic2.tc      |    6 ++++++
 .../selftests/ftrace/test.d/00basic/basic3.tc      |    8 ++++++++
 3 files changed, 17 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/test.d/00basic/basic1.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/00basic/basic3.tc

diff --git a/tools/testing/selftests/ftrace/test.d/00basic/basic1.tc b/tools/testing/selftests/ftrace/test.d/00basic/basic1.tc
new file mode 100644
index 0000000..9980ff1
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/00basic/basic1.tc
@@ -0,0 +1,3 @@
+#!/bin/sh
+# description: Basic trace file check
+test -f README -a -f trace -a -f tracing_on -a -f trace_pipe
diff --git a/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc b/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
new file mode 100644
index 0000000..b04f30d
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
@@ -0,0 +1,6 @@
+#!/bin/sh
+# description: Basic test for tracers
+for t in `cat available_tracers`; do
+  echo $t > current_tracer || exit 1
+done
+echo nop > current_tracer
diff --git a/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc b/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
new file mode 100644
index 0000000..0c1a3a2
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
@@ -0,0 +1,8 @@
+#!/bin/sh
+# description: Basic trace clock test
+[ -f trace_clock ] || exit 1
+for c in `cat trace_clock | tr  -d \[\]`; do
+  echo $c > trace_clock || exit 1
+  grep '\['$c'\]' trace_clock || exit 1
+done
+echo local > trace_clock



^ permalink raw reply related	[flat|nested] 14+ messages in thread
* [PATCH v4 3/4] ftracetest: Add kprobe basic testcases
  2014-09-18 12:04 [PATCH v4 0/4] ftrace: Add a ftrace test collection Masami Hiramatsu
  2014-09-18 12:04 ` [PATCH v4 1/4] ftracetest: Initial commit for ftracetest Masami Hiramatsu
  2014-09-18 12:04 ` [PATCH v4 2/4] ftracetest: Add ftrace basic testcases Masami Hiramatsu
@ 2014-09-18 12:04 ` Masami Hiramatsu
  2014-09-18 12:05 ` [PATCH v4 4/4] ftracetest: Add POSIX.3 standard and XFAIL result codes Masami Hiramatsu
  3 siblings, 0 replies; 14+ messages in thread
From: Masami Hiramatsu @ 2014-09-18 12:04 UTC (permalink / raw)
  To: Shuah Khan, Tom Zanussi, Oleg Nesterov, Steven Rostedt,
	Namhyung Kim, Fengguang Wu, Ingo Molnar
  Cc: Linux Kernel Mailing List

Add basic testcases for kprobe dynamic events.
This also shows that the ftracetest accepts sub-directory
for new testcases.

Changes in v2:
  - Change shell to sh instead of bash.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 .../ftrace/test.d/kprobe/add_and_remove.tc         |   11 +++++++++++
 .../selftests/ftrace/test.d/kprobe/busy_check.tc   |   14 ++++++++++++++
 2 files changed, 25 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
 create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc

diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc b/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
new file mode 100644
index 0000000..5ddfb47
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
@@ -0,0 +1,11 @@
+#!/bin/sh
+# description: Kprobe dynamic event - adding and removing
+
+[ -f kprobe_events ] || exit 1
+
+echo 0 > events/enable || exit 1
+echo > kprobe_events || exit 1
+echo p:myevent do_fork > kprobe_events || exit 1
+grep myevent kprobe_events || exit 1
+[ -d events/kprobes/myevent ] || exit 1
+echo > kprobe_events
diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc b/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
new file mode 100644
index 0000000..588fde97
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
@@ -0,0 +1,14 @@
+#!/bin/sh
+# description: Kprobe dynamic event - busy event check
+
+[ -f kprobe_events ] || exit 1
+
+echo 0 > events/enable || exit 1
+echo > kprobe_events || exit 1
+echo p:myevent do_fork > kprobe_events || exit 1
+[ -d events/kprobes/myevent ] || exit 1
+echo 1 > events/kprobes/myevent/enable || exit 1
+echo > kprobe_events && exit 1 # this must fail
+echo 0 > events/kprobes/myevent/enable || exit 1
+echo > kprobe_events # this must succeed
+



^ permalink raw reply related	[flat|nested] 14+ messages in thread
* [PATCH v4 4/4] ftracetest: Add POSIX.3 standard and XFAIL result codes
  2014-09-18 12:04 [PATCH v4 0/4] ftrace: Add a ftrace test collection Masami Hiramatsu
                   ` (2 preceding siblings ...)
  2014-09-18 12:04 ` [PATCH v4 3/4] ftracetest: Add kprobe " Masami Hiramatsu
@ 2014-09-18 12:05 ` Masami Hiramatsu
  2014-09-22 15:47   ` Namhyung Kim
  3 siblings, 1 reply; 14+ messages in thread
From: Masami Hiramatsu @ 2014-09-18 12:05 UTC (permalink / raw)
  To: Shuah Khan, Tom Zanussi, Oleg Nesterov, Steven Rostedt,
	Namhyung Kim, Fengguang Wu, Ingo Molnar
  Cc: Linux Kernel Mailing List

Add XFAIL and POSIX 1003.3 standard codes (UNRESOLVED/
UNTESTED/UNSUPPORTED) as result codes. These are used for the
results that test case is expected to fail or unsupported
feature (by config).

To return these result code, this introduces exit_unresolved,
exit_untested, exit_unsupported and exit_xfail functions,
which use real-time signals to notify the result code to
ftracetest.

This also set "errexit" option for the testcases, so that
the tests don't need to exit explicitly.

Changes in v4:
 - Remove XPASS and short-names.
 - Use real-time signals for result code.
 - Simplify test cases by using errexit option.
 - Add sample scripts for result code.

Changes in v3:
 - Add UNRESOLVED/UNTESTED codes.
 - Fix to handle undefined codes.
 - Add a document about return codes.
 - Fix to show failure logs.
 - Don't use -e option for echo since dash doesn't support it.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/testing/selftests/ftrace/README              |   38 ++++++
 tools/testing/selftests/ftrace/ftracetest          |  122 ++++++++++++++++++--
 tools/testing/selftests/ftrace/samples/fail.tc     |    4 +
 tools/testing/selftests/ftrace/samples/pass.tc     |    3 
 .../testing/selftests/ftrace/samples/unresolved.tc |    4 +
 .../selftests/ftrace/samples/unsupported.tc        |    3 
 tools/testing/selftests/ftrace/samples/untested.tc |    3 
 tools/testing/selftests/ftrace/samples/xfail.tc    |    3 
 .../selftests/ftrace/test.d/00basic/basic2.tc      |    3 
 .../selftests/ftrace/test.d/00basic/basic3.tc      |    6 -
 .../ftrace/test.d/kprobe/add_and_remove.tc         |   12 +-
 .../selftests/ftrace/test.d/kprobe/busy_check.tc   |   15 +-
 tools/testing/selftests/ftrace/test.d/template     |    5 +
 13 files changed, 187 insertions(+), 34 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/samples/fail.tc
 create mode 100644 tools/testing/selftests/ftrace/samples/pass.tc
 create mode 100644 tools/testing/selftests/ftrace/samples/unresolved.tc
 create mode 100644 tools/testing/selftests/ftrace/samples/unsupported.tc
 create mode 100644 tools/testing/selftests/ftrace/samples/untested.tc
 create mode 100644 tools/testing/selftests/ftrace/samples/xfail.tc

diff --git a/tools/testing/selftests/ftrace/README b/tools/testing/selftests/ftrace/README
index ee2cef7..182e76f 100644
--- a/tools/testing/selftests/ftrace/README
+++ b/tools/testing/selftests/ftrace/README
@@ -38,9 +38,45 @@ extension) and rewrite the test description line.
  * The test cases should run on dash (busybox shell) for testing on
    minimal cross-build environments.
 
+ * Note that the tests are run with "set -e" (errexit) option. If any
+   command fails, the test will be terminated immediately.
+
+ * The tests can return some result codes instead of pass or fail by
+   using exit_unresolved, exit_untested, exit_unsupported and exit_xfail.
+
+Result code
+===========
+
+Ftracetest supports following result codes.
+
+ * PASS: The test succeeded as expected. The test which exits with 0 is
+         counted as passed test.
+
+ * FAIL: The test failed, but was expected to succeed. The test which exits
+         with !0 is counted as failed test.
+
+ * UNRESOLVED: The test produced unclear or intermidiate results.
+             for example, the test was interrupted
+                       or the test depends on a previous test, which failed.
+                       or the test was set up incorrectly
+             The test which is in above situation, must call exit_unresolved.
+
+ * UNTESTED: The test was not run, currently just a placeholder.
+             In this case, the test must call exit_untested.
+
+ * UNSUPPORTED: The test failed because of lack of feature.
+               In this case, the test must call exit_unsupported.
+
+ * XFAIL: The test failed, and was expected to fail.
+          To return XFAIL, call exit_xfail from the test.
+
+There are some sample test scripts for result code under samples/.
+You can also run samples as below:
+
+  # ./ftracetest samples/
+
 TODO
 ====
 
  * Fancy colored output :)
 
- * Integrate with selftest?
diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
index 02f4e7a..2f7f8da0 100755
--- a/tools/testing/selftests/ftrace/ftracetest
+++ b/tools/testing/selftests/ftrace/ftracetest
@@ -112,22 +112,105 @@ catlog() { #file
 }
 
 # Testcase management
+# Test result codes - Dejagnu extended code
+PASS=0	# The test succeeded.
+FAIL=1	# The test failed, but was expected to succeed.
+UNRESOLVED=2  # The test produced indeterminate results. (e.g. interrupted)
+UNTESTED=3    # The test was not run, currently just a placeholder.
+UNSUPPORTED=4 # The test failed because of lack of feature.
+XFAIL=5	# The test failed, and was expected to fail.
+
+# Accumulations
 PASSED_CASES=
 FAILED_CASES=
+UNRESOLVED_CASES=
+UNTESTED_CASES=
+UNSUPPORTED_CASES=
+XFAILED_CASES=
+UNDEFINED_CASES=
+
 CASENO=0
 testcase() { # testfile
   CASENO=$((CASENO+1))
   prlog -n "[$CASENO]"`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
 }
-failed() {
-  prlog "	[FAIL]"
-  FAILED_CASES="$FAILED_CASES $CASENO"
+
+eval_result() { # retval sigval
+  local retval=$2
+  if [ $2 -eq 0 ]; then
+    test $1 -ne 0 && retval=$FAIL
+  fi
+  case $retval in
+    $PASS)
+      prlog "	[PASS]"
+      PASSED_CASES="$PASSED_CASES $CASENO"
+      return 0
+    ;;
+    $FAIL)
+      prlog "	[FAIL]"
+      FAILED_CASES="$FAILED_CASES $CASENO"
+      return 1 # this is a bug.
+    ;;
+    $UNRESOLVED)
+      prlog "	[UNRESOLVED]"
+      UNRESOLVED_CASES="$UNRESOLVED_CASES $CASENO"
+      return 1 # this is a kind of bug.. something happened.
+    ;;
+    $UNTESTED)
+      prlog "	[UNTESTED]"
+      UNTESTED_CASES="$UNTESTED_CASES $CASENO"
+      return 0
+    ;;
+    $UNSUPPORTED)
+      prlog "	[UNSUPPORTED]"
+      UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO"
+      return 0 # this is not a bug
+    ;;
+    $XFAIL)
+      prlog "	[XFAIL]"
+      XFAILED_CASES="$XFAILED_CASES $CASENO"
+      return 0
+    ;;
+    *)
+      prlog "	[UNDEFINED]"
+      UNDEFINED_CASES="$UNDEFINED_CASES $CASENO"
+      return 1 # this must be a test bug
+    ;;
+  esac
+}
+
+# Signal handling for result codes
+SIG_RESULT=
+SIG_BASE=36	# Use realtime signals
+SIG_PID=$$
+
+SIG_UNRESOLVED=$((SIG_BASE + UNRESOLVED))
+exit_unresolved () {
+  kill -s $SIG_UNRESOLVED $SIG_PID
+  exit 0
+}
+trap 'SIG_RESULT=$UNRESOLVED' $SIG_UNRESOLVED
+
+SIG_UNTESTED=$((SIG_BASE + UNTESTED))
+exit_untested () {
+  kill -s $SIG_UNTESTED $SIG_PID
+  exit 0
 }
-passed() {
-  prlog "	[PASS]"
-  PASSED_CASES="$PASSED_CASES $CASENO"
+trap 'SIG_RESULT=$UNTESTED' $SIG_UNTESTED
+
+SIG_UNSUPPORTED=$((SIG_BASE + UNSUPPORTED))
+exit_unsupported () {
+  kill -s $SIG_UNSUPPORTED $SIG_PID
+  exit 0
 }
+trap 'SIG_RESULT=$UNSUPPORTED' $SIG_UNSUPPORTED
 
+SIG_XFAIL=$((SIG_BASE + XFAIL))
+exit_xfail () {
+  kill -s $SIG_XFAIL $SIG_PID
+  exit 0
+}
+trap 'SIG_RESULT=$XFAIL' $SIG_XFAIL
 
 # Run one test case
 run_test() { # testfile
@@ -135,14 +218,16 @@ run_test() { # testfile
   local testlog=`mktemp --tmpdir=$LOG_DIR ${testname}-XXXXXX.log`
   testcase $1
   echo "execute: "$1 > $testlog
-  (cd $TRACING_DIR; set -x ; . $1) >> $testlog 2>&1
-  ret=$?
-  if [ $ret -ne 0 ]; then
-    failed
-    catlog $testlog
-  else
-    passed
+  SIG_RESULT=0
+  # setup PID and PPID, $$ is not updated.
+  (cd $TRACING_DIR; read PID _ < /proc/self/stat ;
+   set -e; set -x; . $1) >> $testlog 2>&1
+  eval_result $? $SIG_RESULT
+  if [ $? -eq 0 ]; then
+    # Remove test log if the test was done as it was expected.
     [ $KEEP_LOG -eq 0 ] && rm $testlog
+  else
+    catlog $testlog
   fi
 }
 
@@ -150,8 +235,15 @@ run_test() { # testfile
 for t in $TEST_CASES; do
   run_test $t
 done
+
 prlog ""
 prlog "# of passed: " `echo $PASSED_CASES | wc -w`
 prlog "# of failed: " `echo $FAILED_CASES | wc -w`
-
-test -z "$FAILED_CASES" # if no error, return 0
+prlog "# of unresolved: " `echo $UNRESOLVED_CASES | wc -w`
+prlog "# of untested: " `echo $UNTESTED_CASES | wc -w`
+prlog "# of unsupported: " `echo $UNSUPPORTED_CASES | wc -w`
+prlog "# of xfailed: " `echo $XFAILED_CASES | wc -w`
+prlog "# of undefined(test bug): " `echo $UNDEFINED_CASES | wc -w`
+
+# if no error, return 0
+test -z "$FAILED_CASES$UNDEFINED_CASES$UNRESOLVED_CASES"
diff --git a/tools/testing/selftests/ftrace/samples/fail.tc b/tools/testing/selftests/ftrace/samples/fail.tc
new file mode 100644
index 0000000..15e35b9
--- /dev/null
+++ b/tools/testing/selftests/ftrace/samples/fail.tc
@@ -0,0 +1,4 @@
+#!/bin/sh
+# description: failure-case example
+cat non-exist-file
+echo "this is not executed"
diff --git a/tools/testing/selftests/ftrace/samples/pass.tc b/tools/testing/selftests/ftrace/samples/pass.tc
new file mode 100644
index 0000000..d015493
--- /dev/null
+++ b/tools/testing/selftests/ftrace/samples/pass.tc
@@ -0,0 +1,3 @@
+#!/bin/sh
+# description: pass-case example
+return 0
diff --git a/tools/testing/selftests/ftrace/samples/unresolved.tc b/tools/testing/selftests/ftrace/samples/unresolved.tc
new file mode 100644
index 0000000..41e99d3
--- /dev/null
+++ b/tools/testing/selftests/ftrace/samples/unresolved.tc
@@ -0,0 +1,4 @@
+#!/bin/sh
+# description: unresolved-case example
+trap exit_unresolved INT
+kill -INT $PID
diff --git a/tools/testing/selftests/ftrace/samples/unsupported.tc b/tools/testing/selftests/ftrace/samples/unsupported.tc
new file mode 100644
index 0000000..45910ff
--- /dev/null
+++ b/tools/testing/selftests/ftrace/samples/unsupported.tc
@@ -0,0 +1,3 @@
+#!/bin/sh
+# description: unsupported-case example
+exit_unsupported
diff --git a/tools/testing/selftests/ftrace/samples/untested.tc b/tools/testing/selftests/ftrace/samples/untested.tc
new file mode 100644
index 0000000..35a4594
--- /dev/null
+++ b/tools/testing/selftests/ftrace/samples/untested.tc
@@ -0,0 +1,3 @@
+#!/bin/sh
+# description: untested-case example
+exit_untested
diff --git a/tools/testing/selftests/ftrace/samples/xfail.tc b/tools/testing/selftests/ftrace/samples/xfail.tc
new file mode 100644
index 0000000..9dd3953
--- /dev/null
+++ b/tools/testing/selftests/ftrace/samples/xfail.tc
@@ -0,0 +1,3 @@
+#!/bin/sh
+# description: xfail-case example
+cat non-exist-file || exit_xfail
diff --git a/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc b/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
index b04f30d..bf9a7b0 100644
--- a/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
+++ b/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
@@ -1,6 +1,7 @@
 #!/bin/sh
 # description: Basic test for tracers
+test -f available_tracers
 for t in `cat available_tracers`; do
-  echo $t > current_tracer || exit 1
+  echo $t > current_tracer
 done
 echo nop > current_tracer
diff --git a/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc b/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
index 0c1a3a2..bde6625 100644
--- a/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
+++ b/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
@@ -1,8 +1,8 @@
 #!/bin/sh
 # description: Basic trace clock test
-[ -f trace_clock ] || exit 1
+test -f trace_clock
 for c in `cat trace_clock | tr  -d \[\]`; do
-  echo $c > trace_clock || exit 1
-  grep '\['$c'\]' trace_clock || exit 1
+  echo $c > trace_clock
+  grep '\['$c'\]' trace_clock
 done
 echo local > trace_clock
diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc b/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
index 5ddfb47..1b8b665 100644
--- a/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
+++ b/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
@@ -1,11 +1,11 @@
 #!/bin/sh
 # description: Kprobe dynamic event - adding and removing
 
-[ -f kprobe_events ] || exit 1
+[ -f kprobe_events ] || exit_unsupported # this is configurable
 
-echo 0 > events/enable || exit 1
-echo > kprobe_events || exit 1
-echo p:myevent do_fork > kprobe_events || exit 1
-grep myevent kprobe_events || exit 1
-[ -d events/kprobes/myevent ] || exit 1
+echo 0 > events/enable
+echo > kprobe_events
+echo p:myevent do_fork > kprobe_events
+grep myevent kprobe_events
+test -d events/kprobes/myevent
 echo > kprobe_events
diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc b/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
index 588fde97..b55c840 100644
--- a/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
+++ b/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
@@ -1,14 +1,13 @@
 #!/bin/sh
 # description: Kprobe dynamic event - busy event check
 
-[ -f kprobe_events ] || exit 1
+[ -f kprobe_events ] || exit_unsupported
 
-echo 0 > events/enable || exit 1
-echo > kprobe_events || exit 1
-echo p:myevent do_fork > kprobe_events || exit 1
-[ -d events/kprobes/myevent ] || exit 1
-echo 1 > events/kprobes/myevent/enable || exit 1
+echo 0 > events/enable
+echo > kprobe_events
+echo p:myevent do_fork > kprobe_events
+test -d events/kprobes/myevent
+echo 1 > events/kprobes/myevent/enable
 echo > kprobe_events && exit 1 # this must fail
-echo 0 > events/kprobes/myevent/enable || exit 1
+echo 0 > events/kprobes/myevent/enable
 echo > kprobe_events # this must succeed
-
diff --git a/tools/testing/selftests/ftrace/test.d/template b/tools/testing/selftests/ftrace/test.d/template
index ce5f735..5448f7a 100644
--- a/tools/testing/selftests/ftrace/test.d/template
+++ b/tools/testing/selftests/ftrace/test.d/template
@@ -1,4 +1,9 @@
 #!/bin/sh
 # description: %HERE DESCRIBE WHAT THIS DOES%
 # you have to add ".tc" extention for your testcase file
+# Note that all tests are run with "errexit" option.
+
 exit 0 # Return 0 if the test is passed, otherwise return !0
+# If the test could not run because of lack of feature, call exit_unsupported
+# If the test returned unclear results, call exit_unresolved
+# If the test is a dummy, or a placeholder, call exit_untested



^ permalink raw reply related	[flat|nested] 14+ messages in thread
* Re: [PATCH v4 1/4] ftracetest: Initial commit for ftracetest
  2014-09-18 12:04 ` [PATCH v4 1/4] ftracetest: Initial commit for ftracetest Masami Hiramatsu
@ 2014-09-18 13:07   ` Shuah Khan
  2014-09-19  1:03     ` Masami Hiramatsu
  0 siblings, 1 reply; 14+ messages in thread
From: Shuah Khan @ 2014-09-18 13:07 UTC (permalink / raw)
  To: Masami Hiramatsu, Shuah Khan, Tom Zanussi, Oleg Nesterov,
	Steven Rostedt, Namhyung Kim, Fengguang Wu, Ingo Molnar
  Cc: Linux Kernel Mailing List, Shuah Khan

On 09/18/2014 06:04 AM, Masami Hiramatsu wrote:
> ftracetest is a collection of testcase shell-scripts for ftrace.
> To avoid regressions of ftrace, these testcases check correct
> ftrace behaviors. If someone would like to add any features on
> ftrace, the patch series should have at least one testcase for
> checking the new behavior.
> 
> Changes in v4:
>  - Move this under selftests. :)
>  - Add a copyright and note of GPLv2.
>  - Fix documents acconding to Steven's comments.
>  - Fix a small bug pointed by Namhyng.
> 
> Changes in v3:
>  - Use "." instead of "source".
>  - Don't use -e option for echo since dash doesn't support it.
> 
> Changes in v2:
>  - Remove unneeded 'function' keyword.
>  - Add abspath and find_testcases.
>  - Make OPT_TEST_CASES a local var.
>  - Ensure given testcase ended with .tc.
>  - Accept a directory option which has multiple testcases.
>  - Change [PASSED]/[FAILED] to [PASS]/[FAIL]
>  - Change the basic shell to sh (dash).
> 
> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> ---
>  MAINTAINERS                                    |    1 
>  tools/testing/selftests/Makefile               |    1 
>  tools/testing/selftests/ftrace/Makefile        |    7 +
>  tools/testing/selftests/ftrace/README          |   46 +++++++
>  tools/testing/selftests/ftrace/ftracetest      |  157 ++++++++++++++++++++++++
>  tools/testing/selftests/ftrace/test.d/template |    4 +
>  6 files changed, 216 insertions(+)
>  create mode 100644 tools/testing/selftests/ftrace/Makefile
>  create mode 100644 tools/testing/selftests/ftrace/README
>  create mode 100755 tools/testing/selftests/ftrace/ftracetest
>  create mode 100644 tools/testing/selftests/ftrace/test.d/template
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5e7866a..29e22db 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -9295,6 +9295,7 @@ F:	include/*/ftrace.h
>  F:	include/linux/trace*.h
>  F:	include/trace/
>  F:	kernel/trace/
> +F:	tools/testing/selftests/ftrace
>  
>  TRIVIAL PATCHES
>  M:	Jiri Kosina <trivial@kernel.org>
> diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
> index 36ff2e4..45f145c 100644
> --- a/tools/testing/selftests/Makefile
> +++ b/tools/testing/selftests/Makefile
> @@ -14,6 +14,7 @@ TARGETS += powerpc
>  TARGETS += user
>  TARGETS += sysctl
>  TARGETS += firmware
> +TARGETS += ftrace
>  
>  TARGETS_HOTPLUG = cpu-hotplug
>  TARGETS_HOTPLUG += memory-hotplug
> diff --git a/tools/testing/selftests/ftrace/Makefile b/tools/testing/selftests/ftrace/Makefile
> new file mode 100644
> index 0000000..76cc9f1
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/Makefile
> @@ -0,0 +1,7 @@
> +all:
> +
> +run_tests:
> +	@/bin/sh ./ftracetest || echo "ftrace selftests: [FAIL]"
> +
> +clean:
> +	rm -rf logs/*
> diff --git a/tools/testing/selftests/ftrace/README b/tools/testing/selftests/ftrace/README
> new file mode 100644
> index 0000000..ee2cef7
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/README
> @@ -0,0 +1,46 @@
> +Linux Ftrace Testcases
> +
> +This is a collection of testcases for ftrace tracing feature in the Linux
> +kernel. Since ftrace exports interfaces via the debugfs, we just need
> +shell scripts for testing. Feel free to add new test cases.
> +
> +Running the ftrace testcases
> +============================
> +
> +At first, you need to be the root user to run this script.
> +To run all testcases:
> +
> +  $ sudo ./ftracetest
> +
> +To run specific testcases:
> +
> +  # ./ftracetest test.d/basic3.tc
> +
> +Or you can also run testcases under given directory:
> +
> +  # ./ftracetest test.d/kprobe/
> +
> +Contributing new testcases
> +==========================
> +
> +Copy test.d/template to your testcase (whose filename must have *.tc
> +extension) and rewrite the test description line.
> +
> + * The working directory of the script is <debugfs>/tracing/.
> +
> + * Take care with side effects as the tests are run with root privilege.
> +
> + * The tests should not run for a long period of time (more than 1 min.)
> +   These are to be unit tests.
> +
> + * You can add a directory for your testcases under test.d/ if needed.
> +
> + * The test cases should run on dash (busybox shell) for testing on
> +   minimal cross-build environments.
> +
> +TODO
> +====
> +
> + * Fancy colored output :)
> +
> + * Integrate with selftest?

The second TODO can be removed since it is moved under selftests??
If you haven't already done this, could you please make sure it
runs without hangs when run from the "make kselftest" target.

thanks,
-- Shuah

-- 
Shuah Khan
Sr. Linux Kernel Developer
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

^ permalink raw reply	[flat|nested] 14+ messages in thread
* Re: [PATCH v4 1/4] ftracetest: Initial commit for ftracetest
  2014-09-18 13:07   ` Shuah Khan
@ 2014-09-19  1:03     ` Masami Hiramatsu
  2014-09-19 14:05       ` Shuah Khan
  0 siblings, 1 reply; 14+ messages in thread
From: Masami Hiramatsu @ 2014-09-19  1:03 UTC (permalink / raw)
  To: Shuah Khan
  Cc: Shuah Khan, Tom Zanussi, Oleg Nesterov, Steven Rostedt,
	Namhyung Kim, Fengguang Wu, Ingo Molnar,
	Linux Kernel Mailing List

(2014/09/18 22:07), Shuah Khan wrote:
> On 09/18/2014 06:04 AM, Masami Hiramatsu wrote:
>> ftracetest is a collection of testcase shell-scripts for ftrace.
>> To avoid regressions of ftrace, these testcases check correct
>> ftrace behaviors. If someone would like to add any features on
>> ftrace, the patch series should have at least one testcase for
>> checking the new behavior.
>>
>> Changes in v4:
>>  - Move this under selftests. :)
>>  - Add a copyright and note of GPLv2.
>>  - Fix documents acconding to Steven's comments.
>>  - Fix a small bug pointed by Namhyng.
>>
>> Changes in v3:
>>  - Use "." instead of "source".
>>  - Don't use -e option for echo since dash doesn't support it.
>>
>> Changes in v2:
>>  - Remove unneeded 'function' keyword.
>>  - Add abspath and find_testcases.
>>  - Make OPT_TEST_CASES a local var.
>>  - Ensure given testcase ended with .tc.
>>  - Accept a directory option which has multiple testcases.
>>  - Change [PASSED]/[FAILED] to [PASS]/[FAIL]
>>  - Change the basic shell to sh (dash).
>>
>> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
>> ---
>>  MAINTAINERS                                    |    1 
>>  tools/testing/selftests/Makefile               |    1 
>>  tools/testing/selftests/ftrace/Makefile        |    7 +
>>  tools/testing/selftests/ftrace/README          |   46 +++++++
>>  tools/testing/selftests/ftrace/ftracetest      |  157 ++++++++++++++++++++++++
>>  tools/testing/selftests/ftrace/test.d/template |    4 +
>>  6 files changed, 216 insertions(+)
>>  create mode 100644 tools/testing/selftests/ftrace/Makefile
>>  create mode 100644 tools/testing/selftests/ftrace/README
>>  create mode 100755 tools/testing/selftests/ftrace/ftracetest
>>  create mode 100644 tools/testing/selftests/ftrace/test.d/template
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 5e7866a..29e22db 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -9295,6 +9295,7 @@ F:	include/*/ftrace.h
>>  F:	include/linux/trace*.h
>>  F:	include/trace/
>>  F:	kernel/trace/
>> +F:	tools/testing/selftests/ftrace
>>  
>>  TRIVIAL PATCHES
>>  M:	Jiri Kosina <trivial@kernel.org>
>> diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
>> index 36ff2e4..45f145c 100644
>> --- a/tools/testing/selftests/Makefile
>> +++ b/tools/testing/selftests/Makefile
>> @@ -14,6 +14,7 @@ TARGETS += powerpc
>>  TARGETS += user
>>  TARGETS += sysctl
>>  TARGETS += firmware
>> +TARGETS += ftrace
>>  
>>  TARGETS_HOTPLUG = cpu-hotplug
>>  TARGETS_HOTPLUG += memory-hotplug
>> diff --git a/tools/testing/selftests/ftrace/Makefile b/tools/testing/selftests/ftrace/Makefile
>> new file mode 100644
>> index 0000000..76cc9f1
>> --- /dev/null
>> +++ b/tools/testing/selftests/ftrace/Makefile
>> @@ -0,0 +1,7 @@
>> +all:
>> +
>> +run_tests:
>> +	@/bin/sh ./ftracetest || echo "ftrace selftests: [FAIL]"
>> +
>> +clean:
>> +	rm -rf logs/*
>> diff --git a/tools/testing/selftests/ftrace/README b/tools/testing/selftests/ftrace/README
>> new file mode 100644
>> index 0000000..ee2cef7
>> --- /dev/null
>> +++ b/tools/testing/selftests/ftrace/README
>> @@ -0,0 +1,46 @@
>> +Linux Ftrace Testcases
>> +
>> +This is a collection of testcases for ftrace tracing feature in the Linux
>> +kernel. Since ftrace exports interfaces via the debugfs, we just need
>> +shell scripts for testing. Feel free to add new test cases.
>> +
>> +Running the ftrace testcases
>> +============================
>> +
>> +At first, you need to be the root user to run this script.
>> +To run all testcases:
>> +
>> +  $ sudo ./ftracetest
>> +
>> +To run specific testcases:
>> +
>> +  # ./ftracetest test.d/basic3.tc
>> +
>> +Or you can also run testcases under given directory:
>> +
>> +  # ./ftracetest test.d/kprobe/
>> +
>> +Contributing new testcases
>> +==========================
>> +
>> +Copy test.d/template to your testcase (whose filename must have *.tc
>> +extension) and rewrite the test description line.
>> +
>> + * The working directory of the script is <debugfs>/tracing/.
>> +
>> + * Take care with side effects as the tests are run with root privilege.
>> +
>> + * The tests should not run for a long period of time (more than 1 min.)
>> +   These are to be unit tests.
>> +
>> + * You can add a directory for your testcases under test.d/ if needed.
>> +
>> + * The test cases should run on dash (busybox shell) for testing on
>> +   minimal cross-build environments.
>> +
>> +TODO
>> +====
>> +
>> + * Fancy colored output :)
>> +
>> + * Integrate with selftest?
> 
> The second TODO can be removed since it is moved under selftests??

Ah, I missed to remove that in this commit. It's already integrated.

- * Integrate with selftest?

Oops, and I removed it in 4/4 ...

> If you haven't already done this, could you please make sure it
> runs without hangs when run from the "make kselftest" target.

OK, here is the last lines of the test result.

== Testing sysctl behavior against /proc/sys/vm/swappiness ==
Writing test file ... ok
Checking sysctl is not set to test value ... ok
Writing sysctl from shell ... ok
Resetting sysctl to original value ... ok
Writing entire sysctl in single write ... ok
Writing middle of sysctl after synchronized seek ... Writing beyond end of sysctl ... Writing sysctl with multiple long writes ... ok
fw_filesystem: [FAIL]
[1] Basic trace file check      [PASS]
[2] Basic test for tracers      [PASS]
[3] Basic trace clock test      [PASS]
[4] Kprobe dynamic event - adding and removing  [PASS]
[5] Kprobe dynamic event - busy event check     [PASS]

# of passed:  5
# of failed:  0
# of unresolved:  0
# of untested:  0
# of unsupported:  0
# of xfailed:  0
# of undefined(test bug):  0

Hmm, I think it should have a separation line before running ftracetest.

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com



^ permalink raw reply	[flat|nested] 14+ messages in thread
* Re: [PATCH v4 1/4] ftracetest: Initial commit for ftracetest
  2014-09-19  1:03     ` Masami Hiramatsu
@ 2014-09-19 14:05       ` Shuah Khan
  2014-09-19 14:42         ` Steven Rostedt
  2014-09-20 17:26         ` Masami Hiramatsu
  0 siblings, 2 replies; 14+ messages in thread
From: Shuah Khan @ 2014-09-19 14:05 UTC (permalink / raw)
  To: Masami Hiramatsu, Steven Rostedt
  Cc: Shuah Khan, Tom Zanussi, Oleg Nesterov, Namhyung Kim,
	Fengguang Wu, Ingo Molnar, Linux Kernel Mailing List, Shuah Khan

On 09/18/2014 07:03 PM, Masami Hiramatsu wrote:
> (2014/09/18 22:07), Shuah Khan wrote:
>> On 09/18/2014 06:04 AM, Masami Hiramatsu wrote:
>>> ftracetest is a collection of testcase shell-scripts for ftrace.
>>> To avoid regressions of ftrace, these testcases check correct
>>> ftrace behaviors. If someone would like to add any features on
>>> ftrace, the patch series should have at least one testcase for
>>> checking the new behavior.
>>>
>>> Changes in v4:
>>>  - Move this under selftests. :)
>>>  - Add a copyright and note of GPLv2.
>>>  - Fix documents acconding to Steven's comments.
>>>  - Fix a small bug pointed by Namhyng.
>>>
>>> Changes in v3:
>>>  - Use "." instead of "source".
>>>  - Don't use -e option for echo since dash doesn't support it.
>>>
>>> Changes in v2:
>>>  - Remove unneeded 'function' keyword.
>>>  - Add abspath and find_testcases.
>>>  - Make OPT_TEST_CASES a local var.
>>>  - Ensure given testcase ended with .tc.
>>>  - Accept a directory option which has multiple testcases.
>>>  - Change [PASSED]/[FAILED] to [PASS]/[FAIL]
>>>  - Change the basic shell to sh (dash).
>>>
>>> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
>>> ---
>>>  MAINTAINERS                                    |    1 
>>>  tools/testing/selftests/Makefile               |    1 
>>>  tools/testing/selftests/ftrace/Makefile        |    7 +
>>>  tools/testing/selftests/ftrace/README          |   46 +++++++
>>>  tools/testing/selftests/ftrace/ftracetest      |  157 ++++++++++++++++++++++++
>>>  tools/testing/selftests/ftrace/test.d/template |    4 +
>>>  6 files changed, 216 insertions(+)
>>>  create mode 100644 tools/testing/selftests/ftrace/Makefile
>>>  create mode 100644 tools/testing/selftests/ftrace/README
>>>  create mode 100755 tools/testing/selftests/ftrace/ftracetest
>>>  create mode 100644 tools/testing/selftests/ftrace/test.d/template
>>>
>>> diff --git a/MAINTAINERS b/MAINTAINERS
>>> index 5e7866a..29e22db 100644
>>> --- a/MAINTAINERS
>>> +++ b/MAINTAINERS
>>> @@ -9295,6 +9295,7 @@ F:	include/*/ftrace.h
>>>  F:	include/linux/trace*.h
>>>  F:	include/trace/
>>>  F:	kernel/trace/
>>> +F:	tools/testing/selftests/ftrace
>>>  
>>>  TRIVIAL PATCHES
>>>  M:	Jiri Kosina <trivial@kernel.org>
>>> diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
>>> index 36ff2e4..45f145c 100644
>>> --- a/tools/testing/selftests/Makefile
>>> +++ b/tools/testing/selftests/Makefile
>>> @@ -14,6 +14,7 @@ TARGETS += powerpc
>>>  TARGETS += user
>>>  TARGETS += sysctl
>>>  TARGETS += firmware
>>> +TARGETS += ftrace
>>>  
>>>  TARGETS_HOTPLUG = cpu-hotplug
>>>  TARGETS_HOTPLUG += memory-hotplug
>>> diff --git a/tools/testing/selftests/ftrace/Makefile b/tools/testing/selftests/ftrace/Makefile
>>> new file mode 100644
>>> index 0000000..76cc9f1
>>> --- /dev/null
>>> +++ b/tools/testing/selftests/ftrace/Makefile
>>> @@ -0,0 +1,7 @@
>>> +all:
>>> +
>>> +run_tests:
>>> +	@/bin/sh ./ftracetest || echo "ftrace selftests: [FAIL]"
>>> +
>>> +clean:
>>> +	rm -rf logs/*
>>> diff --git a/tools/testing/selftests/ftrace/README b/tools/testing/selftests/ftrace/README
>>> new file mode 100644
>>> index 0000000..ee2cef7
>>> --- /dev/null
>>> +++ b/tools/testing/selftests/ftrace/README
>>> @@ -0,0 +1,46 @@
>>> +Linux Ftrace Testcases
>>> +
>>> +This is a collection of testcases for ftrace tracing feature in the Linux
>>> +kernel. Since ftrace exports interfaces via the debugfs, we just need
>>> +shell scripts for testing. Feel free to add new test cases.
>>> +
>>> +Running the ftrace testcases
>>> +============================
>>> +
>>> +At first, you need to be the root user to run this script.
>>> +To run all testcases:
>>> +
>>> +  $ sudo ./ftracetest
>>> +
>>> +To run specific testcases:
>>> +
>>> +  # ./ftracetest test.d/basic3.tc
>>> +
>>> +Or you can also run testcases under given directory:
>>> +
>>> +  # ./ftracetest test.d/kprobe/
>>> +
>>> +Contributing new testcases
>>> +==========================
>>> +
>>> +Copy test.d/template to your testcase (whose filename must have *.tc
>>> +extension) and rewrite the test description line.
>>> +
>>> + * The working directory of the script is <debugfs>/tracing/.
>>> +
>>> + * Take care with side effects as the tests are run with root privilege.
>>> +
>>> + * The tests should not run for a long period of time (more than 1 min.)
>>> +   These are to be unit tests.
>>> +
>>> + * You can add a directory for your testcases under test.d/ if needed.
>>> +
>>> + * The test cases should run on dash (busybox shell) for testing on
>>> +   minimal cross-build environments.
>>> +
>>> +TODO
>>> +====
>>> +
>>> + * Fancy colored output :)
>>> +
>>> + * Integrate with selftest?
>>
>> The second TODO can be removed since it is moved under selftests??
> 
> Ah, I missed to remove that in this commit. It's already integrated.
> 
> - * Integrate with selftest?
> 
> Oops, and I removed it in 4/4 ...

If you could fix this README, that would be great.

> 
>> If you haven't already done this, could you please make sure it
>> runs without hangs when run from the "make kselftest" target.
> 
> OK, here is the last lines of the test result.
> 
> == Testing sysctl behavior against /proc/sys/vm/swappiness ==
> Writing test file ... ok
> Checking sysctl is not set to test value ... ok
> Writing sysctl from shell ... ok
> Resetting sysctl to original value ... ok
> Writing entire sysctl in single write ... ok
> Writing middle of sysctl after synchronized seek ... Writing beyond end of sysctl ... Writing sysctl with multiple long writes ... ok
> fw_filesystem: [FAIL]
> [1] Basic trace file check      [PASS]
> [2] Basic test for tracers      [PASS]
> [3] Basic trace clock test      [PASS]
> [4] Kprobe dynamic event - adding and removing  [PASS]
> [5] Kprobe dynamic event - busy event check     [PASS]
> 
> # of passed:  5
> # of failed:  0
> # of unresolved:  0
> # of untested:  0
> # of unsupported:  0
> # of xfailed:  0
> # of undefined(test bug):  0
> 
> Hmm, I think it should have a separation line before running ftracetest.
> 

I am not concerned about the spacing.

Thanks for doing running the make kselftest target and sharing
the results. selftests don't get built or run in integ test
rings. I am working on addressing this at the moment.

Steven!

Could you please take this through your tree. You have my

Acked-by: Shuah Khan <shuahkh@osg.samsung.com>

for the selftests Makefile

-- Shuah

-- 
Shuah Khan
Sr. Linux Kernel Developer
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

^ permalink raw reply	[flat|nested] 14+ messages in thread
* Re: [PATCH v4 1/4] ftracetest: Initial commit for ftracetest
  2014-09-19 14:05       ` Shuah Khan
@ 2014-09-19 14:42         ` Steven Rostedt
  2014-09-20 17:33           ` Masami Hiramatsu
  2014-09-20 17:26         ` Masami Hiramatsu
  1 sibling, 1 reply; 14+ messages in thread
From: Steven Rostedt @ 2014-09-19 14:42 UTC (permalink / raw)
  To: Shuah Khan
  Cc: Masami Hiramatsu, Shuah Khan, Tom Zanussi, Oleg Nesterov,
	Namhyung Kim, Fengguang Wu, Ingo Molnar,
	Linux Kernel Mailing List

On Fri, 19 Sep 2014 08:05:23 -0600
Shuah Khan <shuahkh@osg.samsung.com> wrote:


> I am not concerned about the spacing.
> 
> Thanks for doing running the make kselftest target and sharing
> the results. selftests don't get built or run in integ test
> rings. I am working on addressing this at the moment.
> 
> Steven!
> 
> Could you please take this through your tree. You have my
> 
> Acked-by: Shuah Khan <shuahkh@osg.samsung.com>
> 
> for the selftests Makefile

OK, thanks!

I'll start trying to get my own personal tests working here.

It may take some work as my tests may run for 10s of minutes, as they
are more stress tests than a pass/fail thing. But I should have
something other people can use. I'll continue using both my own
personal tests as well as trying the new stuff that gets put here.

-- Steve


^ permalink raw reply	[flat|nested] 14+ messages in thread
* Re: Re: [PATCH v4 1/4] ftracetest: Initial commit for ftracetest
  2014-09-19 14:05       ` Shuah Khan
  2014-09-19 14:42         ` Steven Rostedt
@ 2014-09-20 17:26         ` Masami Hiramatsu
  2014-09-22 16:00           ` Steven Rostedt
  1 sibling, 1 reply; 14+ messages in thread
From: Masami Hiramatsu @ 2014-09-20 17:26 UTC (permalink / raw)
  To: Shuah Khan
  Cc: Steven Rostedt, Shuah Khan, Tom Zanussi, Oleg Nesterov,
	Namhyung Kim, Fengguang Wu, Ingo Molnar,
	Linux Kernel Mailing List

(2014/09/19 23:05), Shuah Khan wrote:
>>>> +TODO
>>>> +====
>>>> +
>>>> + * Fancy colored output :)
>>>> +
>>>> + * Integrate with selftest?
>>>
>>> The second TODO can be removed since it is moved under selftests??
>>
>> Ah, I missed to remove that in this commit. It's already integrated.
>>
>> - * Integrate with selftest?
>>
>> Oops, and I removed it in 4/4 ...
> 
> If you could fix this README, that would be great.

Yeah, but I guess if all of this series are merged, it may be not
such a big flaw...
Steven, would I better resend updates?

>>> If you haven't already done this, could you please make sure it
>>> runs without hangs when run from the "make kselftest" target.
>>
>> OK, here is the last lines of the test result.
>>
>> == Testing sysctl behavior against /proc/sys/vm/swappiness ==
>> Writing test file ... ok
>> Checking sysctl is not set to test value ... ok
>> Writing sysctl from shell ... ok
>> Resetting sysctl to original value ... ok
>> Writing entire sysctl in single write ... ok
>> Writing middle of sysctl after synchronized seek ... Writing beyond end of sysctl ... Writing sysctl with multiple long writes ... ok
>> fw_filesystem: [FAIL]
>> [1] Basic trace file check      [PASS]
>> [2] Basic test for tracers      [PASS]
>> [3] Basic trace clock test      [PASS]
>> [4] Kprobe dynamic event - adding and removing  [PASS]
>> [5] Kprobe dynamic event - busy event check     [PASS]
>>
>> # of passed:  5
>> # of failed:  0
>> # of unresolved:  0
>> # of untested:  0
>> # of unsupported:  0
>> # of xfailed:  0
>> # of undefined(test bug):  0
>>
>> Hmm, I think it should have a separation line before running ftracetest.
>>
> 
> I am not concerned about the spacing.

OK, anyway I can make an update for that afterwords.

> Thanks for doing running the make kselftest target and sharing
> the results. selftests don't get built or run in integ test
> rings. I am working on addressing this at the moment.
> 
> Steven!
> 
> Could you please take this through your tree. You have my
> 
> Acked-by: Shuah Khan <shuahkh@osg.samsung.com>
> 
> for the selftests Makefile

Thanks!

> 
> -- Shuah
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com



^ permalink raw reply	[flat|nested] 14+ messages in thread
* Re: [PATCH v4 1/4] ftracetest: Initial commit for ftracetest
  2014-09-19 14:42         ` Steven Rostedt
@ 2014-09-20 17:33           ` Masami Hiramatsu
  0 siblings, 0 replies; 14+ messages in thread
From: Masami Hiramatsu @ 2014-09-20 17:33 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Shuah Khan, Shuah Khan, Tom Zanussi, Oleg Nesterov, Namhyung Kim,
	Fengguang Wu, Ingo Molnar, Linux Kernel Mailing List

(2014/09/19 23:42), Steven Rostedt wrote:
> On Fri, 19 Sep 2014 08:05:23 -0600
> Shuah Khan <shuahkh@osg.samsung.com> wrote:
> 
> 
>> I am not concerned about the spacing.
>>
>> Thanks for doing running the make kselftest target and sharing
>> the results. selftests don't get built or run in integ test
>> rings. I am working on addressing this at the moment.
>>
>> Steven!
>>
>> Could you please take this through your tree. You have my
>>
>> Acked-by: Shuah Khan <shuahkh@osg.samsung.com>
>>
>> for the selftests Makefile
> 
> OK, thanks!
> 
> I'll start trying to get my own personal tests working here.

Great :)

> 
> It may take some work as my tests may run for 10s of minutes, as they
> are more stress tests than a pass/fail thing. But I should have
> something other people can use. I'll continue using both my own
> personal tests as well as trying the new stuff that gets put here.

I think you can also put your stress tests in the ftrace/ directory
(or just make a sub-directory for them). Even ftracetest doesn't
run them, we can update Makefile so that kselftest can run them.
For example, I will add a testcase for my IPMODFIY as a separated
test(ftrace/ipmodify), since it involves some special kernel modules
and doesn't use ftrace debugfs interface but tests ftrace(function trace).

Thank you,


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com



^ permalink raw reply	[flat|nested] 14+ messages in thread
* Re: [PATCH v4 4/4] ftracetest: Add POSIX.3 standard and XFAIL result codes
  2014-09-18 12:05 ` [PATCH v4 4/4] ftracetest: Add POSIX.3 standard and XFAIL result codes Masami Hiramatsu
@ 2014-09-22 15:47   ` Namhyung Kim
  0 siblings, 0 replies; 14+ messages in thread
From: Namhyung Kim @ 2014-09-22 15:47 UTC (permalink / raw)
  To: Masami Hiramatsu
  Cc: Shuah Khan, Tom Zanussi, Oleg Nesterov, Steven Rostedt,
	Fengguang Wu, Ingo Molnar, Linux Kernel Mailing List

Hi Masami,

2014-09-18 (목), 12:05 +0000, Masami Hiramatsu:
> Add XFAIL and POSIX 1003.3 standard codes (UNRESOLVED/
> UNTESTED/UNSUPPORTED) as result codes. These are used for the
> results that test case is expected to fail or unsupported
> feature (by config).
> 
> To return these result code, this introduces exit_unresolved,
> exit_untested, exit_unsupported and exit_xfail functions,
> which use real-time signals to notify the result code to
> ftracetest.
> 
> This also set "errexit" option for the testcases, so that
> the tests don't need to exit explicitly.

Looks much better than before!

Thanks,
Namhyung


> 
> Changes in v4:
>  - Remove XPASS and short-names.
>  - Use real-time signals for result code.
>  - Simplify test cases by using errexit option.
>  - Add sample scripts for result code.
> 
> Changes in v3:
>  - Add UNRESOLVED/UNTESTED codes.
>  - Fix to handle undefined codes.
>  - Add a document about return codes.
>  - Fix to show failure logs.
>  - Don't use -e option for echo since dash doesn't support it.
> 
> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> ---
>  tools/testing/selftests/ftrace/README              |   38 ++++++
>  tools/testing/selftests/ftrace/ftracetest          |  122 ++++++++++++++++++--
>  tools/testing/selftests/ftrace/samples/fail.tc     |    4 +
>  tools/testing/selftests/ftrace/samples/pass.tc     |    3 
>  .../testing/selftests/ftrace/samples/unresolved.tc |    4 +
>  .../selftests/ftrace/samples/unsupported.tc        |    3 
>  tools/testing/selftests/ftrace/samples/untested.tc |    3 
>  tools/testing/selftests/ftrace/samples/xfail.tc    |    3 
>  .../selftests/ftrace/test.d/00basic/basic2.tc      |    3 
>  .../selftests/ftrace/test.d/00basic/basic3.tc      |    6 -
>  .../ftrace/test.d/kprobe/add_and_remove.tc         |   12 +-
>  .../selftests/ftrace/test.d/kprobe/busy_check.tc   |   15 +-
>  tools/testing/selftests/ftrace/test.d/template     |    5 +
>  13 files changed, 187 insertions(+), 34 deletions(-)
>  create mode 100644 tools/testing/selftests/ftrace/samples/fail.tc
>  create mode 100644 tools/testing/selftests/ftrace/samples/pass.tc
>  create mode 100644 tools/testing/selftests/ftrace/samples/unresolved.tc
>  create mode 100644 tools/testing/selftests/ftrace/samples/unsupported.tc
>  create mode 100644 tools/testing/selftests/ftrace/samples/untested.tc
>  create mode 100644 tools/testing/selftests/ftrace/samples/xfail.tc
> 
> diff --git a/tools/testing/selftests/ftrace/README b/tools/testing/selftests/ftrace/README
> index ee2cef7..182e76f 100644
> --- a/tools/testing/selftests/ftrace/README
> +++ b/tools/testing/selftests/ftrace/README
> @@ -38,9 +38,45 @@ extension) and rewrite the test description line.
>   * The test cases should run on dash (busybox shell) for testing on
>     minimal cross-build environments.
>  
> + * Note that the tests are run with "set -e" (errexit) option. If any
> +   command fails, the test will be terminated immediately.
> +
> + * The tests can return some result codes instead of pass or fail by
> +   using exit_unresolved, exit_untested, exit_unsupported and exit_xfail.
> +
> +Result code
> +===========
> +
> +Ftracetest supports following result codes.
> +
> + * PASS: The test succeeded as expected. The test which exits with 0 is
> +         counted as passed test.
> +
> + * FAIL: The test failed, but was expected to succeed. The test which exits
> +         with !0 is counted as failed test.
> +
> + * UNRESOLVED: The test produced unclear or intermidiate results.
> +             for example, the test was interrupted
> +                       or the test depends on a previous test, which failed.
> +                       or the test was set up incorrectly
> +             The test which is in above situation, must call exit_unresolved.
> +
> + * UNTESTED: The test was not run, currently just a placeholder.
> +             In this case, the test must call exit_untested.
> +
> + * UNSUPPORTED: The test failed because of lack of feature.
> +               In this case, the test must call exit_unsupported.
> +
> + * XFAIL: The test failed, and was expected to fail.
> +          To return XFAIL, call exit_xfail from the test.
> +
> +There are some sample test scripts for result code under samples/.
> +You can also run samples as below:
> +
> +  # ./ftracetest samples/
> +
>  TODO
>  ====
>  
>   * Fancy colored output :)
>  
> - * Integrate with selftest?
> diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
> index 02f4e7a..2f7f8da0 100755
> --- a/tools/testing/selftests/ftrace/ftracetest
> +++ b/tools/testing/selftests/ftrace/ftracetest
> @@ -112,22 +112,105 @@ catlog() { #file
>  }
>  
>  # Testcase management
> +# Test result codes - Dejagnu extended code
> +PASS=0	# The test succeeded.
> +FAIL=1	# The test failed, but was expected to succeed.
> +UNRESOLVED=2  # The test produced indeterminate results. (e.g. interrupted)
> +UNTESTED=3    # The test was not run, currently just a placeholder.
> +UNSUPPORTED=4 # The test failed because of lack of feature.
> +XFAIL=5	# The test failed, and was expected to fail.
> +
> +# Accumulations
>  PASSED_CASES=
>  FAILED_CASES=
> +UNRESOLVED_CASES=
> +UNTESTED_CASES=
> +UNSUPPORTED_CASES=
> +XFAILED_CASES=
> +UNDEFINED_CASES=
> +
>  CASENO=0
>  testcase() { # testfile
>    CASENO=$((CASENO+1))
>    prlog -n "[$CASENO]"`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
>  }
> -failed() {
> -  prlog "	[FAIL]"
> -  FAILED_CASES="$FAILED_CASES $CASENO"
> +
> +eval_result() { # retval sigval
> +  local retval=$2
> +  if [ $2 -eq 0 ]; then
> +    test $1 -ne 0 && retval=$FAIL
> +  fi
> +  case $retval in
> +    $PASS)
> +      prlog "	[PASS]"
> +      PASSED_CASES="$PASSED_CASES $CASENO"
> +      return 0
> +    ;;
> +    $FAIL)
> +      prlog "	[FAIL]"
> +      FAILED_CASES="$FAILED_CASES $CASENO"
> +      return 1 # this is a bug.
> +    ;;
> +    $UNRESOLVED)
> +      prlog "	[UNRESOLVED]"
> +      UNRESOLVED_CASES="$UNRESOLVED_CASES $CASENO"
> +      return 1 # this is a kind of bug.. something happened.
> +    ;;
> +    $UNTESTED)
> +      prlog "	[UNTESTED]"
> +      UNTESTED_CASES="$UNTESTED_CASES $CASENO"
> +      return 0
> +    ;;
> +    $UNSUPPORTED)
> +      prlog "	[UNSUPPORTED]"
> +      UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO"
> +      return 0 # this is not a bug
> +    ;;
> +    $XFAIL)
> +      prlog "	[XFAIL]"
> +      XFAILED_CASES="$XFAILED_CASES $CASENO"
> +      return 0
> +    ;;
> +    *)
> +      prlog "	[UNDEFINED]"
> +      UNDEFINED_CASES="$UNDEFINED_CASES $CASENO"
> +      return 1 # this must be a test bug
> +    ;;
> +  esac
> +}
> +
> +# Signal handling for result codes
> +SIG_RESULT=
> +SIG_BASE=36	# Use realtime signals
> +SIG_PID=$$
> +
> +SIG_UNRESOLVED=$((SIG_BASE + UNRESOLVED))
> +exit_unresolved () {
> +  kill -s $SIG_UNRESOLVED $SIG_PID
> +  exit 0
> +}
> +trap 'SIG_RESULT=$UNRESOLVED' $SIG_UNRESOLVED
> +
> +SIG_UNTESTED=$((SIG_BASE + UNTESTED))
> +exit_untested () {
> +  kill -s $SIG_UNTESTED $SIG_PID
> +  exit 0
>  }
> -passed() {
> -  prlog "	[PASS]"
> -  PASSED_CASES="$PASSED_CASES $CASENO"
> +trap 'SIG_RESULT=$UNTESTED' $SIG_UNTESTED
> +
> +SIG_UNSUPPORTED=$((SIG_BASE + UNSUPPORTED))
> +exit_unsupported () {
> +  kill -s $SIG_UNSUPPORTED $SIG_PID
> +  exit 0
>  }
> +trap 'SIG_RESULT=$UNSUPPORTED' $SIG_UNSUPPORTED
>  
> +SIG_XFAIL=$((SIG_BASE + XFAIL))
> +exit_xfail () {
> +  kill -s $SIG_XFAIL $SIG_PID
> +  exit 0
> +}
> +trap 'SIG_RESULT=$XFAIL' $SIG_XFAIL
>  
>  # Run one test case
>  run_test() { # testfile
> @@ -135,14 +218,16 @@ run_test() { # testfile
>    local testlog=`mktemp --tmpdir=$LOG_DIR ${testname}-XXXXXX.log`
>    testcase $1
>    echo "execute: "$1 > $testlog
> -  (cd $TRACING_DIR; set -x ; . $1) >> $testlog 2>&1
> -  ret=$?
> -  if [ $ret -ne 0 ]; then
> -    failed
> -    catlog $testlog
> -  else
> -    passed
> +  SIG_RESULT=0
> +  # setup PID and PPID, $$ is not updated.
> +  (cd $TRACING_DIR; read PID _ < /proc/self/stat ;
> +   set -e; set -x; . $1) >> $testlog 2>&1
> +  eval_result $? $SIG_RESULT
> +  if [ $? -eq 0 ]; then
> +    # Remove test log if the test was done as it was expected.
>      [ $KEEP_LOG -eq 0 ] && rm $testlog
> +  else
> +    catlog $testlog
>    fi
>  }
>  
> @@ -150,8 +235,15 @@ run_test() { # testfile
>  for t in $TEST_CASES; do
>    run_test $t
>  done
> +
>  prlog ""
>  prlog "# of passed: " `echo $PASSED_CASES | wc -w`
>  prlog "# of failed: " `echo $FAILED_CASES | wc -w`
> -
> -test -z "$FAILED_CASES" # if no error, return 0
> +prlog "# of unresolved: " `echo $UNRESOLVED_CASES | wc -w`
> +prlog "# of untested: " `echo $UNTESTED_CASES | wc -w`
> +prlog "# of unsupported: " `echo $UNSUPPORTED_CASES | wc -w`
> +prlog "# of xfailed: " `echo $XFAILED_CASES | wc -w`
> +prlog "# of undefined(test bug): " `echo $UNDEFINED_CASES | wc -w`
> +
> +# if no error, return 0
> +test -z "$FAILED_CASES$UNDEFINED_CASES$UNRESOLVED_CASES"
> diff --git a/tools/testing/selftests/ftrace/samples/fail.tc b/tools/testing/selftests/ftrace/samples/fail.tc
> new file mode 100644
> index 0000000..15e35b9
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/fail.tc
> @@ -0,0 +1,4 @@
> +#!/bin/sh
> +# description: failure-case example
> +cat non-exist-file
> +echo "this is not executed"
> diff --git a/tools/testing/selftests/ftrace/samples/pass.tc b/tools/testing/selftests/ftrace/samples/pass.tc
> new file mode 100644
> index 0000000..d015493
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/pass.tc
> @@ -0,0 +1,3 @@
> +#!/bin/sh
> +# description: pass-case example
> +return 0
> diff --git a/tools/testing/selftests/ftrace/samples/unresolved.tc b/tools/testing/selftests/ftrace/samples/unresolved.tc
> new file mode 100644
> index 0000000..41e99d3
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/unresolved.tc
> @@ -0,0 +1,4 @@
> +#!/bin/sh
> +# description: unresolved-case example
> +trap exit_unresolved INT
> +kill -INT $PID
> diff --git a/tools/testing/selftests/ftrace/samples/unsupported.tc b/tools/testing/selftests/ftrace/samples/unsupported.tc
> new file mode 100644
> index 0000000..45910ff
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/unsupported.tc
> @@ -0,0 +1,3 @@
> +#!/bin/sh
> +# description: unsupported-case example
> +exit_unsupported
> diff --git a/tools/testing/selftests/ftrace/samples/untested.tc b/tools/testing/selftests/ftrace/samples/untested.tc
> new file mode 100644
> index 0000000..35a4594
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/untested.tc
> @@ -0,0 +1,3 @@
> +#!/bin/sh
> +# description: untested-case example
> +exit_untested
> diff --git a/tools/testing/selftests/ftrace/samples/xfail.tc b/tools/testing/selftests/ftrace/samples/xfail.tc
> new file mode 100644
> index 0000000..9dd3953
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/xfail.tc
> @@ -0,0 +1,3 @@
> +#!/bin/sh
> +# description: xfail-case example
> +cat non-exist-file || exit_xfail
> diff --git a/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc b/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
> index b04f30d..bf9a7b0 100644
> --- a/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
> +++ b/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
> @@ -1,6 +1,7 @@
>  #!/bin/sh
>  # description: Basic test for tracers
> +test -f available_tracers
>  for t in `cat available_tracers`; do
> -  echo $t > current_tracer || exit 1
> +  echo $t > current_tracer
>  done
>  echo nop > current_tracer
> diff --git a/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc b/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
> index 0c1a3a2..bde6625 100644
> --- a/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
> +++ b/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
> @@ -1,8 +1,8 @@
>  #!/bin/sh
>  # description: Basic trace clock test
> -[ -f trace_clock ] || exit 1
> +test -f trace_clock
>  for c in `cat trace_clock | tr  -d \[\]`; do
> -  echo $c > trace_clock || exit 1
> -  grep '\['$c'\]' trace_clock || exit 1
> +  echo $c > trace_clock
> +  grep '\['$c'\]' trace_clock
>  done
>  echo local > trace_clock
> diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc b/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
> index 5ddfb47..1b8b665 100644
> --- a/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
> +++ b/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
> @@ -1,11 +1,11 @@
>  #!/bin/sh
>  # description: Kprobe dynamic event - adding and removing
>  
> -[ -f kprobe_events ] || exit 1
> +[ -f kprobe_events ] || exit_unsupported # this is configurable
>  
> -echo 0 > events/enable || exit 1
> -echo > kprobe_events || exit 1
> -echo p:myevent do_fork > kprobe_events || exit 1
> -grep myevent kprobe_events || exit 1
> -[ -d events/kprobes/myevent ] || exit 1
> +echo 0 > events/enable
> +echo > kprobe_events
> +echo p:myevent do_fork > kprobe_events
> +grep myevent kprobe_events
> +test -d events/kprobes/myevent
>  echo > kprobe_events
> diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc b/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
> index 588fde97..b55c840 100644
> --- a/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
> +++ b/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
> @@ -1,14 +1,13 @@
>  #!/bin/sh
>  # description: Kprobe dynamic event - busy event check
>  
> -[ -f kprobe_events ] || exit 1
> +[ -f kprobe_events ] || exit_unsupported
>  
> -echo 0 > events/enable || exit 1
> -echo > kprobe_events || exit 1
> -echo p:myevent do_fork > kprobe_events || exit 1
> -[ -d events/kprobes/myevent ] || exit 1
> -echo 1 > events/kprobes/myevent/enable || exit 1
> +echo 0 > events/enable
> +echo > kprobe_events
> +echo p:myevent do_fork > kprobe_events
> +test -d events/kprobes/myevent
> +echo 1 > events/kprobes/myevent/enable
>  echo > kprobe_events && exit 1 # this must fail
> -echo 0 > events/kprobes/myevent/enable || exit 1
> +echo 0 > events/kprobes/myevent/enable
>  echo > kprobe_events # this must succeed
> -
> diff --git a/tools/testing/selftests/ftrace/test.d/template b/tools/testing/selftests/ftrace/test.d/template
> index ce5f735..5448f7a 100644
> --- a/tools/testing/selftests/ftrace/test.d/template
> +++ b/tools/testing/selftests/ftrace/test.d/template
> @@ -1,4 +1,9 @@
>  #!/bin/sh
>  # description: %HERE DESCRIBE WHAT THIS DOES%
>  # you have to add ".tc" extention for your testcase file
> +# Note that all tests are run with "errexit" option.
> +
>  exit 0 # Return 0 if the test is passed, otherwise return !0
> +# If the test could not run because of lack of feature, call exit_unsupported
> +# If the test returned unclear results, call exit_unresolved
> +# If the test is a dummy, or a placeholder, call exit_untested
> 
> 


-- 
Regards,
Namhyung Kim



^ permalink raw reply	[flat|nested] 14+ messages in thread
* Re: [PATCH v4 1/4] ftracetest: Initial commit for ftracetest
  2014-09-20 17:26         ` Masami Hiramatsu
@ 2014-09-22 16:00           ` Steven Rostedt
  2014-09-22 23:37             ` Masami Hiramatsu
  0 siblings, 1 reply; 14+ messages in thread
From: Steven Rostedt @ 2014-09-22 16:00 UTC (permalink / raw)
  To: Masami Hiramatsu
  Cc: Shuah Khan, Shuah Khan, Tom Zanussi, Oleg Nesterov, Namhyung Kim,
	Fengguang Wu, Ingo Molnar, Linux Kernel Mailing List

On Sun, 21 Sep 2014 02:26:37 +0900
Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:


> Steven, would I better resend updates?
> 

Yeah, can you please send a v5 with all the updates, just so I don't
miss anything. I'm still not at 100% capacity and this will help a lot.

Also, can you please add everyone's acked-bys and reviewed-bys that
have posted so far.

Thanks!

-- Steve

^ permalink raw reply	[flat|nested] 14+ messages in thread
* Re: Re: [PATCH v4 1/4] ftracetest: Initial commit for ftracetest
  2014-09-22 16:00           ` Steven Rostedt
@ 2014-09-22 23:37             ` Masami Hiramatsu
  0 siblings, 0 replies; 14+ messages in thread
From: Masami Hiramatsu @ 2014-09-22 23:37 UTC (permalink / raw)
  To: Steven Rostedt
  Cc: Shuah Khan, Shuah Khan, Tom Zanussi, Oleg Nesterov, Namhyung Kim,
	Fengguang Wu, Ingo Molnar, Linux Kernel Mailing List

(2014/09/23 1:00), Steven Rostedt wrote:
> On Sun, 21 Sep 2014 02:26:37 +0900
> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
> 
> 
>> Steven, would I better resend updates?
>>
> 
> Yeah, can you please send a v5 with all the updates, just so I don't
> miss anything. I'm still not at 100% capacity and this will help a lot.
> 
> Also, can you please add everyone's acked-bys and reviewed-bys that
> have posted so far.

OK, I'll update it soon :)

Thank you,

> 
> Thanks!
> 
> -- Steve
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com



^ permalink raw reply	[flat|nested] 14+ messages in thread
end of thread, other threads:[~2014-09-22 23:37 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-09-18 12:04 [PATCH v4 0/4] ftrace: Add a ftrace test collection Masami Hiramatsu
2014-09-18 12:04 ` [PATCH v4 1/4] ftracetest: Initial commit for ftracetest Masami Hiramatsu
2014-09-18 13:07   ` Shuah Khan
2014-09-19  1:03     ` Masami Hiramatsu
2014-09-19 14:05       ` Shuah Khan
2014-09-19 14:42         ` Steven Rostedt
2014-09-20 17:33           ` Masami Hiramatsu
2014-09-20 17:26         ` Masami Hiramatsu
2014-09-22 16:00           ` Steven Rostedt
2014-09-22 23:37             ` Masami Hiramatsu
2014-09-18 12:04 ` [PATCH v4 2/4] ftracetest: Add ftrace basic testcases Masami Hiramatsu
2014-09-18 12:04 ` [PATCH v4 3/4] ftracetest: Add kprobe " Masami Hiramatsu
2014-09-18 12:05 ` [PATCH v4 4/4] ftracetest: Add POSIX.3 standard and XFAIL result codes Masami Hiramatsu
2014-09-22 15:47   ` Namhyung Kim

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).