[LTP] [RFC PATCH v3 1/2] tst_test.sh: Add TST_TEST_DATA and TST_TEST_DATA_IFS

[LTP] [RFC PATCH v3 1/2] tst_test.sh: Add TST_TEST_DATA and TST_TEST_DATA_IFS

[Petr Vorel]

This is specific only for shell.

Each run of tst_run gets passed sequence number of a test being run
as '$1' and corresponding part of data from TST_TEST_DATA as '$2'.

Also create internal functions tst_run_tests() and tst_run_test()
to reduce duplicity.

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
Changes v2->v3:
* Don't pass $TST_TEST_DATA as whole argument for $TST_CNT (Cyril)
* Create tst_run_tests() for cleanup (Cyril).
* Create tst_run_test() - more cleanup

Maybe we should try to "hide" somehow these API functions (underscore at
the front?).


Kind regards,
Petr
---
 doc/test-writing-guidelines.txt | 74 +++++++++++++++++++++++++++++++++++++----
 testcases/lib/tst_test.sh       | 61 +++++++++++++++++++++------------
 2 files changed, 106 insertions(+), 29 deletions(-)

diff --git a/doc/test-writing-guidelines.txt b/doc/test-writing-guidelines.txt
index de47443eb..531b828ff 100644
--- a/doc/test-writing-guidelines.txt
+++ b/doc/test-writing-guidelines.txt
@@ -1442,12 +1442,12 @@ TST_CNT=2
 
 test1()
 {
-	tst_res TPASS "Test 1 passed"
+	tst_res TPASS "Test $1 passed"
 }
 
 test2()
 {
-	tst_res TPASS "Test 2 passed"
+	tst_res TPASS "Test $1 passed"
 }
 
 tst_run
@@ -1455,7 +1455,8 @@ tst_run
 
 If '$TST_CNT' is set, the test library looks if there are functions named
 '$\{TST_TESTFUNC\}1', ..., '$\{TST_TESTFUNC\}$\{TST_CNT\}' and if these are
-found they are executed one by one.
+found they are executed one by one. The test number is passed to it in the '$1'.
+
 
 [source,sh]
 -------------------------------------------------------------------------------
@@ -1471,8 +1472,8 @@ TST_CNT=2
 do_test()
 {
 	case $1 in
-	1) tst_res TPASS "Test 1 passed";;
-	2) tst_res TPASS "Test 2 passed";;
+	1) tst_res TPASS "Test $1 passed";;
+	2) tst_res TPASS "Test $1 passed";;
 	esac
 }
 
@@ -1483,6 +1484,65 @@ Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc.,
 the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed
 to it in the '$1'.
 
+[source,sh]
+-------------------------------------------------------------------------------
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA and
+# $TST_TEST_DATA_IFS
+#
+
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo:bar:d dd"
+TST_TEST_DATA_IFS=":"
+. tst_test.sh
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo'
+# test 2 TPASS: Test 2 passed with data 'bar'
+# test 3 TPASS: Test 3 passed with data 'd dd'
+
+-------------------------------------------------------------------------------
+
+It's possible to pass data for function with '$TST_TEST_DATA'. Optional
+'$TST_TEST_DATA_IFS' is used for splitting, default value is space.
+
+[source,sh]
+-------------------------------------------------------------------------------
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT
+#
+
+TST_TESTFUNC=do_test
+TST_CNT=2
+TST_TEST_DATA="foo:bar:d dd"
+. tst_test.sh
+
+do_test()
+{
+	case $1 in
+	1) tst_res TPASS "Test $1 passed with data '$2'";;
+	2) tst_res TPASS "Test $1 passed with data '$2'";;
+	esac
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
+# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
+
+-------------------------------------------------------------------------------
+When '$TST_TEST_DATA' is used with '$TST_CNT', it's passed as whole string in
+'$2' ($1 is for the test number), '$TST_TEST_DATA_IFS' is ignored. Similar
+would be when using these variables with separate functions.
+
 2.3.2 Library variables
 ^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -1587,8 +1647,8 @@ these can be listed with passing help '-h' option to any test.
 The function that prints the usage is passed in '$TST_USAGE', the help for
 the options implemented in the library is appended when usage is printed.
 
-Lastly the fucntion '$PARSE_ARGS' is called with the option name in '$1' and,
-if option has argument, its value in '$2'.
+Lastly the fucntion '$PARSE_ARGS' is called with the option name in the '$1'
+and, if option has argument, its value in the '$2'.
 
 [source,sh]
 -------------------------------------------------------------------------------
diff --git a/testcases/lib/tst_test.sh b/testcases/lib/tst_test.sh
index 464c4c41e..7680aa462 100644
--- a/testcases/lib/tst_test.sh
+++ b/testcases/lib/tst_test.sh
@@ -246,7 +246,7 @@ tst_rescmp()
 
 tst_run()
 {
-	local tst_i
+	local tst_i tst_data
 
 	if [ -n "$TST_TEST_PATH" ]; then
 		for tst_i in $(grep TST_ "$TST_TEST_PATH" | sed 's/.*TST_//; s/[="} \t\/:`].*//'); do
@@ -255,7 +255,7 @@ tst_run()
 			OPTS|USAGE|PARSE_ARGS|POS_ARGS);;
 			NEEDS_ROOT|NEEDS_TMPDIR|NEEDS_DEVICE|DEVICE);;
 			NEEDS_CMDS|NEEDS_MODULE|MODPATH|DATAROOT);;
-			IPV6);;
+			IPV6|TEST_DATA|TEST_DATA_IFS);;
 			*) tst_res TWARN "Reserved variable TST_$tst_i used!";;
 			esac
 		done
@@ -348,27 +348,17 @@ tst_run()
 
 	#TODO check that test reports some results for each test function call
 	while [ $TST_ITERATIONS -gt 0 ]; do
-		if [ -n "$TST_CNT" ]; then
-			if type test1 > /dev/null 2>&1; then
-				for tst_i in $(seq $TST_CNT); do
-					local res=$(tst_resstr)
-					$TST_TESTFUNC$tst_i
-					tst_rescmp "$res"
-					TST_COUNT=$((TST_COUNT+1))
-				done
-			else
-				for tst_i in $(seq $TST_CNT); do
-					local res=$(tst_resstr)
-					$TST_TESTFUNC $tst_i
-					tst_rescmp "$res"
-					TST_COUNT=$((TST_COUNT+1))
-				done
-			fi
+		if [ -n "$TST_TEST_DATA" ]; then
+			tst_i=1
+			tst_check_cmds cut
+			while true; do
+				tst_data="$(echo "$TST_TEST_DATA" | cut -d"$TST_TEST_DATA_IFS" -f$tst_i)"
+				[ -z "$tst_data" ] && break
+				tst_run_tests "$tst_data"
+				tst_i=$((tst_i+1))
+			done
 		else
-			local res=$(tst_resstr)
-			$TST_TESTFUNC
-			tst_rescmp "$res"
-			TST_COUNT=$((TST_COUNT+1))
+			tst_run_tests
 		fi
 		TST_ITERATIONS=$((TST_ITERATIONS-1))
 	done
@@ -376,6 +366,31 @@ tst_run()
 	tst_do_exit
 }
 
+tst_run_tests()
+{
+	local tst_data="$1"
+	local tst_i
+
+	for tst_i in $(seq ${TST_CNT:-1}); do
+		if type test1 > /dev/null 2>&1; then
+			tst_run_test "$TST_TESTFUNC$tst_i" $tst_i "$tst_data"
+		else
+			tst_run_test "$TST_TESTFUNC" $tst_i "$tst_data"
+		fi
+	done
+}
+
+tst_run_test()
+{
+	local res=$(tst_resstr)
+	local tst_fnc="$1"
+	shift
+
+	$tst_fnc "$@"
+	tst_rescmp "$res"
+	TST_COUNT=$((TST_COUNT+1))
+}
+
 if [ -z "$TST_ID" ]; then
 	filename=$(basename $0)
 	TST_ID=${filename%%.*}
@@ -400,6 +415,8 @@ if [ -z "$TST_NO_DEFAULT_RUN" ]; then
 		tst_brk TBROK "TST_TESTFUNC is not defined"
 	fi
 
+	TST_TEST_DATA_IFS="${TST_TEST_DATA_IFS:- }"
+
 	if [ -n "$TST_CNT" ]; then
 		if ! tst_is_int "$TST_CNT"; then
 			tst_brk TBROK "TST_CNT must be integer"
-- 
2.16.3

[LTP] [RFC PATCH v3 2/2] lib: Add tests

[Petr Vorel]

Not-signed-off-by: Petr Vorel <pvorel@suse.cz>
---
This is not intended to be merged. It's just a helper which you can use
for testing the previous patch.

(Side note: if we ever want to test tst_test.sh, it'd be possible to
dump it into file and compare with saved output. It'd require to remove
lines with 'stime=')
---
 lib/newlib_tests/Makefile                          |  1 +
 lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh | 22 ++++++++++++
 lib/newlib_tests/test.TST_TEST.TST_CNT.sh          | 20 +++++++++++
 lib/newlib_tests/test.TST_TEST.getopts.sh          | 40 +++++++++++++++++++++
 lib/newlib_tests/test.TST_TEST.sh                  | 25 +++++++++++++
 .../test.TST_TEST_DATA.TST_CNT.separate.sh         | 26 ++++++++++++++
 lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh     | 24 +++++++++++++
 lib/newlib_tests/test.TST_TEST_DATA.getopts.sh     | 41 +++++++++++++++++++++
 lib/newlib_tests/test.TST_TEST_DATA.sh             | 18 ++++++++++
 lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh | 42 ++++++++++++++++++++++
 lib/newlib_tests/test.TST_TEST_DATA_IFS.sh         | 21 +++++++++++
 runtest/test                                       | 13 +++++++
 12 files changed, 293 insertions(+)
 create mode 100755 lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh
 create mode 100755 lib/newlib_tests/test.TST_TEST.TST_CNT.sh
 create mode 100755 lib/newlib_tests/test.TST_TEST.getopts.sh
 create mode 100755 lib/newlib_tests/test.TST_TEST.sh
 create mode 100755 lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.separate.sh
 create mode 100755 lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh
 create mode 100755 lib/newlib_tests/test.TST_TEST_DATA.getopts.sh
 create mode 100755 lib/newlib_tests/test.TST_TEST_DATA.sh
 create mode 100755 lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh
 create mode 100755 lib/newlib_tests/test.TST_TEST_DATA_IFS.sh
 create mode 100644 runtest/test

diff --git a/lib/newlib_tests/Makefile b/lib/newlib_tests/Makefile
index afa09373e..cc20920be 100644
--- a/lib/newlib_tests/Makefile
+++ b/lib/newlib_tests/Makefile
@@ -15,5 +15,6 @@ ifeq ($(ANDROID),1)
 FILTER_OUT_MAKE_TARGETS	+= test08
 endif
 
+INSTALL_TARGETS		:= *.sh
 
 include $(top_srcdir)/include/mk/generic_leaf_target.mk
diff --git a/lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh b/lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh
new file mode 100755
index 000000000..e26e2cb14
--- /dev/null
+++ b/lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh
@@ -0,0 +1,22 @@
+#!/bin/sh
+#
+# Example test with tests in separate functions
+#
+
+TST_TESTFUNC=test
+TST_CNT=2
+. tst_test.sh
+
+test1()
+{
+	tst_res TPASS "Test $1 passed with no data ('$2')"
+}
+
+test2()
+{
+	tst_res TPASS "Test $1 passed with no data ('$2')"
+}
+
+tst_run
+# test 1 TPASS: Test 1 passed with no data ('')
+# test 2 TPASS: Test 2 passed with no data ('')
diff --git a/lib/newlib_tests/test.TST_TEST.TST_CNT.sh b/lib/newlib_tests/test.TST_TEST.TST_CNT.sh
new file mode 100755
index 000000000..d84457fea
--- /dev/null
+++ b/lib/newlib_tests/test.TST_TEST.TST_CNT.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+#
+# Example test with tests in a single function
+#
+
+TST_TESTFUNC=do_test
+TST_CNT=2
+. tst_test.sh
+
+do_test()
+{
+	case $1 in
+	1) tst_res TPASS "Test $1 passed with no data ('$2')";;
+	2) tst_res TPASS "Test $1 passed with no data ('$2')";;
+	esac
+}
+
+tst_run
+# test 1 TPASS: Test 1 passed with no data ('')
+# test 2 TPASS: Test 2 passed with no data ('')
diff --git a/lib/newlib_tests/test.TST_TEST.getopts.sh b/lib/newlib_tests/test.TST_TEST.getopts.sh
new file mode 100755
index 000000000..0ba848a0a
--- /dev/null
+++ b/lib/newlib_tests/test.TST_TEST.getopts.sh
@@ -0,0 +1,40 @@
+#!/bin/sh
+#
+# Optional test command line parameters
+#
+
+TST_OPTS="af:"
+TST_USAGE=usage
+TST_PARSE_ARGS=parse_args
+TST_TESTFUNC=do_test
+
+. tst_test.sh
+
+ALTERNATIVE=0
+MODE="foo"
+
+usage()
+{
+	cat << EOF
+usage: $0 [-a] [-f <foo|bar>]
+
+OPTIONS
+-a     Enable support for alternative foo
+-f     Specify foo or bar mode
+EOF
+}
+
+parse_args()
+{
+	case $1 in
+	a) ALTERNATIVE=1;;
+	f) MODE="$2";;
+	esac
+}
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
+}
+
+tst_run
diff --git a/lib/newlib_tests/test.TST_TEST.sh b/lib/newlib_tests/test.TST_TEST.sh
new file mode 100755
index 000000000..6ecae710d
--- /dev/null
+++ b/lib/newlib_tests/test.TST_TEST.sh
@@ -0,0 +1,25 @@
+#!/bin/sh
+#
+# This is a basic test for true shell buildin
+#
+
+TST_TESTFUNC=do_test
+. tst_test.sh
+
+do_test()
+{
+	true
+	ret=$?
+
+	tst_res TINFO "Test $1 passed with no data ('$2')"
+
+	if [ $ret -eq 0 ]; then
+		tst_res TPASS "true returned 0"
+	else
+		tst_res TFAIL "true returned $ret"
+	fi
+}
+
+tst_run
+# test 1 TINFO: Test 1 passed with no data ('')
+# test 1 TPASS: true returned 0
diff --git a/lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.separate.sh b/lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.separate.sh
new file mode 100755
index 000000000..9a81ad27c
--- /dev/null
+++ b/lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.separate.sh
@@ -0,0 +1,26 @@
+#!/bin/sh
+#
+# Example test with tests in separate functions, using $TST_TEST_DATA and $TST_CNT
+#
+
+TST_TESTFUNC=test
+TST_CNT=2
+TST_TEST_DATA="foo:bar:d dd"
+. tst_test.sh
+
+test1()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+test2()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
+# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
+# test 3 TPASS: Test 1 passed with data 'dd'
+# test 4 TPASS: Test 2 passed with data 'dd'
diff --git a/lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh b/lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh
new file mode 100755
index 000000000..798ad9621
--- /dev/null
+++ b/lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT
+#
+
+TST_TESTFUNC=do_test
+TST_CNT=2
+TST_TEST_DATA="foo:bar:d dd"
+. tst_test.sh
+
+do_test()
+{
+	case $1 in
+	1) tst_res TPASS "Test $1 passed with data '$2'";;
+	2) tst_res TPASS "Test $1 passed with data '$2'";;
+	esac
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
+# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
+# test 3 TPASS: Test 1 passed with data 'dd'
+# test 4 TPASS: Test 2 passed with data 'dd'
diff --git a/lib/newlib_tests/test.TST_TEST_DATA.getopts.sh b/lib/newlib_tests/test.TST_TEST_DATA.getopts.sh
new file mode 100755
index 000000000..1cd150618
--- /dev/null
+++ b/lib/newlib_tests/test.TST_TEST_DATA.getopts.sh
@@ -0,0 +1,41 @@
+#!/bin/sh
+#
+# Optional test command line parameters
+#
+
+TST_OPTS="af:"
+TST_USAGE=usage
+TST_PARSE_ARGS=parse_args
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo0:bar:d dd"
+
+. tst_test.sh
+
+ALTERNATIVE=0
+MODE="foo"
+
+usage()
+{
+	cat << EOF
+usage: $0 [-a] [-f <foo|bar>]
+
+OPTIONS
+-a     Enable support for alternative foo
+-f     Specify foo or bar mode
+EOF
+}
+
+parse_args()
+{
+	case $1 in
+	a) ALTERNATIVE=1;;
+	f) MODE="$2";;
+	esac
+}
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
+}
+
+tst_run
diff --git a/lib/newlib_tests/test.TST_TEST_DATA.sh b/lib/newlib_tests/test.TST_TEST_DATA.sh
new file mode 100755
index 000000000..4cb51c669
--- /dev/null
+++ b/lib/newlib_tests/test.TST_TEST_DATA.sh
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA
+#
+
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo:bar:d dd"
+. tst_test.sh
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
+# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
diff --git a/lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh b/lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh
new file mode 100755
index 000000000..59dbd9598
--- /dev/null
+++ b/lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh
@@ -0,0 +1,42 @@
+#!/bin/sh
+#
+# Optional test command line parameters
+#
+
+TST_OPTS="af:"
+TST_USAGE=usage
+TST_PARSE_ARGS=parse_args
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo0:bar:d dd"
+TST_TEST_DATA_IFS=":"
+
+. tst_test.sh
+
+ALTERNATIVE=0
+MODE="foo"
+
+usage()
+{
+	cat << EOF
+usage: $0 [-a] [-f <foo|bar>]
+
+OPTIONS
+-a     Enable support for alternative foo
+-f     Specify foo or bar mode
+EOF
+}
+
+parse_args()
+{
+	case $1 in
+	a) ALTERNATIVE=1;;
+	f) MODE="$2";;
+	esac
+}
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
+}
+
+tst_run
diff --git a/lib/newlib_tests/test.TST_TEST_DATA_IFS.sh b/lib/newlib_tests/test.TST_TEST_DATA_IFS.sh
new file mode 100755
index 000000000..52a2e6d47
--- /dev/null
+++ b/lib/newlib_tests/test.TST_TEST_DATA_IFS.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA and
+# $TST_TEST_DATA_IFS
+#
+
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo:bar:d dd"
+TST_TEST_DATA_IFS=":"
+. tst_test.sh
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo'
+# test 2 TPASS: Test 2 passed with data 'bar'
+# test 3 TPASS: Test 3 passed with data 'd dd'
diff --git a/runtest/test b/runtest/test
new file mode 100644
index 000000000..469e3b787
--- /dev/null
+++ b/runtest/test
@@ -0,0 +1,13 @@
+TST_TEST_DATA_IFS test.TST_TEST_DATA_IFS.sh
+TST_TEST_DATA test.TST_TEST_DATA.sh
+TST_TEST_DATA.TST_CNT.separate test.TST_TEST_DATA.TST_CNT.separate.sh
+TST_TEST_DATA.TST_CNT test.TST_TEST_DATA.TST_CNT.sh
+TST_TEST test.TST_TEST.sh
+TST_TEST.TST_CNT.separate test.TST_TEST.TST_CNT.separate.sh
+TST_TEST.TST_CNT test.TST_TEST.TST_CNT.sh
+TST_TEST.getopts test.TST_TEST.getopts.sh
+TST_TEST.getopts_a_f test.TST_TEST.getopts.sh -a -f foo1
+TST_TEST_DATA.getopts test.TST_TEST_DATA.getopts.sh
+TST_TEST_DATA.getopts_a_f test.TST_TEST_DATA.getopts.sh -a -f foo2
+TST_TEST_DATA_IFS.getopts test.TST_TEST_DATA_IFS.getopts.sh
+TST_TEST_DATA_IFS.getopts_a_f test.TST_TEST_DATA_IFS.getopts.sh -a -f foo3
-- 
2.16.3

[LTP] [RFC PATCH v3 1/2] tst_test.sh: Add TST_TEST_DATA and TST_TEST_DATA_IFS

[Cyril Hrubis]

Hi!
> Signed-off-by: Petr Vorel <pvorel@suse.cz>
> ---
> Changes v2->v3:
> * Don't pass $TST_TEST_DATA as whole argument for $TST_CNT (Cyril)
> * Create tst_run_tests() for cleanup (Cyril).
> * Create tst_run_test() - more cleanup
> 
> Maybe we should try to "hide" somehow these API functions (underscore at
> the front?).

Actually this is a good idea, we can follow the python underscore
notation that uses it for private variables and functions. Hence the
counters will become _tst_i, etc.

But please do that in a separate patch.

> ---
>  doc/test-writing-guidelines.txt | 74 +++++++++++++++++++++++++++++++++++++----
>  testcases/lib/tst_test.sh       | 61 +++++++++++++++++++++------------
>  2 files changed, 106 insertions(+), 29 deletions(-)
> 
> diff --git a/doc/test-writing-guidelines.txt b/doc/test-writing-guidelines.txt
> index de47443eb..531b828ff 100644
> --- a/doc/test-writing-guidelines.txt
> +++ b/doc/test-writing-guidelines.txt
> @@ -1442,12 +1442,12 @@ TST_CNT=2
>  
>  test1()
>  {
> -	tst_res TPASS "Test 1 passed"
> +	tst_res TPASS "Test $1 passed"
>  }
>  
>  test2()
>  {
> -	tst_res TPASS "Test 2 passed"
> +	tst_res TPASS "Test $1 passed"
>  }
>  
>  tst_run
> @@ -1455,7 +1455,8 @@ tst_run
>  
>  If '$TST_CNT' is set, the test library looks if there are functions named
>  '$\{TST_TESTFUNC\}1', ..., '$\{TST_TESTFUNC\}$\{TST_CNT\}' and if these are
> -found they are executed one by one.
> +found they are executed one by one. The test number is passed to it in the '$1'.
> +
>  
>  [source,sh]
>  -------------------------------------------------------------------------------
> @@ -1471,8 +1472,8 @@ TST_CNT=2
>  do_test()
>  {
>  	case $1 in
> -	1) tst_res TPASS "Test 1 passed";;
> -	2) tst_res TPASS "Test 2 passed";;
> +	1) tst_res TPASS "Test $1 passed";;
> +	2) tst_res TPASS "Test $1 passed";;
>  	esac
>  }
>  
> @@ -1483,6 +1484,65 @@ Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc.,
>  the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed
>  to it in the '$1'.
>  
> +[source,sh]
> +-------------------------------------------------------------------------------
> +#!/bin/sh
> +#
> +# Example test with tests in a single function, using $TST_TEST_DATA and
> +# $TST_TEST_DATA_IFS
> +#
> +
> +TST_TESTFUNC=do_test
> +TST_TEST_DATA="foo:bar:d dd"
> +TST_TEST_DATA_IFS=":"
> +. tst_test.sh
> +
> +do_test()
> +{
> +	tst_res TPASS "Test $1 passed with data '$2'"
> +}
> +
> +tst_run
> +# output:
> +# test 1 TPASS: Test 1 passed with data 'foo'
> +# test 2 TPASS: Test 2 passed with data 'bar'
> +# test 3 TPASS: Test 3 passed with data 'd dd'
> +
> +-------------------------------------------------------------------------------
> +
> +It's possible to pass data for function with '$TST_TEST_DATA'. Optional
> +'$TST_TEST_DATA_IFS' is used for splitting, default value is space.
> +
> +[source,sh]
> +-------------------------------------------------------------------------------
> +#!/bin/sh
> +#
> +# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT
> +#
> +
> +TST_TESTFUNC=do_test
> +TST_CNT=2
> +TST_TEST_DATA="foo:bar:d dd"
> +. tst_test.sh
> +
> +do_test()
> +{
> +	case $1 in
> +	1) tst_res TPASS "Test $1 passed with data '$2'";;
> +	2) tst_res TPASS "Test $1 passed with data '$2'";;
> +	esac
> +}
> +
> +tst_run
> +# output:
> +# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
> +# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
> +
> +-------------------------------------------------------------------------------
> +When '$TST_TEST_DATA' is used with '$TST_CNT', it's passed as whole string in
> +'$2' ($1 is for the test number), '$TST_TEST_DATA_IFS' is ignored. Similar
> +would be when using these variables with separate functions.

Now that we support both TST_CNT and TST_TEST_DATA_IFS can we change
this example to include the TST_TEST_DATA_IFS=":" as well?

>  2.3.2 Library variables
>  ^^^^^^^^^^^^^^^^^^^^^^^
>  
> @@ -1587,8 +1647,8 @@ these can be listed with passing help '-h' option to any test.
>  The function that prints the usage is passed in '$TST_USAGE', the help for
>  the options implemented in the library is appended when usage is printed.
>  
> -Lastly the fucntion '$PARSE_ARGS' is called with the option name in '$1' and,
> -if option has argument, its value in '$2'.
> +Lastly the fucntion '$PARSE_ARGS' is called with the option name in the '$1'
> +and, if option has argument, its value in the '$2'.
>  
>  [source,sh]
>  -------------------------------------------------------------------------------
> diff --git a/testcases/lib/tst_test.sh b/testcases/lib/tst_test.sh
> index 464c4c41e..7680aa462 100644
> --- a/testcases/lib/tst_test.sh
> +++ b/testcases/lib/tst_test.sh
> @@ -246,7 +246,7 @@ tst_rescmp()
>  
>  tst_run()
>  {
> -	local tst_i
> +	local tst_i tst_data
>  
>  	if [ -n "$TST_TEST_PATH" ]; then
>  		for tst_i in $(grep TST_ "$TST_TEST_PATH" | sed 's/.*TST_//; s/[="} \t\/:`].*//'); do
> @@ -255,7 +255,7 @@ tst_run()
>  			OPTS|USAGE|PARSE_ARGS|POS_ARGS);;
>  			NEEDS_ROOT|NEEDS_TMPDIR|NEEDS_DEVICE|DEVICE);;
>  			NEEDS_CMDS|NEEDS_MODULE|MODPATH|DATAROOT);;
> -			IPV6);;
> +			IPV6|TEST_DATA|TEST_DATA_IFS);;
>  			*) tst_res TWARN "Reserved variable TST_$tst_i used!";;
>  			esac
>  		done
> @@ -348,27 +348,17 @@ tst_run()
>  
>  	#TODO check that test reports some results for each test function call
>  	while [ $TST_ITERATIONS -gt 0 ]; do
> -		if [ -n "$TST_CNT" ]; then
> -			if type test1 > /dev/null 2>&1; then
> -				for tst_i in $(seq $TST_CNT); do
> -					local res=$(tst_resstr)
> -					$TST_TESTFUNC$tst_i
> -					tst_rescmp "$res"
> -					TST_COUNT=$((TST_COUNT+1))
> -				done
> -			else
> -				for tst_i in $(seq $TST_CNT); do
> -					local res=$(tst_resstr)
> -					$TST_TESTFUNC $tst_i
> -					tst_rescmp "$res"
> -					TST_COUNT=$((TST_COUNT+1))
> -				done
> -			fi
> +		if [ -n "$TST_TEST_DATA" ]; then
> +			tst_i=1
> +			tst_check_cmds cut
> +			while true; do
> +				tst_data="$(echo "$TST_TEST_DATA" | cut -d"$TST_TEST_DATA_IFS" -f$tst_i)"
> +				[ -z "$tst_data" ] && break
> +				tst_run_tests "$tst_data"
> +				tst_i=$((tst_i+1))
> +			done
>  		else
> -			local res=$(tst_resstr)
> -			$TST_TESTFUNC
> -			tst_rescmp "$res"
> -			TST_COUNT=$((TST_COUNT+1))
> +			tst_run_tests
>  		fi
>  		TST_ITERATIONS=$((TST_ITERATIONS-1))
>  	done
> @@ -376,6 +366,31 @@ tst_run()
>  	tst_do_exit
>  }
>  
> +tst_run_tests()
> +{
> +	local tst_data="$1"
> +	local tst_i
> +
> +	for tst_i in $(seq ${TST_CNT:-1}); do
> +		if type test1 > /dev/null 2>&1; then

I know that this is not your fault but this should be probably:

if type ${TST_TESTFUNC}1 > /dev/null 2>&1; ...

> +			tst_run_test "$TST_TESTFUNC$tst_i" $tst_i "$tst_data"
> +		else
> +			tst_run_test "$TST_TESTFUNC" $tst_i "$tst_data"
> +		fi
> +	done
> +}
> +
> +tst_run_test()
> +{
> +	local res=$(tst_resstr)

Can we please prefix the res here with tst_ as well since we are
touching the code?


> +	local tst_fnc="$1"
> +	shift
> +
> +	$tst_fnc "$@"
> +	tst_rescmp "$res"
> +	TST_COUNT=$((TST_COUNT+1))
> +}
> +
>  if [ -z "$TST_ID" ]; then
>  	filename=$(basename $0)
>  	TST_ID=${filename%%.*}
> @@ -400,6 +415,8 @@ if [ -z "$TST_NO_DEFAULT_RUN" ]; then
>  		tst_brk TBROK "TST_TESTFUNC is not defined"
>  	fi
>  
> +	TST_TEST_DATA_IFS="${TST_TEST_DATA_IFS:- }"
> +
>  	if [ -n "$TST_CNT" ]; then
>  		if ! tst_is_int "$TST_CNT"; then
>  			tst_brk TBROK "TST_CNT must be integer"
> -- 
> 2.16.3
> 

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH v3 2/2] lib: Add tests

[Cyril Hrubis]

Hi!
> This is not intended to be merged. It's just a helper which you can use
> for testing the previous patch.
>
> (Side note: if we ever want to test tst_test.sh, it'd be possible to
> dump it into file and compare with saved output. It'd require to remove
> lines with 'stime=')

That sounds like a good idea actually, I'm all for unit testing the
shell library.

I'm not that sure that we would want to put it into the runtest file
though, maybe we just need a runtest.sh script in the lib/ directory
that would loop over all *.sh file in a certain directory and compared
the output against corresponding file names without the .sh suffix.

We can then do something as lib/tests/ with runtest.sh that would loop
over subdirectories in lib/tests/ executing actual testcases.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH v3 1/2] tst_test.sh: Add TST_TEST_DATA and TST_TEST_DATA_IFS

[Petr Vorel]

Hi Cyril,

Thanks for your comments

> > Signed-off-by: Petr Vorel <pvorel@suse.cz>
> > ---
> > Changes v2->v3:
> > * Don't pass $TST_TEST_DATA as whole argument for $TST_CNT (Cyril)
> > * Create tst_run_tests() for cleanup (Cyril).
> > * Create tst_run_test() - more cleanup

> > Maybe we should try to "hide" somehow these API functions (underscore at
> > the front?).

> Actually this is a good idea, we can follow the python underscore
> notation that uses it for private variables and functions. Hence the
> counters will become _tst_i, etc.

> But please do that in a separate patch.
I meant to add underscore before new functions i.e. _tst_run_tests() and _tst_run_test().
Variables: do you mean to change local variables in tst_run()?
That would be really for separate patch.

> > ---
> >  doc/test-writing-guidelines.txt | 74 +++++++++++++++++++++++++++++++++++++----
> >  testcases/lib/tst_test.sh       | 61 +++++++++++++++++++++------------
> >  2 files changed, 106 insertions(+), 29 deletions(-)

...
> > +TST_TESTFUNC=do_test
> > +TST_CNT=2
> > +TST_TEST_DATA="foo:bar:d dd"
> > +. tst_test.sh
> > +
> > +do_test()
> > +{
> > +	case $1 in
> > +	1) tst_res TPASS "Test $1 passed with data '$2'";;
> > +	2) tst_res TPASS "Test $1 passed with data '$2'";;
> > +	esac
> > +}
> > +
> > +tst_run
> > +# output:
> > +# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
> > +# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
> > +
> > +-------------------------------------------------------------------------------
> > +When '$TST_TEST_DATA' is used with '$TST_CNT', it's passed as whole string in
> > +'$2' ($1 is for the test number), '$TST_TEST_DATA_IFS' is ignored. Similar
> > +would be when using these variables with separate functions.

> Now that we support both TST_CNT and TST_TEST_DATA_IFS can we change
> this example to include the TST_TEST_DATA_IFS=":" as well?
I wanted to demonstrate that TST_TEST_DATA_IFS has a default value ' ').
But ok, I'll change it to define TST_TEST_DATA_IFS as well.

...
> > +	for tst_i in $(seq ${TST_CNT:-1}); do
> > +		if type test1 > /dev/null 2>&1; then

> I know that this is not your fault but this should be probably:

> if type ${TST_TESTFUNC}1 > /dev/null 2>&1; ...
Thanks, I'll fix it, in a separate commit.

> > +tst_run_test()
> > +{
> > +	local res=$(tst_resstr)

> Can we please prefix the res here with tst_ as well since we are
> touching the code?
Sure. To be honest I wasn't sure whether there is some reason for $res being named without
prefix or not.


Kind regards,
Petr

[LTP] [RFC PATCH v3 2/2] lib: Add tests

[Petr Vorel]

> Hi!
> > This is not intended to be merged. It's just a helper which you can use
> > for testing the previous patch.

> > (Side note: if we ever want to test tst_test.sh, it'd be possible to
> > dump it into file and compare with saved output. It'd require to remove
> > lines with 'stime=')

> That sounds like a good idea actually, I'm all for unit testing the
> shell library.
I was inspired by recent change in kernel's kconfig. Masahiro Yamada added tests, which
compare output of various make calls with saved. It's in scripts/kconfig/tests/, using
python's pytest, you can see it by running 'make testconfig'.

> I'm not that sure that we would want to put it into the runtest file
> though, maybe we just need a runtest.sh script in the lib/ directory
> that would loop over all *.sh file in a certain directory and compared
> the output against corresponding file names without the .sh suffix.

> We can then do something as lib/tests/ with runtest.sh that would loop
> over subdirectories in lib/tests/ executing actual testcases.

Sounds reasonable. I added it into runtest file just to help myself to test. And publish
it just in case anyone wanted to use it for testing the patch.


Kind regards,
Petr

[LTP] [RFC PATCH v3 1/2] tst_test.sh: Add TST_TEST_DATA and TST_TEST_DATA_IFS

[Cyril Hrubis]

Hi!
> > Actually this is a good idea, we can follow the python underscore
> > notation that uses it for private variables and functions. Hence the
> > counters will become _tst_i, etc.
> 
> > But please do that in a separate patch.
> I meant to add underscore before new functions i.e. _tst_run_tests() and _tst_run_test().
> Variables: do you mean to change local variables in tst_run()?
> That would be really for separate patch.

As I written, change everything private that leaks to the test context
with _tst_ so that it's clear what the test is supposed to set/call.

> > > ---
> > >  doc/test-writing-guidelines.txt | 74 +++++++++++++++++++++++++++++++++++++----
> > >  testcases/lib/tst_test.sh       | 61 +++++++++++++++++++++------------
> > >  2 files changed, 106 insertions(+), 29 deletions(-)
> 
> ...
> > > +TST_TESTFUNC=do_test
> > > +TST_CNT=2
> > > +TST_TEST_DATA="foo:bar:d dd"
> > > +. tst_test.sh
> > > +
> > > +do_test()
> > > +{
> > > +	case $1 in
> > > +	1) tst_res TPASS "Test $1 passed with data '$2'";;
> > > +	2) tst_res TPASS "Test $1 passed with data '$2'";;
> > > +	esac
> > > +}
> > > +
> > > +tst_run
> > > +# output:
> > > +# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
> > > +# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
> > > +
> > > +-------------------------------------------------------------------------------
> > > +When '$TST_TEST_DATA' is used with '$TST_CNT', it's passed as whole string in
> > > +'$2' ($1 is for the test number), '$TST_TEST_DATA_IFS' is ignored. Similar
> > > +would be when using these variables with separate functions.
> 
> > Now that we support both TST_CNT and TST_TEST_DATA_IFS can we change
> > this example to include the TST_TEST_DATA_IFS=":" as well?
> I wanted to demonstrate that TST_TEST_DATA_IFS has a default value ' ').
> But ok, I'll change it to define TST_TEST_DATA_IFS as well.

No problem with that, but then we should get rid of the : from the
example data and keep it only at "foo bar".

> > Can we please prefix the res here with tst_ as well since we are
> > touching the code?
> Sure. To be honest I wasn't sure whether there is some reason for $res being named without
> prefix or not.

I guess that it haven't been caught in the review process when I first
wrote the library...

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [PATCH 1/2] Make shell lib tests standalone

[Christian Lanig]

---
Please do not merge! See following message for details.

 lib/newlib_tests/Makefile                          |  1 -
 .../{ => shell}/test.TST_TEST.TST_CNT.separate.sh  |  2 +-
 .../{ => shell}/test.TST_TEST.TST_CNT.sh           |  2 +-
 .../{ => shell}/test.TST_TEST.getopts.sh           |  2 +-
 lib/newlib_tests/{ => shell}/test.TST_TEST.sh      |  2 +-
 .../test.TST_TEST_DATA.TST_CNT.separate.sh         |  2 +-
 .../{ => shell}/test.TST_TEST_DATA.TST_CNT.sh      |  2 +-
 .../{ => shell}/test.TST_TEST_DATA.getopts.sh      |  2 +-
 lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.sh |  2 +-
 .../{ => shell}/test.TST_TEST_DATA_IFS.getopts.sh  |  2 +-
 .../{ => shell}/test.TST_TEST_DATA_IFS.sh          |  2 +-
 lib/newlib_tests/test.shell_lib.sh                 | 94 ++++++++++++++++++++++
 runtest/test                                       | 13 ---
 testcases/lib/tst_test.sh                          |  2 +-
 14 files changed, 105 insertions(+), 25 deletions(-)
 rename lib/newlib_tests/{ => shell}/test.TST_TEST.TST_CNT.separate.sh (95%)
 rename lib/newlib_tests/{ => shell}/test.TST_TEST.TST_CNT.sh (95%)
 rename lib/newlib_tests/{ => shell}/test.TST_TEST.getopts.sh (96%)
 rename lib/newlib_tests/{ => shell}/test.TST_TEST.sh (95%)
 rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.TST_CNT.separate.sh (96%)
 rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.TST_CNT.sh (96%)
 rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.getopts.sh (96%)
 rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.sh (95%)
 rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA_IFS.getopts.sh (97%)
 rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA_IFS.sh (96%)
 create mode 100755 lib/newlib_tests/test.shell_lib.sh
 delete mode 100644 runtest/test

diff --git a/lib/newlib_tests/Makefile b/lib/newlib_tests/Makefile
index ad087b440..2fc50160a 100644
--- a/lib/newlib_tests/Makefile
+++ b/lib/newlib_tests/Makefile
@@ -16,6 +16,5 @@ ifeq ($(ANDROID),1)
 FILTER_OUT_MAKE_TARGETS	+= test08
 endif
 
-INSTALL_TARGETS		:= *.sh
 
 include $(top_srcdir)/include/mk/generic_leaf_target.mk
diff --git a/lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh b/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh
similarity index 95%
rename from lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh
rename to lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh
index e26e2cb14..75ab680f1 100755
--- a/lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh
@@ -5,7 +5,7 @@
 
 TST_TESTFUNC=test
 TST_CNT=2
-. tst_test.sh
+. ./tst_test.sh
 
 test1()
 {
diff --git a/lib/newlib_tests/test.TST_TEST.TST_CNT.sh b/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh
similarity index 95%
rename from lib/newlib_tests/test.TST_TEST.TST_CNT.sh
rename to lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh
index d84457fea..31b15204f 100755
--- a/lib/newlib_tests/test.TST_TEST.TST_CNT.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh
@@ -5,7 +5,7 @@
 
 TST_TESTFUNC=do_test
 TST_CNT=2
-. tst_test.sh
+. ./tst_test.sh
 
 do_test()
 {
diff --git a/lib/newlib_tests/test.TST_TEST.getopts.sh b/lib/newlib_tests/shell/test.TST_TEST.getopts.sh
similarity index 96%
rename from lib/newlib_tests/test.TST_TEST.getopts.sh
rename to lib/newlib_tests/shell/test.TST_TEST.getopts.sh
index 0ba848a0a..090133756 100755
--- a/lib/newlib_tests/test.TST_TEST.getopts.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST.getopts.sh
@@ -8,7 +8,7 @@ TST_USAGE=usage
 TST_PARSE_ARGS=parse_args
 TST_TESTFUNC=do_test
 
-. tst_test.sh
+. ./tst_test.sh
 
 ALTERNATIVE=0
 MODE="foo"
diff --git a/lib/newlib_tests/test.TST_TEST.sh b/lib/newlib_tests/shell/test.TST_TEST.sh
similarity index 95%
rename from lib/newlib_tests/test.TST_TEST.sh
rename to lib/newlib_tests/shell/test.TST_TEST.sh
index 6ecae710d..42f14151b 100755
--- a/lib/newlib_tests/test.TST_TEST.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST.sh
@@ -4,7 +4,7 @@
 #
 
 TST_TESTFUNC=do_test
-. tst_test.sh
+. ./tst_test.sh
 
 do_test()
 {
diff --git a/lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.separate.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh
similarity index 96%
rename from lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.separate.sh
rename to lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh
index 9a81ad27c..6d4a90141 100755
--- a/lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.separate.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh
@@ -6,7 +6,7 @@
 TST_TESTFUNC=test
 TST_CNT=2
 TST_TEST_DATA="foo:bar:d dd"
-. tst_test.sh
+. ./tst_test.sh
 
 test1()
 {
diff --git a/lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh
similarity index 96%
rename from lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh
rename to lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh
index 798ad9621..8dd25acae 100755
--- a/lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh
@@ -6,7 +6,7 @@
 TST_TESTFUNC=do_test
 TST_CNT=2
 TST_TEST_DATA="foo:bar:d dd"
-. tst_test.sh
+. ./tst_test.sh
 
 do_test()
 {
diff --git a/lib/newlib_tests/test.TST_TEST_DATA.getopts.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh
similarity index 96%
rename from lib/newlib_tests/test.TST_TEST_DATA.getopts.sh
rename to lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh
index 1cd150618..66e534ae5 100755
--- a/lib/newlib_tests/test.TST_TEST_DATA.getopts.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh
@@ -9,7 +9,7 @@ TST_PARSE_ARGS=parse_args
 TST_TESTFUNC=do_test
 TST_TEST_DATA="foo0:bar:d dd"
 
-. tst_test.sh
+. ./tst_test.sh
 
 ALTERNATIVE=0
 MODE="foo"
diff --git a/lib/newlib_tests/test.TST_TEST_DATA.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA.sh
similarity index 95%
rename from lib/newlib_tests/test.TST_TEST_DATA.sh
rename to lib/newlib_tests/shell/test.TST_TEST_DATA.sh
index 4cb51c669..568cec01e 100755
--- a/lib/newlib_tests/test.TST_TEST_DATA.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA.sh
@@ -5,7 +5,7 @@
 
 TST_TESTFUNC=do_test
 TST_TEST_DATA="foo:bar:d dd"
-. tst_test.sh
+. ./tst_test.sh
 
 do_test()
 {
diff --git a/lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh
similarity index 97%
rename from lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh
rename to lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh
index 59dbd9598..41e3f99de 100755
--- a/lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh
@@ -10,7 +10,7 @@ TST_TESTFUNC=do_test
 TST_TEST_DATA="foo0:bar:d dd"
 TST_TEST_DATA_IFS=":"
 
-. tst_test.sh
+. ./tst_test.sh
 
 ALTERNATIVE=0
 MODE="foo"
diff --git a/lib/newlib_tests/test.TST_TEST_DATA_IFS.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh
similarity index 96%
rename from lib/newlib_tests/test.TST_TEST_DATA_IFS.sh
rename to lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh
index 52a2e6d47..4e73f5030 100755
--- a/lib/newlib_tests/test.TST_TEST_DATA_IFS.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh
@@ -7,7 +7,7 @@
 TST_TESTFUNC=do_test
 TST_TEST_DATA="foo:bar:d dd"
 TST_TEST_DATA_IFS=":"
-. tst_test.sh
+. ./tst_test.sh
 
 do_test()
 {
diff --git a/lib/newlib_tests/test.shell_lib.sh b/lib/newlib_tests/test.shell_lib.sh
new file mode 100755
index 000000000..599eec78a
--- /dev/null
+++ b/lib/newlib_tests/test.shell_lib.sh
@@ -0,0 +1,94 @@
+# !/bin/sh
+#
+# This script iterates over all test cases for the new shell lib and verifies
+# the output.
+# Do NOT use newline symbols in the names of files containing test cases!
+#
+
+setup() {
+	color_green="\033[1;32m"
+	color_red="\033[1;31m"
+	standard_color="\033[0m"
+	start_dir="$PWD"
+	cd "$(dirname "$0")""/../../" || exit 1;
+	sh_lib_dir=""$PWD"/testcases/lib/"
+	sh_lib_test_dir=""$PWD"/lib/newlib_tests/shell/"
+	tst_cases=$(ls "$sh_lib_test_dir" | \
+			sed "s/"$TEST_NAME".sh/"$TEST_NAME"/g")
+	cd "$sh_lib_dir" || exit 1
+}
+
+check_requirements() {
+	case "$0" in
+		-*)
+			printf "Please execute this script. Sourcing ";
+			printf "(. <SCRIPT>) is not supported. \n";
+			return 1;;
+		*)
+			true;
+	esac
+}
+
+verify_output() {
+	local output_found=1
+	local wanted_output=
+	local parsed_line=
+	while read line;
+	do
+		if [ -z "$wanted_output" ] && [ "$line" = "# output:" ]
+		then
+			output_found=0
+		elif [ $output_found -eq 0 ] || [ -n "$wanted_output" ]
+		then
+			if printf "$line" | grep "# " > /dev/null;
+			then
+				if [ $output_found -eq 0 ]
+				then
+					parsed_line=$(printf "$line" | \
+							sed "s/^.\{2\}//")
+					output_found=1
+				else
+					parsed_line="\n"$(printf "$line" | \
+							sed "s/^.\{2\}//")
+				fi
+			elif printf "$line" | grep "#" > /dev/null;
+			then
+				parsed_line="\n"$(printf "$line" | \
+						sed "s/^.\{1\}//")
+			else true;
+			fi
+			wanted_output=""$wanted_output""$parsed_line""
+		else true;
+		fi
+
+	done < ""$sh_lib_test_dir""$1".sh"
+	wanted_output=$(printf "$wanted_output")
+	local actual_output=$(""$sh_lib_test_dir""$1".sh")
+	actual_output=$(printf "$actual_output")
+	if [ "$wanted_output" = "$actual_output" ]
+	then
+		return 0
+	else
+		return 1
+	fi
+}
+
+run_tests() {
+	for tst_case in $tst_cases
+	do
+		printf "Running Test: \""$tst_case"\"...\n"
+		if verify_output "$tst_case";
+		then
+			printf ""$color_green"TPASS: "$standard_color""
+			printf "Test "$tst_case" was successful.\n\n"
+		else
+			printf ""$color_red"TFAIL:"$standard_color""
+			printf "Test "$tst_case" was unsuccessful.\n\n"
+		fi
+	done
+}
+
+check_requirements
+setup
+run_tests
+exit 0
diff --git a/runtest/test b/runtest/test
deleted file mode 100644
index 469e3b787..000000000
--- a/runtest/test
+++ /dev/null
@@ -1,13 +0,0 @@
-TST_TEST_DATA_IFS test.TST_TEST_DATA_IFS.sh
-TST_TEST_DATA test.TST_TEST_DATA.sh
-TST_TEST_DATA.TST_CNT.separate test.TST_TEST_DATA.TST_CNT.separate.sh
-TST_TEST_DATA.TST_CNT test.TST_TEST_DATA.TST_CNT.sh
-TST_TEST test.TST_TEST.sh
-TST_TEST.TST_CNT.separate test.TST_TEST.TST_CNT.separate.sh
-TST_TEST.TST_CNT test.TST_TEST.TST_CNT.sh
-TST_TEST.getopts test.TST_TEST.getopts.sh
-TST_TEST.getopts_a_f test.TST_TEST.getopts.sh -a -f foo1
-TST_TEST_DATA.getopts test.TST_TEST_DATA.getopts.sh
-TST_TEST_DATA.getopts_a_f test.TST_TEST_DATA.getopts.sh -a -f foo2
-TST_TEST_DATA_IFS.getopts test.TST_TEST_DATA_IFS.getopts.sh
-TST_TEST_DATA_IFS.getopts_a_f test.TST_TEST_DATA_IFS.getopts.sh -a -f foo3
diff --git a/testcases/lib/tst_test.sh b/testcases/lib/tst_test.sh
index e553b496d..769ed66c2 100644
--- a/testcases/lib/tst_test.sh
+++ b/testcases/lib/tst_test.sh
@@ -33,7 +33,7 @@ export TST_ITERATIONS=1
 export TST_TMPDIR_RHOST=0
 export TST_LIB_LOADED=1
 
-. tst_ansi_color.sh
+. ./tst_ansi_color.sh
 
 # default trap function
 trap "tst_brk TBROK 'test interrupted'" INT
-- 
2.16.4

[LTP] [PATCH 2/2] Add wanted output to shell lib test case

[Christian Lanig]

---
Please do not merge!
I would like to get some feedback for this one and the previous patch. It is
based on Petr Vorel's [RFC,v3,2/2] lib: Add tests from May 22nd 2018 with the
message ID 20180522193430.20117-2-pvorel@suse.cz.
These two patches transfer Petr's test cases in a separate folder and a shell
script loops over all files as suggested in the replies to the initial message.
The output is verified against a comment section in the test case file itself.
This section is started with the line "# output:" and it is expected that it
goes to the end of the document without interception.
I have run the script with various Shell implementations to make sure it's
portable.

Limitations:
The script is currently unable to test e.g. whether TFAIL is displayed with red
color.
So far I have only added the correct output line in a single test case file so
just one test is currently passing.
Before proceeding I would like to know whether this is a good way to go.

 lib/newlib_tests/shell/test.TST_TEST.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/newlib_tests/shell/test.TST_TEST.sh b/lib/newlib_tests/shell/test.TST_TEST.sh
index 42f14151b..aa4e20377 100755
--- a/lib/newlib_tests/shell/test.TST_TEST.sh
+++ b/lib/newlib_tests/shell/test.TST_TEST.sh
@@ -21,5 +21,12 @@ do_test()
 }
 
 tst_run
+# output:
 # test 1 TINFO: Test 1 passed with no data ('')
 # test 1 TPASS: true returned 0
+#
+# Summary:
+# passed   1
+# failed   0
+# skipped  0
+# warnings 0
-- 
2.16.4

[LTP] [PATCH 1/2] Make shell lib tests standalone

[Petr Vorel]

Hi Christian,

this expects have applied my patch
https://patchwork.ozlabs.org/patch/918549/
http://lists.linux.it/pipermail/ltp/2018-May/008191.html
Christian, next time merge it with your changes (these 3 commits should be just
one, with you as author.)

Before I delve into the details, just comment the functionality.

$ ./lib/newlib_tests/test.shell_lib.sh
Running Test: "test.TST_TEST_DATA.getopts"...
TFAIL:Test test.TST_TEST_DATA.getopts was unsuccessful.

Running Test: "test.TST_TEST_DATA_IFS.getopts"...
TFAIL:Test test.TST_TEST_DATA_IFS.getopts was unsuccessful.

Running Test: "test.TST_TEST_DATA_IFS"...
TFAIL:Test test.TST_TEST_DATA_IFS was unsuccessful.

Running Test: "test.TST_TEST_DATA"...
TFAIL:Test test.TST_TEST_DATA was unsuccessful.

Running Test: "test.TST_TEST_DATA.TST_CNT.separate"...
TFAIL:Test test.TST_TEST_DATA.TST_CNT.separate was unsuccessful.

Running Test: "test.TST_TEST_DATA.TST_CNT"...
TFAIL:Test test.TST_TEST_DATA.TST_CNT was unsuccessful.

Running Test: "test.TST_TEST.getopts"...
TFAIL:Test test.TST_TEST.getopts was unsuccessful.

Running Test: "test.TST_TEST"...
TPASS: Test test.TST_TEST was successful.

Running Test: "test.TST_TEST.TST_CNT.separate"...
TFAIL:Test test.TST_TEST.TST_CNT.separate was unsuccessful.

Running Test: "test.TST_TEST.TST_CNT"...
TFAIL:Test test.TST_TEST.TST_CNT was unsuccessful.

---
These tests must be all green before merging them.
I suppose the failures are because only 5 of 10 tests have "# output:" (parsed
by verify_output()).

About the output. See Cyril's comments about my patch for running C tests
http://lists.linux.it/pipermail/ltp/2018-August/009119.html

Inconsistent output: space after 'TPASS:', not after 'TFAIL:'. I wouldn't use ':'.

Output doesn't tell, what's wrong. IMHO there should be diff of expected output
vs. real output. This is done in xfstests.

+ It could be less verbose. Let's use:

$ ./lib/newlib_tests/test.shell_lib.sh
TFAIL test.TST_TEST_DATA.getopts
[ diff output whats wrong ]
TPASS test.TST_TEST

Summary:
passed   1
failed   1

The name itself would IMHO be better 
test_tst_test.sh or test.tst_test.sh (but we don't use often dot in name).
ATM we have old and new legacy API, test.shell_lib.sh doesn't say which it
tests.

> ---
> Please do not merge! See following message for details.
You can use --rfc switch for git format-patch next time.

...
> --- a/lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh
> +++ b/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh
> @@ -5,7 +5,7 @@

>  TST_TESTFUNC=test
>  TST_CNT=2
> -. tst_test.sh
> +. ./tst_test.sh
This is not good. Much better is to make sure, that testcases/lib/ directory
(where tst_test.sh is) is in PATH variable. The reasons are that you don't
expect current working directory be in the same directory as the test. See:
$ cd lib/newlib_tests
$ ./shell/test.TST_TEST.TST_CNT.separate.sh
bash: ./shell/test.TST_TEST.TST_CNT.separate.sh: No such file or directory

And the tests are supposed to be also examples in docs, where we prefer not
forcing cd into /opt/ltp/testcases/bin.
...

> diff --git a/lib/newlib_tests/test.shell_lib.sh b/lib/newlib_tests/test.shell_lib.sh
> new file mode 100755
> index 000000000..599eec78a
> --- /dev/null
> +++ b/lib/newlib_tests/test.shell_lib.sh
> @@ -0,0 +1,94 @@
> +# !/bin/sh
    ^ Please no space after.

checkbashisms.pl [1] warns you about it:
$ checkbashisms.pl -f test.shell_lib.sh
script test.shell_lib.sh does not appear to have a #! interpreter line;
you may get strange results

> +#
> +# This script iterates over all test cases for the new shell lib and verifies
> +# the output.
> +# Do NOT use newline symbols in the names of files containing test cases!
> +#
Maybe you want to add your copyright here.
You should add license (GPL v2 + any other). I prefer SPDX-License-Identifier as
it's short:
# SPDX-License-Identifier: GPL-2.0-or-later

> +
> +setup() {
> +	color_green="\033[1;32m"
> +	color_red="\033[1;31m"
> +	standard_color="\033[0m"
> +	start_dir="$PWD"
> +	cd "$(dirname "$0")""/../../" || exit 1;
> +	sh_lib_dir=""$PWD"/testcases/lib/"

> +	sh_lib_test_dir=""$PWD"/lib/newlib_tests/shell/"
Wrong quote usage. In this interpretation $PWD isn't quoted.
It should be:
sh_lib_test_dir="$PWD/lib/newlib_tests/shell/"

You probably want to setup it as global variable (out of all functions)
- in this case global isn't that bad. Also I'd prefer simpler name.
Using dirname does not require to be in git root: 

TESTDIR="$(dirname $0)/shell/"

> +	tst_cases=$(ls "$sh_lib_test_dir" | \
> +			sed "s/"$TEST_NAME".sh/"$TEST_NAME"/g")
> +	cd "$sh_lib_dir" || exit 1
> +}
> +
> +check_requirements() {
> +	case "$0" in
> +		-*)
> +			printf "Please execute this script. Sourcing ";
> +			printf "(. <SCRIPT>) is not supported. \n";
> +			return 1;;
> +		*)
> +			true;
> +	esac
> +}
Is this really needed? I think it's obvious.

Now some syntax details.
Good example of shell script is the library itself (testcases/lib/tst_test.sh)

> +verify_output() {

New line before { }
verify_output()
{

> +	local output_found=1
> +	local wanted_output=
> +	local parsed_line=
> +	while read line;
> +	do
while read line; do
Keep 'do', 'then', 'else' on the same line

> +		if [ -z "$wanted_output" ] && [ "$line" = "# output:" ]
> +		then
if [ -z "$wanted_output" ] && [ "$line" = "# output:" ]; then

> +			output_found=0
> +		elif [ $output_found -eq 0 ] || [ -n "$wanted_output" ]
> +		then
elif [ $output_found -eq 0 ] || [ -n "$wanted_output" ], then

> +			if printf "$line" | grep "# " > /dev/null;
> +			then
> +				if [ $output_found -eq 0 ]
> +				then
> +					parsed_line=$(printf "$line" | \
> +							sed "s/^.\{2\}//")
> +					output_found=1
> +				else
> +					parsed_line="\n"$(printf "$line" | \
> +							sed "s/^.\{2\}//")
> +				fi
> +			elif printf "$line" | grep "#" > /dev/null;
> +			then
> +				parsed_line="\n"$(printf "$line" | \
> +						sed "s/^.\{1\}//")
> +			else true;
What is this for? You don't compare $?, which would be set by true.
And it's ugly.

> +			fi
> +			wanted_output=""$wanted_output""$parsed_line""
> +		else true;
> +		fi
> +
> +	done < ""$sh_lib_test_dir""$1".sh"

The easiest way is to store the output in separate file. Your way parsing the
output is interesting, but maybe too complicated for updating the pattern.
Even if you use this way, I'd prefer to get the output with sed/awk. Maybe there
are better ways, but this would work:

$ sed -ne '/^# output:/,$ p;' lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh | sed '1d'
# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
# test 3 TPASS: Test 1 passed with data 'dd'
# test 4 TPASS: Test 2 passed with data 'dd'

As I wrote, IMHO echo expected and actual output into the file and then run
'diff -u' on it would show, what's wrong.

> +	wanted_output=$(printf "$wanted_output")
You probably don't want this (adding to wanted_output variable what is already
there :) ).

> +	local actual_output=$(""$sh_lib_test_dir""$1".sh")
Again, wrong quotes usage.
> +	actual_output=$(printf "$actual_output")
Same here.

Maybe use less verbose variable name (detail).

> +	if [ "$wanted_output" = "$actual_output" ]
> +	then
> +		return 0
> +	else
> +		return 1
> +	fi
You can use
[ "$wanted_output" = "$actual_output" ] && return 0
return 1
> +}

> +run_tests() {
> +	for tst_case in $tst_cases
> +	do
> +		printf "Running Test: \""$tst_case"\"...\n"
> +		if verify_output "$tst_case";
> +		then
> +			printf ""$color_green"TPASS: "$standard_color""
Again, wrong quotes usage.

> +			printf "Test "$tst_case" was successful.\n\n"
> +		else
> +			printf ""$color_red"TFAIL:"$standard_color""
> +			printf "Test "$tst_case" was unsuccessful.\n\n"
> +		fi
Again use '; do/then/else'.

> +	done
> +}
> +
> +check_requirements
> +setup
> +run_tests

> +exit 0
That's wrong. I'd like exit code was 0 for everything pass or 1 for failure.


Kind regards,
Petr

[1] https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl

[LTP] [PATCH 1/2] Make shell lib tests standalone

[Petr Vorel]

Hi Christian,

> $ sed -ne '/^# output:/,$ p;' lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh | sed '1d'
> # test 1 TPASS: Test 1 passed with data 'foo:bar:d'
> # test 2 TPASS: Test 2 passed with data 'foo:bar:d'
> # test 3 TPASS: Test 1 passed with data 'dd'
> # test 4 TPASS: Test 2 passed with data 'dd'

Sorry, this is the correct command:
$ sed -ne '/^# output:/,$ p;' lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh | sed '1d; s/^# //'
test 1 TPASS: Test 1 passed with data 'foo:bar:d'
test 2 TPASS: Test 2 passed with data 'foo:bar:d'
test 3 TPASS: Test 1 passed with data 'dd'
test 4 TPASS: Test 2 passed with data 'dd'

But I bet other will be against keeping the output into the file.

+ It'd be nice to describe, how testing works - how to modify the output
(in README.md or at least in commit message).

Kind regards,
Petr

[LTP] [PATCH 1/2] Make shell lib tests standalone

[Cyril Hrubis]

Hi!
>  rename lib/newlib_tests/{ => shell}/test.TST_TEST.TST_CNT.separate.sh (95%)
>  rename lib/newlib_tests/{ => shell}/test.TST_TEST.TST_CNT.sh (95%)
>  rename lib/newlib_tests/{ => shell}/test.TST_TEST.getopts.sh (96%)
>  rename lib/newlib_tests/{ => shell}/test.TST_TEST.sh (95%)
>  rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.TST_CNT.separate.sh (96%)
>  rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.TST_CNT.sh (96%)
>  rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.getopts.sh (96%)
>  rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.sh (95%)
>  rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA_IFS.getopts.sh (97%)
>  rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA_IFS.sh (96%)

Just small comment, I do wonder if top level lib/ directory is correct
for the shell tests since the shell library is located at testcases/lib/
(for a historical reasons). Maybe we should just create a 'test' directory
under 'testcases/lib/' instead.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 0/1] Add automated tests for shell lib

[Christian Lanig]

The following patch adds the test cases by Petr Vorel. They can be executed by
automatically. doc/write-tests-for-shell-lib.txt describes how everything works.
The output tells whether each test has passed and when it failed, the diff is
printed.

I apologize for the badly formatted message I sent before and hope that this
time I used the right format options.

@Petr

> + It'd be nice to describe, how testing works - how to modify the output
> (in README.md or at least in commit message).
I am not sure that the text in doc/write-tests-for-shell-lib.txt should really
go in README.md. Probably just a link or something. But I can change it if
necessary.

In regard of the other comments, I have removed all double quotes where nothing
should go wrong as long as the person executing the script doesn't have
unsupported file or directory naming or pretty uncommon bugs in his or her
shell.

I have also removed unnecessary else statements (which I thought would be
necessary to stay compatible but this appeared to be a wrong impression when I
read the text about it more accurately a second time).
Also a -h / --help option was added which should be explanatory enough for
people who intuitively run the script with this option and to replace comments
about functionality in the header of the script file.

Another feature is to specify single test cases as arguments to only run these
and no other.

Both outputs, the desired and the actual one, are now stored in a temporary
folder and the PATH environment variable is modified temporarily, which implied
the necessity to add a cleanup function.

Regards,
Christian

Christian Lanig (1):
  Add automated tests for shell lib

 doc/write-tests-for-shell-lib.txt                  |  61 ++++++++++++
 .../shell/test.TST_TEST.TST_CNT.separate.sh        |  29 ++++++
 lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh    |  27 ++++++
 lib/newlib_tests/shell/test.TST_TEST.getopts.sh    |  48 ++++++++++
 lib/newlib_tests/shell/test.TST_TEST.sh            |  32 +++++++
 .../shell/test.TST_TEST_DATA.TST_CNT.separate.sh   |  32 +++++++
 .../shell/test.TST_TEST_DATA.TST_CNT.sh            |  30 ++++++
 .../shell/test.TST_TEST_DATA.getopts.sh            |  50 ++++++++++
 lib/newlib_tests/shell/test.TST_TEST_DATA.sh       |  24 +++++
 .../shell/test.TST_TEST_DATA_IFS.getopts.sh        |  52 ++++++++++
 lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh   |  27 ++++++
 lib/newlib_tests/test_sh_newlib.sh                 | 105 +++++++++++++++++++++
 12 files changed, 517 insertions(+)
 create mode 100644 doc/write-tests-for-shell-lib.txt
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST.getopts.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh
 create mode 100755 lib/newlib_tests/test_sh_newlib.sh

-- 
2.16.4

[LTP] [RFC PATCH 1/1] Add automated tests for shell lib

[Christian Lanig]

Signed-off-by: Christian Lanig <clanig@suse.com>
---
 doc/write-tests-for-shell-lib.txt                  |  61 ++++++++++++
 .../shell/test.TST_TEST.TST_CNT.separate.sh        |  29 ++++++
 lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh    |  27 ++++++
 lib/newlib_tests/shell/test.TST_TEST.getopts.sh    |  48 ++++++++++
 lib/newlib_tests/shell/test.TST_TEST.sh            |  32 +++++++
 .../shell/test.TST_TEST_DATA.TST_CNT.separate.sh   |  32 +++++++
 .../shell/test.TST_TEST_DATA.TST_CNT.sh            |  30 ++++++
 .../shell/test.TST_TEST_DATA.getopts.sh            |  50 ++++++++++
 lib/newlib_tests/shell/test.TST_TEST_DATA.sh       |  24 +++++
 .../shell/test.TST_TEST_DATA_IFS.getopts.sh        |  52 ++++++++++
 lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh   |  27 ++++++
 lib/newlib_tests/test_sh_newlib.sh                 | 105 +++++++++++++++++++++
 12 files changed, 517 insertions(+)
 create mode 100644 doc/write-tests-for-shell-lib.txt
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST.getopts.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh
 create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh
 create mode 100755 lib/newlib_tests/test_sh_newlib.sh

diff --git a/doc/write-tests-for-shell-lib.txt b/doc/write-tests-for-shell-lib.txt
new file mode 100644
index 000000000..6f3ff4854
--- /dev/null
+++ b/doc/write-tests-for-shell-lib.txt
@@ -0,0 +1,61 @@
+How to format tests in order to test the shell library
+======================================================
+
+It is of big importance not only to test the Linux kernel functionality but also
+to make sure LTP is running correctly itself. For this reason it is useful to
+test intrinsic functionality of LTP - to reasonable extend.
+
+1. Running tests for the shell library
+--------------------------------------
+The test cases reside in the folder `lib/newlib_tests/shell`. A script executing
+them one by one is in the folder `lib/newlib_tests`. You can execute this script
+to test all cases or specify test cases to be run. The script is called
+`test_sh_newlib.sh`.
+
+2. Writing tests for the shell library
+--------------------------------------
+The tests are written like all other test cases using the shell library.
+Additionally, at the end of the file the desired output is added. As an example:
+
+[source,shell]
+-------------------------------------------------------------------------------
+#!/bin/sh
+#
+# This is a basic test for true shell buildin
+#
+
+TST_TESTFUNC=do_test
+. tst_test.sh
+
+do_test()
+{
+	true
+	ret=$?
+
+	tst_res TINFO "Test $1 passed with no data ('$2')"
+
+	if [ $ret -eq 0 ]; then
+		tst_res TPASS "true returned 0"
+	else
+		tst_res TFAIL "true returned $ret"
+	fi
+}
+
+tst_run
+# output:
+# test 1 TINFO: Test 1 passed with no data ('')
+# test 1 TPASS: true returned 0
+# 
+# Summary:
+# passed   1
+# failed   0
+# skipped  0
+# warnings 0
+-------------------------------------------------------------------------------
+
+The most noticeable thing is to add the line `# output:` to show the parser that
+parsing should start in the following line. For the following lines the `# `
+will be stripped before the output is then compared with the actual output that
+gets printed on the terminal when running the test. Also make sure that there
+are no trailing lines at the end of the document. But make sure to add one
+trailing space after the `#` when the line will be actually empty.
diff --git a/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh b/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh
new file mode 100755
index 000000000..5e8d4bfb5
--- /dev/null
+++ b/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh
@@ -0,0 +1,29 @@
+#!/bin/sh
+#
+# Example test with tests in separate functions
+#
+
+TST_TESTFUNC=test
+TST_CNT=2
+. tst_test.sh
+
+test1()
+{
+	tst_res TPASS "Test $1 passed with no data ('$2')"
+}
+
+test2()
+{
+	tst_res TPASS "Test $1 passed with no data ('$2')"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with no data ('')
+# test 2 TPASS: Test 2 passed with no data ('')
+# 
+# Summary:
+# passed   2
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh b/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh
new file mode 100755
index 000000000..62fa652cf
--- /dev/null
+++ b/lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+#
+# Example test with tests in a single function
+#
+
+TST_TESTFUNC=do_test
+TST_CNT=2
+. tst_test.sh
+
+do_test()
+{
+	case $1 in
+	1) tst_res TPASS "Test $1 passed with no data ('$2')";;
+	2) tst_res TPASS "Test $1 passed with no data ('$2')";;
+	esac
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with no data ('')
+# test 2 TPASS: Test 2 passed with no data ('')
+# 
+# Summary:
+# passed   2
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/test.TST_TEST.getopts.sh b/lib/newlib_tests/shell/test.TST_TEST.getopts.sh
new file mode 100755
index 000000000..65810b5e8
--- /dev/null
+++ b/lib/newlib_tests/shell/test.TST_TEST.getopts.sh
@@ -0,0 +1,48 @@
+#!/bin/sh
+#
+# Optional test command line parameters
+#
+
+TST_OPTS="af:"
+TST_USAGE=usage
+TST_PARSE_ARGS=parse_args
+TST_TESTFUNC=do_test
+
+. tst_test.sh
+
+ALTERNATIVE=0
+MODE="foo"
+
+usage()
+{
+	cat << EOF
+usage: $0 [-a] [-f <foo|bar>]
+
+OPTIONS
+-a     Enable support for alternative foo
+-f     Specify foo or bar mode
+EOF
+}
+
+parse_args()
+{
+	case $1 in
+	a) ALTERNATIVE=1;;
+	f) MODE="$2";;
+	esac
+}
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data '': a: '0', f: 'foo'
+# 
+# Summary:
+# passed   1
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/test.TST_TEST.sh b/lib/newlib_tests/shell/test.TST_TEST.sh
new file mode 100755
index 000000000..fa6aaa57f
--- /dev/null
+++ b/lib/newlib_tests/shell/test.TST_TEST.sh
@@ -0,0 +1,32 @@
+#!/bin/sh
+#
+# This is a basic test for true shell buildin
+#
+
+TST_TESTFUNC=do_test
+. tst_test.sh
+
+do_test()
+{
+	true
+	ret=$?
+
+	tst_res TINFO "Test $1 passed with no data ('$2')"
+
+	if [ $ret -eq 0 ]; then
+		tst_res TPASS "true returned 0"
+	else
+		tst_res TFAIL "true returned $ret"
+	fi
+}
+
+tst_run
+# output:
+# test 1 TINFO: Test 1 passed with no data ('')
+# test 1 TPASS: true returned 0
+# 
+# Summary:
+# passed   1
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh
new file mode 100755
index 000000000..0ae773519
--- /dev/null
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh
@@ -0,0 +1,32 @@
+#!/bin/sh
+#
+# Example test with tests in separate functions, using $TST_TEST_DATA and $TST_CNT
+#
+
+TST_TESTFUNC=test
+TST_CNT=2
+TST_TEST_DATA="foo:bar:d dd"
+. tst_test.sh
+
+test1()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+test2()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
+# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
+# test 3 TPASS: Test 1 passed with data 'dd'
+# test 4 TPASS: Test 2 passed with data 'dd'
+# 
+# Summary:
+# passed   4
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh
new file mode 100755
index 000000000..71e6acac7
--- /dev/null
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT
+#
+
+TST_TESTFUNC=do_test
+TST_CNT=2
+TST_TEST_DATA="foo:bar:d dd"
+. tst_test.sh
+
+do_test()
+{
+	case $1 in
+	1) tst_res TPASS "Test $1 passed with data '$2'";;
+	2) tst_res TPASS "Test $1 passed with data '$2'";;
+	esac
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
+# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
+# test 3 TPASS: Test 1 passed with data 'dd'
+# test 4 TPASS: Test 2 passed with data 'dd'
+# 
+# Summary:
+# passed   4
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh
new file mode 100755
index 000000000..e1f18ba3a
--- /dev/null
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh
@@ -0,0 +1,50 @@
+#!/bin/sh
+#
+# Optional test command line parameters
+#
+
+TST_OPTS="af:"
+TST_USAGE=usage
+TST_PARSE_ARGS=parse_args
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo0:bar:d dd"
+
+. tst_test.sh
+
+ALTERNATIVE=0
+MODE="foo"
+
+usage()
+{
+	cat << EOF
+usage: $0 [-a] [-f <foo|bar>]
+
+OPTIONS
+-a     Enable support for alternative foo
+-f     Specify foo or bar mode
+EOF
+}
+
+parse_args()
+{
+	case $1 in
+	a) ALTERNATIVE=1;;
+	f) MODE="$2";;
+	esac
+}
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo0:bar:d': a: '0', f: 'foo'
+# test 2 TPASS: Test 1 passed with data 'dd': a: '0', f: 'foo'
+# 
+# Summary:
+# passed   2
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/test.TST_TEST_DATA.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA.sh
new file mode 100755
index 000000000..d3ced1954
--- /dev/null
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA
+#
+
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo:bar:d dd"
+. tst_test.sh
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
+# test 2 TPASS: Test 1 passed with data 'dd'
+# 
+# Summary:
+# passed   2
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh
new file mode 100755
index 000000000..6075904a5
--- /dev/null
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh
@@ -0,0 +1,52 @@
+#!/bin/sh
+#
+# Optional test command line parameters
+#
+
+TST_OPTS="af:"
+TST_USAGE=usage
+TST_PARSE_ARGS=parse_args
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo0:bar:d dd"
+TST_TEST_DATA_IFS=":"
+
+. tst_test.sh
+
+ALTERNATIVE=0
+MODE="foo"
+
+usage()
+{
+	cat << EOF
+usage: $0 [-a] [-f <foo|bar>]
+
+OPTIONS
+-a     Enable support for alternative foo
+-f     Specify foo or bar mode
+EOF
+}
+
+parse_args()
+{
+	case $1 in
+	a) ALTERNATIVE=1;;
+	f) MODE="$2";;
+	esac
+}
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo0': a: '0', f: 'foo'
+# test 2 TPASS: Test 1 passed with data 'bar': a: '0', f: 'foo'
+# test 3 TPASS: Test 1 passed with data 'd dd': a: '0', f: 'foo'
+# 
+# Summary:
+# passed   3
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh b/lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh
new file mode 100755
index 000000000..9a1bb9cf7
--- /dev/null
+++ b/lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA and
+# $TST_TEST_DATA_IFS
+#
+
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo:bar:d dd"
+TST_TEST_DATA_IFS=":"
+. tst_test.sh
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+tst_run
+# output:
+# test 1 TPASS: Test 1 passed with data 'foo'
+# test 2 TPASS: Test 1 passed with data 'bar'
+# test 3 TPASS: Test 1 passed with data 'd dd'
+# 
+# Summary:
+# passed   3
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/test_sh_newlib.sh b/lib/newlib_tests/test_sh_newlib.sh
new file mode 100755
index 000000000..bf335dd6e
--- /dev/null
+++ b/lib/newlib_tests/test_sh_newlib.sh
@@ -0,0 +1,105 @@
+#!/bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+# © 2018 SUSE LLC.
+# 
+# Author: Christian Lanig <clanig@suse.com>
+
+setup()
+{
+	path_sav=$PATH
+	PATH=""$PATH":$(dirname $0)/../../testcases/lib/"
+	color_blue='\033[1;34m'
+	color_green='\033[1;32m'
+	color_red='\033[1;31m'
+	standard_color='\033[0m'
+	tmp_dir="/tmp/sh_lib_tst-$$/"
+	mkdir $tmp_dir || cleanup 1
+	sh_lib_tst_dir="$(dirname $0)/shell/"
+	tst_files="$(ls $sh_lib_tst_dir)"
+	return 0
+}
+
+cleanup()
+{
+	[ -d "$tmp_dir" ] && rm -rf "$tmp_dir"
+	PATH="$PATH_sav"
+	exit $1
+}
+
+print_help()
+{
+	printf "#############################################################\n"
+	printf "# This script iterates over test cases for the new shell    #\n"
+	printf "# library and verifies the output.                          #\n"
+	printf "#############################################################\n"
+	printf "\n"
+	printf "Usage:\n"
+	printf "\t$(basename $0)\n"
+	printf "\t$(basename $0) [test file 1] [test file 2] ...\n\n"
+	exit 0;
+}
+
+parse_params()
+{
+	[ -n "$1" ] && tst_files=
+	while [ -n "$1" ]; do
+		case "$1" in
+			--help) print_help;;
+			-h) print_help;;
+			-*)
+				printf "Unknown positional parameter "$1".\n"
+				cleanup 1;;
+			*) tst_files="$tst_files $1";;
+		esac
+		shift
+	done
+	return 0
+}
+
+verify_output()
+{
+	[ -e "$sh_lib_tst_dir$tst" ] || { printf "$tst not found\n" && \
+			cleanup 1 ;}
+	# read all lines after line: `# output:`, and strip `# ` from beginning
+	sed -n -e '/^# output:/,$ p;' "$sh_lib_tst_dir$tst" | sed '1d; s/^# //'\
+			> "$tmp_dir$tst.wanted" || cleanup 1
+
+	actual_output=$($sh_lib_tst_dir$tst)
+	# remove control signs, add newline at EOF and store in temporary file
+	actual_output=$(printf "$actual_output\n" > $tmp_dir$tst".actual") ||
+			cleanup 1
+
+	cmp $tmp_dir$tst".actual" $tmp_dir$tst".wanted" > /dev/null && return 0
+	return 1
+}
+
+run_tests()
+{
+	pass_cnt=0
+	fail_cnt=0
+	printf "\n"
+	for tst in $tst_files; do
+		if verify_output; then
+			pass_cnt=$(expr $pass_cnt + 1)
+			printf ""$color_green"TPASS$standard_color $tst"
+			printf "\n"
+		else
+			fail_cnt=$(expr $fail_cnt + 1)
+			printf ""$color_red"TFAIL$standard_color $tst\n"
+			printf ""$color_blue"Diff:$standard_color\n"
+			diff -u "$tmp_dir$tst.actual" "$tmp_dir$tst.wanted"
+			printf "\n"
+		fi
+	done
+	printf "\nSummary:\n"
+	printf ""$color_red"Failed:$standard_color $fail_cnt\n"
+	printf ""$color_green"Passed:$standard_color $pass_cnt\n\n"
+	return $fail_cnt
+}
+
+setup
+parse_params "$@"
+run_tests
+res=$?
+cleanup $res
-- 
2.16.4

[LTP] [RFC PATCH 1/1] Add automated tests for shell lib

[Cyril Hrubis]

Hi!
> diff --git a/lib/newlib_tests/shell/test.TST_TEST.getopts.sh b/lib/newlib_tests/shell/test.TST_TEST.getopts.sh
> new file mode 100755
> index 000000000..65810b5e8
> --- /dev/null
> +++ b/lib/newlib_tests/shell/test.TST_TEST.getopts.sh
> @@ -0,0 +1,48 @@
> +#!/bin/sh
> +#
> +# Optional test command line parameters
> +#
> +
> +TST_OPTS="af:"
> +TST_USAGE=usage
> +TST_PARSE_ARGS=parse_args
> +TST_TESTFUNC=do_test
> +
> +. tst_test.sh
> +
> +ALTERNATIVE=0
> +MODE="foo"
> +
> +usage()
> +{
> +	cat << EOF
> +usage: $0 [-a] [-f <foo|bar>]
> +
> +OPTIONS
> +-a     Enable support for alternative foo
> +-f     Specify foo or bar mode
> +EOF
> +}
> +
> +parse_args()
> +{
> +	case $1 in
> +	a) ALTERNATIVE=1;;
> +	f) MODE="$2";;
> +	esac
> +}
> +
> +do_test()
> +{
> +	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
> +}
> +
> +tst_run
> +# output:
> +# test 1 TPASS: Test 1 passed with data '': a: '0', f: 'foo'
> +# 
> +# Summary:
> +# passed   1
> +# failed   0
> +# skipped  0
> +# warnings 0

We do need a way how to pass parameters to the tests so that we can test
this functionality properly (and also the default options that are
implemented in the test library).

Obvious way how to do this is to allow to have several output: blocks at
the end of the test with a different parameters, something as:

# output:
# test 1 TPASS: Test 1 passed with data '': a: '0', f: 'foo'
# 
# Summary:
# passed   1
# failed   0
# skipped  0
# warnings 0

# output:
# params: -a
# test 1 TPASS: Test 1 passed with data '': a: '1', f: 'foo'
# 
# Summary:
# passed   1
# failed   0
# skipped  0
# warnings 0

...

> diff --git a/lib/newlib_tests/test_sh_newlib.sh b/lib/newlib_tests/test_sh_newlib.sh
> new file mode 100755
> index 000000000..bf335dd6e
> --- /dev/null
> +++ b/lib/newlib_tests/test_sh_newlib.sh
> @@ -0,0 +1,105 @@
> +#!/bin/sh
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# ?? 2018 SUSE LLC.
> +# 
> +# Author: Christian Lanig <clanig@suse.com>
> +
> +setup()
> +{
> +	path_sav=$PATH

There is no need to save and restore the PATH, the script will be
executed in subshell and the changes to variables are never propagated
to the parent shell.

The only way how variables can be changed in the currently running shell
is to source the script, in that case the current shell will execute the
commands.

> +	PATH=""$PATH":$(dirname $0)/../../testcases/lib/"
> +
> +	color_blue='\033[1;34m'
> +	color_green='\033[1;32m'
> +	color_red='\033[1;31m'
> +	standard_color='\033[0m'
          ^
	  This is more of 'reset text attributes' as it not only changes
	  the color to default but also turns off bold, underscore,
	  blink, etc. So it would be better named something as
	  reset_attrs='\033[0m'

> +	tmp_dir="/tmp/sh_lib_tst-$$/"
> +	mkdir $tmp_dir || cleanup 1
> +	sh_lib_tst_dir="$(dirname $0)/shell/"
> +	tst_files="$(ls $sh_lib_tst_dir)"
> +	return 0
> +}
> +
> +cleanup()
> +{
> +	[ -d "$tmp_dir" ] && rm -rf "$tmp_dir"
> +	PATH="$PATH_sav"
> +	exit $1
> +}
> +
> +print_help()
> +{
> +	printf "#############################################################\n"
> +	printf "# This script iterates over test cases for the new shell    #\n"
> +	printf "# library and verifies the output.                          #\n"
> +	printf "#############################################################\n"
> +	printf "\n"
> +	printf "Usage:\n"
> +	printf "\t$(basename $0)\n"
> +	printf "\t$(basename $0) [test file 1] [test file 2] ...\n\n"
> +	exit 0;
> +}
> +
> +parse_params()
> +{
> +	[ -n "$1" ] && tst_files=
> +	while [ -n "$1" ]; do
> +		case "$1" in
> +			--help) print_help;;
> +			-h) print_help;;
> +			-*)
> +				printf "Unknown positional parameter "$1".\n"
> +				cleanup 1;;
> +			*) tst_files="$tst_files $1";;
> +		esac
> +		shift
> +	done
> +	return 0
> +}
> +
> +verify_output()
> +{
> +	[ -e "$sh_lib_tst_dir$tst" ] || { printf "$tst not found\n" && \
> +			cleanup 1 ;}
> +	# read all lines after line: `# output:`, and strip `# ` from beginning
> +	sed -n -e '/^# output:/,$ p;' "$sh_lib_tst_dir$tst" | sed '1d; s/^# //'\
> +			> "$tmp_dir$tst.wanted" || cleanup 1

Usually doing more complex text manipulation is easier with awk

In this case you can use gsub() function that will replace substring and
print all lines that were replaced (i.e. print all comments without the
hash and space):

awk 'gsub("^# ", "", $0)' script.sh

Then you can even get one of the output: blocks with:

awk -vb=1 'gsub("^# ", "", $0) {if (/output:/) {p++}; if (p==b && !/output:/) {print $0}};' script.sh
       ^
       This gives you content of comments starting at first # output:
       line up to the next # output: or end of line, which is exactly
       what we need if we want to support passing parameters to
       testcases. Then we can simply count how many # output: lines are
       in a file and then run this command in a loop extracting one
       example output after another.

> +	actual_output=$($sh_lib_tst_dir$tst)
> +	# remove control signs, add newline at EOF and store in temporary file
> +	actual_output=$(printf "$actual_output\n" > $tmp_dir$tst".actual") ||
> +			cleanup 1
> +
> +	cmp $tmp_dir$tst".actual" $tmp_dir$tst".wanted" > /dev/null && return 0
> +	return 1
> +}
> +
> +run_tests()
> +{
> +	pass_cnt=0
> +	fail_cnt=0
> +	printf "\n"
> +	for tst in $tst_files; do
> +		if verify_output; then
> +			pass_cnt=$(expr $pass_cnt + 1)

You can use arithmetic expansion here, i.e. pass_cnt=$((pass_cnt+1))

> +			printf ""$color_green"TPASS$standard_color $tst"
> +			printf "\n"
> +		else
> +			fail_cnt=$(expr $fail_cnt + 1)

And here as well.

> +			printf ""$color_red"TFAIL$standard_color $tst\n"
> +			printf ""$color_blue"Diff:$standard_color\n"
> +			diff -u "$tmp_dir$tst.actual" "$tmp_dir$tst.wanted"
> +			printf "\n"
> +		fi
> +	done
> +	printf "\nSummary:\n"
> +	printf ""$color_red"Failed:$standard_color $fail_cnt\n"
> +	printf ""$color_green"Passed:$standard_color $pass_cnt\n\n"
> +	return $fail_cnt
> +}
> +
> +setup
> +parse_params "$@"
> +run_tests
> +res=$?
> +cleanup $res
> -- 
> 2.16.4
> 

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 1/1] Add automated tests for shell lib

[Petr Vorel]

Hi,

> > +do_test()
> > +{
> > +	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
> > +}
> > +
> > +tst_run
> > +# output:
> > +# test 1 TPASS: Test 1 passed with data '': a: '0', f: 'foo'
> > +# 
> > +# Summary:
> > +# passed   1
> > +# failed   0
> > +# skipped  0
> > +# warnings 0

> We do need a way how to pass parameters to the tests so that we can test
> this functionality properly (and also the default options that are
> implemented in the test library).

> Obvious way how to do this is to allow to have several output: blocks at
> the end of the test with a different parameters, something as:

> # output:
> # test 1 TPASS: Test 1 passed with data '': a: '0', f: 'foo'

> # Summary:
> # passed   1
> # failed   0
> # skipped  0
> # warnings 0

> # output:
> # params: -a
> # test 1 TPASS: Test 1 passed with data '': a: '1', f: 'foo'

> # Summary:
> # passed   1
> # failed   0
> # skipped  0
> # warnings 0

> ...

Good idea!

...
> > +verify_output()
> > +{
> > +	[ -e "$sh_lib_tst_dir$tst" ] || { printf "$tst not found\n" && \
> > +			cleanup 1 ;}
> > +	# read all lines after line: `# output:`, and strip `# ` from beginning
> > +	sed -n -e '/^# output:/,$ p;' "$sh_lib_tst_dir$tst" | sed '1d; s/^# //'\
> > +			> "$tmp_dir$tst.wanted" || cleanup 1

> Usually doing more complex text manipulation is easier with awk

> In this case you can use gsub() function that will replace substring and
> print all lines that were replaced (i.e. print all comments without the
> hash and space):

> awk 'gsub("^# ", "", $0)' script.sh

> Then you can even get one of the output: blocks with:

> awk -vb=1 'gsub("^# ", "", $0) {if (/output:/) {p++}; if (p==b && !/output:/) {print $0}};' script.sh
>        ^
>        This gives you content of comments starting at first # output:
>        line up to the next # output: or end of line, which is exactly
>        what we need if we want to support passing parameters to
>        testcases. Then we can simply count how many # output: lines are
>        in a file and then run this command in a loop extracting one
>        example output after another.
I wonder whether "extracting one example output after another" is still good to
do in awk/sed/shell (with simple readable code). Probably yes, but I wouldn't
mind to have either python (pytest) or some perl code doing it.
Awk is everywhere and definitely more capable than sed or even shell, but not
everybody is fluent in it.


Kind regards,
Petr

[LTP] [RFC PATCH 1/1] Add automated tests for shell lib

[Petr Vorel]

Hi Christian,

some more comments (beside these mentioned by Cyril).
Mostly just shell code style comments.

> +test2()
> +{
> +	tst_res TPASS "Test $1 passed with no data ('$2')"
> +}
> +
> +tst_run
> +# output:
> +# test 1 TPASS: Test 1 passed with no data ('')
> +# test 2 TPASS: Test 2 passed with no data ('')
> +# 
Note, I'd prefer not have trailing whitespace after '#' (actually anywhere).
Not only it's unnecessary it's also problematic for 'git am' applying patches.
In my case all the tests failed because of it.
Also some people setup their editors / IDE to delete trailing whitespace.
(Not pointing all whitespace here.)

> +# Summary:
> +# passed   2
> +# failed   0
> +# skipped  0
> +# warnings 0

> diff --git a/lib/newlib_tests/test_sh_newlib.sh b/lib/newlib_tests/test_sh_newlib.sh
...
> +setup()
> +{
> +	path_sav=$PATH
> +	PATH=""$PATH":$(dirname $0)/../../testcases/lib/"
> +	color_blue='\033[1;34m'
> +	color_green='\033[1;32m'
> +	color_red='\033[1;31m'
> +	standard_color='\033[0m'

Maybe this variables can be outside of setup (no benefit to have them in setup.

> +	tmp_dir="/tmp/sh_lib_tst-$$/"

Although /tmp is everywhere, maybe using $TMPDIR would be good. See tst_test.sh
code:
		if [ -z "$TMPDIR" ]; then
			export TMPDIR="/tmp"
		fi

		TST_TMPDIR=$(mktemp -d "$TMPDIR/LTP_$TST_ID.XXXXXXXXXX")

		chmod 777 "$TST_TMPDIR"

> +	mkdir $tmp_dir || cleanup 1
> +	sh_lib_tst_dir="$(dirname $0)/shell/"
Maybe just tstdir ?
> +	tst_files="$(ls $sh_lib_tst_dir)"

> +print_help()
> +{
> +	printf "#############################################################\n"
> +	printf "# This script iterates over test cases for the new shell    #\n"
> +	printf "# library and verifies the output.                          #\n"
> +	printf "#############################################################\n"
> +	printf "\n"
> +	printf "Usage:\n"
> +	printf "\t$(basename $0)\n"
> +	printf "\t$(basename $0) [test file 1] [test file 2] ...\n\n"
> +	exit 0;

Maybe just:
    cat <<EOF
Unit tests for new shell API
Usage: $(basename $0) [ TEST1 ] [ TEST2 ] 
EOF

> +}
> +
> +parse_params()
> +{
> +	[ -n "$1" ] && tst_files=
> +	while [ -n "$1" ]; do
> +		case "$1" in
> +			--help) print_help;;
> +			-h) print_help;;
> +			-*)
> +				printf "Unknown positional parameter "$1".\n"
> +				cleanup 1;;
> +			*) tst_files="$tst_files $1";;
> +		esac
> +		shift
> +	done
> +	return 0
Why return, when you don't use it elsewhere?

...
> +verify_output()
> +{
> +	[ -e "$sh_lib_tst_dir$tst" ] || { printf "$tst not found\n" && \
> +			cleanup 1 ;}
This might be more readable (nitpicking):
if [ -n "$sh_lib_tst_dir$tst" ]; then
	echo "$tst not found"
	cleanup 1
fi

> +	# read all lines after line: `# output:`, and strip `# ` from beginning
> +	sed -n -e '/^# output:/,$ p;' "$sh_lib_tst_dir$tst" | sed '1d; s/^# //'\
> +			> "$tmp_dir$tst.wanted" || cleanup 1
> +
> +	actual_output=$($sh_lib_tst_dir$tst)

> +	# remove control signs, add newline at EOF and store in temporary file
> +	actual_output=$(printf "$actual_output\n" > $tmp_dir$tst".actual") ||
> +			cleanup 1
Suppose this part will be rewritten. But why do you assign empty string?
echo "$actual_output" > $tmp_dir$tst".actual || cleanup 1

...
> +
> +	cmp $tmp_dir$tst".actual" $tmp_dir$tst".wanted" > /dev/null && return 0
> +	return 1
This is enough (cmp returns it for you):
cmp -s "$tmp_dir$tst.actual" "$tmp_dir$tst.wanted"
> +}

...
> +run_tests()
> +{
> +	pass_cnt=0
> +	fail_cnt=0
> +	printf "\n"
> +	for tst in $tst_files; do
> +		if verify_output; then
> +			pass_cnt=$(expr $pass_cnt + 1)
> +			printf ""$color_green"TPASS$standard_color $tst"
                   ^
"" is empty string, why?
> +			printf "\n"
You can also use {} to separate variable name:
echo "${color_green}TPASS$standard_color $tst"
echo adds new line for you (for simple usage it's portable).
It's safer to quote whole string (with variables).

> +		else
> +			fail_cnt=$(expr $fail_cnt + 1)
> +			printf ""$color_red"TFAIL$standard_color $tst\n"
> +			printf ""$color_blue"Diff:$standard_color\n"
> +			diff -u "$tmp_dir$tst.actual" "$tmp_dir$tst.wanted"
> +			printf "\n"
> +		fi
> +	done
> +	printf "\nSummary:\n"
> +	printf ""$color_red"Failed:$standard_color $fail_cnt\n"
> +	printf ""$color_green"Passed:$standard_color $pass_cnt\n\n"

And how about adding helper functions print_red, print_green, print_blue?
Maybe not worth for this little script.

print_red()
{
	printf "$red$1$reset"
}
print_red "TFAIL
echo "$tst"

> +	return $fail_cnt
> +}
> +
> +setup
> +parse_params "$@"

> +run_tests
> +res=$?
> +cleanup $res
No need to have $res:
run_tests
cleanup $?


Kind regards,
Petr

[LTP] [RFC PATCH v2 0/1] Add automated tests for shell lib

[Christian Lanig]

Hi,

I am sorry that my reply took so long. Thank you very much for your comments
which helped me to improve the code.
I have made an overhaul of the code based on your suggestions but an option to
pass parameters to the tests is still not implemented. 
I'm sending you what I have so far because it would help me when you could
provide me more information regarding this.

I agree that it is difficult to find a way to process text the way needed here
without disregarding the Linux Kernel Coding Style that asks for code which is
transparent and easy to read when we only consider Shell commands. To address
this issue I have drafted a Python script that might go in the tools folder and
may perhaps be helpful in other cases as well. Since there are already Python
files in the project it might be a valid option.

Thank you very much for your patience.

Regards,
Christian Lanig

Christian Lanig (1):
  Add automated tests for shell lib

 doc/write-tests-for-shell-lib.txt                  |  59 ++++++++++
 lib/newlib_tests/shell/test_sh_newlib.sh           | 102 ++++++++++++++++++
 .../testcases/test.TST_TEST.TST_CNT.separate.sh    |  30 ++++++
 .../shell/testcases/test.TST_TEST.TST_CNT.sh       |  28 +++++
 .../shell/testcases/test.TST_TEST.getopts.sh       |  49 +++++++++
 lib/newlib_tests/shell/testcases/test.TST_TEST.sh  |  33 ++++++
 .../test.TST_TEST_DATA.TST_CNT.separate.sh         |  33 ++++++
 .../shell/testcases/test.TST_TEST_DATA.TST_CNT.sh  |  31 ++++++
 .../shell/testcases/test.TST_TEST_DATA.getopts.sh  |  51 +++++++++
 .../shell/testcases/test.TST_TEST_DATA.sh          |  25 +++++
 .../testcases/test.TST_TEST_DATA_IFS.getopts.sh    |  53 +++++++++
 .../shell/testcases/test.TST_TEST_DATA_IFS.sh      |  28 +++++
 tools/lookup_split_cut.py                          | 120 +++++++++++++++++++++
 13 files changed, 642 insertions(+)
 create mode 100644 doc/write-tests-for-shell-lib.txt
 create mode 100755 lib/newlib_tests/shell/test_sh_newlib.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.separate.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.getopts.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.separate.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.getopts.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.getopts.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.sh
 create mode 100755 tools/lookup_split_cut.py

-- 
2.16.4

[LTP] [RFC PATCH v2 1/1] Add automated tests for shell lib

[Christian Lanig]

Signed-off-by: Christian Lanig <clanig@suse.com>
---
 doc/write-tests-for-shell-lib.txt                  |  59 ++++++++++
 lib/newlib_tests/shell/test_sh_newlib.sh           | 102 ++++++++++++++++++
 .../testcases/test.TST_TEST.TST_CNT.separate.sh    |  30 ++++++
 .../shell/testcases/test.TST_TEST.TST_CNT.sh       |  28 +++++
 .../shell/testcases/test.TST_TEST.getopts.sh       |  49 +++++++++
 lib/newlib_tests/shell/testcases/test.TST_TEST.sh  |  33 ++++++
 .../test.TST_TEST_DATA.TST_CNT.separate.sh         |  33 ++++++
 .../shell/testcases/test.TST_TEST_DATA.TST_CNT.sh  |  31 ++++++
 .../shell/testcases/test.TST_TEST_DATA.getopts.sh  |  51 +++++++++
 .../shell/testcases/test.TST_TEST_DATA.sh          |  25 +++++
 .../testcases/test.TST_TEST_DATA_IFS.getopts.sh    |  53 +++++++++
 .../shell/testcases/test.TST_TEST_DATA_IFS.sh      |  28 +++++
 tools/lookup_split_cut.py                          | 120 +++++++++++++++++++++
 13 files changed, 642 insertions(+)
 create mode 100644 doc/write-tests-for-shell-lib.txt
 create mode 100755 lib/newlib_tests/shell/test_sh_newlib.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.separate.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.getopts.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.separate.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.getopts.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.getopts.sh
 create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.sh
 create mode 100755 tools/lookup_split_cut.py

diff --git a/doc/write-tests-for-shell-lib.txt b/doc/write-tests-for-shell-lib.txt
new file mode 100644
index 000000000..7e4e6f566
--- /dev/null
+++ b/doc/write-tests-for-shell-lib.txt
@@ -0,0 +1,59 @@
+How to format tests in order to test the shell library
+======================================================
+
+It is important to test the Linux kernel functionality but also to make sure
+that LTP is running correctly itself. For this reason it is useful to test
+intrinsic functionality of LTP.
+
+1. Running tests for the shell library
+--------------------------------------
+The test cases reside in the folder `lib/newlib_tests/shell`. A script executing
+them one by one is located in the folder `lib/newlib_tests`. You can execute
+this script to test all cases or specify test cases to be run. The script is
+called `test_sh_newlib.sh`.
+
+2. Writing tests for the shell library
+--------------------------------------
+The tests are written like all other test cases using the shell library.
+Additionally, at the end of the file the desired output is added. As an example:
+
+[source,shell]
+-------------------------------------------------------------------------------
+#!/bin/sh
+#
+# This is a basic test for true shell buildin
+#
+
+TST_TESTFUNC=do_test
+. tst_test.sh
+
+do_test()
+{
+	true
+	ret=$?
+
+	tst_res TINFO "Test $1 passed with no data ('$2')"
+
+	if [ $ret -eq 0 ]; then
+		tst_res TPASS "true returned 0"
+	else
+		tst_res TFAIL "true returned $ret"
+	fi
+}
+
+tst_run
+# output:
+# test 1 TINFO: Test 1 passed with no data ('')
+# test 1 TPASS: true returned 0
+#
+# Summary:
+# passed   1
+# failed   0
+# skipped  0
+# warnings 0
+-------------------------------------------------------------------------------
+
+The most noticeable thing is to add the line `# output:` to show the parser that
+parsing should start in the following line. For the following lines the `# `
+will be stripped before the output is then compared with the actual output that
+gets printed on the terminal when running the test.
diff --git a/lib/newlib_tests/shell/test_sh_newlib.sh b/lib/newlib_tests/shell/test_sh_newlib.sh
new file mode 100755
index 000000000..4aa19555b
--- /dev/null
+++ b/lib/newlib_tests/shell/test_sh_newlib.sh
@@ -0,0 +1,102 @@
+#!/bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+# (c) 2019 SUSE LLC
+#
+# Author: Christian Lanig <clanig@suse.com>
+
+PATH="${PATH}:$(dirname $(readlink -f $0))/../../../testcases/lib/"
+if [ -z "$TMPDIR" ]; then
+	export TMPDIR="/tmp"
+fi
+color_blue='\033[1;34m'
+color_green='\033[1;32m'
+color_red='\033[1;31m'
+reset_attr='\033[0m'
+tmp="${TMPDIR}/sh_lib_tst-${$}/"
+mkdir $tmp || cleanup 1
+parent_dir=$(dirname $(readlink -f $0))/
+tooldir=${parent_dir}/../../../tools/
+testdir=${parent_dir}testcases/
+tst_files=$(ls $testdir)
+
+cleanup()
+{
+	[ -d "$tmp" ] && rm -rf "$tmp"
+	exit $1
+}
+
+print_help()
+{
+	cat <<EOF
+
+????????????????????????????????????????????????????????????????????????????????
+? This Shell script iterates over test cases for the new Shell library and     ?
+? verifies the output.                                                         ?
+????????????????????????????????????????????????????????????????????????????????
+
+	Usage:
+		$(basename $0) [TEST_FILE_1] [TEST_FILE_2]
+
+EOF
+	exit 0
+}
+
+parse_params()
+{
+	[ -n "$1" ] && tst_files=
+	while [ -n "$1" ]; do
+		case "$1" in
+			--help) print_help;;
+			-h) print_help;;
+			-*)
+				printf "Unknown positional parameter ${1}.\n"
+				cleanup 1;;
+			*) tst_files="$tst_files $1";;
+		esac
+		shift
+	done
+}
+
+verify_output()
+{
+	if [ ! -e "${testdir}$tst" ]; then
+		printf "$tst not found\n"
+		cleanup 1
+	fi
+
+	${tooldir}lookup_split_cut.py -f ${testdir}$tst -d $tmp \
+			-s '# output:\n' -c '# {0,1}' || cleanup 1
+
+	"${testdir}$tst" > "${tmp}$tst.actual" || cleanup 1
+	cmp -s "${tmp}$tst.actual" "${tmp}${tst}_out/out.1" && return 0
+	return 1
+}
+
+run_tests()
+{
+	pass_cnt=0
+	fail_cnt=0
+	printf "\n"
+	for tst in $tst_files; do
+		if verify_output; then
+			pass_cnt=$(($pass_cnt + 1))
+			printf "${color_green}TPASS$reset_attr ${tst}\n"
+		else
+			fail_cnt=$(($fail_cnt + 1))
+			printf "${color_red}TFAIL$reset_attr ${tst}\n"
+			printf "${color_blue}Diff:${reset_attr}\n"
+			diff -u "${tmp}${tst}.actual" \
+					"${tmp}${tst}_out/out.1"
+			printf "\n"
+		fi
+	done
+	printf "\nSummary:\n"
+	printf "${color_red}Failed:$reset_attr $fail_cnt\n"
+	printf "${color_green}Passed:$reset_attr $pass_cnt\n\n"
+	return $fail_cnt
+}
+
+parse_params "$@"
+run_tests
+cleanup $?
diff --git a/lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.separate.sh b/lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.separate.sh
new file mode 100755
index 000000000..333c4f5fa
--- /dev/null
+++ b/lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.separate.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+#
+# Example test with tests in separate functions
+#
+
+TST_TESTFUNC=test
+TST_CNT=2
+. tst_test.sh
+
+test1()
+{
+	tst_res TPASS "Test $1 passed with no data ('$2')"
+}
+
+test2()
+{
+	tst_res TPASS "Test $1 passed with no data ('$2')"
+}
+
+tst_run
+# output:
+# test 1 TINFO: timeout per run is 0h 5m 0s
+# test 1 TPASS: Test 1 passed with no data ('')
+# test 2 TPASS: Test 2 passed with no data ('')
+#
+# Summary:
+# passed   2
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.sh b/lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.sh
new file mode 100755
index 000000000..73abfc8b3
--- /dev/null
+++ b/lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+#
+# Example test with tests in a single function
+#
+
+TST_TESTFUNC=do_test
+TST_CNT=2
+. tst_test.sh
+
+do_test()
+{
+	case $1 in
+	1) tst_res TPASS "Test $1 passed with no data ('$2')";;
+	2) tst_res TPASS "Test $1 passed with no data ('$2')";;
+	esac
+}
+
+tst_run
+# output:
+# test 1 TINFO: timeout per run is 0h 5m 0s
+# test 1 TPASS: Test 1 passed with no data ('')
+# test 2 TPASS: Test 2 passed with no data ('')
+#
+# Summary:
+# passed   2
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/testcases/test.TST_TEST.getopts.sh b/lib/newlib_tests/shell/testcases/test.TST_TEST.getopts.sh
new file mode 100755
index 000000000..35a700bb2
--- /dev/null
+++ b/lib/newlib_tests/shell/testcases/test.TST_TEST.getopts.sh
@@ -0,0 +1,49 @@
+#!/bin/sh
+#
+# Optional test command line parameters
+#
+
+TST_OPTS="af:"
+TST_USAGE=usage
+TST_PARSE_ARGS=parse_args
+TST_TESTFUNC=do_test
+
+. tst_test.sh
+
+ALTERNATIVE=0
+MODE="foo"
+
+usage()
+{
+	cat << EOF
+usage: $0 [-a] [-f <foo|bar>]
+
+OPTIONS
+-a     Enable support for alternative foo
+-f     Specify foo or bar mode
+EOF
+}
+
+parse_args()
+{
+	case $1 in
+	a) ALTERNATIVE=1;;
+	f) MODE="$2";;
+	esac
+}
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
+}
+
+tst_run
+# output:
+# test 1 TINFO: timeout per run is 0h 5m 0s
+# test 1 TPASS: Test 1 passed with data '': a: '0', f: 'foo'
+#
+# Summary:
+# passed   1
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/testcases/test.TST_TEST.sh b/lib/newlib_tests/shell/testcases/test.TST_TEST.sh
new file mode 100755
index 000000000..930900e1d
--- /dev/null
+++ b/lib/newlib_tests/shell/testcases/test.TST_TEST.sh
@@ -0,0 +1,33 @@
+#!/bin/sh
+#
+# This is a basic test for true shell buildin
+#
+
+TST_TESTFUNC=do_test
+. tst_test.sh
+
+do_test()
+{
+	true
+	ret=$?
+
+	tst_res TINFO "Test $1 passed with no data ('$2')"
+
+	if [ $ret -eq 0 ]; then
+		tst_res TPASS "true returned 0"
+	else
+		tst_res TFAIL "true returned $ret"
+	fi
+}
+
+tst_run
+# output:
+# test 1 TINFO: timeout per run is 0h 5m 0s
+# test 1 TINFO: Test 1 passed with no data ('')
+# test 1 TPASS: true returned 0
+#
+# Summary:
+# passed   1
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.separate.sh b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.separate.sh
new file mode 100755
index 000000000..1faa01a57
--- /dev/null
+++ b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.separate.sh
@@ -0,0 +1,33 @@
+#!/bin/sh
+#
+# Example test with tests in separate functions, using $TST_TEST_DATA and $TST_CNT
+#
+
+TST_TESTFUNC=test
+TST_CNT=2
+TST_TEST_DATA="foo:bar:d dd"
+. tst_test.sh
+
+test1()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+test2()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+tst_run
+# output:
+# test 1 TINFO: timeout per run is 0h 5m 0s
+# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
+# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
+# test 3 TPASS: Test 1 passed with data 'dd'
+# test 4 TPASS: Test 2 passed with data 'dd'
+#
+# Summary:
+# passed   4
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.sh b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.sh
new file mode 100755
index 000000000..889fc09c3
--- /dev/null
+++ b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT
+#
+
+TST_TESTFUNC=do_test
+TST_CNT=2
+TST_TEST_DATA="foo:bar:d dd"
+. tst_test.sh
+
+do_test()
+{
+	case $1 in
+	1) tst_res TPASS "Test $1 passed with data '$2'";;
+	2) tst_res TPASS "Test $1 passed with data '$2'";;
+	esac
+}
+
+tst_run
+# output:
+# test 1 TINFO: timeout per run is 0h 5m 0s
+# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
+# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
+# test 3 TPASS: Test 1 passed with data 'dd'
+# test 4 TPASS: Test 2 passed with data 'dd'
+#
+# Summary:
+# passed   4
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.getopts.sh b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.getopts.sh
new file mode 100755
index 000000000..ba880f891
--- /dev/null
+++ b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.getopts.sh
@@ -0,0 +1,51 @@
+#!/bin/sh
+#
+# Optional test command line parameters
+#
+
+TST_OPTS="af:"
+TST_USAGE=usage
+TST_PARSE_ARGS=parse_args
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo0:bar:d dd"
+
+. tst_test.sh
+
+ALTERNATIVE=0
+MODE="foo"
+
+usage()
+{
+	cat << EOF
+usage: $0 [-a] [-f <foo|bar>]
+
+OPTIONS
+-a     Enable support for alternative foo
+-f     Specify foo or bar mode
+EOF
+}
+
+parse_args()
+{
+	case $1 in
+	a) ALTERNATIVE=1;;
+	f) MODE="$2";;
+	esac
+}
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
+}
+
+tst_run
+# output:
+# test 1 TINFO: timeout per run is 0h 5m 0s
+# test 1 TPASS: Test 1 passed with data 'foo0:bar:d': a: '0', f: 'foo'
+# test 2 TPASS: Test 1 passed with data 'dd': a: '0', f: 'foo'
+#
+# Summary:
+# passed   2
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.sh b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.sh
new file mode 100755
index 000000000..083943833
--- /dev/null
+++ b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.sh
@@ -0,0 +1,25 @@
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA
+#
+
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo:bar:d dd"
+. tst_test.sh
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+tst_run
+# output:
+# test 1 TINFO: timeout per run is 0h 5m 0s
+# test 1 TPASS: Test 1 passed with data 'foo:bar:d'
+# test 2 TPASS: Test 1 passed with data 'dd'
+#
+# Summary:
+# passed   2
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.getopts.sh b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.getopts.sh
new file mode 100755
index 000000000..eb83d2e34
--- /dev/null
+++ b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.getopts.sh
@@ -0,0 +1,53 @@
+#!/bin/sh
+#
+# Optional test command line parameters
+#
+
+TST_OPTS="af:"
+TST_USAGE=usage
+TST_PARSE_ARGS=parse_args
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo0:bar:d dd"
+TST_TEST_DATA_IFS=":"
+
+. tst_test.sh
+
+ALTERNATIVE=0
+MODE="foo"
+
+usage()
+{
+	cat << EOF
+usage: $0 [-a] [-f <foo|bar>]
+
+OPTIONS
+-a     Enable support for alternative foo
+-f     Specify foo or bar mode
+EOF
+}
+
+parse_args()
+{
+	case $1 in
+	a) ALTERNATIVE=1;;
+	f) MODE="$2";;
+	esac
+}
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'"
+}
+
+tst_run
+# output:
+# test 1 TINFO: timeout per run is 0h 5m 0s
+# test 1 TPASS: Test 1 passed with data 'foo0': a: '0', f: 'foo'
+# test 2 TPASS: Test 1 passed with data 'bar': a: '0', f: 'foo'
+# test 3 TPASS: Test 1 passed with data 'd dd': a: '0', f: 'foo'
+#
+# Summary:
+# passed   3
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.sh b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.sh
new file mode 100755
index 000000000..6c6f904b7
--- /dev/null
+++ b/lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+#
+# Example test with tests in a single function, using $TST_TEST_DATA and
+# $TST_TEST_DATA_IFS
+#
+
+TST_TESTFUNC=do_test
+TST_TEST_DATA="foo:bar:d dd"
+TST_TEST_DATA_IFS=":"
+. tst_test.sh
+
+do_test()
+{
+	tst_res TPASS "Test $1 passed with data '$2'"
+}
+
+tst_run
+# output:
+# test 1 TINFO: timeout per run is 0h 5m 0s
+# test 1 TPASS: Test 1 passed with data 'foo'
+# test 2 TPASS: Test 1 passed with data 'bar'
+# test 3 TPASS: Test 1 passed with data 'd dd'
+#
+# Summary:
+# passed   3
+# failed   0
+# skipped  0
+# warnings 0
diff --git a/tools/lookup_split_cut.py b/tools/lookup_split_cut.py
new file mode 100755
index 000000000..2b3388ada
--- /dev/null
+++ b/tools/lookup_split_cut.py
@@ -0,0 +1,120 @@
+#!/usr/bin/env python
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+# (c) 2019 SUSE LLC
+#
+# Author: Christian Lanig <clanig@suse.com>
+
+import sys
+import os
+import re
+from sys import argv
+from os import makedirs, path
+
+src_file_path = None
+dest = None
+delim = None
+pattern = None
+perim = None
+
+argv.reverse()
+basename = path.split(argv.pop())[1]
+
+def print_help():
+
+	help = """
+	This script can look up a passage in a specified text pattern, split a
+	text file and cut a pattern. The operation chain is:
+
+		lookup > split > cut
+
+	The output files are written to the specified destination directory.
+
+	Usage:
+	"""
+	help += "\n\t\t" + basename + " -f [PATH] -d [PATH] -l " \
+			+ "[PERIMETER] -s [DELIMITER] \n" \
+			+ "\t\t\t\t -c [PATTERN]\n\n"
+	help += """
+	-h, --help
+		print this help
+	-f, --file
+		source file to be processed
+	-d, --destination
+		destination path
+	-l, --lookup
+		look for data in text passage
+	-s, --split
+		split file by delimiter
+	-c, --cut
+		cut pattern from file
+	"""
+
+	print(help)
+	sys.exit(0)
+
+def get_next_arg():
+	if argv:
+		return argv.pop()
+	else:
+		print("Missing parameter. Run with \"--help\" for information.")
+		sys.exit(1)
+
+while argv:
+	arg = argv.pop()
+	if arg in ["-h", "--help"]:
+		print_help()
+	elif arg in ["-f", "--file"]:
+		src_file_path = get_next_arg()
+	elif arg in ["-d", "--destination"]:
+		dest = get_next_arg()
+	elif arg in ["-l", "--lookup"]:
+		perim = get_next_arg()
+	elif arg in ["-s", "--split"]:
+		delim = get_next_arg()
+	elif arg in ["-c", "--cut"]:
+		pattern = get_next_arg()
+	else:
+		print("Illegal argument. Run with \"--help\" for information.")
+		sys.exit(1)
+
+if not src_file_path:
+	print("Input file has to be specified.")
+	sys.exit(1)
+
+if not delim and not pattern and not perim:
+	print("Missing parameters. Run with \"--help\" for information.")
+	sys.exit(1)
+
+src_file = open(src_file_path, "r")
+src = src_file.read()
+src_file.close()
+
+capture = 0
+if perim:
+	try:
+		capture = re.search(perim, src).group(1)
+	except:
+		pass
+
+if delim:
+	src_file_name = path.split(src_file_path)[1]
+	out_dir = dest + "/" + src_file_name + "_out"
+
+	if not path.exists(out_dir):
+		makedirs(out_dir)
+
+	src = re.split(delim, src)
+else:
+	src = [src]
+
+if pattern:
+	for i in range(len(src)):
+		src[i] = re.sub(pattern, "", src[i])
+if delim or pattern:
+	for i in range(len(src)):
+		out_file = open(out_dir + "/out." + str(i), "w")
+		out_file.write(src[i])
+		out_file.close()
+
+sys.exit(capture)
-- 
2.16.4

[LTP] [RFC PATCH v2 0/1] Add automated tests for shell lib

[Petr Vorel]

Hi Christian,

> Hi,

> I am sorry that my reply took so long. Thank you very much for your comments
> which helped me to improve the code.
> I have made an overhaul of the code based on your suggestions but an option to
> pass parameters to the tests is still not implemented. 
> I'm sending you what I have so far because it would help me when you could
> provide me more information regarding this.

> I agree that it is difficult to find a way to process text the way needed here
> without disregarding the Linux Kernel Coding Style that asks for code which is
> transparent and easy to read when we only consider Shell commands. To address
> this issue I have drafted a Python script that might go in the tools folder and
> may perhaps be helpful in other cases as well. Since there are already Python
> files in the project it might be a valid option.

> Thank you very much for your patience.
No problem, thanks continue for it.

I'm also not sure if we should put them into separate testcases/ directory.
I'd put them into shell/ directory and rename test_sh_newlib.sh to run.sh or
something simple.

BTW (nit, no hard feeling :) as lib/newlib_tests/test.*.sh are based on my
previous work, you could might mention it in the change log. You updated them to
have timeout output, so it's your work, (if not, it'd be better to send them as
mine separate

Kind regards,
Petr

[LTP] [RFC PATCH v2 1/1] Add automated tests for shell lib

[Petr Vorel]

Hi Christian,

thank you for working on it.

TL;DR: looks good to me, I have some code style objections,
(use echo instead of printf, use $foo instead of ${foo} when possible)
+ some minor suggestions below.

> +++ b/doc/write-tests-for-shell-lib.txt
> @@ -0,0 +1,59 @@
> +How to format tests in order to test the shell library
> +======================================================
> +
> +It is important to test the Linux kernel functionality but also to make sure
> +that LTP is running correctly itself. For this reason it is useful to test
> +intrinsic functionality of LTP.
> +
> +1. Running tests for the shell library
> +--------------------------------------
> +The test cases reside in the folder `lib/newlib_tests/shell`. A script executing
> +them one by one is located in the folder `lib/newlib_tests`. You can execute
> +this script to test all cases or specify test cases to be run. The script is
> +called `test_sh_newlib.sh`.
> +
> +2. Writing tests for the shell library
> +--------------------------------------
> +The tests are written like all other test cases using the shell library.
> +Additionally, at the end of the file the desired output is added. As an example:
Nice. I wonder if it should be in doc/test-writing-guidelines.txt. But this
section is already too big, so it's probably good that it's separate.
Then we need to add page this to wiki (simple).

BTW I plan to introduce make check, which will run this and other checks as well.

> +
> +[source,shell]
> +-------------------------------------------------------------------------------
> +#!/bin/sh
> +#
> +# This is a basic test for true shell buildin
typo: builtin
> +#
nit: I'd remove this empty lines just with '#'
> +
> +TST_TESTFUNC=do_test
> +. tst_test.sh
> +
> +do_test()
> +{
> +	true
> +	ret=$?
> +
> +	tst_res TINFO "Test $1 passed with no data ('$2')"
> +
> +	if [ $ret -eq 0 ]; then
> +		tst_res TPASS "true returned 0"
> +	else
> +		tst_res TFAIL "true returned $ret"
> +	fi
> +}
> +
> +tst_run
> +# output:
> +# test 1 TINFO: Test 1 passed with no data ('')
> +# test 1 TPASS: true returned 0
> +#
> +# Summary:
> +# passed   1
> +# failed   0
> +# skipped  0
> +# warnings 0
> +-------------------------------------------------------------------------------
> +
> +The most noticeable thing is to add the line `# output:` to show the parser that
> +parsing should start in the following line. For the following lines the `# `
> +will be stripped before the output is then compared with the actual output that
> +gets printed on the terminal when running the test.
> diff --git a/lib/newlib_tests/shell/test_sh_newlib.sh b/lib/newlib_tests/shell/test_sh_newlib.sh
> new file mode 100755
> index 000000000..4aa19555b
> --- /dev/null
> +++ b/lib/newlib_tests/shell/test_sh_newlib.sh
> @@ -0,0 +1,102 @@
> +#!/bin/sh
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# (c) 2019 SUSE LLC
> +#
> +# Author: Christian Lanig <clanig@suse.com>
> +
> +PATH="${PATH}:$(dirname $(readlink -f $0))/../../../testcases/lib/"
> +if [ -z "$TMPDIR" ]; then
> +	export TMPDIR="/tmp"
> +fi
> +color_blue='\033[1;34m'
> +color_green='\033[1;32m'
> +color_red='\033[1;31m'
> +reset_attr='\033[0m'
I'd prefer not reimplementing tst_ansi_color.sh.
IMHO it'd be better either use part of it (or tst_ansi_color.sh)
or not use colors at all.

> +tmp="${TMPDIR}/sh_lib_tst-${$}/"
nit: it can be $$ (instead of ${$})
> +mkdir $tmp || cleanup 1
> +parent_dir=$(dirname $(readlink -f $0))/
> +tooldir=${parent_dir}/../../../tools/
> +testdir=${parent_dir}testcases/
> +tst_files=$(ls $testdir)
> +
> +cleanup()
> +{
> +	[ -d "$tmp" ] && rm -rf "$tmp"
nit: 1 could be a default parameter.
> +	exit $1
> +}
> +
> +print_help()
> +{
> +	cat <<EOF
> +
> +????????????????????????????????????????????????????????????????????????????????
> +? This Shell script iterates over test cases for the new Shell library and     ?
> +? verifies the output.                                                         ?
> +????????????????????????????????????????????????????????????????????????????????
nit: I'd prefer to use ASCII (-) than unicode (?).

> +
> +	Usage:
> +		$(basename $0) [TEST_FILE_1] [TEST_FILE_2]
> +
nit: another space not needed.
> +EOF
> +	exit 0
> +}
> +
> +parse_params()
> +{
> +	[ -n "$1" ] && tst_files=
> +	while [ -n "$1" ]; do
We usually prefer to use getopts, which does not allow long opts.
It's ok for this small usage, but for more complicated output it's better not
reimplementing it.
> +		case "$1" in
> +			--help) print_help;;
> +			-h) print_help;;
> +			-*)
> +				printf "Unknown positional parameter ${1}.\n"
maybe use simple this simpler form:
				echo "Unknown positional parameter $1"

> +				cleanup 1;;
> +			*) tst_files="$tst_files $1";;
> +		esac
> +		shift
> +	done
> +}
> +
> +verify_output()
> +{
> +	if [ ! -e "${testdir}$tst" ]; then
This can safely be "$testdir$tst"
> +		printf "$tst not found\n"
Again, use echo
> +		cleanup 1
> +	fi
> +
> +	${tooldir}lookup_split_cut.py -f ${testdir}$tst -d $tmp \
> +			-s '# output:\n' -c '# {0,1}' || cleanup 1
> +
> +	"${testdir}$tst" > "${tmp}$tst.actual" || cleanup 1
Again, ${} is not necessary.
> +	cmp -s "${tmp}$tst.actual" "${tmp}${tst}_out/out.1" && return 0
+ check for cmp.
> +	return 1
> +}
> +
> +run_tests()
> +{
> +	pass_cnt=0
> +	fail_cnt=0
> +	printf "\n"
again, echo is better here.
> +	for tst in $tst_files; do
> +		if verify_output; then
> +			pass_cnt=$(($pass_cnt + 1))
> +			printf "${color_green}TPASS$reset_attr ${tst}\n"
> +		else
> +			fail_cnt=$(($fail_cnt + 1))
> +			printf "${color_red}TFAIL$reset_attr ${tst}\n"
> +			printf "${color_blue}Diff:${reset_attr}\n"
> +			diff -u "${tmp}${tst}.actual" \
We might want to check for diff being available before we use.
> +					"${tmp}${tst}_out/out.1"
> +			printf "\n"
> +		fi
> +	done
> +	printf "\nSummary:\n"
> +	printf "${color_red}Failed:$reset_attr $fail_cnt\n"
> +	printf "${color_green}Passed:$reset_attr $pass_cnt\n\n"
> +	return $fail_cnt
> +}
... (more tests)

> diff --git a/tools/lookup_split_cut.py b/tools/lookup_split_cut.py
> new file mode 100755
> index 000000000..2b3388ada
> --- /dev/null
> +++ b/tools/lookup_split_cut.py
> @@ -0,0 +1,120 @@
> +#!/usr/bin/env python
I guess this should be python3.
I'd be a bit careful to bring python as another dependency,
(there was some awk solution for this, proposed by Cyril),
but as python is everywhere, it shouldn't be a problem.
(We definitely don't want python on SUT, these tests will be eventually
rewritten into C or shell.)

> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# (c) 2019 SUSE LLC
> +#
> +# Author: Christian Lanig <clanig@suse.com>
> +
> +import sys
> +import os
> +import re
> +from sys import argv
> +from os import makedirs, path
> +
> +src_file_path = None
> +dest = None
> +delim = None
> +pattern = None
> +perim = None
> +
> +argv.reverse()
> +basename = path.split(argv.pop())[1]
> +
> +def print_help():
> +
> +	help = """
> +	This script can look up a passage in a specified text pattern, split a
> +	text file and cut a pattern. The operation chain is:
> +
> +		lookup > split > cut
> +
> +	The output files are written to the specified destination directory.
> +
> +	Usage:
> +	"""
> +	help += "\n\t\t" + basename + " -f [PATH] -d [PATH] -l " \
> +			+ "[PERIMETER] -s [DELIMITER] \n" \
> +			+ "\t\t\t\t -c [PATTERN]\n\n"
nit: you can use format() to use variables in multiline strings.

> +	help += """
> +	-h, --help
> +		print this help
> +	-f, --file
> +		source file to be processed
> +	-d, --destination
> +		destination path
> +	-l, --lookup
> +		look for data in text passage
> +	-s, --split
> +		split file by delimiter
> +	-c, --cut
> +		cut pattern from file
> +	"""
> +
> +	print(help)
> +	sys.exit(0)
> +
> +def get_next_arg():
> +	if argv:
> +		return argv.pop()
> +	else:
> +		print("Missing parameter. Run with \"--help\" for information.")
> +		sys.exit(1)
> +
> +while argv:
> +	arg = argv.pop()
> +	if arg in ["-h", "--help"]:
> +		print_help()
> +	elif arg in ["-f", "--file"]:
> +		src_file_path = get_next_arg()
> +	elif arg in ["-d", "--destination"]:
> +		dest = get_next_arg()
> +	elif arg in ["-l", "--lookup"]:
> +		perim = get_next_arg()
> +	elif arg in ["-s", "--split"]:
> +		delim = get_next_arg()
> +	elif arg in ["-c", "--cut"]:
> +		pattern = get_next_arg()
> +	else:
> +		print("Illegal argument. Run with \"--help\" for information.")
I'd print help here.
> +		sys.exit(1)
> +
> +if not src_file_path:
> +	print("Input file has to be specified.")
> +	sys.exit(1)
> +
> +if not delim and not pattern and not perim:
> +	print("Missing parameters. Run with \"--help\" for information.")
> +	sys.exit(1)
> +
> +src_file = open(src_file_path, "r")
> +src = src_file.read()
> +src_file.close()
> +
> +capture = 0
> +if perim:
> +	try:
> +		capture = re.search(perim, src).group(1)
> +	except:
> +		pass
> +
> +if delim:
> +	src_file_name = path.split(src_file_path)[1]
> +	out_dir = dest + "/" + src_file_name + "_out"
> +
> +	if not path.exists(out_dir):
> +		makedirs(out_dir)
> +
> +	src = re.split(delim, src)
> +else:
> +	src = [src]
> +
> +if pattern:
> +	for i in range(len(src)):
> +		src[i] = re.sub(pattern, "", src[i])
> +if delim or pattern:
> +	for i in range(len(src)):
> +		out_file = open(out_dir + "/out." + str(i), "w")
> +		out_file.write(src[i])
> +		out_file.close()
> +
> +sys.exit(capture)

Kind regards,
Petr

[LTP] [RFC PATCH v2 1/1] Add automated tests for shell lib

[Clemens Famulla-Conrad]

Hi Christian,

This tests are nice! Lets keep going with it.

On Thu, 2019-08-22 at 21:12 +0200, Christian Lanig wrote:
<snip>
> diff --git a/lib/newlib_tests/shell/test_sh_newlib.sh
> b/lib/newlib_tests/shell/test_sh_newlib.sh
> new file mode 100755
> index 000000000..4aa19555b
> --- /dev/null
> +++ b/lib/newlib_tests/shell/test_sh_newlib.sh
> @@ -0,0 +1,102 @@
> +#!/bin/sh
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# (c) 2019 SUSE LLC
> +#
> +# Author: Christian Lanig <clanig@suse.com>
> +
> +PATH="${PATH}:$(dirname $(readlink -f $0))/../../../testcases/lib/"
> +if [ -z "$TMPDIR" ]; then
> +	export TMPDIR="/tmp"
> +fi
> +color_blue='\033[1;34m'
> +color_green='\033[1;32m'
> +color_red='\033[1;31m'
> +reset_attr='\033[0m'
> +tmp="${TMPDIR}/sh_lib_tst-${$}/"
> +mkdir $tmp || cleanup 1
> +parent_dir=$(dirname $(readlink -f $0))/
> +tooldir=${parent_dir}/../../../tools/
> +testdir=${parent_dir}testcases/
> +tst_files=$(ls $testdir)
> +
> +cleanup()

You use cleanup as a default handler for error handing.
For instance, if a test doesn't have a `# output:` section we silently
quit with exitcode 1. I would like to be informed more about such
errors. Maybe we could just printout $tst if `$! != 0`.

> +{
> +	[ -d "$tmp" ] && rm -rf "$tmp"
> +	exit $1
> +}
> +
> +print_help()
> +{
> +	cat <<EOF
> +
> +????????????????????????????????????????????????????????????????????
> ????????????
> +? This Shell script iterates over test cases for the new Shell
> library and     ?
> +? verifies the
> output.                                                         ?
> +????????????????????????????????????????????????????????????????????
> ????????????
> +
> +	Usage:
> +		$(basename $0) [TEST_FILE_1] [TEST_FILE_2]
> +
> +EOF
> +	exit 0
> +}
> +
> +parse_params()
> +{
> +	[ -n "$1" ] && tst_files=
> +	while [ -n "$1" ]; do
> +		case "$1" in
> +			--help) print_help;;
> +			-h) print_help;;
> +			-*)
> +				printf "Unknown positional parameter
> ${1}.\n"
> +				cleanup 1;;
> +			*) tst_files="$tst_files $1";;
> +		esac
> +		shift
> +	done
> +}
> +
> +verify_output()
> +{
> +	if [ ! -e "${testdir}$tst" ]; then
> +		printf "$tst not found\n"
> +		cleanup 1
> +	fi
> +
> +	${tooldir}lookup_split_cut.py -f ${testdir}$tst -d $tmp \
> +			-s '# output:\n' -c '# {0,1}' || cleanup 1

just an idea, in perl ( I'm not sure if we have perl already as
dependency). My feeling is, that lookup_split_cut.py is to much for
that task.

cat $testdir$tst | perl -e '$o = 0; while (<STDIN>) {print substr($_,
2) if $o;  $o = 1 if /^# output:/; }' > $tmp${tst}expected_output

> +
> +	"${testdir}$tst" > "${tmp}$tst.actual" || cleanup 1

We should keep going on failed test. We need this to test timeout
functionally or error handling...

> +	cmp -s "${tmp}$tst.actual" "${tmp}${tst}_out/out.1" &&
> return 0
> +	return 1
> +}
> +
> +run_tests()
> +{
> +	pass_cnt=0
> +	fail_cnt=0
> +	printf "\n"
> +	for tst in $tst_files; do
> +		if verify_output; then
> +			pass_cnt=$(($pass_cnt + 1))
> +			printf "${color_green}TPASS$reset_attr
> ${tst}\n"
> +		else
> +			fail_cnt=$(($fail_cnt + 1))
> +			printf "${color_red}TFAIL$reset_attr
> ${tst}\n"
> +			printf "${color_blue}Diff:${reset_attr}\n"
> +			diff -u "${tmp}${tst}.actual" \
> +					"${tmp}${tst}_out/out.1"
> +			printf "\n"
> +		fi
> +	done
> +	printf "\nSummary:\n"
> +	printf "${color_red}Failed:$reset_attr $fail_cnt\n"
> +	printf "${color_green}Passed:$reset_attr $pass_cnt\n\n"
> +	return $fail_cnt
> +}
> +
> +parse_params "$@"
> +run_tests
> +cleanup $?
> 
<snip>

Thanks
Clemens

[LTP] [RFC PATCH v2 1/1] Add automated tests for shell lib

[Christian Lanig]

Hi Petr and Clemens,

thank you for the reasonable comments and suggestions.

I just want to tell you that I will not be able to work on this until
next week. I appreciate your patience very much.

Regards,

Christian

-- 
Christian Lanig - SUSE Engineering Apprentice
SUSE Linux GmbH, Maxfeldstra?e 5, D-90409 N?rnberg
Tel: +49-911-74053-0; Fax: +49-911-7417755;  https://www.suse.com/
GF: Felix Imend?rffer, Jane Smithard, Graham Norton, HRB 21284 (AG N?rnberg)