[LTP] [PATCH v5 0/4] Run tests in CI

[LTP] [PATCH v5 0/4] Run tests in CI

[Petr Vorel]

Hi,

changes v4->v5:
* don't exit on test failure, but continue testing (Cyril)
* rename tst_{brk,res}() to runtest_{brk,res}() (Cyril)
* rewritten summary, put everything to the end
* merged 3 already acked commits

$ make test
...

runtest TINFO: === C TEST RESULTS ===
runtest TINFO: 13x TPASS: test05 test07 test09 test12 test15 test18 tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02 tst_fuzzy_sync03
runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 0x TCONF:

...

runtest TINFO: === SHELL TEST RESULTS ===
runtest TINFO: 2x TPASS: shell/tst_check_driver.sh shell/net/tst_ipaddr_un.sh
runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 1x TCONF: shell/net/tst_rhost_run.sh

runtest TINFO: === FINAL TEST RESULTS ===
runtest TPASS: All C tests TCONF/TPASS
runtest TPASS: All shell tests TCONF/TPASS

Petr Vorel (4):
  lib: Add script for running tests
  make: Add make test{, -c, -shell} targets
  build.sh: Add support for make test
  CI: Run also make test

 .github/workflows/ci.yml    |   5 ++
 Makefile                    |  23 +++++
 build.sh                    |  22 ++++-
 lib/newlib_tests/runtest.sh | 175 ++++++++++++++++++++++++++++++++++++
 4 files changed, 224 insertions(+), 1 deletion(-)
 create mode 100755 lib/newlib_tests/runtest.sh

-- 
2.32.0

[LTP] [PATCH v5 1/4] lib: Add script for running tests

[Petr Vorel]

For now run only tests which TPASS or TCONF.

tst_fuzzy_sync01: sporadically fails, thus disabled:

../../include/tst_fuzzy_sync.h:685: TINFO: Exceeded execution loops, requesting exit
tst_fuzzy_sync01.c:227: TFAIL: acs:3  act:1  art:1  | =:23   -:46   +:2999931
...
Summary:
passed   21
failed   3

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 lib/newlib_tests/runtest.sh | 175 ++++++++++++++++++++++++++++++++++++
 1 file changed, 175 insertions(+)
 create mode 100755 lib/newlib_tests/runtest.sh

diff --git a/lib/newlib_tests/runtest.sh b/lib/newlib_tests/runtest.sh
new file mode 100755
index 000000000..207553aaa
--- /dev/null
+++ b/lib/newlib_tests/runtest.sh
@@ -0,0 +1,175 @@
+#!/bin/sh
+# Copyright (c) 2021 Petr Vorel <pvorel@suse.cz>
+
+LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18
+tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02
+tst_fuzzy_sync03}"
+
+LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/net/*.sh}"
+
+cd $(dirname $0)
+PATH="$PWD/../../testcases/lib/:$PATH"
+
+. tst_ansi_color.sh
+
+usage()
+{
+	cat << EOF
+Usage: $0 [-b DIR ] [-c|-s]
+-b DIR  build directory (required for out-of-tree build)
+-c      run C API tests only
+-s      run shell API tests only
+-h      print this help
+EOF
+}
+
+tst_flag2mask()
+{
+	case "$1" in
+	TPASS) return 0;;
+	TFAIL) return 1;;
+	TBROK) return 2;;
+	TWARN) return 4;;
+	TINFO) return 16;;
+	TCONF) return 32;;
+	esac
+}
+
+runtest_res()
+{
+	if [ $# -eq 0 ]; then
+		echo >&2
+		return
+	fi
+
+	local res="$1"
+	shift
+
+	tst_color_enabled
+	local color=$?
+
+	printf "runtest " >&2
+	tst_print_colored $res "$res: " >&2
+	echo "$@" >&2
+
+}
+
+runtest_brk()
+{
+	local res="$1"
+	shift
+
+	tst_flag2mask "$res"
+	local mask=$?
+
+	runtest_res
+	runtest_res $res $@
+
+	exit $mask
+}
+
+run_tests()
+{
+	local target="$1"
+	local i res ret=0 tbrok tconf tfail tpass twarn vars
+
+	eval vars="\$LTP_${target}_API_TESTS"
+
+	runtest_res TINFO "=== Run $target tests ==="
+
+	for i in $vars; do
+		runtest_res TINFO "* $i"
+		./$i
+		res=$?
+
+		[ $res -ne 0 -a $res -ne 32 ] && ret=1
+
+		case $res in
+			0) tpass="$tpass $i";;
+			1) tfail="$tfail $i";;
+			2) tbrok="$tbrok $i";;
+			4) twarn="$twarn $i";;
+			32) tconf="$tconf $i";;
+			127) runtest_brk TBROK "Error: file not found (wrong PATH? out-of-tree build without -b?), exit code: $res";;
+			*) runtest_brk TBROK "Error: unknown failure, exit code: $res";;
+		esac
+		runtest_res
+	done
+
+	runtest_res TINFO "=== $target TEST RESULTS ==="
+	runtest_res TINFO "$(echo $tpass | wc -w)x TPASS:$tpass"
+	runtest_res TINFO "$(echo $tfail | wc -w)x TFAIL:$tfail"
+	runtest_res TINFO "$(echo $tbrok | wc -w)x TBROK:$tbrok"
+	runtest_res TINFO "$(echo $twarn | wc -w)x TWARN:$twarn"
+	runtest_res TINFO "$(echo $tconf | wc -w)x TCONF:$tconf"
+	runtest_res
+
+	return $ret
+}
+
+run_c_tests()
+{
+	if [ "$builddir" ]; then
+		cd $builddir/lib/newlib_tests
+	fi
+
+	run_tests "C"
+
+	if [ "$builddir" ]; then
+		cd -
+	fi
+}
+
+run_shell_tests()
+{
+	run_tests "SHELL"
+}
+
+
+print_result()
+{
+	local res="$1"
+	local target="$2"
+
+	if [ -z "$res" ]; then
+		runtest_res TCONF "$target tests skipped"
+	elif [ $res -eq 0 ]; then
+		runtest_res TPASS "All $target tests TCONF/TPASS"
+	else
+		runtest_res TFAIL "Some $target test(s) TBROK/TFAIL/TWARN"
+	fi
+}
+
+builddir=
+c_fail=
+run=
+shell_fail=
+
+while getopts b:chs opt; do
+	case $opt in
+		'h') usage; exit 0;;
+		'b') builddir=$OPTARG; PATH="$builddir/testcases/lib:$PATH";;
+		'c') run="c";;
+		's') run="s";;
+		*) usage; runtest_brk TBROK "Error: invalid option";;
+	esac
+done
+
+runtest_res TINFO "PATH='$PATH'"
+
+if [ -z "$run" -o "$run" = "c" ]; then
+	run_c_tests
+	c_fail=$?
+fi
+
+if [ -z "$run" -o "$run" = "s" ]; then
+	run_shell_tests
+	shell_fail=$?
+fi
+
+runtest_res TINFO "=== FINAL TEST RESULTS ==="
+
+print_result "$c_fail" "C"
+print_result "$shell_fail" "shell"
+
+exit $((c_fail|shell_fail))
-- 
2.32.0

[LTP] [PATCH v5 2/4] make: Add make test{, -c, -shell} targets

[Petr Vorel]

For testing C and shell API.

Reviewed-by: Cyril Hrubis <chrubis@suse.cz>
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 Makefile | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/Makefile b/Makefile
index 56812d77b..ff4a70eec 100644
--- a/Makefile
+++ b/Makefile
@@ -192,6 +192,29 @@ $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir)
 ## Install
 install: $(INSTALL_TARGETS)
 
+## Test
+define _test
+	@set -e; $(top_srcdir)/lib/newlib_tests/runtest.sh -b $(abs_builddir) $(1)
+endef
+
+test: lib-all
+ifneq ($(build),$(host))
+	$(error running tests on cross-compile build not supported)
+endif
+	$(call _test)
+
+test-c: lib-all
+ifneq ($(build),$(host))
+	$(error running tests on cross-compile build not supported)
+endif
+	$(call _test,-c)
+
+test-shell: lib-all
+ifneq ($(build),$(host))
+	$(error running tests on cross-compile build not supported)
+endif
+	$(call _test,-s)
+
 ## Help
 .PHONY: help
 help:
-- 
2.32.0

[LTP] [PATCH v5 3/4] build.sh: Add support for make test

[Petr Vorel]

Reviewed-by: Cyril Hrubis <chrubis@suse.cz>
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 build.sh | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/build.sh b/build.sh
index 240ce8e68..9cb26e899 100755
--- a/build.sh
+++ b/build.sh
@@ -119,6 +119,17 @@ build_out_tree()
 	make $MAKE_OPTS_OUT_TREE
 }
 
+test_in_tree()
+{
+	make test
+}
+
+test_out_tree()
+{
+	cd $BUILD_DIR
+	make $MAKE_OPTS_OUT_TREE test
+}
+
 install_in_tree()
 {
 	make $MAKE_OPTS install
@@ -165,6 +176,7 @@ RUN:
 autotools   run only 'make autotools'
 configure   run only 'configure'
 build       run only 'make'
+test        run only 'make test' (not supported for cross-compile build)
 install     run only 'make install'
 
 Default configure options:
@@ -192,7 +204,7 @@ while getopts "c:hio:p:r:t:" opt; do
 		esac;;
 	p) prefix="$OPTARG";;
 	r) case "$OPTARG" in
-		autotools|configure|build|install) run="$OPTARG";;
+		autotools|configure|build|test|install) run="$OPTARG";;
 		*) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;;
 		esac;;
 	t) case "$OPTARG" in
@@ -218,6 +230,14 @@ if [ -z "$run" -o "$run" = "build" ]; then
 	eval build_${tree}_tree
 fi
 
+if [ -z "$run" -o "$run" = "test" ]; then
+	if [ "$build" = "cross" ]; then
+		echo "cross-compile build, skipping running tests" >&2
+	else
+		eval test_${tree}_tree
+	fi
+fi
+
 if [ -z "$run" -o "$run" = "install" ]; then
 	if [ "$install" = 1 ]; then
 		eval install_${tree}_tree
-- 
2.32.0

[LTP] [PATCH v5 4/4] CI: Run also make test

[Petr Vorel]

on all targets

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 .github/workflows/ci.yml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f67f14927..776dbf646 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -151,6 +151,11 @@ jobs:
     - name: Compile
       run: ./build.sh -r build -o ${TREE:-in}
 
+    - name: Test
+      run: |
+        case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac
+        ./build.sh -r test -o ${TREE:-in} -t $BUILD
+
     - name: Install
       run: |
         if [ "$MAKE_INSTALL" = 1 ]; then INSTALL_OPT="-i"; fi
-- 
2.32.0

[LTP] [PATCH v5 0/4] Run tests in CI

[Petr Vorel]

Hi all,

tested:
https://github.com/pevik/ltp/actions/runs/1030551185

One thing which bothers me is mangled output of several tests together:
https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554

runtest TINFO: * tst_res_hexd
/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
/__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1
/__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72
/__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00

runtest TINFO: * tst_strstatus
/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s

Summary:
passed   35
failed   0
broken   0
skipped  0
warnings 0

Summary:
passed   1
failed   0
broken   0
skipped  0
warnings 0

Kind regards,
Petr

[LTP] [PATCH v5 0/4] Run tests in CI

[Cyril Hrubis]

Hi!
> tested:
> https://github.com/pevik/ltp/actions/runs/1030551185
> 
> One thing which bothers me is mangled output of several tests together:
> https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554
> 
> runtest TINFO: * tst_res_hexd
> /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
> /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1
> /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72
> /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00
> 
> runtest TINFO: * tst_strstatus
> /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
> 
> Summary:
> passed   35
> failed   0
> broken   0
> skipped  0
> warnings 0
> 
> Summary:
> passed   1
> failed   0
> broken   0
> skipped  0
> warnings 0

That is indeed strange, you are running these in a loop, the buffers
should have been flushed once controll returns back to the shell from
the test.

I guess that this is a combination of:

* The stdout and stderr are block buffered by the gitlab CI and flushed
  at different times

* The summary is written into stdout in contrast with the message
  so we end up with summary printed later than the rest that goes into
  stderr

I guess this should fix it:

diff --git a/lib/tst_test.c b/lib/tst_test.c
index f4d9f8e3b..084a83c9e 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -734,12 +734,12 @@ static void do_exit(int ret)
                if (results->broken)
                        ret |= TBROK;

-               printf("\nSummary:\n");
-               printf("passed   %d\n", results->passed);
-               printf("failed   %d\n", results->failed);
-               printf("broken   %d\n", results->broken);
-               printf("skipped  %d\n", results->skipped);
-               printf("warnings %d\n", results->warnings);
+               fprintf(stderr, "\nSummary:\n");
+               fprintf(stderr, "passed   %d\n", results->passed);
+               fprintf(stderr, "failed   %d\n", results->failed);
+               fprintf(stderr, "broken   %d\n", results->broken);
+               fprintf(stderr, "skipped  %d\n", results->skipped);
+               fprintf(stderr, "warnings %d\n", results->warnings);
        }

        do_cleanup();

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [PATCH v5 0/4] Run tests in CI

[Petr Vorel]

Hi Cyril,

> Hi!
> > tested:
> > https://github.com/pevik/ltp/actions/runs/1030551185

> > One thing which bothers me is mangled output of several tests together:
> > https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554

> > runtest TINFO: * tst_res_hexd
> > /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
> > /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1
> > /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72
> > /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00

> > runtest TINFO: * tst_strstatus
> > /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s

> > Summary:
> > passed   35
> > failed   0
> > broken   0
> > skipped  0
> > warnings 0

> > Summary:
> > passed   1
> > failed   0
> > broken   0
> > skipped  0
> > warnings 0

> That is indeed strange, you are running these in a loop, the buffers
> should have been flushed once controll returns back to the shell from
> the test.

> I guess that this is a combination of:

> * The stdout and stderr are block buffered by the gitlab CI and flushed
>   at different times

> * The summary is written into stdout in contrast with the message
>   so we end up with summary printed later than the rest that goes into
>   stderr

Yep, it could be.

> I guess this should fix it:

> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index f4d9f8e3b..084a83c9e 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -734,12 +734,12 @@ static void do_exit(int ret)
>                 if (results->broken)
>                         ret |= TBROK;

> -               printf("\nSummary:\n");
> -               printf("passed   %d\n", results->passed);
> -               printf("failed   %d\n", results->failed);
> -               printf("broken   %d\n", results->broken);
> -               printf("skipped  %d\n", results->skipped);
> -               printf("warnings %d\n", results->warnings);
> +               fprintf(stderr, "\nSummary:\n");
> +               fprintf(stderr, "passed   %d\n", results->passed);
> +               fprintf(stderr, "failed   %d\n", results->failed);
> +               fprintf(stderr, "broken   %d\n", results->broken);
> +               fprintf(stderr, "skipped  %d\n", results->skipped);
> +               fprintf(stderr, "warnings %d\n", results->warnings);
>         }

Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963

Kind regards,
Petr

[LTP] [PATCH v5 0/4] Run tests in CI

[Petr Vorel]

Hi Cyril,

...
> > I guess this should fix it:

> > diff --git a/lib/tst_test.c b/lib/tst_test.c
> > index f4d9f8e3b..084a83c9e 100644
> > --- a/lib/tst_test.c
> > +++ b/lib/tst_test.c
> > @@ -734,12 +734,12 @@ static void do_exit(int ret)
> >                 if (results->broken)
> >                         ret |= TBROK;

> > -               printf("\nSummary:\n");
> > -               printf("passed   %d\n", results->passed);
> > -               printf("failed   %d\n", results->failed);
> > -               printf("broken   %d\n", results->broken);
> > -               printf("skipped  %d\n", results->skipped);
> > -               printf("warnings %d\n", results->warnings);
> > +               fprintf(stderr, "\nSummary:\n");
> > +               fprintf(stderr, "passed   %d\n", results->passed);
> > +               fprintf(stderr, "failed   %d\n", results->failed);
> > +               fprintf(stderr, "broken   %d\n", results->broken);
> > +               fprintf(stderr, "skipped  %d\n", results->skipped);
> > +               fprintf(stderr, "warnings %d\n", results->warnings);
> >         }

> Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963

The result is much better but still not correct:
https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707

runtest TINFO: * shell/net/tst_ipaddr_un.sh

Summary:
passed   0
failed   0
broken   0
skipped  1
warnings 0
tst_ipaddr_un 1 TINFO: timeout per run is 0h 5m 0s

Could you please merge your fix? And I'll try sync after each test.
https://github.com/pevik/ltp/actions/runs/1033070696

What is worse is that tests are green, but I see TBROK and TWARN which are not
reflected by results:

tst_check_driver 1 TBROK: Test still running, sending SIGKILL
...
tst_rhost_run 1 TCONF: (null)      0  TWARN  :  tst_kernel.c:105: expected file /lib/modules/5.8.0-1036-azure/modules.dep does not exist or not a file

veth(null)      0  TWARN  :  tst_kernel.c:105: expected file /lib/modules/5.8.0-1036-azure/modules.builtin does not exist or not a file driver not available
Summary:
passed   0
failed   0
broken   0
skipped  1
warnings 0

runtest TINFO: === SHELL TEST RESULTS ===
runtest TINFO: 1x TPASS: shell/net/tst_ipaddr_un.sh
runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 2x TCONF: shell/tst_check_driver.sh shell/net/tst_rhost_run.sh

Kind regards,
Petr

> Kind regards,
> Petr

[LTP] [PATCH v5 0/4] Run tests in CI

[Petr Vorel]

Hi Cyril,

...
> > Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963

> The result is much better but still not correct:
> https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707

> runtest TINFO: * shell/net/tst_ipaddr_un.sh

> Summary:
> passed   0
> failed   0
> broken   0
> skipped  1
> warnings 0
> tst_ipaddr_un 1 TINFO: timeout per run is 0h 5m 0s

> Could you please merge your fix? And I'll try sync after each test.
> https://github.com/pevik/ltp/actions/runs/1033070696

There is still some output printed later, at least
In RPN: A B | C ! ! &
In RPN: A B | C ! ! &
which is from tst_bool_expr.c, but printed after tst_strstatus
https://github.com/pevik/ltp/runs/3074335940?check_suite_focus=true#step:11:587
But it's much better.

I also found a bug in final evaluation on one of my VM:

tst_test.c:1313: TBROK: Test killed by SIGSEGV!
...
runtest TINFO: 1x TBROK: tst_bool_expr
...
runtest TINFO: === FINAL TEST RESULTS ===
runtest TPASS: All C tests TCONF/TPASS <- THIS IS WRONG
runtest TPASS: All shell tests TCONF/TPASS

Kind regards,
Petr

[LTP] [PATCH v5 0/4] Run tests in CI

[Cyril Hrubis]

Hi!
> > Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
> 
> The result is much better but still not correct:
> https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707

If tests are printing into stdout as well it will be easier to redirect
the test output in the script so that the CI gets it all into a single
file descriptor.

What about doing ./$i 1>&2 in the loop that runs the tests instead?

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [PATCH v5 0/4] Run tests in CI

[Petr Vorel]

> Hi!
> > > Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963

> > The result is much better but still not correct:
> > https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707

> If tests are printing into stdout as well it will be easier to redirect
> the test output in the script so that the CI gets it all into a single
> file descriptor.

> What about doing ./$i 1>&2 in the loop that runs the tests instead?
How simple, that should work, I'll test that.

FYI I also need to install iproute (runtime shell API test dependency).
Simple enough to just merge it after this gets merged.