* [LTP] [PATCH v5 0/4] Run tests in CI
@ 2021-07-14 14:19 Petr Vorel
2021-07-14 14:19 ` [LTP] [PATCH v5 1/4] lib: Add script for running tests Petr Vorel
` (4 more replies)
0 siblings, 5 replies; 12+ messages in thread
From: Petr Vorel @ 2021-07-14 14:19 UTC (permalink / raw)
To: ltp
Hi,
changes v4->v5:
* don't exit on test failure, but continue testing (Cyril)
* rename tst_{brk,res}() to runtest_{brk,res}() (Cyril)
* rewritten summary, put everything to the end
* merged 3 already acked commits
$ make test
...
runtest TINFO: === C TEST RESULTS ===
runtest TINFO: 13x TPASS: test05 test07 test09 test12 test15 test18 tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02 tst_fuzzy_sync03
runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 0x TCONF:
...
runtest TINFO: === SHELL TEST RESULTS ===
runtest TINFO: 2x TPASS: shell/tst_check_driver.sh shell/net/tst_ipaddr_un.sh
runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 1x TCONF: shell/net/tst_rhost_run.sh
runtest TINFO: === FINAL TEST RESULTS ===
runtest TPASS: All C tests TCONF/TPASS
runtest TPASS: All shell tests TCONF/TPASS
Petr Vorel (4):
lib: Add script for running tests
make: Add make test{, -c, -shell} targets
build.sh: Add support for make test
CI: Run also make test
.github/workflows/ci.yml | 5 ++
Makefile | 23 +++++
build.sh | 22 ++++-
lib/newlib_tests/runtest.sh | 175 ++++++++++++++++++++++++++++++++++++
4 files changed, 224 insertions(+), 1 deletion(-)
create mode 100755 lib/newlib_tests/runtest.sh
--
2.32.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 1/4] lib: Add script for running tests
2021-07-14 14:19 [LTP] [PATCH v5 0/4] Run tests in CI Petr Vorel
@ 2021-07-14 14:19 ` Petr Vorel
2021-07-14 14:19 ` [LTP] [PATCH v5 2/4] make: Add make test{, -c, -shell} targets Petr Vorel
` (3 subsequent siblings)
4 siblings, 0 replies; 12+ messages in thread
From: Petr Vorel @ 2021-07-14 14:19 UTC (permalink / raw)
To: ltp
For now run only tests which TPASS or TCONF.
tst_fuzzy_sync01: sporadically fails, thus disabled:
../../include/tst_fuzzy_sync.h:685: TINFO: Exceeded execution loops, requesting exit
tst_fuzzy_sync01.c:227: TFAIL: acs:3 act:1 art:1 | =:23 -:46 +:2999931
...
Summary:
passed 21
failed 3
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
lib/newlib_tests/runtest.sh | 175 ++++++++++++++++++++++++++++++++++++
1 file changed, 175 insertions(+)
create mode 100755 lib/newlib_tests/runtest.sh
diff --git a/lib/newlib_tests/runtest.sh b/lib/newlib_tests/runtest.sh
new file mode 100755
index 000000000..207553aaa
--- /dev/null
+++ b/lib/newlib_tests/runtest.sh
@@ -0,0 +1,175 @@
+#!/bin/sh
+# Copyright (c) 2021 Petr Vorel <pvorel@suse.cz>
+
+LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18
+tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02
+tst_fuzzy_sync03}"
+
+LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/net/*.sh}"
+
+cd $(dirname $0)
+PATH="$PWD/../../testcases/lib/:$PATH"
+
+. tst_ansi_color.sh
+
+usage()
+{
+ cat << EOF
+Usage: $0 [-b DIR ] [-c|-s]
+-b DIR build directory (required for out-of-tree build)
+-c run C API tests only
+-s run shell API tests only
+-h print this help
+EOF
+}
+
+tst_flag2mask()
+{
+ case "$1" in
+ TPASS) return 0;;
+ TFAIL) return 1;;
+ TBROK) return 2;;
+ TWARN) return 4;;
+ TINFO) return 16;;
+ TCONF) return 32;;
+ esac
+}
+
+runtest_res()
+{
+ if [ $# -eq 0 ]; then
+ echo >&2
+ return
+ fi
+
+ local res="$1"
+ shift
+
+ tst_color_enabled
+ local color=$?
+
+ printf "runtest " >&2
+ tst_print_colored $res "$res: " >&2
+ echo "$@" >&2
+
+}
+
+runtest_brk()
+{
+ local res="$1"
+ shift
+
+ tst_flag2mask "$res"
+ local mask=$?
+
+ runtest_res
+ runtest_res $res $@
+
+ exit $mask
+}
+
+run_tests()
+{
+ local target="$1"
+ local i res ret=0 tbrok tconf tfail tpass twarn vars
+
+ eval vars="\$LTP_${target}_API_TESTS"
+
+ runtest_res TINFO "=== Run $target tests ==="
+
+ for i in $vars; do
+ runtest_res TINFO "* $i"
+ ./$i
+ res=$?
+
+ [ $res -ne 0 -a $res -ne 32 ] && ret=1
+
+ case $res in
+ 0) tpass="$tpass $i";;
+ 1) tfail="$tfail $i";;
+ 2) tbrok="$tbrok $i";;
+ 4) twarn="$twarn $i";;
+ 32) tconf="$tconf $i";;
+ 127) runtest_brk TBROK "Error: file not found (wrong PATH? out-of-tree build without -b?), exit code: $res";;
+ *) runtest_brk TBROK "Error: unknown failure, exit code: $res";;
+ esac
+ runtest_res
+ done
+
+ runtest_res TINFO "=== $target TEST RESULTS ==="
+ runtest_res TINFO "$(echo $tpass | wc -w)x TPASS:$tpass"
+ runtest_res TINFO "$(echo $tfail | wc -w)x TFAIL:$tfail"
+ runtest_res TINFO "$(echo $tbrok | wc -w)x TBROK:$tbrok"
+ runtest_res TINFO "$(echo $twarn | wc -w)x TWARN:$twarn"
+ runtest_res TINFO "$(echo $tconf | wc -w)x TCONF:$tconf"
+ runtest_res
+
+ return $ret
+}
+
+run_c_tests()
+{
+ if [ "$builddir" ]; then
+ cd $builddir/lib/newlib_tests
+ fi
+
+ run_tests "C"
+
+ if [ "$builddir" ]; then
+ cd -
+ fi
+}
+
+run_shell_tests()
+{
+ run_tests "SHELL"
+}
+
+
+print_result()
+{
+ local res="$1"
+ local target="$2"
+
+ if [ -z "$res" ]; then
+ runtest_res TCONF "$target tests skipped"
+ elif [ $res -eq 0 ]; then
+ runtest_res TPASS "All $target tests TCONF/TPASS"
+ else
+ runtest_res TFAIL "Some $target test(s) TBROK/TFAIL/TWARN"
+ fi
+}
+
+builddir=
+c_fail=
+run=
+shell_fail=
+
+while getopts b:chs opt; do
+ case $opt in
+ 'h') usage; exit 0;;
+ 'b') builddir=$OPTARG; PATH="$builddir/testcases/lib:$PATH";;
+ 'c') run="c";;
+ 's') run="s";;
+ *) usage; runtest_brk TBROK "Error: invalid option";;
+ esac
+done
+
+runtest_res TINFO "PATH='$PATH'"
+
+if [ -z "$run" -o "$run" = "c" ]; then
+ run_c_tests
+ c_fail=$?
+fi
+
+if [ -z "$run" -o "$run" = "s" ]; then
+ run_shell_tests
+ shell_fail=$?
+fi
+
+runtest_res TINFO "=== FINAL TEST RESULTS ==="
+
+print_result "$c_fail" "C"
+print_result "$shell_fail" "shell"
+
+exit $((c_fail|shell_fail))
--
2.32.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 2/4] make: Add make test{, -c, -shell} targets
2021-07-14 14:19 [LTP] [PATCH v5 0/4] Run tests in CI Petr Vorel
2021-07-14 14:19 ` [LTP] [PATCH v5 1/4] lib: Add script for running tests Petr Vorel
@ 2021-07-14 14:19 ` Petr Vorel
2021-07-14 14:20 ` [LTP] [PATCH v5 3/4] build.sh: Add support for make test Petr Vorel
` (2 subsequent siblings)
4 siblings, 0 replies; 12+ messages in thread
From: Petr Vorel @ 2021-07-14 14:19 UTC (permalink / raw)
To: ltp
For testing C and shell API.
Reviewed-by: Cyril Hrubis <chrubis@suse.cz>
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
Makefile | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/Makefile b/Makefile
index 56812d77b..ff4a70eec 100644
--- a/Makefile
+++ b/Makefile
@@ -192,6 +192,29 @@ $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir)
## Install
install: $(INSTALL_TARGETS)
+## Test
+define _test
+ @set -e; $(top_srcdir)/lib/newlib_tests/runtest.sh -b $(abs_builddir) $(1)
+endef
+
+test: lib-all
+ifneq ($(build),$(host))
+ $(error running tests on cross-compile build not supported)
+endif
+ $(call _test)
+
+test-c: lib-all
+ifneq ($(build),$(host))
+ $(error running tests on cross-compile build not supported)
+endif
+ $(call _test,-c)
+
+test-shell: lib-all
+ifneq ($(build),$(host))
+ $(error running tests on cross-compile build not supported)
+endif
+ $(call _test,-s)
+
## Help
.PHONY: help
help:
--
2.32.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 3/4] build.sh: Add support for make test
2021-07-14 14:19 [LTP] [PATCH v5 0/4] Run tests in CI Petr Vorel
2021-07-14 14:19 ` [LTP] [PATCH v5 1/4] lib: Add script for running tests Petr Vorel
2021-07-14 14:19 ` [LTP] [PATCH v5 2/4] make: Add make test{, -c, -shell} targets Petr Vorel
@ 2021-07-14 14:20 ` Petr Vorel
2021-07-14 14:20 ` [LTP] [PATCH v5 4/4] CI: Run also " Petr Vorel
2021-07-14 14:27 ` [LTP] [PATCH v5 0/4] Run tests in CI Petr Vorel
4 siblings, 0 replies; 12+ messages in thread
From: Petr Vorel @ 2021-07-14 14:20 UTC (permalink / raw)
To: ltp
Reviewed-by: Cyril Hrubis <chrubis@suse.cz>
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
build.sh | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/build.sh b/build.sh
index 240ce8e68..9cb26e899 100755
--- a/build.sh
+++ b/build.sh
@@ -119,6 +119,17 @@ build_out_tree()
make $MAKE_OPTS_OUT_TREE
}
+test_in_tree()
+{
+ make test
+}
+
+test_out_tree()
+{
+ cd $BUILD_DIR
+ make $MAKE_OPTS_OUT_TREE test
+}
+
install_in_tree()
{
make $MAKE_OPTS install
@@ -165,6 +176,7 @@ RUN:
autotools run only 'make autotools'
configure run only 'configure'
build run only 'make'
+test run only 'make test' (not supported for cross-compile build)
install run only 'make install'
Default configure options:
@@ -192,7 +204,7 @@ while getopts "c:hio:p:r:t:" opt; do
esac;;
p) prefix="$OPTARG";;
r) case "$OPTARG" in
- autotools|configure|build|install) run="$OPTARG";;
+ autotools|configure|build|test|install) run="$OPTARG";;
*) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;;
esac;;
t) case "$OPTARG" in
@@ -218,6 +230,14 @@ if [ -z "$run" -o "$run" = "build" ]; then
eval build_${tree}_tree
fi
+if [ -z "$run" -o "$run" = "test" ]; then
+ if [ "$build" = "cross" ]; then
+ echo "cross-compile build, skipping running tests" >&2
+ else
+ eval test_${tree}_tree
+ fi
+fi
+
if [ -z "$run" -o "$run" = "install" ]; then
if [ "$install" = 1 ]; then
eval install_${tree}_tree
--
2.32.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 4/4] CI: Run also make test
2021-07-14 14:19 [LTP] [PATCH v5 0/4] Run tests in CI Petr Vorel
` (2 preceding siblings ...)
2021-07-14 14:20 ` [LTP] [PATCH v5 3/4] build.sh: Add support for make test Petr Vorel
@ 2021-07-14 14:20 ` Petr Vorel
2021-07-14 14:27 ` [LTP] [PATCH v5 0/4] Run tests in CI Petr Vorel
4 siblings, 0 replies; 12+ messages in thread
From: Petr Vorel @ 2021-07-14 14:20 UTC (permalink / raw)
To: ltp
on all targets
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
.github/workflows/ci.yml | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f67f14927..776dbf646 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -151,6 +151,11 @@ jobs:
- name: Compile
run: ./build.sh -r build -o ${TREE:-in}
+ - name: Test
+ run: |
+ case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac
+ ./build.sh -r test -o ${TREE:-in} -t $BUILD
+
- name: Install
run: |
if [ "$MAKE_INSTALL" = 1 ]; then INSTALL_OPT="-i"; fi
--
2.32.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 0/4] Run tests in CI
2021-07-14 14:19 [LTP] [PATCH v5 0/4] Run tests in CI Petr Vorel
` (3 preceding siblings ...)
2021-07-14 14:20 ` [LTP] [PATCH v5 4/4] CI: Run also " Petr Vorel
@ 2021-07-14 14:27 ` Petr Vorel
2021-07-14 15:00 ` Cyril Hrubis
4 siblings, 1 reply; 12+ messages in thread
From: Petr Vorel @ 2021-07-14 14:27 UTC (permalink / raw)
To: ltp
Hi all,
tested:
https://github.com/pevik/ltp/actions/runs/1030551185
One thing which bothers me is mangled output of several tests together:
https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554
runtest TINFO: * tst_res_hexd
/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
/__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1
/__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72
/__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00
runtest TINFO: * tst_strstatus
/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
Summary:
passed 35
failed 0
broken 0
skipped 0
warnings 0
Summary:
passed 1
failed 0
broken 0
skipped 0
warnings 0
Kind regards,
Petr
^ permalink raw reply [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 0/4] Run tests in CI
2021-07-14 14:27 ` [LTP] [PATCH v5 0/4] Run tests in CI Petr Vorel
@ 2021-07-14 15:00 ` Cyril Hrubis
2021-07-14 16:43 ` Petr Vorel
0 siblings, 1 reply; 12+ messages in thread
From: Cyril Hrubis @ 2021-07-14 15:00 UTC (permalink / raw)
To: ltp
Hi!
> tested:
> https://github.com/pevik/ltp/actions/runs/1030551185
>
> One thing which bothers me is mangled output of several tests together:
> https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554
>
> runtest TINFO: * tst_res_hexd
> /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
> /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1
> /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72
> /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00
>
> runtest TINFO: * tst_strstatus
> /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
>
> Summary:
> passed 35
> failed 0
> broken 0
> skipped 0
> warnings 0
>
> Summary:
> passed 1
> failed 0
> broken 0
> skipped 0
> warnings 0
That is indeed strange, you are running these in a loop, the buffers
should have been flushed once controll returns back to the shell from
the test.
I guess that this is a combination of:
* The stdout and stderr are block buffered by the gitlab CI and flushed
at different times
* The summary is written into stdout in contrast with the message
so we end up with summary printed later than the rest that goes into
stderr
I guess this should fix it:
diff --git a/lib/tst_test.c b/lib/tst_test.c
index f4d9f8e3b..084a83c9e 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -734,12 +734,12 @@ static void do_exit(int ret)
if (results->broken)
ret |= TBROK;
- printf("\nSummary:\n");
- printf("passed %d\n", results->passed);
- printf("failed %d\n", results->failed);
- printf("broken %d\n", results->broken);
- printf("skipped %d\n", results->skipped);
- printf("warnings %d\n", results->warnings);
+ fprintf(stderr, "\nSummary:\n");
+ fprintf(stderr, "passed %d\n", results->passed);
+ fprintf(stderr, "failed %d\n", results->failed);
+ fprintf(stderr, "broken %d\n", results->broken);
+ fprintf(stderr, "skipped %d\n", results->skipped);
+ fprintf(stderr, "warnings %d\n", results->warnings);
}
do_cleanup();
--
Cyril Hrubis
chrubis@suse.cz
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 0/4] Run tests in CI
2021-07-14 15:00 ` Cyril Hrubis
@ 2021-07-14 16:43 ` Petr Vorel
2021-07-15 7:38 ` Petr Vorel
0 siblings, 1 reply; 12+ messages in thread
From: Petr Vorel @ 2021-07-14 16:43 UTC (permalink / raw)
To: ltp
Hi Cyril,
> Hi!
> > tested:
> > https://github.com/pevik/ltp/actions/runs/1030551185
> > One thing which bothers me is mangled output of several tests together:
> > https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554
> > runtest TINFO: * tst_res_hexd
> > /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
> > /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1
> > /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72
> > /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00
> > runtest TINFO: * tst_strstatus
> > /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
> > Summary:
> > passed 35
> > failed 0
> > broken 0
> > skipped 0
> > warnings 0
> > Summary:
> > passed 1
> > failed 0
> > broken 0
> > skipped 0
> > warnings 0
> That is indeed strange, you are running these in a loop, the buffers
> should have been flushed once controll returns back to the shell from
> the test.
> I guess that this is a combination of:
> * The stdout and stderr are block buffered by the gitlab CI and flushed
> at different times
> * The summary is written into stdout in contrast with the message
> so we end up with summary printed later than the rest that goes into
> stderr
Yep, it could be.
> I guess this should fix it:
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index f4d9f8e3b..084a83c9e 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -734,12 +734,12 @@ static void do_exit(int ret)
> if (results->broken)
> ret |= TBROK;
> - printf("\nSummary:\n");
> - printf("passed %d\n", results->passed);
> - printf("failed %d\n", results->failed);
> - printf("broken %d\n", results->broken);
> - printf("skipped %d\n", results->skipped);
> - printf("warnings %d\n", results->warnings);
> + fprintf(stderr, "\nSummary:\n");
> + fprintf(stderr, "passed %d\n", results->passed);
> + fprintf(stderr, "failed %d\n", results->failed);
> + fprintf(stderr, "broken %d\n", results->broken);
> + fprintf(stderr, "skipped %d\n", results->skipped);
> + fprintf(stderr, "warnings %d\n", results->warnings);
> }
Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
Kind regards,
Petr
^ permalink raw reply [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 0/4] Run tests in CI
2021-07-14 16:43 ` Petr Vorel
@ 2021-07-15 7:38 ` Petr Vorel
2021-07-15 7:48 ` Petr Vorel
2021-07-15 8:51 ` Cyril Hrubis
0 siblings, 2 replies; 12+ messages in thread
From: Petr Vorel @ 2021-07-15 7:38 UTC (permalink / raw)
To: ltp
Hi Cyril,
...
> > I guess this should fix it:
> > diff --git a/lib/tst_test.c b/lib/tst_test.c
> > index f4d9f8e3b..084a83c9e 100644
> > --- a/lib/tst_test.c
> > +++ b/lib/tst_test.c
> > @@ -734,12 +734,12 @@ static void do_exit(int ret)
> > if (results->broken)
> > ret |= TBROK;
> > - printf("\nSummary:\n");
> > - printf("passed %d\n", results->passed);
> > - printf("failed %d\n", results->failed);
> > - printf("broken %d\n", results->broken);
> > - printf("skipped %d\n", results->skipped);
> > - printf("warnings %d\n", results->warnings);
> > + fprintf(stderr, "\nSummary:\n");
> > + fprintf(stderr, "passed %d\n", results->passed);
> > + fprintf(stderr, "failed %d\n", results->failed);
> > + fprintf(stderr, "broken %d\n", results->broken);
> > + fprintf(stderr, "skipped %d\n", results->skipped);
> > + fprintf(stderr, "warnings %d\n", results->warnings);
> > }
> Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
The result is much better but still not correct:
https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707
runtest TINFO: * shell/net/tst_ipaddr_un.sh
Summary:
passed 0
failed 0
broken 0
skipped 1
warnings 0
tst_ipaddr_un 1 TINFO: timeout per run is 0h 5m 0s
Could you please merge your fix? And I'll try sync after each test.
https://github.com/pevik/ltp/actions/runs/1033070696
What is worse is that tests are green, but I see TBROK and TWARN which are not
reflected by results:
tst_check_driver 1 TBROK: Test still running, sending SIGKILL
...
tst_rhost_run 1 TCONF: (null) 0 TWARN : tst_kernel.c:105: expected file /lib/modules/5.8.0-1036-azure/modules.dep does not exist or not a file
veth(null) 0 TWARN : tst_kernel.c:105: expected file /lib/modules/5.8.0-1036-azure/modules.builtin does not exist or not a file driver not available
Summary:
passed 0
failed 0
broken 0
skipped 1
warnings 0
runtest TINFO: === SHELL TEST RESULTS ===
runtest TINFO: 1x TPASS: shell/net/tst_ipaddr_un.sh
runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 2x TCONF: shell/tst_check_driver.sh shell/net/tst_rhost_run.sh
Kind regards,
Petr
> Kind regards,
> Petr
^ permalink raw reply [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 0/4] Run tests in CI
2021-07-15 7:38 ` Petr Vorel
@ 2021-07-15 7:48 ` Petr Vorel
2021-07-15 8:51 ` Cyril Hrubis
1 sibling, 0 replies; 12+ messages in thread
From: Petr Vorel @ 2021-07-15 7:48 UTC (permalink / raw)
To: ltp
Hi Cyril,
...
> > Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
> The result is much better but still not correct:
> https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707
> runtest TINFO: * shell/net/tst_ipaddr_un.sh
> Summary:
> passed 0
> failed 0
> broken 0
> skipped 1
> warnings 0
> tst_ipaddr_un 1 TINFO: timeout per run is 0h 5m 0s
> Could you please merge your fix? And I'll try sync after each test.
> https://github.com/pevik/ltp/actions/runs/1033070696
There is still some output printed later, at least
In RPN: A B | C ! ! &
In RPN: A B | C ! ! &
which is from tst_bool_expr.c, but printed after tst_strstatus
https://github.com/pevik/ltp/runs/3074335940?check_suite_focus=true#step:11:587
But it's much better.
I also found a bug in final evaluation on one of my VM:
tst_test.c:1313: TBROK: Test killed by SIGSEGV!
...
runtest TINFO: 1x TBROK: tst_bool_expr
...
runtest TINFO: === FINAL TEST RESULTS ===
runtest TPASS: All C tests TCONF/TPASS <- THIS IS WRONG
runtest TPASS: All shell tests TCONF/TPASS
Kind regards,
Petr
^ permalink raw reply [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 0/4] Run tests in CI
2021-07-15 7:38 ` Petr Vorel
2021-07-15 7:48 ` Petr Vorel
@ 2021-07-15 8:51 ` Cyril Hrubis
2021-07-15 9:22 ` Petr Vorel
1 sibling, 1 reply; 12+ messages in thread
From: Cyril Hrubis @ 2021-07-15 8:51 UTC (permalink / raw)
To: ltp
Hi!
> > Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
>
> The result is much better but still not correct:
> https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707
If tests are printing into stdout as well it will be easier to redirect
the test output in the script so that the CI gets it all into a single
file descriptor.
What about doing ./$i 1>&2 in the loop that runs the tests instead?
--
Cyril Hrubis
chrubis@suse.cz
^ permalink raw reply [flat|nested] 12+ messages in thread
* [LTP] [PATCH v5 0/4] Run tests in CI
2021-07-15 8:51 ` Cyril Hrubis
@ 2021-07-15 9:22 ` Petr Vorel
0 siblings, 0 replies; 12+ messages in thread
From: Petr Vorel @ 2021-07-15 9:22 UTC (permalink / raw)
To: ltp
> Hi!
> > > Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
> > The result is much better but still not correct:
> > https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707
> If tests are printing into stdout as well it will be easier to redirect
> the test output in the script so that the CI gets it all into a single
> file descriptor.
> What about doing ./$i 1>&2 in the loop that runs the tests instead?
How simple, that should work, I'll test that.
FYI I also need to install iproute (runtime shell API test dependency).
Simple enough to just merge it after this gets merged.
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-07-15 9:22 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-14 14:19 [LTP] [PATCH v5 0/4] Run tests in CI Petr Vorel
2021-07-14 14:19 ` [LTP] [PATCH v5 1/4] lib: Add script for running tests Petr Vorel
2021-07-14 14:19 ` [LTP] [PATCH v5 2/4] make: Add make test{, -c, -shell} targets Petr Vorel
2021-07-14 14:20 ` [LTP] [PATCH v5 3/4] build.sh: Add support for make test Petr Vorel
2021-07-14 14:20 ` [LTP] [PATCH v5 4/4] CI: Run also " Petr Vorel
2021-07-14 14:27 ` [LTP] [PATCH v5 0/4] Run tests in CI Petr Vorel
2021-07-14 15:00 ` Cyril Hrubis
2021-07-14 16:43 ` Petr Vorel
2021-07-15 7:38 ` Petr Vorel
2021-07-15 7:48 ` Petr Vorel
2021-07-15 8:51 ` Cyril Hrubis
2021-07-15 9:22 ` Petr Vorel
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.