* [LTP] [RFC PATCH v2 1/1] make: Add make test{, -c, -c-run, -shell-run} targets
@ 2021-06-18 19:12 Petr Vorel
2021-06-21 8:41 ` Richard Palethorpe
2021-06-22 8:59 ` Cyril Hrubis
0 siblings, 2 replies; 6+ messages in thread
From: Petr Vorel @ 2021-06-18 19:12 UTC (permalink / raw)
To: ltp
For testing C and shell API.
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
Hi all,
another attempt for make test target.
In a long term, I'd like to wrap the execution with some script, as I
suppose there will be some metadata in test, allowing to run tests which
don't TPASS or TBROK (we have quite a lot of them). Then the summary
would be even more useful.
FIXME: broken in lib/, it runs all test twice.
changes v1->v2:
Not sure if it's a good approach to process each test separately,
there is no loop as Cyril suggested in [1], thus there is no summary
what happen.
I previously had in similar loop in include/mk/generic_leaf_target.inc:
test-c-run: $(TEST_TARGETS)
@set -e; echo; echo "===== Test C API ====="; \
for i in $(TEST_TARGETS); do \
echo; echo "* $$i"; \
echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \
PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \
done; echo; \
echo "All C API tests passed or TCONF"
test-shell-run:
@set -e; echo; echo "===== Test shell API ====="; \
for i in $(TEST_SHELL_TARGETS); do \
echo; echo "* $$i"; \
PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \
done; echo; \
echo "All shell API tests passed or TCONF"
but that was problematic in lib (which uses generic_leaf_target.inc).
Kind regards,
Petr
[1] https://patchwork.ozlabs.org/project/ltp/patch/20210603183827.24339-2-pvorel@suse.cz/#2695533
Makefile | 23 ++++++++++++++++++++++-
include/mk/generic_leaf_target.inc | 25 ++++++++++++++++++++++---
include/mk/generic_trunk_target.inc | 2 +-
lib/newlib_tests/Makefile | 13 +++++++++++++
4 files changed, 58 insertions(+), 5 deletions(-)
diff --git a/Makefile b/Makefile
index 56812d77b..40cd4b1cb 100644
--- a/Makefile
+++ b/Makefile
@@ -75,10 +75,14 @@ endif
INSTALL_TARGETS += $(COMMON_TARGETS)
CLEAN_TARGETS += $(COMMON_TARGETS) lib libs
BOOTSTRAP_TARGETS := $(sort $(COMMON_TARGETS) $(CLEAN_TARGETS) $(INSTALL_TARGETS))
+TEST_TARGETS := lib
+TEST_SHELL_TARGETS := lib
CLEAN_TARGETS := $(addsuffix -clean,$(CLEAN_TARGETS))
INSTALL_TARGETS := $(addsuffix -install,$(INSTALL_TARGETS))
MAKE_TARGETS := $(addsuffix -all,$(filter-out lib,$(COMMON_TARGETS)))
+TEST_TARGETS := $(addsuffix -test-c-run,$(TEST_TARGETS))
+TEST_SHELL_TARGETS := $(addsuffix -test-shell-run,$(TEST_SHELL_TARGETS))
# There's no reason why we should run `all' twice. Otherwise we're just wasting
# 3+ mins of useful CPU cycles on a modern machine, and even more time on an
@@ -189,8 +193,25 @@ INSTALL_TARGETS += $(addprefix $(DESTDIR)/$(bindir)/,$(BINDIR_INSTALL_SCRIPTS))
$(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir)
+## Test library API
+.PHONY: test test-c test-c-run test-shell-run
+
+$(TEST_TARGETS): lib-all
+ $(MAKE) -C "$(subst -test-c-run,,$@)" \
+ -f "$(abs_top_srcdir)/$(subst -test-c-run,,$@)/Makefile" test-c-run
+
+$(TEST_SHELL_TARGETS): lib-all
+ $(MAKE) -C "$(subst -test-shell-run,,$@)" \
+ -f "$(abs_top_srcdir)/$(subst -test-shell-run,,$@)/Makefile" test-shell-run
+
+test-c-run: $(TEST_TARGETS)
+
+test-shell-run: $(TEST_SHELL_TARGETS)
+
+test: test-c test-c-run test-shell-run
+
## Install
-install: $(INSTALL_TARGETS)
+install: lib-all $(INSTALL_TARGETS)
## Help
.PHONY: help
diff --git a/include/mk/generic_leaf_target.inc b/include/mk/generic_leaf_target.inc
index 64953f89a..19e238f92 100644
--- a/include/mk/generic_leaf_target.inc
+++ b/include/mk/generic_leaf_target.inc
@@ -63,8 +63,12 @@
# real files or directories). This will automatically append
# adds the .o suffix to all files referenced by
# $(MAKE_TARGETS)) to CLEAN_TARGETS, if MAKE_TARGETS wasn't
-# defined (see
-# $(MAKE_TARGETS)).
+# defined (see $(MAKE_TARGETS)).
+#
+# $(TEST_TARGETS) : LTP C API tests to be runned with test-c-run.
+#
+# $(TEST_SHELL_TARGETS) : LTP shell API tests to be runned with test-shell-run.
+#
# $(INSTALL_MODE) : What mode should we using when calling
# install(1)?
#
@@ -92,7 +96,7 @@
# INSTALL_DIR := $(libdir)
#
-.PHONY: all clean install
+.PHONY: all clean install test test-c test-c-run test-shell-run
ifneq ($(strip $(MAKE_TARGETS)),)
$(MAKE_TARGETS) += $(HOST_MAKE_TARGETS)
@@ -102,11 +106,26 @@ $(MAKE_TARGETS): | $(MAKE_DEPS)
all: $(MAKE_TARGETS)
+TEST_TARGETS_RUN := $(addsuffix -run,$(TEST_TARGETS))
+TEST_SHELL_TARGETS_RUN := $(addsuffix -run,$(TEST_SHELL_TARGETS))
+
+$(TEST_TARGETS_RUN): | $(TEST_TARGETS)
+test-c-run: $(TEST_TARGETS_RUN)
+test-shell-run: $(TEST_SHELL_TARGETS_RUN)
+
+.PHONY: $(TEST_TARGETS_RUN) $(TEST_SHELL_TARGETS_RUN)
+$(TEST_TARGETS_RUN) $(TEST_SHELL_TARGETS_RUN):
+ @set -e; echo; echo "* $(subst -run,,$@)"; \
+ PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$(subst -run,,$@) || [ $$? -eq 32 ]; \
+
+test: test-c-run test-shell-run
+
clean:: $(CLEAN_DEPS)
-$(RM) -f -r $(CLEAN_TARGETS)
$(INSTALL_FILES): | $(INSTALL_DEPS)
install: $(INSTALL_FILES)
+test-c: $(TEST_TARGETS)
# vim: syntax=make
diff --git a/include/mk/generic_trunk_target.inc b/include/mk/generic_trunk_target.inc
index fc59f944f..4427a0f86 100644
--- a/include/mk/generic_trunk_target.inc
+++ b/include/mk/generic_trunk_target.inc
@@ -48,7 +48,7 @@
include $(top_srcdir)/include/mk/functions.mk
-RECURSIVE_TARGETS ?= all install
+RECURSIVE_TARGETS ?= all install test test-c test-c-run test-shell-run
$(eval $(get_make_dirs))
diff --git a/lib/newlib_tests/Makefile b/lib/newlib_tests/Makefile
index 30ca6810c..7ac28660d 100644
--- a/lib/newlib_tests/Makefile
+++ b/lib/newlib_tests/Makefile
@@ -19,5 +19,18 @@ ifeq ($(ANDROID),1)
FILTER_OUT_MAKE_TARGETS += test08
endif
+# C API tests
+# NOTE: only those which exit 0 (TPASS) or 32 (TCONF)
+# TODO: commented out for quicker demonstration
+# TEST_TARGETS += test05 test07 test09 test12 test15 test16 test18
+# TEST_TARGETS += test_exec test_timer tst_bool_expr tst_res_hexd
+# TEST_TARGETS += tst_strstatus tst_fuzzy_sync01 tst_fuzzy_sync02
+TEST_TARGETS += tst_strstatus test05
+
+# shell API tests
+# NOTE: only those which exit 0 (TPASS) or 32 (TCONF)
+TEST_SHELL_TARGETS += shell/tst_check_driver.sh
+# TODO: commented out for quicker demonstration
+# TEST_SHELL_TARGETS += shell/net/*.sh
include $(top_srcdir)/include/mk/generic_leaf_target.mk
--
2.32.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [LTP] [RFC PATCH v2 1/1] make: Add make test{, -c, -c-run, -shell-run} targets
2021-06-18 19:12 [LTP] [RFC PATCH v2 1/1] make: Add make test{, -c, -c-run, -shell-run} targets Petr Vorel
@ 2021-06-21 8:41 ` Richard Palethorpe
2021-06-21 8:48 ` Petr Vorel
2021-06-22 8:59 ` Cyril Hrubis
1 sibling, 1 reply; 6+ messages in thread
From: Richard Palethorpe @ 2021-06-21 8:41 UTC (permalink / raw)
To: ltp
Hello,
Petr Vorel <pvorel@suse.cz> writes:
> For testing C and shell API.
>
> Signed-off-by: Petr Vorel <pvorel@suse.cz>
> ---
> Hi all,
>
> another attempt for make test target.
>
> In a long term, I'd like to wrap the execution with some script, as I
> suppose there will be some metadata in test, allowing to run tests which
> don't TPASS or TBROK (we have quite a lot of them). Then the summary
> would be even more useful.
For each test you could have an tcl/expect (or equivalent Perl/shell if
we don't already require tcl) script which matches the output including
return value and stderr/stdout.
For most tests it would just call a common function to check for
TPASS/TBROK. For tests where we want to look for a given message, it
could match the output. If we sometimes expect TCONF then it could
perform a check to verify that it really should return TCONF.
I guess you could just put some data in a comment. I think that is
likely to be harder though. At least with the number of tests we current
have.
--
Thank you,
Richard.
^ permalink raw reply [flat|nested] 6+ messages in thread
* [LTP] [RFC PATCH v2 1/1] make: Add make test{, -c, -c-run, -shell-run} targets
2021-06-21 8:41 ` Richard Palethorpe
@ 2021-06-21 8:48 ` Petr Vorel
0 siblings, 0 replies; 6+ messages in thread
From: Petr Vorel @ 2021-06-21 8:48 UTC (permalink / raw)
To: ltp
Hi Richie,
> Hello,
> Petr Vorel <pvorel@suse.cz> writes:
> > For testing C and shell API.
> > Signed-off-by: Petr Vorel <pvorel@suse.cz>
> > ---
> > Hi all,
> > another attempt for make test target.
> > In a long term, I'd like to wrap the execution with some script, as I
> > suppose there will be some metadata in test, allowing to run tests which
> > don't TPASS or TBROK (we have quite a lot of them). Then the summary
> > would be even more useful.
> For each test you could have an tcl/expect (or equivalent Perl/shell if
> we don't already require tcl) script which matches the output including
> return value and stderr/stdout.
I'd prefer to avoid tcl/expect. FYI There was some effort [1], based on my
previous work [1], I plan to get back to it.
> For most tests it would just call a common function to check for
> TPASS/TBROK. For tests where we want to look for a given message, it
> could match the output. If we sometimes expect TCONF then it could
> perform a check to verify that it really should return TCONF.
As I noted in [2] today, I might create very simple library for these test
wrappers (small subset of things needed in test wrappers like
lib/newlib_tests/shell/test_timeout.sh).
> I guess you could just put some data in a comment. I think that is
> likely to be harder though. At least with the number of tests we current
> have.
Yep, that's what was used in [1].
Kind regards,
Petr
[1] https://patchwork.ozlabs.org/project/ltp/patch/ce675759672af52bea02c11d51bd7d10f0bcb5cb.1566500817.git.clanig@suse.com/
[2] https://lists.linux.it/pipermail/ltp/2021-June/023306.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* [LTP] [RFC PATCH v2 1/1] make: Add make test{, -c, -c-run, -shell-run} targets
2021-06-18 19:12 [LTP] [RFC PATCH v2 1/1] make: Add make test{, -c, -c-run, -shell-run} targets Petr Vorel
2021-06-21 8:41 ` Richard Palethorpe
@ 2021-06-22 8:59 ` Cyril Hrubis
2021-06-22 9:50 ` Petr Vorel
1 sibling, 1 reply; 6+ messages in thread
From: Cyril Hrubis @ 2021-06-22 8:59 UTC (permalink / raw)
To: ltp
Hi!
> For testing C and shell API.
>
> Signed-off-by: Petr Vorel <pvorel@suse.cz>
> ---
> Hi all,
>
> another attempt for make test target.
>
> In a long term, I'd like to wrap the execution with some script, as I
> suppose there will be some metadata in test, allowing to run tests which
> don't TPASS or TBROK (we have quite a lot of them). Then the summary
> would be even more useful.
>
> FIXME: broken in lib/, it runs all test twice.
>
> changes v1->v2:
> Not sure if it's a good approach to process each test separately,
> there is no loop as Cyril suggested in [1], thus there is no summary
> what happen.
>
> I previously had in similar loop in include/mk/generic_leaf_target.inc:
>
> test-c-run: $(TEST_TARGETS)
> @set -e; echo; echo "===== Test C API ====="; \
> for i in $(TEST_TARGETS); do \
> echo; echo "* $$i"; \
> echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \
> PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \
> done; echo; \
> echo "All C API tests passed or TCONF"
>
> test-shell-run:
> @set -e; echo; echo "===== Test shell API ====="; \
> for i in $(TEST_SHELL_TARGETS); do \
> echo; echo "* $$i"; \
> PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \
> done; echo; \
> echo "All shell API tests passed or TCONF"
>
> but that was problematic in lib (which uses generic_leaf_target.inc).
Why not just add runtest.sh into newlib_tests/ that would do all the
work, then we can forget about all the trickery and just run the scritp
on 'make test-c'. Well we would have to call make in newlib_tests first
to make sure it was compiled, but that should be it.
And the same for testcases/lib/.
I do not think that adding a target per a testcase is sane.
--
Cyril Hrubis
chrubis@suse.cz
^ permalink raw reply [flat|nested] 6+ messages in thread
* [LTP] [RFC PATCH v2 1/1] make: Add make test{, -c, -c-run, -shell-run} targets
2021-06-22 8:59 ` Cyril Hrubis
@ 2021-06-22 9:50 ` Petr Vorel
2021-06-22 10:10 ` Petr Vorel
0 siblings, 1 reply; 6+ messages in thread
From: Petr Vorel @ 2021-06-22 9:50 UTC (permalink / raw)
To: ltp
Hi Cyril,
> Hi!
> > For testing C and shell API.
> > Signed-off-by: Petr Vorel <pvorel@suse.cz>
> > ---
> > Hi all,
> > another attempt for make test target.
> > In a long term, I'd like to wrap the execution with some script, as I
> > suppose there will be some metadata in test, allowing to run tests which
> > don't TPASS or TBROK (we have quite a lot of them). Then the summary
> > would be even more useful.
> > FIXME: broken in lib/, it runs all test twice.
> > changes v1->v2:
> > Not sure if it's a good approach to process each test separately,
> > there is no loop as Cyril suggested in [1], thus there is no summary
> > what happen.
> > I previously had in similar loop in include/mk/generic_leaf_target.inc:
> > test-c-run: $(TEST_TARGETS)
> > @set -e; echo; echo "===== Test C API ====="; \
> > for i in $(TEST_TARGETS); do \
> > echo; echo "* $$i"; \
> > echo "PATH $(top_srcdir)/testcases/lib:$$PATH"; \
> > PATH="$(top_srcdir)/testcases/lib:$$PATH" ./$$i || [ $$? -eq 32 ]; \
> > done; echo; \
> > echo "All C API tests passed or TCONF"
> > test-shell-run:
> > @set -e; echo; echo "===== Test shell API ====="; \
> > for i in $(TEST_SHELL_TARGETS); do \
> > echo; echo "* $$i"; \
> > PATH="$(abs_top_srcdir)/testcases/lib:$(abs_top_builddir)/testcases/lib:$$PATH" $(abs_srcdir)/$$i || [ $$? -eq 32 ]; \
> > done; echo; \
> > echo "All shell API tests passed or TCONF"
> > but that was problematic in lib (which uses generic_leaf_target.inc).
> Why not just add runtest.sh into newlib_tests/ that would do all the
> work, then we can forget about all the trickery and just run the scritp
> on 'make test-c'. Well we would have to call make in newlib_tests first
> to make sure it was compiled, but that should be it.
> And the same for testcases/lib/.
> I do not think that adding a target per a testcase is sane.
Thanks for your input, makes sense. Actually, adding a script which would run it
was my approach in 2 years old patch [1].
I'll add runtest.sh which will run all tests which by default TPASS/TCONF now,
but possible to specify just some tests (speedup the development).
After we merge it I'll continue in Christian's effort to add metadata of
expected output [2], which allows us to run all tests (including these which
fail).
I suggest to add 3 targets: make test test-c test-shell (test will run both
test-c test-shell).
Kind regards,
Petr
[1] https://patchwork.ozlabs.org/project/ltp/patch/20190924182841.4528-1-pvorel@suse.cz/
[2] https://patchwork.ozlabs.org/project/ltp/patch/ce675759672af52bea02c11d51bd7d10f0bcb5cb.1566500817.git.clanig@suse.com/
^ permalink raw reply [flat|nested] 6+ messages in thread
* [LTP] [RFC PATCH v2 1/1] make: Add make test{, -c, -c-run, -shell-run} targets
2021-06-22 9:50 ` Petr Vorel
@ 2021-06-22 10:10 ` Petr Vorel
0 siblings, 0 replies; 6+ messages in thread
From: Petr Vorel @ 2021-06-22 10:10 UTC (permalink / raw)
To: ltp
Hi all,
...
> > Why not just add runtest.sh into newlib_tests/ that would do all the
> > work, then we can forget about all the trickery and just run the scritp
> > on 'make test-c'. Well we would have to call make in newlib_tests first
> > to make sure it was compiled, but that should be it.
> > And the same for testcases/lib/.
> > I do not think that adding a target per a testcase is sane.
> Thanks for your input, makes sense. Actually, adding a script which would run it
> was my approach in 2 years old patch [1].
> I'll add runtest.sh which will run all tests which by default TPASS/TCONF now,
> but possible to specify just some tests (speedup the development).
Also make install works also on lib/newlib_tests. Thus I'll install also runtest.sh
and try to setup PATH a way it works on both running from git and running from
make install. It might be worth to have lib tests runnable on SUT.
Kind regards,
Petr
> After we merge it I'll continue in Christian's effort to add metadata of
> expected output [2], which allows us to run all tests (including these which
> fail).
> I suggest to add 3 targets: make test test-c test-shell (test will run both
> test-c test-shell).
> Kind regards,
> Petr
> [1] https://patchwork.ozlabs.org/project/ltp/patch/20190924182841.4528-1-pvorel@suse.cz/
> [2] https://patchwork.ozlabs.org/project/ltp/patch/ce675759672af52bea02c11d51bd7d10f0bcb5cb.1566500817.git.clanig@suse.com/
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-06-22 10:10 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-18 19:12 [LTP] [RFC PATCH v2 1/1] make: Add make test{, -c, -c-run, -shell-run} targets Petr Vorel
2021-06-21 8:41 ` Richard Palethorpe
2021-06-21 8:48 ` Petr Vorel
2021-06-22 8:59 ` Cyril Hrubis
2021-06-22 9:50 ` Petr Vorel
2021-06-22 10:10 ` Petr Vorel
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.