[PATCH v2 0/2] Fix KSFT toplevel makefile behaviour

[PATCH v2 0/2] Fix KSFT toplevel makefile behaviour

[Cristian Marussi]

Hi

so this series carries two small fixes to the toplevel KSelfTest makefile
which I found useful especially while attempting to run the suite in
automation.

[1/2]
While it is already possible to specify a limited list of TARGETS to run,
it is not instead easily possible to state a list of targets NOT to run
(say due to specific instability issues).
Moreover providing such a skip list through a stripped down list of TARGETS
it is cumbersome and fragile since this poses the risk to stick to an old
stale stripped TARGETS list once upstream decides to add more default
targets.
A new SKIP_TARGETS Makefile variable is provided by this patch to easily
specify a skiplist for target subsystems.

[2/2]
Currently when some target fails to build, KSFT Makefile just carries on
building as much subsystems as it can: unfortunately this is not properly
reflected also in the generation of the runlist inside run_kselftest.sh.
This patch rectifies this behaviour checking for the existence of a target
directory in the INSTALL_PATH before adding the related snippet to the
run_kselftest.sh script.


Thanks

Cristian

Changelog

v1 --> v2
- added Documentation
- various typos fixed
- added a proper override when filtering-out SKIP_TARGETS from TARGETS
  to make it work also when TARGETS is provided too from the cmdline

Cristian Marussi (2):
  kselftest: add capability to skip chosen TARGETS
  kselftest: exclude failed TARGETS from runlist

 Documentation/dev-tools/kselftest.rst | 11 +++++++++++
 tools/testing/selftests/Makefile      | 11 +++++++++++
 2 files changed, 22 insertions(+)

-- 
2.17.1

[PATCH v2 1/2] kselftest: add capability to skip chosen TARGETS

[Cristian Marussi]

Let the user specify an optional TARGETS skiplist through the new optional
SKIP_TARGETS Makefile variable.

It is easier to skip at will using a reduced and well defined list of
possibly problematic targets with SKIP_TARGETS than to provide a partially
stripped down list of good targets using the usual TARGETS variable.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
v1 --> v2
- fixed commit msg
- added documentation fix
- added forced override to allow TARGETS= and SKIP_TARGETS
  being both passed on the cmdline
---
 Documentation/dev-tools/kselftest.rst | 16 ++++++++++++++++
 tools/testing/selftests/Makefile      |  7 +++++++
 2 files changed, 23 insertions(+)

diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index 25604904fa6e..ecdfdc9d4b03 100644
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -89,6 +89,22 @@ To build, save output files in a separate directory with KBUILD_OUTPUT ::
 
   $ export KBUILD_OUTPUT=/tmp/kselftest; make TARGETS="size timers" kselftest
 
+Additionally you can use the "SKIP_TARGETS" variable on the make command
+line to specify one or more targets to exclude from the TARGETS list.
+
+To run all tests but a single subsystem::
+
+  $ make -C tools/testing/selftests SKIP_TARGETS=ptrace run_tests
+
+You can specify multiple tests to skip::
+
+  $  make SKIP_TARGETS="size timers" kselftest
+
+You can also specify a restricted list of tests to run together with a
+dedicated skiplist::
+
+  $  make TARGETS="bpf breakpoints size timers" SKIP_TARGETS=bpf kselftest
+
 See the top-level tools/testing/selftests/Makefile for the list of all
 possible targets.
 
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 25b43a8c2b15..9621b26ee433 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -63,6 +63,13 @@ TARGETS += zram
 TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
+# User can optionally provide a TARGETS skiplist.
+SKIP_TARGETS ?=
+ifneq ($(SKIP_TARGETS),)
+	TMP := $(filter-out $(SKIP_TARGETS), $(TARGETS))
+	override TARGETS := $(TMP)
+endif
+
 # Clear LDFLAGS and MAKEFLAGS if called from main
 # Makefile to avoid test build failures when test
 # Makefile doesn't have explicit build rules.
-- 
2.17.1

[PATCH v2 2/2] kselftest: exclude failed TARGETS from runlist

[Cristian Marussi]

A TARGET which failed to be built/installed should not be included in the
runlist generated inside the run_kselftest.sh script.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
As an example, here BPF failed to compile but was included in the runlist
and attempted to run anyway:

...
./KSFT_LAST_2/run_kselftest.sh: 37: cd: can't cd to bpf
TAP version 13
1..49
\# selftests: KSFT_LAST_2: test_verifier
\# Warning: file test_verifier is missing!
not ok 1 selftests: KSFT_LAST_2: test_verifier
\# selftests: KSFT_LAST_2: test_tag
\# Warning: file test_tag is missing!
---
 tools/testing/selftests/Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 9621b26ee433..d7a9df186837 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -205,8 +205,12 @@ ifdef INSTALL_PATH
 	echo "  cat /dev/null > \$$logfile" >> $(ALL_SCRIPT)
 	echo "fi" >> $(ALL_SCRIPT)
 
+	@# While building run_kselftest.sh skip also non-existent TARGET dirs:
+	@# they could be the result of a build failure and should NOT be
+	@# included in the generated runlist.
 	for TARGET in $(TARGETS); do \
 		BUILD_TARGET=$$BUILD/$$TARGET;	\
+		[ ! -d $$INSTALL_PATH/$$TARGET ] && echo "Skipping non-existent dir: $$TARGET" && continue; \
 		echo "[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \
 		echo "cd $$TARGET" >> $(ALL_SCRIPT); \
 		echo -n "run_many" >> $(ALL_SCRIPT); \
-- 
2.17.1