[PATCH 1/2] kselftest: add capability to skip chosen TARGETS

[PATCH 1/2] kselftest: add capability to skip chosen TARGETS

[Cristian Marussi]

Let the user specify an optional TARGETS skiplist through the new optional
SKIP_TARGETS Makefile variable.

It is easier to skip at will a reduced and well defined list of possibly
problematic targets with SKIP_TARGETS then to provide a partially stripped
down list of good targets using the usual TARGETS variable.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 tools/testing/selftests/Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 25b43a8c2b15..103936faa46d 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -132,6 +132,10 @@ else
 		ARCH=$(ARCH) -C $(top_srcdir) headers_install
 endif
 
+# User can optionally provide a TARGETS skiplist.
+SKIP_TARGETS ?=
+TARGETS := $(filter-out $(SKIP_TARGETS), $(TARGETS))
+
 all: khdr
 	@for TARGET in $(TARGETS); do		\
 		BUILD_TARGET=$$BUILD/$$TARGET;	\
-- 
2.17.1

[PATCH 2/2] kselftest: exclude failed TARGETS from runlist

[Cristian Marussi]

A TARGET which failed to be built/installed should not be included in the
runlist generated inside the run_kselftest.sh script.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
As an example, here BPF failed to compile but was included in the runlist
and attempted to run anyway:

...
./KSFT_LAST_2/run_kselftest.sh: 37: cd: can't cd to bpf
TAP version 13
1..49
\# selftests: KSFT_LAST_2: test_verifier
\# Warning: file test_verifier is missing!
not ok 1 selftests: KSFT_LAST_2: test_verifier
\# selftests: KSFT_LAST_2: test_tag
\# Warning: file test_tag is missing!
---
 tools/testing/selftests/Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 103936faa46d..e11ace11b07c 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -202,8 +202,12 @@ ifdef INSTALL_PATH
 	echo "  cat /dev/null > \$$logfile" >> $(ALL_SCRIPT)
 	echo "fi" >> $(ALL_SCRIPT)
 
+	@# While building run_kselftest.sh skip also non-existent TARGET dirs:
+	@# they could be the result of a build failure and should NOT be
+	@# included in the generated runlist.
 	for TARGET in $(TARGETS); do \
 		BUILD_TARGET=$$BUILD/$$TARGET;	\
+		[ ! -d $$INSTALL_PATH/$$TARGET ] && echo "Skipping non-existent dir: $$TARGET" && continue; \
 		echo "[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \
 		echo "cd $$TARGET" >> $(ALL_SCRIPT); \
 		echo -n "run_many" >> $(ALL_SCRIPT); \
-- 
2.17.1

Re: [PATCH 2/2] kselftest: exclude failed TARGETS from runlist

[shuah]

On 9/25/19 7:24 AM, Cristian Marussi wrote:
> A TARGET which failed to be built/installed should not be included in the
> runlist generated inside the run_kselftest.sh script.
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
> As an example, here BPF failed to compile but was included in the runlist
> and attempted to run anyway:
> 
> ...
> ./KSFT_LAST_2/run_kselftest.sh: 37: cd: can't cd to bpf
> TAP version 13
> 1..49
> \# selftests: KSFT_LAST_2: test_verifier
> \# Warning: file test_verifier is missing!
> not ok 1 selftests: KSFT_LAST_2: test_verifier
> \# selftests: KSFT_LAST_2: test_tag
> \# Warning: file test_tag is missing!
> ---
>   tools/testing/selftests/Makefile | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
> index 103936faa46d..e11ace11b07c 100644
> --- a/tools/testing/selftests/Makefile
> +++ b/tools/testing/selftests/Makefile
> @@ -202,8 +202,12 @@ ifdef INSTALL_PATH
>   	echo "  cat /dev/null > \$$logfile" >> $(ALL_SCRIPT)
>   	echo "fi" >> $(ALL_SCRIPT)
>   
> +	@# While building run_kselftest.sh skip also non-existent TARGET dirs:
> +	@# they could be the result of a build failure and should NOT be
> +	@# included in the generated runlist.
>   	for TARGET in $(TARGETS); do \
>   		BUILD_TARGET=$$BUILD/$$TARGET;	\
> +		[ ! -d $$INSTALL_PATH/$$TARGET ] && echo "Skipping non-existent dir: $$TARGET" && continue; \
>   		echo "[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \
>   		echo "cd $$TARGET" >> $(ALL_SCRIPT); \
>   		echo -n "run_many" >> $(ALL_SCRIPT); \
> 

This is great. Thanks for the patch.

-- Shuah

Re: [PATCH 1/2] kselftest: add capability to skip chosen TARGETS

[shuah]

On 9/25/19 7:24 AM, Cristian Marussi wrote:
> Let the user specify an optional TARGETS skiplist through the new optional
> SKIP_TARGETS Makefile variable.
> 
> It is easier to skip at will a reduced and well defined list of possibly
> problematic targets with SKIP_TARGETS then to provide a partially stripped
> down list of good targets using the usual TARGETS variable.
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>   tools/testing/selftests/Makefile | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
> index 25b43a8c2b15..103936faa46d 100644
> --- a/tools/testing/selftests/Makefile
> +++ b/tools/testing/selftests/Makefile
> @@ -132,6 +132,10 @@ else
>   		ARCH=$(ARCH) -C $(top_srcdir) headers_install
>   endif
>   
> +# User can optionally provide a TARGETS skiplist.
> +SKIP_TARGETS ?=
> +TARGETS := $(filter-out $(SKIP_TARGETS), $(TARGETS))
> +
>   all: khdr
>   	@for TARGET in $(TARGETS); do		\
>   		BUILD_TARGET=$$BUILD/$$TARGET;	\
> 

Thanks for doing this. This looks good to me. Would you like to
update the documentation file?

thanks,
-- Shuah

Re: [PATCH 1/2] kselftest: add capability to skip chosen TARGETS

[Cristian Marussi]

On 25/09/2019 20:36, shuah wrote:
> On 9/25/19 7:24 AM, Cristian Marussi wrote:
>> Let the user specify an optional TARGETS skiplist through the new optional
>> SKIP_TARGETS Makefile variable.
>>
>> It is easier to skip at will a reduced and well defined list of possibly
>> problematic targets with SKIP_TARGETS then to provide a partially stripped
>> down list of good targets using the usual TARGETS variable.
>>
>> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
>> ---
>>   tools/testing/selftests/Makefile | 4 ++++
>>   1 file changed, 4 insertions(+)
>>
>> diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
>> index 25b43a8c2b15..103936faa46d 100644
>> --- a/tools/testing/selftests/Makefile
>> +++ b/tools/testing/selftests/Makefile
>> @@ -132,6 +132,10 @@ else
>>   		ARCH=$(ARCH) -C $(top_srcdir) headers_install
>>   endif
>>   
>> +# User can optionally provide a TARGETS skiplist.
>> +SKIP_TARGETS ?=
>> +TARGETS := $(filter-out $(SKIP_TARGETS), $(TARGETS))
>> +
>>   all: khdr
>>   	@for TARGET in $(TARGETS); do		\
>>   		BUILD_TARGET=$$BUILD/$$TARGET;	\
>>
> 
> Thanks for doing this. This looks good to me. Would you like to
> update the documentation file?
> 

Thanks for the review Shuah, I'll add doc and fixes in V2
with a proper cover letter

Thanks

Cristian
> thanks,
> -- Shuah
> 

RE: [PATCH 1/2] kselftest: add capability to skip chosen TARGETS

[Tim.Bird]

Just a few nits inline below.

> -----Original Message-----
> From: Cristian Marussi on Wednesday, September 25, 2019 3:24 AM
> 
> Let the user specify an optional TARGETS skiplist through the new optional
> SKIP_TARGETS Makefile variable.
> 
> It is easier to skip at will a reduced and well defined list of possibly

It seems like there's a word missing.
at will a -> at will using a

> problematic targets with SKIP_TARGETS then to provide a partially stripped

then -> than

> down list of good targets using the usual TARGETS variable.
> 
> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  tools/testing/selftests/Makefile | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/tools/testing/selftests/Makefile
> b/tools/testing/selftests/Makefile
> index 25b43a8c2b15..103936faa46d 100644
> --- a/tools/testing/selftests/Makefile
> +++ b/tools/testing/selftests/Makefile
> @@ -132,6 +132,10 @@ else
>  		ARCH=$(ARCH) -C $(top_srcdir) headers_install
>  endif
> 
> +# User can optionally provide a TARGETS skiplist.
> +SKIP_TARGETS ?=
> +TARGETS := $(filter-out $(SKIP_TARGETS), $(TARGETS))
> +
>  all: khdr
>  	@for TARGET in $(TARGETS); do		\
>  		BUILD_TARGET=$$BUILD/$$TARGET;	\
> --
> 2.17.1

Great feature!  Thanks.
 -- Tim

Re: [PATCH 1/2] kselftest: add capability to skip chosen TARGETS

[Cristian Marussi]

Hi Tim

thanks for the review.

On 25/09/2019 21:20, Tim.Bird@sony.com wrote:
> Just a few nits inline below.
> 
>> -----Original Message-----
>> From: Cristian Marussi on Wednesday, September 25, 2019 3:24 AM
>>
>> Let the user specify an optional TARGETS skiplist through the new optional
>> SKIP_TARGETS Makefile variable.
>>
>> It is easier to skip at will a reduced and well defined list of possibly
> 
> It seems like there's a word missing.
> at will a -> at will using a
> 
>> problematic targets with SKIP_TARGETS then to provide a partially stripped
> 
> then -> than
> 
>> down list of good targets using the usual TARGETS variable.
>>
>> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
>> ---
>>  tools/testing/selftests/Makefile | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/tools/testing/selftests/Makefile
>> b/tools/testing/selftests/Makefile
>> index 25b43a8c2b15..103936faa46d 100644
>> --- a/tools/testing/selftests/Makefile
>> +++ b/tools/testing/selftests/Makefile
>> @@ -132,6 +132,10 @@ else
>>  		ARCH=$(ARCH) -C $(top_srcdir) headers_install
>>  endif
>>
>> +# User can optionally provide a TARGETS skiplist.
>> +SKIP_TARGETS ?=
>> +TARGETS := $(filter-out $(SKIP_TARGETS), $(TARGETS))
>> +
>>  all: khdr
>>  	@for TARGET in $(TARGETS); do		\
>>  		BUILD_TARGET=$$BUILD/$$TARGET;	\
>> --
>> 2.17.1
> 
> Great feature!  Thanks.
>  -- Tim
> 

I'll fix all in V2.

Thanks

Cristian