[PATCH v3 00/20] kselftest install target feature

[PATCH v3 00/20] kselftest install target feature

[Shuah Khan]

This patch series adds a new kselftest_install make target
to enable selftest install. When make kselftest_install is
run, selftests are installed on the system. A new install
target is added to selftests Makefile which will install
targets for the tests that are specified in INSTALL_TARGETS.
During install, a script is generated to run tests that are
installed. This script will be installed in the selftest install
directory. Individual test Makefiles are changed to add to the
script. This will allow new tests to add install and run test
commands to the generated kselftest script. kselftest target
now depends on kselftest_install and runs the generated kselftest
script to reduce duplicate work and for common look and feel when
running tests.

This approach leverages and extends the existing framework that
uses makefile targets to implement run_tests and adds install
target. This will scale well as new tests get added and makes
it easier for test writers to add install target at the same
time new test gets added.

This series is uploaded to the following experimental branch
for anybody that is interested in playing with it:
git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git devel

Some benchmark numbers: no relation to this patch series,
I just happened to get some timing numbers, this is nice
and low.
kselftest full run from install dir using kselftest.sh
9.41user 3.55system 0:24.86elapsed

This v3 series reduces duplicate code to generate script
in indiviual test Makefiles and consolidates support in
selftests main Makefile. In the main Makefile, it does
minimal work to set and export install path. In this
series exec and powerpc tests are not included in the
install, this work will be done in future patches. exec
and powerpc are still run when make kselftest is invoked.

v2 series: addressed the duplicate code in install and
run_tests targets in individual test Makefiles.
Reference: https://lkml.org/lkml/2014/11/4/707

Shuah Khan (20):
  selftests/breakpoints: add install target to enable test install
  selftests/cpu-hotplug: add install target to enable test install
  selftests/efivarfs: add install target to enable test install
  selftests/firmware: add install target to enable test install
  selftests/ftrace: add install target to enable test install
  selftests/ipc: add install target to enable test install
  selftests/kcmp: add install target to enable test install
  selftests/memfd: add install target to enable test install
  selftests/memory-hotplug: add install target to enable test install
  selftests/mount: add install target to enable test install
  selftests/mqueue: add install target to enable test install
  selftests/net: add install target to enable test install
  selftests/ptrace: add install target to enable test install
  selftests/size: add install target to enable test install
  selftests/sysctl: add install target to enable test install
  selftests/timers: add install target to enable test install
  selftests/user: add install target to enable test install
  selftests/vm: add install target to enable test install
  selftests: add install target to enable test install
  kbuild: add a new kselftest_install make target to install selftests

 Makefile                                           | 14 +++++-
 tools/testing/selftests/Makefile                   | 54 +++++++++++++++++++++-
 tools/testing/selftests/breakpoints/Makefile       | 19 +++++++-
 tools/testing/selftests/cpu-hotplug/Makefile       | 14 +++++-
 .../{on-off-test.sh => cpu-on-off-test.sh}         |  0
 tools/testing/selftests/efivarfs/Makefile          | 16 ++++++-
 tools/testing/selftests/firmware/Makefile          | 43 ++++++++++-------
 tools/testing/selftests/ftrace/Makefile            | 11 ++++-
 tools/testing/selftests/ipc/Makefile               | 19 +++++++-
 tools/testing/selftests/kcmp/Makefile              | 13 +++++-
 tools/testing/selftests/memfd/Makefile             | 17 +++++--
 tools/testing/selftests/memory-hotplug/Makefile    | 14 +++++-
 .../{on-off-test.sh => mem-on-off-test.sh}         |  0
 tools/testing/selftests/mount/Makefile             | 12 ++++-
 tools/testing/selftests/mqueue/Makefile            | 18 ++++++--
 tools/testing/selftests/net/Makefile               | 20 ++++++--
 tools/testing/selftests/ptrace/Makefile            | 16 +++++--
 tools/testing/selftests/size/Makefile              | 12 ++++-
 tools/testing/selftests/sysctl/Makefile            | 17 ++++++-
 tools/testing/selftests/timers/Makefile            | 12 ++++-
 tools/testing/selftests/user/Makefile              | 12 ++++-
 tools/testing/selftests/vm/Makefile                | 11 ++++-
 22 files changed, 315 insertions(+), 49 deletions(-)
 rename tools/testing/selftests/cpu-hotplug/{on-off-test.sh => cpu-on-off-test.sh} (100%)
 rename tools/testing/selftests/memory-hotplug/{on-off-test.sh => mem-on-off-test.sh} (100%)

-- 
2.1.0

[PATCH v3 01/20] selftests/breakpoints: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/breakpoints/Makefile | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/breakpoints/Makefile b/tools/testing/selftests/breakpoints/Makefile
index e18b42b..b5a5fe8 100644
--- a/tools/testing/selftests/breakpoints/Makefile
+++ b/tools/testing/selftests/breakpoints/Makefile
@@ -8,6 +8,7 @@ ifeq ($(ARCH),x86_64)
 	ARCH := x86
 endif
 
+TEST_STR = ./breakpoint_test || echo breakpoints selftests: [FAIL]
 
 all:
 ifeq ($(ARCH),x86)
@@ -16,8 +17,22 @@ else
 	echo "Not an x86 target, can't build breakpoints selftests"
 endif
 
-run_tests:
-	@./breakpoint_test || echo "breakpoints selftests: [FAIL]"
+install:
+ifdef INSTALL_KSFT_PATH
+ifeq ($(ARCH),x86)
+	install ./breakpoint_test $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Not an x86 target, unable to install breakpoints selftests
+endif
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
+run_tests: all
+ifeq ($(ARCH),x86)
+	@$(TEST_STR)
+endif
 
 clean:
 	rm -fr breakpoint_test
-- 
2.1.0

[PATCH v3 02/20] selftests/cpu-hotplug: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/cpu-hotplug/Makefile               | 14 ++++++++++++--
 .../cpu-hotplug/{on-off-test.sh => cpu-on-off-test.sh}     |  0
 2 files changed, 12 insertions(+), 2 deletions(-)
 rename tools/testing/selftests/cpu-hotplug/{on-off-test.sh => cpu-on-off-test.sh} (100%)

diff --git a/tools/testing/selftests/cpu-hotplug/Makefile b/tools/testing/selftests/cpu-hotplug/Makefile
index e9c28d8..c9e15ee 100644
--- a/tools/testing/selftests/cpu-hotplug/Makefile
+++ b/tools/testing/selftests/cpu-hotplug/Makefile
@@ -1,9 +1,19 @@
+TEST_STR=/bin/bash ./cpu-on-off-test.sh || echo cpu-hotplug selftests: [FAIL]
+
 all:
 
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./cpu-on-off-test.sh $(INSTALL_KSFT_PATH)/cpu-on-off-test.sh
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
 run_tests:
-	@/bin/bash ./on-off-test.sh || echo "cpu-hotplug selftests: [FAIL]"
+	@$(TEST_STR)
 
 run_full_test:
-	@/bin/bash ./on-off-test.sh -a || echo "cpu-hotplug selftests: [FAIL]"
+	@/bin/bash ./cpu-on-off-test.sh -a || echo "cpu-hotplug selftests: [FAIL]"
 
 clean:
diff --git a/tools/testing/selftests/cpu-hotplug/on-off-test.sh b/tools/testing/selftests/cpu-hotplug/cpu-on-off-test.sh
similarity index 100%
rename from tools/testing/selftests/cpu-hotplug/on-off-test.sh
rename to tools/testing/selftests/cpu-hotplug/cpu-on-off-test.sh
-- 
2.1.0

[PATCH v3 03/20] selftests/efivarfs: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/efivarfs/Makefile | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/efivarfs/Makefile b/tools/testing/selftests/efivarfs/Makefile
index 29e8c6b..aaf404b 100644
--- a/tools/testing/selftests/efivarfs/Makefile
+++ b/tools/testing/selftests/efivarfs/Makefile
@@ -3,10 +3,22 @@ CFLAGS = -Wall
 
 test_objs = open-unlink create-read
 
-all: $(test_objs)
+TEST_STR = /bin/bash ./efivarfs.sh || echo efivarfs selftests: [FAIL]
+
+all:
+	gcc open-unlink.c -o open-unlink
+	gcc create-read.c -o create-read
+
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./efivarfs.sh $(test_objs) $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
 
 run_tests: all
-	@/bin/bash ./efivarfs.sh || echo "efivarfs selftests: [FAIL]"
+	@$(TEST_STR)
 
 clean:
 	rm -f $(test_objs)
-- 
2.1.0

[PATCH v3 04/20] selftests/firmware: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/firmware/Makefile | 43 +++++++++++++++++++------------
 1 file changed, 26 insertions(+), 17 deletions(-)

diff --git a/tools/testing/selftests/firmware/Makefile b/tools/testing/selftests/firmware/Makefile
index e23cce0..7ac1cf3 100644
--- a/tools/testing/selftests/firmware/Makefile
+++ b/tools/testing/selftests/firmware/Makefile
@@ -1,25 +1,34 @@
 # Makefile for firmware loading selftests
 
 # No binaries, but make sure arg-less "make" doesn't trigger "run_tests"
+
+__fw_filesystem:
+fw_filesystem  = if /bin/sh ./fw_filesystem.sh ; then
+fw_filesystem += echo fw_filesystem: ok;
+fw_filesystem += else echo fw_filesystem: [FAIL];
+fw_filesystem += fi
+
+__fw_userhelper:
+fw_userhelper  = if /bin/sh ./fw_userhelper.sh ; then
+fw_userhelper += echo fw_userhelper: ok;
+fw_userhelper += else
+fw_userhelper += echo fw_userhelper: [FAIL];
+fw_userhelper += fi
+
 all:
 
-fw_filesystem:
-	@if /bin/sh ./fw_filesystem.sh ; then \
-                echo "fw_filesystem: ok"; \
-        else \
-                echo "fw_filesystem: [FAIL]"; \
-                exit 1; \
-        fi
-
-fw_userhelper:
-	@if /bin/sh ./fw_userhelper.sh ; then \
-                echo "fw_userhelper: ok"; \
-        else \
-                echo "fw_userhelper: [FAIL]"; \
-                exit 1; \
-        fi
-
-run_tests: all fw_filesystem fw_userhelper
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./fw_filesystem.sh ./fw_userhelper.sh $(INSTALL_KSFT_PATH)
+	@echo "$(fw_filesystem)" >> $(KSELFTEST)
+	@echo "$(fw_userhelper)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
+run_tests:
+	@$(fw_filesystem)
+	@$(fw_userhelper)
 
 # Nothing to clean up.
 clean:
-- 
2.1.0

[PATCH v3 05/20] selftests/ftrace: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/ftrace/Makefile | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/ftrace/Makefile b/tools/testing/selftests/ftrace/Makefile
index 76cc9f1..7c7cf42 100644
--- a/tools/testing/selftests/ftrace/Makefile
+++ b/tools/testing/selftests/ftrace/Makefile
@@ -1,7 +1,16 @@
+TEST_STR = /bin/sh ./ftracetest || echo ftrace selftests: [FAIL]
+
 all:
 
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./ftracetest $(INSTALL_KSFT_PATH)
+	@cp -r test.d $(INSTALL_KSFT_PATH)
+	echo "$(TEST_STR)" >> $(KSELFTEST)
+endif
+
 run_tests:
-	@/bin/sh ./ftracetest || echo "ftrace selftests: [FAIL]"
+	@$(TEST_STR)
 
 clean:
 	rm -rf logs/*
-- 
2.1.0

Re: [PATCH v3 05/20] selftests/ftrace: add install target to enable test install

[Steven Rostedt]

On Wed, 24 Dec 2014 09:27:41 -0700
Shuah Khan <shuahkh@osg.samsung.com> wrote:

> Add a new make target to enable installing test. This target
> installs test in the kselftest install location and add to the
> kselftest script to run the test. Install target can be run
> only from top level kernel source directory.
> 
> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
> ---
>  tools/testing/selftests/ftrace/Makefile | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/testing/selftests/ftrace/Makefile b/tools/testing/selftests/ftrace/Makefile
> index 76cc9f1..7c7cf42 100644
> --- a/tools/testing/selftests/ftrace/Makefile
> +++ b/tools/testing/selftests/ftrace/Makefile
> @@ -1,7 +1,16 @@
> +TEST_STR = /bin/sh ./ftracetest || echo ftrace selftests: [FAIL]

Is it ok that this removes the quotes around the echo string? I don't
see anything wrong about it, but I don't know if there's a shell out
there that will fail due to it.

Other than than,

Acked-by: Steven Rostedt <rostedt@goodmis.org>

-- Steve


> +
>  all:
>  
> +install:
> +ifdef INSTALL_KSFT_PATH
> +	install ./ftracetest $(INSTALL_KSFT_PATH)
> +	@cp -r test.d $(INSTALL_KSFT_PATH)
> +	echo "$(TEST_STR)" >> $(KSELFTEST)
> +endif
> +
>  run_tests:
> -	@/bin/sh ./ftracetest || echo "ftrace selftests: [FAIL]"
> +	@$(TEST_STR)
>  
>  clean:
>  	rm -rf logs/*

Re: Re: [PATCH v3 05/20] selftests/ftrace: add install target to enable test install

[Masami Hiramatsu]

(2015/01/03 0:45), Steven Rostedt wrote:
> On Wed, 24 Dec 2014 09:27:41 -0700
> Shuah Khan <shuahkh@osg.samsung.com> wrote:
> 
>> Add a new make target to enable installing test. This target
>> installs test in the kselftest install location and add to the
>> kselftest script to run the test. Install target can be run
>> only from top level kernel source directory.
>>
>> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
>> ---
>>  tools/testing/selftests/ftrace/Makefile | 11 ++++++++++-
>>  1 file changed, 10 insertions(+), 1 deletion(-)
>>
>> diff --git a/tools/testing/selftests/ftrace/Makefile b/tools/testing/selftests/ftrace/Makefile
>> index 76cc9f1..7c7cf42 100644
>> --- a/tools/testing/selftests/ftrace/Makefile
>> +++ b/tools/testing/selftests/ftrace/Makefile
>> @@ -1,7 +1,16 @@
>> +TEST_STR = /bin/sh ./ftracetest || echo ftrace selftests: [FAIL]
> 
> Is it ok that this removes the quotes around the echo string? I don't
> see anything wrong about it, but I don't know if there's a shell out
> there that will fail due to it.

Agreed, if possible, we'd better add quotes around the echo string
as you did in 2/20 in this series.

Anyway, you can also add my ack to this patch.

Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>

Thank you!

> 
> Other than than,
> 
> Acked-by: Steven Rostedt <rostedt@goodmis.org>
> 
> -- Steve
> 
> 
>> +
>>  all:
>>  
>> +install:
>> +ifdef INSTALL_KSFT_PATH
>> +	install ./ftracetest $(INSTALL_KSFT_PATH)
>> +	@cp -r test.d $(INSTALL_KSFT_PATH)
>> +	echo "$(TEST_STR)" >> $(KSELFTEST)
>> +endif
>> +
>>  run_tests:
>> -	@/bin/sh ./ftracetest || echo "ftrace selftests: [FAIL]"
>> +	@$(TEST_STR)
>>  
>>  clean:
>>  	rm -rf logs/*
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Re: [PATCH v3 05/20] selftests/ftrace: add install target to enable test install

[Masami Hiramatsu]

(2015/01/03 0:45), Steven Rostedt wrote:
> On Wed, 24 Dec 2014 09:27:41 -0700
> Shuah Khan <shuahkh@osg.samsung.com> wrote:
> 
>> Add a new make target to enable installing test. This target
>> installs test in the kselftest install location and add to the
>> kselftest script to run the test. Install target can be run
>> only from top level kernel source directory.
>>
>> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
>> ---
>>  tools/testing/selftests/ftrace/Makefile | 11 ++++++++++-
>>  1 file changed, 10 insertions(+), 1 deletion(-)
>>
>> diff --git a/tools/testing/selftests/ftrace/Makefile b/tools/testing/selftests/ftrace/Makefile
>> index 76cc9f1..7c7cf42 100644
>> --- a/tools/testing/selftests/ftrace/Makefile
>> +++ b/tools/testing/selftests/ftrace/Makefile
>> @@ -1,7 +1,16 @@
>> +TEST_STR = /bin/sh ./ftracetest || echo ftrace selftests: [FAIL]
> 
> Is it ok that this removes the quotes around the echo string? I don't
> see anything wrong about it, but I don't know if there's a shell out
> there that will fail due to it.

Agreed, if possible, we'd better add quotes around the echo string
as you did in 2/20 in this series.

Anyway, you can also add my ack to this patch.

Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>

Thank you!

> 
> Other than than,
> 
> Acked-by: Steven Rostedt <rostedt@goodmis.org>
> 
> -- Steve
> 
> 
>> +
>>  all:
>>  
>> +install:
>> +ifdef INSTALL_KSFT_PATH
>> +	install ./ftracetest $(INSTALL_KSFT_PATH)
>> +	@cp -r test.d $(INSTALL_KSFT_PATH)
>> +	echo "$(TEST_STR)" >> $(KSELFTEST)
>> +endif
>> +
>>  run_tests:
>> -	@/bin/sh ./ftracetest || echo "ftrace selftests: [FAIL]"
>> +	@$(TEST_STR)
>>  
>>  clean:
>>  	rm -rf logs/*
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [PATCH v3 05/20] selftests/ftrace: add install target to enable test install

[Shuah Khan]

On 01/02/2015 08:45 AM, Steven Rostedt wrote:
> On Wed, 24 Dec 2014 09:27:41 -0700
> Shuah Khan <shuahkh@osg.samsung.com> wrote:
> 
>> Add a new make target to enable installing test. This target
>> installs test in the kselftest install location and add to the
>> kselftest script to run the test. Install target can be run
>> only from top level kernel source directory.
>>
>> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
>> ---
>>  tools/testing/selftests/ftrace/Makefile | 11 ++++++++++-
>>  1 file changed, 10 insertions(+), 1 deletion(-)
>>
>> diff --git a/tools/testing/selftests/ftrace/Makefile b/tools/testing/selftests/ftrace/Makefile
>> index 76cc9f1..7c7cf42 100644
>> --- a/tools/testing/selftests/ftrace/Makefile
>> +++ b/tools/testing/selftests/ftrace/Makefile
>> @@ -1,7 +1,16 @@
>> +TEST_STR = /bin/sh ./ftracetest || echo ftrace selftests: [FAIL]
> 
> Is it ok that this removes the quotes around the echo string? I don't
> see anything wrong about it, but I don't know if there's a shell out
> there that will fail due to it.

Right. both sh and bash are fine without the quotes. In this case there
are no variables to interpret, so quotes don't do anything. I might as
well play it safe. I will have to fix a few other tests to address this
comment. Will generate v4s for a few tests in this series.

Thanks,
-- Shuah

> 
> Other than than,
> 
> Acked-by: Steven Rostedt <rostedt@goodmis.org>
> 
> -- Steve
> 
> 
>> +
>>  all:
>>  
>> +install:
>> +ifdef INSTALL_KSFT_PATH
>> +	install ./ftracetest $(INSTALL_KSFT_PATH)
>> +	@cp -r test.d $(INSTALL_KSFT_PATH)
>> +	echo "$(TEST_STR)" >> $(KSELFTEST)
>> +endif
>> +
>>  run_tests:
>> -	@/bin/sh ./ftracetest || echo "ftrace selftests: [FAIL]"
>> +	@$(TEST_STR)
>>  
>>  clean:
>>  	rm -rf logs/*
> 


-- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

[PATCH v3 06/20] selftests/ipc: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/ipc/Makefile | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/ipc/Makefile b/tools/testing/selftests/ipc/Makefile
index 74bbefd..b43e8e8 100644
--- a/tools/testing/selftests/ipc/Makefile
+++ b/tools/testing/selftests/ipc/Makefile
@@ -11,6 +11,8 @@ endif
 
 CFLAGS += -I../../../../usr/include/
 
+TEST_STR = ./msgque_test || echo ipc msgque test: [FAIL]
+
 all:
 ifeq ($(ARCH),x86)
 	gcc $(CFLAGS) msgque.c -o msgque_test
@@ -18,8 +20,23 @@ else
 	echo "Not an x86 target, can't build msgque selftest"
 endif
 
+install:
+ifdef INSTALL_KSFT_PATH
+ifeq ($(ARCH),x86)
+	make all
+	install ./msgque_test $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Not an x86 target, unable to install ipc msgque selftests
+endif
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
 run_tests: all
-	./msgque_test
+ifeq ($(ARCH),x86)
+	@$(TEST_STR)
+endif
 
 clean:
 	rm -fr ./msgque_test
-- 
2.1.0

[PATCH v3 07/20] selftests/kcmp: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/kcmp/Makefile | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/kcmp/Makefile b/tools/testing/selftests/kcmp/Makefile
index ff0eefd..8d640d0 100644
--- a/tools/testing/selftests/kcmp/Makefile
+++ b/tools/testing/selftests/kcmp/Makefile
@@ -1,10 +1,21 @@
 CC := $(CROSS_COMPILE)$(CC)
 CFLAGS += -I../../../../usr/include/
 
+TEST_STR = ./kcmp_test || echo kcmp_test: [FAIL]
+
 all: kcmp_test
 
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./kcmp_test $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+	@echo rm -f kcmp-test-file >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
 run_tests: all
-	@./kcmp_test || echo "kcmp_test: [FAIL]"
+	@$(TEST_STR)
 
 clean:
 	$(RM) kcmp_test kcmp-test-file
-- 
2.1.0

[PATCH v3 08/20] selftests/memfd: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/memfd/Makefile | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/memfd/Makefile b/tools/testing/selftests/memfd/Makefile
index b80cd10..5f70b33 100644
--- a/tools/testing/selftests/memfd/Makefile
+++ b/tools/testing/selftests/memfd/Makefile
@@ -2,19 +2,30 @@ CFLAGS += -D_FILE_OFFSET_BITS=64
 CFLAGS += -I../../../../include/uapi/
 CFLAGS += -I../../../../include/
 
+INSTALL_PROGS = memfd_test fuse_test run_fuse_test.sh
+MEMFD_TEST_STR = ./memfd_test || echo memfd_test: [FAIL]
+FUSE_TEST_STR = ./run_fuse_test.sh || echo fuse_test: [FAIL]
+
 all:
 	gcc $(CFLAGS) memfd_test.c -o memfd_test
 
+install:
+ifdef INSTALL_KSFT_PATH
+	install $(INSTALL_PROGS) $(INSTALL_KSFT_PATH)
+	@echo "$(MEMFD_TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
 run_tests: all
-	gcc $(CFLAGS) memfd_test.c -o memfd_test
-	@./memfd_test || echo "memfd_test: [FAIL]"
+	@$(MEMFD_TEST_STR)
 
 build_fuse:
 	gcc $(CFLAGS) fuse_mnt.c `pkg-config fuse --cflags --libs` -o fuse_mnt
 	gcc $(CFLAGS) fuse_test.c -o fuse_test
 
 run_fuse: build_fuse
-	@./run_fuse_test.sh || echo "fuse_test: [FAIL]"
+	@$(FUSE_TEST_STR)
 
 clean:
 	$(RM) memfd_test fuse_test
-- 
2.1.0

[PATCH v3 09/20] selftests/memory-hotplug: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/memory-hotplug/Makefile            | 14 ++++++++++++--
 .../memory-hotplug/{on-off-test.sh => mem-on-off-test.sh}  |  0
 2 files changed, 12 insertions(+), 2 deletions(-)
 rename tools/testing/selftests/memory-hotplug/{on-off-test.sh => mem-on-off-test.sh} (100%)

diff --git a/tools/testing/selftests/memory-hotplug/Makefile b/tools/testing/selftests/memory-hotplug/Makefile
index d46b8d4..561ee2b 100644
--- a/tools/testing/selftests/memory-hotplug/Makefile
+++ b/tools/testing/selftests/memory-hotplug/Makefile
@@ -1,9 +1,19 @@
+TEST_STR=/bin/bash ./mem-on-off-test.sh -r 2 || echo memory-hotplug selftests: [FAIL]
+
 all:
 
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./mem-on-off-test.sh $(INSTALL_KSFT_PATH)/mem-on-off-test.sh
+	@echo "$(TEST_STR)" >> $(KSELFTEST) >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
 run_tests:
-	@/bin/bash ./on-off-test.sh -r 2 || echo "memory-hotplug selftests: [FAIL]"
+	@$(TEST_STR)
 
 run_full_test:
-	@/bin/bash ./on-off-test.sh || echo "memory-hotplug selftests: [FAIL]"
+	@/bin/bash ./mem-on-off-test.sh || echo "memory-hotplug selftests: [FAIL]"
 
 clean:
diff --git a/tools/testing/selftests/memory-hotplug/on-off-test.sh b/tools/testing/selftests/memory-hotplug/mem-on-off-test.sh
similarity index 100%
rename from tools/testing/selftests/memory-hotplug/on-off-test.sh
rename to tools/testing/selftests/memory-hotplug/mem-on-off-test.sh
-- 
2.1.0

[PATCH v3 10/20] selftests/mount: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/mount/Makefile | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/mount/Makefile b/tools/testing/selftests/mount/Makefile
index 337d853..607de4c 100644
--- a/tools/testing/selftests/mount/Makefile
+++ b/tools/testing/selftests/mount/Makefile
@@ -1,5 +1,7 @@
 # Makefile for mount selftests.
 
+TEST_STR = if [ -f /proc/self/uid_map ] ; then ./unprivileged-remount-test ; fi
+
 all: unprivileged-remount-test
 
 unprivileged-remount-test: unprivileged-remount-test.c
@@ -7,7 +9,15 @@ unprivileged-remount-test: unprivileged-remount-test.c
 
 # Allow specific tests to be selected.
 test_unprivileged_remount: unprivileged-remount-test
-	@if [ -f /proc/self/uid_map ] ; then ./unprivileged-remount-test ; fi
+	@$(TEST_STR)
+
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./unprivileged-remount-test $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
 
 run_tests: all test_unprivileged_remount
 
-- 
2.1.0

[PATCH v3 11/20] selftests/mqueue: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/mqueue/Makefile | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/mqueue/Makefile b/tools/testing/selftests/mqueue/Makefile
index 8056e2e..27757b9 100644
--- a/tools/testing/selftests/mqueue/Makefile
+++ b/tools/testing/selftests/mqueue/Makefile
@@ -1,10 +1,22 @@
+MQ_OPEN_TEST_STR = ./mq_open_tests /test1 || echo mq_open_tests: [FAIL]
+MQ_PERF_TEST_STR = ./mq_perf_tests || echo mq_perf_tests: [FAIL]
+
 all:
 	gcc -O2 mq_open_tests.c -o mq_open_tests -lrt
 	gcc -O2 -o mq_perf_tests mq_perf_tests.c -lrt -lpthread -lpopt
 
-run_tests:
-	@./mq_open_tests /test1 || echo "mq_open_tests: [FAIL]"
-	@./mq_perf_tests || echo "mq_perf_tests: [FAIL]"
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./mq_open_tests ./mq_perf_tests $(INSTALL_KSFT_PATH)
+	@echo "$(MQ_OPEN_TEST_STR)" >> $(KSELFTEST)
+	@echo "$(MQ_PERF_TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
+run_tests: all
+	@$(MQ_OPEN_TEST_STR)
+	@$(MQ_PERF_TEST_STR)
 
 clean:
 	rm -f mq_open_tests mq_perf_tests
-- 
2.1.0

[PATCH v3 12/20] selftests/net: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/net/Makefile | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index 62f22cc..2ffc96f 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -6,14 +6,28 @@ CFLAGS = -Wall -O2 -g
 CFLAGS += -I../../../../usr/include/
 
 NET_PROGS = socket psock_fanout psock_tpacket
+INSTALL_PROGS = run_netsocktests run_afpackettests test_bpf.sh $(NET_PROGS)
+NETSOCK_TEST_STR = /bin/sh ./run_netsocktests || echo sockettests: [FAIL]
+AFPKT_TEST_STR = /bin/sh ./run_afpackettests || echo afpackettests: [FAIL]
+BFP_TEST_STR = ./test_bpf.sh
 
 all: $(NET_PROGS)
 %: %.c
 	$(CC) $(CFLAGS) -o $@ $^
 
+install:
+ifdef INSTALL_KSFT_PATH
+	install $(INSTALL_PROGS) $(INSTALL_KSFT_PATH)
+	@echo "$(NETSOCK_TEST_STR)" >> $(KSELFTEST)
+	@echo "$(AFPKT_TEST_STR)" >> $(KSELFTEST)
+	@echo "$(BFP_TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
 run_tests: all
-	@/bin/sh ./run_netsocktests || echo "sockettests: [FAIL]"
-	@/bin/sh ./run_afpackettests || echo "afpackettests: [FAIL]"
-	./test_bpf.sh
+	@$(NETSOCK_TEST_STR)
+	@$(AFPKT_TEST_STR)
+	@$(BFP_TEST_STR)
 clean:
 	$(RM) $(NET_PROGS)
-- 
2.1.0

[PATCH v3 13/20] selftests/ptrace: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/ptrace/Makefile | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/ptrace/Makefile b/tools/testing/selftests/ptrace/Makefile
index 47ae2d3..f238212 100644
--- a/tools/testing/selftests/ptrace/Makefile
+++ b/tools/testing/selftests/ptrace/Makefile
@@ -1,10 +1,20 @@
 CFLAGS += -iquote../../../../include/uapi -Wall
-peeksiginfo: peeksiginfo.c
 
-all: peeksiginfo
+TEST_STR = ./peeksiginfo || echo peeksiginfo selftests: [FAIL]
+
+all:
+	gcc peeksiginfo.c -o peeksiginfo
+
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./peeksiginfo $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
 
 clean:
 	rm -f peeksiginfo
 
 run_tests: all
-	@./peeksiginfo || echo "peeksiginfo selftests: [FAIL]"
+	@$(TEST_STR)
-- 
2.1.0

[PATCH v3 14/20] selftests/size: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/size/Makefile | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/size/Makefile b/tools/testing/selftests/size/Makefile
index 04dc25e..bb7113b 100644
--- a/tools/testing/selftests/size/Makefile
+++ b/tools/testing/selftests/size/Makefile
@@ -1,12 +1,22 @@
 CC = $(CROSS_COMPILE)gcc
 
+TEST_STR = ./get_size || echo get_size selftests: [FAIL]
+
 all: get_size
 
 get_size: get_size.c
 	$(CC) -static -ffreestanding -nostartfiles -s $< -o $@
 
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./get_size $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
 run_tests: all
-	./get_size
+	@$(TEST_STR)
 
 clean:
 	$(RM) get_size
-- 
2.1.0

WIP alternative - was Re: [PATCH v3 14/20] selftests/size: add install target to enable test install

[Tim Bird]

On 12/24/2014 08:27 AM, Shuah Khan wrote:
> Add a new make target to enable installing test. This target
> installs test in the kselftest install location and add to the
> kselftest script to run the test. Install target can be run
> only from top level kernel source directory.
> 
> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
> ---
>  tools/testing/selftests/size/Makefile | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/testing/selftests/size/Makefile b/tools/testing/selftests/size/Makefile
> index 04dc25e..bb7113b 100644
> --- a/tools/testing/selftests/size/Makefile
> +++ b/tools/testing/selftests/size/Makefile
> @@ -1,12 +1,22 @@
>  CC = $(CROSS_COMPILE)gcc
>  
> +TEST_STR = ./get_size || echo get_size selftests: [FAIL]
> +
>  all: get_size
>  
>  get_size: get_size.c
>  	$(CC) -static -ffreestanding -nostartfiles -s $< -o $@
>  
> +install:
> +ifdef INSTALL_KSFT_PATH
> +	install ./get_size $(INSTALL_KSFT_PATH)
> +	@echo "$(TEST_STR)" >> $(KSELFTEST)
> +else
> +	@echo Run make kselftest_install in top level source directory
> +endif
> +
>  run_tests: all
> -	./get_size
> +	@$(TEST_STR)
>  
>  clean:
>  	$(RM) get_size
> 

The install phase is desperately needed for usage of kselftest in
cross-target situations (applicable to almost all embedded).  So this
is great stuff.

I worked a bit on isolating the install stuff to a makefile include file.
This allows simplifying some of the sub-level Makefiles a bit, and allowing
control of some of the install and run logic in less places.

This is still a work in progress, but before I got too far along, I wanted
to post it for people to provide feedback.  A couple of problems cropped
up that are worth discussing, IMHO.

1) I think it should be a requirement that each test has a single
"main" program to execute to run the tests.  If multiple tests are supported
or more flexibility is desired for additional arguments, or that sort of
thing, then that's fine, but the automated script builder should be able
to run just a single program or script to have things work.  This also
makes things more consistent.  In the case of the firmware test, I created
a single fw_both.sh script to do this, instead of having two separate
blocks in the kselftest.sh script.

2) I've added a CROSS_INSTALL variable, which can call an arbitrary program
to place files on the target system (rather than just calling 'install').
In my case, I'd use my own 'ttc cp' command, which I can extend as necessary
to put things on a remote machine.  This works for a single directory,
but things get dicier with sub-directory trees full of files (like
the ftrace test uses.)

If additional items need to be installed to the target, then maybe a setup
program should be used, rather than just copying files.

3) Some of the scripts were using /bin/bash to execute them, rather
than rely on the interpreter line in the script itself (and having
the script have executable privileges).  Is there a reason for this?
I modified a few scripts to be executable, and got rid of the
explicit execution with /bin/bash.

The following is just a start...  Let me know if this direction looks
OK, and I'll finish this up.  The main item to look at is
kselftest.include file.  Note that these patches are based on Shuah's
series - but if you want to use these ideas I can rebase them onto
mainline, and break them out per test sub-level like Shuah did.


Let me know what you think.
From 14164fd3117c97799a050f8cf791dedc93aa5e82 Mon Sep 17 00:00:00 2001
From: Tim Bird <tim.bird@sonymobile.com>
Date: Wed, 31 Dec 2014 18:04:08 -0800
Subject: [PATCH] Switch to using an include file for common kselftest_install
 actions

---
 tools/testing/selftests/cpu-hotplug/Makefile       | 15 +++-------
 .../selftests/cpu-hotplug/cpu-on-off-test.sh       |  0
 tools/testing/selftests/efivarfs/Makefile          | 21 ++++----------
 tools/testing/selftests/efivarfs/efivarfs.sh       |  0
 tools/testing/selftests/firmware/Makefile          | 32 ++++------------------
 tools/testing/selftests/firmware/fw_both.sh        | 13 +++++++++
 tools/testing/selftests/kselftest.include          | 26 ++++++++++++++++++
 tools/testing/selftests/size/Makefile              | 19 +++----------
 9 files changed, 60 insertions(+), 69 deletions(-)
 mode change 100644 => 100755 tools/testing/selftests/cpu-hotplug/cpu-on-off-test.sh
 mode change 100644 => 100755 tools/testing/selftests/efivarfs/efivarfs.sh
 create mode 100755 tools/testing/selftests/firmware/fw_both.sh
 create mode 100644 tools/testing/selftests/kselftest.include

diff --git a/tools/testing/selftests/cpu-hotplug/Makefile b/tools/testing/selftests/cpu-hotplug/Makefile
index c9e15ee..d2e540d 100644
--- a/tools/testing/selftests/cpu-hotplug/Makefile
+++ b/tools/testing/selftests/cpu-hotplug/Makefile
@@ -1,18 +1,11 @@
-TEST_STR=/bin/bash ./cpu-on-off-test.sh || echo cpu-hotplug selftests: [FAIL]
+TEST_TITLE = cpu-hotplug
+TEST_MAIN_PROG = cpu-on-off-test.sh
 
 all:
 
-install:
-ifdef INSTALL_KSFT_PATH
-	install ./cpu-on-off-test.sh $(INSTALL_KSFT_PATH)/cpu-on-off-test.sh
-	@echo "$(TEST_STR)" >> $(KSELFTEST)
-else
-	@echo Run make kselftest_install in top level source directory
-endif
-
-run_tests:
-	@$(TEST_STR)
+include ../kselftest.include
 
+# use -a to get all tests
 run_full_test:
 	@/bin/bash ./cpu-on-off-test.sh -a || echo "cpu-hotplug selftests: [FAIL]"
 
diff --git a/tools/testing/selftests/cpu-hotplug/cpu-on-off-test.sh b/tools/testing/selftests/cpu-hotplug/cpu-on-off-test.sh
old mode 100644
new mode 100755
diff --git a/tools/testing/selftests/efivarfs/Makefile b/tools/testing/selftests/efivarfs/Makefile
index aaf404b..8d48e9c 100644
--- a/tools/testing/selftests/efivarfs/Makefile
+++ b/tools/testing/selftests/efivarfs/Makefile
@@ -1,24 +1,15 @@
-CC = $(CROSS_COMPILE)gcc
-CFLAGS = -Wall
+TEST_TITLE = efivarfs
+TEST_MAIN_PROG = efivarfs.sh
 
-test_objs = open-unlink create-read
+CFLAGS = -Wall
 
-TEST_STR = /bin/bash ./efivarfs.sh || echo efivarfs selftests: [FAIL]
+TEST_PROGS = open-unlink create-read
 
 all:
 	gcc open-unlink.c -o open-unlink
 	gcc create-read.c -o create-read
 
-install:
-ifdef INSTALL_KSFT_PATH
-	install ./efivarfs.sh $(test_objs) $(INSTALL_KSFT_PATH)
-	@echo "$(TEST_STR)" >> $(KSELFTEST)
-else
-	@echo Run make kselftest_install in top level source directory
-endif
-
-run_tests: all
-	@$(TEST_STR)
+include ../kselftest.include
 
 clean:
-	rm -f $(test_objs)
+	rm -f $(TEST_PROGS)
diff --git a/tools/testing/selftests/efivarfs/efivarfs.sh b/tools/testing/selftests/efivarfs/efivarfs.sh
old mode 100644
new mode 100755
diff --git a/tools/testing/selftests/firmware/Makefile b/tools/testing/selftests/firmware/Makefile
index 7ac1cf3..b576308 100644
--- a/tools/testing/selftests/firmware/Makefile
+++ b/tools/testing/selftests/firmware/Makefile
@@ -1,36 +1,14 @@
 # Makefile for firmware loading selftests
+TEST_MAIN_PROG = fw_both.sh
+TEST_TITLE = firmware
+TEST_FILES = ./fw_filesystem.sh ./fw_userhelper.sh
 
 # No binaries, but make sure arg-less "make" doesn't trigger "run_tests"
-
-__fw_filesystem:
-fw_filesystem  = if /bin/sh ./fw_filesystem.sh ; then
-fw_filesystem += echo fw_filesystem: ok;
-fw_filesystem += else echo fw_filesystem: [FAIL];
-fw_filesystem += fi
-
-__fw_userhelper:
-fw_userhelper  = if /bin/sh ./fw_userhelper.sh ; then
-fw_userhelper += echo fw_userhelper: ok;
-fw_userhelper += else
-fw_userhelper += echo fw_userhelper: [FAIL];
-fw_userhelper += fi
-
 all:
 
-install:
-ifdef INSTALL_KSFT_PATH
-	install ./fw_filesystem.sh ./fw_userhelper.sh $(INSTALL_KSFT_PATH)
-	@echo "$(fw_filesystem)" >> $(KSELFTEST)
-	@echo "$(fw_userhelper)" >> $(KSELFTEST)
-else
-	@echo Run make kselftest_install in top level source directory
-endif
-
-run_tests:
-	@$(fw_filesystem)
-	@$(fw_userhelper)
+include ../kselftest.include
 
 # Nothing to clean up.
 clean:
 
-.PHONY: all clean run_tests fw_filesystem fw_userhelper
+.PHONY: all clean run_tests
diff --git a/tools/testing/selftests/firmware/fw_both.sh b/tools/testing/selftests/firmware/fw_both.sh
new file mode 100755
index 0000000..fb7fa8d
--- /dev/null
+++ b/tools/testing/selftests/firmware/fw_both.sh
@@ -0,0 +1,13 @@
+#!/bin/sh
+
+if /bin/sh ./fw_filesystem.sh ; then
+	echo "fw_filesystem: ok";
+else
+	echo "fw_filesystem: [FAIL]";
+fi
+
+if /bin/sh ./fw_userhelper.sh ; then
+	echo "fw_userhelper: ok";
+else
+	echo "fw_userhelper: [FAIL]";
+fi
diff --git a/tools/testing/selftests/kselftest.include b/tools/testing/selftests/kselftest.include
new file mode 100644
index 0000000..7020a82
--- /dev/null
+++ b/tools/testing/selftests/kselftest.include
@@ -0,0 +1,26 @@
+# this make include script expects the following variables to be set:
+#  TEST_MAIN_PROG - a single program to invoke to run the test
+#  TEST_TITLE - the title of the test
+#  TEST_FILES - other files that should be copied to the testing directory
+#  INSTALL_KSFT_PATH - where to put test programs
+#  KSELFTEST - the name of the generated script (which calls the sub-tests)
+#  the 'all' target, which makes sure everything is built
+
+# this file defines the 'install' and 'run_tests' targets
+
+CC = $(CROSS_COMPILE)gcc
+INSTALL = $(CROSS_INSTALL)install
+export INSTALL
+
+TEST_STR = ./$(TEST_MAIN_PROG) || echo $(TEST_TITLE) selftests: [FAIL]
+
+install:
+ifdef INSTALL_KSFT_PATH
+	$(INSTALL) ./$(TEST_MAIN_PROG) $(TEST_FILES) $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in the top level source directory
+endif
+
+run_tests: all
+	@$(TEST_STR)
diff --git a/tools/testing/selftests/size/Makefile b/tools/testing/selftests/size/Makefile
index bb7113b..c88819a 100644
--- a/tools/testing/selftests/size/Makefile
+++ b/tools/testing/selftests/size/Makefile
@@ -1,22 +1,11 @@
-CC = $(CROSS_COMPILE)gcc
+TEST_MAIN_PROG = get_size
 
-TEST_STR = ./get_size || echo get_size selftests: [FAIL]
+all: $(TEST_MAIN_PROG)
 
-all: get_size
+include ../kselftest.include
 
 get_size: get_size.c
 	$(CC) -static -ffreestanding -nostartfiles -s $< -o $@
 
-install:
-ifdef INSTALL_KSFT_PATH
-	install ./get_size $(INSTALL_KSFT_PATH)
-	@echo "$(TEST_STR)" >> $(KSELFTEST)
-else
-	@echo Run make kselftest_install in top level source directory
-endif
-
-run_tests: all
-	@$(TEST_STR)
-
 clean:
-	$(RM) get_size
+	$(RM) $(TEST_MAIN_PROG)
-- 
1.8.2.2

Re: WIP alternative - was Re: [PATCH v3 14/20] selftests/size: add install target to enable test install

[Shuah Khan]

On 12/31/2014 07:31 PM, Tim Bird wrote:
> On 12/24/2014 08:27 AM, Shuah Khan wrote:
>> Add a new make target to enable installing test. This target
>> installs test in the kselftest install location and add to the
>> kselftest script to run the test. Install target can be run
>> only from top level kernel source directory.
>>
>> Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
>> ---
>>  tools/testing/selftests/size/Makefile | 12 +++++++++++-
>>  1 file changed, 11 insertions(+), 1 deletion(-)
>>
>> diff --git a/tools/testing/selftests/size/Makefile b/tools/testing/selftests/size/Makefile
>> index 04dc25e..bb7113b 100644
>> --- a/tools/testing/selftests/size/Makefile
>> +++ b/tools/testing/selftests/size/Makefile
>> @@ -1,12 +1,22 @@
>>  CC = $(CROSS_COMPILE)gcc
>>  
>> +TEST_STR = ./get_size || echo get_size selftests: [FAIL]
>> +
>>  all: get_size
>>  
>>  get_size: get_size.c
>>  	$(CC) -static -ffreestanding -nostartfiles -s $< -o $@
>>  
>> +install:
>> +ifdef INSTALL_KSFT_PATH
>> +	install ./get_size $(INSTALL_KSFT_PATH)
>> +	@echo "$(TEST_STR)" >> $(KSELFTEST)
>> +else
>> +	@echo Run make kselftest_install in top level source directory
>> +endif
>> +
>>  run_tests: all
>> -	./get_size
>> +	@$(TEST_STR)
>>  
>>  clean:
>>  	$(RM) get_size
>>
> 
> The install phase is desperately needed for usage of kselftest in
> cross-target situations (applicable to almost all embedded).  So this
> is great stuff.

Thanks.

> 
> I worked a bit on isolating the install stuff to a makefile include file.
> This allows simplifying some of the sub-level Makefiles a bit, and allowing
> control of some of the install and run logic in less places.
> 
> This is still a work in progress, but before I got too far along, I wanted
> to post it for people to provide feedback.  A couple of problems cropped
> up that are worth discussing, IMHO.
> 
> 1) I think it should be a requirement that each test has a single
> "main" program to execute to run the tests.  If multiple tests are supported
> or more flexibility is desired for additional arguments, or that sort of
> thing, then that's fine, but the automated script builder should be able
> to run just a single program or script to have things work.  This also
> makes things more consistent.  In the case of the firmware test, I created
> a single fw_both.sh script to do this, instead of having two separate
> blocks in the kselftest.sh script.

It is a good goal for individual tests to use a main program to run
tests, even though, I would not make it a requirement. I would like to
leave that decision up to the individual test writer.

> 
> 2) I've added a CROSS_INSTALL variable, which can call an arbitrary program
> to place files on the target system (rather than just calling 'install').
> In my case, I'd use my own 'ttc cp' command, which I can extend as necessary
> to put things on a remote machine.  This works for a single directory,
> but things get dicier with sub-directory trees full of files (like
> the ftrace test uses.)
> 
> If additional items need to be installed to the target, then maybe a setup
> program should be used, rather than just copying files.
> 
> 3) Some of the scripts were using /bin/bash to execute them, rather
> than rely on the interpreter line in the script itself (and having
> the script have executable privileges).  Is there a reason for this?
> I modified a few scripts to be executable, and got rid of the
> explicit execution with /bin/bash.

Probably no reason other than the choice made by the test writer.
It could be cleaned up and made consistent, however, I would see
this as a separate enhancement type work that could be done on its
own and not include it in the install work.

> 
> The following is just a start...  Let me know if this direction looks
> OK, and I'll finish this up.  The main item to look at is
> kselftest.include file.  Note that these patches are based on Shuah's
> series - but if you want to use these ideas I can rebase them onto
> mainline, and break them out per test sub-level like Shuah did.

One of the reasons I picked install target approach is to enable the
feature by extending the existing run_tests support. This way we will
have the feature available quickly. Once that is supported, we can work
on evolving to a generic approach to use the include file approach, as
the changes you are proposing are based on the the series I sent out,
and makes improvements to it.

kselftest.include file approach could work for several tests and tests
that can't use the generic could add their own install support.

I propose evolving to a generic kselftest.include as the second step in
evolving the install feature. Can I count on you do the work and update
the tests to use kselftest.include, CROSS_INSTALL variable support?

thanks,
-- Shuah


Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

Re: WIP alternative - was Re: [PATCH v3 14/20] selftests/size: add install target to enable test install

[Tim Bird]

On 01/05/2015 01:28 PM, Shuah Khan wrote:
> On 12/31/2014 07:31 PM, Tim Bird wrote:
...
>> The install phase is desperately needed for usage of kselftest in
>> cross-target situations (applicable to almost all embedded).  So this
>> is great stuff.
> 
> Thanks.
> 
>>
>> I worked a bit on isolating the install stuff to a makefile include file.
>> This allows simplifying some of the sub-level Makefiles a bit, and allowing
>> control of some of the install and run logic in less places.
>>
>> This is still a work in progress, but before I got too far along, I wanted
>> to post it for people to provide feedback.  A couple of problems cropped
>> up that are worth discussing, IMHO.
>>
>> 1) I think it should be a requirement that each test has a single
>> "main" program to execute to run the tests.  If multiple tests are supported
>> or more flexibility is desired for additional arguments, or that sort of
>> thing, then that's fine, but the automated script builder should be able
>> to run just a single program or script to have things work.  This also
>> makes things more consistent.  In the case of the firmware test, I created
>> a single fw_both.sh script to do this, instead of having two separate
>> blocks in the kselftest.sh script.
> 
> It is a good goal for individual tests to use a main program to run
> tests, even though, I would not make it a requirement. I would like to
> leave that decision up to the individual test writer.
>
OK.  It helps to have a single line when trying to isolate
RUN_TEST creation into the include file, but there may be other
ways to accomplish this.

>>
>> 2) I've added a CROSS_INSTALL variable, which can call an arbitrary program
>> to place files on the target system (rather than just calling 'install').
>> In my case, I'd use my own 'ttc cp' command, which I can extend as necessary
>> to put things on a remote machine.  This works for a single directory,
>> but things get dicier with sub-directory trees full of files (like
>> the ftrace test uses.)
>>
>> If additional items need to be installed to the target, then maybe a setup
>> program should be used, rather than just copying files.
>>
>> 3) Some of the scripts were using /bin/bash to execute them, rather
>> than rely on the interpreter line in the script itself (and having
>> the script have executable privileges).  Is there a reason for this?
>> I modified a few scripts to be executable, and got rid of the
>> explicit execution with /bin/bash.
> 
> Probably no reason other than the choice made by the test writer.
> It could be cleaned up and made consistent, however, I would see
> this as a separate enhancement type work that could be done on its
> own and not include it in the install work.

OK - this was also something that simplified the creation
of the RUN_TEST variable in the isolated include file.
Also, having the interpreter explicit in the invocation line
in the Makefile as well as in the script itself is a bit redundant.
>>
>> The following is just a start...  Let me know if this direction looks
>> OK, and I'll finish this up.  The main item to look at is
>> kselftest.include file.  Note that these patches are based on Shuah's
>> series - but if you want to use these ideas I can rebase them onto
>> mainline, and break them out per test sub-level like Shuah did.
> 
> One of the reasons I picked install target approach is to enable the
> feature by extending the existing run_tests support. This way we will
> have the feature available quickly. Once that is supported, we can work
> on evolving to a generic approach to use the include file approach, as
> the changes you are proposing are based on the the series I sent out,
> and makes improvements to it.
> 
> kselftest.include file approach could work for several tests and tests
> that can't use the generic could add their own install support.
> 
> I propose evolving to a generic kselftest.include as the second step in
> evolving the install feature. Can I count on you do the work and update
> the tests to use kselftest.include, CROSS_INSTALL variable support?

Yes.  I'd be happy to evolve it through phases to support the include
file and cross-target install feature.

Is there anything I can help with in the mean time?  Some of the tests
require a directory tree of files rather than just a few top-level files
(e.g. ftrace).

I was thinking about doing some work to tar-ify the needed directories of
data files during build, and untar it in the execution area during the
install step.  Do you want me to propose something there?
 -- Tim

Re: WIP alternative - was Re: [PATCH v3 14/20] selftests/size: add install target to enable test install

[Shuah Khan]

On 01/05/2015 02:56 PM, Tim Bird wrote:
> On 01/05/2015 01:28 PM, Shuah Khan wrote:
>> On 12/31/2014 07:31 PM, Tim Bird wrote:
> ...
>>> The install phase is desperately needed for usage of kselftest in
>>> cross-target situations (applicable to almost all embedded).  So this
>>> is great stuff.
>>
>> Thanks.
>>
>>>
>>> I worked a bit on isolating the install stuff to a makefile include file.
>>> This allows simplifying some of the sub-level Makefiles a bit, and allowing
>>> control of some of the install and run logic in less places.
>>>
>>> This is still a work in progress, but before I got too far along, I wanted
>>> to post it for people to provide feedback.  A couple of problems cropped
>>> up that are worth discussing, IMHO.
>>>
>>> 1) I think it should be a requirement that each test has a single
>>> "main" program to execute to run the tests.  If multiple tests are supported
>>> or more flexibility is desired for additional arguments, or that sort of
>>> thing, then that's fine, but the automated script builder should be able
>>> to run just a single program or script to have things work.  This also
>>> makes things more consistent.  In the case of the firmware test, I created
>>> a single fw_both.sh script to do this, instead of having two separate
>>> blocks in the kselftest.sh script.
>>
>> It is a good goal for individual tests to use a main program to run
>> tests, even though, I would not make it a requirement. I would like to
>> leave that decision up to the individual test writer.
>>
> OK.  It helps to have a single line when trying to isolate
> RUN_TEST creation into the include file, but there may be other
> ways to accomplish this.
> 
>>>
>>> 2) I've added a CROSS_INSTALL variable, which can call an arbitrary program
>>> to place files on the target system (rather than just calling 'install').
>>> In my case, I'd use my own 'ttc cp' command, which I can extend as necessary
>>> to put things on a remote machine.  This works for a single directory,
>>> but things get dicier with sub-directory trees full of files (like
>>> the ftrace test uses.)
>>>
>>> If additional items need to be installed to the target, then maybe a setup
>>> program should be used, rather than just copying files.
>>>
>>> 3) Some of the scripts were using /bin/bash to execute them, rather
>>> than rely on the interpreter line in the script itself (and having
>>> the script have executable privileges).  Is there a reason for this?
>>> I modified a few scripts to be executable, and got rid of the
>>> explicit execution with /bin/bash.
>>
>> Probably no reason other than the choice made by the test writer.
>> It could be cleaned up and made consistent, however, I would see
>> this as a separate enhancement type work that could be done on its
>> own and not include it in the install work.
> 
> OK - this was also something that simplified the creation
> of the RUN_TEST variable in the isolated include file.
> Also, having the interpreter explicit in the invocation line
> in the Makefile as well as in the script itself is a bit redundant.
>>>
>>> The following is just a start...  Let me know if this direction looks
>>> OK, and I'll finish this up.  The main item to look at is
>>> kselftest.include file.  Note that these patches are based on Shuah's
>>> series - but if you want to use these ideas I can rebase them onto
>>> mainline, and break them out per test sub-level like Shuah did.
>>
>> One of the reasons I picked install target approach is to enable the
>> feature by extending the existing run_tests support. This way we will
>> have the feature available quickly. Once that is supported, we can work
>> on evolving to a generic approach to use the include file approach, as
>> the changes you are proposing are based on the the series I sent out,
>> and makes improvements to it.
>>
>> kselftest.include file approach could work for several tests and tests
>> that can't use the generic could add their own install support.
>>
>> I propose evolving to a generic kselftest.include as the second step in
>> evolving the install feature. Can I count on you do the work and update
>> the tests to use kselftest.include, CROSS_INSTALL variable support?
> 
> Yes.  I'd be happy to evolve it through phases to support the include
> file and cross-target install feature.

Thanks that would be great.

> 
> Is there anything I can help with in the mean time?  Some of the tests
> require a directory tree of files rather than just a few top-level files
> (e.g. ftrace).

I left out couple of tests in this first round. exec and powerpc. Would
you like to add install target support for these tests? That is the next
thing on my list of todo and if you would like to get that working, you
are welcome to.

> 
> I was thinking about doing some work to tar-ify the needed directories of
> data files during build, and untar it in the execution area during the
> install step.  Do you want me to propose something there?

More like generating tar-ball of the install directory for remote
installs perhaps. Since install feature is using existing Makefile,
user can specify a override install location. Is there a need for,
more like use for generating a tar-ball for remote installs?

thanks,
-- Shuah

-- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

[PATCH v3 15/20] selftests/sysctl: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/sysctl/Makefile | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/sysctl/Makefile b/tools/testing/selftests/sysctl/Makefile
index 0a92ada..dfbaa58 100644
--- a/tools/testing/selftests/sysctl/Makefile
+++ b/tools/testing/selftests/sysctl/Makefile
@@ -4,12 +4,25 @@
 # No binaries, but make sure arg-less "make" doesn't trigger "run_tests".
 all:
 
+INSTALL_PROGS = common_tests run_numerictests run_stringtests
+NUMERIC_TEST_STR = /bin/sh ./run_numerictests
+STRING_TEST_STR = /bin/sh ./run_stringtests
+
 # Allow specific tests to be selected.
 test_num:
-	@/bin/sh ./run_numerictests
+	@$(NUMERIC_TEST_STR)
 
 test_string:
-	@/bin/sh ./run_stringtests
+	@$(STRING_TEST_STR)
+
+install: all
+ifdef INSTALL_KSFT_PATH
+	install $(INSTALL_PROGS) $(INSTALL_KSFT_PATH)
+	@echo "$(NUMERIC_TEST_STR)" >> $(KSELFTEST)
+	@echo "$(STRING_TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
 
 run_tests: all test_num test_string
 
-- 
2.1.0

[PATCH v3 16/20] selftests/timers: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/timers/Makefile | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/timers/Makefile b/tools/testing/selftests/timers/Makefile
index eb2859f..922bad5 100644
--- a/tools/testing/selftests/timers/Makefile
+++ b/tools/testing/selftests/timers/Makefile
@@ -1,8 +1,18 @@
+TEST_STR = ./posix_timers
+
 all:
 	gcc posix_timers.c -o posix_timers -lrt
 
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./posix_timers $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
 run_tests: all
-	./posix_timers
+	@$(TEST_STR)
 
 clean:
 	rm -f ./posix_timers
-- 
2.1.0

[PATCH v3 17/20] selftests/user: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/user/Makefile | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/user/Makefile b/tools/testing/selftests/user/Makefile
index 12c9d15..218c5aa 100644
--- a/tools/testing/selftests/user/Makefile
+++ b/tools/testing/selftests/user/Makefile
@@ -1,7 +1,17 @@
 # Makefile for user memory selftests
 
+TEST_STR = ./test_user_copy.sh
+
 # No binaries, but make sure arg-less "make" doesn't trigger "run_tests"
 all:
 
+install:
+ifdef INSTALL_KSFT_PATH
+	install ./test_user_copy.sh $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
 run_tests: all
-	./test_user_copy.sh
+	@$(TEST_STR)
-- 
2.1.0

[PATCH v3 18/20] selftests/vm: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing test. This target
installs test in the kselftest install location and add to the
kselftest script to run the test. Install target can be run
only from top level kernel source directory.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/vm/Makefile | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile
index 4c4b1f6..761d17d 100644
--- a/tools/testing/selftests/vm/Makefile
+++ b/tools/testing/selftests/vm/Makefile
@@ -4,13 +4,22 @@ CC = $(CROSS_COMPILE)gcc
 CFLAGS = -Wall
 BINARIES = hugepage-mmap hugepage-shm map_hugetlb thuge-gen hugetlbfstest
 BINARIES += transhuge-stress
+TEST_STR = /bin/sh ./run_vmtests || echo vmtests: [FAIL]
 
 all: $(BINARIES)
 %: %.c
 	$(CC) $(CFLAGS) -o $@ $^
 
+install:
+ifdef INSTALL_KSFT_PATH
+	install run_vmtests $(BINARIES) $(INSTALL_KSFT_PATH)
+	@echo "$(TEST_STR)" >> $(KSELFTEST)
+else
+	@echo Run make kselftest_install in top level source directory
+endif
+
 run_tests: all
-	@/bin/sh ./run_vmtests || (echo "vmtests: [FAIL]"; exit 1)
+	@$(TEST_STR)
 
 clean:
 	$(RM) $(BINARIES)
-- 
2.1.0

[PATCH v3 19/20] selftests: add install target to enable test install

[Shuah Khan]

Add a new make target to enable installing selftests. This
new target will call install targets for the tests that are
specified in INSTALL_TARGETS. During install, a script is
generated to run tests that are installed. This script will
be installed in the selftest install directory. Individual
test Makefiles are changed to add to the script. This will
allow new tests to add install and run test commands to the
generated kselftest script. run_tests target runs the
generated kselftest script to run tests when it is initiated
from from "make kselftest" from top level source directory.

Approach:

Add a new kselftest_install target:
-- exports kselftest INSTALL_KSFT_PATH
   default $(INSTALL_MOD_PATH)/lib/kselftest/$(KERNELRELEASE)
-- exports INSTALL_KSFT_PATH
-- runs selftests make kselftest_install target:

selftests make install target
-- Sets up environment for sub-makefiles
-- creates kselftest.sh script in install install dir
-- runs install targets for INSTALL_TARGETS
-- install target can be run only from top level source dir.

Individual test make install targets:
-- install test programs and/or scripts in install dir
-- append to the ksefltest.sh file to add commands to run test
-- install target can be run only from top level source dir.

Adds the following new ways to initiate selftests:
-- Installing and running kselftest from install directory
   by running  "make kselftest"
-- Running kselftest script from install directory

Maintains the following ways to run tests:
-- make TARGETS=net kselftest
-- make -C tools/testing/selftests run_tests
-- make -C tools/testing/selftests TARGETS=target run_tests
   Ability specify targets: e.g TARGETS=net
-- make run_tests from tools/testing/selftests
-- make run_tests from individual test directories:
   e.g: make run_tests in tools/testing/selftests/net

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/Makefile | 54 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 53 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 4e51122..e70cdc9 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -22,15 +22,67 @@ TARGETS += vm
 TARGETS_HOTPLUG = cpu-hotplug
 TARGETS_HOTPLUG += memory-hotplug
 
+# Used in only run_tests target when make kselftest is run in
+# top level source directory
+ifeq "$(origin TARGETS)" "command line"
+no_install_run=1
+endif
+
+ifdef INSTALL_KSFT_PATH
+KSELFTEST=$(INSTALL_KSFT_PATH)/kselftest.sh
+export KSELFTEST
+# TODO add install target for SKIP_INSTALL_TARGETS
+SKIP_INSTALL_TARGETS = exec powerpc
+INSTALL_TARGETS = $(filter-out $(SKIP_INSTALL_TARGETS),$(TARGETS)) ipc
+else
+no_install_run=1
+endif
+
+INSTALL_KSFT_ERR = Run make kselftest_install in top level source directory
+
 all:
 	for TARGET in $(TARGETS); do \
 		make -C $$TARGET; \
 	done;
 
-run_tests: all
+install:
+ifdef INSTALL_KSFT_PATH
+	rm -rf $(INSTALL_KSFT_PATH)
+	mkdir -p $(INSTALL_KSFT_PATH)
+
+	make all
+	@echo "#!/bin/sh\n# Kselftest Run Tests ...." >> $(KSELFTEST)
+	@echo "# This file is generated by kselftest_install" >> $(KSELFTEST)
+	@echo "# Please don't change it !!\n"  >> $(KSELFTEST)
+	@echo echo ============================== >> $(KSELFTEST)
+	for TARGET in $(INSTALL_TARGETS); do \
+		echo Installing $$TARGET; \
+		echo echo Start $$TARGET test .... >> $(KSELFTEST); \
+		make -C $$TARGET install; \
+		echo echo End $$TARGET test .... >> $(KSELFTEST); \
+		echo echo ============================== >> $(KSELFTEST); \
+	done;
+	chmod +x $(KSELFTEST)
+else
+	@echo $(INSTALL_KSFT_ERR)
+endif
+
+run_tests:
+ifndef no_install_run
+# ifdef INSTALL_KSFT_PATH
+	make install
+	@cd $(INSTALL_KSFT_PATH); ./kselftest.sh; cd -
+# invoke run_tests for SKIP_INSTALL_TARGETS
+	for TARGET in $(SKIP_INSTALL_TARGETS); do \
+		make -C $$TARGET run_tests; \
+	done;
+# endif
+else
+	make all
 	for TARGET in $(TARGETS); do \
 		make -C $$TARGET run_tests; \
 	done;
+endif
 
 hotplug:
 	for TARGET in $(TARGETS_HOTPLUG); do \
-- 
2.1.0

[PATCH v3 20/20] kbuild: add a new kselftest_install make target to install selftests

[Shuah Khan]

Add a new make target to install to install kernel selftests.
This new target will build and install selftests. kselftest
target now depends on kselftest_install and runs the generated
kselftest script to reduce duplicate work and for common look
and feel when running tests.

make kselftest_target:
-- exports kselftest INSTALL_KSFT_PATH
   default $(INSTALL_MOD_PATH)/lib/kselftest/$(KERNELRELEASE)
-- exports INSTALL_KSFT_PATH
-- runs selftests make install target:

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 Makefile | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index b1c3254..99fac47 100644
--- a/Makefile
+++ b/Makefile
@@ -1072,12 +1072,21 @@ headers_check: headers_install
 	$(Q)$(MAKE) $(hdr-inst)=arch/$(hdr-arch)/include/uapi/asm $(hdr-dst) HDRCHECK=1
 
 # ---------------------------------------------------------------------------
-# Kernel selftest
+# Kernel selftest targets
+
+# Default base path for kselftest install
+INSTALL_KSFT_PATH = $(INSTALL_MOD_PATH)/lib/kselftest/$(KERNELRELEASE)
+export INSTALL_KSFT_PATH
 
 PHONY += kselftest
 kselftest:
 	$(Q)$(MAKE) -C tools/testing/selftests run_tests
 
+# Kernel selftest install
+PHONY += kselftest_install
+kselftest_install:
+	$(Q)$(MAKE) -C tools/testing/selftests install
+
 # ---------------------------------------------------------------------------
 # Modules
 
@@ -1286,6 +1295,9 @@ help:
 	@echo  '                    Build, install, and boot kernel before'
 	@echo  '                    running kselftest on it'
 	@echo  ''
+	@echo  '  kselftest_install - Install Kselftests to INSTALL_KSFT_PATH'
+	@echo  '                      default: $(INSTALL_MOD_PATH)/lib/kselftest/$(KERNELRELEASE)'
+	@echo  ''
 	@echo  'Kernel packaging:'
 	@$(MAKE) $(build)=$(package-dir) help
 	@echo  ''
-- 
2.1.0

Re: [PATCH v3 00/20] kselftest install target feature

[Michael Ellerman]

On Wed, 2014-12-24 at 09:27 -0700, Shuah Khan wrote:
> This patch series adds a new kselftest_install make target
> to enable selftest install. When make kselftest_install is
> run, selftests are installed on the system. A new install
> target is added to selftests Makefile which will install
> targets for the tests that are specified in INSTALL_TARGETS.
> During install, a script is generated to run tests that are
> installed. This script will be installed in the selftest install
> directory. Individual test Makefiles are changed to add to the
> script. This will allow new tests to add install and run test
> commands to the generated kselftest script. kselftest target
> now depends on kselftest_install and runs the generated kselftest
> script to reduce duplicate work and for common look and feel when
> running tests.
> 
> This approach leverages and extends the existing framework that
> uses makefile targets to implement run_tests and adds install
> target. This will scale well as new tests get added and makes
> it easier for test writers to add install target at the same
> time new test gets added.
> 
> This v3 series reduces duplicate code to generate script
> in indiviual test Makefiles and consolidates support in
> selftests main Makefile. In the main Makefile, it does
> minimal work to set and export install path. In this
> series exec and powerpc tests are not included in the
> install, this work will be done in future patches. exec
> and powerpc are still run when make kselftest is invoked.

Any particular reason you excluded the powerpc tests? Going by a quick count,
powerpc has 32 of the 54 self tests, ie. more than half.

Sorry I didn't get a chance to review v1 or v2, but is this really the best
solution we can come up with? It seems to involve a lot of boiler plate getting
repeated in every Makefile.

I'm off this week so I can't immediately come up with something better, I'll
try in the new year.

cheers

Re: [PATCH v3 00/20] kselftest install target feature

[Shuah Khan]

On 12/28/2014 09:53 PM, Michael Ellerman wrote:
> On Wed, 2014-12-24 at 09:27 -0700, Shuah Khan wrote:
>> This patch series adds a new kselftest_install make target
>> to enable selftest install. When make kselftest_install is
>> run, selftests are installed on the system. A new install
>> target is added to selftests Makefile which will install
>> targets for the tests that are specified in INSTALL_TARGETS.
>> During install, a script is generated to run tests that are
>> installed. This script will be installed in the selftest install
>> directory. Individual test Makefiles are changed to add to the
>> script. This will allow new tests to add install and run test
>> commands to the generated kselftest script. kselftest target
>> now depends on kselftest_install and runs the generated kselftest
>> script to reduce duplicate work and for common look and feel when
>> running tests.
>>
>> This approach leverages and extends the existing framework that
>> uses makefile targets to implement run_tests and adds install
>> target. This will scale well as new tests get added and makes
>> it easier for test writers to add install target at the same
>> time new test gets added.
>>
>> This v3 series reduces duplicate code to generate script
>> in indiviual test Makefiles and consolidates support in
>> selftests main Makefile. In the main Makefile, it does
>> minimal work to set and export install path. In this
>> series exec and powerpc tests are not included in the
>> install, this work will be done in future patches. exec
>> and powerpc are still run when make kselftest is invoked.
> 
> Any particular reason you excluded the powerpc tests? Going by a quick count,
> powerpc has 32 of the 54 self tests, ie. more than half.

No particular reason other than not having a good way to test the
changes I need to make. It does have sub-directory structure with
multiple makefiles underneath. I would like to work on this after
this patch series gets in or maybe you can help out on powerpc
changes if you like.

> 
> Sorry I didn't get a chance to review v1 or v2, but is this really the best
> solution we can come up with? It seems to involve a lot of boiler plate getting
> repeated in every Makefile.

This approach extends the existing approach to use makefile targets as
a means to support running tests. Also it gives full control to the
individual test developer in making changes to the targets as needed
without conflicts with work that is in progress on other tests.

There isn't a whole lot of boiler plating code repeated as such in
individual makefiles. They all add their specific targets to the
main script.

-- Shuah


-- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978