[Qemu-devel] [PATCH] tests: Allow GTESTER_OPTIONS variable expansion from command line

[Qemu-devel] [PATCH] tests: Allow GTESTER_OPTIONS variable expansion from command line

[Philippe Mathieu-Daudé]

The gtester utility offers the "-p=TESTPATH" option to
"only run test cases matching TESTPATH".  We already use
a $GTESTER_OPTIONS variable.
Use make variable expansion to allow qtest path matching
from command line:

  $ time make check-qtest-arm GTESTER_OPTIONS="-p /arm/boot-serial"
    GTESTER check-qtest-arm

  user 0m1.548s

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
I am more interested in the "-s" option (exclude pattern) but
can't have it working :(

i.e. "run all but qom/hmp":
make check-qtest-arm GTESTER_OPTIONS="-s /arm/qom -s /arm/hmp"

 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 3b9a5e31a2..b0ff3d988c 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -870,7 +870,7 @@ tests/test-qga$(EXESUF): qemu-ga$(EXESUF)
 tests/test-qga$(EXESUF): tests/test-qga.o $(qtest-obj-y)
 
 SPEED = quick
-GTESTER_OPTIONS = -k $(if $(V),--verbose,-q)
+GTESTER_OPTIONS += -k $(if $(V),--verbose,-q)
 GCOV_OPTIONS = -n $(if $(V),-f,)
 
 # gtester tests, possibly with verbose output
-- 
2.17.0

Re: [Qemu-devel] [PATCH] tests: Allow GTESTER_OPTIONS variable expansion from command line

[Thomas Huth]

On 23.05.2018 13:57, Philippe Mathieu-Daudé wrote:
> The gtester utility offers the "-p=TESTPATH" option to
> "only run test cases matching TESTPATH".  We already use
> a $GTESTER_OPTIONS variable.
> Use make variable expansion to allow qtest path matching
> from command line:
> 
>   $ time make check-qtest-arm GTESTER_OPTIONS="-p /arm/boot-serial"
>     GTESTER check-qtest-arm
> 
>   user 0m1.548s
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> I am more interested in the "-s" option (exclude pattern) but
> can't have it working :(
> 
> i.e. "run all but qom/hmp":
> make check-qtest-arm GTESTER_OPTIONS="-s /arm/qom -s /arm/hmp"
> 
>  tests/Makefile.include | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 3b9a5e31a2..b0ff3d988c 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -870,7 +870,7 @@ tests/test-qga$(EXESUF): qemu-ga$(EXESUF)
>  tests/test-qga$(EXESUF): tests/test-qga.o $(qtest-obj-y)
>  
>  SPEED = quick
> -GTESTER_OPTIONS = -k $(if $(V),--verbose,-q)
> +GTESTER_OPTIONS += -k $(if $(V),--verbose,-q)
>  GCOV_OPTIONS = -n $(if $(V),-f,)
>  
>  # gtester tests, possibly with verbose output

"make check-help" says:

 Default options are -k and (for make V=1) --verbose; they can be
 changed with variable GTESTER_OPTIONS.

So if you change the behavior here to amend to the variable instead, I
think you should also adjust the help text. Not sure if it's a good
idea, though, there might be people who want to run without "-k" and
this can not be disabled anymore with your patch.

 Thomas

Re: [Qemu-devel] [PATCH] tests: Allow GTESTER_OPTIONS variable expansion from command line

[Philippe Mathieu-Daudé]

On 05/23/2018 03:33 PM, Thomas Huth wrote:
> On 23.05.2018 13:57, Philippe Mathieu-Daudé wrote:
>> The gtester utility offers the "-p=TESTPATH" option to
>> "only run test cases matching TESTPATH".  We already use
>> a $GTESTER_OPTIONS variable.
>> Use make variable expansion to allow qtest path matching
>> from command line:
>>
>>   $ time make check-qtest-arm GTESTER_OPTIONS="-p /arm/boot-serial"
>>     GTESTER check-qtest-arm
>>
>>   user 0m1.548s
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> I am more interested in the "-s" option (exclude pattern) but
>> can't have it working :(
>>
>> i.e. "run all but qom/hmp":
>> make check-qtest-arm GTESTER_OPTIONS="-s /arm/qom -s /arm/hmp"
>>
>>  tests/Makefile.include | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 3b9a5e31a2..b0ff3d988c 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -870,7 +870,7 @@ tests/test-qga$(EXESUF): qemu-ga$(EXESUF)
>>  tests/test-qga$(EXESUF): tests/test-qga.o $(qtest-obj-y)
>>  
>>  SPEED = quick
>> -GTESTER_OPTIONS = -k $(if $(V),--verbose,-q)
>> +GTESTER_OPTIONS += -k $(if $(V),--verbose,-q)
>>  GCOV_OPTIONS = -n $(if $(V),-f,)
>>  
>>  # gtester tests, possibly with verbose output
> 
> "make check-help" says:
> 
>  Default options are -k and (for make V=1) --verbose; they can be
>  changed with variable GTESTER_OPTIONS.
> 
> So if you change the behavior here to amend to the variable instead, I
> think you should also adjust the help text. Not sure if it's a good
> idea, though, there might be people who want to run without "-k" and
> this can not be disabled anymore with your patch.

Ok I see.  I'm interested in hearing how people run/use this option.

Anyway I'm fine using:

$ make check-qtest-arm GTESTER_OPTIONS="-p /arm/boot-serial --verbose"

Thanks for your review,

Phil.