Re: [Qemu-devel] [PATCH 5/6] tests: Run the iotests during "make check" again

[Thomas Huth <thuth@redhat.com>]

On 24/04/2019 12.48, Daniel P. Berrangé wrote:
> On Wed, Apr 24, 2019 at 12:37:46PM +0200, Thomas Huth wrote:
>> People often forget to run the iotests before submitting patches or
>> pull requests - this is likely due to the fact that we do not run the
>> tests during our mandatory "make check" tests yet. Now that we've got
>> a new "ci" group of iotests that should be fine to run in every environ-
>> ment, it should be OK to enable the iotests during "make check" again.
>> Thus we now run the "ci" tests by default from the qemu-iotests-quick.sh
>> script, and only use the former "quick" group (that contains some tests
>> that are failing in some environments) when the user decided to run
>> "make check SPEED=thorough" or something similar.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  tests/Makefile.include      |  2 +-
>>  tests/qemu-iotests-quick.sh | 17 ++++++++++++++++-
>>  2 files changed, 17 insertions(+), 2 deletions(-)
> 
>> diff --git a/tests/qemu-iotests-quick.sh b/tests/qemu-iotests-quick.sh
>> index 0e554bb972..416b3fc48b 100755
>> --- a/tests/qemu-iotests-quick.sh
>> +++ b/tests/qemu-iotests-quick.sh
> 
> s/-quick//  perhaps, since it no longer only runs the "quick" tests.
> 
>> @@ -1,8 +1,23 @@
>>  #!/bin/sh
>>  
>> +# Honor the SPEED environment variable, just like we do it for the qtests.
>> +# The default is to run all tests that still work fine in a CI environments,
>> +# but if the user set SPEED=slow or SPEED=thorough, we also run all other
>> +# tests that are still marked as "quick"
>> +if [ "$SPEED" = "slow" -o "$SPEED" = "thorough" ]; then
>> +    group=quick
> 
> When user asks for "slow" tests we run the "quick" tests, which
> presumably implies...
> 
>> +else
>> +    group=ci
> 
> ..."ci" is quicker than the "quick"  tests !
> 
> Confused ? We certainly try to ensure that you will be :-)

... and there is also tests/check-block.sh (which is not used by "make
check-block", by the way, but still talks about "make check-block"),
which is even slower than the "ci" and "quick" group...

I guess that could be cleaned up, too (why do we need two scripts?) ...
just not sure how the final result should look like and if it should be
part of this patch series or rather something separate. Max, Kevin, what
do you think?

Maybe we should merge the two scripts, and depending on the SPEED
variable, we run:

 quick => "ci" group
 thorough => "quick" group
 slow => old behaviour of the check-block.sh script

?

 Thomas