Re: [Qemu-devel] [PATCH 2/6] tests/qemu-iotests/group: Introduce a new "ci" group for CI pipelines

[Thomas Huth <thuth@redhat.com>]

On 24/04/2019 13.25, Daniel P. Berrangé wrote:
> On Wed, Apr 24, 2019 at 12:37:43PM +0200, Thomas Huth wrote:
>> Tests in this group are supposed to run in every possible QEMU configuration.
>> That means they should run with every QEMU binary (also non-x86), without
>> dependencies on an optional features, they must work at least with the qcow2
>> image format and be able to run on all kind of host filesystems and users
>> (i.e. also as "nobody" or "root").
>>
>> The initial list has been created as a subset of the "quick" group, where
>> I've disabled all tests that are failing with qemu-system-aarch64 or
>> qemu-system-tricore or in one of our CI pipelines.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  tests/qemu-iotests/group | 194 ++++++++++++++++++++-------------------
>>  1 file changed, 102 insertions(+), 92 deletions(-)
>>
>> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
>> index bae7718380..2ed42dcc14 100644
>> --- a/tests/qemu-iotests/group
>> +++ b/tests/qemu-iotests/group
>> @@ -4,63 +4,73 @@
>>  # - do not start group names with a digit
>>  #
>>  
>> +#
>> +# Some notes about the groups:
>> +# - Tests in the "quick" group should finish within some few seconds
>> +# - Tests in the "ci" group are suitable for running in CI systems. That
>> +#   means they should run with every QEMU binary (also non-x86), with
>> +#   every QEMU configuration (i.e. no dependency on an optional feature),
>> +#   at least with the qcow2 image format and all kind of host filesystems
>> +#   and users (i.e. also as "nobody" or "root").
> 
> No dependancy on optional features feels a bit restrictive from my POV.
> 
> We have quite alot of testing coverage of stuff that depends on the
> crypto libraries that is important for us to run in "CI". This includes
> LUKS block drivers, qcow2 with LUKS,  NBD with TLS. Personally I expect
> all of those to be tested by CI systems.
> 
> IOW for a "CI" group, the bar should be higher than "no optional features"

Ok, I think I just did not write it properly. What I meant was: the test
should not *fail* because of a missing feature (what some tests are
unfortunately doing). It's fine if the test detects the missing feature
and then marks the test as *skip*.

> Since we control the env used for CI via Docker or VM images, we can ensure
> that they're populated with a set of packages that maximises our coverage.
> All our platforms support gnutls for example, which gets us all the crypto
> bits we need.
> 
> The problem of course is the desire to put this into "make check" by default
> which developers can run in any configuration.
> 
> I think this is a sign that we need two distinct groupings.
> 
> Stuff run by a generic "make check" /should not/ depend on optional features,
> 
> Stuff run by an automated CI system /should/ depend on optional features. 

I think adding two new groups is not really needed here.
First, we already have the "quick" group that should contain most of the
tests with optional features. Second, just make sure that tests that
depend on optional features and still should be included in all CI
pipelines can detect the availability of the feature properly and report
"skip" if the feature is not available.

 Thomas