Re: [Qemu-devel] [PATCH 2/6] tests/qemu-iotests/group: Introduce a new "ci" group for CI pipelines

["Daniel P. Berrangé" <berrange@redhat.com>]

On Wed, Apr 24, 2019 at 02:37:02PM +0200, Thomas Huth wrote:
> On 24/04/2019 13.25, Daniel P. Berrangé wrote:
> > On Wed, Apr 24, 2019 at 12:37:43PM +0200, Thomas Huth wrote:
> >> Tests in this group are supposed to run in every possible QEMU configuration.
> >> That means they should run with every QEMU binary (also non-x86), without
> >> dependencies on an optional features, they must work at least with the qcow2
> >> image format and be able to run on all kind of host filesystems and users
> >> (i.e. also as "nobody" or "root").
> >>
> >> The initial list has been created as a subset of the "quick" group, where
> >> I've disabled all tests that are failing with qemu-system-aarch64 or
> >> qemu-system-tricore or in one of our CI pipelines.
> >>
> >> Signed-off-by: Thomas Huth <thuth@redhat.com>
> >> ---
> >>  tests/qemu-iotests/group | 194 ++++++++++++++++++++-------------------
> >>  1 file changed, 102 insertions(+), 92 deletions(-)
> >>
> >> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> >> index bae7718380..2ed42dcc14 100644
> >> --- a/tests/qemu-iotests/group
> >> +++ b/tests/qemu-iotests/group
> >> @@ -4,63 +4,73 @@
> >>  # - do not start group names with a digit
> >>  #
> >>  
> >> +#
> >> +# Some notes about the groups:
> >> +# - Tests in the "quick" group should finish within some few seconds
> >> +# - Tests in the "ci" group are suitable for running in CI systems. That
> >> +#   means they should run with every QEMU binary (also non-x86), with
> >> +#   every QEMU configuration (i.e. no dependency on an optional feature),
> >> +#   at least with the qcow2 image format and all kind of host filesystems
> >> +#   and users (i.e. also as "nobody" or "root").
> > 
> > No dependancy on optional features feels a bit restrictive from my POV.
> > 
> > We have quite alot of testing coverage of stuff that depends on the
> > crypto libraries that is important for us to run in "CI". This includes
> > LUKS block drivers, qcow2 with LUKS,  NBD with TLS. Personally I expect
> > all of those to be tested by CI systems.
> > 
> > IOW for a "CI" group, the bar should be higher than "no optional features"
> 
> Ok, I think I just did not write it properly. What I meant was: the test
> should not *fail* because of a missing feature (what some tests are
> unfortunately doing). It's fine if the test detects the missing feature
> and then marks the test as *skip*.

Ok, this brings back the topic I don't think we ever resolved around how
to best "detect" features in the iotests. eg if we want to know whether
we can use TLS features, how do we decide this ?

One option is to do a dummy launch of QEMU using TLS & check if it fails,
but that's kind of fragile for CI. A genuine bug in QEMU might cause this
dummy launch to fail, and thus mistakenly skip all the tests instead of
failing.

We can't look at config-host.{h,mak} either as iotests are supposed to be
runnable outside of a build tree.

We could look at whether gnutls is linked to QEMU but again that is going
to be fragile if a bug in configure causes us to mistakenly stop enabling
gnutls.

For developers, dynamically probing features & skipping is reasonable.

For automated CI, I think we should set clear desired featureset upfront
based on what we know we are supposde to have available and not silently
skip tests


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|