From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.3 required=3.0 tests=FROM_EXCESS_BASE64, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_PASS,URIBL_BLOCKED,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30E1BC10F11 for ; Wed, 24 Apr 2019 12:51:53 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 03A09218B0 for ; Wed, 24 Apr 2019 12:51:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 03A09218B0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([127.0.0.1]:41238 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hJHNQ-0000XU-Aq for qemu-devel@archiver.kernel.org; Wed, 24 Apr 2019 08:51:52 -0400 Received: from eggs.gnu.org ([209.51.188.92]:45891) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hJHMb-0008OP-2X for qemu-devel@nongnu.org; Wed, 24 Apr 2019 08:51:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hJHMZ-0006NT-OT for qemu-devel@nongnu.org; Wed, 24 Apr 2019 08:51:01 -0400 Received: from mx1.redhat.com ([209.132.183.28]:59318) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hJHMW-0006B7-Ju; Wed, 24 Apr 2019 08:50:56 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 64DD8307D867; Wed, 24 Apr 2019 12:50:55 +0000 (UTC) Received: from redhat.com (ovpn-112-50.ams2.redhat.com [10.36.112.50]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D5885648A3; Wed, 24 Apr 2019 12:50:49 +0000 (UTC) Date: Wed, 24 Apr 2019 13:50:46 +0100 From: Daniel =?utf-8?B?UC4gQmVycmFuZ8Op?= To: Thomas Huth Message-ID: <20190424125046.GJ31388@redhat.com> References: <20190424103747.10173-1-thuth@redhat.com> <20190424103747.10173-3-thuth@redhat.com> <20190424112537.GF31388@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.3 (2019-02-01) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.48]); Wed, 24 Apr 2019 12:50:55 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: Re: [Qemu-devel] [PATCH 2/6] tests/qemu-iotests/group: Introduce a new "ci" group for CI pipelines X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Daniel =?utf-8?B?UC4gQmVycmFuZ8Op?= Cc: Fam Zheng , Kevin Wolf , Ed Maste , qemu-block@nongnu.org, Philippe =?utf-8?Q?Mathieu-Daud=C3=A9?= , qemu-devel@nongnu.org, Max Reitz , Alex =?utf-8?Q?Benn=C3=A9e?= , Li-Wen Hsu Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Message-ID: <20190424125046.0ZZSou6K_XNBEHRXMZHo3k9at6duOmcb2vH25G6fPr8@z> On Wed, Apr 24, 2019 at 02:37:02PM +0200, Thomas Huth wrote: > On 24/04/2019 13.25, Daniel P. Berrang=C3=A9 wrote: > > On Wed, Apr 24, 2019 at 12:37:43PM +0200, Thomas Huth wrote: > >> Tests in this group are supposed to run in every possible QEMU confi= guration. > >> That means they should run with every QEMU binary (also non-x86), wi= thout > >> dependencies on an optional features, they must work at least with t= he qcow2 > >> image format and be able to run on all kind of host filesystems and = users > >> (i.e. also as "nobody" or "root"). > >> > >> The initial list has been created as a subset of the "quick" group, = where > >> I've disabled all tests that are failing with qemu-system-aarch64 or > >> qemu-system-tricore or in one of our CI pipelines. > >> > >> Signed-off-by: Thomas Huth > >> --- > >> tests/qemu-iotests/group | 194 ++++++++++++++++++++----------------= --- > >> 1 file changed, 102 insertions(+), 92 deletions(-) > >> > >> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group > >> index bae7718380..2ed42dcc14 100644 > >> --- a/tests/qemu-iotests/group > >> +++ b/tests/qemu-iotests/group > >> @@ -4,63 +4,73 @@ > >> # - do not start group names with a digit > >> # > >> =20 > >> +# > >> +# Some notes about the groups: > >> +# - Tests in the "quick" group should finish within some few second= s > >> +# - Tests in the "ci" group are suitable for running in CI systems.= That > >> +# means they should run with every QEMU binary (also non-x86), wi= th > >> +# every QEMU configuration (i.e. no dependency on an optional fea= ture), > >> +# at least with the qcow2 image format and all kind of host files= ystems > >> +# and users (i.e. also as "nobody" or "root"). > >=20 > > No dependancy on optional features feels a bit restrictive from my PO= V. > >=20 > > We have quite alot of testing coverage of stuff that depends on the > > crypto libraries that is important for us to run in "CI". This includ= es > > LUKS block drivers, qcow2 with LUKS, NBD with TLS. Personally I expe= ct > > all of those to be tested by CI systems. > >=20 > > IOW for a "CI" group, the bar should be higher than "no optional feat= ures" >=20 > Ok, I think I just did not write it properly. What I meant was: the tes= t > should not *fail* because of a missing feature (what some tests are > unfortunately doing). It's fine if the test detects the missing feature > and then marks the test as *skip*. Ok, this brings back the topic I don't think we ever resolved around how to best "detect" features in the iotests. eg if we want to know whether we can use TLS features, how do we decide this ? One option is to do a dummy launch of QEMU using TLS & check if it fails, but that's kind of fragile for CI. A genuine bug in QEMU might cause this dummy launch to fail, and thus mistakenly skip all the tests instead of failing. We can't look at config-host.{h,mak} either as iotests are supposed to be runnable outside of a build tree. We could look at whether gnutls is linked to QEMU but again that is going to be fragile if a bug in configure causes us to mistakenly stop enabling gnutls. For developers, dynamically probing features & skipping is reasonable. For automated CI, I think we should set clear desired featureset upfront based on what we know we are supposde to have available and not silently skip tests Regards, Daniel --=20 |: https://berrange.com -o- https://www.flickr.com/photos/dberran= ge :| |: https://libvirt.org -o- https://fstop138.berrange.c= om :| |: https://entangle-photo.org -o- https://www.instagram.com/dberran= ge :|