From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:47300) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hJFiP-0008HY-Tx for qemu-devel@nongnu.org; Wed, 24 Apr 2019 07:05:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hJFdT-0005p6-R5 for qemu-devel@nongnu.org; Wed, 24 Apr 2019 07:00:21 -0400 References: <20190424103747.10173-1-thuth@redhat.com> <20190424103747.10173-6-thuth@redhat.com> <20190424104855.GE31388@redhat.com> From: Thomas Huth Message-ID: Date: Wed, 24 Apr 2019 12:59:59 +0200 MIME-Version: 1.0 In-Reply-To: <20190424104855.GE31388@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [PATCH 5/6] tests: Run the iotests during "make check" again List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: =?UTF-8?Q?Daniel_P=2e_Berrang=c3=a9?= Cc: qemu-devel@nongnu.org, Fam Zheng , Kevin Wolf , Ed Maste , qemu-block@nongnu.org, =?UTF-8?Q?Alex_Benn=c3=a9e?= , Max Reitz , =?UTF-8?Q?Philippe_Mathieu-Daud=c3=a9?= , Li-Wen Hsu On 24/04/2019 12.48, Daniel P. Berrang=C3=A9 wrote: > On Wed, Apr 24, 2019 at 12:37:46PM +0200, Thomas Huth wrote: >> People often forget to run the iotests before submitting patches or >> pull requests - this is likely due to the fact that we do not run the >> tests during our mandatory "make check" tests yet. Now that we've got >> a new "ci" group of iotests that should be fine to run in every enviro= n- >> ment, it should be OK to enable the iotests during "make check" again. >> Thus we now run the "ci" tests by default from the qemu-iotests-quick.= sh >> script, and only use the former "quick" group (that contains some test= s >> that are failing in some environments) when the user decided to run >> "make check SPEED=3Dthorough" or something similar. >> >> Signed-off-by: Thomas Huth >> --- >> tests/Makefile.include | 2 +- >> tests/qemu-iotests-quick.sh | 17 ++++++++++++++++- >> 2 files changed, 17 insertions(+), 2 deletions(-) >=20 >> diff --git a/tests/qemu-iotests-quick.sh b/tests/qemu-iotests-quick.sh >> index 0e554bb972..416b3fc48b 100755 >> --- a/tests/qemu-iotests-quick.sh >> +++ b/tests/qemu-iotests-quick.sh >=20 > s/-quick// perhaps, since it no longer only runs the "quick" tests. >=20 >> @@ -1,8 +1,23 @@ >> #!/bin/sh >> =20 >> +# Honor the SPEED environment variable, just like we do it for the qt= ests. >> +# The default is to run all tests that still work fine in a CI enviro= nments, >> +# but if the user set SPEED=3Dslow or SPEED=3Dthorough, we also run a= ll other >> +# tests that are still marked as "quick" >> +if [ "$SPEED" =3D "slow" -o "$SPEED" =3D "thorough" ]; then >> + group=3Dquick >=20 > When user asks for "slow" tests we run the "quick" tests, which > presumably implies... >=20 >> +else >> + group=3Dci >=20 > ..."ci" is quicker than the "quick" tests ! >=20 > Confused ? We certainly try to ensure that you will be :-) ... and there is also tests/check-block.sh (which is not used by "make check-block", by the way, but still talks about "make check-block"), which is even slower than the "ci" and "quick" group... I guess that could be cleaned up, too (why do we need two scripts?) ... just not sure how the final result should look like and if it should be part of this patch series or rather something separate. Max, Kevin, what do you think? Maybe we should merge the two scripts, and depending on the SPEED variable, we run: quick =3D> "ci" group thorough =3D> "quick" group slow =3D> old behaviour of the check-block.sh script ? Thomas From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B87CC282CE for ; Wed, 24 Apr 2019 11:07:26 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5F0E6218B0 for ; Wed, 24 Apr 2019 11:07:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5F0E6218B0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([127.0.0.1]:39666 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hJFkL-0001gO-EN for qemu-devel@archiver.kernel.org; Wed, 24 Apr 2019 07:07:25 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47300) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hJFiP-0008HY-Tx for qemu-devel@nongnu.org; Wed, 24 Apr 2019 07:05:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hJFdT-0005p6-R5 for qemu-devel@nongnu.org; Wed, 24 Apr 2019 07:00:21 -0400 Received: from mx1.redhat.com ([209.132.183.28]:33244) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hJFdN-0005WX-Im; Wed, 24 Apr 2019 07:00:13 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6FAF07C0A3; Wed, 24 Apr 2019 11:00:09 +0000 (UTC) Received: from thuth.remote.csb (ovpn-116-69.ams2.redhat.com [10.36.116.69]) by smtp.corp.redhat.com (Postfix) with ESMTP id E4531646D9; Wed, 24 Apr 2019 11:00:00 +0000 (UTC) To: =?UTF-8?Q?Daniel_P=2e_Berrang=c3=a9?= References: <20190424103747.10173-1-thuth@redhat.com> <20190424103747.10173-6-thuth@redhat.com> <20190424104855.GE31388@redhat.com> From: Thomas Huth Openpgp: preference=signencrypt Autocrypt: addr=thuth@redhat.com; keydata= xsFNBFH7eUwBEACzyOXKU+5Pcs6wNpKzrlJwzRl3VGZt95VCdb+FgoU9g11m7FWcOafrVRwU yYkTm9+7zBUc0sW5AuPGR/dp3pSLX/yFWsA/UB4nJsHqgDvDU7BImSeiTrnpMOTXb7Arw2a2 4CflIyFqjCpfDM4MuTmzTjXq4Uov1giGE9X6viNo1pxyEpd7PanlKNnf4PqEQp06X4IgUacW tSGj6Gcns1bCuHV8OPWLkf4hkRnu8hdL6i60Yxz4E6TqlrpxsfYwLXgEeswPHOA6Mn4Cso9O 0lewVYfFfsmokfAVMKWzOl1Sr0KGI5T9CpmRfAiSHpthhHWnECcJFwl72NTi6kUcUzG4se81 O6n9d/kTj7pzTmBdfwuOZ0YUSqcqs0W+l1NcASSYZQaDoD3/SLk+nqVeCBB4OnYOGhgmIHNW 0CwMRO/GK+20alxzk//V9GmIM2ACElbfF8+Uug3pqiHkVnKqM7W9/S1NH2qmxB6zMiJUHlTH gnVeZX0dgH27mzstcF786uPcdEqS0KJuxh2kk5IvUSL3Qn3ZgmgdxBMyCPciD/1cb7/Ahazr 3ThHQXSHXkH/aDXdfLsKVuwDzHLVSkdSnZdt5HHh75/NFHxwaTlydgfHmFFwodK8y/TjyiGZ zg2Kje38xnz8zKn9iesFBCcONXS7txENTzX0z80WKBhK+XSFJwARAQABzRxUaG9tYXMgSHV0 aCA8dGguaHV0aEBnbXguZGU+wsF7BBMBAgAlAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIX gAUCUfuWKwIZAQAKCRAu2dd0/nAttbe/EACb9hafyOb2FmhUqeAiBORSsUifFacQ7laVjcgR I4um8CSHvxijYftpkM2EdAtmXIKgbNDpQoXcWLXB9lu9mLgTO4DVT00TRR65ikn3FCWcyT74 ENTOzRKyKLsDCjhXKPblTPIQbYAUCOWElcyAPm0ERd62fA/rKNxgIiNo/l4UODOMoOJm2/Ox ZoTckW68Eqv7k9L7m7j+Hn3hoDTjAmcCBJt+j7pOhzWvCbqoNOIH8C8qvPaNlrba+R/K6jkO 6jZkTbYQpGIofEQJ/TNn38IsNGpI1ALTHWFtoMxp3j2Imz0REO6dRE2fHRN8sVlHgkoeGhmY NbDsDE1jFQOEObFnu0euk//7BXU7tGOHckVAZ8T1smiRPHfQU7UEH2a/grndxJ+PNeM5w7n2 l+FN3cf2KgPotCK2s9MjSdZA7C5e3rFYO8lqiqTJKvc62vqp3e7B0Kjyy5/QtzSOejBij2QL xkKSFNtxIz4MtuxN8e3IDQNxsKry3nF7R4MDvouXlMo6wP9KuyNWb+vFJt9GtbgfDMIFVamp ZfhEWzWRJH4VgksENA4K/BzjEHCcbTUb1TFsiB1VRnBPJ0SqlvifnfKk6HcpkDk6Pg8Q5FOJ gbNHrdgXsm+m/9GF2zUUr+rOlhVbK23TUqKqPfwnD7uxjpakVcJnsVCFqJpZi1F/ga9IN87B TQRR+3lMARAAtp831HniPHb9AuKq3wj83ujZK8lH5RLrfVsB4X1wi47bwo56BqhXpR/zxPTR eOFT0gnbw9UkphVc7uk/alnXMDEmgvnuxv89PwIQX6k3qLABeV7ykJQG/WT5HQ6+2DdGtVw3 2vjYAPiWQeETsgWRRQMDR0/hwp8s8tL/UodwYCScH6Vxx9pdy353L1fK4Bb9G73a+9FPjp9l x+WwKTsltVqSBuSjyZQ3c3EE8qbTidXZxB38JwARH8yN3TX+t65cbBqLl/zRUUUTapHQpUEd yoAsHIml32e4q+3xdLtTdlLi7FgPBItSazcqZPjEcYW73UAuLcmQmfJlQ5PkDiuqcitn+KzH /1pqsTU7QFZjbmSMJyXY0TDErOFuMOjf20b6arcpEqse1V3IKrb+nqqA2azboRm3pEANLAJw iVTwK3qwGRgK5ut6N/Znv20VEHkFUsRAZoOusrIRfR5HFDxlXguAdEz8M/hxXFYYXqOoaCYy 6pJxTjy0Y/tIfmS/g9Bnp8qg9wsrsnk0+XRnDVPak++G3Uq9tJPwpJbyO0vcqEI3vAXkAB7X VXLzvFwi66RrsPUoDkuzj+aCNumtOePDOCpXQGPpKl+l1aYRMN/+lNSk3+1sVuc2C07WnYyE gV/cbEVklPmKrNwu6DeUyD0qI/bVzKMWZAiB1r56hsGeyYcAEQEAAcLBXwQYAQIACQUCUft5 TAIbDAAKCRAu2dd0/nAttYTwEACLAS/THRqXRKb17PQmKwZHerUvZm2klo+lwQ3wNQBHUJAT p2R9ULexyXrJPqjUpy7+voz+FcKiuQBTKyieiIxO46oMxsbXGZ70o3gxjxdYdgimUD6U8PPd JH8tfAL4BR5FZNjspcnscN2jgbF4OrpDeOLyBaj6HPmElNPtECHWCaf1xbIFsZxSDGMA6cUh 0uX3Q8VI7JN1AR2cfiIRY7NrIlWYucJxyKjO3ivWm69nCtsHiJ0wcF8KlVo7F2eLaufo0K8A ynL8SHMF3VEyxsXOP2f1UR9T2Ur30MXcTBpjUxml1TX3RWY5uH89Js/jlIugBwuAmacJ7JYh lTg6sF/GNc4nPb4kk2yktNWTade+TzsllYlJPaorD2Qe8qX0iFUhFC6y9+O6mP4ZvWoYapp9 ezYNuebMgEr93ob1+4sFg3812wNP01WqsGtWCJHnPv/JoonFdMzD/bIkXGEJMk6ks2kxQQZq g6Ik/s/vxOfao/xCn8nHt7GwvVy41795hzK6tbSl+BuyCRp0vfPRP34OnK7+jR2nvQpJu/pU rCELuGwT9hsYkUPjVd4lfylN3mzEc6iAv/wwjsc0DRTSQCpXT3v2ymTAsRKrVaEZLibTXaf+ WslxWek3xNYRiqwwWAJuL652eAlxUgQ5ZS+fXBRTiQpJ+F26I/2lccScRd9G5w== Organization: Red Hat Message-ID: Date: Wed, 24 Apr 2019 12:59:59 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <20190424104855.GE31388@redhat.com> Content-Type: text/plain; charset="UTF-8" Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Wed, 24 Apr 2019 11:00:09 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: Re: [Qemu-devel] [PATCH 5/6] tests: Run the iotests during "make check" again X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , Ed Maste , qemu-block@nongnu.org, =?UTF-8?Q?Philippe_Mathieu-Daud=c3=a9?= , qemu-devel@nongnu.org, Max Reitz , =?UTF-8?Q?Alex_Benn=c3=a9e?= , Li-Wen Hsu Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Message-ID: <20190424105959.7IxVx6blaHvMvAXFussK7kvbUaMy7-OA8zHEs82FRTI@z> On 24/04/2019 12.48, Daniel P. Berrang=C3=A9 wrote: > On Wed, Apr 24, 2019 at 12:37:46PM +0200, Thomas Huth wrote: >> People often forget to run the iotests before submitting patches or >> pull requests - this is likely due to the fact that we do not run the >> tests during our mandatory "make check" tests yet. Now that we've got >> a new "ci" group of iotests that should be fine to run in every enviro= n- >> ment, it should be OK to enable the iotests during "make check" again. >> Thus we now run the "ci" tests by default from the qemu-iotests-quick.= sh >> script, and only use the former "quick" group (that contains some test= s >> that are failing in some environments) when the user decided to run >> "make check SPEED=3Dthorough" or something similar. >> >> Signed-off-by: Thomas Huth >> --- >> tests/Makefile.include | 2 +- >> tests/qemu-iotests-quick.sh | 17 ++++++++++++++++- >> 2 files changed, 17 insertions(+), 2 deletions(-) >=20 >> diff --git a/tests/qemu-iotests-quick.sh b/tests/qemu-iotests-quick.sh >> index 0e554bb972..416b3fc48b 100755 >> --- a/tests/qemu-iotests-quick.sh >> +++ b/tests/qemu-iotests-quick.sh >=20 > s/-quick// perhaps, since it no longer only runs the "quick" tests. >=20 >> @@ -1,8 +1,23 @@ >> #!/bin/sh >> =20 >> +# Honor the SPEED environment variable, just like we do it for the qt= ests. >> +# The default is to run all tests that still work fine in a CI enviro= nments, >> +# but if the user set SPEED=3Dslow or SPEED=3Dthorough, we also run a= ll other >> +# tests that are still marked as "quick" >> +if [ "$SPEED" =3D "slow" -o "$SPEED" =3D "thorough" ]; then >> + group=3Dquick >=20 > When user asks for "slow" tests we run the "quick" tests, which > presumably implies... >=20 >> +else >> + group=3Dci >=20 > ..."ci" is quicker than the "quick" tests ! >=20 > Confused ? We certainly try to ensure that you will be :-) ... and there is also tests/check-block.sh (which is not used by "make check-block", by the way, but still talks about "make check-block"), which is even slower than the "ci" and "quick" group... I guess that could be cleaned up, too (why do we need two scripts?) ... just not sure how the final result should look like and if it should be part of this patch series or rather something separate. Max, Kevin, what do you think? Maybe we should merge the two scripts, and depending on the SPEED variable, we run: quick =3D> "ci" group thorough =3D> "quick" group slow =3D> old behaviour of the check-block.sh script ? Thomas