From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C67B8C07E95 for ; Mon, 19 Jul 2021 15:15:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 43EB760249 for ; Mon, 19 Jul 2021 15:15:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 43EB760249 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:36044 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Uz6-0007Ls-8G for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 11:15:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42154) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5UxL-00052r-FJ for qemu-devel@nongnu.org; Mon, 19 Jul 2021 11:13:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:52334) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5UxJ-0005oV-90 for qemu-devel@nongnu.org; Mon, 19 Jul 2021 11:13:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626707596; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RVUpuAqFGFErV92aNOoehfAOEytBBdNJXM5obZRl5KM=; b=TeF9GJDeTj6CFBy9f6t+0I5AKA4D6LJAab14KQFl13U3L0htI4dgDzNGEYfUj3jjCbYw9e +V2nLvIvahjuyYI2TbJcnPWmPVBbGaqx4iKnFjSstQ8pgfO3YuAAHfZnQefxbkLt6++UT/ tPWdZ4jADboP9/0Weq9FQY02WshYlqw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-142-xpxvMQKdMtCyMYl3Ox-jEg-1; Mon, 19 Jul 2021 11:13:15 -0400 X-MC-Unique: xpxvMQKdMtCyMYl3Ox-jEg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1FF1F1084F60; Mon, 19 Jul 2021 15:13:14 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-114-187.ams2.redhat.com [10.36.114.187]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E6E155C1C5; Mon, 19 Jul 2021 15:13:09 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 6791711326B9; Mon, 19 Jul 2021 17:13:08 +0200 (CEST) From: Markus Armbruster To: Igor Mammedov Subject: Re: [PATCH v3 0/9] tests: Add test cases for TPM 1.2 ACPI tables References: <20210712150949.165725-1-stefanb@linux.vnet.ibm.com> <36bcf543-0b56-7e2f-26e7-648ca3cf58dd@linux.ibm.com> <87a6mpez2b.fsf@dusky.pond.sub.org> <97703096-ad9d-f676-ffcb-46ad4bf340c2@redhat.com> <87a6modt46.fsf@dusky.pond.sub.org> <20210719153837.46fdef08@redhat.com> <871r7ummz8.fsf@dusky.pond.sub.org> <20210719164430.161e9d1e@redhat.com> Date: Mon, 19 Jul 2021 17:13:08 +0200 In-Reply-To: <20210719164430.161e9d1e@redhat.com> (Igor Mammedov's message of "Mon, 19 Jul 2021 16:44:30 +0200") Message-ID: <87v956jq0r.fsf@dusky.pond.sub.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=armbru@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=170.10.133.124; envelope-from=armbru@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.469, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , Stefan Berger , QEMU Developers , =?utf-8?Q?Marc-Andr=C3=A9?= Lureau , Paolo Bonzini , Philippe =?utf-8?Q?Mathieu-Daud=C3=A9?= , Stefan Berger Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Igor Mammedov writes: > On Mon, 19 Jul 2021 15:56:16 +0200 > Philippe Mathieu-Daud=C3=A9 wrote: > >> On Mon, Jul 19, 2021 at 3:50 PM Markus Armbruster wr= ote: >> > Igor Mammedov writes: =20 >> > > On Thu, 15 Jul 2021 07:49:13 +0200 >> > > Markus Armbruster wrote: =20 >> > >> Philippe Mathieu-Daud=C3=A9 writes: =20 >>=20 >> > >> >>> IMO the "right" solution is to check via QMP if TMP is supporte= d >> > >> >>> or not. This is now doable since commit caff255a546 ("tpm: Retu= rn >> > >> >>> QMP error when TPM is disabled in build"). >> > >> >>> >> > >> >>> Long term we'd like to decouple the tests/ build from the vario= us >> > >> >>> QEMU configurations, and build the tests once. =20 >> > >> >> >> > >> >> This argument applies only to macros from target-specific header= s like >> > >> >> $TARGET-config-target.h, not to macros from config-host.h. #ifd= ef >> > >> >> CONFIG_TPM should be fine, shouldn't it? =20 >> > >> > >> > >> > Some definitions depend on the host (OS, libraries installed, ...= ), >> > >> > others depend on the --enable/--disable ./configure options. >> > >> > >> > >> > IMO it would be nice if we could get qtests independent of the la= tter. =20 >> > >> >> > >> Why? =20 >> > > >> > > In another mail-thread Philippe mentioned that there is desire >> > > to use qtest out of tree to test other QEMU binaries. >> > > >> > > However, just probing for features at runtime aren't going >> > > to help with the goal as tests are tailored for the latest >> > > CLI/QMP/ABI. To make it work we would have practically >> > > introduce versioned tests. >> > > >> > > So I wonder why one external acceptance-tests suite is not >> > > sufficient, that we would want to hijack relatively simple >> > > internal qtest at expense of increased resources needed to >> > > run/write unit tests. =20 >> > >> > Yes. qtest was not designed for use with anything but HEAD, and I dou= bt >> > we can make it fit such uses at reasonable expense. =20 >>=20 >> One HEAD but multiple configurations... > > Even assuming reconfigure won't cause world rebuild, > It will be a win only if number of configuration probes > is small. > However it doesn't scale for large numbers and it might be > faster to rebuild affected tests in the end. (worst case: #probes * #targ= ets) > I wonder if we can do probing once & cache it somewhere to avoid ^^^. As far as I can tell, the time to compile these tests is dwarved several times over by the time to run them. In other words, most of the waste to optimize out hides in the test programs, not in compiling them. >> If you want to simplify human time, can we simply run qtests once per >> arch/OS but with all features enabled? Otherwise skip qtests? How build configuration and tests interact is poorly understood. We instead use brute force: run all the tests for all configurations, always. To do better without sacrificing coverage, we need to understand better. Sadly, I don't see that happening.