From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15989C56202 for ; Thu, 26 Nov 2020 10:18:41 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5799C2145D for ; Thu, 26 Nov 2020 10:18:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="fb+osCLJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5799C2145D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:54092 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kiEMJ-0000w4-4N for qemu-devel@archiver.kernel.org; Thu, 26 Nov 2020 05:18:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:43020) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kiELg-0000Ru-CN for qemu-devel@nongnu.org; Thu, 26 Nov 2020 05:18:00 -0500 Received: from mail-ej1-x62e.google.com ([2a00:1450:4864:20::62e]:37370) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kiELe-0005hd-AO for qemu-devel@nongnu.org; Thu, 26 Nov 2020 05:17:59 -0500 Received: by mail-ej1-x62e.google.com with SMTP id z5so2150640ejp.4 for ; Thu, 26 Nov 2020 02:17:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=noZcFbuDvMDzVWlS4J5l58/Q7GYdjsUIYznJBF/nQks=; b=fb+osCLJRooARVnDAuG83r8fqlQUVRSPGergXFhYuwEvEgUAjzrWfb8uQv2UkxF5bB WnYKtIiDfDNtvOLEtt5iia97Y9EM/M6eO7aZtevJZeHjEbm5g1nY485e+7wD/RiG717C tLWfjcfAImNWTjiuYDpxeGKjZrEePz5YaOGO4rpQRku3JPf2dAgEhjzkoRgjshJ+lmmJ gaV2NnqaNdhyS3QkugsuN9KOtWoULq0LYUntRnyWWJuHs9kwLHj9QHuzd/06p0Nd46TH p5HrfsxMY6AUjp18LrJRpVLCejbDkdSQ3Bhy8jDpEetqvG/P3GgcQ2E7k4ZBITGSxrCe pssg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=noZcFbuDvMDzVWlS4J5l58/Q7GYdjsUIYznJBF/nQks=; b=bLehSd0qd/hTiObfH1dSp1uh2kvZYkOVzokMriJBmOUQJN0mTFVBnJhgXtajx+TzIM ZOTDZNE4yMGDwIR9kWN6a3aAVl4OeVlgUYycoEx6smunQ3x4p3HRz5Ptn7RMIbK1XwB6 lv+5w6S2e1NCRcH55jbqo1v8iO36s7cXBi7kJ88bfl5nGOLr6n2IEwXgxpFeXMlyzTzd CrcEUei5nOyvXrgq1TsuQW+MMlV1v/38UINlS80AzEe2Olt/zUUsrhdJGalLgls01pSo AnHgXoLluqXeAJPNCerOPhwQy22be+cgD3aTABgGA84iQM2aMTWkhFCmIQk5GLkXXDlq UOjw== X-Gm-Message-State: AOAM532NY6SB7TBBobcoXUfCWsCRiKAr3Wfp6Fo6xtoP14Phwsh9qUlc tbYYSdoZRoLy/97Y5UPrF2qIQFLWzz57nT/9RZvQ6w== X-Google-Smtp-Source: ABdhPJxkp8DNoJe+WLS3+ZYrbSzyW8u18Zb0dy25rlFt6teaAYoYhaeKeIvQMCYvzWevI6oxLVqfnEPgXfJV7dmhhYo= X-Received: by 2002:a17:906:dc4:: with SMTP id p4mr2023912eji.56.1606385875340; Thu, 26 Nov 2020 02:17:55 -0800 (PST) MIME-Version: 1.0 References: <3a664806-8aa3-feb4-fb30-303d303217a8@redhat.com> In-Reply-To: <3a664806-8aa3-feb4-fb30-303d303217a8@redhat.com> From: Peter Maydell Date: Thu, 26 Nov 2020 10:17:44 +0000 Message-ID: Subject: Re: Proposal for a regular upstream performance testing To: =?UTF-8?B?THVrw6HFoSBEb2t0b3I=?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::62e; envelope-from=peter.maydell@linaro.org; helo=mail-ej1-x62e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Charles Shih , Aleksandar Markovic , QEMU Developers , Stefan Hajnoczi Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Thu, 26 Nov 2020 at 08:13, Luk=C3=A1=C5=A1 Doktor w= rote: > The goal of this initiative is to detect system-wide performance > regressions as well as improvements early, ideally pin-point the > individual commits and notify people that they should fix things. > All in upstream and ideally with least human interaction possible. So, my general view on this is that automated testing of performance is nice, but unless it also comes with people willing to do the work of identifying and fixing the causes of performance regressions there's a risk that it degrades into another automated email or set of graphs that nobody pays much attention to (outside of the obvious "oops this commit dropped us by 50%" mistakes). As with fuzz testing, I'm a bit wary of setting up an automated system that just pushes work onto humans who already have full plates. Is RedHat also planning to set performance requirements and assign engineers to keep us from falling below them ? thanks -- PMM