From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40FF4C433DF for ; Tue, 21 Jul 2020 18:43:08 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 19A2120717 for ; Tue, 21 Jul 2020 18:43:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 19A2120717 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jxxE9-0002Dp-TO; Tue, 21 Jul 2020 18:42:57 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jxxE8-0001xV-NN for xen-devel@lists.xenproject.org; Tue, 21 Jul 2020 18:42:56 +0000 X-Inumbo-ID: e8d9001a-cb81-11ea-85a2-bc764e2007e4 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id e8d9001a-cb81-11ea-85a2-bc764e2007e4; Tue, 21 Jul 2020 18:42:22 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1jxxDZ-0001u7-A6; Tue, 21 Jul 2020 19:42:21 +0100 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH 08/14] Executive: Use index for report__find_test Date: Tue, 21 Jul 2020 19:41:59 +0100 Message-Id: <20200721184205.15232-9-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200721184205.15232-1-ian.jackson@eu.citrix.com> References: <20200721184205.15232-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" After we refactor this query then we can enable the index use. (Both of these things together in this commit because I haven't perf tested the version with just the refactoring.) (We have provided an index that can answer this question really quickly if a version is specified. But the query planner couldn't see that because it works without seeing the bind variables, so doesn't know that the value of name is going to be suitable for this index.) * Convert the two EXISTS subqueries into JOIN/AND with a DISTINCT clause naming the fields on flights, so as to replicate the previous result rows. Then do $selection field last. The subquery is a convenient way to let this do the previous thing for all the values of $selection (including, notably, *). * Add the additional AND clause for r.name, which has no logical effect given the actual values of name, enabling the query planner to use this index. Perf: In my test case the sg-report-flight runtime is now ~8s. I am reasonably confident that this will not make other use cases of this code worse. Perf: runtime of my test case now ~11s Example query before (from the Perl DBI trace): SELECT * FROM flights f WHERE EXISTS ( SELECT 1 FROM runvars r WHERE name=? AND val=? AND r.flight=f.flight AND ( (CASE WHEN (r.job) LIKE 'build-%-prev' THEN 'xprev' WHEN ((r.job) LIKE 'build-%-freebsd' AND 'x' = 'freebsdbuildjob') THEN 'DISCARD' ELSE '' END) = '') ) AND ( (TRUE AND flight <= 151903) AND (blessing='real') ) AND (branch=?) ORDER BY flight DESC LIMIT 1 After: SELECT * FROM ( SELECT DISTINCT flight, started, blessing, branch, intended FROM flights f JOIN runvars r USING (flight) WHERE name=? AND name LIKE 'revision_%' AND val=? AND r.flight=f.flight AND ( (CASE WHEN (r.job) LIKE 'build-%-prev' THEN 'xprev' WHEN ((r.job) LIKE 'build-%-freebsd' AND 'x' = 'freebsdbuildjob') THEN 'DISCARD' ELSE '' END) = '') AND ( (TRUE AND flight <= 151903) AND (blessing='real') ) AND (branch=?) ) AS sub WHERE TRUE ORDER BY flight DESC LIMIT 1 In both cases with bind vars: 'revision_xen' '165f3afbfc3db70fcfdccad07085cde0a03c858b' "xen-unstable" Diff to the example query: @@ -1,10 +1,10 @@ SELECT * + FROM ( SELECT DISTINCT + flight, started, blessing, branch, intended FROM flights f - WHERE - EXISTS ( - SELECT 1 - FROM runvars r + JOIN runvars r USING (flight) WHERE name=? + AND name LIKE 'revision_%' AND val=? AND r.flight=f.flight AND ( (CASE @@ -14,8 +14,8 @@ ELSE '' END) = '') - ) AND ( (TRUE AND flight <= 151903) AND (blessing='real') ) AND (branch=?) +) AS sub WHERE TRUE ORDER BY flight DESC LIMIT 1 CC: George Dunlap Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 20 ++++++++------------ schema/runvars-revision-index.sql | 2 +- 2 files changed, 9 insertions(+), 13 deletions(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index c3dc1261..c272e9f2 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -415,37 +415,32 @@ sub report__find_test ($$$$$$$) { my $querytext = <