From: Ian Jackson <ian.jackson@eu.citrix.com>
To: xen-devel@lists.xenproject.org
Cc: Ian Jackson <ian.jackson@eu.citrix.com>,
George Dunlap <George.Dunlap@citrix.com>
Subject: [OSSTEST PATCH 06/14] sg-report-flight: Use WITH clause to use index for $anypassq
Date: Tue, 21 Jul 2020 19:41:57 +0100 [thread overview]
Message-ID: <20200721184205.15232-7-ian.jackson@eu.citrix.com> (raw)
In-Reply-To: <20200721184205.15232-1-ian.jackson@eu.citrix.com>
Perf: runtime of my test case now ~11s
Example query before (from the Perl DBI trace):
SELECT * FROM flights JOIN steps USING (flight)
WHERE (branch='xen-unstable')
AND job=? and testid=? and status='pass'
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
LIMIT 1
After:
WITH s AS
(
SELECT * FROM steps
WHERE job=? and testid=? and status='pass'
)
SELECT * FROM flights JOIN s USING (flight)
WHERE (branch='xen-unstable')
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
LIMIT 1
In both cases with bind vars:
"test-amd64-i386-xl-pvshim"
"guest-start"
Diff to the query:
- SELECT * FROM flights JOIN steps USING (flight)
+ WITH s AS
+ (
+ SELECT * FROM steps
+ WHERE job=? and testid=? and status='pass'
+ )
+ SELECT * FROM flights JOIN s USING (flight)
WHERE (branch='xen-unstable')
- AND job=? and testid=? and status='pass'
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
LIMIT 1
CC: George Dunlap <George.Dunlap@citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
schema/steps-job-index.sql | 2 +-
sg-report-flight | 14 ++++++++++++--
2 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/schema/steps-job-index.sql b/schema/steps-job-index.sql
index 07dc5a30..2c33af72 100644
--- a/schema/steps-job-index.sql
+++ b/schema/steps-job-index.sql
@@ -1,4 +1,4 @@
--- ##OSSTEST## 006 Preparatory
+-- ##OSSTEST## 006 Needed
--
-- This index helps sg-report-flight find if a test ever passed.
diff --git a/sg-report-flight b/sg-report-flight
index b5398573..b8d948da 100755
--- a/sg-report-flight
+++ b/sg-report-flight
@@ -849,10 +849,20 @@ sub justifyfailures ($;$) {
my @failures= values %{ $fi->{Failures} };
+ # In psql 9.6 this WITH clause makes postgresql do the steps query
+ # first. This is good because if this test never passed we can
+ # determine that really quickly using the new index, without
+ # having to scan the flights table. (If the test passed we will
+ # probably not have to look at many flights to find one, so in
+ # that case this is not much worse.)
my $anypassq= <<END;
- SELECT * FROM flights JOIN steps USING (flight)
+ WITH s AS
+ (
+ SELECT * FROM steps
+ WHERE job=? and testid=? and status='pass'
+ )
+ SELECT * FROM flights JOIN s USING (flight)
WHERE $branches_cond_q
- AND job=? and testid=? and status='pass'
AND $blessingscond
LIMIT 1
END
--
2.20.1
next prev parent reply other threads:[~2020-07-21 18:42 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-21 18:41 [OSSTEST PATCH 00/14] Flight report performance improvements Ian Jackson
2020-07-21 18:41 ` [OSSTEST PATCH 01/14] sg-report-flight: Add a comment re same-flight search narrowing Ian Jackson
2020-07-21 18:41 ` [OSSTEST PATCH 02/14] sg-report-flight: Sort failures by job name as last resort Ian Jackson
2020-07-21 18:41 ` [OSSTEST PATCH 03/14] schema: Provide indices for sg-report-flight Ian Jackson
2020-07-21 18:41 ` [OSSTEST PATCH 04/14] sg-report-flight: Ask the db for flights of interest Ian Jackson
2020-07-22 12:10 ` George Dunlap
2020-07-22 14:03 ` Ian Jackson
2020-07-21 18:41 ` [OSSTEST PATCH 05/14] sg-report-flight: Use WITH to use best index use for $flightsq Ian Jackson
2020-07-22 12:47 ` George Dunlap
2020-07-22 14:06 ` Ian Jackson
2020-07-21 18:41 ` Ian Jackson [this message]
2020-07-27 16:15 ` [OSSTEST PATCH 06/14] sg-report-flight: Use WITH clause to use index for $anypassq George Dunlap
2020-07-31 10:41 ` Ian Jackson
2020-07-21 18:41 ` [OSSTEST PATCH 07/14] sg-report-flight: Use the job row from the intitial query Ian Jackson
2020-07-21 18:41 ` [OSSTEST PATCH 08/14] Executive: Use index for report__find_test Ian Jackson
2020-07-22 11:33 ` George Dunlap
2020-07-22 13:49 ` Ian Jackson
2020-07-21 18:42 ` [OSSTEST PATCH 09/14] duration_estimator: Ignore truncated jobs unless we know the step Ian Jackson
2020-07-21 18:42 ` [OSSTEST PATCH 10/14] duration_estimator: Introduce some _qtxt variables Ian Jackson
2020-07-21 18:42 ` [OSSTEST PATCH 11/14] duration_estimator: Explicitly provide null in general host q Ian Jackson
2020-07-21 18:42 ` [OSSTEST PATCH 12/14] duration_estimator: Return job column in first query Ian Jackson
2020-07-21 18:42 ` [OSSTEST PATCH 13/14] duration_estimator: Move $uptincl_testid to separate @x_params Ian Jackson
2020-07-21 18:42 ` [OSSTEST PATCH 14/14] duration_estimator: Move duration query loop into database Ian Jackson
2020-07-27 17:43 ` George Dunlap
2020-07-31 10:39 ` Ian Jackson
2020-07-31 10:45 ` George Dunlap
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200721184205.15232-7-ian.jackson@eu.citrix.com \
--to=ian.jackson@eu.citrix.com \
--cc=George.Dunlap@citrix.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).