All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ian Jackson <ian.jackson@eu.citrix.com>
To: <xen-devel@lists.xenproject.org>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Subject: [OSSTEST PATCH 44/60] sg-report-job-history (nfc): Drop $hostsq query
Date: Fri, 14 Aug 2020 18:21:49 +0100	[thread overview]
Message-ID: <20200814172205.9624-45-ian.jackson@eu.citrix.com> (raw)
In-Reply-To: <20200814172205.9624-1-ian.jackson@eu.citrix.com>

We have eliminated all the users of @hostvarcols before @hostvarcols2
is calculated from the row data.

The query which produces this is very slow and can't be cached.  We
can abolish it now and just use the @hostvarcols2 calculation.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 22 +---------------------
 1 file changed, 1 insertion(+), 21 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 5d913b96..5684ac02 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -147,25 +147,6 @@ END
 END
     $flightsq->execute(@params);
 
-    my $hostsq= db_prepare(<<END);
-        SELECT DISTINCT name
-	 FROM runvars
-	 JOIN flights USING (flight)
-	WHERE ($cond)
-	  AND flight >= COALESCE(
-             (
-	      SELECT flight $fromstuff
-	      LIMIT 1 OFFSET $offset
-	     ), 0)
-     ORDER BY name;
-END
-    $hostsq->execute(@params, @params); # sql text contains $cond twice
-    my @hostvarcols;
-    while (my ($hostvar) = $hostsq->fetchrow_array()) {
-	next unless $hostvar =~ m/(^|_)host$/;
-	push @hostvarcols, $hostvar;
-    }
-
     my $hostq= db_prepare(<<END);
         SELECT name, val
           FROM runvars
@@ -213,8 +194,7 @@ END
     foreach my $ri (@test_rows) {
 	$hostvarnames{$_}=1 foreach keys %{ $ri->{Hosts} }
     }
-    my @hostvarcols2 = sort keys %hostvarnames;
-    die unless "@hostvarcols" eq "@hostvarcols2";
+    my @hostvarcols = sort keys %hostvarnames;
 
     my $osstestverq= db_prepare(<<END);
         SELECT DISTINCT harness
-- 
2.11.0



  parent reply	other threads:[~2020-08-14 17:27 UTC|newest]

Thread overview: 61+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 01/60] history reporting (nfc): Do not key cache on hostname any more Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 02/60] history reporting (nfc): Add some test runes to the notes Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 03/60] history reporting: Delete two debug prints Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 04/60] history reporting (nfc): Refactor to generalise, cache_read_existing Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 05/60] history reporting (nfc): Make cache_write_entry into a top-level sub Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 06/60] history reporting (nfc): Rename jobquery to cacheable_query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 07/60] history reporting (nfc): Break out cache_row_lookup_prep Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 08/60] history reporting (nfc): Rename "existing" to "previous" Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 09/60] history reporting (nfc): Provide cacheable_fn Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 10/60] history reporting (nfc): Bind by name in cacheable_query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 11/60] history reporting (nfc): Rename $cachehits to $rows_hit Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 12/60] history reporting (nfc): Record query-specific stats Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 13/60] history reporting (nfc): Move cacheable_* further up the file Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 14/60] history reporting (nfc): Use cacheable_fn for power methods Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 15/60] history reporting (nfc): Introduce cache_set_key_cols Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 16/60] history reporting (nfc): Record more row-specific stats Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 17/60] history reporting: Cache stats reporting: Centralise and rework Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 18/60] history reporting (nfc): Remove now-obsolete stats variables Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 19/60] history reporting (nfc): Introduce empty HistoryReport module Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 20/60] history reporting (nfc): Move cache code into " Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 21/60] history reporting (nfc): Rename some module variables, remove "cache" Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 22/60] history reporting: Skip undefined keys Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 23/60] history reporting (nfc): Documentation for the new module Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 24/60] history reporting: Cache data limit now in History module Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 25/60] history reporting: Print debug for cache misses Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 26/60] history reporting: Improve an error message slightly Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 27/60] sg-report-host-history: Write cache entry for unfinished jobs Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 28/60] parallel by fork: Break out into HistoryReport Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 29/60] parallel by fork: Disconnect $dbh_tests as well as undefing it Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 30/60] parallel by fork: Fix a variable name to $task Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 31/60] sg-report-job-history: Prep for fork: Move $buildsq query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 32/60] sg-report-job-history: Prep for fork: Move $revisionsq query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 33/60] sg-report-job-history: Use fork-based parallelism Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 34/60] sg-report-job-history: Use one child per report Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 35/60] sg-report-job-history (nfc): Have main program decide HTML filename Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 36/60] sg-report-job-history: Always write HTML output Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 37/60] sg-report-job-history (nfc): Remove needless conditional Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 38/60] history reporting (nfc): Add another test rune to the notes Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 39/60] history reporting (nfc): Make cacheable_fn work without cache Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 40/60] sg-report-job-history: Refactor "ALL" handling Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 41/60] sg-report-job-history (nfc): Make $ri->{Hosts} a hash Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 42/60] sg-report-job-history (nfc): Add new hostvarcols calculation Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 43/60] sg-report-job-history (nfc): Query hosts runvars in one go Ian Jackson
2020-08-14 17:21 ` Ian Jackson [this message]
2020-08-14 17:21 ` [OSSTEST PATCH 45/60] history reporting (nfc): Provide cache_set_task_print Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 46/60] sg-report-job-history: Introduce use of cache, for hosts query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 47/60] history reporting (nfc): Break out url_quote Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 48/60] history reporting (nfc): Break out url_unquote Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 49/60] history reporting (nfc): Break out $url_ok_chars Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 50/60] history reporting (nfc): url-quoting: quote = too Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 51/60] history reporting (nfc): Quote keys too Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 52/60] sg-report-job-history: Cache the per-flight revisions Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 53/60] sg-report-job-history (nfc): Refactor osstestrevs code Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 54/60] sg-report-job-history: Cache osstestrevs Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 55/60] history reporting: Break out minflight_by_time Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 56/60] sg-report-job-history (nfc): Abolish $fromstuff Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 57/60] sg-report-job-history: Cache report_run_getinfo Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 58/60] sg-report-host-history: " Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 59/60] sg-report-job-history: Provide --time-limit Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 60/60] sg-report-job-history: Increase default limit Ian Jackson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200814172205.9624-45-ian.jackson@eu.citrix.com \
    --to=ian.jackson@eu.citrix.com \
    --cc=xen-devel@lists.xenproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.