[OSSTEST PATCH 00/60] Speed up sg-report-job-history

[OSSTEST PATCH 00/60] Speed up sg-report-job-history

[Ian Jackson]

I think this is the final section of my work to speed up osstest's
reporting db queries etc.

After this is pushed and working, I will look and see what the
interval is now, between one flight finishing and the next starting,
to see ig there is more to be done.

Ian Jackson (60):
  history reporting (nfc): Do not key cache on hostname any more
  history reporting (nfc): Add some test runes to the notes
  history reporting: Delete two debug prints
  history reporting (nfc): Refactor to generalise, cache_read_existing
  history reporting (nfc): Make cache_write_entry into a top-level sub
  history reporting (nfc): Rename jobquery to cacheable_query
  history reporting (nfc): Break out cache_row_lookup_prep
  history reporting (nfc): Rename "existing" to "previous"
  history reporting (nfc): Provide cacheable_fn
  history reporting (nfc): Bind by name in cacheable_query
  history reporting (nfc): Rename $cachehits to $rows_hit
  history reporting (nfc): Record query-specific stats
  history reporting (nfc): Move cacheable_* further up the file
  history reporting (nfc): Use cacheable_fn for power methods
  history reporting (nfc): Introduce cache_set_key_cols
  history reporting (nfc): Record more row-specific stats
  history reporting: Cache stats reporting: Centralise and rework
  history reporting (nfc): Remove now-obsolete stats variables
  history reporting (nfc): Introduce empty HistoryReport module
  history reporting (nfc): Move cache code into HistoryReport module
  history reporting (nfc): Rename some module variables, remove "cache"
  history reporting: Skip undefined keys
  history reporting (nfc): Documentation for the new module
  history reporting: Cache data limit now in History module
  history reporting: Print debug for cache misses
  history reporting: Improve an error message slightly
  sg-report-host-history: Write cache entry for unfinished jobs
  parallel by fork: Break out into HistoryReport
  parallel by fork: Disconnect $dbh_tests as well as undefing it
  parallel by fork: Fix a variable name to $task
  sg-report-job-history: Prep for fork: Move $buildsq query
  sg-report-job-history: Prep for fork: Move $revisionsq query
  sg-report-job-history: Use fork-based parallelism
  sg-report-job-history: Use one child per report
  sg-report-job-history (nfc): Have main program decide HTML filename
  sg-report-job-history: Always write HTML output
  sg-report-job-history (nfc): Remove needless conditional
  history reporting (nfc): Add another test rune to the notes
  history reporting (nfc): Make cacheable_fn work without cache
  sg-report-job-history: Refactor "ALL" handling
  sg-report-job-history (nfc): Make $ri->{Hosts} a hash
  sg-report-job-history (nfc): Add new hostvarcols calculation
  sg-report-job-history (nfc): Query hosts runvars in one go
  sg-report-job-history (nfc): Drop $hostsq query
  history reporting (nfc): Provide cache_set_task_print
  sg-report-job-history: Introduce use of cache, for hosts query
  history reporting (nfc): Break out url_quote
  history reporting (nfc): Break out url_unquote
  history reporting (nfc): Break out $url_ok_chars
  history reporting (nfc): url-quoting: quote = too
  history reporting (nfc): Quote keys too
  sg-report-job-history: Cache the per-flight revisions
  sg-report-job-history (nfc): Refactor osstestrevs code
  sg-report-job-history: Cache osstestrevs
  history reporting: Break out minflight_by_time
  sg-report-job-history (nfc): Abolish $fromstuff
  sg-report-job-history: Cache report_run_getinfo
  sg-report-host-history: Cache report_run_getinfo
  sg-report-job-history: Provide --time-limit
  sg-report-job-history: Increase default limit

 Osstest/HistoryReport.pm | 301 +++++++++++++++++++++++++++++++++++++++++++++
 runes                    |  10 ++
 sg-report-host-history   | 216 +++++++-------------------------
 sg-report-job-history    | 313 +++++++++++++++++++++++++----------------------
 4 files changed, 520 insertions(+), 320 deletions(-)
 create mode 100644 Osstest/HistoryReport.pm

-- 
2.11.0

[OSSTEST PATCH 01/60] history reporting (nfc): Do not key cache on hostname any more

[Ian Jackson]

Now we process only one host at a time, so we don't need to
distinguish.

Replace all references to $tcache with %cache.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 380f8fac..50cc5b58 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -76,7 +76,7 @@ our $restrictflight_cond = restrictflight_cond();
 our $flightcond;
 our $minflight;
 
-our %hcaches;
+our %cache;
 
 sub read_existing_logs ($) {
     my ($hostname) = @_;
@@ -86,8 +86,7 @@ sub read_existing_logs ($) {
         return if $!==ENOENT;
         die "failed to open $html_file: $!";
     }
-    my $tcache = { };
-    $hcaches{$hostname} = $tcache;
+    %cache = ();
     for (;;) {
         $_ = <H> // last;
         next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
@@ -106,7 +105,7 @@ sub read_existing_logs ($) {
 #	    print DEBUG "GOTCACHE $hostname $k\n";
 	}
 #	print DEBUG "GOTCACHE $hostname \@ $jr->{flight} $jr->{job} $jr->{status},$jr->{name}\n";
-	$tcache->{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr;
+	$cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr;
     }
     close H;
 }
@@ -253,8 +252,6 @@ END
     my $inrows = $hosts{$hostname};
     print DEBUG "FOUND ", (scalar @$inrows), " ROWS for $hostname\n";
 
-    my $tcache = $hcaches{$hostname};
-
     # Each entry in @$inrows is a $jr, which is a hash
     # It has keys for the result columns in mainquery
     # It also has keys '%<letter>' (yes, with a literal '%')
@@ -267,7 +264,7 @@ END
 	#print DEBUG "JOB $jr->{flight}.$jr->{job} ";
 
 	my $cacherow =
-	    $tcache->{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}};
+	    $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}};
 	if ($cacherow) {
 	    $jr = $cacherow;
 	    $cachehits++;
@@ -285,7 +282,7 @@ END
     }
 
     print DEBUG "CACHE $hostname $cachehits / ".(scalar @rows)
-	." of ".(scalar %$tcache)."\n";
+	." of ".(scalar %cache)."\n";
 
     my $write_cache_entry = sub {
 	my ($jr) = @_;
-- 
2.11.0

[OSSTEST PATCH 02/60] history reporting (nfc): Add some test runes to the notes

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 runes | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/runes b/runes
index 27ece2af..8139f955 100644
--- a/runes
+++ b/runes
@@ -3,3 +3,11 @@ select * from resource_sharing s full outer join resources r on r.restype = 'sha
 
 select * from jobs full outer join steps using (job, flight) where flight=2364 order by job, stepno;
 
+
+
+
+>v; env DBI_TRACE=2=v time ./sg-report-job-history --debug --html-dir=$PWD/d --flight=152223 --max-flight=152223  2>&1 |tee c |ts -s %.s >c.ts
+
+>v; env DBI_TRACE=2=v time ./sg-report-host-history --debug --html-dir=$PWD/e --max-flight=152223 flight:152223  2>&1 |tee c |ts -s %.s >c.ts
+rm -rf d; mkdir d; cp -a e/. d/.; mv c.ts b.ts; mv c b; mv v u
+cp -a d/. e/.
-- 
2.11.0

[OSSTEST PATCH 03/60] history reporting: Delete two debug prints

[Ian Jackson]

These are going to become invalid and I don't think it's important to
save them.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 2 --
 1 file changed, 2 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 50cc5b58..4c0022e8 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -102,9 +102,7 @@ sub read_existing_logs ($) {
 	    my $k = $1;
 	    s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
 	    $ch->{$k} = $_;
-#	    print DEBUG "GOTCACHE $hostname $k\n";
 	}
-#	print DEBUG "GOTCACHE $hostname \@ $jr->{flight} $jr->{job} $jr->{status},$jr->{name}\n";
 	$cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr;
     }
     close H;
-- 
2.11.0

[OSSTEST PATCH 04/60] history reporting (nfc): Refactor to generalise, cache_read_existing

[Ian Jackson]

* Introduce @cache_row_key_cols and cache_row_key
* Handle $html_file pathname construction and $read_existing
  at the call site.
* Rename cache_read_existing.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 4c0022e8..83344a40 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -77,11 +77,15 @@ our $flightcond;
 our $minflight;
 
 our %cache;
+our @cache_row_key_cols = qw(flight job status name);
 
-sub read_existing_logs ($) {
-    my ($hostname) = @_;
-    return unless $read_existing;
-    my $html_file = "$htmlout/$hostname.html";
+sub cache_row_key ($) {
+    my ($jr) = @_;
+    return join $; , map { $jr->{$_} } @cache_row_key_cols;
+}
+
+sub cache_read_existing ($) {
+    my ($html_file) = @_;
     if (!open H, $html_file) {
         return if $!==ENOENT;
         die "failed to open $html_file: $!";
@@ -103,7 +107,7 @@ sub read_existing_logs ($) {
 	    s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
 	    $ch->{$k} = $_;
 	}
-	$cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr;
+	$cache{cache_row_key($jr)} = $jr;
     }
     close H;
 }
@@ -261,8 +265,7 @@ END
     foreach my $jr (@$inrows) {
 	#print DEBUG "JOB $jr->{flight}.$jr->{job} ";
 
-	my $cacherow =
-	    $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}};
+	my $cacherow = $cache{cache_row_key($jr)};
 	if ($cacherow) {
 	    $jr = $cacherow;
 	    $cachehits++;
@@ -492,7 +495,7 @@ foreach my $host (sort keys %hosts) {
     my $pid = fork // die $!;
     if (!$pid) {
 	opendb_tests();
-	read_existing_logs($host);
+	cache_read_existing("$htmlout/$host.html") if $read_existing;
 	db_retry($dbh_tests, [], sub {
             mainquery($host);
 	    reporthost $host;
-- 
2.11.0

[OSSTEST PATCH 05/60] history reporting (nfc): Make cache_write_entry into a top-level sub

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 46 +++++++++++++++++++++++-----------------------
 1 file changed, 23 insertions(+), 23 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 83344a40..fe5b7cc2 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -112,6 +112,27 @@ sub cache_read_existing ($) {
     close H;
 }
 
+sub cache_write_entry ($$) {
+    my ($fh, $jr) = @_;
+    print $fh "<!-- osstest-report-reuseable";
+    my $whash = sub {
+	my ($h) = @_;
+	foreach my $k (sort keys %$h) {
+	    next if $k =~ m/^\%/;
+	    $_ = $h->{$k};
+	    s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
+	    printf $fh " %s=%s", $k, $_;
+	}
+    };
+    $whash->($jr);
+    foreach my $hk (sort keys %$jr) {
+	next unless $hk =~ m/^\%/;
+	print $fh " $'";
+	$whash->($jr->{$hk});
+    }
+    print $fh " -->\n";
+}
+
 sub computeflightsrange () {
     if ($flightlimit) {
 	my $minflightsq = db_prepare(<<END);
@@ -285,27 +306,6 @@ END
     print DEBUG "CACHE $hostname $cachehits / ".(scalar @rows)
 	." of ".(scalar %cache)."\n";
 
-    my $write_cache_entry = sub {
-	my ($jr) = @_;
-        print H "<!-- osstest-report-reuseable";
-	my $whash = sub {
-	    my ($h) = @_;
-	    foreach my $k (sort keys %$h) {
-		next if $k =~ m/^\%/;
-		$_ = $h->{$k};
-		s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
-		printf H " %s=%s", $k, $_;
-	    }
-	};
-	$whash->($jr);
-	foreach my $hk (sort keys %$jr) {
-	    next unless $hk =~ m/^\%/;
-	    print H " $'";
-	    $whash->($jr->{$hk});
-	}
-	print H " -->\n";
-    };
-
     @rows = sort { $b->{finished} <=> $a->{finished} } @rows;
 
     print DEBUG "SORTED\n";
@@ -317,7 +317,7 @@ END
 
     foreach my $jr (@rows) {
 	if ($wrote++ >= $limit) {
-	    $write_cache_entry->($jr);
+	    cache_write_entry(\*H, $jr);
 	    next;
 	}
 
@@ -417,7 +417,7 @@ END
         print H "<td>" if !$any_power;
 	print H "</td>\n";
 
-	$write_cache_entry->($jr);
+	cache_write_entry(\*H, $jr);
 
 	print H "</tr>\n\n";
 	$alternate ^= 1;
-- 
2.11.0

[OSSTEST PATCH 06/60] history reporting (nfc): Rename jobquery to cacheable_query

[Ian Jackson]

Prep work.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index fe5b7cc2..a8dae159 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -171,7 +171,7 @@ END
 our $jqcachemisses = 0;
 our $jqtotal = 0;
 
-sub jobquery ($$$) {
+sub cacheable_query ($$$) {
     my ($q, $jr, $cachekey) = @_;
     $jqtotal++;
     $cachekey = '%'.$cachekey;
@@ -292,7 +292,7 @@ END
 	    $cachehits++;
 	}
 
-	my $endedrow = jobquery($endedq, $jr, 'e');
+	my $endedrow = cacheable_query($endedq, $jr, 'e');
 	if (!$endedrow) {
 	    #print DEBUG "no-finished\n";
 	    next;
@@ -322,8 +322,8 @@ END
 	}
 
         #print DEBUG "JR $jr->{flight}.$jr->{job}\n";
-	my $ir = jobquery($infoq, $jr, 'i');
-	my $ar = jobquery($allocdq, $jr, 'a');
+	my $ir = cacheable_query($infoq, $jr, 'i');
+	my $ar = cacheable_query($allocdq, $jr, 'a');
 	my $ident = $jr->{name};
 
 	my $altcolour = report_altcolour($alternate);
-- 
2.11.0

[OSSTEST PATCH 07/60] history reporting (nfc): Break out cache_row_lookup_prep

[Ian Jackson]

Prep work.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index a8dae159..2724ef27 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -112,6 +112,16 @@ sub cache_read_existing ($) {
     close H;
 }
 
+sub cache_row_lookup_prep ($) {
+    my ($jrr) = @_;
+
+    my $cacherow = $cache{cache_row_key($$jrr)};
+    if ($cacherow) {
+	$$jrr = $cacherow;
+	$cachehits++;
+    }
+}
+
 sub cache_write_entry ($$) {
     my ($fh, $jr) = @_;
     print $fh "<!-- osstest-report-reuseable";
@@ -286,11 +296,7 @@ END
     foreach my $jr (@$inrows) {
 	#print DEBUG "JOB $jr->{flight}.$jr->{job} ";
 
-	my $cacherow = $cache{cache_row_key($jr)};
-	if ($cacherow) {
-	    $jr = $cacherow;
-	    $cachehits++;
-	}
+	cache_row_lookup_prep(\$jr);
 
 	my $endedrow = cacheable_query($endedq, $jr, 'e');
 	if (!$endedrow) {
-- 
2.11.0

[OSSTEST PATCH 08/60] history reporting (nfc): Rename "existing" to "previous"

[Ian Jackson]

This seems more idiomatic use of English.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 2724ef27..f37dd4a2 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -32,7 +32,7 @@ our $limit= 2000;
 our $timelimit= 86400 * (366 + 14);
 our $flightlimit;
 our $htmlout = ".";
-our $read_existing=1;
+our $read_previous=1;
 our $doinstall=1;
 our $maxjobs=10;
 our @blessings;
@@ -58,7 +58,7 @@ while (@ARGV && $ARGV[0] =~ m/^-/) {
     } elsif (m/^--html-dir=(.*)$/) {
         $htmlout= $1;
     } elsif (m/^--regenerate$/) {
-        $read_existing= 0;
+        $read_previous= 0;
     } elsif (m/^--no-install$/) {
         $doinstall= 0;
     } elsif (m/^--debug/) {
@@ -84,7 +84,7 @@ sub cache_row_key ($) {
     return join $; , map { $jr->{$_} } @cache_row_key_cols;
 }
 
-sub cache_read_existing ($) {
+sub cache_read_previous ($) {
     my ($html_file) = @_;
     if (!open H, $html_file) {
         return if $!==ENOENT;
@@ -501,7 +501,7 @@ foreach my $host (sort keys %hosts) {
     my $pid = fork // die $!;
     if (!$pid) {
 	opendb_tests();
-	cache_read_existing("$htmlout/$host.html") if $read_existing;
+	cache_read_previous("$htmlout/$host.html") if $read_previous;
 	db_retry($dbh_tests, [], sub {
             mainquery($host);
 	    reporthost $host;
-- 
2.11.0

[OSSTEST PATCH 09/60] history reporting (nfc): Provide cacheable_fn

[Ian Jackson]

Prep work.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index f37dd4a2..90369ce4 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -181,20 +181,27 @@ END
 our $jqcachemisses = 0;
 our $jqtotal = 0;
 
-sub cacheable_query ($$$) {
-    my ($q, $jr, $cachekey) = @_;
+sub cacheable_fn ($$$) {
+    my ($jr, $cachekey, $fn) = @_;
     $jqtotal++;
     $cachekey = '%'.$cachekey;
     my $cached = $jr->{$cachekey};
     if (!$cached) {
 	$jqcachemisses++;
-	$q->execute($jr->{flight}, $jr->{job});
-	$cached = $q->fetchrow_hashref();
+	$cached = $fn->();
 	$jr->{$cachekey} = $cached;
     }
     return $cached;
 }
 
+sub cacheable_query ($$$) {
+    my ($q, $jr, $cachekey) = @_;
+    cacheable_fn($jr, $cachekey, sub {
+	$q->execute($jr->{flight}, $jr->{job});
+	return $q->fetchrow_hashref();
+    });
+}
+
 our %hosts;
 
 sub mainquery ($) {
-- 
2.11.0

[OSSTEST PATCH 10/60] history reporting (nfc): Bind by name in cacheable_query

[Ian Jackson]

cacheable_query used to simply pass $jr->{flight} and ->{job}.  But we
want this to be reuseable for other kinds of query, with different
cache keys.

So bind by name: we expect the caller to use :name placeholders in the
query.  We can then look through the prepared query parameters, and
fish the corresponding values out of $jr.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 90369ce4..8c5cd105 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -197,7 +197,11 @@ sub cacheable_fn ($$$) {
 sub cacheable_query ($$$) {
     my ($q, $jr, $cachekey) = @_;
     cacheable_fn($jr, $cachekey, sub {
-	$q->execute($jr->{flight}, $jr->{job});
+	foreach my $k (keys %{ $q->{ParamTypes} }) {
+	    $k =~ m/^:/ or die "$k ?";
+	    $q->bind_param($k, $jr->{$'} // die "$k ?");
+	}
+	$q->execute();
 	return $q->fetchrow_hashref();
     });
 }
@@ -259,7 +263,7 @@ sub reporthost ($) {
     our $endedq //= db_prepare(<<END);
 	SELECT finished, testid, status AS laststepstatus
 	  FROM steps
-	 WHERE flight=? AND job=? AND finished IS NOT NULL
+	 WHERE flight=:flight AND job=:job AND finished IS NOT NULL
 	 ORDER BY finished DESC
 	 LIMIT 1
 END
@@ -267,13 +271,13 @@ END
     our $infoq //= db_prepare(<<END);
 	SELECT blessing, branch, intended
 	  FROM flights
-	 WHERE flight=? AND ?!='X'
+	 WHERE flight=:flight AND :job != 'X'
 END
 
     our $allocdq //= db_prepare(<<END);
 	SELECT testid, finished, status
 	  FROM steps
-	 WHERE flight=? AND job=?
+	 WHERE flight=:flight AND job=:job
 	   AND (testid='hosts-allocate' OR step='ts-hosts-allocate')
 	 ORDER BY finished ASC
 	 LIMIT 1
-- 
2.11.0

[OSSTEST PATCH 11/60] history reporting (nfc): Rename $cachehits to $rows_hit

[Ian Jackson]

Prep work.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 8c5cd105..705367cf 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -118,7 +118,7 @@ sub cache_row_lookup_prep ($) {
     my $cacherow = $cache{cache_row_key($$jrr)};
     if ($cacherow) {
 	$$jrr = $cacherow;
-	$cachehits++;
+	$rows_hit++;
     }
 }
 
@@ -303,7 +303,7 @@ END
     # The contents of $jr for each job is cached across runs.
 
     my @rows;
-    my $cachehits = 0;
+    my $rows_hit = 0;
     foreach my $jr (@$inrows) {
 	#print DEBUG "JOB $jr->{flight}.$jr->{job} ";
 
@@ -320,7 +320,7 @@ END
 	push @rows, { %$jr, %$endedrow };
     }
 
-    print DEBUG "CACHE $hostname $cachehits / ".(scalar @rows)
+    print DEBUG "CACHE $hostname $rows_hit / ".(scalar @rows)
 	." of ".(scalar %cache)."\n";
 
     @rows = sort { $b->{finished} <=> $a->{finished} } @rows;
-- 
2.11.0

[OSSTEST PATCH 12/60] history reporting (nfc): Record query-specific stats

[Ian Jackson]

Prep work.  Nothing uses these yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/sg-report-host-history b/sg-report-host-history
index 705367cf..32414322 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -77,6 +77,10 @@ our $flightcond;
 our $minflight;
 
 our %cache;
+
+our %q_count;
+our %q_misses;
+
 our @cache_row_key_cols = qw(flight job status name);
 
 sub cache_row_key ($) {
@@ -186,8 +190,10 @@ sub cacheable_fn ($$$) {
     $jqtotal++;
     $cachekey = '%'.$cachekey;
     my $cached = $jr->{$cachekey};
+    $q_count{$cachekey}++;
     if (!$cached) {
 	$jqcachemisses++;
+	$q_misses{$cachekey}++;
 	$cached = $fn->();
 	$jr->{$cachekey} = $cached;
     }
-- 
2.11.0

[OSSTEST PATCH 13/60] history reporting (nfc): Move cacheable_* further up the file

[Ian Jackson]

Pure code motion.  Brings these together with the other work-cacheing
code.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 60 +++++++++++++++++++++++++-------------------------
 1 file changed, 30 insertions(+), 30 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 32414322..43ddd2af 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -88,6 +88,36 @@ sub cache_row_key ($) {
     return join $; , map { $jr->{$_} } @cache_row_key_cols;
 }
 
+our $jqcachemisses = 0;
+our $jqtotal = 0;
+
+sub cacheable_fn ($$$) {
+    my ($jr, $cachekey, $fn) = @_;
+    $jqtotal++;
+    $cachekey = '%'.$cachekey;
+    my $cached = $jr->{$cachekey};
+    $q_count{$cachekey}++;
+    if (!$cached) {
+	$jqcachemisses++;
+	$q_misses{$cachekey}++;
+	$cached = $fn->();
+	$jr->{$cachekey} = $cached;
+    }
+    return $cached;
+}
+
+sub cacheable_query ($$$) {
+    my ($q, $jr, $cachekey) = @_;
+    cacheable_fn($jr, $cachekey, sub {
+	foreach my $k (keys %{ $q->{ParamTypes} }) {
+	    $k =~ m/^:/ or die "$k ?";
+	    $q->bind_param($k, $jr->{$'} // die "$k ?");
+	}
+	$q->execute();
+	return $q->fetchrow_hashref();
+    });
+}
+
 sub cache_read_previous ($) {
     my ($html_file) = @_;
     if (!open H, $html_file) {
@@ -182,36 +212,6 @@ END
     print DEBUG "MINFLIGHT $minflight\n";
 }
 
-our $jqcachemisses = 0;
-our $jqtotal = 0;
-
-sub cacheable_fn ($$$) {
-    my ($jr, $cachekey, $fn) = @_;
-    $jqtotal++;
-    $cachekey = '%'.$cachekey;
-    my $cached = $jr->{$cachekey};
-    $q_count{$cachekey}++;
-    if (!$cached) {
-	$jqcachemisses++;
-	$q_misses{$cachekey}++;
-	$cached = $fn->();
-	$jr->{$cachekey} = $cached;
-    }
-    return $cached;
-}
-
-sub cacheable_query ($$$) {
-    my ($q, $jr, $cachekey) = @_;
-    cacheable_fn($jr, $cachekey, sub {
-	foreach my $k (keys %{ $q->{ParamTypes} }) {
-	    $k =~ m/^:/ or die "$k ?";
-	    $q->bind_param($k, $jr->{$'} // die "$k ?");
-	}
-	$q->execute();
-	return $q->fetchrow_hashref();
-    });
-}
-
 our %hosts;
 
 sub mainquery ($) {
-- 
2.11.0

[OSSTEST PATCH 14/60] history reporting (nfc): Use cacheable_fn for power methods

[Ian Jackson]

Avoids open-coding knowledge of the $jr convention at this use site.
%powers becomes %$powers, so it's a bit noisy.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 43ddd2af..9510757f 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -387,12 +387,8 @@ END
 	my $ri = report_run_getinfo({ %$jr, %$ir });
 	print H "<td $ri->{ColourAttr}>$ri->{Content}</td>\n";
 
-	my %powers;
-	if ($jr->{'%p'}) {
-	    %powers = %{ $jr->{'%p'} };
-	    $runvarq_hits++;
-	} else {
-	    $runvarq_misses++;
+	my $powers = cacheable_fn($jr, 'b', sub {
+            my %powers;
 	    $jrunvarq->execute($jr->{flight}, $jr->{job},
 			       db_quote_like_pattern($ident));
 	    my %runvars;
@@ -404,8 +400,8 @@ END
 		next unless $r =~ m{^\Q${ident}\E_power_};
 		$powers{$'} = $runvars{$r};
 	    }
-	    $jr->{'%p'} = { %powers };
-	}
+	    \%powers;
+	});
 	my $skipped = 0;
         my $any_power = 0;
         my $pr_power_colour = sub {
@@ -421,7 +417,7 @@ END
 	    for (; $skipped; $skipped--) {
 		$pr_power_colour->($grey_mid, " - ");
 	    }
-	    my $how = $powers{$wh};
+	    my $how = $powers->{$wh};
 	    my $colour =
 		$how =~ /PDU/      ? $yellow :
 		$how =~ /Combined/ ? $yellow :
@@ -430,11 +426,11 @@ END
             $pr_power_colour->($colour, " $how ");
 	};
 	foreach my $wh (qw(install recover)) {
-	    $skipped++, next unless exists $powers{$wh};
+	    $skipped++, next unless exists $powers->{$wh};
 	    $pr_power->($wh);
-	    delete $powers{$wh};
+	    delete $powers->{$wh};
 	}
-	foreach my $wh (sort keys %powers) {
+	foreach my $wh (sort keys %$powers) {
 	    $pr_power->($wh);
 	}
         print H "<td>" if !$any_power;
-- 
2.11.0

[OSSTEST PATCH 15/60] history reporting (nfc): Introduce cache_set_key_cols

[Ian Jackson]

This setter function will be needed when this becomes its own module.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 9510757f..07e549a0 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -76,12 +76,14 @@ our $restrictflight_cond = restrictflight_cond();
 our $flightcond;
 our $minflight;
 
+our @cache_row_key_cols;
+
 our %cache;
 
 our %q_count;
 our %q_misses;
 
-our @cache_row_key_cols = qw(flight job status name);
+sub cache_set_key_cols { @cache_row_key_cols = @_; }
 
 sub cache_row_key ($) {
     my ($jr) = @_;
@@ -177,6 +179,8 @@ sub cache_write_entry ($$) {
     print $fh " -->\n";
 }
 
+cache_set_key_cols(qw(flight job status name));
+
 sub computeflightsrange () {
     if ($flightlimit) {
 	my $minflightsq = db_prepare(<<END);
-- 
2.11.0

[OSSTEST PATCH 16/60] history reporting (nfc): Record more row-specific stats

[Ian Jackson]

Prep work.  Nothing uses these yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 07e549a0..32cc883d 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -83,6 +83,10 @@ our %cache;
 our %q_count;
 our %q_misses;
 
+our $rows_previous = 0;
+our $rows_today = 0;
+our $rows_hit = 0;
+
 sub cache_set_key_cols { @cache_row_key_cols = @_; }
 
 sub cache_row_key ($) {
@@ -128,6 +132,7 @@ sub cache_read_previous ($) {
     }
     %cache = ();
     for (;;) {
+	$rows_previous++;
         $_ = <H> // last;
         next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
 	my $jr = {};
@@ -151,6 +156,7 @@ sub cache_read_previous ($) {
 sub cache_row_lookup_prep ($) {
     my ($jrr) = @_;
 
+    $rows_today++;
     my $cacherow = $cache{cache_row_key($$jrr)};
     if ($cacherow) {
 	$$jrr = $cacherow;
@@ -313,7 +319,6 @@ END
     # The contents of $jr for each job is cached across runs.
 
     my @rows;
-    my $rows_hit = 0;
     foreach my $jr (@$inrows) {
 	#print DEBUG "JOB $jr->{flight}.$jr->{job} ";
 
-- 
2.11.0

[OSSTEST PATCH 17/60] history reporting: Cache stats reporting: Centralise and rework

[Ian Jackson]

This uses the new variables instead of the old ones.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 32cc883d..9c70b560 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -185,6 +185,17 @@ sub cache_write_entry ($$) {
     print $fh " -->\n";
 }
 
+sub cache_report_stats ($) {
+    my ($what) = @_;
+    print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today";
+    for my $cachekey (sort keys %q_count) {
+	my $total = $q_count{$cachekey};
+	my $hits = $total - ($q_misses{$cachekey} // 0);
+	print ::DEBUG " $cachekey=$hits/$total";
+    }
+    print ::DEBUG "\n";
+}
+
 cache_set_key_cols(qw(flight job status name));
 
 sub computeflightsrange () {
@@ -335,9 +346,6 @@ END
 	push @rows, { %$jr, %$endedrow };
     }
 
-    print DEBUG "CACHE $hostname $rows_hit / ".(scalar @rows)
-	." of ".(scalar %cache)."\n";
-
     @rows = sort { $b->{finished} <=> $a->{finished} } @rows;
 
     print DEBUG "SORTED\n";
@@ -457,8 +465,7 @@ END
     rename "$html_file.new", "$html_file" or die "$html_file $!"
         if $doinstall;
 
-    print DEBUG "HOST CACHE RQ $runvarq_hits / ".
-	  ($runvarq_hits+$runvarq_misses)."\n";
+    cache_report_stats("$hostname");
 }
 
 foreach my $host (@ARGV) {
@@ -528,7 +535,6 @@ foreach my $host (sort keys %hosts) {
             mainquery($host);
 	    reporthost $host;
 	});
-	print DEBUG "JQ CACHE ".($jqtotal-$jqcachemisses)." / $jqtotal\n";
 	exit(0);
     }
     print DEBUG "SPAWNED [$pid] $host\n";
-- 
2.11.0

[OSSTEST PATCH 18/60] history reporting (nfc): Remove now-obsolete stats variables

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 9c70b560..1b4566aa 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -94,17 +94,12 @@ sub cache_row_key ($) {
     return join $; , map { $jr->{$_} } @cache_row_key_cols;
 }
 
-our $jqcachemisses = 0;
-our $jqtotal = 0;
-
 sub cacheable_fn ($$$) {
     my ($jr, $cachekey, $fn) = @_;
-    $jqtotal++;
     $cachekey = '%'.$cachekey;
     my $cached = $jr->{$cachekey};
     $q_count{$cachekey}++;
     if (!$cached) {
-	$jqcachemisses++;
 	$q_misses{$cachekey}++;
 	$cached = $fn->();
 	$jr->{$cachekey} = $cached;
-- 
2.11.0

[OSSTEST PATCH 19/60] history reporting (nfc): Introduce empty HistoryReport module

[Ian Jackson]

This is the boilerplate.  Code will appear in it in a moment.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 43 +++++++++++++++++++++++++++++++++++++++++++
 sg-report-host-history   |  1 +
 2 files changed, 44 insertions(+)
 create mode 100644 Osstest/HistoryReport.pm

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
new file mode 100644
index 00000000..c5e7e226
--- /dev/null
+++ b/Osstest/HistoryReport.pm
@@ -0,0 +1,43 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2020 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package Osstest::HistoryReport;
+
+use strict;
+use warnings;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw(
+			 cache_set_key_cols
+			 cache_read_previous
+			 cache_row_lookup_prep
+			 cacheable_query
+			 cacheable_fn
+			 cache_write_entry
+			 cache_report_stats
+		    );
+    %EXPORT_TAGS = ();
+
+    @EXPORT_OK   = qw();
+}
+
+use POSIX;
+
+1;
diff --git a/sg-report-host-history b/sg-report-host-history
index 1b4566aa..05a2dfe0 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -27,6 +27,7 @@ use HTML::Entities;
 use POSIX;
 
 use Osstest::Executive qw(:DEFAULT :colours);
+use Osstest::HistoryReport;
 
 our $limit= 2000;
 our $timelimit= 86400 * (366 + 14);
-- 
2.11.0

[OSSTEST PATCH 20/60] history reporting (nfc): Move cache code into HistoryReport module

[Ian Jackson]

Finally this is now reuseable code and we can put it in the
HistoryReport module.

Pure cut-and-paste.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 115 +++++++++++++++++++++++++++++++++++++++++++++++
 sg-report-host-history   | 115 -----------------------------------------------
 2 files changed, 115 insertions(+), 115 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index c5e7e226..a0565b6a 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -40,4 +40,119 @@ BEGIN {
 
 use POSIX;
 
+our @cache_row_key_cols;
+
+our %cache;
+
+our %q_count;
+our %q_misses;
+
+our $rows_previous = 0;
+our $rows_today = 0;
+our $rows_hit = 0;
+
+sub cache_set_key_cols { @cache_row_key_cols = @_; }
+
+sub cache_row_key ($) {
+    my ($jr) = @_;
+    return join $; , map { $jr->{$_} } @cache_row_key_cols;
+}
+
+sub cacheable_fn ($$$) {
+    my ($jr, $cachekey, $fn) = @_;
+    $cachekey = '%'.$cachekey;
+    my $cached = $jr->{$cachekey};
+    $q_count{$cachekey}++;
+    if (!$cached) {
+	$q_misses{$cachekey}++;
+	$cached = $fn->();
+	$jr->{$cachekey} = $cached;
+    }
+    return $cached;
+}
+
+sub cacheable_query ($$$) {
+    my ($q, $jr, $cachekey) = @_;
+    cacheable_fn($jr, $cachekey, sub {
+	foreach my $k (keys %{ $q->{ParamTypes} }) {
+	    $k =~ m/^:/ or die "$k ?";
+	    $q->bind_param($k, $jr->{$'} // die "$k ?");
+	}
+	$q->execute();
+	return $q->fetchrow_hashref();
+    });
+}
+
+sub cache_read_previous ($) {
+    my ($html_file) = @_;
+    if (!open H, $html_file) {
+        return if $!==ENOENT;
+        die "failed to open $html_file: $!";
+    }
+    %cache = ();
+    for (;;) {
+	$rows_previous++;
+        $_ = <H> // last;
+        next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
+	my $jr = {};
+	my $ch = $jr;
+	foreach (split / /, $1) {
+	    if (m{^\w+$}) {
+		$ch = { };
+		$jr->{'%'.$&} = $ch;
+		next;
+	    }
+	    s{^(\w+)=}{} or die;
+	    my $k = $1;
+	    s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
+	    $ch->{$k} = $_;
+	}
+	$cache{cache_row_key($jr)} = $jr;
+    }
+    close H;
+}
+
+sub cache_row_lookup_prep ($) {
+    my ($jrr) = @_;
+
+    $rows_today++;
+    my $cacherow = $cache{cache_row_key($$jrr)};
+    if ($cacherow) {
+	$$jrr = $cacherow;
+	$rows_hit++;
+    }
+}
+
+sub cache_write_entry ($$) {
+    my ($fh, $jr) = @_;
+    print $fh "<!-- osstest-report-reuseable";
+    my $whash = sub {
+	my ($h) = @_;
+	foreach my $k (sort keys %$h) {
+	    next if $k =~ m/^\%/;
+	    $_ = $h->{$k};
+	    s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
+	    printf $fh " %s=%s", $k, $_;
+	}
+    };
+    $whash->($jr);
+    foreach my $hk (sort keys %$jr) {
+	next unless $hk =~ m/^\%/;
+	print $fh " $'";
+	$whash->($jr->{$hk});
+    }
+    print $fh " -->\n";
+}
+
+sub cache_report_stats ($) {
+    my ($what) = @_;
+    print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today";
+    for my $cachekey (sort keys %q_count) {
+	my $total = $q_count{$cachekey};
+	my $hits = $total - ($q_misses{$cachekey} // 0);
+	print ::DEBUG " $cachekey=$hits/$total";
+    }
+    print ::DEBUG "\n";
+}
+
 1;
diff --git a/sg-report-host-history b/sg-report-host-history
index 05a2dfe0..a195bb21 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -77,121 +77,6 @@ our $restrictflight_cond = restrictflight_cond();
 our $flightcond;
 our $minflight;
 
-our @cache_row_key_cols;
-
-our %cache;
-
-our %q_count;
-our %q_misses;
-
-our $rows_previous = 0;
-our $rows_today = 0;
-our $rows_hit = 0;
-
-sub cache_set_key_cols { @cache_row_key_cols = @_; }
-
-sub cache_row_key ($) {
-    my ($jr) = @_;
-    return join $; , map { $jr->{$_} } @cache_row_key_cols;
-}
-
-sub cacheable_fn ($$$) {
-    my ($jr, $cachekey, $fn) = @_;
-    $cachekey = '%'.$cachekey;
-    my $cached = $jr->{$cachekey};
-    $q_count{$cachekey}++;
-    if (!$cached) {
-	$q_misses{$cachekey}++;
-	$cached = $fn->();
-	$jr->{$cachekey} = $cached;
-    }
-    return $cached;
-}
-
-sub cacheable_query ($$$) {
-    my ($q, $jr, $cachekey) = @_;
-    cacheable_fn($jr, $cachekey, sub {
-	foreach my $k (keys %{ $q->{ParamTypes} }) {
-	    $k =~ m/^:/ or die "$k ?";
-	    $q->bind_param($k, $jr->{$'} // die "$k ?");
-	}
-	$q->execute();
-	return $q->fetchrow_hashref();
-    });
-}
-
-sub cache_read_previous ($) {
-    my ($html_file) = @_;
-    if (!open H, $html_file) {
-        return if $!==ENOENT;
-        die "failed to open $html_file: $!";
-    }
-    %cache = ();
-    for (;;) {
-	$rows_previous++;
-        $_ = <H> // last;
-        next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
-	my $jr = {};
-	my $ch = $jr;
-	foreach (split / /, $1) {
-	    if (m{^\w+$}) {
-		$ch = { };
-		$jr->{'%'.$&} = $ch;
-		next;
-	    }
-	    s{^(\w+)=}{} or die;
-	    my $k = $1;
-	    s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
-	    $ch->{$k} = $_;
-	}
-	$cache{cache_row_key($jr)} = $jr;
-    }
-    close H;
-}
-
-sub cache_row_lookup_prep ($) {
-    my ($jrr) = @_;
-
-    $rows_today++;
-    my $cacherow = $cache{cache_row_key($$jrr)};
-    if ($cacherow) {
-	$$jrr = $cacherow;
-	$rows_hit++;
-    }
-}
-
-sub cache_write_entry ($$) {
-    my ($fh, $jr) = @_;
-    print $fh "<!-- osstest-report-reuseable";
-    my $whash = sub {
-	my ($h) = @_;
-	foreach my $k (sort keys %$h) {
-	    next if $k =~ m/^\%/;
-	    $_ = $h->{$k};
-	    s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
-	    printf $fh " %s=%s", $k, $_;
-	}
-    };
-    $whash->($jr);
-    foreach my $hk (sort keys %$jr) {
-	next unless $hk =~ m/^\%/;
-	print $fh " $'";
-	$whash->($jr->{$hk});
-    }
-    print $fh " -->\n";
-}
-
-sub cache_report_stats ($) {
-    my ($what) = @_;
-    print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today";
-    for my $cachekey (sort keys %q_count) {
-	my $total = $q_count{$cachekey};
-	my $hits = $total - ($q_misses{$cachekey} // 0);
-	print ::DEBUG " $cachekey=$hits/$total";
-    }
-    print ::DEBUG "\n";
-}
-
 cache_set_key_cols(qw(flight job status name));
 
 sub computeflightsrange () {
-- 
2.11.0

[OSSTEST PATCH 21/60] history reporting (nfc): Rename some module variables, remove "cache"

[Ian Jackson]

This makes the code terser and easier to read.  No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index a0565b6a..61b1ee8f 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -40,7 +40,7 @@ BEGIN {
 
 use POSIX;
 
-our @cache_row_key_cols;
+our @key_cols;
 
 our %cache;
 
@@ -51,11 +51,11 @@ our $rows_previous = 0;
 our $rows_today = 0;
 our $rows_hit = 0;
 
-sub cache_set_key_cols { @cache_row_key_cols = @_; }
+sub cache_set_key_cols { @key_cols = @_; }
 
-sub cache_row_key ($) {
+sub key ($) {
     my ($jr) = @_;
-    return join $; , map { $jr->{$_} } @cache_row_key_cols;
+    return join $; , map { $jr->{$_} } @key_cols;
 }
 
 sub cacheable_fn ($$$) {
@@ -107,7 +107,7 @@ sub cache_read_previous ($) {
 	    s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
 	    $ch->{$k} = $_;
 	}
-	$cache{cache_row_key($jr)} = $jr;
+	$cache{key($jr)} = $jr;
     }
     close H;
 }
@@ -116,7 +116,7 @@ sub cache_row_lookup_prep ($) {
     my ($jrr) = @_;
 
     $rows_today++;
-    my $cacherow = $cache{cache_row_key($$jrr)};
+    my $cacherow = $cache{key($$jrr)};
     if ($cacherow) {
 	$$jrr = $cacherow;
 	$rows_hit++;
-- 
2.11.0

[OSSTEST PATCH 22/60] history reporting: Skip undefined keys

[Ian Jackson]

This makes it work if the caller's cached hash contains an key which
is bound to undef.

sg-report-host-history already does this, which currently causes:

 Use of uninitialized value $_ in substitution (s///) at Osstest/HistoryReport.pm line 134.
 Use of uninitialized value $_ in printf at Osstest/HistoryReport.pm line 135.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 61b1ee8f..cb8c6ca1 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -131,6 +131,7 @@ sub cache_write_entry ($$) {
 	foreach my $k (sort keys %$h) {
 	    next if $k =~ m/^\%/;
 	    $_ = $h->{$k};
+	    next unless defined;
 	    s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
 	    printf $fh " %s=%s", $k, $_;
 	}
-- 
2.11.0

[OSSTEST PATCH 23/60] history reporting (nfc): Documentation for the new module

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 27 +++++++++++++++++++++++++++
 sg-report-host-history   |  6 +-----
 2 files changed, 28 insertions(+), 5 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index cb8c6ca1..6e5ed6ec 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -51,6 +51,33 @@ our $rows_previous = 0;
 our $rows_today = 0;
 our $rows_hit = 0;
 
+
+# History report query and processing cache
+#
+# Basic principle:
+#
+# Each HTML output file corresponds to the rows of one
+# primary query.  That primary query yields rows containing
+# (at least) the columns passed to cache_set_key_cols.
+#
+# The HTML files contain caches of per-primary-row queries and
+# computations.  Each such subcomputation is identified by a
+# $cachekey, which is assigned by the calling code (but should be
+# alphanumeric).  The memoisation is keyed off the key columns and
+# $cachekey, so the subcomputation should not depend on anything else.
+#
+# The API is based on a %$jr.
+#
+# %$jr contains one entry for each for each key columns.
+#
+# It also contains one entry for each cached subcomputation.
+# The keys for these entries are "\%$cachekey".
+
+# These routines all expect the file handle ::DEBUG
+# They are not reentrant and can only be used for one HTML
+# output file in one program run.
+
+
 sub cache_set_key_cols { @key_cols = @_; }
 
 sub key ($) {
diff --git a/sg-report-host-history b/sg-report-host-history
index a195bb21..e281986d 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -204,11 +204,7 @@ END
     my $inrows = $hosts{$hostname};
     print DEBUG "FOUND ", (scalar @$inrows), " ROWS for $hostname\n";
 
-    # Each entry in @$inrows is a $jr, which is a hash
-    # It has keys for the result columns in mainquery
-    # It also has keys '%<letter>' (yes, with a literal '%')
-    # which are the results of per-job queries.
-    # The contents of $jr for each job is cached across runs.
+    # Each entry in @$inrows is a HistoryReport $jr
 
     my @rows;
     foreach my $jr (@$inrows) {
-- 
2.11.0

[OSSTEST PATCH 24/60] history reporting: Cache data limit now in History module

[Ian Jackson]

Replace the ad-hoc query-specific limit strategy in
sg-report-host-history with a new, more principled, arrangement, in
HistoryReport.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 38 ++++++++++++++++++++++++++++++++------
 sg-report-host-history   | 14 +++++---------
 2 files changed, 37 insertions(+), 15 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 6e5ed6ec..b576c02a 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -31,7 +31,7 @@ BEGIN {
 			 cacheable_query
 			 cacheable_fn
 			 cache_write_entry
-			 cache_report_stats
+			 cache_finish
 		    );
     %EXPORT_TAGS = ();
 
@@ -43,6 +43,7 @@ use POSIX;
 our @key_cols;
 
 our %cache;
+our @previous;
 
 our %q_count;
 our %q_misses;
@@ -50,6 +51,7 @@ our %q_misses;
 our $rows_previous = 0;
 our $rows_today = 0;
 our $rows_hit = 0;
+our $rows_extra = 0;
 
 
 # History report query and processing cache
@@ -73,6 +75,12 @@ our $rows_hit = 0;
 # It also contains one entry for each cached subcomputation.
 # The keys for these entries are "\%$cachekey".
 
+# Limit strategy
+#
+# It is up to the caller to make a primary query that returns the rows
+# that it is interested in.  They should be in (perhaps approximate)
+# reverse time order.
+
 # These routines all expect the file handle ::DEBUG
 # They are not reentrant and can only be used for one HTML
 # output file in one program run.
@@ -118,7 +126,6 @@ sub cache_read_previous ($) {
     }
     %cache = ();
     for (;;) {
-	$rows_previous++;
         $_ = <H> // last;
         next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
 	my $jr = {};
@@ -134,6 +141,7 @@ sub cache_read_previous ($) {
 	    s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
 	    $ch->{$k} = $_;
 	}
+	push @previous, $jr;
 	$cache{key($jr)} = $jr;
     }
     close H;
@@ -169,18 +177,36 @@ sub cache_write_entry ($$) {
 	print $fh " $'";
 	$whash->($jr->{$hk});
     }
+    $jr->{'% written'} = 1;
     print $fh " -->\n";
 }
 
-sub cache_report_stats ($) {
-    my ($what) = @_;
-    print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today";
+sub cache_finish ($$) {
+    my ($fh, $what) = @_;
+
+    my $nprevious = @previous;
+
+    # Write as many rows again.  This saves re-querying in case of
+    # flapping etc.
+    for (;;) {
+	last if
+	  $rows_extra >= $rows_today &&
+	  ($rows_today + $rows_extra >= @previous * 0.9);
+	my $jr = shift @previous;
+	last unless $jr;
+	next if $jr->{'% written'};
+	cache_write_entry($fh, $jr);
+	$rows_extra++;
+    }
+
+    print ::DEBUG
+	"CACHE $what read=$nprevious hits $rows_hit/$rows_today";
     for my $cachekey (sort keys %q_count) {
 	my $total = $q_count{$cachekey};
 	my $hits = $total - ($q_misses{$cachekey} // 0);
 	print ::DEBUG " $cachekey=$hits/$total";
     }
-    print ::DEBUG "\n";
+    print ::DEBUG " extra=$rows_extra\n";
 }
 
 1;
diff --git a/sg-report-host-history b/sg-report-host-history
index e281986d..2ad2134d 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -119,7 +119,7 @@ our %hosts;
 sub mainquery ($) {
     my ($host) = @_;
 
-    our $runvarq //= db_prepare(<<END);
+    our $runvarq = db_prepare(<<END.($limit ? <<END_LIMIT : ''));
 	SELECT flight, job, name, status
 	  FROM runvars
           JOIN jobs USING (flight, job)
@@ -128,8 +128,9 @@ sub mainquery ($) {
 	   AND $flightcond
            AND $restrictflight_cond
 	 ORDER BY flight DESC
-         LIMIT $limit * 2
 END
+         LIMIT $limit
+END_LIMIT
 
     print DEBUG "MAINQUERY $host...\n";
     $runvarq->execute($host);
@@ -233,11 +234,6 @@ END
     my $runvarq_misses = 0;
 
     foreach my $jr (@rows) {
-	if ($wrote++ >= $limit) {
-	    cache_write_entry(\*H, $jr);
-	    next;
-	}
-
         #print DEBUG "JR $jr->{flight}.$jr->{job}\n";
 	my $ir = cacheable_query($infoq, $jr, 'i');
 	my $ar = cacheable_query($allocdq, $jr, 'a');
@@ -338,11 +334,11 @@ END
 
     print H "</table></body></html>\n";
 
+    cache_finish(\*H, "$hostname");
+
     close H or die $!;
     rename "$html_file.new", "$html_file" or die "$html_file $!"
         if $doinstall;
-
-    cache_report_stats("$hostname");
 }
 
 foreach my $host (@ARGV) {
-- 
2.11.0

[OSSTEST PATCH 25/60] history reporting: Print debug for cache misses

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index b576c02a..b593f207 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -155,6 +155,9 @@ sub cache_row_lookup_prep ($) {
     if ($cacherow) {
 	$$jrr = $cacherow;
 	$rows_hit++;
+    } else {
+	print ::DEBUG "CACHE MISS ",
+	    (join " ", map { $$jrr->{$_} } @key_cols), "\n";
     }
 }
 
-- 
2.11.0

[OSSTEST PATCH 26/60] history reporting: Improve an error message slightly

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index b593f207..6a23cfba 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -128,6 +128,7 @@ sub cache_read_previous ($) {
     for (;;) {
         $_ = <H> // last;
         next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
+	my $orig = $_;
 	my $jr = {};
 	my $ch = $jr;
 	foreach (split / /, $1) {
@@ -136,7 +137,7 @@ sub cache_read_previous ($) {
 		$jr->{'%'.$&} = $ch;
 		next;
 	    }
-	    s{^(\w+)=}{} or die;
+	    s{^(\w+)=}{} or die "$orig -- $_ ";
 	    my $k = $1;
 	    s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
 	    $ch->{$k} = $_;
-- 
2.11.0

[OSSTEST PATCH 27/60] sg-report-host-history: Write cache entry for unfinished jobs

[Ian Jackson]

We have to also check ->{finished}, rather than the existence of a row
at all, since now unfinished jobs can appear in the cache.

Because the cache key includes the job status, when the job becomes
finished the cache entry will be invalidated.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index 2ad2134d..6bf14aa2 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -214,7 +214,8 @@ END
 	cache_row_lookup_prep(\$jr);
 
 	my $endedrow = cacheable_query($endedq, $jr, 'e');
-	if (!$endedrow) {
+	if (!defined $endedrow->{finished}) {
+	    cache_write_entry(\*H, $jr);
 	    #print DEBUG "no-finished\n";
 	    next;
 	}
-- 
2.11.0

[OSSTEST PATCH 28/60] parallel by fork: Break out into HistoryReport

[Ian Jackson]

Move this code from sg-report-host-history to HistoryReport, so that
it can be reused.

No significant functional change.  Some changes to debug messages.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 51 ++++++++++++++++++++++++++++++++++++++++++++++++
 sg-report-host-history   | 49 +++++++++-------------------------------------
 2 files changed, 60 insertions(+), 40 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 6a23cfba..0b206de4 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -19,6 +19,9 @@ package Osstest::HistoryReport;
 use strict;
 use warnings;
 
+use Osstest;
+use Osstest::Executive;
+
 BEGIN {
     use Exporter ();
     our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
@@ -32,6 +35,7 @@ BEGIN {
 			 cacheable_fn
 			 cache_write_entry
 			 cache_finish
+			 parallel_by_fork
 		    );
     %EXPORT_TAGS = ();
 
@@ -213,4 +217,51 @@ sub cache_finish ($$) {
     print ::DEBUG " extra=$rows_extra\n";
 }
 
+our %children;
+our $worst = 0;
+our $whoami;
+
+sub wait_for_max_children ($) {
+    my ($lim) = @_;
+    while (keys(%children) > $lim) {
+	$!=0; $?=0; my $got = wait;
+	die "$! $got $?" unless exists $children{$got};
+	my $host = $children{$got};
+	delete $children{$got};
+	$worst = $? if $? > $worst;
+	if ($?) {
+	    print STDERR $whoami."[$$]: [$got] failed for $host: $?\n";
+	} else {
+	    print ::DEBUG "REAPED [$got] $host\n";
+	}
+    }
+}
+
+sub parallel_by_fork ($$$$) {
+    my ($set_whoami, $maxjobs, $tasks, $fn) = @_;
+    # does   $fn->($_) foreach @$tasks
+    # but in parallal and then exits.
+    # entries in @$taskarray should be suitable for print in messages.
+    # db is reopened in each child.
+
+    $whoami = $set_whoami;
+    undef $dbh_tests;
+
+    foreach my $task (@$tasks) {
+	wait_for_max_children($maxjobs);
+
+	my $pid = fork // die $!;
+	if (!$pid) {
+	    opendb_tests();
+	    $fn->($task);
+	    exit 0;
+	}
+	print ::DEBUG "SPAWNED [$pid] $task\n";
+	$children{$pid} = $task;
+    }
+
+    wait_for_max_children(0);
+    exit $worst;
+}
+
 1;
diff --git a/sg-report-host-history b/sg-report-host-history
index 6bf14aa2..0a2e9904 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -377,43 +377,12 @@ db_retry($dbh_tests, [], sub {
     computeflightsrange();
 });
 
-undef $dbh_tests;
-
-our %children;
-our $worst = 0;
-
-sub wait_for_max_children ($) {
-    my ($lim) = @_;
-    while (keys(%children) > $lim) {
-	$!=0; $?=0; my $got = wait;
-	die "$! $got $?" unless exists $children{$got};
-	my $host = $children{$got};
-	delete $children{$got};
-	$worst = $? if $? > $worst;
-	if ($?) {
-	    print STDERR "sg-report-flight[: [$got] failed for $host: $?\n";
-	} else {
-	    print DEBUG "REAPED [$got] $host\n";
-	}
-    }
-}
-
-foreach my $host (sort keys %hosts) {
-    wait_for_max_children($maxjobs);
-
-    my $pid = fork // die $!;
-    if (!$pid) {
-	opendb_tests();
-	cache_read_previous("$htmlout/$host.html") if $read_previous;
-	db_retry($dbh_tests, [], sub {
-            mainquery($host);
-	    reporthost $host;
-	});
-	exit(0);
-    }
-    print DEBUG "SPAWNED [$pid] $host\n";
-    $children{$pid} = $host;
-}
-
-wait_for_max_children(0);
-exit $worst;
+parallel_by_fork('sg-report-flight', $maxjobs, [ sort keys %hosts ], sub {
+    my ($host) = @_;
+    cache_read_previous("$htmlout/$host.html") if $read_previous;
+    db_retry($dbh_tests, [], sub {
+        mainquery($host);
+	reporthost $host;
+    });
+    exit(0);
+});
-- 
2.11.0

[OSSTEST PATCH 29/60] parallel by fork: Disconnect $dbh_tests as well as undefing it

[Ian Jackson]

If the caller is buggy and has statement handles still open, they can
still "work" even if we have thrown away the db handle.

Where, after forking, "work" means "use the same connection in
multiple processes simultaneously, without locking".  This could
result in arbitrary crazy nbehaviour (eg, TLS crypto failures).

No functional change with existing callers since they don't have this
bug.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 0b206de4..f0571189 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -245,6 +245,7 @@ sub parallel_by_fork ($$$$) {
     # db is reopened in each child.
 
     $whoami = $set_whoami;
+    $dbh_tests->disconnect();
     undef $dbh_tests;
 
     foreach my $task (@$tasks) {
-- 
2.11.0

[OSSTEST PATCH 30/60] parallel by fork: Fix a variable name to $task

[Ian Jackson]

This code came from sg-report-host-history where tasks were hosts.
But in the more general context, the names are wrong.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index f0571189..695d744a 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -226,13 +226,13 @@ sub wait_for_max_children ($) {
     while (keys(%children) > $lim) {
 	$!=0; $?=0; my $got = wait;
 	die "$! $got $?" unless exists $children{$got};
-	my $host = $children{$got};
+	my $task = $children{$got};
 	delete $children{$got};
 	$worst = $? if $? > $worst;
 	if ($?) {
-	    print STDERR $whoami."[$$]: [$got] failed for $host: $?\n";
+	    print STDERR $whoami."[$$]: [$got] failed for $task: $?\n";
 	} else {
-	    print ::DEBUG "REAPED [$got] $host\n";
+	    print ::DEBUG "REAPED [$got] $task\n";
 	}
     }
 }
-- 
2.11.0

[OSSTEST PATCH 31/60] sg-report-job-history: Prep for fork: Move $buildsq query

[Ian Jackson]

We will need to prepare this once per (job,branch) so that it works
when we do each of those in a different process.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 6008ca72..47fc7a62 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -106,15 +106,15 @@ sub add_revisions ($$$$) {
     }
 }
 
-our $buildsq= db_prepare(<<END);
+sub processjobbranch ($$) {
+    my ($j,$bra) = @_;
+
+    my $buildsq= db_prepare(<<END);
         SELECT * FROM runvars
          WHERE flight=? AND job=?
            AND name LIKE '%buildjob'
 END
 
-sub processjobbranch ($$) {
-    my ($j,$bra) = @_;
-
     print DEBUG "processjobbranch('$j',", ($bra ? "'$bra'" : 'undef'), ")\n";
 
     my %rev_grid_col;
-- 
2.11.0

[OSSTEST PATCH 32/60] sg-report-job-history: Prep for fork: Move $revisionsq query

[Ian Jackson]

We will need to prepare this in add_revisions so that it works when we
do each (job,branch) in a different process.

It is OK that it is still global, becauswe we only call add_revisions
in the children.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 47fc7a62..424053f1 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -89,15 +89,16 @@ if (defined($flight)) {
     push @jobs, $job;
 }
 
-our $revisionsq= db_prepare(<<END);
+sub add_revisions ($$$$) {
+    my ($revmap, $flightnum, $j, $sfx) = @_;
+
+    our $revisionsq //= db_prepare(<<END);
         SELECT * FROM runvars
          WHERE flight=? AND job=?
            AND name LIKE 'built\\_revision\\_%'
 END
-# (We report on non-main-revision jobs just as for main-revision ones.)
+    # (We report on non-main-revision jobs just as for main-revision ones.)
 
-sub add_revisions ($$$$) {
-    my ($revmap, $flightnum, $j, $sfx) = @_;
     $revisionsq->execute($flightnum, $j);
     while (my $row= $revisionsq->fetchrow_hashref()) {
         my $n= $row->{name};
-- 
2.11.0

[OSSTEST PATCH 33/60] sg-report-job-history: Use fork-based parallelism

[Ian Jackson]

For now, one child per job (for all branches).  This is already a
speedup.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 424053f1..384a7670 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -26,11 +26,13 @@ use IO::Handle;
 use HTML::Entities;
 
 use Osstest::Executive;
+use Osstest::HistoryReport;
 
 our (@blessings,@branches);
 our $limit= 100;
 our $htmlout;
 our $flight;
+our $maxjobs=10;
 our $job;
 
 open DEBUG, ">/dev/null";
@@ -38,7 +40,7 @@ open DEBUG, ">/dev/null";
 while (@ARGV && $ARGV[0] =~ m/^-/) {
     $_= shift @ARGV;
     last if m/^--?$/;
-    if (m/^--(job|flight)\=(.*)$/) {
+    if (m/^--(job|flight|maxjobs)\=(.*)$/) {
         $$1= $2;
     } elsif (m/^--(limit)\=([1-9]\d*)$/) {
         $$1= $2;
@@ -300,6 +302,9 @@ sub processjob ($) {
     processjobbranch($j,$_) foreach @branches;
 }
 
-db_begin_work($dbh_tests, []);
-db_readonly_report();
-foreach my $j (@jobs) { processjob($j); }
+parallel_by_fork('sg-report-job-history', $maxjobs, \@jobs, sub {
+    my ($j) = @_;
+    db_retry($dbh_tests, [], sub {
+        processjob($j);
+    });
+});
-- 
2.11.0

[OSSTEST PATCH 34/60] sg-report-job-history: Use one child per report

[Ian Jackson]

Rather than one child per job, which then did one report per branch.

This will mean we can use the cache machinery, which is rather global
so wouldn't cope well with processing multiple job history reports
within a process.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 384a7670..3307b281 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -296,15 +296,16 @@ END
     }
 }
 
-sub processjob ($) {
-    my ($j) = @_;
-    processjobbranch($j,undef);
-    processjobbranch($j,$_) foreach @branches;
+my @tasks;
+foreach my $j (@jobs) {
+    push @tasks, $j;
+    push @tasks, "$j $_" foreach @branches;
 }
 
-parallel_by_fork('sg-report-job-history', $maxjobs, \@jobs, sub {
-    my ($j) = @_;
+parallel_by_fork('sg-report-job-history', $maxjobs, \@tasks, sub {
+    my ($task) = @_;
+    my ($job, $branch) = split / /, $task; # $branch might be undef
     db_retry($dbh_tests, [], sub {
-        processjob($j);
+        processjobbranch($job, $branch);
     });
 });
-- 
2.11.0

[OSSTEST PATCH 35/60] sg-report-job-history (nfc): Have main program decide HTML filename

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 3307b281..3f10a74c 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -109,8 +109,8 @@ END
     }
 }
 
-sub processjobbranch ($$) {
-    my ($j,$bra) = @_;
+sub processjobbranch ($$$) {
+    my ($j,$bra,$html_file) = @_;
 
     my $buildsq= db_prepare(<<END);
         SELECT * FROM runvars
@@ -210,15 +210,13 @@ END
 END
 
     if (defined $htmlout) {
-        my ($title,$html_file,$url);
+        my ($title,$url);
 	ensuredir "$htmlout/history";
 	ensuredir "$htmlout/history/$j";
         if (defined $bra) {
             $title= "$j (branch $bra)";
-            $html_file= "history/$j/$bra.html";
         } else {
             $title= "$j (all branches)";
-            $html_file= "history/$j/ALL.html";
         }
         $html_file= "$htmlout/$html_file";
         open H, "> $html_file.new" or die "$html_file $!";
@@ -305,7 +303,8 @@ foreach my $j (@jobs) {
 parallel_by_fork('sg-report-job-history', $maxjobs, \@tasks, sub {
     my ($task) = @_;
     my ($job, $branch) = split / /, $task; # $branch might be undef
+    my $html_file = sprintf "history/%s/%s.html", $job, ($branch // 'ALL');
     db_retry($dbh_tests, [], sub {
-        processjobbranch($job, $branch);
+        processjobbranch($job, $branch, $html_file);
     });
 });
-- 
2.11.0

[OSSTEST PATCH 36/60] sg-report-job-history: Always write HTML output

[Ian Jackson]

Previously, unlike sg-report-host-history, if you didn't specify
--html-dir, it would would do a lot work to no effect.

This is not useful and nothing calls it this way.  So abolish this
notion.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 3f10a74c..7197a5f6 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -30,7 +30,7 @@ use Osstest::HistoryReport;
 
 our (@blessings,@branches);
 our $limit= 100;
-our $htmlout;
+our $htmlout = '.';
 our $flight;
 our $maxjobs=10;
 our $job;
-- 
2.11.0

[OSSTEST PATCH 37/60] sg-report-job-history (nfc): Remove needless conditional

[Ian Jackson]

$htmlout is now always defined.

Nothing other than indentation change, and removal of the surrounding
if block.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 162 +++++++++++++++++++++++++-------------------------
 1 file changed, 80 insertions(+), 82 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 7197a5f6..3b45992f 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -209,89 +209,87 @@ END
          WHERE flight=?
 END
 
-    if (defined $htmlout) {
-        my ($title,$url);
-	ensuredir "$htmlout/history";
-	ensuredir "$htmlout/history/$j";
-        if (defined $bra) {
-            $title= "$j (branch $bra)";
-        } else {
-            $title= "$j (all branches)";
-        }
-        $html_file= "$htmlout/$html_file";
-        open H, "> $html_file.new" or die "$html_file $!";
-        $title= encode_entities($title);
-        print H "<html><head><title>$title</title></head><body>\n";
-        print H "<h1>$title</h1>\n";
-        print H "<table rules=all>";
-        print H "<tr><th>started</th><th>flight</th>",
-	        "<th>branch</th><th>failure</th>\n";
-        print H "<th>", (join ", ", @hostvarcols), "</th>\n";
-        print H "<th>osstest</th>";
-        foreach my $c (@rev_grid_cols) {
-            print H "<th>".encode_entities($c)."</th>\n";
-        }
-        print H "</tr>\n";
-        my $alternate= 0;
-        my @last_revs;
-        my @alt_revs;
-        my $alt_hosts;
-        my $alt_osstest;
-        foreach my $r (@test_rows) {
-            my $altcolour= report_altcolour($alternate);
-            print H "<tr $altcolour>";
-	    my $started = $r->{Flight}{started};
-	    print H "<td>";
-	    print H show_abs_time $started if $started;
-	    print H "</td>\n";
-            my $flt= $r->{Flight}{flight};
-            $url= "$c{ReportHtmlPubBaseUrl}/$flt";
-            print H "<td><a href=\"$url\">$flt</a></td>\n";
-            print H "<td>".encode_entities($r->{Flight}{branch})."</td>\n";
-            $url= "$c{ReportHtmlPubBaseUrl}/$flt/".encode_entities($j).
-		"/info.html";
-            print H "<td $r->{ColourAttr}><a href=\"$url\">".
-                $r->{Content}."</a></td>\n";
-            my $lastrev;
-            my $hosts = join ", ", map { $_ // "-" } @{ $r->{Hosts} };
-            my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts);
-            print H "<td $hosts_colour>".encode_entities($hosts)."</td>\n";
-            $osstestverq->execute($r->{Flight}{flight});
-            my $osstestrevs = join ' ',
-                 map {
-                    $_ = $_->{harness};
-                    s/^([0-9a-f]{12})[0-9a-f]+\b/$1/;
-                    "<kbd>".encode_entities($_)."</kbd>";
-                 }
-                 @{ $osstestverq->fetchall_arrayref({}) };
-            my $osstest_colour = report_altchangecolour(\$alt_osstest, $osstestrevs);
-            print H "<td $osstest_colour>$osstestrevs</td>\n";
-            foreach my $i (0..$#rev_grid_cols) {
-                my $v= $r->{Revisions}[$i];
-		my $rev_colour = report_altchangecolour(\$alt_revs[$i], $v);
-                print H "<td $rev_colour>";
-                if (defined $v) {
-                    my $vp= $v;
-                    if (defined $lastrev && $v eq $lastrev) {
-                        $vp= '<-';
-                    } else {
-                        $vp =~ s/^((?:\d+\:)?[0-9a-f]{12})([0-9a-f].*)$/$1/;
-                    }
-                    print H "<kbd>".encode_entities($vp)."</kbd>";
-                }
-                $lastrev= $v;
-                $last_revs[$i]= $v;
-                print H "</td>";
-            }
-            print H "</tr>\n";
-            $alternate ^= 1;
-        }
-        print H "</table></body></html>\n";
-        H->error and die $!;
-        close H or die $!;
-        rename "$html_file.new", $html_file or die "$html_file $!";
-        #print "wrote $html_file\n";
+    my ($title,$url);
+    ensuredir "$htmlout/history";
+    ensuredir "$htmlout/history/$j";
+    if (defined $bra) {
+	$title= "$j (branch $bra)";
+    } else {
+	$title= "$j (all branches)";
+    }
+    $html_file= "$htmlout/$html_file";
+    open H, "> $html_file.new" or die "$html_file $!";
+    $title= encode_entities($title);
+    print H "<html><head><title>$title</title></head><body>\n";
+    print H "<h1>$title</h1>\n";
+    print H "<table rules=all>";
+    print H "<tr><th>started</th><th>flight</th>",
+	    "<th>branch</th><th>failure</th>\n";
+    print H "<th>", (join ", ", @hostvarcols), "</th>\n";
+    print H "<th>osstest</th>";
+    foreach my $c (@rev_grid_cols) {
+	print H "<th>".encode_entities($c)."</th>\n";
+    }
+    print H "</tr>\n";
+    my $alternate= 0;
+    my @last_revs;
+    my @alt_revs;
+    my $alt_hosts;
+    my $alt_osstest;
+    foreach my $r (@test_rows) {
+	my $altcolour= report_altcolour($alternate);
+	print H "<tr $altcolour>";
+	my $started = $r->{Flight}{started};
+	print H "<td>";
+	print H show_abs_time $started if $started;
+	print H "</td>\n";
+	my $flt= $r->{Flight}{flight};
+	$url= "$c{ReportHtmlPubBaseUrl}/$flt";
+	print H "<td><a href=\"$url\">$flt</a></td>\n";
+	print H "<td>".encode_entities($r->{Flight}{branch})."</td>\n";
+	$url= "$c{ReportHtmlPubBaseUrl}/$flt/".encode_entities($j).
+	    "/info.html";
+	print H "<td $r->{ColourAttr}><a href=\"$url\">".
+	    $r->{Content}."</a></td>\n";
+	my $lastrev;
+	my $hosts = join ", ", map { $_ // "-" } @{ $r->{Hosts} };
+	my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts);
+	print H "<td $hosts_colour>".encode_entities($hosts)."</td>\n";
+	$osstestverq->execute($r->{Flight}{flight});
+	my $osstestrevs = join ' ',
+	     map {
+		$_ = $_->{harness};
+		s/^([0-9a-f]{12})[0-9a-f]+\b/$1/;
+		"<kbd>".encode_entities($_)."</kbd>";
+	     }
+	     @{ $osstestverq->fetchall_arrayref({}) };
+	my $osstest_colour = report_altchangecolour(\$alt_osstest, $osstestrevs);
+	print H "<td $osstest_colour>$osstestrevs</td>\n";
+	foreach my $i (0..$#rev_grid_cols) {
+	    my $v= $r->{Revisions}[$i];
+	    my $rev_colour = report_altchangecolour(\$alt_revs[$i], $v);
+	    print H "<td $rev_colour>";
+	    if (defined $v) {
+		my $vp= $v;
+		if (defined $lastrev && $v eq $lastrev) {
+		    $vp= '<-';
+		} else {
+		    $vp =~ s/^((?:\d+\:)?[0-9a-f]{12})([0-9a-f].*)$/$1/;
+		}
+		print H "<kbd>".encode_entities($vp)."</kbd>";
+	    }
+	    $lastrev= $v;
+	    $last_revs[$i]= $v;
+	    print H "</td>";
+	}
+	print H "</tr>\n";
+	$alternate ^= 1;
     }
+    print H "</table></body></html>\n";
+    H->error and die $!;
+    close H or die $!;
+    rename "$html_file.new", $html_file or die "$html_file $!";
+    #print "wrote $html_file\n";
 }
 
 my @tasks;
-- 
2.11.0

[OSSTEST PATCH 38/60] history reporting (nfc): Add another test rune to the notes

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 runes | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/runes b/runes
index 8139f955..829e6106 100644
--- a/runes
+++ b/runes
@@ -11,3 +11,5 @@ select * from jobs full outer join steps using (job, flight) where flight=2364 o
 >v; env DBI_TRACE=2=v time ./sg-report-host-history --debug --html-dir=$PWD/e --max-flight=152223 flight:152223  2>&1 |tee c |ts -s %.s >c.ts
 rm -rf d; mkdir d; cp -a e/. d/.; mv c.ts b.ts; mv c b; mv v u
 cp -a d/. e/.
+
+>v; env DBI_TRACE=2=v time ./sg-report-job-history --debug --html-dir=$PWD/e --max-flight=152223 --flight=152223  2>&1 |tee c |ts -s %.s >c.ts
-- 
2.11.0

[OSSTEST PATCH 39/60] history reporting (nfc): Make cacheable_fn work without cache

[Ian Jackson]

This would allow us to use this in call sites without a cache.

I changed my mind about the code that prompted this, but it still
seems plausibly useful, so I'm keeping this commit.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 695d744a..5d2c83b7 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -99,6 +99,7 @@ sub key ($) {
 
 sub cacheable_fn ($$$) {
     my ($jr, $cachekey, $fn) = @_;
+    return $fn->() if !defined $cachekey;
     $cachekey = '%'.$cachekey;
     my $cached = $jr->{$cachekey};
     $q_count{$cachekey}++;
-- 
2.11.0

[OSSTEST PATCH 40/60] sg-report-job-history: Refactor "ALL" handling

[Ian Jackson]

* Make an explicit entry ALL in @branches, rather than implicitly
  processing ALL as well.

* Consequently, put explicit ALL entries in @tasks too, rather than
  putting in entries without a branch name.

* Pass ALL to processjobbranch rather than undef, and turn it into
  the internally-used undef at the start.

When used with --flight (findflight), this has no functional change.
When used with --job, ALL must now be included in the branch
list passed to --branches.  The only in-tree call is with --flight.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 3b45992f..8932458e 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -72,7 +72,7 @@ sub findflight () {
         SELECT branch FROM flights WHERE flight=?
 END
     die "$flight ? @$branches ?" if @$branches!=1;
-    @branches= @$branches;
+    @branches= ('ALL', @$branches);
 
     my $selectq= db_prepare(<<END);
         SELECT job FROM jobs WHERE flight=? ORDER BY JOB
@@ -110,7 +110,10 @@ END
 }
 
 sub processjobbranch ($$$) {
-    my ($j,$bra,$html_file) = @_;
+    my ($j,$branch_or_all,$html_file) = @_;
+
+    my $bra = $branch_or_all;
+    undef $bra if $bra eq 'ALL';
 
     my $buildsq= db_prepare(<<END);
         SELECT * FROM runvars
@@ -294,14 +297,13 @@ END
 
 my @tasks;
 foreach my $j (@jobs) {
-    push @tasks, $j;
     push @tasks, "$j $_" foreach @branches;
 }
 
 parallel_by_fork('sg-report-job-history', $maxjobs, \@tasks, sub {
     my ($task) = @_;
     my ($job, $branch) = split / /, $task; # $branch might be undef
-    my $html_file = sprintf "history/%s/%s.html", $job, ($branch // 'ALL');
+    my $html_file = sprintf "history/%s/%s.html", $job, $branch;
     db_retry($dbh_tests, [], sub {
         processjobbranch($job, $branch, $html_file);
     });
-- 
2.11.0

[OSSTEST PATCH 41/60] sg-report-job-history (nfc): Make $ri->{Hosts} a hash

[Ian Jackson]

This will make it easier to cache this.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 8932458e..118926c6 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -173,11 +173,11 @@ END
     while (my $f= $flightsq->fetchrow_hashref()) {
         my $ri= report_run_getinfo($f);
 
-	$ri->{Hosts} = [ ];
+	$ri->{Hosts} = { };
 	foreach my $hostvar (@hostvarcols) {
 	    $hostq->execute($f->{flight}, $f->{job}, $hostvar);
 	    my ($host) = $hostq->fetchrow_array();
-	    push @{ $ri->{Hosts} }, ($host // "-");
+	    $ri->{Hosts}{$hostvar} = $host;
 	}
 
         my %revisions;
@@ -255,7 +255,7 @@ END
 	print H "<td $r->{ColourAttr}><a href=\"$url\">".
 	    $r->{Content}."</a></td>\n";
 	my $lastrev;
-	my $hosts = join ", ", map { $_ // "-" } @{ $r->{Hosts} };
+	my $hosts = join ", ", map { $r->{Hosts}{$_} // "-" } @hostvarcols;
 	my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts);
 	print H "<td $hosts_colour>".encode_entities($hosts)."</td>\n";
 	$osstestverq->execute($r->{Flight}{flight});
-- 
2.11.0

[OSSTEST PATCH 42/60] sg-report-job-history (nfc): Add new hostvarcols calculation

[Ian Jackson]

We are going to want to replace the existing @hostvarcols
calculation.  Provide a new one based on $ri->{Hosts}.

Right now, die if they don't produce the same answers.  This still
works, which shows that the calculation is right.

Tested-by: Ian Jackson <ian.jackson@eu.citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/sg-report-job-history b/sg-report-job-history
index 118926c6..fb2930a9 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -206,6 +206,13 @@ END
         push @test_rows, $ri;
     }
 
+    my %hostvarnames;
+    foreach my $ri (@test_rows) {
+	$hostvarnames{$_}=1 foreach keys %{ $ri->{Hosts} }
+    }
+    my @hostvarcols2 = sort keys %hostvarnames;
+    die unless "@hostvarcols" eq "@hostvarcols2";
+
     my $osstestverq= db_prepare(<<END);
         SELECT DISTINCT harness
           FROM flights_harness_touched
-- 
2.11.0

[OSSTEST PATCH 43/60] sg-report-job-history (nfc): Query hosts runvars in one go

[Ian Jackson]

Rather than doing one query for each entry in @hostvarcols, do one
query for all the relevant runvars.  This is quite a bit faster and
will enable us to use the cache.

This is correct because @hostvarcols was the union of all the host
runvars, so this produces the same answers as the individual queries.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index fb2930a9..5d913b96 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -167,16 +167,19 @@ END
     }
 
     my $hostq= db_prepare(<<END);
-        SELECT val FROM runvars WHERE flight=? AND job=? AND name=?
+        SELECT name, val
+          FROM runvars
+         WHERE flight=?
+           AND job=?
+           AND (name = 'host' OR name LIKE '%\\_host')
 END
 
     while (my $f= $flightsq->fetchrow_hashref()) {
         my $ri= report_run_getinfo($f);
 
 	$ri->{Hosts} = { };
-	foreach my $hostvar (@hostvarcols) {
-	    $hostq->execute($f->{flight}, $f->{job}, $hostvar);
-	    my ($host) = $hostq->fetchrow_array();
+	$hostq->execute($f->{flight}, $f->{job});
+	while (my ($hostvar,$host) = $hostq->fetchrow_array()) {
 	    $ri->{Hosts}{$hostvar} = $host;
 	}
 
-- 
2.11.0

[OSSTEST PATCH 44/60] sg-report-job-history (nfc): Drop $hostsq query

[Ian Jackson]

We have eliminated all the users of @hostvarcols before @hostvarcols2
is calculated from the row data.

The query which produces this is very slow and can't be cached.  We
can abolish it now and just use the @hostvarcols2 calculation.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 22 +---------------------
 1 file changed, 1 insertion(+), 21 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 5d913b96..5684ac02 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -147,25 +147,6 @@ END
 END
     $flightsq->execute(@params);
 
-    my $hostsq= db_prepare(<<END);
-        SELECT DISTINCT name
-	 FROM runvars
-	 JOIN flights USING (flight)
-	WHERE ($cond)
-	  AND flight >= COALESCE(
-             (
-	      SELECT flight $fromstuff
-	      LIMIT 1 OFFSET $offset
-	     ), 0)
-     ORDER BY name;
-END
-    $hostsq->execute(@params, @params); # sql text contains $cond twice
-    my @hostvarcols;
-    while (my ($hostvar) = $hostsq->fetchrow_array()) {
-	next unless $hostvar =~ m/(^|_)host$/;
-	push @hostvarcols, $hostvar;
-    }
-
     my $hostq= db_prepare(<<END);
         SELECT name, val
           FROM runvars
@@ -213,8 +194,7 @@ END
     foreach my $ri (@test_rows) {
 	$hostvarnames{$_}=1 foreach keys %{ $ri->{Hosts} }
     }
-    my @hostvarcols2 = sort keys %hostvarnames;
-    die unless "@hostvarcols" eq "@hostvarcols2";
+    my @hostvarcols = sort keys %hostvarnames;
 
     my $osstestverq= db_prepare(<<END);
         SELECT DISTINCT harness
-- 
2.11.0

[OSSTEST PATCH 45/60] history reporting (nfc): Provide cache_set_task_print

[Ian Jackson]

This takes a string which gets added to the cache messages.  This
will allow us to distinguish the output from different processes
when using parallel by fork.

Nothing sets this yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 5d2c83b7..bfbee28c 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -29,6 +29,7 @@ BEGIN {
     @ISA         = qw(Exporter);
     @EXPORT      = qw(
 			 cache_set_key_cols
+			 cache_set_task_print
 			 cache_read_previous
 			 cache_row_lookup_prep
 			 cacheable_query
@@ -46,6 +47,7 @@ use POSIX;
 
 our @key_cols;
 
+our $taskprint='';
 our %cache;
 our @previous;
 
@@ -91,6 +93,7 @@ our $rows_extra = 0;
 
 
 sub cache_set_key_cols { @key_cols = @_; }
+sub cache_set_task_print ($) { ($taskprint)=@_; $taskprint =~ s/\s*$/ /; }
 
 sub key ($) {
     my ($jr) = @_;
@@ -162,7 +165,7 @@ sub cache_row_lookup_prep ($) {
 	$$jrr = $cacherow;
 	$rows_hit++;
     } else {
-	print ::DEBUG "CACHE MISS ",
+	print ::DEBUG "CACHE MISS $taskprint",
 	    (join " ", map { $$jrr->{$_} } @key_cols), "\n";
     }
 }
@@ -209,7 +212,7 @@ sub cache_finish ($$) {
     }
 
     print ::DEBUG
-	"CACHE $what read=$nprevious hits $rows_hit/$rows_today";
+	"CACHE $taskprint$what read=$nprevious hits $rows_hit/$rows_today";
     for my $cachekey (sort keys %q_count) {
 	my $total = $q_count{$cachekey};
 	my $hits = $total - ($q_misses{$cachekey} // 0);
-- 
2.11.0

[OSSTEST PATCH 46/60] sg-report-job-history: Introduce use of cache, for hosts query

[Ian Jackson]

* Set up the cache.
* Call the per-row setup hook.
* Cache the computation of $ri->{Hosts}.
* Call the per-row cache write hook.
* Finalise the cache.

Output is the same, but with cache information in the output html, and
faster.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 5684ac02..9f1216a6 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -121,6 +121,10 @@ sub processjobbranch ($$$) {
            AND name LIKE '%buildjob'
 END
 
+    cache_set_task_print("$j $branch_or_all");
+    cache_set_key_cols(qw(flight status)); # we process one job per process
+    cache_read_previous("$htmlout/$html_file");
+
     print DEBUG "processjobbranch('$j',", ($bra ? "'$bra'" : 'undef'), ")\n";
 
     my %rev_grid_col;
@@ -156,13 +160,17 @@ END
 END
 
     while (my $f= $flightsq->fetchrow_hashref()) {
+	cache_row_lookup_prep(\$f);
         my $ri= report_run_getinfo($f);
 
-	$ri->{Hosts} = { };
-	$hostq->execute($f->{flight}, $f->{job});
-	while (my ($hostvar,$host) = $hostq->fetchrow_array()) {
-	    $ri->{Hosts}{$hostvar} = $host;
-	}
+	$ri->{Hosts} = cacheable_fn($f, 'h', sub {
+            my %hosts;
+	    $hostq->execute($f->{flight}, $f->{job});
+	    while (my ($hostvar,$host) = $hostq->fetchrow_array()) {
+		$hosts{$hostvar} = $host;
+	    }
+	    \%hosts;
+	});
 
         my %revisions;
         add_revisions(\%revisions, $f->{flight}, $f->{job}, '');
@@ -276,8 +284,10 @@ END
 	    print H "</td>";
 	}
 	print H "</tr>\n";
+	cache_write_entry(\*H, $r->{Flight});
 	$alternate ^= 1;
     }
+    cache_finish(\*H, "");
     print H "</table></body></html>\n";
     H->error and die $!;
     close H or die $!;
-- 
2.11.0

[OSSTEST PATCH 47/60] history reporting (nfc): Break out url_quote

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index bfbee28c..f30ded5b 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -156,6 +156,12 @@ sub cache_read_previous ($) {
     close H;
 }
 
+sub url_quote ($) {
+    local ($_) = "$_[0]";
+    s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
+    $_;
+}
+
 sub cache_row_lookup_prep ($) {
     my ($jrr) = @_;
 
@@ -179,8 +185,7 @@ sub cache_write_entry ($$) {
 	    next if $k =~ m/^\%/;
 	    $_ = $h->{$k};
 	    next unless defined;
-	    s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
-	    printf $fh " %s=%s", $k, $_;
+	    printf $fh " %s=%s", $k, url_quote($_);
 	}
     };
     $whash->($jr);
-- 
2.11.0

[OSSTEST PATCH 48/60] history reporting (nfc): Break out url_unquote

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index f30ded5b..f8a49955 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -147,8 +147,7 @@ sub cache_read_previous ($) {
 	    }
 	    s{^(\w+)=}{} or die "$orig -- $_ ";
 	    my $k = $1;
-	    s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
-	    $ch->{$k} = $_;
+	    $ch->{$k} = url_unquote($_);
 	}
 	push @previous, $jr;
 	$cache{key($jr)} = $jr;
@@ -156,6 +155,12 @@ sub cache_read_previous ($) {
     close H;
 }
 
+sub url_unquote ($) {
+    local ($_) = "$_[0]";
+    s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
+    $_;
+}
+
 sub url_quote ($) {
     local ($_) = "$_[0]";
     s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
-- 
2.11.0

[OSSTEST PATCH 49/60] history reporting (nfc): Break out $url_ok_chars

[Ian Jackson]

We will want this in a moment.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index f8a49955..75ec75f3 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -45,6 +45,8 @@ BEGIN {
 
 use POSIX;
 
+our $url_ok_chars = '-+=/~:;_.,\w';
+
 our @key_cols;
 
 our $taskprint='';
@@ -163,7 +165,7 @@ sub url_unquote ($) {
 
 sub url_quote ($) {
     local ($_) = "$_[0]";
-    s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
+    s{[^$url_ok_chars]}{ sprintf "%%%02x", ord $& }ge;
     $_;
 }
 
-- 
2.11.0

[OSSTEST PATCH 50/60] history reporting (nfc): url-quoting: quote = too

[Ian Jackson]

We are going to want to url-encode keys.  If key contains =, we still
need to be able to tell where it ends, so it must be encoded.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 75ec75f3..20913d27 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -45,7 +45,7 @@ BEGIN {
 
 use POSIX;
 
-our $url_ok_chars = '-+=/~:;_.,\w';
+our $url_ok_chars = '-+/~:;_.,\w';
 
 our @key_cols;
 
-- 
2.11.0

[OSSTEST PATCH 51/60] history reporting (nfc): Quote keys too

[Ian Jackson]

Right now all the callers have keys which don't need quoting.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 20913d27..3e2ad0d3 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -147,8 +147,9 @@ sub cache_read_previous ($) {
 		$jr->{'%'.$&} = $ch;
 		next;
 	    }
-	    s{^(\w+)=}{} or die "$orig -- $_ ";
-	    my $k = $1;
+	    # Tolerate %, which will be de-quoted by url_unquote
+	    s{^([\%$url_ok_chars]+)=}{}i or die "$orig -- $_ ";
+	    my $k = url_unquote($1);
 	    $ch->{$k} = url_unquote($_);
 	}
 	push @previous, $jr;
@@ -192,7 +193,7 @@ sub cache_write_entry ($$) {
 	    next if $k =~ m/^\%/;
 	    $_ = $h->{$k};
 	    next unless defined;
-	    printf $fh " %s=%s", $k, url_quote($_);
+	    printf $fh " %s=%s", url_quote($k), url_quote($_);
 	}
     };
     $whash->($jr);
-- 
2.11.0

[OSSTEST PATCH 52/60] sg-report-job-history: Cache the per-flight revisions

[Ian Jackson]

This involves changing %revisions to %$revisions in the code which
uses the value.

No logical change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 9f1216a6..39ebe024 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -172,25 +172,29 @@ END
 	    \%hosts;
 	});
 
-        my %revisions;
-        add_revisions(\%revisions, $f->{flight}, $f->{job}, '');
+	my $revisions = cacheable_fn($f, 'r', sub {
+            my %revisions;
+	    add_revisions(\%revisions, $f->{flight}, $f->{job}, '');
         
-        $buildsq->execute($f->{flight}, $f->{job});
-        while (my $bld= $buildsq->fetchrow_hashref()) {
-            my $bn= $bld->{name};
-            $bn =~ s/buildjob$//;
-            my ($oflight,$ojob) = flight_otherjob($f->{flight}, $bld->{val});
-            add_revisions(\%revisions, $oflight, $ojob, " ($bn)");
-        }
+	    $buildsq->execute($f->{flight}, $f->{job});
+	    while (my $bld= $buildsq->fetchrow_hashref()) {
+		my $bn= $bld->{name};
+		$bn =~ s/buildjob$//;
+		my ($oflight,$ojob) =
+		  flight_otherjob($f->{flight}, $bld->{val});
+		add_revisions(\%revisions, $oflight, $ojob, " ($bn)");
+	    }
+	    \%revisions;
+	});
         my @rev_cols;
-        foreach my $revk (sort keys %revisions) {
+        foreach my $revk (sort keys %$revisions) {
             my $col= $rev_grid_col{$revk};
             if (!defined $col) {
                 $col= @rev_grid_cols;
                 push @rev_grid_cols, $revk;
                 $rev_grid_col{$revk}= $col;
             }
-            $rev_cols[$col]= $revisions{$revk};
+            $rev_cols[$col]= $$revisions{$revk};
         }
         $ri->{Revisions}= \@rev_cols;
         $ri->{Flight}= $f;
-- 
2.11.0

[OSSTEST PATCH 53/60] sg-report-job-history (nfc): Refactor osstestrevs code

[Ian Jackson]

Split this into (1) get the data from the db (2) process it into the
form we want.

This will make it easy to cache (1).

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 39ebe024..d5e37bcd 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -260,14 +260,17 @@ END
 	my $hosts = join ", ", map { $r->{Hosts}{$_} // "-" } @hostvarcols;
 	my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts);
 	print H "<td $hosts_colour>".encode_entities($hosts)."</td>\n";
+	my %osstestrevs;
 	$osstestverq->execute($r->{Flight}{flight});
+	while (my ($harness) = $osstestverq->fetchrow_array()) {
+	    $osstestrevs{$harness}++;
+	}
 	my $osstestrevs = join ' ',
 	     map {
-		$_ = $_->{harness};
 		s/^([0-9a-f]{12})[0-9a-f]+\b/$1/;
 		"<kbd>".encode_entities($_)."</kbd>";
 	     }
-	     @{ $osstestverq->fetchall_arrayref({}) };
+	     sort keys %osstestrevs;
 	my $osstest_colour = report_altchangecolour(\$alt_osstest, $osstestrevs);
 	print H "<td $osstest_colour>$osstestrevs</td>\n";
 	foreach my $i (0..$#rev_grid_cols) {
-- 
2.11.0

[OSSTEST PATCH 54/60] sg-report-job-history: Cache osstestrevs

[Ian Jackson]

No logical change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index d5e37bcd..48dc2f57 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -260,17 +260,20 @@ END
 	my $hosts = join ", ", map { $r->{Hosts}{$_} // "-" } @hostvarcols;
 	my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts);
 	print H "<td $hosts_colour>".encode_entities($hosts)."</td>\n";
-	my %osstestrevs;
-	$osstestverq->execute($r->{Flight}{flight});
-	while (my ($harness) = $osstestverq->fetchrow_array()) {
-	    $osstestrevs{$harness}++;
-	}
+	my $osstestrevmap = cacheable_fn($r->{Flight}, 'o', sub {
+            my %osstestrevs;
+	    $osstestverq->execute($r->{Flight}{flight});
+	    while (my ($harness) = $osstestverq->fetchrow_array()) {
+		$osstestrevs{$harness}++;
+	    }
+	    \%osstestrevs;
+	});
 	my $osstestrevs = join ' ',
 	     map {
 		s/^([0-9a-f]{12})[0-9a-f]+\b/$1/;
 		"<kbd>".encode_entities($_)."</kbd>";
 	     }
-	     sort keys %osstestrevs;
+	     sort keys %$osstestrevmap;
 	my $osstest_colour = report_altchangecolour(\$alt_osstest, $osstestrevs);
 	print H "<td $osstest_colour>$osstestrevs</td>\n";
 	foreach my $i (0..$#rev_grid_cols) {
-- 
2.11.0

[OSSTEST PATCH 55/60] history reporting: Break out minflight_by_time

[Ian Jackson]

Move this from sg-report-host-history so we can reuse it.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HistoryReport.pm | 16 ++++++++++++++++
 sg-report-host-history   | 13 ++-----------
 2 files changed, 18 insertions(+), 11 deletions(-)

diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 3e2ad0d3..b851ae45 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -37,6 +37,7 @@ BEGIN {
 			 cache_write_entry
 			 cache_finish
 			 parallel_by_fork
+			 minflight_by_time
 		    );
     %EXPORT_TAGS = ();
 
@@ -282,4 +283,19 @@ sub parallel_by_fork ($$$$) {
     exit $worst;
 }
 
+sub minflight_by_time ($) {
+    my ($timelimit) = @_;
+    my $minflightsq = db_prepare(<<END);
+	    SELECT flight
+              FROM flights
+             WHERE started >= ?
+          ORDER BY flight ASC
+             LIMIT 1
+END
+    my $now = time // die $!;
+    $minflightsq->execute($now - $timelimit);
+    my ($minflight,) = $minflightsq->fetchrow_array();
+    $minflight //= 0;
+}
+
 1;
diff --git a/sg-report-host-history b/sg-report-host-history
index 0a2e9904..efe652bb 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -95,19 +95,10 @@ sub computeflightsrange () {
 END
 	$minflightsq->execute();
 	($minflight,) = $minflightsq->fetchrow_array();
+	$minflight //= 0;
     } else {
-	my $minflightsq = db_prepare(<<END);
-	    SELECT flight
-              FROM flights
-             WHERE started >= ?
-          ORDER BY flight ASC
-             LIMIT 1
-END
-	my $now = time // die $!;
-        $minflightsq->execute($now - $timelimit);
-	($minflight,) = $minflightsq->fetchrow_array();
+	$minflight = minflight_by_time($timelimit);
     }
-    $minflight //= 0;
 
     $flightcond = "(flight > $minflight)";
 
-- 
2.11.0

[OSSTEST PATCH 56/60] sg-report-job-history (nfc): Abolish $fromstuff

[Ian Jackson]

This used to be reused, but that is no longer the case.  Do away with
it, for clarity and simplicity.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 48dc2f57..2cac123b 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -140,13 +140,11 @@ END
     }
     my $offset= $limit-1;
 
-    my $fromstuff= <<END;
+    my $flightsq= db_prepare(<<END);
+        SELECT * 
           FROM jobs JOIN flights USING (flight)
          WHERE ($cond)
       ORDER BY flight DESC
-END
-    my $flightsq= db_prepare(<<END);
-        SELECT * $fromstuff
          LIMIT $limit
 END
     $flightsq->execute(@params);
-- 
2.11.0

[OSSTEST PATCH 57/60] sg-report-job-history: Cache report_run_getinfo

[Ian Jackson]

No logical change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 2cac123b..5d2a7e15 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -159,7 +159,7 @@ END
 
     while (my $f= $flightsq->fetchrow_hashref()) {
 	cache_row_lookup_prep(\$f);
-        my $ri= report_run_getinfo($f);
+        my $ri= cacheable_fn($f, 'rri', sub { report_run_getinfo($f) });
 
 	$ri->{Hosts} = cacheable_fn($f, 'h', sub {
             my %hosts;
-- 
2.11.0

[OSSTEST PATCH 58/60] sg-report-host-history: Cache report_run_getinfo

[Ian Jackson]

No logical change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-host-history | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/sg-report-host-history b/sg-report-host-history
index efe652bb..a814ef37 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -266,7 +266,9 @@ END
 
 	print H "<td><a href=\"$url\">$jr->{job}</td>\n";
 
-	my $ri = report_run_getinfo({ %$jr, %$ir });
+	my $ri = cacheable_fn($jr, 'rri', sub {
+            report_run_getinfo({ %$jr, %$ir })
+	});
 	print H "<td $ri->{ColourAttr}>$ri->{Content}</td>\n";
 
 	my $powers = cacheable_fn($jr, 'b', sub {
-- 
2.11.0

[OSSTEST PATCH 59/60] sg-report-job-history: Provide --time-limit

[Ian Jackson]

Calculate a minflight based on the time limit, and set the time limit
to a year ago by default.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/sg-report-job-history b/sg-report-job-history
index 5d2a7e15..e09c694f 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -30,6 +30,7 @@ use Osstest::HistoryReport;
 
 our (@blessings,@branches);
 our $limit= 100;
+our $timelimit= 86400 * (366 + 14);
 our $htmlout = '.';
 our $flight;
 our $maxjobs=10;
@@ -44,6 +45,8 @@ while (@ARGV && $ARGV[0] =~ m/^-/) {
         $$1= $2;
     } elsif (m/^--(limit)\=([1-9]\d*)$/) {
         $$1= $2;
+    } elsif (m/^--time-limit\=([1-9]\d*)$/) {
+        $timelimit= $1;
     } elsif (restrictflight_arg($_)) {
 	# Handled by Executive
     } elsif (m/^--html-dir=(.*)$/) {
@@ -67,6 +70,9 @@ csreadconfig();
 
 our @jobs;
 
+our $minflight = minflight_by_time($timelimit);
+print DEBUG "MINFLIGHT $minflight\n";
+
 sub findflight () {
     my $branches= $dbh_tests->selectcol_arrayref(<<END, {}, $flight);
         SELECT branch FROM flights WHERE flight=?
@@ -144,6 +150,7 @@ END
         SELECT * 
           FROM jobs JOIN flights USING (flight)
          WHERE ($cond)
+           AND flight > $minflight
       ORDER BY flight DESC
          LIMIT $limit
 END
-- 
2.11.0

[OSSTEST PATCH 60/60] sg-report-job-history: Increase default limit

[Ian Jackson]

Now this is a *lot* faster, we can print a lot more history.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-report-job-history | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index e09c694f..843dd776 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -29,7 +29,7 @@ use Osstest::Executive;
 use Osstest::HistoryReport;
 
 our (@blessings,@branches);
-our $limit= 100;
+our $limit= 2000;
 our $timelimit= 86400 * (366 + 14);
 our $htmlout = '.';
 our $flight;
-- 
2.11.0