* [OSSTEST PATCH 00/60] Speed up sg-report-job-history
@ 2020-08-14 17:21 Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 01/60] history reporting (nfc): Do not key cache on hostname any more Ian Jackson
` (59 more replies)
0 siblings, 60 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
I think this is the final section of my work to speed up osstest's
reporting db queries etc.
After this is pushed and working, I will look and see what the
interval is now, between one flight finishing and the next starting,
to see ig there is more to be done.
Ian Jackson (60):
history reporting (nfc): Do not key cache on hostname any more
history reporting (nfc): Add some test runes to the notes
history reporting: Delete two debug prints
history reporting (nfc): Refactor to generalise, cache_read_existing
history reporting (nfc): Make cache_write_entry into a top-level sub
history reporting (nfc): Rename jobquery to cacheable_query
history reporting (nfc): Break out cache_row_lookup_prep
history reporting (nfc): Rename "existing" to "previous"
history reporting (nfc): Provide cacheable_fn
history reporting (nfc): Bind by name in cacheable_query
history reporting (nfc): Rename $cachehits to $rows_hit
history reporting (nfc): Record query-specific stats
history reporting (nfc): Move cacheable_* further up the file
history reporting (nfc): Use cacheable_fn for power methods
history reporting (nfc): Introduce cache_set_key_cols
history reporting (nfc): Record more row-specific stats
history reporting: Cache stats reporting: Centralise and rework
history reporting (nfc): Remove now-obsolete stats variables
history reporting (nfc): Introduce empty HistoryReport module
history reporting (nfc): Move cache code into HistoryReport module
history reporting (nfc): Rename some module variables, remove "cache"
history reporting: Skip undefined keys
history reporting (nfc): Documentation for the new module
history reporting: Cache data limit now in History module
history reporting: Print debug for cache misses
history reporting: Improve an error message slightly
sg-report-host-history: Write cache entry for unfinished jobs
parallel by fork: Break out into HistoryReport
parallel by fork: Disconnect $dbh_tests as well as undefing it
parallel by fork: Fix a variable name to $task
sg-report-job-history: Prep for fork: Move $buildsq query
sg-report-job-history: Prep for fork: Move $revisionsq query
sg-report-job-history: Use fork-based parallelism
sg-report-job-history: Use one child per report
sg-report-job-history (nfc): Have main program decide HTML filename
sg-report-job-history: Always write HTML output
sg-report-job-history (nfc): Remove needless conditional
history reporting (nfc): Add another test rune to the notes
history reporting (nfc): Make cacheable_fn work without cache
sg-report-job-history: Refactor "ALL" handling
sg-report-job-history (nfc): Make $ri->{Hosts} a hash
sg-report-job-history (nfc): Add new hostvarcols calculation
sg-report-job-history (nfc): Query hosts runvars in one go
sg-report-job-history (nfc): Drop $hostsq query
history reporting (nfc): Provide cache_set_task_print
sg-report-job-history: Introduce use of cache, for hosts query
history reporting (nfc): Break out url_quote
history reporting (nfc): Break out url_unquote
history reporting (nfc): Break out $url_ok_chars
history reporting (nfc): url-quoting: quote = too
history reporting (nfc): Quote keys too
sg-report-job-history: Cache the per-flight revisions
sg-report-job-history (nfc): Refactor osstestrevs code
sg-report-job-history: Cache osstestrevs
history reporting: Break out minflight_by_time
sg-report-job-history (nfc): Abolish $fromstuff
sg-report-job-history: Cache report_run_getinfo
sg-report-host-history: Cache report_run_getinfo
sg-report-job-history: Provide --time-limit
sg-report-job-history: Increase default limit
Osstest/HistoryReport.pm | 301 +++++++++++++++++++++++++++++++++++++++++++++
runes | 10 ++
sg-report-host-history | 216 +++++++-------------------------
sg-report-job-history | 313 +++++++++++++++++++++++++----------------------
4 files changed, 520 insertions(+), 320 deletions(-)
create mode 100644 Osstest/HistoryReport.pm
--
2.11.0
^ permalink raw reply [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 01/60] history reporting (nfc): Do not key cache on hostname any more
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 02/60] history reporting (nfc): Add some test runes to the notes Ian Jackson
` (58 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Now we process only one host at a time, so we don't need to
distinguish.
Replace all references to $tcache with %cache.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 13 +++++--------
1 file changed, 5 insertions(+), 8 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 380f8fac..50cc5b58 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -76,7 +76,7 @@ our $restrictflight_cond = restrictflight_cond();
our $flightcond;
our $minflight;
-our %hcaches;
+our %cache;
sub read_existing_logs ($) {
my ($hostname) = @_;
@@ -86,8 +86,7 @@ sub read_existing_logs ($) {
return if $!==ENOENT;
die "failed to open $html_file: $!";
}
- my $tcache = { };
- $hcaches{$hostname} = $tcache;
+ %cache = ();
for (;;) {
$_ = <H> // last;
next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
@@ -106,7 +105,7 @@ sub read_existing_logs ($) {
# print DEBUG "GOTCACHE $hostname $k\n";
}
# print DEBUG "GOTCACHE $hostname \@ $jr->{flight} $jr->{job} $jr->{status},$jr->{name}\n";
- $tcache->{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr;
+ $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr;
}
close H;
}
@@ -253,8 +252,6 @@ END
my $inrows = $hosts{$hostname};
print DEBUG "FOUND ", (scalar @$inrows), " ROWS for $hostname\n";
- my $tcache = $hcaches{$hostname};
-
# Each entry in @$inrows is a $jr, which is a hash
# It has keys for the result columns in mainquery
# It also has keys '%<letter>' (yes, with a literal '%')
@@ -267,7 +264,7 @@ END
#print DEBUG "JOB $jr->{flight}.$jr->{job} ";
my $cacherow =
- $tcache->{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}};
+ $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}};
if ($cacherow) {
$jr = $cacherow;
$cachehits++;
@@ -285,7 +282,7 @@ END
}
print DEBUG "CACHE $hostname $cachehits / ".(scalar @rows)
- ." of ".(scalar %$tcache)."\n";
+ ." of ".(scalar %cache)."\n";
my $write_cache_entry = sub {
my ($jr) = @_;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 02/60] history reporting (nfc): Add some test runes to the notes
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 01/60] history reporting (nfc): Do not key cache on hostname any more Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 03/60] history reporting: Delete two debug prints Ian Jackson
` (57 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
runes | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/runes b/runes
index 27ece2af..8139f955 100644
--- a/runes
+++ b/runes
@@ -3,3 +3,11 @@ select * from resource_sharing s full outer join resources r on r.restype = 'sha
select * from jobs full outer join steps using (job, flight) where flight=2364 order by job, stepno;
+
+
+
+>v; env DBI_TRACE=2=v time ./sg-report-job-history --debug --html-dir=$PWD/d --flight=152223 --max-flight=152223 2>&1 |tee c |ts -s %.s >c.ts
+
+>v; env DBI_TRACE=2=v time ./sg-report-host-history --debug --html-dir=$PWD/e --max-flight=152223 flight:152223 2>&1 |tee c |ts -s %.s >c.ts
+rm -rf d; mkdir d; cp -a e/. d/.; mv c.ts b.ts; mv c b; mv v u
+cp -a d/. e/.
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 03/60] history reporting: Delete two debug prints
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 01/60] history reporting (nfc): Do not key cache on hostname any more Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 02/60] history reporting (nfc): Add some test runes to the notes Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 04/60] history reporting (nfc): Refactor to generalise, cache_read_existing Ian Jackson
` (56 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
These are going to become invalid and I don't think it's important to
save them.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 2 --
1 file changed, 2 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 50cc5b58..4c0022e8 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -102,9 +102,7 @@ sub read_existing_logs ($) {
my $k = $1;
s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
$ch->{$k} = $_;
-# print DEBUG "GOTCACHE $hostname $k\n";
}
-# print DEBUG "GOTCACHE $hostname \@ $jr->{flight} $jr->{job} $jr->{status},$jr->{name}\n";
$cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr;
}
close H;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 04/60] history reporting (nfc): Refactor to generalise, cache_read_existing
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (2 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 03/60] history reporting: Delete two debug prints Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 05/60] history reporting (nfc): Make cache_write_entry into a top-level sub Ian Jackson
` (55 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
* Introduce @cache_row_key_cols and cache_row_key
* Handle $html_file pathname construction and $read_existing
at the call site.
* Rename cache_read_existing.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 19 +++++++++++--------
1 file changed, 11 insertions(+), 8 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 4c0022e8..83344a40 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -77,11 +77,15 @@ our $flightcond;
our $minflight;
our %cache;
+our @cache_row_key_cols = qw(flight job status name);
-sub read_existing_logs ($) {
- my ($hostname) = @_;
- return unless $read_existing;
- my $html_file = "$htmlout/$hostname.html";
+sub cache_row_key ($) {
+ my ($jr) = @_;
+ return join $; , map { $jr->{$_} } @cache_row_key_cols;
+}
+
+sub cache_read_existing ($) {
+ my ($html_file) = @_;
if (!open H, $html_file) {
return if $!==ENOENT;
die "failed to open $html_file: $!";
@@ -103,7 +107,7 @@ sub read_existing_logs ($) {
s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
$ch->{$k} = $_;
}
- $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} = $jr;
+ $cache{cache_row_key($jr)} = $jr;
}
close H;
}
@@ -261,8 +265,7 @@ END
foreach my $jr (@$inrows) {
#print DEBUG "JOB $jr->{flight}.$jr->{job} ";
- my $cacherow =
- $cache{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}};
+ my $cacherow = $cache{cache_row_key($jr)};
if ($cacherow) {
$jr = $cacherow;
$cachehits++;
@@ -492,7 +495,7 @@ foreach my $host (sort keys %hosts) {
my $pid = fork // die $!;
if (!$pid) {
opendb_tests();
- read_existing_logs($host);
+ cache_read_existing("$htmlout/$host.html") if $read_existing;
db_retry($dbh_tests, [], sub {
mainquery($host);
reporthost $host;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 05/60] history reporting (nfc): Make cache_write_entry into a top-level sub
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (3 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 04/60] history reporting (nfc): Refactor to generalise, cache_read_existing Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 06/60] history reporting (nfc): Rename jobquery to cacheable_query Ian Jackson
` (54 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 46 +++++++++++++++++++++++-----------------------
1 file changed, 23 insertions(+), 23 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 83344a40..fe5b7cc2 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -112,6 +112,27 @@ sub cache_read_existing ($) {
close H;
}
+sub cache_write_entry ($$) {
+ my ($fh, $jr) = @_;
+ print $fh "<!-- osstest-report-reuseable";
+ my $whash = sub {
+ my ($h) = @_;
+ foreach my $k (sort keys %$h) {
+ next if $k =~ m/^\%/;
+ $_ = $h->{$k};
+ s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
+ printf $fh " %s=%s", $k, $_;
+ }
+ };
+ $whash->($jr);
+ foreach my $hk (sort keys %$jr) {
+ next unless $hk =~ m/^\%/;
+ print $fh " $'";
+ $whash->($jr->{$hk});
+ }
+ print $fh " -->\n";
+}
+
sub computeflightsrange () {
if ($flightlimit) {
my $minflightsq = db_prepare(<<END);
@@ -285,27 +306,6 @@ END
print DEBUG "CACHE $hostname $cachehits / ".(scalar @rows)
." of ".(scalar %cache)."\n";
- my $write_cache_entry = sub {
- my ($jr) = @_;
- print H "<!-- osstest-report-reuseable";
- my $whash = sub {
- my ($h) = @_;
- foreach my $k (sort keys %$h) {
- next if $k =~ m/^\%/;
- $_ = $h->{$k};
- s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
- printf H " %s=%s", $k, $_;
- }
- };
- $whash->($jr);
- foreach my $hk (sort keys %$jr) {
- next unless $hk =~ m/^\%/;
- print H " $'";
- $whash->($jr->{$hk});
- }
- print H " -->\n";
- };
-
@rows = sort { $b->{finished} <=> $a->{finished} } @rows;
print DEBUG "SORTED\n";
@@ -317,7 +317,7 @@ END
foreach my $jr (@rows) {
if ($wrote++ >= $limit) {
- $write_cache_entry->($jr);
+ cache_write_entry(\*H, $jr);
next;
}
@@ -417,7 +417,7 @@ END
print H "<td>" if !$any_power;
print H "</td>\n";
- $write_cache_entry->($jr);
+ cache_write_entry(\*H, $jr);
print H "</tr>\n\n";
$alternate ^= 1;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 06/60] history reporting (nfc): Rename jobquery to cacheable_query
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (4 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 05/60] history reporting (nfc): Make cache_write_entry into a top-level sub Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 07/60] history reporting (nfc): Break out cache_row_lookup_prep Ian Jackson
` (53 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Prep work.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index fe5b7cc2..a8dae159 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -171,7 +171,7 @@ END
our $jqcachemisses = 0;
our $jqtotal = 0;
-sub jobquery ($$$) {
+sub cacheable_query ($$$) {
my ($q, $jr, $cachekey) = @_;
$jqtotal++;
$cachekey = '%'.$cachekey;
@@ -292,7 +292,7 @@ END
$cachehits++;
}
- my $endedrow = jobquery($endedq, $jr, 'e');
+ my $endedrow = cacheable_query($endedq, $jr, 'e');
if (!$endedrow) {
#print DEBUG "no-finished\n";
next;
@@ -322,8 +322,8 @@ END
}
#print DEBUG "JR $jr->{flight}.$jr->{job}\n";
- my $ir = jobquery($infoq, $jr, 'i');
- my $ar = jobquery($allocdq, $jr, 'a');
+ my $ir = cacheable_query($infoq, $jr, 'i');
+ my $ar = cacheable_query($allocdq, $jr, 'a');
my $ident = $jr->{name};
my $altcolour = report_altcolour($alternate);
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 07/60] history reporting (nfc): Break out cache_row_lookup_prep
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (5 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 06/60] history reporting (nfc): Rename jobquery to cacheable_query Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 08/60] history reporting (nfc): Rename "existing" to "previous" Ian Jackson
` (52 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Prep work.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index a8dae159..2724ef27 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -112,6 +112,16 @@ sub cache_read_existing ($) {
close H;
}
+sub cache_row_lookup_prep ($) {
+ my ($jrr) = @_;
+
+ my $cacherow = $cache{cache_row_key($$jrr)};
+ if ($cacherow) {
+ $$jrr = $cacherow;
+ $cachehits++;
+ }
+}
+
sub cache_write_entry ($$) {
my ($fh, $jr) = @_;
print $fh "<!-- osstest-report-reuseable";
@@ -286,11 +296,7 @@ END
foreach my $jr (@$inrows) {
#print DEBUG "JOB $jr->{flight}.$jr->{job} ";
- my $cacherow = $cache{cache_row_key($jr)};
- if ($cacherow) {
- $jr = $cacherow;
- $cachehits++;
- }
+ cache_row_lookup_prep(\$jr);
my $endedrow = cacheable_query($endedq, $jr, 'e');
if (!$endedrow) {
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 08/60] history reporting (nfc): Rename "existing" to "previous"
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (6 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 07/60] history reporting (nfc): Break out cache_row_lookup_prep Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 09/60] history reporting (nfc): Provide cacheable_fn Ian Jackson
` (51 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This seems more idiomatic use of English.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 2724ef27..f37dd4a2 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -32,7 +32,7 @@ our $limit= 2000;
our $timelimit= 86400 * (366 + 14);
our $flightlimit;
our $htmlout = ".";
-our $read_existing=1;
+our $read_previous=1;
our $doinstall=1;
our $maxjobs=10;
our @blessings;
@@ -58,7 +58,7 @@ while (@ARGV && $ARGV[0] =~ m/^-/) {
} elsif (m/^--html-dir=(.*)$/) {
$htmlout= $1;
} elsif (m/^--regenerate$/) {
- $read_existing= 0;
+ $read_previous= 0;
} elsif (m/^--no-install$/) {
$doinstall= 0;
} elsif (m/^--debug/) {
@@ -84,7 +84,7 @@ sub cache_row_key ($) {
return join $; , map { $jr->{$_} } @cache_row_key_cols;
}
-sub cache_read_existing ($) {
+sub cache_read_previous ($) {
my ($html_file) = @_;
if (!open H, $html_file) {
return if $!==ENOENT;
@@ -501,7 +501,7 @@ foreach my $host (sort keys %hosts) {
my $pid = fork // die $!;
if (!$pid) {
opendb_tests();
- cache_read_existing("$htmlout/$host.html") if $read_existing;
+ cache_read_previous("$htmlout/$host.html") if $read_previous;
db_retry($dbh_tests, [], sub {
mainquery($host);
reporthost $host;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 09/60] history reporting (nfc): Provide cacheable_fn
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (7 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 08/60] history reporting (nfc): Rename "existing" to "previous" Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 10/60] history reporting (nfc): Bind by name in cacheable_query Ian Jackson
` (50 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Prep work.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index f37dd4a2..90369ce4 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -181,20 +181,27 @@ END
our $jqcachemisses = 0;
our $jqtotal = 0;
-sub cacheable_query ($$$) {
- my ($q, $jr, $cachekey) = @_;
+sub cacheable_fn ($$$) {
+ my ($jr, $cachekey, $fn) = @_;
$jqtotal++;
$cachekey = '%'.$cachekey;
my $cached = $jr->{$cachekey};
if (!$cached) {
$jqcachemisses++;
- $q->execute($jr->{flight}, $jr->{job});
- $cached = $q->fetchrow_hashref();
+ $cached = $fn->();
$jr->{$cachekey} = $cached;
}
return $cached;
}
+sub cacheable_query ($$$) {
+ my ($q, $jr, $cachekey) = @_;
+ cacheable_fn($jr, $cachekey, sub {
+ $q->execute($jr->{flight}, $jr->{job});
+ return $q->fetchrow_hashref();
+ });
+}
+
our %hosts;
sub mainquery ($) {
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 10/60] history reporting (nfc): Bind by name in cacheable_query
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (8 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 09/60] history reporting (nfc): Provide cacheable_fn Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 11/60] history reporting (nfc): Rename $cachehits to $rows_hit Ian Jackson
` (49 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
cacheable_query used to simply pass $jr->{flight} and ->{job}. But we
want this to be reuseable for other kinds of query, with different
cache keys.
So bind by name: we expect the caller to use :name placeholders in the
query. We can then look through the prepared query parameters, and
fish the corresponding values out of $jr.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 90369ce4..8c5cd105 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -197,7 +197,11 @@ sub cacheable_fn ($$$) {
sub cacheable_query ($$$) {
my ($q, $jr, $cachekey) = @_;
cacheable_fn($jr, $cachekey, sub {
- $q->execute($jr->{flight}, $jr->{job});
+ foreach my $k (keys %{ $q->{ParamTypes} }) {
+ $k =~ m/^:/ or die "$k ?";
+ $q->bind_param($k, $jr->{$'} // die "$k ?");
+ }
+ $q->execute();
return $q->fetchrow_hashref();
});
}
@@ -259,7 +263,7 @@ sub reporthost ($) {
our $endedq //= db_prepare(<<END);
SELECT finished, testid, status AS laststepstatus
FROM steps
- WHERE flight=? AND job=? AND finished IS NOT NULL
+ WHERE flight=:flight AND job=:job AND finished IS NOT NULL
ORDER BY finished DESC
LIMIT 1
END
@@ -267,13 +271,13 @@ END
our $infoq //= db_prepare(<<END);
SELECT blessing, branch, intended
FROM flights
- WHERE flight=? AND ?!='X'
+ WHERE flight=:flight AND :job != 'X'
END
our $allocdq //= db_prepare(<<END);
SELECT testid, finished, status
FROM steps
- WHERE flight=? AND job=?
+ WHERE flight=:flight AND job=:job
AND (testid='hosts-allocate' OR step='ts-hosts-allocate')
ORDER BY finished ASC
LIMIT 1
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 11/60] history reporting (nfc): Rename $cachehits to $rows_hit
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (9 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 10/60] history reporting (nfc): Bind by name in cacheable_query Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 12/60] history reporting (nfc): Record query-specific stats Ian Jackson
` (48 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Prep work.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 8c5cd105..705367cf 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -118,7 +118,7 @@ sub cache_row_lookup_prep ($) {
my $cacherow = $cache{cache_row_key($$jrr)};
if ($cacherow) {
$$jrr = $cacherow;
- $cachehits++;
+ $rows_hit++;
}
}
@@ -303,7 +303,7 @@ END
# The contents of $jr for each job is cached across runs.
my @rows;
- my $cachehits = 0;
+ my $rows_hit = 0;
foreach my $jr (@$inrows) {
#print DEBUG "JOB $jr->{flight}.$jr->{job} ";
@@ -320,7 +320,7 @@ END
push @rows, { %$jr, %$endedrow };
}
- print DEBUG "CACHE $hostname $cachehits / ".(scalar @rows)
+ print DEBUG "CACHE $hostname $rows_hit / ".(scalar @rows)
." of ".(scalar %cache)."\n";
@rows = sort { $b->{finished} <=> $a->{finished} } @rows;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 12/60] history reporting (nfc): Record query-specific stats
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (10 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 11/60] history reporting (nfc): Rename $cachehits to $rows_hit Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 13/60] history reporting (nfc): Move cacheable_* further up the file Ian Jackson
` (47 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Prep work. Nothing uses these yet.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/sg-report-host-history b/sg-report-host-history
index 705367cf..32414322 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -77,6 +77,10 @@ our $flightcond;
our $minflight;
our %cache;
+
+our %q_count;
+our %q_misses;
+
our @cache_row_key_cols = qw(flight job status name);
sub cache_row_key ($) {
@@ -186,8 +190,10 @@ sub cacheable_fn ($$$) {
$jqtotal++;
$cachekey = '%'.$cachekey;
my $cached = $jr->{$cachekey};
+ $q_count{$cachekey}++;
if (!$cached) {
$jqcachemisses++;
+ $q_misses{$cachekey}++;
$cached = $fn->();
$jr->{$cachekey} = $cached;
}
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 13/60] history reporting (nfc): Move cacheable_* further up the file
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (11 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 12/60] history reporting (nfc): Record query-specific stats Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 14/60] history reporting (nfc): Use cacheable_fn for power methods Ian Jackson
` (46 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Pure code motion. Brings these together with the other work-cacheing
code.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 60 +++++++++++++++++++++++++-------------------------
1 file changed, 30 insertions(+), 30 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 32414322..43ddd2af 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -88,6 +88,36 @@ sub cache_row_key ($) {
return join $; , map { $jr->{$_} } @cache_row_key_cols;
}
+our $jqcachemisses = 0;
+our $jqtotal = 0;
+
+sub cacheable_fn ($$$) {
+ my ($jr, $cachekey, $fn) = @_;
+ $jqtotal++;
+ $cachekey = '%'.$cachekey;
+ my $cached = $jr->{$cachekey};
+ $q_count{$cachekey}++;
+ if (!$cached) {
+ $jqcachemisses++;
+ $q_misses{$cachekey}++;
+ $cached = $fn->();
+ $jr->{$cachekey} = $cached;
+ }
+ return $cached;
+}
+
+sub cacheable_query ($$$) {
+ my ($q, $jr, $cachekey) = @_;
+ cacheable_fn($jr, $cachekey, sub {
+ foreach my $k (keys %{ $q->{ParamTypes} }) {
+ $k =~ m/^:/ or die "$k ?";
+ $q->bind_param($k, $jr->{$'} // die "$k ?");
+ }
+ $q->execute();
+ return $q->fetchrow_hashref();
+ });
+}
+
sub cache_read_previous ($) {
my ($html_file) = @_;
if (!open H, $html_file) {
@@ -182,36 +212,6 @@ END
print DEBUG "MINFLIGHT $minflight\n";
}
-our $jqcachemisses = 0;
-our $jqtotal = 0;
-
-sub cacheable_fn ($$$) {
- my ($jr, $cachekey, $fn) = @_;
- $jqtotal++;
- $cachekey = '%'.$cachekey;
- my $cached = $jr->{$cachekey};
- $q_count{$cachekey}++;
- if (!$cached) {
- $jqcachemisses++;
- $q_misses{$cachekey}++;
- $cached = $fn->();
- $jr->{$cachekey} = $cached;
- }
- return $cached;
-}
-
-sub cacheable_query ($$$) {
- my ($q, $jr, $cachekey) = @_;
- cacheable_fn($jr, $cachekey, sub {
- foreach my $k (keys %{ $q->{ParamTypes} }) {
- $k =~ m/^:/ or die "$k ?";
- $q->bind_param($k, $jr->{$'} // die "$k ?");
- }
- $q->execute();
- return $q->fetchrow_hashref();
- });
-}
-
our %hosts;
sub mainquery ($) {
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 14/60] history reporting (nfc): Use cacheable_fn for power methods
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (12 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 13/60] history reporting (nfc): Move cacheable_* further up the file Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 15/60] history reporting (nfc): Introduce cache_set_key_cols Ian Jackson
` (45 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Avoids open-coding knowledge of the $jr convention at this use site.
%powers becomes %$powers, so it's a bit noisy.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 20 ++++++++------------
1 file changed, 8 insertions(+), 12 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 43ddd2af..9510757f 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -387,12 +387,8 @@ END
my $ri = report_run_getinfo({ %$jr, %$ir });
print H "<td $ri->{ColourAttr}>$ri->{Content}</td>\n";
- my %powers;
- if ($jr->{'%p'}) {
- %powers = %{ $jr->{'%p'} };
- $runvarq_hits++;
- } else {
- $runvarq_misses++;
+ my $powers = cacheable_fn($jr, 'b', sub {
+ my %powers;
$jrunvarq->execute($jr->{flight}, $jr->{job},
db_quote_like_pattern($ident));
my %runvars;
@@ -404,8 +400,8 @@ END
next unless $r =~ m{^\Q${ident}\E_power_};
$powers{$'} = $runvars{$r};
}
- $jr->{'%p'} = { %powers };
- }
+ \%powers;
+ });
my $skipped = 0;
my $any_power = 0;
my $pr_power_colour = sub {
@@ -421,7 +417,7 @@ END
for (; $skipped; $skipped--) {
$pr_power_colour->($grey_mid, " - ");
}
- my $how = $powers{$wh};
+ my $how = $powers->{$wh};
my $colour =
$how =~ /PDU/ ? $yellow :
$how =~ /Combined/ ? $yellow :
@@ -430,11 +426,11 @@ END
$pr_power_colour->($colour, " $how ");
};
foreach my $wh (qw(install recover)) {
- $skipped++, next unless exists $powers{$wh};
+ $skipped++, next unless exists $powers->{$wh};
$pr_power->($wh);
- delete $powers{$wh};
+ delete $powers->{$wh};
}
- foreach my $wh (sort keys %powers) {
+ foreach my $wh (sort keys %$powers) {
$pr_power->($wh);
}
print H "<td>" if !$any_power;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 15/60] history reporting (nfc): Introduce cache_set_key_cols
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (13 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 14/60] history reporting (nfc): Use cacheable_fn for power methods Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 16/60] history reporting (nfc): Record more row-specific stats Ian Jackson
` (44 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This setter function will be needed when this becomes its own module.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 9510757f..07e549a0 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -76,12 +76,14 @@ our $restrictflight_cond = restrictflight_cond();
our $flightcond;
our $minflight;
+our @cache_row_key_cols;
+
our %cache;
our %q_count;
our %q_misses;
-our @cache_row_key_cols = qw(flight job status name);
+sub cache_set_key_cols { @cache_row_key_cols = @_; }
sub cache_row_key ($) {
my ($jr) = @_;
@@ -177,6 +179,8 @@ sub cache_write_entry ($$) {
print $fh " -->\n";
}
+cache_set_key_cols(qw(flight job status name));
+
sub computeflightsrange () {
if ($flightlimit) {
my $minflightsq = db_prepare(<<END);
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 16/60] history reporting (nfc): Record more row-specific stats
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (14 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 15/60] history reporting (nfc): Introduce cache_set_key_cols Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 17/60] history reporting: Cache stats reporting: Centralise and rework Ian Jackson
` (43 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Prep work. Nothing uses these yet.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 07e549a0..32cc883d 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -83,6 +83,10 @@ our %cache;
our %q_count;
our %q_misses;
+our $rows_previous = 0;
+our $rows_today = 0;
+our $rows_hit = 0;
+
sub cache_set_key_cols { @cache_row_key_cols = @_; }
sub cache_row_key ($) {
@@ -128,6 +132,7 @@ sub cache_read_previous ($) {
}
%cache = ();
for (;;) {
+ $rows_previous++;
$_ = <H> // last;
next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
my $jr = {};
@@ -151,6 +156,7 @@ sub cache_read_previous ($) {
sub cache_row_lookup_prep ($) {
my ($jrr) = @_;
+ $rows_today++;
my $cacherow = $cache{cache_row_key($$jrr)};
if ($cacherow) {
$$jrr = $cacherow;
@@ -313,7 +319,6 @@ END
# The contents of $jr for each job is cached across runs.
my @rows;
- my $rows_hit = 0;
foreach my $jr (@$inrows) {
#print DEBUG "JOB $jr->{flight}.$jr->{job} ";
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 17/60] history reporting: Cache stats reporting: Centralise and rework
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (15 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 16/60] history reporting (nfc): Record more row-specific stats Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 18/60] history reporting (nfc): Remove now-obsolete stats variables Ian Jackson
` (42 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This uses the new variables instead of the old ones.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 18 ++++++++++++------
1 file changed, 12 insertions(+), 6 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 32cc883d..9c70b560 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -185,6 +185,17 @@ sub cache_write_entry ($$) {
print $fh " -->\n";
}
+sub cache_report_stats ($) {
+ my ($what) = @_;
+ print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today";
+ for my $cachekey (sort keys %q_count) {
+ my $total = $q_count{$cachekey};
+ my $hits = $total - ($q_misses{$cachekey} // 0);
+ print ::DEBUG " $cachekey=$hits/$total";
+ }
+ print ::DEBUG "\n";
+}
+
cache_set_key_cols(qw(flight job status name));
sub computeflightsrange () {
@@ -335,9 +346,6 @@ END
push @rows, { %$jr, %$endedrow };
}
- print DEBUG "CACHE $hostname $rows_hit / ".(scalar @rows)
- ." of ".(scalar %cache)."\n";
-
@rows = sort { $b->{finished} <=> $a->{finished} } @rows;
print DEBUG "SORTED\n";
@@ -457,8 +465,7 @@ END
rename "$html_file.new", "$html_file" or die "$html_file $!"
if $doinstall;
- print DEBUG "HOST CACHE RQ $runvarq_hits / ".
- ($runvarq_hits+$runvarq_misses)."\n";
+ cache_report_stats("$hostname");
}
foreach my $host (@ARGV) {
@@ -528,7 +535,6 @@ foreach my $host (sort keys %hosts) {
mainquery($host);
reporthost $host;
});
- print DEBUG "JQ CACHE ".($jqtotal-$jqcachemisses)." / $jqtotal\n";
exit(0);
}
print DEBUG "SPAWNED [$pid] $host\n";
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 18/60] history reporting (nfc): Remove now-obsolete stats variables
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (16 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 17/60] history reporting: Cache stats reporting: Centralise and rework Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 19/60] history reporting (nfc): Introduce empty HistoryReport module Ian Jackson
` (41 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 5 -----
1 file changed, 5 deletions(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 9c70b560..1b4566aa 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -94,17 +94,12 @@ sub cache_row_key ($) {
return join $; , map { $jr->{$_} } @cache_row_key_cols;
}
-our $jqcachemisses = 0;
-our $jqtotal = 0;
-
sub cacheable_fn ($$$) {
my ($jr, $cachekey, $fn) = @_;
- $jqtotal++;
$cachekey = '%'.$cachekey;
my $cached = $jr->{$cachekey};
$q_count{$cachekey}++;
if (!$cached) {
- $jqcachemisses++;
$q_misses{$cachekey}++;
$cached = $fn->();
$jr->{$cachekey} = $cached;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 19/60] history reporting (nfc): Introduce empty HistoryReport module
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (17 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 18/60] history reporting (nfc): Remove now-obsolete stats variables Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 20/60] history reporting (nfc): Move cache code into " Ian Jackson
` (40 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This is the boilerplate. Code will appear in it in a moment.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 43 +++++++++++++++++++++++++++++++++++++++++++
sg-report-host-history | 1 +
2 files changed, 44 insertions(+)
create mode 100644 Osstest/HistoryReport.pm
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
new file mode 100644
index 00000000..c5e7e226
--- /dev/null
+++ b/Osstest/HistoryReport.pm
@@ -0,0 +1,43 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2020 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+package Osstest::HistoryReport;
+
+use strict;
+use warnings;
+
+BEGIN {
+ use Exporter ();
+ our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+ $VERSION = 1.00;
+ @ISA = qw(Exporter);
+ @EXPORT = qw(
+ cache_set_key_cols
+ cache_read_previous
+ cache_row_lookup_prep
+ cacheable_query
+ cacheable_fn
+ cache_write_entry
+ cache_report_stats
+ );
+ %EXPORT_TAGS = ();
+
+ @EXPORT_OK = qw();
+}
+
+use POSIX;
+
+1;
diff --git a/sg-report-host-history b/sg-report-host-history
index 1b4566aa..05a2dfe0 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -27,6 +27,7 @@ use HTML::Entities;
use POSIX;
use Osstest::Executive qw(:DEFAULT :colours);
+use Osstest::HistoryReport;
our $limit= 2000;
our $timelimit= 86400 * (366 + 14);
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 20/60] history reporting (nfc): Move cache code into HistoryReport module
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (18 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 19/60] history reporting (nfc): Introduce empty HistoryReport module Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 21/60] history reporting (nfc): Rename some module variables, remove "cache" Ian Jackson
` (39 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Finally this is now reuseable code and we can put it in the
HistoryReport module.
Pure cut-and-paste.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 115 +++++++++++++++++++++++++++++++++++++++++++++++
sg-report-host-history | 115 -----------------------------------------------
2 files changed, 115 insertions(+), 115 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index c5e7e226..a0565b6a 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -40,4 +40,119 @@ BEGIN {
use POSIX;
+our @cache_row_key_cols;
+
+our %cache;
+
+our %q_count;
+our %q_misses;
+
+our $rows_previous = 0;
+our $rows_today = 0;
+our $rows_hit = 0;
+
+sub cache_set_key_cols { @cache_row_key_cols = @_; }
+
+sub cache_row_key ($) {
+ my ($jr) = @_;
+ return join $; , map { $jr->{$_} } @cache_row_key_cols;
+}
+
+sub cacheable_fn ($$$) {
+ my ($jr, $cachekey, $fn) = @_;
+ $cachekey = '%'.$cachekey;
+ my $cached = $jr->{$cachekey};
+ $q_count{$cachekey}++;
+ if (!$cached) {
+ $q_misses{$cachekey}++;
+ $cached = $fn->();
+ $jr->{$cachekey} = $cached;
+ }
+ return $cached;
+}
+
+sub cacheable_query ($$$) {
+ my ($q, $jr, $cachekey) = @_;
+ cacheable_fn($jr, $cachekey, sub {
+ foreach my $k (keys %{ $q->{ParamTypes} }) {
+ $k =~ m/^:/ or die "$k ?";
+ $q->bind_param($k, $jr->{$'} // die "$k ?");
+ }
+ $q->execute();
+ return $q->fetchrow_hashref();
+ });
+}
+
+sub cache_read_previous ($) {
+ my ($html_file) = @_;
+ if (!open H, $html_file) {
+ return if $!==ENOENT;
+ die "failed to open $html_file: $!";
+ }
+ %cache = ();
+ for (;;) {
+ $rows_previous++;
+ $_ = <H> // last;
+ next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
+ my $jr = {};
+ my $ch = $jr;
+ foreach (split / /, $1) {
+ if (m{^\w+$}) {
+ $ch = { };
+ $jr->{'%'.$&} = $ch;
+ next;
+ }
+ s{^(\w+)=}{} or die;
+ my $k = $1;
+ s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
+ $ch->{$k} = $_;
+ }
+ $cache{cache_row_key($jr)} = $jr;
+ }
+ close H;
+}
+
+sub cache_row_lookup_prep ($) {
+ my ($jrr) = @_;
+
+ $rows_today++;
+ my $cacherow = $cache{cache_row_key($$jrr)};
+ if ($cacherow) {
+ $$jrr = $cacherow;
+ $rows_hit++;
+ }
+}
+
+sub cache_write_entry ($$) {
+ my ($fh, $jr) = @_;
+ print $fh "<!-- osstest-report-reuseable";
+ my $whash = sub {
+ my ($h) = @_;
+ foreach my $k (sort keys %$h) {
+ next if $k =~ m/^\%/;
+ $_ = $h->{$k};
+ s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
+ printf $fh " %s=%s", $k, $_;
+ }
+ };
+ $whash->($jr);
+ foreach my $hk (sort keys %$jr) {
+ next unless $hk =~ m/^\%/;
+ print $fh " $'";
+ $whash->($jr->{$hk});
+ }
+ print $fh " -->\n";
+}
+
+sub cache_report_stats ($) {
+ my ($what) = @_;
+ print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today";
+ for my $cachekey (sort keys %q_count) {
+ my $total = $q_count{$cachekey};
+ my $hits = $total - ($q_misses{$cachekey} // 0);
+ print ::DEBUG " $cachekey=$hits/$total";
+ }
+ print ::DEBUG "\n";
+}
+
1;
diff --git a/sg-report-host-history b/sg-report-host-history
index 05a2dfe0..a195bb21 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -77,121 +77,6 @@ our $restrictflight_cond = restrictflight_cond();
our $flightcond;
our $minflight;
-our @cache_row_key_cols;
-
-our %cache;
-
-our %q_count;
-our %q_misses;
-
-our $rows_previous = 0;
-our $rows_today = 0;
-our $rows_hit = 0;
-
-sub cache_set_key_cols { @cache_row_key_cols = @_; }
-
-sub cache_row_key ($) {
- my ($jr) = @_;
- return join $; , map { $jr->{$_} } @cache_row_key_cols;
-}
-
-sub cacheable_fn ($$$) {
- my ($jr, $cachekey, $fn) = @_;
- $cachekey = '%'.$cachekey;
- my $cached = $jr->{$cachekey};
- $q_count{$cachekey}++;
- if (!$cached) {
- $q_misses{$cachekey}++;
- $cached = $fn->();
- $jr->{$cachekey} = $cached;
- }
- return $cached;
-}
-
-sub cacheable_query ($$$) {
- my ($q, $jr, $cachekey) = @_;
- cacheable_fn($jr, $cachekey, sub {
- foreach my $k (keys %{ $q->{ParamTypes} }) {
- $k =~ m/^:/ or die "$k ?";
- $q->bind_param($k, $jr->{$'} // die "$k ?");
- }
- $q->execute();
- return $q->fetchrow_hashref();
- });
-}
-
-sub cache_read_previous ($) {
- my ($html_file) = @_;
- if (!open H, $html_file) {
- return if $!==ENOENT;
- die "failed to open $html_file: $!";
- }
- %cache = ();
- for (;;) {
- $rows_previous++;
- $_ = <H> // last;
- next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
- my $jr = {};
- my $ch = $jr;
- foreach (split / /, $1) {
- if (m{^\w+$}) {
- $ch = { };
- $jr->{'%'.$&} = $ch;
- next;
- }
- s{^(\w+)=}{} or die;
- my $k = $1;
- s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
- $ch->{$k} = $_;
- }
- $cache{cache_row_key($jr)} = $jr;
- }
- close H;
-}
-
-sub cache_row_lookup_prep ($) {
- my ($jrr) = @_;
-
- $rows_today++;
- my $cacherow = $cache{cache_row_key($$jrr)};
- if ($cacherow) {
- $$jrr = $cacherow;
- $rows_hit++;
- }
-}
-
-sub cache_write_entry ($$) {
- my ($fh, $jr) = @_;
- print $fh "<!-- osstest-report-reuseable";
- my $whash = sub {
- my ($h) = @_;
- foreach my $k (sort keys %$h) {
- next if $k =~ m/^\%/;
- $_ = $h->{$k};
- s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
- printf $fh " %s=%s", $k, $_;
- }
- };
- $whash->($jr);
- foreach my $hk (sort keys %$jr) {
- next unless $hk =~ m/^\%/;
- print $fh " $'";
- $whash->($jr->{$hk});
- }
- print $fh " -->\n";
-}
-
-sub cache_report_stats ($) {
- my ($what) = @_;
- print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today";
- for my $cachekey (sort keys %q_count) {
- my $total = $q_count{$cachekey};
- my $hits = $total - ($q_misses{$cachekey} // 0);
- print ::DEBUG " $cachekey=$hits/$total";
- }
- print ::DEBUG "\n";
-}
-
cache_set_key_cols(qw(flight job status name));
sub computeflightsrange () {
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 21/60] history reporting (nfc): Rename some module variables, remove "cache"
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (19 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 20/60] history reporting (nfc): Move cache code into " Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 22/60] history reporting: Skip undefined keys Ian Jackson
` (38 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This makes the code terser and easier to read. No functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index a0565b6a..61b1ee8f 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -40,7 +40,7 @@ BEGIN {
use POSIX;
-our @cache_row_key_cols;
+our @key_cols;
our %cache;
@@ -51,11 +51,11 @@ our $rows_previous = 0;
our $rows_today = 0;
our $rows_hit = 0;
-sub cache_set_key_cols { @cache_row_key_cols = @_; }
+sub cache_set_key_cols { @key_cols = @_; }
-sub cache_row_key ($) {
+sub key ($) {
my ($jr) = @_;
- return join $; , map { $jr->{$_} } @cache_row_key_cols;
+ return join $; , map { $jr->{$_} } @key_cols;
}
sub cacheable_fn ($$$) {
@@ -107,7 +107,7 @@ sub cache_read_previous ($) {
s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
$ch->{$k} = $_;
}
- $cache{cache_row_key($jr)} = $jr;
+ $cache{key($jr)} = $jr;
}
close H;
}
@@ -116,7 +116,7 @@ sub cache_row_lookup_prep ($) {
my ($jrr) = @_;
$rows_today++;
- my $cacherow = $cache{cache_row_key($$jrr)};
+ my $cacherow = $cache{key($$jrr)};
if ($cacherow) {
$$jrr = $cacherow;
$rows_hit++;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 22/60] history reporting: Skip undefined keys
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (20 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 21/60] history reporting (nfc): Rename some module variables, remove "cache" Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 23/60] history reporting (nfc): Documentation for the new module Ian Jackson
` (37 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This makes it work if the caller's cached hash contains an key which
is bound to undef.
sg-report-host-history already does this, which currently causes:
Use of uninitialized value $_ in substitution (s///) at Osstest/HistoryReport.pm line 134.
Use of uninitialized value $_ in printf at Osstest/HistoryReport.pm line 135.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 61b1ee8f..cb8c6ca1 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -131,6 +131,7 @@ sub cache_write_entry ($$) {
foreach my $k (sort keys %$h) {
next if $k =~ m/^\%/;
$_ = $h->{$k};
+ next unless defined;
s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
printf $fh " %s=%s", $k, $_;
}
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 23/60] history reporting (nfc): Documentation for the new module
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (21 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 22/60] history reporting: Skip undefined keys Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 24/60] history reporting: Cache data limit now in History module Ian Jackson
` (36 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 27 +++++++++++++++++++++++++++
sg-report-host-history | 6 +-----
2 files changed, 28 insertions(+), 5 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index cb8c6ca1..6e5ed6ec 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -51,6 +51,33 @@ our $rows_previous = 0;
our $rows_today = 0;
our $rows_hit = 0;
+
+# History report query and processing cache
+#
+# Basic principle:
+#
+# Each HTML output file corresponds to the rows of one
+# primary query. That primary query yields rows containing
+# (at least) the columns passed to cache_set_key_cols.
+#
+# The HTML files contain caches of per-primary-row queries and
+# computations. Each such subcomputation is identified by a
+# $cachekey, which is assigned by the calling code (but should be
+# alphanumeric). The memoisation is keyed off the key columns and
+# $cachekey, so the subcomputation should not depend on anything else.
+#
+# The API is based on a %$jr.
+#
+# %$jr contains one entry for each for each key columns.
+#
+# It also contains one entry for each cached subcomputation.
+# The keys for these entries are "\%$cachekey".
+
+# These routines all expect the file handle ::DEBUG
+# They are not reentrant and can only be used for one HTML
+# output file in one program run.
+
+
sub cache_set_key_cols { @key_cols = @_; }
sub key ($) {
diff --git a/sg-report-host-history b/sg-report-host-history
index a195bb21..e281986d 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -204,11 +204,7 @@ END
my $inrows = $hosts{$hostname};
print DEBUG "FOUND ", (scalar @$inrows), " ROWS for $hostname\n";
- # Each entry in @$inrows is a $jr, which is a hash
- # It has keys for the result columns in mainquery
- # It also has keys '%<letter>' (yes, with a literal '%')
- # which are the results of per-job queries.
- # The contents of $jr for each job is cached across runs.
+ # Each entry in @$inrows is a HistoryReport $jr
my @rows;
foreach my $jr (@$inrows) {
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 24/60] history reporting: Cache data limit now in History module
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (22 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 23/60] history reporting (nfc): Documentation for the new module Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 25/60] history reporting: Print debug for cache misses Ian Jackson
` (35 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Replace the ad-hoc query-specific limit strategy in
sg-report-host-history with a new, more principled, arrangement, in
HistoryReport.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 38 ++++++++++++++++++++++++++++++++------
sg-report-host-history | 14 +++++---------
2 files changed, 37 insertions(+), 15 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 6e5ed6ec..b576c02a 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -31,7 +31,7 @@ BEGIN {
cacheable_query
cacheable_fn
cache_write_entry
- cache_report_stats
+ cache_finish
);
%EXPORT_TAGS = ();
@@ -43,6 +43,7 @@ use POSIX;
our @key_cols;
our %cache;
+our @previous;
our %q_count;
our %q_misses;
@@ -50,6 +51,7 @@ our %q_misses;
our $rows_previous = 0;
our $rows_today = 0;
our $rows_hit = 0;
+our $rows_extra = 0;
# History report query and processing cache
@@ -73,6 +75,12 @@ our $rows_hit = 0;
# It also contains one entry for each cached subcomputation.
# The keys for these entries are "\%$cachekey".
+# Limit strategy
+#
+# It is up to the caller to make a primary query that returns the rows
+# that it is interested in. They should be in (perhaps approximate)
+# reverse time order.
+
# These routines all expect the file handle ::DEBUG
# They are not reentrant and can only be used for one HTML
# output file in one program run.
@@ -118,7 +126,6 @@ sub cache_read_previous ($) {
}
%cache = ();
for (;;) {
- $rows_previous++;
$_ = <H> // last;
next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
my $jr = {};
@@ -134,6 +141,7 @@ sub cache_read_previous ($) {
s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
$ch->{$k} = $_;
}
+ push @previous, $jr;
$cache{key($jr)} = $jr;
}
close H;
@@ -169,18 +177,36 @@ sub cache_write_entry ($$) {
print $fh " $'";
$whash->($jr->{$hk});
}
+ $jr->{'% written'} = 1;
print $fh " -->\n";
}
-sub cache_report_stats ($) {
- my ($what) = @_;
- print ::DEBUG "CACHE $what read=$rows_previous hits $rows_hit/$rows_today";
+sub cache_finish ($$) {
+ my ($fh, $what) = @_;
+
+ my $nprevious = @previous;
+
+ # Write as many rows again. This saves re-querying in case of
+ # flapping etc.
+ for (;;) {
+ last if
+ $rows_extra >= $rows_today &&
+ ($rows_today + $rows_extra >= @previous * 0.9);
+ my $jr = shift @previous;
+ last unless $jr;
+ next if $jr->{'% written'};
+ cache_write_entry($fh, $jr);
+ $rows_extra++;
+ }
+
+ print ::DEBUG
+ "CACHE $what read=$nprevious hits $rows_hit/$rows_today";
for my $cachekey (sort keys %q_count) {
my $total = $q_count{$cachekey};
my $hits = $total - ($q_misses{$cachekey} // 0);
print ::DEBUG " $cachekey=$hits/$total";
}
- print ::DEBUG "\n";
+ print ::DEBUG " extra=$rows_extra\n";
}
1;
diff --git a/sg-report-host-history b/sg-report-host-history
index e281986d..2ad2134d 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -119,7 +119,7 @@ our %hosts;
sub mainquery ($) {
my ($host) = @_;
- our $runvarq //= db_prepare(<<END);
+ our $runvarq = db_prepare(<<END.($limit ? <<END_LIMIT : ''));
SELECT flight, job, name, status
FROM runvars
JOIN jobs USING (flight, job)
@@ -128,8 +128,9 @@ sub mainquery ($) {
AND $flightcond
AND $restrictflight_cond
ORDER BY flight DESC
- LIMIT $limit * 2
END
+ LIMIT $limit
+END_LIMIT
print DEBUG "MAINQUERY $host...\n";
$runvarq->execute($host);
@@ -233,11 +234,6 @@ END
my $runvarq_misses = 0;
foreach my $jr (@rows) {
- if ($wrote++ >= $limit) {
- cache_write_entry(\*H, $jr);
- next;
- }
-
#print DEBUG "JR $jr->{flight}.$jr->{job}\n";
my $ir = cacheable_query($infoq, $jr, 'i');
my $ar = cacheable_query($allocdq, $jr, 'a');
@@ -338,11 +334,11 @@ END
print H "</table></body></html>\n";
+ cache_finish(\*H, "$hostname");
+
close H or die $!;
rename "$html_file.new", "$html_file" or die "$html_file $!"
if $doinstall;
-
- cache_report_stats("$hostname");
}
foreach my $host (@ARGV) {
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 25/60] history reporting: Print debug for cache misses
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (23 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 24/60] history reporting: Cache data limit now in History module Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 26/60] history reporting: Improve an error message slightly Ian Jackson
` (34 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 3 +++
1 file changed, 3 insertions(+)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index b576c02a..b593f207 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -155,6 +155,9 @@ sub cache_row_lookup_prep ($) {
if ($cacherow) {
$$jrr = $cacherow;
$rows_hit++;
+ } else {
+ print ::DEBUG "CACHE MISS ",
+ (join " ", map { $$jrr->{$_} } @key_cols), "\n";
}
}
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 26/60] history reporting: Improve an error message slightly
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (24 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 25/60] history reporting: Print debug for cache misses Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 27/60] sg-report-host-history: Write cache entry for unfinished jobs Ian Jackson
` (33 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index b593f207..6a23cfba 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -128,6 +128,7 @@ sub cache_read_previous ($) {
for (;;) {
$_ = <H> // last;
next unless m{^\<\!-- osstest-report-reuseable (.*)--\>$};
+ my $orig = $_;
my $jr = {};
my $ch = $jr;
foreach (split / /, $1) {
@@ -136,7 +137,7 @@ sub cache_read_previous ($) {
$jr->{'%'.$&} = $ch;
next;
}
- s{^(\w+)=}{} or die;
+ s{^(\w+)=}{} or die "$orig -- $_ ";
my $k = $1;
s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
$ch->{$k} = $_;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 27/60] sg-report-host-history: Write cache entry for unfinished jobs
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (25 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 26/60] history reporting: Improve an error message slightly Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 28/60] parallel by fork: Break out into HistoryReport Ian Jackson
` (32 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
We have to also check ->{finished}, rather than the existence of a row
at all, since now unfinished jobs can appear in the cache.
Because the cache key includes the job status, when the job becomes
finished the cache entry will be invalidated.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index 2ad2134d..6bf14aa2 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -214,7 +214,8 @@ END
cache_row_lookup_prep(\$jr);
my $endedrow = cacheable_query($endedq, $jr, 'e');
- if (!$endedrow) {
+ if (!defined $endedrow->{finished}) {
+ cache_write_entry(\*H, $jr);
#print DEBUG "no-finished\n";
next;
}
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 28/60] parallel by fork: Break out into HistoryReport
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (26 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 27/60] sg-report-host-history: Write cache entry for unfinished jobs Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 29/60] parallel by fork: Disconnect $dbh_tests as well as undefing it Ian Jackson
` (31 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Move this code from sg-report-host-history to HistoryReport, so that
it can be reused.
No significant functional change. Some changes to debug messages.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 51 ++++++++++++++++++++++++++++++++++++++++++++++++
sg-report-host-history | 49 +++++++++-------------------------------------
2 files changed, 60 insertions(+), 40 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 6a23cfba..0b206de4 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -19,6 +19,9 @@ package Osstest::HistoryReport;
use strict;
use warnings;
+use Osstest;
+use Osstest::Executive;
+
BEGIN {
use Exporter ();
our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
@@ -32,6 +35,7 @@ BEGIN {
cacheable_fn
cache_write_entry
cache_finish
+ parallel_by_fork
);
%EXPORT_TAGS = ();
@@ -213,4 +217,51 @@ sub cache_finish ($$) {
print ::DEBUG " extra=$rows_extra\n";
}
+our %children;
+our $worst = 0;
+our $whoami;
+
+sub wait_for_max_children ($) {
+ my ($lim) = @_;
+ while (keys(%children) > $lim) {
+ $!=0; $?=0; my $got = wait;
+ die "$! $got $?" unless exists $children{$got};
+ my $host = $children{$got};
+ delete $children{$got};
+ $worst = $? if $? > $worst;
+ if ($?) {
+ print STDERR $whoami."[$$]: [$got] failed for $host: $?\n";
+ } else {
+ print ::DEBUG "REAPED [$got] $host\n";
+ }
+ }
+}
+
+sub parallel_by_fork ($$$$) {
+ my ($set_whoami, $maxjobs, $tasks, $fn) = @_;
+ # does $fn->($_) foreach @$tasks
+ # but in parallal and then exits.
+ # entries in @$taskarray should be suitable for print in messages.
+ # db is reopened in each child.
+
+ $whoami = $set_whoami;
+ undef $dbh_tests;
+
+ foreach my $task (@$tasks) {
+ wait_for_max_children($maxjobs);
+
+ my $pid = fork // die $!;
+ if (!$pid) {
+ opendb_tests();
+ $fn->($task);
+ exit 0;
+ }
+ print ::DEBUG "SPAWNED [$pid] $task\n";
+ $children{$pid} = $task;
+ }
+
+ wait_for_max_children(0);
+ exit $worst;
+}
+
1;
diff --git a/sg-report-host-history b/sg-report-host-history
index 6bf14aa2..0a2e9904 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -377,43 +377,12 @@ db_retry($dbh_tests, [], sub {
computeflightsrange();
});
-undef $dbh_tests;
-
-our %children;
-our $worst = 0;
-
-sub wait_for_max_children ($) {
- my ($lim) = @_;
- while (keys(%children) > $lim) {
- $!=0; $?=0; my $got = wait;
- die "$! $got $?" unless exists $children{$got};
- my $host = $children{$got};
- delete $children{$got};
- $worst = $? if $? > $worst;
- if ($?) {
- print STDERR "sg-report-flight[: [$got] failed for $host: $?\n";
- } else {
- print DEBUG "REAPED [$got] $host\n";
- }
- }
-}
-
-foreach my $host (sort keys %hosts) {
- wait_for_max_children($maxjobs);
-
- my $pid = fork // die $!;
- if (!$pid) {
- opendb_tests();
- cache_read_previous("$htmlout/$host.html") if $read_previous;
- db_retry($dbh_tests, [], sub {
- mainquery($host);
- reporthost $host;
- });
- exit(0);
- }
- print DEBUG "SPAWNED [$pid] $host\n";
- $children{$pid} = $host;
-}
-
-wait_for_max_children(0);
-exit $worst;
+parallel_by_fork('sg-report-flight', $maxjobs, [ sort keys %hosts ], sub {
+ my ($host) = @_;
+ cache_read_previous("$htmlout/$host.html") if $read_previous;
+ db_retry($dbh_tests, [], sub {
+ mainquery($host);
+ reporthost $host;
+ });
+ exit(0);
+});
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 29/60] parallel by fork: Disconnect $dbh_tests as well as undefing it
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (27 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 28/60] parallel by fork: Break out into HistoryReport Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 30/60] parallel by fork: Fix a variable name to $task Ian Jackson
` (30 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
If the caller is buggy and has statement handles still open, they can
still "work" even if we have thrown away the db handle.
Where, after forking, "work" means "use the same connection in
multiple processes simultaneously, without locking". This could
result in arbitrary crazy nbehaviour (eg, TLS crypto failures).
No functional change with existing callers since they don't have this
bug.
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 0b206de4..f0571189 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -245,6 +245,7 @@ sub parallel_by_fork ($$$$) {
# db is reopened in each child.
$whoami = $set_whoami;
+ $dbh_tests->disconnect();
undef $dbh_tests;
foreach my $task (@$tasks) {
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 30/60] parallel by fork: Fix a variable name to $task
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (28 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 29/60] parallel by fork: Disconnect $dbh_tests as well as undefing it Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 31/60] sg-report-job-history: Prep for fork: Move $buildsq query Ian Jackson
` (29 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This code came from sg-report-host-history where tasks were hosts.
But in the more general context, the names are wrong.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index f0571189..695d744a 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -226,13 +226,13 @@ sub wait_for_max_children ($) {
while (keys(%children) > $lim) {
$!=0; $?=0; my $got = wait;
die "$! $got $?" unless exists $children{$got};
- my $host = $children{$got};
+ my $task = $children{$got};
delete $children{$got};
$worst = $? if $? > $worst;
if ($?) {
- print STDERR $whoami."[$$]: [$got] failed for $host: $?\n";
+ print STDERR $whoami."[$$]: [$got] failed for $task: $?\n";
} else {
- print ::DEBUG "REAPED [$got] $host\n";
+ print ::DEBUG "REAPED [$got] $task\n";
}
}
}
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 31/60] sg-report-job-history: Prep for fork: Move $buildsq query
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (29 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 30/60] parallel by fork: Fix a variable name to $task Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 32/60] sg-report-job-history: Prep for fork: Move $revisionsq query Ian Jackson
` (28 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
We will need to prepare this once per (job,branch) so that it works
when we do each of those in a different process.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 6008ca72..47fc7a62 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -106,15 +106,15 @@ sub add_revisions ($$$$) {
}
}
-our $buildsq= db_prepare(<<END);
+sub processjobbranch ($$) {
+ my ($j,$bra) = @_;
+
+ my $buildsq= db_prepare(<<END);
SELECT * FROM runvars
WHERE flight=? AND job=?
AND name LIKE '%buildjob'
END
-sub processjobbranch ($$) {
- my ($j,$bra) = @_;
-
print DEBUG "processjobbranch('$j',", ($bra ? "'$bra'" : 'undef'), ")\n";
my %rev_grid_col;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 32/60] sg-report-job-history: Prep for fork: Move $revisionsq query
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (30 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 31/60] sg-report-job-history: Prep for fork: Move $buildsq query Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 33/60] sg-report-job-history: Use fork-based parallelism Ian Jackson
` (27 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
We will need to prepare this in add_revisions so that it works when we
do each (job,branch) in a different process.
It is OK that it is still global, becauswe we only call add_revisions
in the children.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 47fc7a62..424053f1 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -89,15 +89,16 @@ if (defined($flight)) {
push @jobs, $job;
}
-our $revisionsq= db_prepare(<<END);
+sub add_revisions ($$$$) {
+ my ($revmap, $flightnum, $j, $sfx) = @_;
+
+ our $revisionsq //= db_prepare(<<END);
SELECT * FROM runvars
WHERE flight=? AND job=?
AND name LIKE 'built\\_revision\\_%'
END
-# (We report on non-main-revision jobs just as for main-revision ones.)
+ # (We report on non-main-revision jobs just as for main-revision ones.)
-sub add_revisions ($$$$) {
- my ($revmap, $flightnum, $j, $sfx) = @_;
$revisionsq->execute($flightnum, $j);
while (my $row= $revisionsq->fetchrow_hashref()) {
my $n= $row->{name};
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 33/60] sg-report-job-history: Use fork-based parallelism
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (31 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 32/60] sg-report-job-history: Prep for fork: Move $revisionsq query Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 34/60] sg-report-job-history: Use one child per report Ian Jackson
` (26 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
For now, one child per job (for all branches). This is already a
speedup.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 424053f1..384a7670 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -26,11 +26,13 @@ use IO::Handle;
use HTML::Entities;
use Osstest::Executive;
+use Osstest::HistoryReport;
our (@blessings,@branches);
our $limit= 100;
our $htmlout;
our $flight;
+our $maxjobs=10;
our $job;
open DEBUG, ">/dev/null";
@@ -38,7 +40,7 @@ open DEBUG, ">/dev/null";
while (@ARGV && $ARGV[0] =~ m/^-/) {
$_= shift @ARGV;
last if m/^--?$/;
- if (m/^--(job|flight)\=(.*)$/) {
+ if (m/^--(job|flight|maxjobs)\=(.*)$/) {
$$1= $2;
} elsif (m/^--(limit)\=([1-9]\d*)$/) {
$$1= $2;
@@ -300,6 +302,9 @@ sub processjob ($) {
processjobbranch($j,$_) foreach @branches;
}
-db_begin_work($dbh_tests, []);
-db_readonly_report();
-foreach my $j (@jobs) { processjob($j); }
+parallel_by_fork('sg-report-job-history', $maxjobs, \@jobs, sub {
+ my ($j) = @_;
+ db_retry($dbh_tests, [], sub {
+ processjob($j);
+ });
+});
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 34/60] sg-report-job-history: Use one child per report
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (32 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 33/60] sg-report-job-history: Use fork-based parallelism Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 35/60] sg-report-job-history (nfc): Have main program decide HTML filename Ian Jackson
` (25 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Rather than one child per job, which then did one report per branch.
This will mean we can use the cache machinery, which is rather global
so wouldn't cope well with processing multiple job history reports
within a process.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 384a7670..3307b281 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -296,15 +296,16 @@ END
}
}
-sub processjob ($) {
- my ($j) = @_;
- processjobbranch($j,undef);
- processjobbranch($j,$_) foreach @branches;
+my @tasks;
+foreach my $j (@jobs) {
+ push @tasks, $j;
+ push @tasks, "$j $_" foreach @branches;
}
-parallel_by_fork('sg-report-job-history', $maxjobs, \@jobs, sub {
- my ($j) = @_;
+parallel_by_fork('sg-report-job-history', $maxjobs, \@tasks, sub {
+ my ($task) = @_;
+ my ($job, $branch) = split / /, $task; # $branch might be undef
db_retry($dbh_tests, [], sub {
- processjob($j);
+ processjobbranch($job, $branch);
});
});
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 35/60] sg-report-job-history (nfc): Have main program decide HTML filename
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (33 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 34/60] sg-report-job-history: Use one child per report Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 36/60] sg-report-job-history: Always write HTML output Ian Jackson
` (24 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 3307b281..3f10a74c 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -109,8 +109,8 @@ END
}
}
-sub processjobbranch ($$) {
- my ($j,$bra) = @_;
+sub processjobbranch ($$$) {
+ my ($j,$bra,$html_file) = @_;
my $buildsq= db_prepare(<<END);
SELECT * FROM runvars
@@ -210,15 +210,13 @@ END
END
if (defined $htmlout) {
- my ($title,$html_file,$url);
+ my ($title,$url);
ensuredir "$htmlout/history";
ensuredir "$htmlout/history/$j";
if (defined $bra) {
$title= "$j (branch $bra)";
- $html_file= "history/$j/$bra.html";
} else {
$title= "$j (all branches)";
- $html_file= "history/$j/ALL.html";
}
$html_file= "$htmlout/$html_file";
open H, "> $html_file.new" or die "$html_file $!";
@@ -305,7 +303,8 @@ foreach my $j (@jobs) {
parallel_by_fork('sg-report-job-history', $maxjobs, \@tasks, sub {
my ($task) = @_;
my ($job, $branch) = split / /, $task; # $branch might be undef
+ my $html_file = sprintf "history/%s/%s.html", $job, ($branch // 'ALL');
db_retry($dbh_tests, [], sub {
- processjobbranch($job, $branch);
+ processjobbranch($job, $branch, $html_file);
});
});
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 36/60] sg-report-job-history: Always write HTML output
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (34 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 35/60] sg-report-job-history (nfc): Have main program decide HTML filename Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 37/60] sg-report-job-history (nfc): Remove needless conditional Ian Jackson
` (23 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Previously, unlike sg-report-host-history, if you didn't specify
--html-dir, it would would do a lot work to no effect.
This is not useful and nothing calls it this way. So abolish this
notion.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 3f10a74c..7197a5f6 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -30,7 +30,7 @@ use Osstest::HistoryReport;
our (@blessings,@branches);
our $limit= 100;
-our $htmlout;
+our $htmlout = '.';
our $flight;
our $maxjobs=10;
our $job;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 37/60] sg-report-job-history (nfc): Remove needless conditional
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (35 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 36/60] sg-report-job-history: Always write HTML output Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 38/60] history reporting (nfc): Add another test rune to the notes Ian Jackson
` (22 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
$htmlout is now always defined.
Nothing other than indentation change, and removal of the surrounding
if block.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 162 +++++++++++++++++++++++++-------------------------
1 file changed, 80 insertions(+), 82 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 7197a5f6..3b45992f 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -209,89 +209,87 @@ END
WHERE flight=?
END
- if (defined $htmlout) {
- my ($title,$url);
- ensuredir "$htmlout/history";
- ensuredir "$htmlout/history/$j";
- if (defined $bra) {
- $title= "$j (branch $bra)";
- } else {
- $title= "$j (all branches)";
- }
- $html_file= "$htmlout/$html_file";
- open H, "> $html_file.new" or die "$html_file $!";
- $title= encode_entities($title);
- print H "<html><head><title>$title</title></head><body>\n";
- print H "<h1>$title</h1>\n";
- print H "<table rules=all>";
- print H "<tr><th>started</th><th>flight</th>",
- "<th>branch</th><th>failure</th>\n";
- print H "<th>", (join ", ", @hostvarcols), "</th>\n";
- print H "<th>osstest</th>";
- foreach my $c (@rev_grid_cols) {
- print H "<th>".encode_entities($c)."</th>\n";
- }
- print H "</tr>\n";
- my $alternate= 0;
- my @last_revs;
- my @alt_revs;
- my $alt_hosts;
- my $alt_osstest;
- foreach my $r (@test_rows) {
- my $altcolour= report_altcolour($alternate);
- print H "<tr $altcolour>";
- my $started = $r->{Flight}{started};
- print H "<td>";
- print H show_abs_time $started if $started;
- print H "</td>\n";
- my $flt= $r->{Flight}{flight};
- $url= "$c{ReportHtmlPubBaseUrl}/$flt";
- print H "<td><a href=\"$url\">$flt</a></td>\n";
- print H "<td>".encode_entities($r->{Flight}{branch})."</td>\n";
- $url= "$c{ReportHtmlPubBaseUrl}/$flt/".encode_entities($j).
- "/info.html";
- print H "<td $r->{ColourAttr}><a href=\"$url\">".
- $r->{Content}."</a></td>\n";
- my $lastrev;
- my $hosts = join ", ", map { $_ // "-" } @{ $r->{Hosts} };
- my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts);
- print H "<td $hosts_colour>".encode_entities($hosts)."</td>\n";
- $osstestverq->execute($r->{Flight}{flight});
- my $osstestrevs = join ' ',
- map {
- $_ = $_->{harness};
- s/^([0-9a-f]{12})[0-9a-f]+\b/$1/;
- "<kbd>".encode_entities($_)."</kbd>";
- }
- @{ $osstestverq->fetchall_arrayref({}) };
- my $osstest_colour = report_altchangecolour(\$alt_osstest, $osstestrevs);
- print H "<td $osstest_colour>$osstestrevs</td>\n";
- foreach my $i (0..$#rev_grid_cols) {
- my $v= $r->{Revisions}[$i];
- my $rev_colour = report_altchangecolour(\$alt_revs[$i], $v);
- print H "<td $rev_colour>";
- if (defined $v) {
- my $vp= $v;
- if (defined $lastrev && $v eq $lastrev) {
- $vp= '<-';
- } else {
- $vp =~ s/^((?:\d+\:)?[0-9a-f]{12})([0-9a-f].*)$/$1/;
- }
- print H "<kbd>".encode_entities($vp)."</kbd>";
- }
- $lastrev= $v;
- $last_revs[$i]= $v;
- print H "</td>";
- }
- print H "</tr>\n";
- $alternate ^= 1;
- }
- print H "</table></body></html>\n";
- H->error and die $!;
- close H or die $!;
- rename "$html_file.new", $html_file or die "$html_file $!";
- #print "wrote $html_file\n";
+ my ($title,$url);
+ ensuredir "$htmlout/history";
+ ensuredir "$htmlout/history/$j";
+ if (defined $bra) {
+ $title= "$j (branch $bra)";
+ } else {
+ $title= "$j (all branches)";
+ }
+ $html_file= "$htmlout/$html_file";
+ open H, "> $html_file.new" or die "$html_file $!";
+ $title= encode_entities($title);
+ print H "<html><head><title>$title</title></head><body>\n";
+ print H "<h1>$title</h1>\n";
+ print H "<table rules=all>";
+ print H "<tr><th>started</th><th>flight</th>",
+ "<th>branch</th><th>failure</th>\n";
+ print H "<th>", (join ", ", @hostvarcols), "</th>\n";
+ print H "<th>osstest</th>";
+ foreach my $c (@rev_grid_cols) {
+ print H "<th>".encode_entities($c)."</th>\n";
+ }
+ print H "</tr>\n";
+ my $alternate= 0;
+ my @last_revs;
+ my @alt_revs;
+ my $alt_hosts;
+ my $alt_osstest;
+ foreach my $r (@test_rows) {
+ my $altcolour= report_altcolour($alternate);
+ print H "<tr $altcolour>";
+ my $started = $r->{Flight}{started};
+ print H "<td>";
+ print H show_abs_time $started if $started;
+ print H "</td>\n";
+ my $flt= $r->{Flight}{flight};
+ $url= "$c{ReportHtmlPubBaseUrl}/$flt";
+ print H "<td><a href=\"$url\">$flt</a></td>\n";
+ print H "<td>".encode_entities($r->{Flight}{branch})."</td>\n";
+ $url= "$c{ReportHtmlPubBaseUrl}/$flt/".encode_entities($j).
+ "/info.html";
+ print H "<td $r->{ColourAttr}><a href=\"$url\">".
+ $r->{Content}."</a></td>\n";
+ my $lastrev;
+ my $hosts = join ", ", map { $_ // "-" } @{ $r->{Hosts} };
+ my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts);
+ print H "<td $hosts_colour>".encode_entities($hosts)."</td>\n";
+ $osstestverq->execute($r->{Flight}{flight});
+ my $osstestrevs = join ' ',
+ map {
+ $_ = $_->{harness};
+ s/^([0-9a-f]{12})[0-9a-f]+\b/$1/;
+ "<kbd>".encode_entities($_)."</kbd>";
+ }
+ @{ $osstestverq->fetchall_arrayref({}) };
+ my $osstest_colour = report_altchangecolour(\$alt_osstest, $osstestrevs);
+ print H "<td $osstest_colour>$osstestrevs</td>\n";
+ foreach my $i (0..$#rev_grid_cols) {
+ my $v= $r->{Revisions}[$i];
+ my $rev_colour = report_altchangecolour(\$alt_revs[$i], $v);
+ print H "<td $rev_colour>";
+ if (defined $v) {
+ my $vp= $v;
+ if (defined $lastrev && $v eq $lastrev) {
+ $vp= '<-';
+ } else {
+ $vp =~ s/^((?:\d+\:)?[0-9a-f]{12})([0-9a-f].*)$/$1/;
+ }
+ print H "<kbd>".encode_entities($vp)."</kbd>";
+ }
+ $lastrev= $v;
+ $last_revs[$i]= $v;
+ print H "</td>";
+ }
+ print H "</tr>\n";
+ $alternate ^= 1;
}
+ print H "</table></body></html>\n";
+ H->error and die $!;
+ close H or die $!;
+ rename "$html_file.new", $html_file or die "$html_file $!";
+ #print "wrote $html_file\n";
}
my @tasks;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 38/60] history reporting (nfc): Add another test rune to the notes
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (36 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 37/60] sg-report-job-history (nfc): Remove needless conditional Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 39/60] history reporting (nfc): Make cacheable_fn work without cache Ian Jackson
` (21 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
runes | 2 ++
1 file changed, 2 insertions(+)
diff --git a/runes b/runes
index 8139f955..829e6106 100644
--- a/runes
+++ b/runes
@@ -11,3 +11,5 @@ select * from jobs full outer join steps using (job, flight) where flight=2364 o
>v; env DBI_TRACE=2=v time ./sg-report-host-history --debug --html-dir=$PWD/e --max-flight=152223 flight:152223 2>&1 |tee c |ts -s %.s >c.ts
rm -rf d; mkdir d; cp -a e/. d/.; mv c.ts b.ts; mv c b; mv v u
cp -a d/. e/.
+
+>v; env DBI_TRACE=2=v time ./sg-report-job-history --debug --html-dir=$PWD/e --max-flight=152223 --flight=152223 2>&1 |tee c |ts -s %.s >c.ts
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 39/60] history reporting (nfc): Make cacheable_fn work without cache
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (37 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 38/60] history reporting (nfc): Add another test rune to the notes Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 40/60] sg-report-job-history: Refactor "ALL" handling Ian Jackson
` (20 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This would allow us to use this in call sites without a cache.
I changed my mind about the code that prompted this, but it still
seems plausibly useful, so I'm keeping this commit.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 695d744a..5d2c83b7 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -99,6 +99,7 @@ sub key ($) {
sub cacheable_fn ($$$) {
my ($jr, $cachekey, $fn) = @_;
+ return $fn->() if !defined $cachekey;
$cachekey = '%'.$cachekey;
my $cached = $jr->{$cachekey};
$q_count{$cachekey}++;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 40/60] sg-report-job-history: Refactor "ALL" handling
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (38 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 39/60] history reporting (nfc): Make cacheable_fn work without cache Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 41/60] sg-report-job-history (nfc): Make $ri->{Hosts} a hash Ian Jackson
` (19 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
* Make an explicit entry ALL in @branches, rather than implicitly
processing ALL as well.
* Consequently, put explicit ALL entries in @tasks too, rather than
putting in entries without a branch name.
* Pass ALL to processjobbranch rather than undef, and turn it into
the internally-used undef at the start.
When used with --flight (findflight), this has no functional change.
When used with --job, ALL must now be included in the branch
list passed to --branches. The only in-tree call is with --flight.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 3b45992f..8932458e 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -72,7 +72,7 @@ sub findflight () {
SELECT branch FROM flights WHERE flight=?
END
die "$flight ? @$branches ?" if @$branches!=1;
- @branches= @$branches;
+ @branches= ('ALL', @$branches);
my $selectq= db_prepare(<<END);
SELECT job FROM jobs WHERE flight=? ORDER BY JOB
@@ -110,7 +110,10 @@ END
}
sub processjobbranch ($$$) {
- my ($j,$bra,$html_file) = @_;
+ my ($j,$branch_or_all,$html_file) = @_;
+
+ my $bra = $branch_or_all;
+ undef $bra if $bra eq 'ALL';
my $buildsq= db_prepare(<<END);
SELECT * FROM runvars
@@ -294,14 +297,13 @@ END
my @tasks;
foreach my $j (@jobs) {
- push @tasks, $j;
push @tasks, "$j $_" foreach @branches;
}
parallel_by_fork('sg-report-job-history', $maxjobs, \@tasks, sub {
my ($task) = @_;
my ($job, $branch) = split / /, $task; # $branch might be undef
- my $html_file = sprintf "history/%s/%s.html", $job, ($branch // 'ALL');
+ my $html_file = sprintf "history/%s/%s.html", $job, $branch;
db_retry($dbh_tests, [], sub {
processjobbranch($job, $branch, $html_file);
});
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 41/60] sg-report-job-history (nfc): Make $ri->{Hosts} a hash
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (39 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 40/60] sg-report-job-history: Refactor "ALL" handling Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 42/60] sg-report-job-history (nfc): Add new hostvarcols calculation Ian Jackson
` (18 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This will make it easier to cache this.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 8932458e..118926c6 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -173,11 +173,11 @@ END
while (my $f= $flightsq->fetchrow_hashref()) {
my $ri= report_run_getinfo($f);
- $ri->{Hosts} = [ ];
+ $ri->{Hosts} = { };
foreach my $hostvar (@hostvarcols) {
$hostq->execute($f->{flight}, $f->{job}, $hostvar);
my ($host) = $hostq->fetchrow_array();
- push @{ $ri->{Hosts} }, ($host // "-");
+ $ri->{Hosts}{$hostvar} = $host;
}
my %revisions;
@@ -255,7 +255,7 @@ END
print H "<td $r->{ColourAttr}><a href=\"$url\">".
$r->{Content}."</a></td>\n";
my $lastrev;
- my $hosts = join ", ", map { $_ // "-" } @{ $r->{Hosts} };
+ my $hosts = join ", ", map { $r->{Hosts}{$_} // "-" } @hostvarcols;
my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts);
print H "<td $hosts_colour>".encode_entities($hosts)."</td>\n";
$osstestverq->execute($r->{Flight}{flight});
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 42/60] sg-report-job-history (nfc): Add new hostvarcols calculation
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (40 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 41/60] sg-report-job-history (nfc): Make $ri->{Hosts} a hash Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 43/60] sg-report-job-history (nfc): Query hosts runvars in one go Ian Jackson
` (17 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
We are going to want to replace the existing @hostvarcols
calculation. Provide a new one based on $ri->{Hosts}.
Right now, die if they don't produce the same answers. This still
works, which shows that the calculation is right.
Tested-by: Ian Jackson <ian.jackson@eu.citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/sg-report-job-history b/sg-report-job-history
index 118926c6..fb2930a9 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -206,6 +206,13 @@ END
push @test_rows, $ri;
}
+ my %hostvarnames;
+ foreach my $ri (@test_rows) {
+ $hostvarnames{$_}=1 foreach keys %{ $ri->{Hosts} }
+ }
+ my @hostvarcols2 = sort keys %hostvarnames;
+ die unless "@hostvarcols" eq "@hostvarcols2";
+
my $osstestverq= db_prepare(<<END);
SELECT DISTINCT harness
FROM flights_harness_touched
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 43/60] sg-report-job-history (nfc): Query hosts runvars in one go
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (41 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 42/60] sg-report-job-history (nfc): Add new hostvarcols calculation Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 44/60] sg-report-job-history (nfc): Drop $hostsq query Ian Jackson
` (16 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Rather than doing one query for each entry in @hostvarcols, do one
query for all the relevant runvars. This is quite a bit faster and
will enable us to use the cache.
This is correct because @hostvarcols was the union of all the host
runvars, so this produces the same answers as the individual queries.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index fb2930a9..5d913b96 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -167,16 +167,19 @@ END
}
my $hostq= db_prepare(<<END);
- SELECT val FROM runvars WHERE flight=? AND job=? AND name=?
+ SELECT name, val
+ FROM runvars
+ WHERE flight=?
+ AND job=?
+ AND (name = 'host' OR name LIKE '%\\_host')
END
while (my $f= $flightsq->fetchrow_hashref()) {
my $ri= report_run_getinfo($f);
$ri->{Hosts} = { };
- foreach my $hostvar (@hostvarcols) {
- $hostq->execute($f->{flight}, $f->{job}, $hostvar);
- my ($host) = $hostq->fetchrow_array();
+ $hostq->execute($f->{flight}, $f->{job});
+ while (my ($hostvar,$host) = $hostq->fetchrow_array()) {
$ri->{Hosts}{$hostvar} = $host;
}
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 44/60] sg-report-job-history (nfc): Drop $hostsq query
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (42 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 43/60] sg-report-job-history (nfc): Query hosts runvars in one go Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 45/60] history reporting (nfc): Provide cache_set_task_print Ian Jackson
` (15 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
We have eliminated all the users of @hostvarcols before @hostvarcols2
is calculated from the row data.
The query which produces this is very slow and can't be cached. We
can abolish it now and just use the @hostvarcols2 calculation.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 22 +---------------------
1 file changed, 1 insertion(+), 21 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 5d913b96..5684ac02 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -147,25 +147,6 @@ END
END
$flightsq->execute(@params);
- my $hostsq= db_prepare(<<END);
- SELECT DISTINCT name
- FROM runvars
- JOIN flights USING (flight)
- WHERE ($cond)
- AND flight >= COALESCE(
- (
- SELECT flight $fromstuff
- LIMIT 1 OFFSET $offset
- ), 0)
- ORDER BY name;
-END
- $hostsq->execute(@params, @params); # sql text contains $cond twice
- my @hostvarcols;
- while (my ($hostvar) = $hostsq->fetchrow_array()) {
- next unless $hostvar =~ m/(^|_)host$/;
- push @hostvarcols, $hostvar;
- }
-
my $hostq= db_prepare(<<END);
SELECT name, val
FROM runvars
@@ -213,8 +194,7 @@ END
foreach my $ri (@test_rows) {
$hostvarnames{$_}=1 foreach keys %{ $ri->{Hosts} }
}
- my @hostvarcols2 = sort keys %hostvarnames;
- die unless "@hostvarcols" eq "@hostvarcols2";
+ my @hostvarcols = sort keys %hostvarnames;
my $osstestverq= db_prepare(<<END);
SELECT DISTINCT harness
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 45/60] history reporting (nfc): Provide cache_set_task_print
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (43 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 44/60] sg-report-job-history (nfc): Drop $hostsq query Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 46/60] sg-report-job-history: Introduce use of cache, for hosts query Ian Jackson
` (14 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This takes a string which gets added to the cache messages. This
will allow us to distinguish the output from different processes
when using parallel by fork.
Nothing sets this yet.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 5d2c83b7..bfbee28c 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -29,6 +29,7 @@ BEGIN {
@ISA = qw(Exporter);
@EXPORT = qw(
cache_set_key_cols
+ cache_set_task_print
cache_read_previous
cache_row_lookup_prep
cacheable_query
@@ -46,6 +47,7 @@ use POSIX;
our @key_cols;
+our $taskprint='';
our %cache;
our @previous;
@@ -91,6 +93,7 @@ our $rows_extra = 0;
sub cache_set_key_cols { @key_cols = @_; }
+sub cache_set_task_print ($) { ($taskprint)=@_; $taskprint =~ s/\s*$/ /; }
sub key ($) {
my ($jr) = @_;
@@ -162,7 +165,7 @@ sub cache_row_lookup_prep ($) {
$$jrr = $cacherow;
$rows_hit++;
} else {
- print ::DEBUG "CACHE MISS ",
+ print ::DEBUG "CACHE MISS $taskprint",
(join " ", map { $$jrr->{$_} } @key_cols), "\n";
}
}
@@ -209,7 +212,7 @@ sub cache_finish ($$) {
}
print ::DEBUG
- "CACHE $what read=$nprevious hits $rows_hit/$rows_today";
+ "CACHE $taskprint$what read=$nprevious hits $rows_hit/$rows_today";
for my $cachekey (sort keys %q_count) {
my $total = $q_count{$cachekey};
my $hits = $total - ($q_misses{$cachekey} // 0);
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 46/60] sg-report-job-history: Introduce use of cache, for hosts query
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (44 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 45/60] history reporting (nfc): Provide cache_set_task_print Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 47/60] history reporting (nfc): Break out url_quote Ian Jackson
` (13 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
* Set up the cache.
* Call the per-row setup hook.
* Cache the computation of $ri->{Hosts}.
* Call the per-row cache write hook.
* Finalise the cache.
Output is the same, but with cache information in the output html, and
faster.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 5684ac02..9f1216a6 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -121,6 +121,10 @@ sub processjobbranch ($$$) {
AND name LIKE '%buildjob'
END
+ cache_set_task_print("$j $branch_or_all");
+ cache_set_key_cols(qw(flight status)); # we process one job per process
+ cache_read_previous("$htmlout/$html_file");
+
print DEBUG "processjobbranch('$j',", ($bra ? "'$bra'" : 'undef'), ")\n";
my %rev_grid_col;
@@ -156,13 +160,17 @@ END
END
while (my $f= $flightsq->fetchrow_hashref()) {
+ cache_row_lookup_prep(\$f);
my $ri= report_run_getinfo($f);
- $ri->{Hosts} = { };
- $hostq->execute($f->{flight}, $f->{job});
- while (my ($hostvar,$host) = $hostq->fetchrow_array()) {
- $ri->{Hosts}{$hostvar} = $host;
- }
+ $ri->{Hosts} = cacheable_fn($f, 'h', sub {
+ my %hosts;
+ $hostq->execute($f->{flight}, $f->{job});
+ while (my ($hostvar,$host) = $hostq->fetchrow_array()) {
+ $hosts{$hostvar} = $host;
+ }
+ \%hosts;
+ });
my %revisions;
add_revisions(\%revisions, $f->{flight}, $f->{job}, '');
@@ -276,8 +284,10 @@ END
print H "</td>";
}
print H "</tr>\n";
+ cache_write_entry(\*H, $r->{Flight});
$alternate ^= 1;
}
+ cache_finish(\*H, "");
print H "</table></body></html>\n";
H->error and die $!;
close H or die $!;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 47/60] history reporting (nfc): Break out url_quote
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (45 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 46/60] sg-report-job-history: Introduce use of cache, for hosts query Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 48/60] history reporting (nfc): Break out url_unquote Ian Jackson
` (12 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index bfbee28c..f30ded5b 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -156,6 +156,12 @@ sub cache_read_previous ($) {
close H;
}
+sub url_quote ($) {
+ local ($_) = "$_[0]";
+ s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
+ $_;
+}
+
sub cache_row_lookup_prep ($) {
my ($jrr) = @_;
@@ -179,8 +185,7 @@ sub cache_write_entry ($$) {
next if $k =~ m/^\%/;
$_ = $h->{$k};
next unless defined;
- s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
- printf $fh " %s=%s", $k, $_;
+ printf $fh " %s=%s", $k, url_quote($_);
}
};
$whash->($jr);
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 48/60] history reporting (nfc): Break out url_unquote
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (46 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 47/60] history reporting (nfc): Break out url_quote Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 49/60] history reporting (nfc): Break out $url_ok_chars Ian Jackson
` (11 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index f30ded5b..f8a49955 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -147,8 +147,7 @@ sub cache_read_previous ($) {
}
s{^(\w+)=}{} or die "$orig -- $_ ";
my $k = $1;
- s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
- $ch->{$k} = $_;
+ $ch->{$k} = url_unquote($_);
}
push @previous, $jr;
$cache{key($jr)} = $jr;
@@ -156,6 +155,12 @@ sub cache_read_previous ($) {
close H;
}
+sub url_unquote ($) {
+ local ($_) = "$_[0]";
+ s{\%([0-9a-f]{2})}{ chr hex $1 }ge;
+ $_;
+}
+
sub url_quote ($) {
local ($_) = "$_[0]";
s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 49/60] history reporting (nfc): Break out $url_ok_chars
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (47 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 48/60] history reporting (nfc): Break out url_unquote Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 50/60] history reporting (nfc): url-quoting: quote = too Ian Jackson
` (10 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
We will want this in a moment.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index f8a49955..75ec75f3 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -45,6 +45,8 @@ BEGIN {
use POSIX;
+our $url_ok_chars = '-+=/~:;_.,\w';
+
our @key_cols;
our $taskprint='';
@@ -163,7 +165,7 @@ sub url_unquote ($) {
sub url_quote ($) {
local ($_) = "$_[0]";
- s{[^-+=/~:;_.,\w]}{ sprintf "%%%02x", ord $& }ge;
+ s{[^$url_ok_chars]}{ sprintf "%%%02x", ord $& }ge;
$_;
}
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 50/60] history reporting (nfc): url-quoting: quote = too
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (48 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 49/60] history reporting (nfc): Break out $url_ok_chars Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 51/60] history reporting (nfc): Quote keys too Ian Jackson
` (9 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
We are going to want to url-encode keys. If key contains =, we still
need to be able to tell where it ends, so it must be encoded.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 75ec75f3..20913d27 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -45,7 +45,7 @@ BEGIN {
use POSIX;
-our $url_ok_chars = '-+=/~:;_.,\w';
+our $url_ok_chars = '-+/~:;_.,\w';
our @key_cols;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 51/60] history reporting (nfc): Quote keys too
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (49 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 50/60] history reporting (nfc): url-quoting: quote = too Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 52/60] sg-report-job-history: Cache the per-flight revisions Ian Jackson
` (8 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Right now all the callers have keys which don't need quoting.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 20913d27..3e2ad0d3 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -147,8 +147,9 @@ sub cache_read_previous ($) {
$jr->{'%'.$&} = $ch;
next;
}
- s{^(\w+)=}{} or die "$orig -- $_ ";
- my $k = $1;
+ # Tolerate %, which will be de-quoted by url_unquote
+ s{^([\%$url_ok_chars]+)=}{}i or die "$orig -- $_ ";
+ my $k = url_unquote($1);
$ch->{$k} = url_unquote($_);
}
push @previous, $jr;
@@ -192,7 +193,7 @@ sub cache_write_entry ($$) {
next if $k =~ m/^\%/;
$_ = $h->{$k};
next unless defined;
- printf $fh " %s=%s", $k, url_quote($_);
+ printf $fh " %s=%s", url_quote($k), url_quote($_);
}
};
$whash->($jr);
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 52/60] sg-report-job-history: Cache the per-flight revisions
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (50 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 51/60] history reporting (nfc): Quote keys too Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 53/60] sg-report-job-history (nfc): Refactor osstestrevs code Ian Jackson
` (7 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This involves changing %revisions to %$revisions in the code which
uses the value.
No logical change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 26 +++++++++++++++-----------
1 file changed, 15 insertions(+), 11 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 9f1216a6..39ebe024 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -172,25 +172,29 @@ END
\%hosts;
});
- my %revisions;
- add_revisions(\%revisions, $f->{flight}, $f->{job}, '');
+ my $revisions = cacheable_fn($f, 'r', sub {
+ my %revisions;
+ add_revisions(\%revisions, $f->{flight}, $f->{job}, '');
- $buildsq->execute($f->{flight}, $f->{job});
- while (my $bld= $buildsq->fetchrow_hashref()) {
- my $bn= $bld->{name};
- $bn =~ s/buildjob$//;
- my ($oflight,$ojob) = flight_otherjob($f->{flight}, $bld->{val});
- add_revisions(\%revisions, $oflight, $ojob, " ($bn)");
- }
+ $buildsq->execute($f->{flight}, $f->{job});
+ while (my $bld= $buildsq->fetchrow_hashref()) {
+ my $bn= $bld->{name};
+ $bn =~ s/buildjob$//;
+ my ($oflight,$ojob) =
+ flight_otherjob($f->{flight}, $bld->{val});
+ add_revisions(\%revisions, $oflight, $ojob, " ($bn)");
+ }
+ \%revisions;
+ });
my @rev_cols;
- foreach my $revk (sort keys %revisions) {
+ foreach my $revk (sort keys %$revisions) {
my $col= $rev_grid_col{$revk};
if (!defined $col) {
$col= @rev_grid_cols;
push @rev_grid_cols, $revk;
$rev_grid_col{$revk}= $col;
}
- $rev_cols[$col]= $revisions{$revk};
+ $rev_cols[$col]= $$revisions{$revk};
}
$ri->{Revisions}= \@rev_cols;
$ri->{Flight}= $f;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 53/60] sg-report-job-history (nfc): Refactor osstestrevs code
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (51 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 52/60] sg-report-job-history: Cache the per-flight revisions Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 54/60] sg-report-job-history: Cache osstestrevs Ian Jackson
` (6 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Split this into (1) get the data from the db (2) process it into the
form we want.
This will make it easy to cache (1).
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 39ebe024..d5e37bcd 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -260,14 +260,17 @@ END
my $hosts = join ", ", map { $r->{Hosts}{$_} // "-" } @hostvarcols;
my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts);
print H "<td $hosts_colour>".encode_entities($hosts)."</td>\n";
+ my %osstestrevs;
$osstestverq->execute($r->{Flight}{flight});
+ while (my ($harness) = $osstestverq->fetchrow_array()) {
+ $osstestrevs{$harness}++;
+ }
my $osstestrevs = join ' ',
map {
- $_ = $_->{harness};
s/^([0-9a-f]{12})[0-9a-f]+\b/$1/;
"<kbd>".encode_entities($_)."</kbd>";
}
- @{ $osstestverq->fetchall_arrayref({}) };
+ sort keys %osstestrevs;
my $osstest_colour = report_altchangecolour(\$alt_osstest, $osstestrevs);
print H "<td $osstest_colour>$osstestrevs</td>\n";
foreach my $i (0..$#rev_grid_cols) {
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 54/60] sg-report-job-history: Cache osstestrevs
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (52 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 53/60] sg-report-job-history (nfc): Refactor osstestrevs code Ian Jackson
@ 2020-08-14 17:21 ` Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 55/60] history reporting: Break out minflight_by_time Ian Jackson
` (5 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:21 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
No logical change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index d5e37bcd..48dc2f57 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -260,17 +260,20 @@ END
my $hosts = join ", ", map { $r->{Hosts}{$_} // "-" } @hostvarcols;
my $hosts_colour = report_altchangecolour(\$alt_hosts, $hosts);
print H "<td $hosts_colour>".encode_entities($hosts)."</td>\n";
- my %osstestrevs;
- $osstestverq->execute($r->{Flight}{flight});
- while (my ($harness) = $osstestverq->fetchrow_array()) {
- $osstestrevs{$harness}++;
- }
+ my $osstestrevmap = cacheable_fn($r->{Flight}, 'o', sub {
+ my %osstestrevs;
+ $osstestverq->execute($r->{Flight}{flight});
+ while (my ($harness) = $osstestverq->fetchrow_array()) {
+ $osstestrevs{$harness}++;
+ }
+ \%osstestrevs;
+ });
my $osstestrevs = join ' ',
map {
s/^([0-9a-f]{12})[0-9a-f]+\b/$1/;
"<kbd>".encode_entities($_)."</kbd>";
}
- sort keys %osstestrevs;
+ sort keys %$osstestrevmap;
my $osstest_colour = report_altchangecolour(\$alt_osstest, $osstestrevs);
print H "<td $osstest_colour>$osstestrevs</td>\n";
foreach my $i (0..$#rev_grid_cols) {
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 55/60] history reporting: Break out minflight_by_time
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (53 preceding siblings ...)
2020-08-14 17:21 ` [OSSTEST PATCH 54/60] sg-report-job-history: Cache osstestrevs Ian Jackson
@ 2020-08-14 17:22 ` Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 56/60] sg-report-job-history (nfc): Abolish $fromstuff Ian Jackson
` (4 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:22 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Move this from sg-report-host-history so we can reuse it.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/HistoryReport.pm | 16 ++++++++++++++++
sg-report-host-history | 13 ++-----------
2 files changed, 18 insertions(+), 11 deletions(-)
diff --git a/Osstest/HistoryReport.pm b/Osstest/HistoryReport.pm
index 3e2ad0d3..b851ae45 100644
--- a/Osstest/HistoryReport.pm
+++ b/Osstest/HistoryReport.pm
@@ -37,6 +37,7 @@ BEGIN {
cache_write_entry
cache_finish
parallel_by_fork
+ minflight_by_time
);
%EXPORT_TAGS = ();
@@ -282,4 +283,19 @@ sub parallel_by_fork ($$$$) {
exit $worst;
}
+sub minflight_by_time ($) {
+ my ($timelimit) = @_;
+ my $minflightsq = db_prepare(<<END);
+ SELECT flight
+ FROM flights
+ WHERE started >= ?
+ ORDER BY flight ASC
+ LIMIT 1
+END
+ my $now = time // die $!;
+ $minflightsq->execute($now - $timelimit);
+ my ($minflight,) = $minflightsq->fetchrow_array();
+ $minflight //= 0;
+}
+
1;
diff --git a/sg-report-host-history b/sg-report-host-history
index 0a2e9904..efe652bb 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -95,19 +95,10 @@ sub computeflightsrange () {
END
$minflightsq->execute();
($minflight,) = $minflightsq->fetchrow_array();
+ $minflight //= 0;
} else {
- my $minflightsq = db_prepare(<<END);
- SELECT flight
- FROM flights
- WHERE started >= ?
- ORDER BY flight ASC
- LIMIT 1
-END
- my $now = time // die $!;
- $minflightsq->execute($now - $timelimit);
- ($minflight,) = $minflightsq->fetchrow_array();
+ $minflight = minflight_by_time($timelimit);
}
- $minflight //= 0;
$flightcond = "(flight > $minflight)";
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 56/60] sg-report-job-history (nfc): Abolish $fromstuff
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (54 preceding siblings ...)
2020-08-14 17:22 ` [OSSTEST PATCH 55/60] history reporting: Break out minflight_by_time Ian Jackson
@ 2020-08-14 17:22 ` Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 57/60] sg-report-job-history: Cache report_run_getinfo Ian Jackson
` (3 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:22 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
This used to be reused, but that is no longer the case. Do away with
it, for clarity and simplicity.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 48dc2f57..2cac123b 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -140,13 +140,11 @@ END
}
my $offset= $limit-1;
- my $fromstuff= <<END;
+ my $flightsq= db_prepare(<<END);
+ SELECT *
FROM jobs JOIN flights USING (flight)
WHERE ($cond)
ORDER BY flight DESC
-END
- my $flightsq= db_prepare(<<END);
- SELECT * $fromstuff
LIMIT $limit
END
$flightsq->execute(@params);
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 57/60] sg-report-job-history: Cache report_run_getinfo
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (55 preceding siblings ...)
2020-08-14 17:22 ` [OSSTEST PATCH 56/60] sg-report-job-history (nfc): Abolish $fromstuff Ian Jackson
@ 2020-08-14 17:22 ` Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 58/60] sg-report-host-history: " Ian Jackson
` (2 subsequent siblings)
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:22 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
No logical change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index 2cac123b..5d2a7e15 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -159,7 +159,7 @@ END
while (my $f= $flightsq->fetchrow_hashref()) {
cache_row_lookup_prep(\$f);
- my $ri= report_run_getinfo($f);
+ my $ri= cacheable_fn($f, 'rri', sub { report_run_getinfo($f) });
$ri->{Hosts} = cacheable_fn($f, 'h', sub {
my %hosts;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 58/60] sg-report-host-history: Cache report_run_getinfo
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (56 preceding siblings ...)
2020-08-14 17:22 ` [OSSTEST PATCH 57/60] sg-report-job-history: Cache report_run_getinfo Ian Jackson
@ 2020-08-14 17:22 ` Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 59/60] sg-report-job-history: Provide --time-limit Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 60/60] sg-report-job-history: Increase default limit Ian Jackson
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:22 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
No logical change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-host-history | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/sg-report-host-history b/sg-report-host-history
index efe652bb..a814ef37 100755
--- a/sg-report-host-history
+++ b/sg-report-host-history
@@ -266,7 +266,9 @@ END
print H "<td><a href=\"$url\">$jr->{job}</td>\n";
- my $ri = report_run_getinfo({ %$jr, %$ir });
+ my $ri = cacheable_fn($jr, 'rri', sub {
+ report_run_getinfo({ %$jr, %$ir })
+ });
print H "<td $ri->{ColourAttr}>$ri->{Content}</td>\n";
my $powers = cacheable_fn($jr, 'b', sub {
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 59/60] sg-report-job-history: Provide --time-limit
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (57 preceding siblings ...)
2020-08-14 17:22 ` [OSSTEST PATCH 58/60] sg-report-host-history: " Ian Jackson
@ 2020-08-14 17:22 ` Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 60/60] sg-report-job-history: Increase default limit Ian Jackson
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:22 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Calculate a minflight based on the time limit, and set the time limit
to a year ago by default.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/sg-report-job-history b/sg-report-job-history
index 5d2a7e15..e09c694f 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -30,6 +30,7 @@ use Osstest::HistoryReport;
our (@blessings,@branches);
our $limit= 100;
+our $timelimit= 86400 * (366 + 14);
our $htmlout = '.';
our $flight;
our $maxjobs=10;
@@ -44,6 +45,8 @@ while (@ARGV && $ARGV[0] =~ m/^-/) {
$$1= $2;
} elsif (m/^--(limit)\=([1-9]\d*)$/) {
$$1= $2;
+ } elsif (m/^--time-limit\=([1-9]\d*)$/) {
+ $timelimit= $1;
} elsif (restrictflight_arg($_)) {
# Handled by Executive
} elsif (m/^--html-dir=(.*)$/) {
@@ -67,6 +70,9 @@ csreadconfig();
our @jobs;
+our $minflight = minflight_by_time($timelimit);
+print DEBUG "MINFLIGHT $minflight\n";
+
sub findflight () {
my $branches= $dbh_tests->selectcol_arrayref(<<END, {}, $flight);
SELECT branch FROM flights WHERE flight=?
@@ -144,6 +150,7 @@ END
SELECT *
FROM jobs JOIN flights USING (flight)
WHERE ($cond)
+ AND flight > $minflight
ORDER BY flight DESC
LIMIT $limit
END
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [OSSTEST PATCH 60/60] sg-report-job-history: Increase default limit
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
` (58 preceding siblings ...)
2020-08-14 17:22 ` [OSSTEST PATCH 59/60] sg-report-job-history: Provide --time-limit Ian Jackson
@ 2020-08-14 17:22 ` Ian Jackson
59 siblings, 0 replies; 61+ messages in thread
From: Ian Jackson @ 2020-08-14 17:22 UTC (permalink / raw)
To: xen-devel; +Cc: Ian Jackson
Now this is a *lot* faster, we can print a lot more history.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
sg-report-job-history | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sg-report-job-history b/sg-report-job-history
index e09c694f..843dd776 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -29,7 +29,7 @@ use Osstest::Executive;
use Osstest::HistoryReport;
our (@blessings,@branches);
-our $limit= 100;
+our $limit= 2000;
our $timelimit= 86400 * (366 + 14);
our $htmlout = '.';
our $flight;
--
2.11.0
^ permalink raw reply related [flat|nested] 61+ messages in thread
end of thread, other threads:[~2020-08-14 17:27 UTC | newest]
Thread overview: 61+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-14 17:21 [OSSTEST PATCH 00/60] Speed up sg-report-job-history Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 01/60] history reporting (nfc): Do not key cache on hostname any more Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 02/60] history reporting (nfc): Add some test runes to the notes Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 03/60] history reporting: Delete two debug prints Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 04/60] history reporting (nfc): Refactor to generalise, cache_read_existing Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 05/60] history reporting (nfc): Make cache_write_entry into a top-level sub Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 06/60] history reporting (nfc): Rename jobquery to cacheable_query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 07/60] history reporting (nfc): Break out cache_row_lookup_prep Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 08/60] history reporting (nfc): Rename "existing" to "previous" Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 09/60] history reporting (nfc): Provide cacheable_fn Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 10/60] history reporting (nfc): Bind by name in cacheable_query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 11/60] history reporting (nfc): Rename $cachehits to $rows_hit Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 12/60] history reporting (nfc): Record query-specific stats Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 13/60] history reporting (nfc): Move cacheable_* further up the file Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 14/60] history reporting (nfc): Use cacheable_fn for power methods Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 15/60] history reporting (nfc): Introduce cache_set_key_cols Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 16/60] history reporting (nfc): Record more row-specific stats Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 17/60] history reporting: Cache stats reporting: Centralise and rework Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 18/60] history reporting (nfc): Remove now-obsolete stats variables Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 19/60] history reporting (nfc): Introduce empty HistoryReport module Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 20/60] history reporting (nfc): Move cache code into " Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 21/60] history reporting (nfc): Rename some module variables, remove "cache" Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 22/60] history reporting: Skip undefined keys Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 23/60] history reporting (nfc): Documentation for the new module Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 24/60] history reporting: Cache data limit now in History module Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 25/60] history reporting: Print debug for cache misses Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 26/60] history reporting: Improve an error message slightly Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 27/60] sg-report-host-history: Write cache entry for unfinished jobs Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 28/60] parallel by fork: Break out into HistoryReport Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 29/60] parallel by fork: Disconnect $dbh_tests as well as undefing it Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 30/60] parallel by fork: Fix a variable name to $task Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 31/60] sg-report-job-history: Prep for fork: Move $buildsq query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 32/60] sg-report-job-history: Prep for fork: Move $revisionsq query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 33/60] sg-report-job-history: Use fork-based parallelism Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 34/60] sg-report-job-history: Use one child per report Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 35/60] sg-report-job-history (nfc): Have main program decide HTML filename Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 36/60] sg-report-job-history: Always write HTML output Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 37/60] sg-report-job-history (nfc): Remove needless conditional Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 38/60] history reporting (nfc): Add another test rune to the notes Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 39/60] history reporting (nfc): Make cacheable_fn work without cache Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 40/60] sg-report-job-history: Refactor "ALL" handling Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 41/60] sg-report-job-history (nfc): Make $ri->{Hosts} a hash Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 42/60] sg-report-job-history (nfc): Add new hostvarcols calculation Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 43/60] sg-report-job-history (nfc): Query hosts runvars in one go Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 44/60] sg-report-job-history (nfc): Drop $hostsq query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 45/60] history reporting (nfc): Provide cache_set_task_print Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 46/60] sg-report-job-history: Introduce use of cache, for hosts query Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 47/60] history reporting (nfc): Break out url_quote Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 48/60] history reporting (nfc): Break out url_unquote Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 49/60] history reporting (nfc): Break out $url_ok_chars Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 50/60] history reporting (nfc): url-quoting: quote = too Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 51/60] history reporting (nfc): Quote keys too Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 52/60] sg-report-job-history: Cache the per-flight revisions Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 53/60] sg-report-job-history (nfc): Refactor osstestrevs code Ian Jackson
2020-08-14 17:21 ` [OSSTEST PATCH 54/60] sg-report-job-history: Cache osstestrevs Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 55/60] history reporting: Break out minflight_by_time Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 56/60] sg-report-job-history (nfc): Abolish $fromstuff Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 57/60] sg-report-job-history: Cache report_run_getinfo Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 58/60] sg-report-host-history: " Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 59/60] sg-report-job-history: Provide --time-limit Ian Jackson
2020-08-14 17:22 ` [OSSTEST PATCH 60/60] sg-report-job-history: Increase default limit Ian Jackson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).