[OSSTEST PATCH 1/4] cs-bisection-step: Change some url. references to job.

[OSSTEST PATCH 1/4] cs-bisection-step: Change some url. references to job.

[Ian Jackson]

This query is going to turn into two variants, one of which doesn't
have the url join.

In the output, prefer to refer to fields from job.  They are
constrained to be equal (and they are not null) so this has no
functional change.

Also swap the conditions over for clarity.

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cs-bisection-step | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/cs-bisection-step b/cs-bisection-step
index 16227234..f9ef1558 100755
--- a/cs-bisection-step
+++ b/cs-bisection-step
@@ -231,8 +231,8 @@ END
 
         SELECT url.val AS uval,
 	       rev.val AS rval,
-	       url.job AS job,
-      ${\ other_revision_job_suffix('url.job','url.use',' ') } AS othrev,
+	       rev.job AS job,
+      ${\ other_revision_job_suffix('rev.job','rev.use',' ') } AS othrev,
 	       url.name AS longname
 
 	    FROM tmp_build_info AS rev
@@ -241,8 +241,8 @@ END
            WHERE (rev.name LIKE E'built\\_revision\\_%' OR
                   rev.name LIKE E'revision\\_%')
   	     AND  url.name LIKE E'tree\\_%'
-	     AND  rev.use = url.use
-	     AND  rev.job = url.job
+	     AND  url.use = rev.use
+	     AND  url.job = rev.job
 	     AND (rev.name = 'built_revision_' || substr(url.name,6) OR
                   rev.name = 'revision_'       || substr(url.name,6))
 
-- 
2.20.1

[OSSTEST PATCH 2/4] cs-bisection-step: flight_rmap Disassemble the revisions query

[Ian Jackson]

Break out various pieces that we are going to need to reuse for the
other version of this query (which won't have the url join).

Also, rather than retrieving the `tree_<tree>' runvar and calculating
the tree name from that, use the `[built_]revision_<tree>' runvar from
rev.

No overall functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cs-bisection-step | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)

diff --git a/cs-bisection-step b/cs-bisection-step
index f9ef1558..b36bac05 100755
--- a/cs-bisection-step
+++ b/cs-bisection-step
@@ -227,19 +227,30 @@ END
 	       AND flight = ?
 END
 
-    my $sth= db_prepare(<<END);
-
-        SELECT url.val AS uval,
+    my $qtxt_common_results = <<END;
+               rev.name AS revname,
 	       rev.val AS rval,
 	       rev.job AS job,
       ${\ other_revision_job_suffix('rev.job','rev.use',' ') } AS othrev,
-	       url.name AS longname
-
+END
+    my $qtxt_common_tables = <<END;
 	    FROM tmp_build_info AS rev
+END
+    my $qtxt_common_rev_condition = <<END;
+                 (rev.name LIKE E'built\\_revision\\_%' OR
+                  rev.name LIKE E'revision\\_%')
+END
+
+    my $sth= db_prepare(<<END);
+        SELECT
+$qtxt_common_results
+	       url.val AS uval
+
+$qtxt_common_tables
       CROSS JOIN tmp_build_info AS url
 
-           WHERE (rev.name LIKE E'built\\_revision\\_%' OR
-                  rev.name LIKE E'revision\\_%')
+           WHERE
+$qtxt_common_rev_condition
   	     AND  url.name LIKE E'tree\\_%'
 	     AND  url.use = rev.use
 	     AND  url.job = rev.job
@@ -247,7 +258,6 @@ END
                   rev.name = 'revision_'       || substr(url.name,6))
 
 	   ORDER by url.val;
-	  
 END
     $sth->execute();
     my $row;
@@ -255,7 +265,7 @@ END
     my (@ttreenames, @ttreeurls, @trevisions);
     while ($row= $sth->fetchrow_hashref()) {
         next if $row->{othrev} eq 'DISCARD';
-        $row->{longname} =~ m/^tree_/ or die "$row->{longname} ?";
+        $row->{revname} =~ m/^(?:built_)?revision_/ or die "$row->{revname} ?";
         my $name= $'; #'
         print DEBUG " $flight.$row->{job} uval=$row->{uval}".
             " rval=$row->{rval} name=$name othrev=\`$row->{othrev}'\n";
-- 
2.20.1

[OSSTEST PATCH 3/4] cs-bisection-step: Provide no-urls variant of the main query

[Ian Jackson]

This variant just returns '' for urls.  Unlike the with-urls variant,
it does not ignore trees without urls.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cs-bisection-step | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/cs-bisection-step b/cs-bisection-step
index b36bac05..2f75313e 100755
--- a/cs-bisection-step
+++ b/cs-bisection-step
@@ -181,6 +181,7 @@ END
 
 sub flight_rmap ($) {
     my ($flight) = @_;
+    my $need_urls = 1;
 
     $dbh_tests->do(<<END, {});
           CREATE TEMP TABLE tmp_build_info (
@@ -241,7 +242,19 @@ END
                   rev.name LIKE E'revision\\_%')
 END
 
-    my $sth= db_prepare(<<END);
+    my $sth= db_prepare(!$need_urls ? <<END_NOURLS : <<END_URLS);
+        SELECT
+$qtxt_common_results
+	       '' AS uval
+
+$qtxt_common_tables
+
+           WHERE
+$qtxt_common_rev_condition
+
+	   ORDER by rev.name;
+
+END_NOURLS
         SELECT
 $qtxt_common_results
 	       url.val AS uval
@@ -258,7 +271,8 @@ $qtxt_common_rev_condition
                   rev.name = 'revision_'       || substr(url.name,6))
 
 	   ORDER by url.val;
-END
+END_URLS
+
     $sth->execute();
     my $row;
     my $mixed=0;
-- 
2.20.1

[OSSTEST PATCH 4/4] cs-bisection-step: Do not insist on urls in main history search

[Ian Jackson]

If a Xen build fails, but we are trying to bisect something involving
libvirt, the libvirt job does not really run.  It does not populate
the tree_<tree> values for its git submodules - that would involve
actually booking out a host and cloning it.

The effect is that xen build failures which occur somewhere in the
range of a libvirt bisection cause total failure (actually, looping
bisection) rather than treating the build failure as `blocked'.

Fix this by tolerating trees with missing urls - but only during the
main history search.  Most of the rest of the time we need the urls,
mainly because we are going to feed them to adhoc-revtuple-generator
and mention them in our output.

It could now happen that trees appear in the main bisection history
which weren't in our bases.  These end up being ignored by
flight_rtuple.  This is not ideal but it will do for now.  In any case
that aspect is no worse than before.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cs-bisection-step | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/cs-bisection-step b/cs-bisection-step
index 2f75313e..478baa5b 100755
--- a/cs-bisection-step
+++ b/cs-bisection-step
@@ -179,9 +179,8 @@ sub relevant_hosts ($) {
 END
 }
 
-sub flight_rmap ($) {
-    my ($flight) = @_;
-    my $need_urls = 1;
+sub flight_rmap ($$) {
+    my ($flight, $need_urls) = @_;
 
     $dbh_tests->do(<<END, {});
           CREATE TEMP TABLE tmp_build_info (
@@ -322,7 +321,7 @@ END
 
 sub flight_rtuple ($) {
     my ($flight) = @_;
-    my $map= flight_rmap($flight);
+    my $map= flight_rmap($flight,0);
     return () if !defined $map;
     my @revisions= ();
     die unless @treeinfos;
@@ -396,12 +395,12 @@ END
 
         print STDERR " [$failhosts] ";
 
-	my $failrmap = flight_rmap($tryfail->{flight});
+	my $failrmap = flight_rmap($tryfail->{flight},1);
 	next unless $failrmap;
 
 	my $checkbasisvcs = sub {
 	    my ($trybasisflight) = @_;
-	    my $basisrmap = flight_rmap($trybasisflight);
+	    my $basisrmap = flight_rmap($trybasisflight,1);
 	    my @bad;
 #print STDERR Dumper($failrmap, $basisrmap);
 	    foreach my $tree (keys %$failrmap) {
@@ -480,8 +479,8 @@ END
 our (%nodes, @latest_rtuple, @basispass_rtuple);
 
 sub digraph_whole () {
-    my $latest_rmap= flight_rmap($latest_flight);
-    my $basispass_rmap= flight_rmap($basispass_flight);
+    my $latest_rmap= flight_rmap($latest_flight,1);
+    my $basispass_rmap= flight_rmap($basispass_flight,1);
     if (!defined $basispass_rmap) {
 	die "Basis pass $basispass_flight rmap indeterminate/wrong\n";
     }
-- 
2.20.1