[OSSTEST PATCH 1/4] host allocation: Support -B<blessing> option

[OSSTEST PATCH 1/4] host allocation: Support -B<blessing> option

[Ian Jackson]

The flight's intended affects the hostflags required, the duration
searches, and other decisions.  It is particularly useful for
debugging, where it can be desirable to try replaying a production
job's allocation with a "play" job.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-hosts-allocate-Executive |   17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 706ab6d..7c32847 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -31,12 +31,15 @@ open DEBUG, ">/dev/null" or die $!;
 
 our $compressdebug=1;
 our $fake; # never allocate
+our $blessing;
 
 while (@ARGV and $ARGV[0] =~ m/^-/) {
     $_= shift @ARGV;
     last if m/^--$/;
     while (m/^-./) {
-        if (s/^-U/-/) {
+        if (s/^-B(.*)/-/) {
+	    $blessing=$1; # default is flight's intended blessing
+	} elsif (s/^-U/-/) {
 	    $compressdebug=0;
 	} elsif (s/^-n/-/) {
 	    $fake=1;
@@ -84,7 +87,9 @@ sub setup () {
         SELECT * FROM flights
          WHERE flight = ?
 END
-    logm("flight $flight intended $fi->{intended} branch $fi->{branch}");
+
+    $blessing //= $fi->{intended};
+    logm("flight $flight intended $blessing branch $fi->{branch}");
 
     $jobinfo= $dbh_tests->selectrow_hashref(<<END, {}, $flight, $job);
         SELECT * FROM jobs
@@ -121,7 +126,7 @@ END
 		      AND hostflag LIKE 'equiv-%'
 END
 
-    $duration_estimator= duration_estimator($fi->{branch}, $fi->{intended},
+    $duration_estimator= duration_estimator($fi->{branch}, $blessing,
                                             sub { print DEBUG "@_\n"; });
 
     $resprop_q= $dbh_tests->prepare(<<END);
@@ -284,14 +289,14 @@ sub hid_find_possibilities ($$) {
 
     my $use= $hid->{OverrideUse} || $r{ $hid->{Ident} };
 
-    print DEBUG "HID $hid->{Ident} INTENDED $fi->{intended}\n";
+    print DEBUG "HID $hid->{Ident} INTENDED $blessing\n";
     my $findhostsq = <<END;
 	    SELECT *
 	      FROM resources JOIN hostflags
 		ON (restype='host' AND shareix=0 AND
 		    resname=hostname AND hostflag=?)
 END
-    my @findhostsqvars = ("blessed-$fi->{intended}");
+    my @findhostsqvars = ("blessed-$blessing");
 
     if (defined $use) {
         print DEBUG "HID $hid->{Ident} USE $use\n";
@@ -431,7 +436,7 @@ END
         find_recent_duration($dbg,$hid,$candrow);
 
 	if ($candrow->{restype} eq 'host') {
-	    $equivstatusq->execute($job,$fi->{intended},$fi->{branch},
+	    $equivstatusq->execute($job,$blessing,$fi->{branch},
 				   $hid->{Ident},$candrow->{resname});
 	    my $esrow = $equivstatusq->fetchrow_hashref();
 	    $candrow->{EquivMostRecentStatus} = $esrow->{status};
-- 
1.7.10.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 2/4] host allocation: Pass a debug function in find_recent_duration

[Ian Jackson]

We provide only a commented-out debug print.  This produces more
copius output than is desirable even to a compressed debug log.

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-hosts-allocate-Executive |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 7c32847..9d8fbfd 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -487,7 +487,10 @@ sub find_recent_duration ($$) {
     ($candrow->{Duration},
      $candrow->{MostRecentStarted},
      $candrow->{MostRecentStatus}) =
-        $duration_estimator->($job, $hid->{Ident}, $candrow->{resname});
+        $duration_estimator->($job, $hid->{Ident}, $candrow->{resname},
+                              sub {
+#                                  print DEBUG "$dbg DUR-EST @_\n";
+                              });
 }
 
 
-- 
1.7.10.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 3/4] host allocation: Dedupe "other hosts" flights in duration estimation

[Ian Jackson]

The old query would return one row for each step in each relevant
flight.  But we are really only interested in the flight.

Group by the flight and sort on max(finished).

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Executive.pm |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index 69f0319..c82a9c7 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -1051,14 +1051,16 @@ sub duration_estimator ($$;$$) {
 END
 
     my $duration_anyref_q= $dbh_tests->prepare(<<END);
-            SELECT f.flight AS flight
+            SELECT f.flight AS flight,
+                   max(s.finished) AS max_finished
 		      FROM steps s JOIN flights f
 		        ON s.flight=f.flight
 		     WHERE s.job=? AND f.blessing=? AND f.branch=?
                        AND s.finished IS NOT NULL
                        AND f.started IS NOT NULL
                        AND f.started >= ?
-                     ORDER BY s.finished DESC
+                     GROUP BY f.flight
+                     ORDER BY max_finished DESC
 END
     # s J J J # fix perl-mode
 
-- 
1.7.10.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 4/4] host allocation: Fix duration estimate to not include host allocation

[Ian Jackson]

In 720f08cb9052 "Executive: Previous duration estimator: use overall
time, not sum of steps" we introduced a bug: the condition to exclude
the host allocation time is now not effective if there are any steps
before host allocation.  Usually there are.

This means that the host allocation duration estimator has been
including the host allocation time from previous jobs, which is quite
wrong.

Fix this by subtracting the maximum duration of any host allocation
step.  Hopefully there will only be one.

If any host allocation runs concurrently with other steps (including
other host allocations) then this will start to give wrong answers.
But there are other reasons why we wouldn't want to do that.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Executive.pm |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index c82a9c7..137eb44 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -1071,10 +1071,15 @@ END
               FROM steps
              WHERE flight=? AND job=?
         )
+        SELECT (
             SELECT max(finished)-min(started)
-                AS duration
               FROM tsteps
-             WHERE step != 'ts-hosts-allocate'
+          ) - (
+            SELECT sum(finished-started)
+              FROM tsteps
+             WHERE step = 'ts-hosts-allocate'
+          )
+                AS duration
 END
 
     if ($will_uptoincl_testid) {
-- 
1.7.10.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel