[OSSTEST PATCH 00/13] Miscellaneous improvements

[OSSTEST PATCH 00/13] Miscellaneous improvements

[Ian Jackson]

These patches were accumulated during the Xen 4.16 freeze and as RM I
chose to defer them.  I am submitting them to osstest pretest now.

Ian Jackson (12):
  make-flight: do_examine_one: add firmware argument
  ts-memdisk-try-append: Reindent (nfc)
  ts-memdisk-try-append: Enable perl warnings
  ts-memdisk-try-append: More defensive case test
  examination: skip memdisk on non-BIOS hosts, run per-firmware on x86
  starvation: Only count "pass" and "fail" as done jobs
  sg-run-job: Do not set a step status to pass unless it's running
  TestSupport: sub broken: break out "$msg"
  TestSupport: sub broken: Add a newline
  TestSupport: sub broken: exit with status 0 for skip
  TestSupport: sub broken: document that it does not return.
  ts-memdisk-try-append: Fix for fact that broken() does not return

Roger Pau Monne (1):
  ts-xen-install: enable timestamp on guests logs

 Osstest/Executive.pm        |  3 ++-
 Osstest/TestSupport.pm      | 13 ++++++++++--
 make-flight                 | 24 +++++++++++++++++++--
 mfi-common                  |  1 +
 tcl/JobDB-Executive.tcl     |  1 +
 ts-hosts-allocate-Executive |  5 ++++-
 ts-memdisk-try-append       | 42 +++++++++++++++++++++++++------------
 ts-xen-install              |  2 +-
 8 files changed, 71 insertions(+), 20 deletions(-)

-- 
2.20.1

[OSSTEST PATCH 01/13] make-flight: do_examine_one: add firmware argument

[Ian Jackson]

Nothing sets this yet.

No functional change (checked with standalone-generate-dump-flight-runvars)

Signed-off-by: Ian Jackson <iwj@xenproject.org>
Release-Acked-by: Ian Jackson <iwj@xenproject.org>
---
 make-flight | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/make-flight b/make-flight
index ecbb195bc..acb19b113 100755
--- a/make-flight
+++ b/make-flight
@@ -687,20 +687,31 @@ do_pv_debian_tests () {
 }
 
 do_examine_one () {
+  local firmware="$1"
+
   case "$branch" in
     xen-unstable)		;; # only likely to regress on -unstable
     osstest)			;; # very likely to regress
     linux-*)			;; # often seems to regress
     *)			return	;; # stuff used for guests is irrelevant
   esac
+
+  local firmware_suffix
+  local firmware_hostflag
+  if [ "$firmware" ]; then
+    firmware_suffix=-$firmware
+    firmware_hostflag=,PropEq:Firmware:bios:$firmware
+  fi
+
   local freebsd_runvars
   # set_freebsd_runvars expects $arch to be set to the desired FreeBSD arch.
   local arch=$dom0arch
   # Pass true to not append any hostflags when creating the FreeBSD runvars.
   set_freebsd_runvars true
-  job_create_test test-$xenarch$kern-$dom0arch-examine \
+  job_create_test test-$xenarch$kern-$dom0arch-examine${firmware_suffix} \
                   host-examine-xen xl $xenarch $dom0arch \
-                  all_hostflags=$most_hostflags $freebsd_runvars
+                  all_hostflags=$most_hostflags$firmware_hostflag \
+                  $freebsd_runvars
 }
 
 test_matrix_do_one () {
-- 
2.20.1

[OSSTEST PATCH 02/13] ts-memdisk-try-append: Reindent (nfc)

[Ian Jackson]

Signed-off-by: Ian Jackson <iwj@xenproject.org>
Release-Acked-by: Ian Jackson <iwj@xenproject.org>
---
 ts-memdisk-try-append | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/ts-memdisk-try-append b/ts-memdisk-try-append
index 4b54eff01..a35751eb3 100755
--- a/ts-memdisk-try-append
+++ b/ts-memdisk-try-append
@@ -19,17 +19,17 @@
 set -xe -o posix
 
 arch=`perl -I. -e '
-                use Osstest;
-                use Osstest::TestSupport;
+    use Osstest;
+    use Osstest::TestSupport;
 
-                tsreadconfig();
+    tsreadconfig();
 
-		our $whhost = pop @ARGV; # arg parsing bodge
-		$whhost ||= "host";
-		our $ho = selecthost($whhost);
+    our $whhost = pop @ARGV; # arg parsing bodge
+    $whhost ||= "host";
+    our $ho = selecthost($whhost);
 
-                print $ho->{Arch} or die $!;
-              ' "$@"`
+    print $ho->{Arch} or die $!;
+    ' "$@"`
 
 case "$arch" in
 amd64)
-- 
2.20.1

[OSSTEST PATCH 03/13] ts-memdisk-try-append: Enable perl warnings

[Ian Jackson]

Signed-off-by: Ian Jackson <iwj@xenproject.org>
Release-Acked-by: Ian Jackson <iwj@xenproject.org>
---
 ts-memdisk-try-append | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-memdisk-try-append b/ts-memdisk-try-append
index a35751eb3..bfb36caf7 100755
--- a/ts-memdisk-try-append
+++ b/ts-memdisk-try-append
@@ -18,7 +18,7 @@
 
 set -xe -o posix
 
-arch=`perl -I. -e '
+arch=`perl -I. -we '
     use Osstest;
     use Osstest::TestSupport;
 
-- 
2.20.1

[OSSTEST PATCH 04/13] ts-memdisk-try-append: More defensive case test

[Ian Jackson]

Signed-off-by: Ian Jackson <iwj@xenproject.org>
Release-Acked-by: Ian Jackson <iwj@xenproject.org>
---
 ts-memdisk-try-append | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/ts-memdisk-try-append b/ts-memdisk-try-append
index bfb36caf7..d5c12866f 100755
--- a/ts-memdisk-try-append
+++ b/ts-memdisk-try-append
@@ -34,14 +34,14 @@ arch=`perl -I. -we '
 case "$arch" in
 amd64)
     ;;
-'')
-    echo >&2 'Job arch retrieval failed?'
-    exit 1
-    ;;
-*)
+[a-z]*)
     echo "Arch $arch not supported for memdisk tests"
     exit 0
     ;;
+*)
+    echo >&2 'Job arch retrieval failed?'
+    exit 1
+    ;;
 esac
 
 if ./ts-freebsd-host-install --test-boot --record-append $@; then
-- 
2.20.1

[OSSTEST PATCH 05/13] examination: skip memdisk on non-BIOS hosts, run per-firmware on x86

[Ian Jackson]

This is a combination of two changes:

ts-memdisk-try-append: skip memdisk test on non-BIOS hosts

make-flight: examine: Insist on -bios and -uefi tests on x86

  This will let us skip ts-memdisk-try-append on non-bios platforms
  without risking regression.  It will also definitely spot
  regressions which will occur on any uefi host.

  standalone-generate-dump-flight-runvars reveals the changes are as
  follows:

  New jobs
     test-amd64-i386-examine-bios     test-amd64-i386-examine-uefi
     test-amd64-amd64-examine-bios    test-amd64-amd64-examine-uefi
  added everywhere that has the corresponding plain job, namely
     osstest
     linux-*
     xen-unstable

  These jobs are just like the plain jobs, except that one of
     ,PropEq:Firmware:bios:bios
     ,PropEq:Firmware:bios:uefi
  has been added to the end of all_hostflags.

Signed-off-by: Ian Jackson <iwj@xenproject.org>
Release-Acked-by: Ian Jackson <iwj@xenproject.org>
---
 make-flight           |  9 +++++++++
 mfi-common            |  1 +
 ts-memdisk-try-append | 18 ++++++++++++++++++
 3 files changed, 28 insertions(+)

diff --git a/make-flight b/make-flight
index acb19b113..d7eeb4ebb 100755
--- a/make-flight
+++ b/make-flight
@@ -736,6 +736,15 @@ test_matrix_do_one () {
   do_credit12_tests
 
   do_examine_one
+  case "$dom0arch" in
+    i386|amd64)
+      # Test on BIOS, which makes sure we test one where FreeBSD should
+      # work.  See ts-memdisk-try-append, and set_freebsd_runvars in
+      # mfi-common.
+      do_examine_one bios
+      do_examine_one uefi
+      ;;
+  esac
 
   # No further arm tests at the moment
   if [ $dom0arch = armhf -o $dom0arch = arm64 ]; then
diff --git a/mfi-common b/mfi-common
index 771843f05..59e712f4e 100644
--- a/mfi-common
+++ b/mfi-common
@@ -163,6 +163,7 @@ set_freebsd_runvars () {
     if [ x$no_hostflags != xtrue ]; then
         # osstest doesn't yet know how to install FreeBSD on UEFI hosts, so
         # limit the usable hardware to boxes that boot from BIOS.
+        # See also in ts-memdisk-try-append.
         freebsd_runvars="all_hostflags,=PropEq:Firmware:bios:bios"
     fi
 
diff --git a/ts-memdisk-try-append b/ts-memdisk-try-append
index d5c12866f..18c4aaa5c 100755
--- a/ts-memdisk-try-append
+++ b/ts-memdisk-try-append
@@ -28,6 +28,20 @@ arch=`perl -I. -we '
     $whhost ||= "host";
     our $ho = selecthost($whhost);
 
+    if (get_target_property($ho,"firmware") ne "bios") {
+        # osstest does not yet know how to install FreeBSD on UEFI hosts
+	# See also in set_freebsd_runvars in mfi_common
+
+	# Skipping this test on non-BIOS does not lead to pushes
+	# where this test is broken, because make-flght test_matrix_do_one
+	# arranges to run this test on BIOS too.
+
+        broken("skipping memdisk because firmware is not bios",
+	       "skip");
+	print "SKIP\n" or die $!;
+	exit 0;
+    }
+
     print $ho->{Arch} or die $!;
     ' "$@"`
 
@@ -38,6 +52,10 @@ amd64)
     echo "Arch $arch not supported for memdisk tests"
     exit 0
     ;;
+SKIP)
+    echo "Step skipped."
+    exit 0
+    ;;
 *)
     echo >&2 'Job arch retrieval failed?'
     exit 1
-- 
2.20.1

[OSSTEST PATCH 06/13] starvation: Only count "pass" and "fail" as done jobs

[Ian Jackson]

The main effect is to treat "broken" and "blocked" jobs as not part of
the flight for starvation calculations.

This is correct for commissioning flights and OK for other flights.

Signed-off-by: Ian Jackson <iwj@xenproject.org>
Release-Acked-by: Ian Jackson <iwj@xenproject.org>
---
 Osstest/Executive.pm        | 3 ++-
 ts-hosts-allocate-Executive | 5 ++++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index d95d848dc..9d7c71369 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -1148,7 +1148,8 @@ END
 #        to allow for slow jobs, compared to other jobs
 #    W   number of jobs waiting - strictly, jobs in states
 #          preparing queued running
-#    D   number of jobs done - strictly, other states
+#    D   number of jobs done - strictly, jobs in states
+#          pass fail
 #  tuning parameters:
 #    Xt  X when D=9 W=1, ie cancel one job out of ten
 #    Xh  X when D=1 W=1, ie cancel one job out of two
diff --git a/ts-hosts-allocate-Executive b/ts-hosts-allocate-Executive
index 4dfcd0cd1..19047d49e 100755
--- a/ts-hosts-allocate-Executive
+++ b/ts-hosts-allocate-Executive
@@ -846,11 +846,14 @@ sub starving ($$$) {
 	    # is indistinguishable from a flight which is at the head
 	    # of the queue for a small set of resources.
 	    return (0, "job $j status $st, don't give up just yet");
-	} else {
+	} elsif ($st eq 'pass' ||
+		 $st eq 'fail') {
 	    $d++;
 	    return (0, "job $j status $st but no step finished time!")
 		unless defined $fin;
 	    $maxfin = $fin if $fin > $maxfin;
+	} else {
+	    # disregard - neither W or D
 	}
     }
     # we quit if the total time from the start of the flight
-- 
2.20.1

[OSSTEST PATCH 07/13] sg-run-job: Do not set a step status to pass unless it's running

[Ian Jackson]

This avoids skip etc. being treated as pass.

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 tcl/JobDB-Executive.tcl | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tcl/JobDB-Executive.tcl b/tcl/JobDB-Executive.tcl
index 38248823e..8e0062f59 100644
--- a/tcl/JobDB-Executive.tcl
+++ b/tcl/JobDB-Executive.tcl
@@ -326,6 +326,7 @@ proc step-set-status {flight job stepno st} {
                AND status<>'broken'
                AND status<>'starved'
                AND status<>'fail'
+               AND NOT ('$st' = 'pass' AND status <> 'running')
         "
         set pause 0
         db-execute-array stopinfo "
-- 
2.20.1

[OSSTEST PATCH 08/13] TestSupport: sub broken: break out "$msg"

[Ian Jackson]

No functional change.

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 Osstest/TestSupport.pm | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index a0ca69437..b17bf08a6 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -281,8 +281,10 @@ sub broken ($;$) {
                AND (status='queued' OR status='running' OR status='preparing')
 END
     });
-    die uc($newst).": $m; ". ($affected>0 ? "marked $flight.$job $newst"
-                         : "($flight.$job not marked $newst)");
+    my $msg = uc($newst).": $m; ".
+      ($affected>0 ? "marked $flight.$job $newst"
+                   : "($flight.$job not marked $newst)");
+    die $msg;
 }
 
 sub complete_testid ($) {
-- 
2.20.1

[OSSTEST PATCH 09/13] TestSupport: sub broken: Add a newline

[Ian Jackson]

This suppresses printing of the the line number, which is not useful.

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 Osstest/TestSupport.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index b17bf08a6..ff890f96c 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -283,7 +283,8 @@ END
     });
     my $msg = uc($newst).": $m; ".
       ($affected>0 ? "marked $flight.$job $newst"
-                   : "($flight.$job not marked $newst)");
+                   : "($flight.$job not marked $newst)").
+		     "\n";
     die $msg;
 }
 
-- 
2.20.1

[OSSTEST PATCH 10/13] TestSupport: sub broken: exit with status 0 for skip

[Ian Jackson]

Otherwise, whatever calls us will set the step to fail (or something).

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 Osstest/TestSupport.pm | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index ff890f96c..ebe436275 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -285,7 +285,12 @@ END
       ($affected>0 ? "marked $flight.$job $newst"
                    : "($flight.$job not marked $newst)").
 		     "\n";
-    die $msg;
+    if ($newst =~ m{^(?:skip)$}) {
+	print STDERR $msg;
+	exit 0;
+    } else {
+	die $msg;
+    }
 }
 
 sub complete_testid ($) {
-- 
2.20.1

[OSSTEST PATCH 11/13] TestSupport: sub broken: document that it does not return.

[Ian Jackson]

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 Osstest/TestSupport.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index ebe436275..8103ea1d3 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -261,6 +261,7 @@ sub fail ($) {
 sub broken ($;$) {
     my ($m, $newst) = @_;
     # must be run outside transaction
+    # dies or exits
     my $affected;
     $newst= 'broken' unless defined $newst;
     eval {
-- 
2.20.1

[OSSTEST PATCH 12/13] ts-memdisk-try-append: Fix for fact that broken() does not return

[Ian Jackson]

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 ts-memdisk-try-append | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/ts-memdisk-try-append b/ts-memdisk-try-append
index 18c4aaa5c..e34415b05 100755
--- a/ts-memdisk-try-append
+++ b/ts-memdisk-try-append
@@ -38,8 +38,6 @@ arch=`perl -I. -we '
 
         broken("skipping memdisk because firmware is not bios",
 	       "skip");
-	print "SKIP\n" or die $!;
-	exit 0;
     }
 
     print $ho->{Arch} or die $!;
@@ -52,7 +50,7 @@ amd64)
     echo "Arch $arch not supported for memdisk tests"
     exit 0
     ;;
-SKIP)
+'')
     echo "Step skipped."
     exit 0
     ;;
-- 
2.20.1

[OSSTEST PATCH 13/13] ts-xen-install: enable timestamp on guests logs

[Ian Jackson]

From: Roger Pau Monne <roger.pau@citrix.com>

Enable the timestamp feature of xenconsoled so guests logs have a time
reference. Can be helpful when debugging boot related slowness.

This requires using the XENCONSOLED_ARGS option and setting both the
log and the timestamp options. Note that setting XENCONSOLED_TRACE
will override any options in XENCONSOLED_ARGS, so they can not be used
in conjunction.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <iwj@xenproject.org>
---
 ts-xen-install | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-xen-install b/ts-xen-install
index 47865eb60..bf55d4e5c 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -140,7 +140,7 @@ sub adjustconfig () {
 
     my @commons_config =
         (
-	   "XENCONSOLED_TRACE" => "guest",
+	   "XENCONSOLED_ARGS" => '"--log=guest --timestamp=all"',
 	);
 
     my $xenstored = target_var($ho, 'xenstored');
-- 
2.20.1