[OSSTEST PATCH 0/5] Only run ts-memdisk-try-append on BIOS hosts

[OSSTEST PATCH 0/5] Only run ts-memdisk-try-append on BIOS hosts

[Ian Jackson]

There is an infelicity in handling of FreeBSD's "memdisk" test.

The use of this step is as part of host examination.  There, it may
fail on hosts which cannot cope with the FreeBSD version in use.  That
is OK (from a general osstest/CI POV) because each host has its own
job name and therefore its own regression tracking.  So this test
failing on those hosts is not troublesome.

But as part of the tests for osstest itself, and Xen and Linux, we
also check that host examination still works.  This is appropriate
because host examination also checks that we can retrieve accurate
serial logs, etc.

To this end many general flights have test-*-examine jobs.

Like any job, these are peripatetic.  The x86 ones might run on BIOS
hosts, or UEFI hosts.  But our support for FreeBSD does not cope with
UEFI yet.  The result is that whether this test passes or fails
depends on the host, but this is not a heisenbug which should be fixed
right away.

Sort this out as follows:

* Have the examine test only run the FreeBSD step when the host
  firmware is BIOS.  On non-FreeBSD hosts the test step will go from
  "fail" to "skip", which is appropriate.

* To make sure we catch actual regressions in this step, make a copy
  of the examine job which is always run on a BIOS host.

* While we are here, explicitly check that the examine job works on
  x86 UEFI too.


4.16 release implications:

There are two sets of changes here:

I. flight composition:

The runvar dump shows that this only adds new jobs.  I think these new
tests are good, but of course they might be broken somehow.  If they
are broken, they are new jobs which won't count as regressions, and if
they flap we can force push while we disable them again.

II. behavioural change to ts-memdisk-try-append

This change is *intended* to make the test "skip" in cases where it is
currently "fail".  I have done an ad-hoc check that this seems to DTRT
but it might be bad somehow.

However, this test is currently actually-failing on the xen-unstable
branch.  And it could be force pushed if necessary.

III. Benefits

The benefit of all this is to improve the testing for 4.16.  In
particular, the examine test ought now to run properly, and no longer
cause blockages.

I think at this stage of the release this change to osstest ought to
have a release-ack but I think I have convinced myself it's OK, so:
Release-Acked-By: Ian Jackson <iwj@xenproject.org>


Ian Jackson (5):
  make-flight: do_examine_one: add firmware argument
  ts-memdisk-try-append: Reindent (nfc)
  ts-memdisk-try-append: Enable perl warnings
  ts-memdisk-try-append: More defensive case test
  examination: skip memdisk on non-BIOS hosts, run on some BIOS hosts

 make-flight           | 23 ++++++++++++++++++++--
 mfi-common            |  1 +
 ts-memdisk-try-append | 45 +++++++++++++++++++++++++++++--------------
 3 files changed, 53 insertions(+), 16 deletions(-)

-- 
2.20.1

[OSSTEST PATCH 1/5] make-flight: do_examine_one: add firmware argument

[Ian Jackson]

Nothing sets this yet.

No functional change (checked with standalone-generate-dump-flight-runvars)

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 make-flight | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/make-flight b/make-flight
index ecbb195bc..acb19b113 100755
--- a/make-flight
+++ b/make-flight
@@ -687,20 +687,31 @@ do_pv_debian_tests () {
 }
 
 do_examine_one () {
+  local firmware="$1"
+
   case "$branch" in
     xen-unstable)		;; # only likely to regress on -unstable
     osstest)			;; # very likely to regress
     linux-*)			;; # often seems to regress
     *)			return	;; # stuff used for guests is irrelevant
   esac
+
+  local firmware_suffix
+  local firmware_hostflag
+  if [ "$firmware" ]; then
+    firmware_suffix=-$firmware
+    firmware_hostflag=,PropEq:Firmware:bios:$firmware
+  fi
+
   local freebsd_runvars
   # set_freebsd_runvars expects $arch to be set to the desired FreeBSD arch.
   local arch=$dom0arch
   # Pass true to not append any hostflags when creating the FreeBSD runvars.
   set_freebsd_runvars true
-  job_create_test test-$xenarch$kern-$dom0arch-examine \
+  job_create_test test-$xenarch$kern-$dom0arch-examine${firmware_suffix} \
                   host-examine-xen xl $xenarch $dom0arch \
-                  all_hostflags=$most_hostflags $freebsd_runvars
+                  all_hostflags=$most_hostflags$firmware_hostflag \
+                  $freebsd_runvars
 }
 
 test_matrix_do_one () {
-- 
2.20.1

[OSSTEST PATCH 2/5] ts-memdisk-try-append: Reindent (nfc)

[Ian Jackson]

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 ts-memdisk-try-append | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/ts-memdisk-try-append b/ts-memdisk-try-append
index 4b54eff01..a35751eb3 100755
--- a/ts-memdisk-try-append
+++ b/ts-memdisk-try-append
@@ -19,17 +19,17 @@
 set -xe -o posix
 
 arch=`perl -I. -e '
-                use Osstest;
-                use Osstest::TestSupport;
+    use Osstest;
+    use Osstest::TestSupport;
 
-                tsreadconfig();
+    tsreadconfig();
 
-		our $whhost = pop @ARGV; # arg parsing bodge
-		$whhost ||= "host";
-		our $ho = selecthost($whhost);
+    our $whhost = pop @ARGV; # arg parsing bodge
+    $whhost ||= "host";
+    our $ho = selecthost($whhost);
 
-                print $ho->{Arch} or die $!;
-              ' "$@"`
+    print $ho->{Arch} or die $!;
+    ' "$@"`
 
 case "$arch" in
 amd64)
-- 
2.20.1

[OSSTEST PATCH 3/5] ts-memdisk-try-append: Enable perl warnings

[Ian Jackson]

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 ts-memdisk-try-append | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-memdisk-try-append b/ts-memdisk-try-append
index a35751eb3..bfb36caf7 100755
--- a/ts-memdisk-try-append
+++ b/ts-memdisk-try-append
@@ -18,7 +18,7 @@
 
 set -xe -o posix
 
-arch=`perl -I. -e '
+arch=`perl -I. -we '
     use Osstest;
     use Osstest::TestSupport;
 
-- 
2.20.1

[OSSTEST PATCH 4/5] ts-memdisk-try-append: More defensive case test

[Ian Jackson]

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 ts-memdisk-try-append | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/ts-memdisk-try-append b/ts-memdisk-try-append
index bfb36caf7..d5c12866f 100755
--- a/ts-memdisk-try-append
+++ b/ts-memdisk-try-append
@@ -34,14 +34,14 @@ arch=`perl -I. -we '
 case "$arch" in
 amd64)
     ;;
-'')
-    echo >&2 'Job arch retrieval failed?'
-    exit 1
-    ;;
-*)
+[a-z]*)
     echo "Arch $arch not supported for memdisk tests"
     exit 0
     ;;
+*)
+    echo >&2 'Job arch retrieval failed?'
+    exit 1
+    ;;
 esac
 
 if ./ts-freebsd-host-install --test-boot --record-append $@; then
-- 
2.20.1

[OSSTEST PATCH 5/5] examination: skip memdisk on non-BIOS hosts, run per-firmware on x86

[Ian Jackson]

This is a combination of two changes:

ts-memdisk-try-append: skip memdisk test on non-BIOS hosts

make-flight: examine: Insist on -bios and -uefi tests on x86

  This will let us skip ts-memdisk-try-append on non-bios platforms
  without risking regression.  It will also definitely spot
  regressions which will occur on any uefi host.

  standalone-generate-dump-flight-runvars reveals the changes are as
  follows:

  New jobs
     test-amd64-i386-examine-bios     test-amd64-i386-examine-uefi
     test-amd64-amd64-examine-bios    test-amd64-amd64-examine-uefi
  added everywhere that has the corresponding plain job, namely
     osstest
     linux-*
     xen-unstable

  These jobs are just like the plain jobs, except that one of
     ,PropEq:Firmware:bios:bios
     ,PropEq:Firmware:bios:uefi
  has been added to the end of all_hostflags.

Signed-off-by: Ian Jackson <iwj@xenproject.org>
---
 make-flight           |  9 +++++++++
 mfi-common            |  1 +
 ts-memdisk-try-append | 17 +++++++++++++++++
 3 files changed, 27 insertions(+)

diff --git a/make-flight b/make-flight
index acb19b113..d7eeb4ebb 100755
--- a/make-flight
+++ b/make-flight
@@ -736,6 +736,15 @@ test_matrix_do_one () {
   do_credit12_tests
 
   do_examine_one
+  case "$dom0arch" in
+    i386|amd64)
+      # Test on BIOS, which makes sure we test one where FreeBSD should
+      # work.  See ts-memdisk-try-append, and set_freebsd_runvars in
+      # mfi-common.
+      do_examine_one bios
+      do_examine_one uefi
+      ;;
+  esac
 
   # No further arm tests at the moment
   if [ $dom0arch = armhf -o $dom0arch = arm64 ]; then
diff --git a/mfi-common b/mfi-common
index 771843f05..59e712f4e 100644
--- a/mfi-common
+++ b/mfi-common
@@ -163,6 +163,7 @@ set_freebsd_runvars () {
     if [ x$no_hostflags != xtrue ]; then
         # osstest doesn't yet know how to install FreeBSD on UEFI hosts, so
         # limit the usable hardware to boxes that boot from BIOS.
+        # See also in ts-memdisk-try-append.
         freebsd_runvars="all_hostflags,=PropEq:Firmware:bios:bios"
     fi
 
diff --git a/ts-memdisk-try-append b/ts-memdisk-try-append
index d5c12866f..8fd6540b8 100755
--- a/ts-memdisk-try-append
+++ b/ts-memdisk-try-append
@@ -28,6 +28,20 @@ arch=`perl -I. -we '
     $whhost ||= "host";
     our $ho = selecthost($whhost);
 
+    if (get_target_property($ho,"firmware") ne "bios") {
+        # osstest does not yet know how to install FreeBSD on UEFI hosts
+	# See also in set_freebsd_runvars in mfi_common
+
+	# Skipping this test on non-BIOS does not lead to pushes
+	# where this test is broken, because make-flght test_matrix_do_one
+	# arranges to run this test on BIOS too.
+
+        broken("skipping memdisk because firmware is not bios",
+	       "skip");
+	print "SKIP\n" or die $!;
+	exit 0;
+    }
+
     print $ho->{Arch} or die $!;
     ' "$@"`
 
@@ -37,6 +51,9 @@ amd64)
 [a-z]*)
     echo "Arch $arch not supported for memdisk tests"
     exit 0
+SKIP)
+    echo "Step skipped."
+    exit 0
     ;;
 *)
     echo >&2 'Job arch retrieval failed?'
-- 
2.20.1