[OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)

[OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)

[Ian Jackson]

This series looks about as ready as it is going to be.  Unfortunately
there are still two issues, each of which cropped up once in my final
formal retest.  See below.

What are people's opinions?  Should I push this to osstest pretest
soon after the Xen codefreeze (eg, after we get the first push after
freeze) ?

The downside would be introducing new low-probability heisenbugs.  But
in my tests it seems that some existing heisenbugs may be abated.
It's hard to say for sure.  The upside is a lower risk that Debian
will break our stuff and also being able to commission a number of
machines which do not work with stretch (Debian oldstable).

The two issues:

1. In one test, libvirt segfaulted.  This is with osstest's own-built
   tested version of libvirt; osstest's own-build tested version of
   Linux; and osstest's own-built xen.git#master.

   http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-i386-libvirt/info.html
   http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-i386-libvirt/rimava1---var-log-kern.log.gz
   | May 28 21:39:11 rimava1 kernel: [ 1146.326593] libvirtd[16868]: segfault at 820 ip b72535b0 sp abce5090 error 4 in libc-2.28.so[b71f0000+14e000]

2. In one test, the Debian installer hung.  The kernel printed many
   soft lockup messages.  This is with osstest's own dom0 kernel and
   hypervisor, but the 4.19.67-2+deb10u1 kernel from Debian.

   http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-amd64-xl-qemut-debianhvm-i386-xsm/info.html
   http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-amd64-xl-qemut-debianhvm-i386-xsm/godello1---var-log-xen-qemu-dm-debianhvm.guest.osstest.log.gz

Ian Jackson (49):
  ts-logs-capture: Cope if xl shutdown leaves domain running for a bit
  ts-xen-build-prep: Install rsync
  lvcreate argments: pass --yes -Z y -W y
  TestSupport: allow more time for apt
  Booting: Use `--' rather than `---' to introduce host cmdline
  di_installcmdline_core: Pass locale on d-i command line
  setupboot_grub2: Drop $submenu variable
  ts-leak-check: Ignore buster's udevd too
  Bodge systemd random seed arrangements
  Debian guests made with xen-tools: Write systemd random seed file
  ts-debian-di-install: Provide guest with more RAM
  Debian: preseed: use priority= alias
  Debian: Specify `priority=critical' rather than locale
  Honour 'LinuxSerialConsole <suite>' host property
  buster: make-hosts-flight: Add to possible suites for hosts flight
  buster: Extend grub2 uefi no install workaround
  buster: ts-host-install: Extend net.ifnames workaround
  buster: Deinstall the "systemd" package
  buster: preseed partman-auto-lvm/guided_size
  buster: ts-host-install: NTP not honoured bug remains
  buster: Extend ARM clock workaround
  buster: Extend guest bootloader workaround
  Honour DebianImageFile_SUITE_ARCH
  buster: Specify DebianImageFile_SUITE_ARCH
  20_linux_xen: Copy Debian buster version into our initramfs area
  20_linux_xen: Adhoc template substitution
  20_linux_xen: Ignore xenpolicy and config files too
  20_linux_xen: Support Xen Security Modules (XSM/FLASK)
  mg-debian-installer-update: support overlay-intramfs-SUITE
  overlay-initrd-buster/sbin/reopen-console: Copy from Debian
  overlay-initrd-buster/sbin/reopen-console: Fix #932416
  buster: chiark-scripts: Install a new version on buster too
  buster: Provide TftpDiVersion
  buster: grub, arm64: extend chainloading workaround
  buster: setupboot_grub2: Note what files exist in /boot
  buster: setupboot_grub2: Handle missing policy file bug
  buster: Extend workaround for dhcpd EROFS bug
  ts-xen-install: Add $ho argument to some_extradebs
  ts-xen-install: Move some_extradebs to Debian.pm
  Debian.pm: Break out standard_extradebs
  Debian.pm: Move standard_extradebs to ts-host-install
  buster: Install own linux-libc-dev package (!)
  setupboot_grub2: Insist on space after directives
  setupboot_grub2: Print line number of entry we are using
  setupboot_grub2: Recognise --nounzip for initramfs
  setupboot_grub2: Copy hv command line from grub to xen.cfg
  setupboot_grub2: Do not boot with XSM policy etc. unless xsm=1
  buster 20_linux_xen: Only load policy in XSM-enabled builds
  buster: Switch to Debian buster as the default suite

 Osstest.pm                                    |   2 +-
 Osstest/Debian.pm                             | 128 +++++--
 Osstest/TestSupport.pm                        |  15 +-
 make-hosts-flight                             |   2 +-
 mfi-common                                    |   9 +
 mg-debian-installer-update                    |  20 ++
 overlay-buster/etc/grub.d/20_linux_xen        | 327 ++++++++++++++++++
 overlay-initrd-buster/sbin/reopen-console     | 126 +++++++
 .../override.conf                             |   3 +
 overlay/usr/local/bin/random-seed-add         |  33 ++
 production-config                             |   5 +
 ts-debian-di-install                          |   2 +-
 ts-debian-fixup                               |   6 +
 ts-host-install                               |   6 +-
 ts-leak-check                                 |   1 +
 ts-logs-capture                               |   1 +
 ts-xen-build-prep                             |   6 +-
 ts-xen-install                                |  41 +--
 18 files changed, 656 insertions(+), 77 deletions(-)
 create mode 100755 overlay-buster/etc/grub.d/20_linux_xen
 create mode 100755 overlay-initrd-buster/sbin/reopen-console
 create mode 100644 overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf
 create mode 100755 overlay/usr/local/bin/random-seed-add

-- 
2.20.1

[OSSTEST PATCH 01/49] ts-logs-capture: Cope if xl shutdown leaves domain running for a bit

[Ian Jackson]

This seems mostly to affect buster but it could in principle affect
earlier releases too I think.

In principle it would be nice to fix this bug, and to have a proper
test for it, but a reliable test is hard and an unreliable one is not
useful.  So I guess we are going to have this workaround
indefinitely...

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-logs-capture | 1 +
 1 file changed, 1 insertion(+)

diff --git a/ts-logs-capture b/ts-logs-capture
index 0320a5a5..d75a2fda 100755
--- a/ts-logs-capture
+++ b/ts-logs-capture
@@ -272,6 +272,7 @@ sub shutdown_guests () {
 		( xl shutdown -a -F -w ; echo y ) &
 	    ) | (
 		read x
+		sleep 10 # xl shutdown is a bit racy :-/
 		xl list | awk '!/^Domain-0 |^Name / {print $2}' \
 		| xargs -t -r -n1 xl destroy ||:
 	    )
-- 
2.20.1

[OSSTEST PATCH 02/49] ts-xen-build-prep: Install rsync

[Ian Jackson]

osstest uses this for transferring configuration, build artefacts, and
so on.

In Debian stretch and earlier, rsync happened to be pulled in by
something else.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-xen-build-prep | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index e9298d54..8e73f763 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -197,7 +197,7 @@ END
 }
 
 sub prep () {
-    my @packages = qw(mercurial
+    my @packages = qw(mercurial rsync
                       build-essential bin86 bcc iasl bc
                       flex bison cmake
                       libpci-dev libncurses5-dev libssl-dev python-dev
-- 
2.20.1

[OSSTEST PATCH 03/49] lvcreate argments: pass --yes -Z y -W y

[Ian Jackson]

The documentation seesm to think this is the default but empirically
it isn't.  In our environment --yes is fine.

I have reported this to Debian as #953183.  Also vaguely related (and
discovered by me at the same time) is #953185.

This came up while trying to get things work on buster.  I don't know
what has changed.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm | 2 +-
 ts-xen-build-prep      | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 1e7da676..43766ee3 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -935,7 +935,7 @@ sub lv_create ($$$$) {
     my ($ho, $vg, $lv, $mb) = @_;
     my $lvdev = "/dev/$vg/$lv";
     target_cmd_root($ho, "lvremove -f $lvdev ||:");
-    target_cmd_root($ho, "lvcreate -L ${mb}M -n $lv $vg");
+    target_cmd_root($ho, "lvcreate --yes -Z y -W y -L ${mb}M -n $lv $vg");
     target_cmd_root($ho, "dd if=/dev/zero of=$lvdev count=10");
     return $lvdev;
 }
diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 8e73f763..dabb9921 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -61,7 +61,7 @@ sub determine_vg_lv () {
 sub lvextend_stage1 () {
     target_cmd_root($ho, <<END);
         set -ex; if ! test -f /root/swap_osstest_enabled; then
-            lvcreate -L 10G -n swap_osstest_build $vg ||:
+            lvcreate --yes -Z y -W y -L 10G -n swap_osstest_build $vg ||:
             mkswap /dev/$vg/swap_osstest_build ||:
             swapon /dev/$vg/swap_osstest_build
             touch /root/swap_osstest_enabled
@@ -84,7 +84,7 @@ sub vginfo () {
 
 sub lvcreate () {
     target_cmd_output_root($ho,
-			   "lvdisplay $lv || lvcreate -l 1 -n $lvleaf $vg");
+			   "lvdisplay $lv || lvcreate --yes -Z y -W y -l 1 -n $lvleaf $vg");
 }
 
 sub lvextend1 ($$$) {
-- 
2.20.1

[OSSTEST PATCH 04/49] TestSupport: allow more time for apt

[Ian Jackson]

Empirically some of these operations can take longer than 30s,
especially with a cold cache.

Note that because of host sharing and our on-host apt lock, the
timeout needs to be the same for every apt operation: a fast operation
could be blocked behind a slow one.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 43766ee3..f4e9414c 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -637,12 +637,12 @@ sub target_install_packages_nonfree_nonconcurrent ($@) {
     my ($ho, @packages) = @_;
     my $slist= '/etc/apt/sources.list';
     my $xsuites= 'contrib non-free';
-    target_cmd_root($ho, <<END, 30);
+    target_cmd_root($ho, <<END, 300);
     perl -i~ -pe 'next unless m/^deb/; s{ main\$}{\$& $xsuites};' $slist
     apt-get update
 END
     target_run_pkgmanager_install($ho,\@packages);
-    target_cmd_root($ho, <<END, 30);
+    target_cmd_root($ho, <<END, 300);
     mv $slist~ $slist
     apt-get update
 END
-- 
2.20.1

[OSSTEST PATCH 05/49] Booting: Use `--' rather than `---' to introduce host cmdline

[Ian Jackson]

Because systemd did something obnoxious, the kernel retaliated in the
game of Core Wars by hiding all arguments before `--' from userspace.
So use `---' instead so that all the arguments remain visible.

This in some sense now applies to host installs a change we had
already made to Debian HVM guests.  See osstest#493b7395
  ts-debian-hvm-install: Use ---, and no longer duplicate $gconsole
and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007
  Kernel command line handling change breaks d-i user-params functionality

This change is fine for all non-ancient versions of Debian, so I have
not made it conditional.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index f4e9414c..ff8103f2 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -2909,7 +2909,7 @@ label overwrite
 	menu label ^Overwrite
 	menu default
 	kernel $kern
-	append $dicmd initrd=$initrd -- $hocmd
+	append $dicmd initrd=$initrd --- $hocmd
 	ipappend $xopts{ipappend}
 	$dtbs
 default overwrite
@@ -2956,7 +2956,7 @@ sub setup_netboot_di_uefi ($$$$$;%) {
 set default=0
 set timeout=5
 menuentry 'overwrite' {
-  linux $kern $dicmd -- $hocmd
+  linux $kern $dicmd --- $hocmd
   initrd $initrd
 }
 END
-- 
2.20.1

[OSSTEST PATCH 06/49] di_installcmdline_core: Pass locale on d-i command line

[Ian Jackson]

In buster, d-i wants when setting up the network, ie before the
preseed is loaded.

We leave it in the preseed too because why not.

I think this change should be fine for older versions of Debian.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 6e9d2072..ba975b87 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -699,7 +699,8 @@ sub di_installcmdline_core ($$;@) {
                "hostname=$tho->{Name}",
                "$xopts{PreseedScheme}=$ps_url",
                "netcfg/dhcp_timeout=150",
-               "netcfg/choose_interface=$netcfg_interface"
+               "netcfg/choose_interface=$netcfg_interface",
+               "debian-installer/locale=en_GB",
                );
 
     my $debconf_priority= $xopts{DebconfPriority};
-- 
2.20.1

[OSSTEST PATCH 07/49] setupboot_grub2: Drop $submenu variable

[Ian Jackson]

We really only used this to check how many levels deep in { we are.
That can be done by checking $#offsets, which is >0 if we are in a
submenu and not otherwise.  We lose the ability to report the start
line of the submenu, but that's OK.

But as a bonus, we no longer bomb out on nested submenus: previously
the first } would cause $submenu to be undef.  Now we pop from
@offsets and all is fine.

Nested submenus are present in Debian buster.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index ba975b87..b8bf67dc 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -452,15 +452,13 @@ sub setupboot_grub2 ($$$$) {
         my @offsets = (0);
         my $entry;
         my $chainentry;
-        my $submenu;
         while (<$f>) {
             next if m/^\s*\#/ || !m/\S/;
             if (m/^\s*\}\s*$/) {
-                die unless $entry || $submenu;
-                if (!$entry && $submenu) {
-                    logm("Met end of a submenu $submenu->{StartLine}..$.. ".
+                die unless $entry || $#offsets;
+                if (!$entry && $#offsets) {
+                    logm("Met end of a submenu at $. (@offsets) ".
                         "Our want kern is $want_kernver");
-                    $submenu= undef;
                     pop @offsets;
                     $offsets[$#offsets]++;
                     next;
@@ -510,7 +508,6 @@ sub setupboot_grub2 ($$$$) {
                 $offsets[$#offsets]++;
             }
             if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
-                $submenu={ StartLine =>$., MenuEntryPath => join ">", @offsets };
                 push @offsets,(0);
             }
             if (m/^\s*chainloader\s*\/EFI\/osstest\/xen.efi/) {
-- 
2.20.1

[OSSTEST PATCH 08/49] ts-leak-check: Ignore buster's udevd too

[Ian Jackson]

For reasons I don't propose to investigate, on buster udevd shows up
like this:

  2019-11-26 18:13:48 Z LEAKED [process 2633 /lib/systemd/systemd-udevd] process: root      2633  1555  0 18:10 ?        00:00:00 /lib/systemd/systemd-udevd

This does not match our suppression.  Add an additional suppression.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-leak-check | 1 +
 1 file changed, 1 insertion(+)

diff --git a/ts-leak-check b/ts-leak-check
index 41e6245d..f3cca8aa 100755
--- a/ts-leak-check
+++ b/ts-leak-check
@@ -202,6 +202,7 @@ xenstore /vm
 xenstore /libxl
 
 process .* udevd
+process .* .*/systemd-udevd
 process .* /.+/systemd-shim
 
 file /var/run/xenstored/db
-- 
2.20.1

[OSSTEST PATCH 09/49] Bodge systemd random seed arrangements

[Ian Jackson]

systemd does not regard the contents of the random seed file as useful
for the purposes of placating the kernel's entropy tracker.  As a
result, the system hangs at boot waiting for entropy.

Fix this by providing a small program which can be used to load a seed
file into /dev/random and also call RNDADDTOENTCNT to add the
appropriate amount to the kernel's counter.

Arrange to run this program instead of
   /lib/systemd/systemd-random-seed load

With systemd the random seed file is in /var/lib/systemd/random-seed
rather than /var/lib/urandom/random-seed.

And, provide an initial contents of this file, via a d-i late_command.

Unfortunately we must hardcode the actual numerical value of
RNDADDTOENTCNT because we don't have a suitable compiler anywhere
nearby.  It seems to have the same value on i386, amd64, armhf and
arm64, our currently supported architectures.

Thanks to Colin Watson for pointers to the systemd random unit and
Matthew Vernon for instructions on overriding just ExecStart.

I think this change should be a no-op on non-systemd systems.

In principle this is a bug in Debian or in systemd, that ought to be
reported upstream.  However, it has been extensively discussed on
debian-devel and it does not seem that any improvement is likely.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm                             | 18 ++++++++++
 .../override.conf                             |  3 ++
 overlay/usr/local/bin/random-seed-add         | 33 +++++++++++++++++++
 3 files changed, 54 insertions(+)
 create mode 100644 overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf
 create mode 100755 overlay/usr/local/bin/random-seed-add

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index b8bf67dc..8ccacc79 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -49,6 +49,7 @@ BEGIN {
                       di_installcmdline_core
                       di_vg_name
                       debian_dhcp_rofs_fix
+		      debian_write_random_seed_command
                       );
     %EXPORT_TAGS = ( );
 
@@ -1087,6 +1088,13 @@ ln -s . /target/boot/boot
 END
     }
 
+    my $cmd = debian_write_random_seed_command('/target');
+    preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+$cmd
+END
+
     $preseed_file .= preseed_hook_cmds();
 
     return create_webfile($ho, "preseed$sfx", $preseed_file);
@@ -1612,4 +1620,14 @@ mv '$script.new' '$script'
 END
 }
 
+sub debian_write_random_seed_command ($) {
+    my ($mountpoint) = @_;
+    my $dir = "$mountpoint/var/lib/systemd";
+    return <<END;
+        umask 077
+        test -d $dir || mkdir -m 0755 $dir
+        dd if=/dev/urandom of=$dir/random-seed bs=1k count=1
+END
+}
+
 1;
diff --git a/overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf b/overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf
new file mode 100644
index 00000000..f6cc0f84
--- /dev/null
+++ b/overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf
@@ -0,0 +1,3 @@
+[Service]
+ExecStart=
+ExecStart=/usr/local/bin/random-seed-add /var/lib/systemd/random-seed
diff --git a/overlay/usr/local/bin/random-seed-add b/overlay/usr/local/bin/random-seed-add
new file mode 100755
index 00000000..89e75c4d
--- /dev/null
+++ b/overlay/usr/local/bin/random-seed-add
@@ -0,0 +1,33 @@
+#!/usr/bin/perl -w
+use strict;
+
+open R, '>', '/dev/random' or die "open /dev/random: $!\n";
+R->autoflush(1);
+
+sub rndaddtoentcnt ($) {
+    my ($bits) = @_;
+    my $x = pack 'L', $bits;
+    my $r = ioctl R, 0x40045201, $x;
+    defined $r or die "RNDADDTOENTCNT: $!\n";
+}
+
+sub process_stdin ($) {
+    my ($f) = @_;
+    my $got = read STDIN, $_, 512;
+    defined $got or die "read $f: $!\n";
+    last if !$got;
+    print R $_ or die "write /dev/random: $!\n";
+    my $bits = length($_) * 8;
+    rndaddtoentcnt($bits);
+}
+
+if (!@ARGV) {
+    process_stdin('stdin');
+} else {
+    die "no options supported\n" if $ARGV[0] =~ m/^\-/;
+    foreach my $f (@ARGV) {
+        open STDIN, '<', $f or die "open for reading $f: $!\n";
+        process_stdin($f);
+    }
+}
+
-- 
2.20.1

[OSSTEST PATCH 10/49] Debian guests made with xen-tools: Write systemd random seed file

[Ian Jackson]

When the Debian guest is not made with d-i, we must still provide this
random seed file.  This can be done in ts-debian-fixup.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-debian-fixup | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/ts-debian-fixup b/ts-debian-fixup
index fef9836e..dfeb4d39 100755
--- a/ts-debian-fixup
+++ b/ts-debian-fixup
@@ -102,6 +102,11 @@ sub console () {
     logm("extra: $extra");
 }
 
+sub randomseed () {
+    my $cmd = debian_write_random_seed_command($mountpoint);
+    target_cmd_root($ho, "set -ex\n".cmd);
+}
+
 sub filesystems () {
     my $rootdev= $r{"$gho->{Guest}_rootdev"};
     return unless defined($rootdev) && length($rootdev);
@@ -215,6 +220,7 @@ END
 target_cmd_root($ho, debian_dhcp_rofs_fix($ho, $mountpoint));
 
 console();
+randomseed();
 filesystems();
 otherfixupcfg();
 writecfg();
-- 
2.20.1

[OSSTEST PATCH 11/49] ts-debian-di-install: Provide guest with more RAM

[Ian Jackson]

buster cannot boot in so little because its initramfs and kernel are
too large.  Bump it to 2G.

However, our armhf test nodes have very little RAM.  And the Debian
armhf does fit in them as a guest still, so use a smaller value there.

Keying this off the architecture rather than the available host memory
is better because you do need the bigger value precisely if you are
not using armhf, and this makes osstest less dependent on a completely
accurate and populated host properties database.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-debian-di-install | 2 +-
 ts-debian-fixup      | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/ts-debian-di-install b/ts-debian-di-install
index 9abb4956..d84407cf 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -64,7 +64,7 @@ $gn ||= 'debian';
 
 our $ho= selecthost($whhost);
 
-our $ram_mb=    512;
+our $ram_mb= $r{arch} =~ m/^armhf/ ? 768 : 2048;
 our $disk_mb= 10000;
 
 our $guesthost= $gn.
diff --git a/ts-debian-fixup b/ts-debian-fixup
index dfeb4d39..528fb03b 100755
--- a/ts-debian-fixup
+++ b/ts-debian-fixup
@@ -104,7 +104,7 @@ sub console () {
 
 sub randomseed () {
     my $cmd = debian_write_random_seed_command($mountpoint);
-    target_cmd_root($ho, "set -ex\n".cmd);
+    target_cmd_root($ho, "set -ex\n".$cmd);
 }
 
 sub filesystems () {
-- 
2.20.1

[OSSTEST PATCH 12/49] Debian: preseed: use priority= alias

[Ian Jackson]

This marginally reduces command line clobber.  This alias has been
supported approximately forever.  (And this code is currently only
used when DebconfPriority is set, which it generally isn't.)

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 8ccacc79..345aff68 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -702,7 +702,7 @@ sub di_installcmdline_core ($$;@) {
                );
 
     my $debconf_priority= $xopts{DebconfPriority};
-    push @cl, "debconf/priority=$debconf_priority"
+    push @cl, "priority=$debconf_priority"
         if defined $debconf_priority;
     push @cl, "rescue/enable=true" if $xopts{RescueMode};
 
-- 
2.20.1

[OSSTEST PATCH 13/49] Debian: Specify `priority=critical' rather than locale

[Ian Jackson]

In buster, it appears that specifying locale on the command line is
not sufficient.  Rather than adding more things to the command line,
instead, just say `priority=critical', by defaulting $debconf_priority
to 'critical'.

I think this change should be fine for earlier suites too.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 345aff68..7b311a14 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -698,12 +698,10 @@ sub di_installcmdline_core ($$;@) {
                "$xopts{PreseedScheme}=$ps_url",
                "netcfg/dhcp_timeout=150",
                "netcfg/choose_interface=$netcfg_interface",
-               "debian-installer/locale=en_GB",
                );
 
-    my $debconf_priority= $xopts{DebconfPriority};
-    push @cl, "priority=$debconf_priority"
-        if defined $debconf_priority;
+    my $debconf_priority= $xopts{DebconfPriority} // 'critical';
+    push @cl, "priority=$debconf_priority";
     push @cl, "rescue/enable=true" if $xopts{RescueMode};
 
     if ($r{syslog_server}) {
-- 
2.20.1

[OSSTEST PATCH 14/49] Honour 'LinuxSerialConsole <suite>' host property

[Ian Jackson]

This works like LinuxSerialConsole.

I originally wrote this to try to work around #940028, where multiple
d-i autoinstalls run in parallel leading to hard-to-debug lossage!
Explicitly specing the console causes it to run only on that one.

However, it turns out that explicitly specifying the console does not
always work and a better fix is needed.  Nevertheless, having added
this feature it seems foolish to throw it away.

Currently there are no hosts with this property so no functiaonal
change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index ff8103f2..7eeac49f 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1447,7 +1447,10 @@ sub get_target_property ($$;$) {
 sub get_host_native_linux_console ($) {
     my ($ho) = @_;
 
-    my $console = get_host_property($ho, "LinuxSerialConsole", "ttyS0");
+    my $console;
+    $console //= get_host_property($ho, "LinuxSerialConsole $ho->{Suite}")
+	if $ho->{Suite};
+    $console //= get_host_property($ho, "LinuxSerialConsole", "ttyS0");
     return $console if $console eq 'NONE';
 
     return "$console,$c{Baud}n8";
-- 
2.20.1

[OSSTEST PATCH 15/49] buster: make-hosts-flight: Add to possible suites for hosts flight

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 make-hosts-flight | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/make-hosts-flight b/make-hosts-flight
index 92da1c7c..e2c3776a 100755
--- a/make-hosts-flight
+++ b/make-hosts-flight
@@ -26,7 +26,7 @@ blessing=$4
 buildflight=$5
 
 : ${ALL_ARCHES:=amd64 i386 arm64 armhf}
-: ${ALL_SUITES:=stretch jessie}
+: ${ALL_SUITES:=buster stretch jessie}
 # ^ most preferred suite first
 
 : ${PERHOST_MAXWAIT:=20000} # seconds
-- 
2.20.1

[OSSTEST PATCH 16/49] buster: Extend grub2 uefi no install workaround

[Ian Jackson]

src:grub2 is RFH in Debian, which is a contributory factor to these
patches in #789798 and #792547 languishing.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 7b311a14..9b4ef967 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -1466,7 +1466,7 @@ d-i partman-auto/expert_recipe string					\\
 END
 
     if (get_host_property($ho, "firmware") eq "uefi") {
-	die unless $ho->{Suite} =~ m/jessie|stretch/;
+	die unless $ho->{Suite} =~ m/jessie|stretch|buster/;
 	# Prevent grub-install from making a new Debian boot entry, so
 	# we always reboot from the network. Debian bug #789798 proposes a
 	# properly preseedable solution to this.
-- 
2.20.1

[OSSTEST PATCH 17/49] buster: ts-host-install: Extend net.ifnames workaround

[Ian Jackson]

Really we should fix this by making a .deb in Debian that we could
install.  But this is a longer-term project.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-host-install | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-host-install b/ts-host-install
index 7a72a867..fe26f70f 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -282,7 +282,7 @@ END
 
     # Don't use "Predictable Network Interface Names"
     # https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
-    push @hocmdline, "net.ifnames=0" if $ho->{Suite} =~ m/stretch/;
+    push @hocmdline, "net.ifnames=0" if $ho->{Suite} =~ m/stretch|buster/;
 
     push @hocmdline,
         get_host_property($ho, "linux-boot-append $ho->{Suite}", ''),
-- 
2.20.1

[OSSTEST PATCH 18/49] buster: Deinstall the "systemd" package

[Ian Jackson]

This installs a pam rule which causes logins to hang.  It also seems
to cause some kind of udev wedge.

We are using sysvinit so this package is not desirable.  Empirically,
removing it makes the system work.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 9b4ef967..a328160e 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -842,6 +842,7 @@ sub preseed_base ($$$;@) {
     if ( $suite !~ /squeeze|wheezy/ ) {
        preseed_hook_command($ho, 'late_command', $sfx, <<END)
 in-target apt-get install -y sysvinit-core
+in-target apt-get remove -y systemd
 END
     }
 
-- 
2.20.1

[OSSTEST PATCH 19/49] buster: preseed partman-auto-lvm/guided_size

[Ian Jackson]

Otherwise we get this question:

  | You may use the whole volume group for guided partitioning, or part
  | of it.  [...]
  | Amount of volume group to use for guided partitioning:

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index a328160e..bac33d00 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -938,6 +938,7 @@ d-i partman/choose_partition select finish
 d-i partman/confirm boolean true
 d-i partman-lvm/confirm boolean true
 d-i partman-lvm/device_remove_lvm_span boolean true
+d-i partman-auto-lvm/guided_size string max
 
 d-i partman/confirm_nooverwrite true
 d-i partman-lvm/confirm_nooverwrite true
-- 
2.20.1

[OSSTEST PATCH 20/49] buster: ts-host-install: NTP not honoured bug remains

[Ian Jackson]

Debian #778564 remains open.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-host-install | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-host-install b/ts-host-install
index fe26f70f..253dbb5d 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -152,7 +152,7 @@ END
 	    my $done= 0;
 	    while (<EI>) {
 		if (m/^server\b|^pool\b\s/) {
-		    if ($ho->{Suite} =~ m/lenny|squeeze|wheezy|jessie|stretch/) {
+		    if ($ho->{Suite} =~ m/lenny|squeeze|wheezy|jessie|stretch|buster/) {
 			$_= $done ? "" : "server $ntpserver\n";
 		    } else {
 			m/^server \Q$ntpserver\E\s/ or
-- 
2.20.1

[OSSTEST PATCH 21/49] buster: Extend ARM clock workaround

[Ian Jackson]

CC: Julien Grall <julien@xen.org>
CC: Stefano Stabellini <sstabellini@kernel.org>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index bac33d00..71167351 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -248,7 +248,7 @@ END
 	push @xenkopt, $xenkopt;
 	# https://bugs.xenproject.org/xen/bug/45
 	push @xenkopt, "clk_ignore_unused"
-	    if $ho->{Suite} =~ m/wheezy|jessie|stretch/;
+	    if $ho->{Suite} =~ m/wheezy|jessie|stretch|buster/;
 
 	$xenkopt = join ' ', @xenkopt;
 	logm("Dom0 Linux options: $xenkopt");
-- 
2.20.1

[OSSTEST PATCH 22/49] buster: Extend guest bootloader workaround

[Ian Jackson]

CC: Julien Grall <julien@xen.org>
CC: Stefano Stabellini <sstabellini@kernel.org>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 71167351..3fc9e555 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -1064,7 +1064,7 @@ END
     logm("\$arch is $arch, \$suite is $suite");
     if ($xopts{PvMenuLst} &&
 	$arch =~ /^arm/ &&
-	$suite =~ /wheezy|jessie|stretch|sid/ ) {
+	$suite =~ /wheezy|jessie|stretch|buster|sid/ ) {
 
 	# Debian doesn't currently know what bootloader to install in
 	# a Xen guest on ARM. We install pv-grub-menu above which
-- 
2.20.1

[OSSTEST PATCH 23/49] Honour DebianImageFile_SUITE_ARCH

[Ian Jackson]

This lets us specify the whole filename, not just a version.
This is needed because for buster we are going to use
   debian-10.2.0-ARCH-xfce-CD-1.iso

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mfi-common | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/mfi-common b/mfi-common
index b40f057e..640cf328 100644
--- a/mfi-common
+++ b/mfi-common
@@ -522,6 +522,15 @@ job_create_test () {
 
 usual_debianhvm_image () {
   local arch=$1; shift
+  if [ -n "$DEBIAN_IMAGE_FILE" ]; then
+      echo $DEBIAN_IMAGE_FILE
+      return
+  fi
+  local file=`getconfig DebianImageFile_${guestsuite}_${arch}`
+  if [ -n "$file " ]; then
+      echo $file
+      return
+  fi
   local ver=$DEBIAN_IMAGE_VERSION
   if [ -z "$ver" ] ; then
       ver=`getconfig DebianImageVersion_$guestsuite`
-- 
2.20.1

[OSSTEST PATCH 24/49] buster: Specify DebianImageFile_SUITE_ARCH

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 production-config | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/production-config b/production-config
index 103b8915..f0ddc132 100644
--- a/production-config
+++ b/production-config
@@ -98,6 +98,9 @@ DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/2019020
 DebianImageVersion_wheezy 7.2.0
 DebianImageVersion_jessie 8.2.0
 DebianImageVersion_stretch 9.4.0
+DebianImageFile_buster_amd64 debian-10.2.0-amd64-xfce-CD-1.iso
+DebianImageFile_buster_i386 debian-10.2.0-i386-xfce-CD-1.iso
+
 
 # Update with ./mg-netgrub-loader-update
 TftpGrubVersion XXXX-XX-XX
-- 
2.20.1

[OSSTEST PATCH 25/49] 20_linux_xen: Copy Debian buster version into our initramfs area

[Ian Jackson]

This is from 41e42571ebc50fa351cd63ce40044946652c5c72 in Debian's grub
package.

We are going to want to modify this to support XSM/FLASK and cope with
upstream build outputs.

In this commit we dump the exact file contents across.  It's not
effective right now because of the ".in" extension.  In fact, the file
is a template.

At the time of writing I am trying to send our substantive changes
upstream via Debian's Gitlab:
  https://salsa.debian.org/grub-team/grub/-/merge_requests/18

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 overlay-buster/etc/grub.d/20_linux_xen.in | 299 ++++++++++++++++++++++
 1 file changed, 299 insertions(+)
 create mode 100644 overlay-buster/etc/grub.d/20_linux_xen.in

diff --git a/overlay-buster/etc/grub.d/20_linux_xen.in b/overlay-buster/etc/grub.d/20_linux_xen.in
new file mode 100644
index 00000000..98ef163c
--- /dev/null
+++ b/overlay-buster/etc/grub.d/20_linux_xen.in
@@ -0,0 +1,299 @@
+#! /bin/sh
+set -e
+
+# grub-mkconfig helper script.
+# Copyright (C) 2006,2007,2008,2009,2010  Free Software Foundation, Inc.
+#
+# GRUB is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# GRUB is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
+
+prefix="@prefix@"
+exec_prefix="@exec_prefix@"
+datarootdir="@datarootdir@"
+
+. "$pkgdatadir/grub-mkconfig_lib"
+
+export TEXTDOMAIN=@PACKAGE@
+export TEXTDOMAINDIR="@localedir@"
+
+CLASS="--class gnu-linux --class gnu --class os --class xen"
+SUPPORTED_INITS="sysvinit:/lib/sysvinit/init systemd:/lib/systemd/systemd upstart:/sbin/upstart"
+
+if [ "x${GRUB_DISTRIBUTOR}" = "x" ] ; then
+  OS=GNU/Linux
+else
+  OS="${GRUB_DISTRIBUTOR} GNU/Linux"
+  CLASS="--class $(echo ${GRUB_DISTRIBUTOR} | tr 'A-Z' 'a-z' | cut -d' ' -f1|LC_ALL=C sed 's,[^[:alnum:]_],_,g') ${CLASS}"
+fi
+
+# loop-AES arranges things so that /dev/loop/X can be our root device, but
+# the initrds that Linux uses don't like that.
+case ${GRUB_DEVICE} in
+  /dev/loop/*|/dev/loop[0-9])
+    GRUB_DEVICE=`losetup ${GRUB_DEVICE} | sed -e "s/^[^(]*(\([^)]\+\)).*/\1/"`
+    # We can't cope with devices loop-mounted from files here.
+    case ${GRUB_DEVICE} in
+      /dev/*) ;;
+      *) exit 0 ;;
+    esac
+  ;;
+esac
+
+# btrfs may reside on multiple devices. We cannot pass them as value of root= parameter
+# and mounting btrfs requires user space scanning, so force UUID in this case.
+if [ "x${GRUB_DEVICE_UUID}" = "x" ] || [ "x${GRUB_DISABLE_LINUX_UUID}" = "xtrue" ] \
+    || ! test -e "/dev/disk/by-uuid/${GRUB_DEVICE_UUID}" \
+    || ( test -e "${GRUB_DEVICE}" && uses_abstraction "${GRUB_DEVICE}" lvm ); then
+  LINUX_ROOT_DEVICE=${GRUB_DEVICE}
+else
+  LINUX_ROOT_DEVICE=UUID=${GRUB_DEVICE_UUID}
+fi
+
+# Allow overriding GRUB_CMDLINE_LINUX and GRUB_CMDLINE_LINUX_DEFAULT.
+if [ "${GRUB_CMDLINE_LINUX_XEN_REPLACE}" ]; then
+  GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX_XEN_REPLACE}"
+fi
+if [ "${GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT}" ]; then
+  GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT}"
+fi
+
+case x"$GRUB_FS" in
+    xbtrfs)
+	rootsubvol="`make_system_path_relative_to_its_root /`"
+	rootsubvol="${rootsubvol#/}"
+	if [ "x${rootsubvol}" != x ]; then
+	    GRUB_CMDLINE_LINUX="rootflags=subvol=${rootsubvol} ${GRUB_CMDLINE_LINUX}"
+	fi;;
+    xzfs)
+	rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
+	bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
+	LINUX_ROOT_DEVICE="ZFS=${rpool}${bootfs%/}"
+	;;
+esac
+
+title_correction_code=
+
+linux_entry ()
+{
+  os="$1"
+  version="$2"
+  xen_version="$3"
+  type="$4"
+  args="$5"
+  xen_args="$6"
+  if [ -z "$boot_device_id" ]; then
+      boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")"
+  fi
+  if [ x$type != xsimple ] ; then
+      if [ x$type = xrecovery ] ; then
+	  title="$(gettext_printf "%s, with Xen %s and Linux %s (%s)" "${os}" "${xen_version}" "${version}" "$(gettext "${GRUB_RECOVERY_TITLE}")")"
+      elif [ "${type#init-}" != "$type" ] ; then
+	  title="$(gettext_printf "%s, with Xen %s and Linux %s (%s)" "${os}" "${xen_version}" "${version}" "${type#init-}")"
+      else
+	  title="$(gettext_printf "%s, with Xen %s and Linux %s" "${os}" "${xen_version}" "${version}")"
+      fi
+      replacement_title="$(echo "Advanced options for ${OS}" | sed 's,>,>>,g')>$(echo "$title" | sed 's,>,>>,g')"
+      if [ x"Xen ${xen_version}>$title" = x"$GRUB_ACTUAL_DEFAULT" ]; then
+         quoted="$(echo "$GRUB_ACTUAL_DEFAULT" | grub_quote)"
+         title_correction_code="${title_correction_code}if [ \"x\$default\" = '$quoted' ]; then default='$(echo "$replacement_title" | grub_quote)'; fi;"
+         grub_warn "$(gettext_printf "Please don't use old title \`%s' for GRUB_DEFAULT, use \`%s' (for versions before 2.00) or \`%s' (for 2.00 or later)" "$GRUB_ACTUAL_DEFAULT" "$replacement_title" "gnulinux-advanced-$boot_device_id>gnulinux-$version-$type-$boot_device_id")"
+      fi
+      echo "menuentry '$(echo "$title" | grub_quote)' ${CLASS} \$menuentry_id_option 'xen-gnulinux-$version-$type-$boot_device_id' {" | sed "s/^/$submenu_indentation/"
+  else
+      title="$(gettext_printf "%s, with Xen hypervisor" "${os}")"
+      echo "menuentry '$(echo "$title" | grub_quote)' ${CLASS} \$menuentry_id_option 'xen-gnulinux-simple-$boot_device_id' {" | sed "s/^/$submenu_indentation/"
+  fi
+  if [ x$type != xrecovery ] ; then
+      save_default_entry | grub_add_tab | sed "s/^/$submenu_indentation/"
+  fi
+
+  if [ -z "${prepare_boot_cache}" ]; then
+    prepare_boot_cache="$(prepare_grub_to_access_device ${GRUB_DEVICE_BOOT} | grub_add_tab)"
+  fi
+  printf '%s\n' "${prepare_boot_cache}" | sed "s/^/$submenu_indentation/"
+  xmessage="$(gettext_printf "Loading Xen %s ..." ${xen_version})"
+  lmessage="$(gettext_printf "Loading Linux %s ..." ${version})"
+  sed "s/^/$submenu_indentation/" << EOF
+	echo	'$(echo "$xmessage" | grub_quote)'
+        if [ "\$grub_platform" = "pc" -o "\$grub_platform" = "" ]; then
+            xen_rm_opts=
+        else
+            xen_rm_opts="no-real-mode edd=off"
+        fi
+	${xen_loader}	${rel_xen_dirname}/${xen_basename} placeholder ${xen_args} \${xen_rm_opts}
+	echo	'$(echo "$lmessage" | grub_quote)'
+	${module_loader}	${rel_dirname}/${basename} placeholder root=${linux_root_device_thisversion} ro ${args}
+EOF
+  if test -n "${initrd}" ; then
+    # TRANSLATORS: ramdisk isn't identifier. Should be translated.
+    message="$(gettext_printf "Loading initial ramdisk ...")"
+    sed "s/^/$submenu_indentation/" << EOF
+	echo	'$(echo "$message" | grub_quote)'
+	${module_loader}	--nounzip   ${rel_dirname}/${initrd}
+EOF
+  fi
+  sed "s/^/$submenu_indentation/" << EOF
+}
+EOF
+}
+
+linux_list=
+for i in /boot/vmlinu[xz]-* /vmlinu[xz]-* /boot/kernel-*; do
+    if grub_file_is_not_garbage "$i"; then
+    	basename=$(basename $i)
+	version=$(echo $basename | sed -e "s,^[^0-9]*-,,g")
+	dirname=$(dirname $i)
+	config=
+	for j in "${dirname}/config-${version}" "${dirname}/config-${alt_version}" "/etc/kernels/kernel-config-${version}" ; do
+	    if test -e "${j}" ; then
+		config="${j}"
+		break
+	    fi
+	done
+        if (grep -qx "CONFIG_XEN_DOM0=y" "${config}" 2> /dev/null || grep -qx "CONFIG_XEN_PRIVILEGED_GUEST=y" "${config}" 2> /dev/null); then linux_list="$linux_list $i" ; fi
+    fi
+done
+if [ "x${linux_list}" = "x" ] ; then
+    exit 0
+fi
+
+file_is_not_sym () {
+    case "$1" in
+	*/xen-syms-*)
+	    return 1;;
+	*)
+	    return 0;;
+    esac
+}
+
+xen_list=
+for i in /boot/xen*; do
+    if grub_file_is_not_garbage "$i" && file_is_not_sym "$i" ; then xen_list="$xen_list $i" ; fi
+done
+prepare_boot_cache=
+boot_device_id=
+
+title_correction_code=
+
+machine=`uname -m`
+
+case "$machine" in
+    i?86) GENKERNEL_ARCH="x86" ;;
+    mips|mips64) GENKERNEL_ARCH="mips" ;;
+    mipsel|mips64el) GENKERNEL_ARCH="mipsel" ;;
+    arm*) GENKERNEL_ARCH="arm" ;;
+    *) GENKERNEL_ARCH="$machine" ;;
+esac
+
+# Extra indentation to add to menu entries in a submenu. We're not in a submenu
+# yet, so it's empty. In a submenu it will be equal to '\t' (one tab).
+submenu_indentation=""
+
+is_top_level=true
+
+while [ "x${xen_list}" != "x" ] ; do
+    list="${linux_list}"
+    current_xen=`version_find_latest $xen_list`
+    xen_basename=`basename ${current_xen}`
+    xen_dirname=`dirname ${current_xen}`
+    rel_xen_dirname=`make_system_path_relative_to_its_root $xen_dirname`
+    xen_version=`echo $xen_basename | sed -e "s,.gz$,,g;s,^xen-,,g"`
+    if [ -z "$boot_device_id" ]; then
+	boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")"
+    fi
+    if [ "x$is_top_level" != xtrue ]; then
+	echo "	submenu '$(gettext_printf "Xen hypervisor, version %s" "${xen_version}" | grub_quote)' \$menuentry_id_option 'xen-hypervisor-$xen_version-$boot_device_id' {"
+    fi
+    if ($grub_file --is-x86-multiboot2 $current_xen); then
+	xen_loader="multiboot2"
+	module_loader="module2"
+    else
+	xen_loader="multiboot"
+	module_loader="module"
+    fi
+    while [ "x$list" != "x" ] ; do
+	linux=`version_find_latest $list`
+	gettext_printf "Found linux image: %s\n" "$linux" >&2
+	basename=`basename $linux`
+	dirname=`dirname $linux`
+	rel_dirname=`make_system_path_relative_to_its_root $dirname`
+	version=`echo $basename | sed -e "s,^[^0-9]*-,,g"`
+	alt_version=`echo $version | sed -e "s,\.old$,,g"`
+	linux_root_device_thisversion="${LINUX_ROOT_DEVICE}"
+
+	initrd=
+	for i in "initrd.img-${version}" "initrd-${version}.img" "initrd-${version}.gz" \
+	   "initrd-${version}" "initramfs-${version}.img" \
+	   "initrd.img-${alt_version}" "initrd-${alt_version}.img" \
+	   "initrd-${alt_version}" "initramfs-${alt_version}.img" \
+	   "initramfs-genkernel-${version}" \
+	   "initramfs-genkernel-${alt_version}" \
+	   "initramfs-genkernel-${GENKERNEL_ARCH}-${version}" \
+	   "initramfs-genkernel-${GENKERNEL_ARCH}-${alt_version}" ; do
+	    if test -e "${dirname}/${i}" ; then
+		initrd="$i"
+		break
+	    fi
+	done
+	if test -n "${initrd}" ; then
+	    gettext_printf "Found initrd image: %s\n" "${dirname}/${initrd}" >&2
+	else
+    # "UUID=" magic is parsed by initrds.  Since there's no initrd, it can't work here.
+	    linux_root_device_thisversion=${GRUB_DEVICE}
+	fi
+
+	if [ "x$is_top_level" = xtrue ] && [ "x${GRUB_DISABLE_SUBMENU}" != xy ]; then
+	    linux_entry "${OS}" "${version}" "${xen_version}" simple \
+		"${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}" "${GRUB_CMDLINE_XEN} ${GRUB_CMDLINE_XEN_DEFAULT}"
+
+	    submenu_indentation="$grub_tab$grub_tab"
+    
+	    if [ -z "$boot_device_id" ]; then
+		boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")"
+	    fi
+            # TRANSLATORS: %s is replaced with an OS name
+	    echo "submenu '$(gettext_printf "Advanced options for %s (with Xen hypervisor)" "${OS}" | grub_quote)' \$menuentry_id_option 'gnulinux-advanced-$boot_device_id' {"
+	echo "	submenu '$(gettext_printf "Xen hypervisor, version %s" "${xen_version}" | grub_quote)' \$menuentry_id_option 'xen-hypervisor-$xen_version-$boot_device_id' {"
+	   is_top_level=false
+	fi
+
+	linux_entry "${OS}" "${version}" "${xen_version}" advanced \
+	    "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}" "${GRUB_CMDLINE_XEN} ${GRUB_CMDLINE_XEN_DEFAULT}"
+	for supported_init in ${SUPPORTED_INITS}; do
+	    init_path="${supported_init#*:}"
+	    if [ -x "${init_path}" ] && [ "$(readlink -f /sbin/init)" != "$(readlink -f "${init_path}")" ]; then
+		linux_entry "${OS}" "${version}" "${xen_version}" "init-${supported_init%%:*}" \
+		    "${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} init=${init_path}" "${GRUB_CMDLINE_XEN} ${GRUB_CMDLINE_XEN_DEFAULT}"
+
+	    fi
+	done
+	if [ "x${GRUB_DISABLE_RECOVERY}" != "xtrue" ]; then
+	    linux_entry "${OS}" "${version}" "${xen_version}" recovery \
+		"single ${GRUB_CMDLINE_LINUX}" "${GRUB_CMDLINE_XEN}"
+	fi
+
+	list=`echo $list | tr ' ' '\n' | fgrep -vx "$linux" | tr '\n' ' '`
+    done
+    if [ x"$is_top_level" != xtrue ]; then
+	echo '	}'
+    fi
+    xen_list=`echo $xen_list | tr ' ' '\n' | fgrep -vx "$current_xen" | tr '\n' ' '`
+done
+
+# If at least one kernel was found, then we need to
+# add a closing '}' for the submenu command.
+if [ x"$is_top_level" != xtrue ]; then
+  echo '}'
+fi
+
+echo "$title_correction_code"
-- 
2.20.1

[OSSTEST PATCH 26/49] 20_linux_xen: Adhoc template substitution

[Ian Jackson]

This file is a template that various build-time variables get
substituted into.  Make thos substitutions by hand (actually, by
copying the values our file for stretch).  And rename the file.

So now we are using our file instead of the grub package's.  But it is
the same...

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 .../etc/grub.d/{20_linux_xen.in => 20_linux_xen}       | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
 rename overlay-buster/etc/grub.d/{20_linux_xen.in => 20_linux_xen} (98%)
 mode change 100644 => 100755

diff --git a/overlay-buster/etc/grub.d/20_linux_xen.in b/overlay-buster/etc/grub.d/20_linux_xen
old mode 100644
new mode 100755
similarity index 98%
rename from overlay-buster/etc/grub.d/20_linux_xen.in
rename to overlay-buster/etc/grub.d/20_linux_xen
index 98ef163c..fb3ed82f
--- a/overlay-buster/etc/grub.d/20_linux_xen.in
+++ b/overlay-buster/etc/grub.d/20_linux_xen
@@ -17,14 +17,14 @@ set -e
 # You should have received a copy of the GNU General Public License
 # along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
 
-prefix="@prefix@"
-exec_prefix="@exec_prefix@"
-datarootdir="@datarootdir@"
+prefix="/usr"
+exec_prefix="/usr"
+datarootdir="/usr/share"
 
 . "$pkgdatadir/grub-mkconfig_lib"
 
-export TEXTDOMAIN=@PACKAGE@
-export TEXTDOMAINDIR="@localedir@"
+export TEXTDOMAIN=grub
+export TEXTDOMAINDIR="${datarootdir}/locale"
 
 CLASS="--class gnu-linux --class gnu --class os --class xen"
 SUPPORTED_INITS="sysvinit:/lib/sysvinit/init systemd:/lib/systemd/systemd upstart:/sbin/upstart"
-- 
2.20.1

[OSSTEST PATCH 27/49] 20_linux_xen: Ignore xenpolicy and config files too

[Ian Jackson]

"file_is_not_sym" currently only checks for xen-syms.  Extend it to
disregard xenpolicy (XSM policy files) and files ending .config (which
are built by the Xen upstream build system in some configurations and
can therefore end up in /boot).

Rename the function accordingly, to "file_is_not_xen_garbage".

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 overlay-buster/etc/grub.d/20_linux_xen | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/overlay-buster/etc/grub.d/20_linux_xen b/overlay-buster/etc/grub.d/20_linux_xen
index fb3ed82f..01dfcb57 100755
--- a/overlay-buster/etc/grub.d/20_linux_xen
+++ b/overlay-buster/etc/grub.d/20_linux_xen
@@ -167,10 +167,14 @@ if [ "x${linux_list}" = "x" ] ; then
     exit 0
 fi
 
-file_is_not_sym () {
+file_is_not_xen_garbage () {
     case "$1" in
 	*/xen-syms-*)
 	    return 1;;
+	*/xenpolicy-*)
+	    return 1;;
+	*/*.config)
+	    return 1;;
 	*)
 	    return 0;;
     esac
@@ -178,7 +182,7 @@ file_is_not_sym () {
 
 xen_list=
 for i in /boot/xen*; do
-    if grub_file_is_not_garbage "$i" && file_is_not_sym "$i" ; then xen_list="$xen_list $i" ; fi
+    if grub_file_is_not_garbage "$i" && file_is_not_xen_garbage "$i" ; then xen_list="$xen_list $i" ; fi
 done
 prepare_boot_cache=
 boot_device_id=
-- 
2.20.1

[OSSTEST PATCH 28/49] 20_linux_xen: Support Xen Security Modules (XSM/FLASK)

[Ian Jackson]

XSM is enabled by adding "flask=enforcing" as a Xen command line
argument, and providing the policy file as a grub module.

We make entries for both with and without XSM.  If XSM is not compiled
into Xen, then there are no policy files, so no change to the boot
options.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 overlay-buster/etc/grub.d/20_linux_xen | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/overlay-buster/etc/grub.d/20_linux_xen b/overlay-buster/etc/grub.d/20_linux_xen
index 01dfcb57..4d3294a2 100755
--- a/overlay-buster/etc/grub.d/20_linux_xen
+++ b/overlay-buster/etc/grub.d/20_linux_xen
@@ -84,6 +84,11 @@ esac
 title_correction_code=
 
 linux_entry ()
+{
+  linux_entry_xsm "$@" false
+  linux_entry_xsm "$@" true
+}
+linux_entry_xsm ()
 {
   os="$1"
   version="$2"
@@ -91,6 +96,18 @@ linux_entry ()
   type="$4"
   args="$5"
   xen_args="$6"
+  xsm="$7"
+  # If user wants to enable XSM support, make sure there's
+  # corresponding policy file.
+  if ${xsm} ; then
+      xenpolicy="xenpolicy-$xen_version"
+      if test ! -e "${xen_dirname}/${xenpolicy}" ; then
+	  return
+      fi
+      xen_args="$xen_args flask=enforcing"
+      xen_version="$(gettext_printf "%s (XSM enabled)" "$xen_version")"
+      # xen_version is used for messages only; actual file is xen_basename
+  fi
   if [ -z "$boot_device_id" ]; then
       boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")"
   fi
@@ -140,6 +157,13 @@ EOF
     sed "s/^/$submenu_indentation/" << EOF
 	echo	'$(echo "$message" | grub_quote)'
 	${module_loader}	--nounzip   ${rel_dirname}/${initrd}
+EOF
+  fi
+  if test -n "${xenpolicy}" ; then
+    message="$(gettext_printf "Loading XSM policy ...")"
+    sed "s/^/$submenu_indentation/" << EOF
+	echo	'$(echo "$message" | grub_quote)'
+	${module_loader}     ${rel_dirname}/${xenpolicy}
 EOF
   fi
   sed "s/^/$submenu_indentation/" << EOF
-- 
2.20.1

[OSSTEST PATCH 29/49] mg-debian-installer-update: support overlay-intramfs-SUITE

[Ian Jackson]

This lets us patch the installer more easily.

No uses yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-debian-installer-update | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index f1e682f9..fb4fe2ab 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -33,6 +33,8 @@ sbase=$site/dists/$suite
 
 src=$sbase/main/installer-$arch/current/images/netboot/
 
+osstest_dir="$(pwd)"
+
 case ${suite}_${arch} in
     lenny_armhf|squeeze_armhf|lenny_arm64|squeeze_arm64|wheezy_arm64)
         # No such thing.
@@ -188,6 +190,24 @@ if [ "x$specialkernel" != x ]; then
     rm -rf x
 fi
 
+overlay_initrd=$osstest_dir/overlay-initrd-$suite
+if [ -e "$overlay_initrd" ]; then
+    for f in $files; do
+        s=${f/:*} ; d=${f/*:}
+        case "$d" in
+            *initrd*)
+                echo "adding $overlay_initrd to $d"
+                (
+                    set -e
+                    cd "$overlay_initrd"
+                    find -print0 | cpio -0 -Hnewc -o \
+                        | gzip -9nf
+                ) >>$d.new
+                ;;
+        esac
+    done
+fi
+
 for f in $files; do
         s=${f/:*} ; d=${f/*:}
         mv -f $d.new $d
-- 
2.20.1

[OSSTEST PATCH 30/49] overlay-initrd-buster/sbin/reopen-console: Copy from Debian

[Ian Jackson]

We are going to patch this file to work around a bug, using the new
overlay mechanism.

The first step is to include the file in our overlay so we overwrite
it.  Currently, this is a no-op, so no functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 overlay-initrd-buster/sbin/reopen-console | 94 +++++++++++++++++++++++
 1 file changed, 94 insertions(+)
 create mode 100755 overlay-initrd-buster/sbin/reopen-console

diff --git a/overlay-initrd-buster/sbin/reopen-console b/overlay-initrd-buster/sbin/reopen-console
new file mode 100755
index 00000000..dd354deb
--- /dev/null
+++ b/overlay-initrd-buster/sbin/reopen-console
@@ -0,0 +1,94 @@
+#!/bin/sh
+
+# First find the enabled consoles from the kernel, noting if one is 'preferred'
+# Record these.
+# Run the startup scripts on the preferred console
+
+# In order to have D-I appear on all consoles, modify the inittab to
+# add one entry for each console, running debian-installer.
+# Finally HUP init so that it runs those installers
+# (but doesn't rerun the sysinit startup stuff, including this script)
+
+
+NL="
+"
+
+LOGGER_UP=0
+LOG_FILE=/var/log/reopen-console
+
+log() {
+	# In very early startup we don't have syslog. Log to file that
+	# we can flush out later so we can at least see what happened
+	# at early startup
+	if [ $LOGGER_UP -eq 1 ]; then
+	        logger -t reopen-console "$@"
+	else
+		echo "$@" >> $LOG_FILE
+	fi
+}
+
+flush_logger () {
+	cat $LOG_FILE | logger -t reopen-console
+	rm $LOG_FILE
+}
+
+consoles=
+preferred=
+# Retrieve all enabled consoles from kernel; ignore those
+# for which no device file exists
+
+kernelconsoles="$(cat /proc/consoles)"
+for cons in $(echo "$kernelconsoles" | sed -n -r -e 's/(^.*)  .*\((.*)\).*$/\1/p' )
+do
+	log "Looking at console $cons from /proc/consoles"
+	status=$(echo "$kernelconsoles" | grep $cons | sed -n -r -e 's/(^.*) *.*\((.*)\).*$/\2/p' )
+	if [ -e "/dev/$cons" ] && [ $(echo "$status" | grep -o 'E') ]; then
+		consoles="${consoles:+$consoles$NL}$cons"
+		log "   Adding $cons to consoles list"
+	fi
+	# 'C' console is 'most prefered'.
+	if [ $(echo "$status" | grep -o 'C') ]; then
+		preferred="$cons"
+		log "   $cons is preferred"
+	fi
+done
+
+if [ -z "$consoles" ]; then
+	# Nothing found? Default to /dev/console.
+	log "Found no consoles! Defaulting to /dev/console"
+	consoles=console
+fi
+if [ -z "$preferred" ]; then
+	#None marked preferred? Use the first one
+	preferred=$(echo "$consoles" | head -n 1)
+	log "Found no preferred console. Picking $preferred"
+fi
+
+for cons in $consoles
+do
+	echo "/dev/$cons " >> /var/run/console-devices
+done
+echo "/dev/$preferred " > /var/run/console-preferred
+
+
+# Add debian-installer lines into inittab - one per console
+for cons in $consoles
+do
+	log "Adding inittab entry for $cons"
+	echo "$cons::respawn:/sbin/debian-installer" >> /etc/inittab
+done
+
+# Run the startup scripts once, using the preferred console
+cons=$(cat /var/run/console-preferred)
+# Some other session may have that console as ctty. Steal it from them
+/sbin/steal-ctty $cons "$@"
+
+# Now we should have syslog running, so flush our log entries
+LOGGER_UP=1
+flush_logger
+
+# Finally restart init to run debian-installer on discovered consoles
+log "Restarting init to start d-i on the consoles we found"
+kill -HUP 1
+
+exit 0
-- 
2.20.1

[OSSTEST PATCH 31/49] overlay-initrd-buster/sbin/reopen-console: Fix #932416

[Ian Jackson]

This bug affects us.  Cherry pick the changes to the relevant file
from the commit in the upstream debian-installer repo:

  https://salsa.debian.org/installer-team/rootskel/commit/0ee43d05b83f8ef5a856f3282e002a111809cef9

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 overlay-initrd-buster/sbin/reopen-console | 36 +++++++++++++++++++++--
 1 file changed, 34 insertions(+), 2 deletions(-)

diff --git a/overlay-initrd-buster/sbin/reopen-console b/overlay-initrd-buster/sbin/reopen-console
index dd354deb..13b15a33 100755
--- a/overlay-initrd-buster/sbin/reopen-console
+++ b/overlay-initrd-buster/sbin/reopen-console
@@ -16,6 +16,17 @@ NL="
 LOGGER_UP=0
 LOG_FILE=/var/log/reopen-console
 
+# If we're running with preseeding, we have a problem with running d-i
+# on multiple consoles. We'll end up running each of those d-i
+# instances in parallel with all kinds of hilarious undefined
+# behaviour as they trip over each other! If we detect that we're
+# preseeding (via any of the possible preseed methods), DO NOT run d-i
+# multiple times. Instead, fall back to the older, more simple
+# behaviour and run it once. If the user wants to see or interact with
+# their preseed on a specific console, they get to tell us which one
+# they want to use.
+PRESEEDING=0
+
 log() {
 	# In very early startup we don't have syslog. Log to file that
 	# we can flush out later so we can at least see what happened
@@ -32,6 +43,20 @@ flush_logger () {
 	rm $LOG_FILE
 }
 
+# If we have a preseed.cfg in the initramfs
+if [ -e /preseed.cfg ]; then
+    log "Found /preseed.cfg; falling back to simple mode for preseeding"
+    PRESEEDING=1
+fi
+
+# Have we been told to do preseeding stuff on the boot command line?
+for WORD in auto url; do
+    if (grep -qw "$WORD" /proc/cmdline); then
+	log "Found \"$WORD\" in the command line; falling back to simple mode for preseeding"
+	PRESEEDING=1
+    fi
+done
+
 consoles=
 preferred=
 # Retrieve all enabled consoles from kernel; ignore those
@@ -44,7 +69,7 @@ do
 	status=$(echo "$kernelconsoles" | grep $cons | sed -n -r -e 's/(^.*) *.*\((.*)\).*$/\2/p' )
 	if [ -e "/dev/$cons" ] && [ $(echo "$status" | grep -o 'E') ]; then
 		consoles="${consoles:+$consoles$NL}$cons"
-		log "   Adding $cons to consoles list"
+		log "   Adding $cons to possible consoles list"
 	fi
 	# 'C' console is 'most prefered'.
 	if [ $(echo "$status" | grep -o 'C') ]; then
@@ -64,6 +89,13 @@ if [ -z "$preferred" ]; then
 	log "Found no preferred console. Picking $preferred"
 fi
 
+# If we're preseeding, do simple stuff here (see above). We just
+# want one console. Let's pick the preferred one ONLY
+if [ $PRESEEDING = 1 ]; then
+    log "Running with preseeding. Picking preferred $preferred ONLY"
+    consoles=$preferred
+fi
+
 for cons in $consoles
 do
 	echo "/dev/$cons " >> /var/run/console-devices
@@ -88,7 +120,7 @@ LOGGER_UP=1
 flush_logger
 
 # Finally restart init to run debian-installer on discovered consoles
-log "Restarting init to start d-i on the consoles we found"
+log "Restarting init to start d-i on the console(s) we found"
 kill -HUP 1
 
 exit 0
-- 
2.20.1

[OSSTEST PATCH 32/49] buster: chiark-scripts: Install a new version on buster too

[Ian Jackson]

We need various fixes that are not in buster, sadly.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 production-config | 1 +
 1 file changed, 1 insertion(+)

diff --git a/production-config b/production-config
index f0ddc132..e3870d47 100644
--- a/production-config
+++ b/production-config
@@ -107,6 +107,7 @@ TftpGrubVersion XXXX-XX-XX
 
 DebianExtraPackages_jessie chiark-scripts_6.0.3~citrix1_all.deb
 DebianExtraPackages_stretch chiark-scripts_6.0.4~citrix1_all.deb
+DebianExtraPackages_buster chiark-scripts_6.0.5~citrix1_all.deb
 
 DebianExtraPackages_uefi_i386_jessie   extradebs-uefi-i386-2018-04-01/
 DebianExtraPackages_uefi_amd64_jessie  extradebs-uefi-amd64-2018-04-01/
-- 
2.20.1

[OSSTEST PATCH 33/49] buster: Provide TftpDiVersion

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 production-config | 1 +
 1 file changed, 1 insertion(+)

diff --git a/production-config b/production-config
index e3870d47..6372ac9a 100644
--- a/production-config
+++ b/production-config
@@ -91,6 +91,7 @@ TftpNetbootGroup osstest
 TftpDiVersion_wheezy 2016-06-08
 TftpDiVersion_jessie 2018-06-26
 TftpDiVersion_stretch 2020-02-10
+TftpDiVersion_buster 2020-05-19
 
 DebianSnapshotBackports_jessie http://snapshot.debian.org/archive/debian/20190206T211314Z/
 
-- 
2.20.1

[OSSTEST PATCH 34/49] buster: grub, arm64: extend chainloading workaround

[Ian Jackson]

multiboot[2] isn't supported.

Also link to the bug report.

CC: Julien Grall <julien@xen.org>
CC: Stefano Stabellini <sstabellini@kernel.org>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 3fc9e555..9f1ce1df 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -443,9 +443,10 @@ sub setupboot_grub2 ($$$$) {
     my $kernkey= (defined $xenhopt ? 'KernDom0' : 'KernOnly');
 
     # Grub2 on jessie/stretch ARM* doesn't do multiboot, so we must chainload.
+    # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=884770
     my $need_uefi_chainload =
         get_host_property($ho, "firmware") eq "uefi" &&
-        $ho->{Suite} =~ m/jessie|stretch/ && $ho->{Arch} =~ m/^arm/;
+        $ho->{Suite} =~ m/jessie|stretch|buster/ && $ho->{Arch} =~ m/^arm/;
 
     my $parsemenu= sub {
         my $f= bl_getmenu_open($ho, $rmenu, "$stash/$ho->{Name}--grub.cfg.1");
-- 
2.20.1

[OSSTEST PATCH 35/49] buster: setupboot_grub2: Note what files exist in /boot

[Ian Jackson]

Nothing uses this yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 9f1ce1df..0386ff7a 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -448,6 +448,11 @@ sub setupboot_grub2 ($$$$) {
         get_host_property($ho, "firmware") eq "uefi" &&
         $ho->{Suite} =~ m/jessie|stretch|buster/ && $ho->{Arch} =~ m/^arm/;
 
+    my %bootfiles =
+	map { $_ => 1 }
+	split / /,
+	target_cmd_output_root($ho, "cd /boot && echo *");
+
     my $parsemenu= sub {
         my $f= bl_getmenu_open($ho, $rmenu, "$stash/$ho->{Name}--grub.cfg.1");
     
-- 
2.20.1

[OSSTEST PATCH 36/49] buster: setupboot_grub2: Handle missing policy file bug

[Ian Jackson]

This is a complex interaction between update-grub and the Xen build
system on ARM64.  Not sure exactly who to blame but since we have our
own 20_linux_xen bodge, let's wait until we don't.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 0386ff7a..51217fb4 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -496,7 +496,17 @@ sub setupboot_grub2 ($$$$) {
 			 " kernel $entry->{KernVer}, not $want_kernver)");
 		} elsif ($want_xsm && !defined $entry->{Xenpolicy}) {
 		    logm("(skipping entry at $entry->{StartLine}..$.;".
-			 " XSM policy file not present)");
+			 " XSM policy file not mentioned)");
+		} elsif ($ho->{Suite} =~ m/buster/ &&
+			 defined $entry->{Xenpolicy} &&
+			 !$bootfiles{
+                             $entry->{Xenpolicy} =~ m{^/?} ? $' : die
+						 }) {
+		    # Our 20_linux_xen bodge with buster's update-grub
+		    # generates entries which mention /boot/xenpolicy-xen
+		    # even though that file doesn't exist on ARM64.
+		    logm("(skipping entry at $entry->{StartLine}..$.;".
+			 " XSM policy file not on disk!)");
 		} else {
 		    # yes!
 		    last;
-- 
2.20.1

[OSSTEST PATCH 37/49] buster: Extend workaround for dhcpd EROFS bug

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 51217fb4..49d94b9b 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -1624,7 +1624,7 @@ sub debian_dhcp_rofs_fix ($$) {
     # / is still ro.  In stretch, the isc dhcp client spins requesting
     # an address and then sending a DHCPDECLINE (and then, usually,
     # eventually works).
-    return '' unless $ho->{Suite} =~ m/stretch/;
+    return '' unless $ho->{Suite} =~ m/stretch|buster/;
     my $script = "$rootdir/lib/udev/ifupdown-hotplug";
     <<END.<<'ENDQ'.<<END
 set -ex
-- 
2.20.1

[OSSTEST PATCH 38/49] ts-xen-install: Add $ho argument to some_extradebs

[Ian Jackson]

This is going to move to Debian.pm.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-xen-install | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/ts-xen-install b/ts-xen-install
index 08b4ea23..6196a890 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -71,8 +71,8 @@ sub packages () {
         if toolstack($ho)->{ExtraPackages};
 }
 
-sub some_extradebs ($) {
-    my ($items) = @_;
+sub some_extradebs ($$) {
+    my ($ho, $items) = @_;
     my $cfgvar = join('_', @$items);
     my $specs = $c{$cfgvar};
     if (!length $specs) {
@@ -111,11 +111,11 @@ sub extradebs () {
     my $suite = $ho->{Suite};
 
     # $c{ DebianExtraPackages_<suite> }
-    some_extradebs([ 'DebianExtraPackages', $suite ]);
+    some_extradebs($ho, [ 'DebianExtraPackages', $suite ]);
 
     # $c{ DebianExtraPackages_<firmware>_<arch>_<suite> }
     my $firmware = get_host_property($ho, "firmware");
-    some_extradebs([ 'DebianExtraPackages', $firmware, $ho->{Arch}, $suite ]);
+    some_extradebs($ho, [ 'DebianExtraPackages', $firmware, $ho->{Arch}, $suite ]);
 }
 
 sub extract () {
-- 
2.20.1

[OSSTEST PATCH 39/49] ts-xen-install: Move some_extradebs to Debian.pm

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 38 ++++++++++++++++++++++++++++++++++++++
 ts-xen-install    | 36 ------------------------------------
 2 files changed, 38 insertions(+), 36 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 49d94b9b..d51ac493 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -24,6 +24,7 @@ use POSIX;
 
 use IO::File;
 use File::Copy;
+use File::Basename;
 
 use Osstest;
 use Osstest::TestSupport;
@@ -50,6 +51,7 @@ BEGIN {
                       di_vg_name
                       debian_dhcp_rofs_fix
 		      debian_write_random_seed_command
+		      some_extradebs
                       );
     %EXPORT_TAGS = ( );
 
@@ -1646,4 +1648,40 @@ sub debian_write_random_seed_command ($) {
 END
 }
 
+sub some_extradebs ($$) {
+    my ($ho, $items) = @_;
+    my $cfgvar = join('_', @$items);
+    my $specs = $c{$cfgvar};
+    if (!length $specs) {
+	logm("$cfgvar: no extra debs");
+	return;
+    }
+    my $counter = 0;
+    my $rsync_installed;
+    foreach my $spec (split /\s+/, $specs) {
+	my $path = $spec;
+	$path = "$c{Images}/$path" unless $path =~ m{^/|^\./};
+	my ($ontarget, $dpkgopts);
+	if ($path =~ m{/$}) {
+	    $ontarget = "extrapackages-$cfgvar-$counter"; $counter++;
+	    $dpkgopts = '-iGROEB';
+	    logm("$cfgvar: updating packages from directory $path");
+	    target_install_packages($ho, qw(rsync)) unless $rsync_installed++;
+	    target_putfile_root($ho,300, "$path/.", $ontarget, '-r');
+	} elsif ($path =~ m{\.deb$}) {
+	    $path =~ s{_\.deb}{ "_$ho->{Arch}.deb" }e;
+	    logm("$cfgvar: installing $path");
+	    $ontarget = basename($path);
+	    $dpkgopts = '-iB';
+	    target_putfile_root($ho,300, $path, $ontarget);
+	} else {
+	    die "no / or . deb in $spec ?";
+	}
+	target_cmd_root($ho,
+			"dpkg --force-confold $dpkgopts $ontarget </dev/null",
+			300);
+	target_run_pkgmanager_install($ho, [], 0,1);
+    }
+}
+
 1;
diff --git a/ts-xen-install b/ts-xen-install
index 6196a890..d67cd121 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -71,42 +71,6 @@ sub packages () {
         if toolstack($ho)->{ExtraPackages};
 }
 
-sub some_extradebs ($$) {
-    my ($ho, $items) = @_;
-    my $cfgvar = join('_', @$items);
-    my $specs = $c{$cfgvar};
-    if (!length $specs) {
-	logm("$cfgvar: no extra debs");
-	return;
-    }
-    my $counter = 0;
-    my $rsync_installed;
-    foreach my $spec (split /\s+/, $specs) {
-	my $path = $spec;
-	$path = "$c{Images}/$path" unless $path =~ m{^/|^\./};
-	my ($ontarget, $dpkgopts);
-	if ($path =~ m{/$}) {
-	    $ontarget = "extrapackages-$cfgvar-$counter"; $counter++;
-	    $dpkgopts = '-iGROEB';
-	    logm("$cfgvar: updating packages from directory $path");
-	    target_install_packages($ho, qw(rsync)) unless $rsync_installed++;
-	    target_putfile_root($ho,300, "$path/.", $ontarget, '-r');
-	} elsif ($path =~ m{\.deb$}) {
-	    $path =~ s{_\.deb}{ "_$ho->{Arch}.deb" }e;
-	    logm("$cfgvar: installing $path");
-	    $ontarget = basename($path);
-	    $dpkgopts = '-iB';
-	    target_putfile_root($ho,300, $path, $ontarget);
-	} else {
-	    die "no / or . deb in $spec ?";
-	}
-	target_cmd_root($ho,
-			"dpkg --force-confold $dpkgopts $ontarget </dev/null",
-			300);
-	target_run_pkgmanager_install($ho, [], 0,1);
-    }
-}
-
 sub extradebs () {
     my $suite = $ho->{Suite};
 
-- 
2.20.1

[OSSTEST PATCH 40/49] Debian.pm: Break out standard_extradebs

[Ian Jackson]

Refactor this out of ts-xen-install.  We are going to run it in
ts-host-install.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 7 ++++++-
 ts-xen-install    | 3 +--
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index d51ac493..60393ca9 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -51,7 +51,6 @@ BEGIN {
                       di_vg_name
                       debian_dhcp_rofs_fix
 		      debian_write_random_seed_command
-		      some_extradebs
                       );
     %EXPORT_TAGS = ( );
 
@@ -1684,4 +1683,10 @@ sub some_extradebs ($$) {
     }
 }
 
+sub standard_extradebs ($) {
+    my ($ho) = @_;
+    # $c{ DebianExtraPackages_<suite> }
+    some_extradebs($ho, [ 'DebianExtraPackages', $ho->{Suite} ]);
+}
+
 1;
diff --git a/ts-xen-install b/ts-xen-install
index d67cd121..965fd519 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -74,8 +74,7 @@ sub packages () {
 sub extradebs () {
     my $suite = $ho->{Suite};
 
-    # $c{ DebianExtraPackages_<suite> }
-    some_extradebs($ho, [ 'DebianExtraPackages', $suite ]);
+    standard_extradebs($ho);
 
     # $c{ DebianExtraPackages_<firmware>_<arch>_<suite> }
     my $firmware = get_host_property($ho, "firmware");
-- 
2.20.1

[OSSTEST PATCH 41/49] Debian.pm: Move standard_extradebs to ts-host-install

[Ian Jackson]

This makes it effect builds on Debian, too.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 1 +
 ts-host-install   | 2 ++
 ts-xen-install    | 2 --
 3 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 60393ca9..2d30b3e9 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -51,6 +51,7 @@ BEGIN {
                       di_vg_name
                       debian_dhcp_rofs_fix
 		      debian_write_random_seed_command
+		      some_extradebs standard_extradebs
                       );
     %EXPORT_TAGS = ( );
 
diff --git a/ts-host-install b/ts-host-install
index 253dbb5d..924c1e06 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -146,6 +146,8 @@ END
             qw(amd64-microcode intel-microcode));
     }
 
+    standard_extradebs($ho);
+
     my $ntpserver = get_target_property($ho, 'NtpServer');
     if ($ntpserver) {
 	target_editfile_root($ho, '/etc/ntp.conf', sub {
diff --git a/ts-xen-install b/ts-xen-install
index 965fd519..5d4f8b0d 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -74,8 +74,6 @@ sub packages () {
 sub extradebs () {
     my $suite = $ho->{Suite};
 
-    standard_extradebs($ho);
-
     # $c{ DebianExtraPackages_<firmware>_<arch>_<suite> }
     my $firmware = get_host_property($ho, "firmware");
     some_extradebs($ho, [ 'DebianExtraPackages', $firmware, $ho->{Arch}, $suite ]);
-- 
2.20.1

[OSSTEST PATCH 42/49] buster: Install own linux-libc-dev package (!)

[Ian Jackson]

As reported here:
  https://patchew.org/QEMU/20200513120147.21443-1-f4bug@amsat.org/
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960271
the kernel has broken the build of upstream qemu.  This made it
into a Debian stable kernel update.  This breaks our CI runs almost
completely, when they run with buster.

I spoke to the Debian kernel folks and apparently there is no intent
to fast-track a fix to this.  So instead I have made a kernel source
package with the patch from that bug report, and built the
linux-libc-dev package fromk it.  The source is here for the time
being:
  https://www.chiark.greenend.org.uk/~ijackson/quicksand/2020/libc-kernel-bug.960271/

Deployment note: the source and linux-libc-dev_*.deb are in the
images directory on osstest@test-lab.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 production-config | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/production-config b/production-config
index 6372ac9a..7a8c0fd3 100644
--- a/production-config
+++ b/production-config
@@ -108,7 +108,7 @@ TftpGrubVersion XXXX-XX-XX
 
 DebianExtraPackages_jessie chiark-scripts_6.0.3~citrix1_all.deb
 DebianExtraPackages_stretch chiark-scripts_6.0.4~citrix1_all.deb
-DebianExtraPackages_buster chiark-scripts_6.0.5~citrix1_all.deb
+DebianExtraPackages_buster chiark-scripts_6.0.5~citrix1_all.deb libc-kernel-bug.960271/linux-libc-dev_4.19.118-2.0iwj_.deb
 
 DebianExtraPackages_uefi_i386_jessie   extradebs-uefi-i386-2018-04-01/
 DebianExtraPackages_uefi_amd64_jessie  extradebs-uefi-amd64-2018-04-01/
-- 
2.20.1

[OSSTEST PATCH 43/49] setupboot_grub2: Insist on space after directives

[Ian Jackson]

These parsing regexps were all wrong!

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 2d30b3e9..a20569e5 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -529,17 +529,17 @@ sub setupboot_grub2 ($$$$) {
             if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
                 push @offsets,(0);
             }
-            if (m/^\s*chainloader\s*\/EFI\/osstest\/xen.efi/) {
+            if (m/^\s*chainloader\s+\/EFI\/osstest\/xen.efi/) {
                 die unless $entry;
                 $entry->{Hv}= $1;
                 $entry->{Chainload} = 1;
             }
-            if (m/^\s*multiboot2?\s*(?:\/boot)?\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+            if (m/^\s*multiboot2?\s+(?:\/boot)?\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
                 die unless $entry;
                 $entry->{Hv}= $1;
                 $entry->{Chainload} = 0;
             }
-            if (m/^\s*multiboot2?\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))\s+(.*)/) {
+            if (m/^\s*multiboot2?\s+(?:\/boot)?\/(vmlinu[xz]-(\S+))\s+(.*)/) {
                 die unless $entry;
                 $entry->{KernOnly}= $1;
                 $entry->{KernVer}= $2;
@@ -551,7 +551,7 @@ sub setupboot_grub2 ($$$$) {
                 $entry->{KernVer}= $2;
                 $entry->{KernOpts}= $3;
             }
-            if (m/^\s*module2?\s*(?:\/boot)?\/(initrd\S+)/) {
+            if (m/^\s*module2?\s+(?:\/boot)?\/(initrd\S+)/) {
                 $entry->{Initrd}= $1;
             }
 	    if (m/^\s*module2?\s*\/(xenpolicy\S+)/) {
-- 
2.20.1

[OSSTEST PATCH 44/49] setupboot_grub2: Print line number of entry we are using

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index a20569e5..615047cb 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -562,7 +562,7 @@ sub setupboot_grub2 ($$$$) {
 
         die unless $entry->{Title};
 
-        logm("boot check: grub2, found $entry->{Title}");
+        logm("boot check: grub2, l.$., found $entry->{Title}");
 
 	die unless $entry->{$kernkey};
 	if (defined $xenhopt) {
-- 
2.20.1

[OSSTEST PATCH 45/49] setupboot_grub2: Recognise --nounzip for initramfs

[Ian Jackson]

Modern versions of update-grub like to add this.  We need to spot this
so that under EFI we generate the right things in xen.cfg.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 615047cb..de53c1ac 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -551,7 +551,7 @@ sub setupboot_grub2 ($$$$) {
                 $entry->{KernVer}= $2;
                 $entry->{KernOpts}= $3;
             }
-            if (m/^\s*module2?\s+(?:\/boot)?\/(initrd\S+)/) {
+            if (m/^\s*module2?\s+(?:--nounzip\s+)*(?:\/boot)?\/(initrd\S+)/) {
                 $entry->{Initrd}= $1;
             }
 	    if (m/^\s*module2?\s*\/(xenpolicy\S+)/) {
-- 
2.20.1

[OSSTEST PATCH 46/49] setupboot_grub2: Copy hv command line from grub to xen.cfg

[Ian Jackson]

This reuses all of the stuff that update-grub, etc., have put there.

In particular without this we never have flask=enforcing!

We have to do something about the ${xen_rm_opts} that appear in these
entries.  In principle there might be many variable expansions, but in
practice there is only this one It applies only to x86, and this use
of chainloading to xen.efi and reading xen.cfg applies only to arm64.
And anyway we weren't putting it these things in before.  So OK...

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index de53c1ac..c18bf718 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -534,10 +534,11 @@ sub setupboot_grub2 ($$$$) {
                 $entry->{Hv}= $1;
                 $entry->{Chainload} = 1;
             }
-            if (m/^\s*multiboot2?\s+(?:\/boot)?\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+            if (m/^\s*multiboot2?\s+(?:\/boot)?\/(xen\-[0-9][-+.0-9a-z]*\S+)\s+(.*)/) {
                 die unless $entry;
                 $entry->{Hv}= $1;
                 $entry->{Chainload} = 0;
+		$entry->{HvOpts} = $2;
             }
             if (m/^\s*multiboot2?\s+(?:\/boot)?\/(vmlinu[xz]-(\S+))\s+(.*)/) {
                 die unless $entry;
@@ -575,7 +576,7 @@ sub setupboot_grub2 ($$$$) {
             # Propagate relevant fields of the main entry over to the
             # chain entry for use of subsequent code.
             foreach (qw(KernVer KernDom0 KernOnly KernOpts
-                        Initrd Xenpolicy)) {
+                        Initrd Xenpolicy HvOpts)) {
 		next unless $entry->{$_};
 		die if $chainentry->{$_};
 		$chainentry->{$_} = $entry->{$_};
@@ -604,12 +605,14 @@ sub setupboot_grub2 ($$$$) {
 
 	if ($need_uefi_chainload) {
 	    my $entry= $parsemenu->();
+	    my $hvopts = $entry->{HvOpts};
+	    $hvopts =~ s/\$\{\w+\}//g; # delete these and hope!
 	    my $xencfg = <<END;
 [global]
 default=osstest
 
 [osstest]
-options=$xenhopt
+options=$hvopts
 kernel=vmlinuz $entry->{KernOpts}
 END
             $xencfg .= "ramdisk=initrd.gz\n" if $entry->{Initrd};
-- 
2.20.1

[OSSTEST PATCH 47/49] setupboot_grub2: Do not boot with XSM policy etc. unless xsm=1

[Ian Jackson]

This prevents us from passing an XSM policy file, and
`flask=enforcing', in supposedly-non-XSM tests.

These bootloader entries can appear because the Xen upstream build
ships XSM policy files by default even if XSM is disabled in the
hypervisor, causing update-grub to generate useless `XSM enabled'
entries.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index c18bf718..b140ede2 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -499,6 +499,9 @@ sub setupboot_grub2 ($$$$) {
 		} elsif ($want_xsm && !defined $entry->{Xenpolicy}) {
 		    logm("(skipping entry at $entry->{StartLine}..$.;".
 			 " XSM policy file not mentioned)");
+		} elsif (!$want_xsm && defined $entry->{Xenpolicy}) {
+		    logm("(skipping entry at $entry->{StartLine}..$.;".
+			 " XSM policy file, but we don't want XSM)");
 		} elsif ($ho->{Suite} =~ m/buster/ &&
 			 defined $entry->{Xenpolicy} &&
 			 !$bootfiles{
-- 
2.20.1

[OSSTEST PATCH 48/49] buster 20_linux_xen: Only load policy in XSM-enabled builds

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 overlay-buster/etc/grub.d/20_linux_xen | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/overlay-buster/etc/grub.d/20_linux_xen b/overlay-buster/etc/grub.d/20_linux_xen
index 4d3294a2..6f2a98ba 100755
--- a/overlay-buster/etc/grub.d/20_linux_xen
+++ b/overlay-buster/etc/grub.d/20_linux_xen
@@ -159,7 +159,7 @@ EOF
 	${module_loader}	--nounzip   ${rel_dirname}/${initrd}
 EOF
   fi
-  if test -n "${xenpolicy}" ; then
+  if ${xsm} && test -n "${xenpolicy}" ; then
     message="$(gettext_printf "Loading XSM policy ...")"
     sed "s/^/$submenu_indentation/" << EOF
 	echo	'$(echo "$message" | grub_quote)'
-- 
2.20.1

[OSSTEST PATCH 49/49] buster: Switch to Debian buster as the default suite

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest.pm b/Osstest.pm
index 1e381d8f..63dddd95 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -87,7 +87,7 @@ our %c = qw(
 
     Images images
 
-    DebianSuite stretch
+    DebianSuite buster
     DebianMirrorSubpath debian
 
     TestHostKeypairPath id_rsa_osstest
-- 
2.20.1

RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)

[Paul Durrant]

> -----Original Message-----
> From: Ian Jackson <ian.jackson@eu.citrix.com>
> Sent: 29 May 2020 12:19
> To: xen-devel@lists.xenproject.org; Paul Durrant <paul@xen.org>
> Cc: Ian Jackson <ian.jackson@eu.citrix.com>; committers@xenproject.org
> Subject: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)
> 
> This series looks about as ready as it is going to be.  Unfortunately
> there are still two issues, each of which cropped up once in my final
> formal retest.  See below.
> 
> What are people's opinions?  Should I push this to osstest pretest
> soon after the Xen codefreeze (eg, after we get the first push after
> freeze) ?
> 

That sounds reasonable; I don't think we want to perturb osstest until after the first push.

> The downside would be introducing new low-probability heisenbugs.  But
> in my tests it seems that some existing heisenbugs may be abated.
> It's hard to say for sure.  The upside is a lower risk that Debian
> will break our stuff and also being able to commission a number of
> machines which do not work with stretch (Debian oldstable).
> 

I assume we can revert if things go badly wrong and being able to commission more machines would seem to be quite beneficial at this
stage.

  Paul

> The two issues:
> 
> 1. In one test, libvirt segfaulted.  This is with osstest's own-built
>    tested version of libvirt; osstest's own-build tested version of
>    Linux; and osstest's own-built xen.git#master.
> 
>    http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-i386-libvirt/info.html
>    http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-i386-libvirt/rimava1---var-log-
> kern.log.gz
>    | May 28 21:39:11 rimava1 kernel: [ 1146.326593] libvirtd[16868]: segfault at 820 ip b72535b0 sp
> abce5090 error 4 in libc-2.28.so[b71f0000+14e000]
> 
> 2. In one test, the Debian installer hung.  The kernel printed many
>    soft lockup messages.  This is with osstest's own dom0 kernel and
>    hypervisor, but the 4.19.67-2+deb10u1 kernel from Debian.
> 
>    http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-amd64-xl-qemut-debianhvm-i386-
> xsm/info.html
>    http://logs.test-lab.xenproject.org/osstest/logs/150456/test-amd64-amd64-xl-qemut-debianhvm-i386-
> xsm/godello1---var-log-xen-qemu-dm-debianhvm.guest.osstest.log.gz
> 
> Ian Jackson (49):
>   ts-logs-capture: Cope if xl shutdown leaves domain running for a bit
>   ts-xen-build-prep: Install rsync
>   lvcreate argments: pass --yes -Z y -W y
>   TestSupport: allow more time for apt
>   Booting: Use `--' rather than `---' to introduce host cmdline
>   di_installcmdline_core: Pass locale on d-i command line
>   setupboot_grub2: Drop $submenu variable
>   ts-leak-check: Ignore buster's udevd too
>   Bodge systemd random seed arrangements
>   Debian guests made with xen-tools: Write systemd random seed file
>   ts-debian-di-install: Provide guest with more RAM
>   Debian: preseed: use priority= alias
>   Debian: Specify `priority=critical' rather than locale
>   Honour 'LinuxSerialConsole <suite>' host property
>   buster: make-hosts-flight: Add to possible suites for hosts flight
>   buster: Extend grub2 uefi no install workaround
>   buster: ts-host-install: Extend net.ifnames workaround
>   buster: Deinstall the "systemd" package
>   buster: preseed partman-auto-lvm/guided_size
>   buster: ts-host-install: NTP not honoured bug remains
>   buster: Extend ARM clock workaround
>   buster: Extend guest bootloader workaround
>   Honour DebianImageFile_SUITE_ARCH
>   buster: Specify DebianImageFile_SUITE_ARCH
>   20_linux_xen: Copy Debian buster version into our initramfs area
>   20_linux_xen: Adhoc template substitution
>   20_linux_xen: Ignore xenpolicy and config files too
>   20_linux_xen: Support Xen Security Modules (XSM/FLASK)
>   mg-debian-installer-update: support overlay-intramfs-SUITE
>   overlay-initrd-buster/sbin/reopen-console: Copy from Debian
>   overlay-initrd-buster/sbin/reopen-console: Fix #932416
>   buster: chiark-scripts: Install a new version on buster too
>   buster: Provide TftpDiVersion
>   buster: grub, arm64: extend chainloading workaround
>   buster: setupboot_grub2: Note what files exist in /boot
>   buster: setupboot_grub2: Handle missing policy file bug
>   buster: Extend workaround for dhcpd EROFS bug
>   ts-xen-install: Add $ho argument to some_extradebs
>   ts-xen-install: Move some_extradebs to Debian.pm
>   Debian.pm: Break out standard_extradebs
>   Debian.pm: Move standard_extradebs to ts-host-install
>   buster: Install own linux-libc-dev package (!)
>   setupboot_grub2: Insist on space after directives
>   setupboot_grub2: Print line number of entry we are using
>   setupboot_grub2: Recognise --nounzip for initramfs
>   setupboot_grub2: Copy hv command line from grub to xen.cfg
>   setupboot_grub2: Do not boot with XSM policy etc. unless xsm=1
>   buster 20_linux_xen: Only load policy in XSM-enabled builds
>   buster: Switch to Debian buster as the default suite
> 
>  Osstest.pm                                    |   2 +-
>  Osstest/Debian.pm                             | 128 +++++--
>  Osstest/TestSupport.pm                        |  15 +-
>  make-hosts-flight                             |   2 +-
>  mfi-common                                    |   9 +
>  mg-debian-installer-update                    |  20 ++
>  overlay-buster/etc/grub.d/20_linux_xen        | 327 ++++++++++++++++++
>  overlay-initrd-buster/sbin/reopen-console     | 126 +++++++
>  .../override.conf                             |   3 +
>  overlay/usr/local/bin/random-seed-add         |  33 ++
>  production-config                             |   5 +
>  ts-debian-di-install                          |   2 +-
>  ts-debian-fixup                               |   6 +
>  ts-host-install                               |   6 +-
>  ts-leak-check                                 |   1 +
>  ts-logs-capture                               |   1 +
>  ts-xen-build-prep                             |   6 +-
>  ts-xen-install                                |  41 +--
>  18 files changed, 656 insertions(+), 77 deletions(-)
>  create mode 100755 overlay-buster/etc/grub.d/20_linux_xen
>  create mode 100755 overlay-initrd-buster/sbin/reopen-console
>  create mode 100644 overlay/etc/systemd/system/systemd-random-seed.service.d/override.conf
>  create mode 100755 overlay/usr/local/bin/random-seed-add
> 
> --
> 2.20.1

RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)

[Ian Jackson]

Paul Durrant writes ("RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)"):
> I assume we can revert if things go badly wrong and being able to commission more machines would seem to be quite beneficial at this
> stage.

Thanks for your opinion.

It would be possible to revert the final switch, yes.  Most of the
other changes are supposed to work with stretch too.

I haven't done a test run with the new code, but the old version of
Debian.  I will do that.  That will give confidence that we could, in
fact, revert things.

Ian.

RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)

[Ian Jackson]

Ian Jackson writes ("RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)"):
> Paul Durrant writes ("RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)"):
> > I assume we can revert if things go badly wrong and being able to commission more machines would seem to be quite beneficial at this
> > stage.
> 
> Thanks for your opinion.
> 
> It would be possible to revert the final switch, yes.  Most of the
> other changes are supposed to work with stretch too.
> 
> I haven't done a test run with the new code, but the old version of
> Debian.  I will do that.  That will give confidence that we could, in
> fact, revert things.

I have now done this and it resulted in one bugfix.

So if you are still content to go ahead I would like to push this
series to osstest's own self-push-gate now.

Ian.

RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)

[Paul Durrant]

> -----Original Message-----
> From: Ian Jackson <ian.jackson@citrix.com>
> Sent: 08 June 2020 16:34
> To: paul@xen.org
> Cc: xen-devel@lists.xenproject.org; committers@xenproject.org
> Subject: RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)
> 
> Ian Jackson writes ("RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)"):
> > Paul Durrant writes ("RE: [OSSTEST PATCH v2 00/49] Switch to Debian buster (= Debian stable)"):
> > > I assume we can revert if things go badly wrong and being able to commission more machines would
> seem to be quite beneficial at this
> > > stage.
> >
> > Thanks for your opinion.
> >
> > It would be possible to revert the final switch, yes.  Most of the
> > other changes are supposed to work with stretch too.
> >
> > I haven't done a test run with the new code, but the old version of
> > Debian.  I will do that.  That will give confidence that we could, in
> > fact, revert things.
> 
> I have now done this and it resulted in one bugfix.
> 
> So if you are still content to go ahead I would like to push this
> series to osstest's own self-push-gate now.
> 

Yes, I am content to go ahead.

  Paul