[PATCH OSSTEST v4 00/25] add distro domU testing flight

[PATCH OSSTEST v4 00/25] add distro domU testing flight

[Ian Campbell]

Hi,

It's been a long time since v3 of this series, which was
http://article.gmane.org/gmane.comp.emulators.xen.devel/224415. That
contains the background which I won't repeat here.

Since then I have done the big rebase, addressed the review feed back
from last time (I think!) and added a new feature which is a bit
unrelated but uses the new ability to do d-i based installs to test
qcow, vhd and raw file images in the main flight.

There are some patches in here which I think will be useful to the Intel
folks doing the nested virt testing, specifically the refactoring of how
overlays and ssh host keys are done will be useful for installing a
guest to be treated as the L1 host. LongTao, I've CCd you only on the 3
patches which I think will be of interest instead of the full 24 patch
series.

For the benefit of that work it would be good to get at least the
precursor cleanup/refactoring patches into good shape and into the tree
sooner rather than later.

I've run this as an adhoc job on the Cambridge instance, the results of
which are at:
http://www.chiark.greenend.org.uk/~xensrcts/logs/36995/

The failures mainly seem to be some sort of proxy issue returning old
data (similar to what used to happen to mg-debian-installer-update), so
we boot an older d-i kernel+initrd and end up unable to find any kernel
modules. I've tried to workaround this with a patch near the end
"ts-debian-di-install: Use ftp.debian.org directly", the latter I'm not
sure how to handle other than by bashing the local proxy over the head.

On a small number I see "The installer failed to download a file from
the mirror." messages, which might be the cache/proxy or might be an
actual Debian issue.

On the one armhf attempt which didn't suffer from the first problem
(daily netboot based) it seems the kernel didn't actual halt when
shutdown, which is likely to be a real Debian kernel bug.

I've also run an adhoc xen-unstable test of the new jobs there, results
at http://www.chiark.greenend.org.uk/~xensrcts/logs/37011/ the
interesting jobs are *-pvgrub, *-pygrub for the bootloader tests and
*-{qcow2,raw,vhd} for the image format stuff, LVM is tested by the
existing -{xl,libvirt} jobs.

Results are a bit of a mixed bag. 64-bit pvgrub worked, but 32-bit
pvgrub failed due to Debian installing a non-PAE kernel (which cannot be
booted), that's probably a bug in the preseeding. The pygrub test
passed.

-qcow2 and -vhd tests passed except the armhf one which couldn't find
the appropriate vmlinuz-xen because mg-debian-installer-update doesn't
create it, and the Wheezy baseline doesn't support running as a Xen
guest on ARM anyway. I could suppress this in make-flight until the
Jessie upgrade.

The -raw tests all failed due to a timeout doing:
        dd if=/dev/zero of=/var/lib/xen/images/debian/disk.raw bs=1M count=10000
        
I'm unclear if it timed out after 30 (as the code seems to say) or 60
(as the logs seem to say) but either way something obviously needs
adjusting. I just tried it on my local test box and it took 1m20s and
reported 130 MB/s. I've appended an as yet untested patch which assumed
100 MB/s and calculates a timeout from that.

I could switch to using "count=1 seek=10000" to create a sparse file,
which should be pretty much instantaneous, but allocating the whole
backing file seemed like a good idea.

Ian.

[PATCH OSSTEST v4 01/25] TestSupport: Add helper to fetch a URL on a host

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Make sure wget is installed
---
 Osstest/Debian.pm      | 2 +-
 Osstest/TestSupport.pm | 9 ++++++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 69530fb..5e6bcba 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -617,7 +617,7 @@ d-i apt-setup/another boolean false
 d-i apt-setup/non-free boolean false
 d-i apt-setup/contrib boolean false
 
-d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, $extra_packages
+d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, wget, $extra_packages
 
 $xopts{ExtraPreseed}
 
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 1053016..c4adc48 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -55,7 +55,7 @@ BEGIN {
                       target_putfilecontents_stash
 		      target_putfilecontents_root_stash
                       target_put_guest_image target_editfile
-                      target_editfile_cancel
+                      target_editfile_cancel target_fetchurl
                       target_editfile_root target_file_exists
                       target_run_apt
                       target_install_packages target_install_packages_norec
@@ -1560,6 +1560,13 @@ END
     return $cfgpath;
 }
 
+sub target_fetchurl($$$;$) {
+    my ($ho, $url, $path, $timeo) = @_;
+    $timeo ||= 2000;
+    target_cmd_root($ho, "wget --progress=dot:mega -O $path $url", $timeo);
+}
+
+
 sub target_put_guest_image ($$$) {
     my ($ho, $gho, $default) = @_;
     my $specimage = $r{"$gho->{Guest}_image"};
-- 
2.1.4

Re: [PATCH OSSTEST v4 01/25] TestSupport: Add helper to fetch a URL on a host

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v4 01/25] TestSupport: Add helper to fetch a URL on a host"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

We're not using this to fetch URLs on the public internet, are we ?
If we are then we need to honour $c{HttpProxy}.

Perhaps you could add a comment to the effect that target_fetch_url
should be used for URLs within the test environment, only.

Ian.

Re: [PATCH OSSTEST v4 01/25] TestSupport: Add helper to fetch a URL on a host

[Ian Campbell]

On Thu, 2015-04-02 at 12:24 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v4 01/25] TestSupport: Add helper to fetch a URL on a host"):
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> 
> We're not using this to fetch URLs on the public internet, are we ?
> If we are then we need to honour $c{HttpProxy}.

It is used to fetch the kernel+initrd to test, which previously came
from the apt cache, via $c{DebianMirrorHost}.

Would you recommend $c{HttpProxy} or ${DebianMirrorProxy} for this? I
think given the generic nature only the former, even though the only
caller is currently grabbing apt-cachable things.

> Perhaps you could add a comment to the effect that target_fetch_url
> should be used for URLs within the test environment, only.
> 
> Ian.

[PATCH OSSTEST v4 02/25] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
--
v3: Use CamelCase for xopts, use the correct condition
---
 Osstest/TestSupport.pm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index c4adc48..490e16d 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1533,12 +1533,13 @@ sub prepareguest_part_xencfg ($$$$$) {
     my $onpoweroff= $xopts->{OnPowerOff} || 'destroy';
     my $oncrash= $xopts->{OnCrash} || 'preserve';
     my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
+    my $viftype= $xopts->{VifType} ? "type=$xopts->{VifType}," : "";
     my $xoptcfg= $xopts->{ExtraConfig};
     $xoptcfg='' unless defined $xoptcfg;
     my $xencfg= <<END;
 name        = '$gho->{Name}'
 memory = ${ram_mb}
-vif         = [ 'type=ioemu,mac=$gho->{Ether}' ]
+vif         = [ '${viftype}mac=$gho->{Ether}' ]
 #
 on_poweroff = '$onpoweroff'
 on_reboot   = '$onreboot'
@@ -1633,6 +1634,7 @@ END
         $cfg .= "bios='$bios'\n";
     }
 
+    $xopts{VifType} ||= "ioemu";
     my $cfgpath= prepareguest_part_xencfg($ho, $gho, $ram_mb, \%xopts, $cfg);
     target_cmd_root($ho, <<END);
         (echo $passwd; echo $passwd) | vncpasswd $gho->{Guest}.vncpw
-- 
2.1.4

Re: [PATCH OSSTEST v4 02/25] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v4 02/25] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

[PATCH OSSTEST v4 03/25] create_webfile: Support use with guests as well as hosts.

[Ian Campbell]

In particular make the path unique by ensuring it includes the host
and guest name in the guest case.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: New patch
---
 Osstest/TestSupport.pm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 490e16d..8f67f7e 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1979,6 +1979,8 @@ sub await_webspace_fetch_byleaf ($$$$$) {
 sub create_webfile ($$$) {
     my ($ho, $tail, $contents) = @_; # $contents as for file_link_contents
     my $wf_rhs= $ho->{Name}."_".$tail;
+    # $ho->{Host} is set if $ho is a guest.
+    $wf_rhs= $ho->{Host}->{Name}."_${wf_rhs}" if $ho->{Host};
     my $wf_common= $c{WebspaceCommon}.$wf_rhs;
     my $wf_url= $c{WebspaceUrl}.$wf_common;
     my $wf_file= $c{WebspaceFile}.$wf_common;
-- 
2.1.4

Re: [PATCH OSSTEST v4 03/25] create_webfile: Support use with guests as well as hosts.

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v4 03/25] create_webfile: Support use with guests as well as hosts."):
> In particular make the path unique by ensuring it includes the host
> and guest name in the guest case.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

But I have a minor style quibble:

> +    # $ho->{Host} is set if $ho is a guest.
> +    $wf_rhs= $ho->{Host}->{Name}."_${wf_rhs}" if $ho->{Host};

It would be usual to write

  +    $wf_rhs= $ho->{Host}{Name}."_${wf_rhs}" if $ho->{Host};

since only the first -> is required.

Thanks,
Ian.

[PATCH OSSTEST v4 04/25] Debian: refactor code to add preseed commands to the preseed file

[Ian Campbell]

Call it from ts-debian-hvm-install.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: longtaox.pang@intel.com
---
v3: New patch
---
 Osstest/Debian.pm     | 16 +++++++++++-----
 ts-debian-hvm-install |  3 +++
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 5e6bcba..ef602b1 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -37,7 +37,7 @@ BEGIN {
                       %preseed_cmds
                       preseed_base
                       preseed_create
-                      preseed_hook_command preseed_hook_installscript
+                      preseed_hook_command preseed_hook_installscript preseed_hook_cmds
                       di_installcmdline_core
                       );
     %EXPORT_TAGS = ( );
@@ -907,10 +907,7 @@ d-i partman-auto/expert_recipe string					\\
 
 END
 
-    foreach my $di_key (keys %preseed_cmds) {
-        $preseed_file .= "d-i preseed/$di_key string ".
-            (join ' && ', @{ $preseed_cmds{$di_key} }). "\n";
-    }
+    $preseed_file .= preseed_hook_cmds();
 
     if ($ho->{Flags}{'no-di-kernel'}) {
 	$preseed_file .= <<END;
@@ -954,4 +951,13 @@ chmod +x '$installer_pathname'
 END
 }
 
+sub preseed_hook_cmds () {
+    my $preseed;
+    foreach my $di_key (keys %preseed_cmds) {
+        $preseed .= "d-i preseed/$di_key string ".
+            (join ' && ', @{ $preseed_cmds{$di_key} }). "\n";
+    }
+    return $preseed;
+}
+
 1;
diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index cfd5144..95fce9a 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -77,6 +77,9 @@ d-i preseed/late_command string \\
         in-target mkdir -p /root/.ssh; \\
         in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
 END
+
+    $preseed_file .= preseed_hook_cmds();
+
     return $preseed_file;
 }
 
-- 
2.1.4

Re: [PATCH OSSTEST v4 04/25] Debian: refactor code to add preseed commands to the preseed file

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v4 04/25] Debian: refactor code to add preseed commands to the preseed file"):
> Call it from ts-debian-hvm-install.

Please expand the commit message.  Something like:

 This means that, in future, ts-debian-hvm-install can use
 preseed_hook_command and preseed_hook_installscript.  No functional
 change for now.

Thanks,
Ian.

Re: [PATCH OSSTEST v4 04/25] Debian: refactor code to add preseed commands to the preseed file

[Ian Campbell]

On Thu, 2015-04-02 at 12:31 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v4 04/25] Debian: refactor code to add preseed commands to the preseed file"):
> > Call it from ts-debian-hvm-install.
> 
> Please expand the commit message.  Something like:
> 
>  This means that, in future, ts-debian-hvm-install can use
>  preseed_hook_command and preseed_hook_installscript.  No functional
>  change for now.

I assumed you meant s/hvm/di/ since the hvm case is handled right here,
but done.

Ian

> 
> Thanks,
> Ian.

[PATCH OSSTEST v4 05/25] Debian: refactor preseeding of .ssh directories

[Ian Campbell]

Causes known_hosts to be consistently created as well as ~osstest/.ssh
to be consistently populated (it previsouly wasn't for HVM guests).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: longtaox.pang@intel.com
---
v3: New patch
---
 Osstest/Debian.pm     | 99 +++++++++++++++++++++++++++++----------------------
 ts-debian-hvm-install |  5 ++-
 2 files changed, 59 insertions(+), 45 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index ef602b1..db89326 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -37,6 +37,7 @@ BEGIN {
                       %preseed_cmds
                       preseed_base
                       preseed_create
+                      preseed_ssh
                       preseed_hook_command preseed_hook_installscript preseed_hook_cmds
                       di_installcmdline_core
                       );
@@ -551,6 +552,60 @@ sub di_installcmdline_core ($$;@) {
     return @cl;
 }
 
+sub preseed_ssh ($$) {
+    my ($ho,$sfx) = @_;
+
+    my $authkeys_url= create_webfile($ho, "authkeys$sfx", authorized_keys());
+
+    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key.pub";
+    my $hostkey= get_filecontents($hostkeyfile);
+    chomp($hostkey); $hostkey.="\n";
+    my $knownhosts= '';
+
+    my $hostsq= $dbh_tests->prepare(<<END);
+        SELECT val FROM runvars
+         WHERE flight=? AND name LIKE '%host'
+         GROUP BY val
+END
+    $hostsq->execute($flight);
+    while (my ($node) = $hostsq->fetchrow_array()) {
+        my $longname= "$node.$c{TestHostDomain}";
+        my (@hostent)= gethostbyname($longname);
+        if (!@hostent) {
+            logm("skipping host key for nonexistent host $longname");
+            next;
+        }
+        my $specs= join ',', $longname, $node, map {
+            join '.', unpack 'W4', $_;
+        } @hostent[4..$#hostent];
+        logm("adding host key for $specs");
+        $knownhosts.= "$specs ".$hostkey;
+    }
+    $hostsq->finish();
+
+    $knownhosts.= "localhost,127.0.0.1 ".$hostkey;
+    my $knownhosts_url= create_webfile($ho, "known_hosts$sfx", $knownhosts);
+
+    preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+r=/target/root
+cd \$r
+
+umask 022
+mkdir .ssh
+wget -O .ssh/authorized_keys '$authkeys_url'
+wget -O .ssh/known_hosts     '$knownhosts_url'
+
+u=osstest
+h=/home/\$u
+mkdir /target\$h/.ssh
+cp .ssh/authorized_keys /target\$h/.ssh
+chroot /target chown -R \$u.\$u \$h/.ssh
+END
+}
+
 sub preseed_base ($$$;@) {
     my ($ho,$suite,$extra_packages,%xopts) = @_;
 
@@ -639,43 +694,12 @@ END
 sub preseed_create ($$;@) {
     my ($ho, $sfx, %xopts) = @_;
 
-    my $authkeys_url= create_webfile($ho, "authkeys$sfx", authorized_keys());
-
-    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key.pub";
-    my $hostkey= get_filecontents($hostkeyfile);
-    chomp($hostkey); $hostkey.="\n";
-    my $knownhosts= '';
-
     my $disk= $xopts{DiskDevice} || '/dev/sda';
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
     my $d_i= $ho->{Tftp}{Path}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.
 	$c{TftpDiVersion}.'-'.$ho->{Suite};
 
-    my $hostsq= $dbh_tests->prepare(<<END);
-        SELECT val FROM runvars
-         WHERE flight=? AND name LIKE '%host'
-         GROUP BY val
-END
-    $hostsq->execute($flight);
-    while (my ($node) = $hostsq->fetchrow_array()) {
-        my $longname= "$node.$c{TestHostDomain}";
-        my (@hostent)= gethostbyname($longname);
-        if (!@hostent) {
-            logm("skipping host key for nonexistent host $longname");
-            next;
-        }
-        my $specs= join ',', $longname, $node, map {
-            join '.', unpack 'W4', $_;
-        } @hostent[4..$#hostent];
-        logm("adding host key for $specs");
-        $knownhosts.= "$specs ".$hostkey;
-    }
-    $hostsq->finish();
-
-    $knownhosts.= "localhost,127.0.0.1 ".$hostkey;
-    my $knownhosts_url= create_webfile($ho, "known_hosts$sfx", $knownhosts);
-
     my $overlays= '';
     my $create_overlay= sub {
         my ($srcdir, $tfilename) = @_;
@@ -740,6 +764,8 @@ ls -l /dev/sd*
 true
 END
 
+    preseed_ssh($ho, $sfx);
+
     preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
 set -ex
@@ -747,17 +773,6 @@ set -ex
 r=/target/root
 cd \$r
 
-umask 022
-mkdir .ssh
-wget -O .ssh/authorized_keys '$authkeys_url'
-wget -O .ssh/known_hosts     '$knownhosts_url'
-
-u=osstest
-h=/home/\$u
-mkdir /target\$h/.ssh
-cp .ssh/authorized_keys /target\$h/.ssh
-chroot /target chown -R \$u.\$u \$h/.ssh
-
 echo FANCYTTY=0 >> /target/etc/lsb-base-logging.sh
 
 $overlays
diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 95fce9a..02c142b 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -44,7 +44,6 @@ our $gho;
 sub preseed () {
 
     my $preseed_file = preseed_base($gho,'wheezy','',());
-    my $authkeys = join('\\n', split(/\n/, authorized_keys()));
 
     $preseed_file .= (<<END);
 d-i netcfg/get_hostname string $gn
@@ -74,10 +73,10 @@ d-i apt-setup/cdrom/set-first boolean false
 d-i preseed/late_command string \\
         in-target mkdir -p /boot/efi/EFI/boot; \\
         in-target cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/boot/bootx64.efi ;\\
-        in-target mkdir -p /root/.ssh; \\
-        in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
 END
 
+    preseed_ssh($ho,'');
+
     $preseed_file .= preseed_hook_cmds();
 
     return $preseed_file;
-- 
2.1.4

Re: [PATCH OSSTEST v4 05/25] Debian: refactor preseeding of .ssh directories

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v4 05/25] Debian: refactor preseeding of .ssh directories"):
> Causes known_hosts to be consistently created as well as ~osstest/.ssh
> to be consistently populated (it previsouly wasn't for HVM guests).

Most of this is code motion, AFAICT.  The difference is just that
preseed_ssh is now called from ts-debian-hvm-install, too ?

Why should preseed_ssh not be used by preseed_base ?

Ian.

Re: [PATCH OSSTEST v4 05/25] Debian: refactor preseeding of .ssh directories

[Ian Campbell]

On Thu, 2015-04-02 at 12:33 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v4 05/25] Debian: refactor preseeding of .ssh directories"):
> > Causes known_hosts to be consistently created as well as ~osstest/.ssh
> > to be consistently populated (it previsouly wasn't for HVM guests).
> 
> Most of this is code motion, AFAICT.  The difference is just that
> preseed_ssh is now called from ts-debian-hvm-install, too ?
> 
> Why should preseed_ssh not be used by preseed_base ?

Good question, I think it could be. At least I can't think of any sort
of preseedable thing we would want to install which doesn't want this.
I'll try it for next time.

preseed_base today is only concerned with creating the base text for the
pressed file, whereas preseed_ssh does more stuff like adding webfiles
and adding preseed hooks.

I take it you don't see that as a problem?

Ian.

[PATCH OSSTEST v4 06/25] Debian: Refactor installation of overlays, so it can be used for guests too

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: longtaox.pang@intel.com
---
v3: New patch
---
 Osstest/Debian.pm | 57 ++++++++++++++++++++++++++++---------------------------
 1 file changed, 29 insertions(+), 28 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index db89326..13cd147 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -38,7 +38,9 @@ BEGIN {
                       preseed_base
                       preseed_create
                       preseed_ssh
-                      preseed_hook_command preseed_hook_installscript preseed_hook_cmds
+                      preseed_hook_command preseed_hook_installscript
+                      preseed_hook_overlay
+                      preseed_hook_cmds
                       di_installcmdline_core
                       );
     %EXPORT_TAGS = ( );
@@ -700,26 +702,6 @@ sub preseed_create ($$;@) {
     my $d_i= $ho->{Tftp}{Path}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.
 	$c{TftpDiVersion}.'-'.$ho->{Suite};
 
-    my $overlays= '';
-    my $create_overlay= sub {
-        my ($srcdir, $tfilename) = @_;
-        my $url= create_webfile($ho, "$tfilename$sfx", sub {
-            my ($fh) = @_;
-            contents_make_cpio($fh, 'ustar', $srcdir);
-        });
-        $overlays .= <<END;
-wget -O overlay.tar '$url'
-cd /target
-tar xf \$r/overlay.tar
-cd \$r
-rm overlay.tar
-
-END
-    };
-
-    $create_overlay->('overlay',        'overlay.tar');
-    $create_overlay->($c{OverlayLocal}, 'overlay-local.tar');
-
     preseed_hook_installscript($ho, $sfx,
           '/lib/partman/init.d', '000override-parted-devices', <<END);
 #!/bin/sh
@@ -765,19 +747,14 @@ true
 END
 
     preseed_ssh($ho, $sfx);
+    preseed_hook_overlay($ho, $sfx, 'overlay', 'overlay.tar');
+    preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
 
     preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
 set -ex
 
-r=/target/root
-cd \$r
-
 echo FANCYTTY=0 >> /target/etc/lsb-base-logging.sh
-
-$overlays
-
-echo latecmd done.
 END
 
     my $dtbs = "$d_i/dtbs.tar.gz";
@@ -966,6 +943,30 @@ chmod +x '$installer_pathname'
 END
 }
 
+sub preseed_hook_overlay ($$$$) {
+    my ($ho, $sfx, $srcdir, $tfilename) = @_;
+    my $url= create_webfile($ho, "$tfilename$sfx", sub {
+        my ($fh) = @_;
+        contents_make_cpio($fh, 'ustar', $srcdir);
+    });
+    preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+r=/target/root
+cd \$r
+
+umask 022
+
+wget -O overlay.tar '$url'
+cd /target
+tar xf \$r/overlay.tar
+cd \$r
+rm overlay.tar
+
+END
+}
+
 sub preseed_hook_cmds () {
     my $preseed;
     foreach my $di_key (keys %preseed_cmds) {
-- 
2.1.4

Re: [PATCH OSSTEST v4 06/25] Debian: Refactor installation of overlays, so it can be used for guests too

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v4 06/25] Debian: Refactor installation of overlays, so it can be used for guests too"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

[PATCH OSSTEST v4 07/25] Debian: add preseed_create_guest helper

[Ian Campbell]

Creates a preseed file suitable for use in a PV guest

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Rebase, pass $ho to preseed_base
v3: Handle $xopts{ExtraPreseed} undefined in preseed_base
---
 Osstest/Debian.pm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 13cd147..993acc7 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -37,6 +37,7 @@ BEGIN {
                       %preseed_cmds
                       preseed_base
                       preseed_create
+                      preseed_create_guest
                       preseed_ssh
                       preseed_hook_command preseed_hook_installscript
                       preseed_hook_overlay
@@ -611,6 +612,9 @@ END
 sub preseed_base ($$$;@) {
     my ($ho,$suite,$extra_packages,%xopts) = @_;
 
+    $extra_packages ||= '';
+    $xopts{ExtraPreseed} ||= '';
+
     my $preseed = <<"END";
 d-i mirror/suite string $suite
 
@@ -693,6 +697,31 @@ END
     return $preseed;
 }
 
+sub preseed_create_guest ($$;@) {
+    my ($ho, $sfx, %xopts) = @_;
+
+    my $suite= $xopts{Suite} || $c{DebianSuite};
+
+    my $extra_packages;
+
+    my $preseed_file= preseed_base($ho, $suite, $extra_packages, %xopts);
+    $preseed_file.= (<<END);
+d-i     partman-auto/method             string regular
+d-i     partman-auto/choose_recipe \\
+                select All files in one partition (recommended for new users)
+
+d-i     grub-installer/bootdev          string /dev/xvda
+
+END
+
+    preseed_ssh($ho, $sfx);
+    preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
+
+    $preseed_file .= preseed_hook_cmds();
+
+    return create_webfile($ho, "preseed$sfx", $preseed_file);
+}
+
 sub preseed_create ($$;@) {
     my ($ho, $sfx, %xopts) = @_;
 
-- 
2.1.4

Re: [PATCH OSSTEST v4 07/25] Debian: add preseed_create_guest helper

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v4 07/25] Debian: add preseed_create_guest helper"):
> Creates a preseed file suitable for use in a PV guest
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
> v4: Rebase, pass $ho to preseed_base
> v3: Handle $xopts{ExtraPreseed} undefined in preseed_base
> ---
>  Osstest/Debian.pm | 29 +++++++++++++++++++++++++++++
>  1 file changed, 29 insertions(+)
> 
> diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
> index 13cd147..993acc7 100644
> --- a/Osstest/Debian.pm
> +++ b/Osstest/Debian.pm
> @@ -37,6 +37,7 @@ BEGIN {
>                        %preseed_cmds
>                        preseed_base
>                        preseed_create
> +                      preseed_create_guest
>                        preseed_ssh
>                        preseed_hook_command preseed_hook_installscript
>                        preseed_hook_overlay
> @@ -611,6 +612,9 @@ END
>  sub preseed_base ($$$;@) {
>      my ($ho,$suite,$extra_packages,%xopts) = @_;
>  
> +    $extra_packages ||= '';

This is a little odd.  You are changing preseed_base to tolerate an
unspecified $extra_packages, but it's still a mandatory argument.
Maybe you should mvoe the `;' in the sub prototype.

Or maybe you just wanted to write:
  +    my $extra_packages = '';
instead of:
> +    my $extra_packages;

> +    preseed_ssh($ho, $sfx);
> +    preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');

Perhaps the OverlayLocal should always be done, rather than having to
be explicitly specified by all the callers of preseed_base ?

Thanks,
Ian.

Re: [PATCH OSSTEST v4 07/25] Debian: add preseed_create_guest helper

[Ian Campbell]

On Thu, 2015-04-02 at 12:40 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v4 07/25] Debian: add preseed_create_guest helper"):
> > Creates a preseed file suitable for use in a PV guest
> > 
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> > v4: Rebase, pass $ho to preseed_base
> > v3: Handle $xopts{ExtraPreseed} undefined in preseed_base
> > ---
> >  Osstest/Debian.pm | 29 +++++++++++++++++++++++++++++
> >  1 file changed, 29 insertions(+)
> > 
> > diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
> > index 13cd147..993acc7 100644
> > --- a/Osstest/Debian.pm
> > +++ b/Osstest/Debian.pm
> > @@ -37,6 +37,7 @@ BEGIN {
> >                        %preseed_cmds
> >                        preseed_base
> >                        preseed_create
> > +                      preseed_create_guest
> >                        preseed_ssh
> >                        preseed_hook_command preseed_hook_installscript
> >                        preseed_hook_overlay
> > @@ -611,6 +612,9 @@ END
> >  sub preseed_base ($$$;@) {
> >      my ($ho,$suite,$extra_packages,%xopts) = @_;
> >  
> > +    $extra_packages ||= '';
> 
> This is a little odd.  You are changing preseed_base to tolerate an
> unspecified $extra_packages, but it's still a mandatory argument.
> Maybe you should mvoe the `;' in the sub prototype.
> 
> Or maybe you just wanted to write:
>   +    my $extra_packages = '';
> instead of:
> > +    my $extra_packages;

It looks like I actually do, I suspect that defaulting must be a
leftover.

> 
> > +    preseed_ssh($ho, $sfx);
> > +    preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
> 
> Perhaps the OverlayLocal should always be done, rather than having to
> be explicitly specified by all the callers of preseed_base ?

Having moved preseed_ssh to preseed_base this certainly makes sense.

Ian.

[PATCH OSSTEST v4 08/25] make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs()

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: New patch
---
 make-flight | 4 ----
 mfi-common  | 4 ++++
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/make-flight b/make-flight
index cc8ecdb..ec0a20c 100755
--- a/make-flight
+++ b/make-flight
@@ -36,10 +36,6 @@ defguestsuite=`getconfig GuestDebianSuite`
 
 if [ x$buildflight = x ]; then
 
-  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
-     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
-  fi
-
   create_build_jobs
 
 else
diff --git a/mfi-common b/mfi-common
index 16fc8c5..a9e966f 100644
--- a/mfi-common
+++ b/mfi-common
@@ -63,6 +63,10 @@ create_build_jobs () {
   local enable_ovmf
   local build_hostflags
 
+  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
+     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
+  fi
+
   for arch in ${BUILD_ARCHES- i386 amd64 armhf }; do
 
     if [ "x$arch" = xdisable ]; then continue; fi
-- 
2.1.4

Re: [PATCH OSSTEST v4 08/25] make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs()

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v4 08/25] make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs()"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

[PATCH OSSTEST v4 09/25] distros: add support for installing Debian PV guests via d-i, flight and jobs

[Ian Campbell]

This patch introduces ts-debian-di-install which can install Debian
from a netboot (PXE) debian installer image. By default it installs
from the d-i image used by osstest (using the special Xen PV guest
enabled flavour where necessary) but it can also install the current
release versions from Squeeze onwards (up to and including Sid) and
the d-i daily builds. This is controlled by runvars {Guest}_diver =
osstest|current and {Guest}_dist = squeeze|...|sid.  The resulting
guests boot the distro kernel using pygrub (pvgrub will follow).

The distros flights differ substantially from the existing flights.
Introduce make-distros-flight using the functionality previously
refactored into mfi-common. The new flight tests all versions of
Debian from Squeeze onward as an amd64, i386 and armhf guests (armhf
from Jessie onwards only) using the usual smoke tests.

Test names are suffixed -pygrub pending the addition of pvgrub
variants in a future commit.

Add the new cases to sg-run-job

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: use guest create
v3: $BUILD_LVEXTEND_MAX now handled in mfi-common
    Consolidate setting of ruvars
    Include $flight and $job in tmpdir name
    Use Osstest::Debian::di_installcmdline_core
    Document the usage of get_host_property on a guest object
    Correct ARM netboot paths
    Include bootloader in test name
       Should include -pv too?
    console= repetition for Jessie onwards.
    Wait for up to an hour for the install. I'd seen timeouts right at
    the end of the install with the previous value

fixup: distros: add support for installing Debian PV guests via d-i, flight and jobs
---
 Osstest/TestSupport.pm |   3 +
 make-distros-flight    |  98 +++++++++++++++++++++++++++++
 sg-run-job             |  11 ++++
 ts-debian-di-install   | 168 +++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 280 insertions(+)
 create mode 100755 make-distros-flight
 create mode 100755 ts-debian-di-install

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 8f67f7e..3cd8d22 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -906,8 +906,11 @@ sub propname_massage ($) {
     return $prop;
 }
 
+# It is fine to call this on a guest object too, in which case it will
+# always return $defval.
 sub get_host_property ($$;$) {
     my ($ho, $prop, $defval) = @_;
+    return $defval unless $ho->{Properties};
     my $val = $ho->{Properties}{propname_massage($prop)};
     return defined($val) ? $val : $defval;
 }
diff --git a/make-distros-flight b/make-distros-flight
new file mode 100755
index 0000000..5067342
--- /dev/null
+++ b/make-distros-flight
@@ -0,0 +1,98 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. cri-common
+. ap-common
+. mfi-common
+
+defsuite=`getconfig DebianSuite`
+defguestsuite=`getconfig GuestDebianSuite`
+
+if [ x$buildflight = x ]; then
+
+  WANT_XEND=false REVISION_LINUX_OLD=disable
+
+  create_build_jobs
+
+else
+
+  bfi=$buildflight.
+
+fi
+
+job_create_test_filter_callback () {
+  if [ "$xenarch" = "i386" ]; then return 1; fi
+  return 0
+}
+
+test_matrix_branch_filter_callback () {
+    :
+}
+
+test_do_one_netboot () {
+  job_create_test                                               \
+   test-$xenarch$kern-$dom0arch-$domU-$dist-netboot-pygrub      \
+    test-debian-di xl $xenarch $dom0arch                        \
+      kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
+      debian_arch=$domU                                         \
+      debian_dist=$dist                                         \
+      debian_method=netboot                                     \
+      debian_diver=current                                      \
+      all_hostflags=$most_hostflags
+}
+
+test_matrix_do_one () {
+  case ${xenarch} in
+  amd64) domUarches="amd64 i386";;
+  armhf) domUarches="armhf";;
+  esac
+
+  for domU in $domUarches ; do
+    for dist in squeeze wheezy jessie sid daily ; do
+      case ${domU}_${dist} in
+      armhf_squeeze) continue;; # No armhf in Squeeze
+      armhf_wheezy) continue;; # No armhf guest support in Wheezy
+      *) ;;
+      esac
+
+      test_do_one_netboot
+
+    done
+
+  done
+}
+
+test_matrix_iterate
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:
diff --git a/sg-run-job b/sg-run-job
index 0ec314f..284cd50 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -258,6 +258,17 @@ proc run-job/test-debian {} {
     test-guest debian
 }
 
+proc install-guest-debian-di {} {
+    run-ts . = ts-debian-di-install
+    run-ts . = ts-guest-start + debian
+}
+
+proc need-hosts/test-debian-di {} { return host }
+proc run-job/test-debian-di {} {
+    install-guest-debian-di
+    test-guest debian
+}
+
 proc need-hosts/test-freebsd {} { return host }
 proc run-job/test-freebsd {} {
     run-ts . = ts-freebsd-install
diff --git a/ts-debian-di-install b/ts-debian-di-install
new file mode 100755
index 0000000..a829185
--- /dev/null
+++ b/ts-debian-di-install
@@ -0,0 +1,168 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+$gn ||= 'debian';
+
+our $ho= selecthost($whhost);
+
+our $ram_mb=    512;
+our $disk_mb= 10000;
+
+our $guesthost= "$gn.guest.osstest";
+our $gho;
+
+sub prep () {
+    target_install_packages_norec($ho, qw(lvm2));
+
+    $gho= prepareguest($ho, $gn, $guesthost, 22,
+                       $disk_mb, 40);
+
+    prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
+
+    target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+}
+
+sub setup_netboot($$$)
+{
+    my ($didir, $arch, $suite) = @_;
+
+    my $di_ver= $r{"$gho->{Guest}_diver"} || "osstest";
+
+    my ($kernel,$initrd);
+
+    if ( $di_ver eq "osstest" ) {
+	my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.$c{TftpDiVersion}.'-'.$ho->{Suite};
+
+	$kernel = "$di_path/vmlinuz-xen";
+	$initrd = "$di_path/initrd.gz-xen";
+
+	target_putfile_root($ho, 60, $kernel, "$didir/kernel_${suite}_${arch}");
+	target_putfile_root($ho, 60, $initrd, "$didir/initrd_${suite}_${arch}");
+
+    } else {
+	my $mirror = "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
+
+	my $di_url = $suite eq "daily" ?
+	    "http://d-i.debian.org/daily-images/$arch/daily/netboot" :
+	    "$mirror/dists/$suite/main/installer-$arch/$di_ver/images/netboot";
+
+	$di_url .= "/xen" if $arch =~ m/amd64|i386/;
+	$di_url .= "/debian-installer/arm64" if $arch =~ /arm64/;
+
+	$kernel = "$di_url/vmlinuz";
+	$initrd = "$di_url/initrd.gz";
+
+	target_fetchurl($ho, $kernel, "$didir/kernel_${suite}_${arch}");
+	target_fetchurl($ho, $initrd, "$didir/initrd_${suite}_${arch}");
+    }
+
+    store_runvar("$gho->{Guest}_netboot_kernel", $kernel);
+    store_runvar("$gho->{Guest}_netboot_initrd", $initrd);
+
+    return <<END;
+kernel      = "$didir/kernel_${suite}_${arch}"
+ramdisk     = "$didir/initrd_${suite}_${arch}"
+END
+}
+sub ginstall () {
+    my $arch= $r{"$gho->{Guest}_arch"};
+    my $method= $r{"$gho->{Guest}_method"};
+
+    my $tmpdir= "/root/$flight-$job-di";
+    target_cmd_root($ho, <<END);
+rm -rf $tmpdir
+mkdir $tmpdir
+END
+
+    my ($method_cfg, $ps_url, $extra_disk);
+
+    if ( $method eq "netboot" )
+    {
+	my $suite= $r{"$gho->{Guest}_dist"};
+	logm("$method $suite/$arch");
+
+	$method_cfg = setup_netboot($tmpdir, $arch, $suite);
+
+	$suite = "sid" if $suite eq "daily";
+
+	$ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+
+	$extra_disk = "";
+    }
+    else
+    {
+	die "$method";
+    }
+
+    my @cmdline = ();
+    push @cmdline, "debian-installer/exit/always_halt=true";
+    push @cmdline, "domain=$c{TestHostDomain}";
+    push @cmdline, "console=hvc0";
+    push @cmdline, di_installcmdline_core($gho, $ps_url);
+    push @cmdline, "--";
+    # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007 for
+    # why this is repeated.
+    push @cmdline, "console=hvc0";
+
+    my $cmdline = join(" ", @cmdline);
+
+    my %install_xopts = (
+	OnPowerOff => "preserve"
+    );
+
+    prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
+$method_cfg
+extra       = "$cmdline"
+#
+disk        = [
+            $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+            ]
+END
+
+    guest_create($gho);
+
+    guest_checkrunning($ho, $gho) or die "$gho->{Name} not running";
+
+    guest_await_shutdown($ho,$gho,3600);
+    guest_destroy($gho);
+
+    my $blcfg = <<END;
+bootloader = "pygrub"
+END
+
+    prepareguest_part_xencfg($ho, $gho, $ram_mb, {}, <<END);
+$blcfg
+#
+disk        = [
+            'phy:$gho->{Lvdev},xvda,w'
+            ]
+END
+    return;
+}
+
+prep();
+ginstall();
-- 
2.1.4

[PATCH OSSTEST v4 10/25] distros: support booting Debian PV (d-i installed) guests with pvgrub.

[Ian Campbell]

This requires the use of the pv-grub-menu package which is in Jessie
onwards.  (it is in wheezy-backports which is the subject of a
subsequent patch).

The bootloader to use is specified via a runvar {Guest}_bootloader.

Adjust make-distros-flight to use pvgrub for some subset of i386 and
amd64 guests to get coverage.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Define and use arch_debian2xen and arch_xen2debian
    Avoid pv-grub-x86_64.gz on i386 dom0, we don't built it there.
    Fiddle with py vs pv grub stripy a bit.
---
 Osstest.pm           |  7 +++++++
 Osstest/Debian.pm    |  2 +-
 make-distros-flight  | 21 ++++++++++++++++++++-
 ts-debian-di-install | 11 +++++++++--
 4 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 7989129..c0e3ee7 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -38,6 +38,7 @@ BEGIN {
                       db_begin_work
                       ensuredir get_filecontents_core_quiet system_checked
                       nonempty visible_undef show_abs_time
+                      %arch_debian2xen %arch_xen2debian
                       );
     %EXPORT_TAGS = ( );
 
@@ -49,6 +50,12 @@ our $mjobdb;
 
 our $dbh_tests;
 
+our %arch_debian2xen = qw(i386 x86_32
+			  amd64 x86_64
+			  armhf armhf);
+our %arch_xen2debian;
+$arch_xen2debian{$arch_debian2xen{$_}} = $_ foreach keys %arch_debian2xen;
+
 #---------- static default config settings ----------
 
 our %c = qw(
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 993acc7..cbc7172 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -702,7 +702,7 @@ sub preseed_create_guest ($$;@) {
 
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
-    my $extra_packages;
+    my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
 
     my $preseed_file= preseed_base($ho, $suite, $extra_packages, %xopts);
     $preseed_file.= (<<END);
diff --git a/make-distros-flight b/make-distros-flight
index 5067342..80df21b 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -55,14 +55,33 @@ test_matrix_branch_filter_callback () {
 }
 
 test_do_one_netboot () {
+  stripy bootloader pvgrub pygrub \
+    "$xenarch" "amd64" \
+    "$dom0arch" "i386" \
+    "$domU" "amd64" \
+    "$dist" "sid"
+
+  case ${dom0arch}_${domU}_${dist} in
+    arm*_arm*_*) bootloader="pygrub";; # no pvgrub for arm
+
+    # Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
+    # workedaround in Jessie+ with pv-grub-menu package.
+    *_squeeze) bootloader="pygrub";;
+    *_wheezy) bootloader="pygrub";;
+
+    # pv-grub-x86_64.gz is not built by 32-bit dom0 userspace build.
+    i386_amd64_*) bootloader="pygrub";;
+  esac
+
   job_create_test                                               \
-   test-$xenarch$kern-$dom0arch-$domU-$dist-netboot-pygrub      \
+   test-$xenarch$kern-$dom0arch-$domU-$dist-netboot-$bootloader \
     test-debian-di xl $xenarch $dom0arch                        \
       kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
       debian_arch=$domU                                         \
       debian_dist=$dist                                         \
       debian_method=netboot                                     \
       debian_diver=current                                      \
+      debian_bootloader=$bootloader                             \
       all_hostflags=$most_hostflags
 }
 
diff --git a/ts-debian-di-install b/ts-debian-di-install
index a829185..73cb43d 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -93,6 +93,8 @@ sub ginstall () {
     my $method= $r{"$gho->{Guest}_method"};
 
     my $tmpdir= "/root/$flight-$job-di";
+    my $bl= $r{"$gho->{Guest}_bootloader"};
+
     target_cmd_root($ho, <<END);
 rm -rf $tmpdir
 mkdir $tmpdir
@@ -109,7 +111,7 @@ END
 
 	$suite = "sid" if $suite eq "daily";
 
-	$ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+	$ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
 
 	$extra_disk = "";
     }
@@ -150,7 +152,12 @@ END
     guest_await_shutdown($ho,$gho,3600);
     guest_destroy($gho);
 
-    my $blcfg = <<END;
+    my $xenarch = $arch_debian2xen{$arch};
+    my $pvgrub = "/usr/local/lib/xen/boot/pv-grub-$xenarch.gz";
+    my $blcfg = $bl eq "pvgrub" ? <<END : <<END;
+kernel = "$pvgrub"
+extra = "(hd0,0)/boot/grub/menu.lst"
+END
 bootloader = "pygrub"
 END
 
-- 
2.1.4

[PATCH OSSTEST v4 11/25] distros: Support pvgrub for Wheezy too.

[Ian Campbell]

This requires us to install pv-grub-menu from backports, which we do
using a late_command.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3:
  - Remove spurious () from <<(END) (and the prexisting "" too)
  - Remove $xopts{EnableBackports} and automatically handle the need to add
    backports in preseed_base.
  - Install via late_command not apt-setup, since the former has
    issues, hence subject drops "attempt to..."
---
 Osstest/Debian.pm   | 44 +++++++++++++++++++++++++++++++++++++++++++-
 make-distros-flight |  4 ++--
 2 files changed, 45 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index cbc7172..b37540e 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -689,6 +689,17 @@ END
 d-i clock-setup/ntp-server string $ntpserver
 END
 
+    # deb http://ftp.debian.org/debian/ wheezy-backports main
+    $preseed .= <<END if $extra_packages =~ m#\b/$suite-backports\b#;
+d-i apt-setup/local0/repository string \\
+    http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
+d-i apt-setup/local0/comment string $suite backports
+END
+# Jessie onwards...
+#    $preseed .= <<END if $extra_packages =~ m#\b/$suite-backports\b#;
+#d-i apt-setup/services-select multiselect security, updates, backports
+#END
+
     $preseed .= <<"END";
 
 ### END OF DEBIAN PRESEED BASE
@@ -702,7 +713,38 @@ sub preseed_create_guest ($$;@) {
 
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
-    my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
+    my $extra_packages = "";
+    if ($xopts{PvMenuLst}) {
+        if ($suite =~ m/wheezy/) {
+            # pv-grub-menu/wheezy-backports + using apt-setup to add
+            # backports results in iproute, ifupdown and
+            # isc-dhcp-client getting removed because tasksel's
+            # invocation of apt-get install somehow decides the
+            # iproute2 from wheezy-backports is a thing it wants to
+            # install. So instead lets fake it with a late command...
+            #
+            # This also has the bonus of working round an issue with
+            # 1.2.1~bpo70+1 which created an invalid menu.lst using
+            # "root(/dev/xvda,0)" which pvgrub cannot parse because
+            # the Grub device.map isn't present at pkgsel/include time
+            # but it is by late_command time. This was fixed by
+            # version 1.3 which is in Jessie onwards.
+            preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+(
+    echo
+    echo \\\# $suite backports
+    echo deb http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
+) >> /target/etc/apt/sources.list
+in-target apt-get update
+in-target apt-get install -y -t wheezy-backports pv-grub-menu
+END
+        } else {
+            $extra_packages = "pv-grub-menu";
+        }
+    }
 
     my $preseed_file= preseed_base($ho, $suite, $extra_packages, %xopts);
     $preseed_file.= (<<END);
diff --git a/make-distros-flight b/make-distros-flight
index 80df21b..97df89a 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -65,9 +65,9 @@ test_do_one_netboot () {
     arm*_arm*_*) bootloader="pygrub";; # no pvgrub for arm
 
     # Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
-    # workedaround in Jessie+ with pv-grub-menu package.
+    # workedaround in Wheezy+ with pv-grub-menu package (backports in Wheezy,
+    # in Jessie+ main).
     *_squeeze) bootloader="pygrub";;
-    *_wheezy) bootloader="pygrub";;
 
     # pv-grub-x86_64.gz is not built by 32-bit dom0 userspace build.
     i386_amd64_*) bootloader="pygrub";;
-- 
2.1.4

[PATCH OSSTEST v4 12/25] distros: support PV guest install from Debian netinst media.

[Ian Campbell]

The netinst media are iso images containing a base Debian install and
some (image size dependent) additional tasks.

On x86 the "multiarch" iso flavour contains a Xen capable kernel for
both i386 and amd64 so use that.

This adds support for two classes of ISO, the CD sized ones which are
built nightly and the DVD sized ones which are built weekly.

The images are downloaded using jigdo which sources the majority of
the data from a local Debian mirror, for this reason I have not
worried about the fact that the i386 and amd64 tests are downloading
the same thing (adding a specific download job would require finding
up to 4GB of scratch space for each flight).

The ISOs booted using pygrub which can extract the kernel and initrd
from a ISO image. The resulting guests are also booted with pygrub
since the pv-grub-menu package is not available on the ISO images and
we have pvgrub coverage from the netboot tests.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Support http_proxy in jigdo download
v3:
  - Indent jigdo-lite script line to improve readability
  - Wrap bootloader_args
  - Fetch URL on target, to get a timeout. Use wget since that is arranged to
    be present.
  - include bootloader in test name.
---
 Osstest/Debian.pm    |  7 ++++--
 make-distros-flight  | 23 ++++++++++++++++++
 ts-debian-di-install | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 95 insertions(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index b37540e..831290c 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -616,8 +616,6 @@ sub preseed_base ($$$;@) {
     $xopts{ExtraPreseed} ||= '';
 
     my $preseed = <<"END";
-d-i mirror/suite string $suite
-
 d-i debian-installer/locale string en_GB
 d-i console-keymaps-at/keymap select gb
 d-i keyboard-configuration/xkb-keymap string en_GB
@@ -689,6 +687,11 @@ END
 d-i clock-setup/ntp-server string $ntpserver
 END
 
+    # For CDROM the suite is part of the image
+    $preseed .= <<END unless $xopts{CDROM};
+d-i mirror/suite string $suite
+END
+
     # deb http://ftp.debian.org/debian/ wheezy-backports main
     $preseed .= <<END if $extra_packages =~ m#\b/$suite-backports\b#;
 d-i apt-setup/local0/repository string \\
diff --git a/make-distros-flight b/make-distros-flight
index 97df89a..e3ec8f7 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -85,6 +85,20 @@ test_do_one_netboot () {
       all_hostflags=$most_hostflags
 }
 
+test_do_one_netinst () {
+  # Always pygrub since no pv-grub-menu on CD
+  job_create_test                                               \
+   test-$xenarch$kern-$dom0arch-$domU-$cd-netinst-pygrub        \
+    test-debian-di xl $xenarch $dom0arch                        \
+      kernbuildjob=${bfi}build-$dom0arch-$kernbuild             \
+      debian_arch=$domU                                         \
+      debian_cd=$cd                                             \
+      debian_method=netinst                                     \
+      debian_bootloader=pygrub                                  \
+      all_hostflags=$most_hostflags
+
+}
+
 test_matrix_do_one () {
   case ${xenarch} in
   amd64) domUarches="amd64 i386";;
@@ -103,6 +117,15 @@ test_matrix_do_one () {
 
     done
 
+    for cd in current weekly ; do
+      case ${domU}_${dist} in
+      armhf_*) continue;; # No iso targets for armhf
+      *) ;;
+      esac
+
+      test_do_one_netinst
+
+    done
   done
 }
 
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 73cb43d..62e9bb5 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -46,6 +46,63 @@ sub prep () {
     target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
 }
 
+sub setup_netinst($$$)
+{
+    my ($didir, $arch, $cd) = @_;
+    my %arch_props = (
+	amd64 => { PathArch => "multi-arch", FileArch => "amd64-i386", IsoPath => "/install.amd/xen" },
+	i386  => { PathArch => "multi-arch", FileArch => "amd64-i386", IsoPath => "/install.386/xen" },
+	armhf => { PathArch => "armhf",      FileArch => "armhf",      IsoPath => "/install.armhf" }
+    );
+    my $props = $arch_props{$arch} or die "Unknown arch $arch";
+
+    target_install_packages($ho, qw(jigdo-file));
+
+    my $baseurl = $cd eq "current" ?
+      "http://cdimage.debian.org/debian-cd/current/$props->{PathArch}/jigdo-cd" :
+      "http://cdimage.debian.org/cdimage/weekly-builds/$props->{PathArch}/jigdo-cd";
+
+    my $filebase;
+
+    # Use the MD5SUMs file as an index
+    logm("Fetch index from $baseurl/MD5SUMS");
+    my $idx = target_cmd_output_root($ho, "wget --quiet -O - $baseurl/MD5SUMS");
+    foreach (split /\n/, $idx) {
+	m/^[0-9a-f]{32}  (debian-.*-$props->{FileArch}-netinst)\.iso$/ or next;
+	$filebase = $1;
+	last;
+    }
+
+    die unless $filebase;
+
+    logm("Downloading $baseurl/$filebase.jigdo");
+    my $netinst_jigdo = "$baseurl/$filebase.jigdo";
+    # Jigdo uses wget internally, and so obeys $http_proxy. This seems
+    # simpler than the advice at
+    # https://www.debian.org/CD/jigdo-cd/#faq which is to edit
+    # ~/.jigdo-lite and change the wgetOpts setting.
+    my $useproxy = $c{DebianMirrorProxy} // $c{HttpProxy};
+    my $cmd = '';
+    $cmd .= <<END if $useproxy;
+        export http_proxy=$useproxy
+END
+    $cmd .= <<END;
+        cd $didir && jigdo-lite --noask $netinst_jigdo
+END
+    # Jigdo seems to use /etc/apt/sources.list or something, so this
+    # just works using the already configured mirror without
+    # additional configuration, which is good because there doesn't
+    # seem to be any support for such things, at least in Squeeze.
+    target_cmd_root($ho, $cmd, 3600);
+    store_runvar("$gho->{Guest}_netinst_jigdo", $netinst_jigdo);
+
+    return (<<END, "\"file:$didir/$filebase.iso,xvdd:cdrom,r\",");
+bootloader = "pygrub"
+bootloader_args = ["--kernel=$props->{IsoPath}/vmlinuz",
+                   "--ramdisk=$props->{IsoPath}/initrd.gz"]
+END
+}
+
 sub setup_netboot($$$)
 {
     my ($didir, $arch, $suite) = @_;
@@ -115,6 +172,16 @@ END
 
 	$extra_disk = "";
     }
+    elsif ($method eq "netinst" )
+    {
+	my $cd = $r{"$gho->{Guest}_cd"};
+
+	logm("$method $cd/$arch");
+
+	($method_cfg,$extra_disk) = setup_netinst($tmpdir, $arch, $cd);
+
+	$ps_url = preseed_create_guest($gho, '', CDROM=>1);
+    }
     else
     {
 	die "$method";
-- 
2.1.4

[PATCH OSSTEST v4 13/25] Test pygrub and pvgrub on the regular flights

[Ian Campbell]

Since we now have the ability to test these drop one of each of
pygrub, pvgrub-32 and pvgrub-64 into the standard flights. Omitting
the {Guest}_diver runvar causes ts-debian-di-install to use the d-i
images in the location configured via TftpDiVersion, so they are
Version Controlled along with the d-i version used for the host.

This adds three new jobs:
test-amd64-amd64-amd64-pvgrub:
    all_hostflags             arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
    arch                      amd64
    buildjob                  build-amd64
    debian_arch               amd64
    debian_bootloader         pvgrub
    debian_dist               wheezy
    debian_method             netboot
    kernbuildjob              build-amd64-pvops
    kernkind                  pvops
    toolstack                 xl
    xenbuildjob               build-amd64
test-amd64-amd64-i386-pvgrub:
    all_hostflags             arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
    arch                      amd64
    buildjob                  build-amd64
    debian_arch               i386
    debian_bootloader         pvgrub
    debian_dist               wheezy
    debian_method             netboot
    kernbuildjob              build-amd64-pvops
    kernkind                  pvops
    toolstack                 xl
    xenbuildjob               build-amd64
test-amd64-amd64-pygrub:
    all_hostflags             arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
    arch                      amd64
    buildjob                  build-amd64
    debian_arch               amd64
    debian_bootloader         pygrub
    debian_dist               wheezy
    debian_method             netboot
    kernbuildjob              build-amd64-pvops
    kernkind                  pvops
    toolstack                 xl
    xenbuildjob               build-amd64

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: added runvar details
---
 make-flight | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/make-flight b/make-flight
index ec0a20c..ca48b81 100755
--- a/make-flight
+++ b/make-flight
@@ -303,6 +303,42 @@ do_passthrough_tests () {
   done
 }
 
+do_pygrub_tests () {
+  if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+    return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-pygrub   \
+    test-debian-di xl $xenarch $dom0arch                \
+      debian_arch=amd64                                 \
+      debian_dist=$guestsuite                           \
+      debian_method=netboot                             \
+      debian_bootloader=pygrub                          \
+      all_hostflags=$most_hostflags
+}
+
+do_pvgrub_tests () {
+  if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+    return
+  fi
+
+  job_create_test test-$xenarch$kern-$dom0arch-amd64-pvgrub     \
+    test-debian-di xl $xenarch $dom0arch                        \
+      debian_arch=amd64                                         \
+      debian_dist=$guestsuite                                   \
+      debian_method=netboot                                     \
+      debian_bootloader=pvgrub                                  \
+      all_hostflags=$most_hostflags                             \
+
+  job_create_test test-$xenarch$kern-$dom0arch-i386-pvgrub      \
+    test-debian-di xl $xenarch $dom0arch                        \
+      debian_arch=i386                                          \
+      debian_dist=$guestsuite                                   \
+      debian_method=netboot                                     \
+      debian_bootloader=pvgrub                                  \
+      all_hostflags=$most_hostflags
+}
+
 do_pv_debian_tests () {
   xsms=$(xenbranch_xsm_variants)
 
@@ -427,6 +463,9 @@ test_matrix_do_one () {
 
   fi
   #do_passthrough_tests
+
+  do_pygrub_tests
+  do_pvgrub_tests
 }
 
 test_matrix_iterate
-- 
2.1.4

[PATCH OSSTEST v4 14/25] distros: add branch infrastructure

[Ian Campbell]

Since the distro nightlies are not version controlled we cannot use
the usual mechanisms for detecting regressions. Special case things
appropriately. We use an OLD_REVISION of "flight-NNN" to signify that
the old revision is another flight and not a tree revision.

A grep over $NEW_REVISION needed adjusting since NEW_REVISION is empty
in this mode, leading to "grep <filename>" which hangs waiting for
stdin.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3:
  Handle within cr-daily-branch, since ap-fetch-version* don't make sense for
  a branch such as this.
---
 cr-daily-branch | 36 ++++++++++++++++++++++++++++--------
 cri-common      |  1 +
 2 files changed, 29 insertions(+), 8 deletions(-)

diff --git a/cr-daily-branch b/cr-daily-branch
index c7d1071..55afadf 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -68,23 +68,34 @@ fetch_version () {
 	printf '%s\n' "$fetch_version_result"
 }
 
-treeurl=`./ap-print-url $branch`
+case $branch in
+    distros)
+	treeurl=none;;
+    *)
+	treeurl=`./ap-print-url $branch`;;
+esac
 
 force_baseline=false
 skipidentical=true
 wantpush=$OSSTEST_PUSH
 
-if [ "x$OLD_REVISION" = x ]; then
-        OLD_REVISION="`./ap-fetch-version-old $branch`"
-        export OLD_REVISION
-fi
-
 check_tested () {
 	./sg-check-tested --debug --branch=$branch \
 	  --blessings=${DAILY_BRANCH_TESTED_BLESSING:-$OSSTEST_BLESSING} \
 	  "$@"
 }
 
+if [ "x$OLD_REVISION" = x ]; then
+    case $branch in
+	distros)
+	    OSSTEST_NO_BASELINE=y
+	    OLD_REVISION=flight-`check_tested`
+	    ;;
+	*) OLD_REVISION="`./ap-fetch-version-old $branch`";;
+    esac
+    export OLD_REVISION
+fi
+
 if [ "x$OSSTEST_NO_BASELINE" != xy ] ; then
 	testedflight=`check_tested --revision-$tree="$OLD_REVISION"`
 
@@ -227,6 +238,11 @@ if [ "x$OLD_REVISION" = xdetermine-late ]; then
 	OLD_REVISION="`./ap-fetch-version-baseline-late $branch $NEW_REVISION`"
 fi
 
+case $branch in
+distros) makeflight=./make-distros-flight ;;
+*)       makeflight=./make-flight ;;
+esac
+
 if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
         wantpush=false
 	for checkbranch in x $BRANCHES_ALWAYS; do
@@ -241,7 +257,7 @@ if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
 fi
 
 $DAILY_BRANCH_PREMAKE_HOOK
-flight=`./make-flight $branch $xenbranch $OSSTEST_BLESSING "$@"`
+flight=`$makeflight $branch $xenbranch $OSSTEST_BLESSING "$@"`
 $DAILY_BRANCH_POSTMAKE_HOOK
 
 heading=tmp/$flight.heading-info
@@ -261,6 +277,10 @@ fi
 revlog=tmp/$flight.revision-log
 
 case "$NEW_REVISION/$OLD_REVISION" in
+/flight-[0-9]*)
+	echo >&2 "SGR COMPARISON AGAINST ${OLD_REVISION}"
+	sgr_args+=" --that-flight=${OLD_REVISION#flight-}"
+	;;
 */*[^0-9a-f]* | *[^0-9a-f]*/*)
         echo >&2 "NO SGR COMPARISON badchar $NEW_REVISION/$OLD_REVISION"
         ;;
@@ -321,7 +341,7 @@ start_email $flight $branch "$sgr_args" "$subject_prefix"
 push=false
 if grep '^tolerable$' $mrof >/dev/null 2>&1; then push=$wantpush; fi
 if test -f $branch.force; then push=$OSSTEST_PUSH; fi
-if grep -xF $NEW_REVISION $branch.force-rev; then push=$OSSTEST_PUSH; fi
+if test -n "$NEW_REVISION" && grep -xF $NEW_REVISION $branch.force-rev; then push=$OSSTEST_PUSH; fi
 if test -f $branch.block; then push=false; fi
 
 if test -e $mrof && test -e $tree_bisect && ! grep '^broken' $mrof; then
diff --git a/cri-common b/cri-common
index ad44546..4bf7548 100644
--- a/cri-common
+++ b/cri-common
@@ -72,6 +72,7 @@ select_xenbranch () {
 	rumpuserxen)	      tree=rumpuserxen; xenbranch=xen-unstable ;;
 	seabios)		tree=seabios;	xenbranch=xen-unstable ;;
 	ovmf)			tree=ovmf;	xenbranch=xen-unstable ;;
+	distros)		tree=none;	xenbranch=xen-unstable ;;
 	osstest)		tree=osstest;	xenbranch=xen-unstable ;;
 	esac
 	if [ "x$tree" = xlinux ]; then
-- 
2.1.4

[PATCH OSSTEST v4 15/25] distros: Run a flight over the weekend.

[Ian Campbell]

Once a week should be sufficient for this test. It involves quite a
large number of jobs so schedule it to start early on Saturday so it
can run over the weekend when, in theory, things should be quieter.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 crontab | 1 +
 1 file changed, 1 insertion(+)

diff --git a/crontab b/crontab
index 5dc8675..1917ec4 100644
--- a/crontab
+++ b/crontab
@@ -6,5 +6,6 @@ MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
 18		9	* * 1,3,5	cd testing.git && BRANCHES=linux-next		./cr-for-branches branches -w "./cr-daily-branch --real"
 18		4	* * *		cd testing.git && BRANCHES='linux-linus linux-mingo-tip-master linux-3.0 libvirt rumpuserxen' ./cr-for-branches branches -w "./cr-daily-branch --real"
 6-59/15   	*	* * *		cd testing.git && EXTRA_BRANCHES='linux-linus linux-3.0 rumpuserxen libvirt' ./cr-for-branches bisects -w "./cr-try-bisect --real"
+46		7	* * 6		cd testing.git && BRANCHES=distros		./cr-for-branches branches -w "./cr-daily-branch --real"
 #8-59/5		*	* * *		cd bisects/adhoc.git &&	with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
 3		4	* * *		savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
-- 
2.1.4

[PATCH OSSTEST v4 16/25] Debian: Handle lack of bootloader support in d-i on ARM.

[Ian Campbell]

Debian doesn't currently know what bootloader to install in a Xen
guest on ARM. We install pv-grub-menu above which actually does what
we need, but the installer doesn't treat that as a "bootloader".

Most ARM platforms end up installing a u-boot boot.scr, based on a
platform whitelist. This doesn't seem appropriate for us. Grub is not
available for arm32. For arm64 we will eventually end up with in-guest
UEFI and therefore grub-efi and things will work normally. I'm not
sure what the answer is going to be for arm32.

This patch enables the workaround for Wheezy, Jessie and Sid,
post-Jessie should be enabled as we add them. (Pre-wheezy does not
support running as a Xen guest on ARM so we don't test them at all).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Handle sid too
v3: New
---
 Osstest/Debian.pm    | 14 ++++++++++++--
 ts-debian-di-install |  6 ++++--
 2 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 831290c..7f9deaa 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -711,8 +711,8 @@ END
     return $preseed;
 }
 
-sub preseed_create_guest ($$;@) {
-    my ($ho, $sfx, %xopts) = @_;
+sub preseed_create_guest ($$$;@) {
+    my ($ho, $arch, $sfx, %xopts) = @_;
 
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
@@ -759,6 +759,16 @@ d-i     grub-installer/bootdev          string /dev/xvda
 
 END
 
+    # Debian doesn't currently know what bootloader to install in a
+    # Xen guest on ARM. We install pv-grub-menu above which actually
+    # does what we need, but the installer doesn't treat that as a
+    # "bootloader".
+    logm("\$arch is $arch, \$suite is $suite");
+    $preseed_file.= (<<END) if $arch =~ /^arm/ && $suite =~ /wheezy|jessie|sid/;
+d-i     nobootloader/confirmation_common boolean true
+
+END
+
     preseed_ssh($ho, $sfx);
     preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
 
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 62e9bb5..56b6ad8 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -168,7 +168,9 @@ END
 
 	$suite = "sid" if $suite eq "daily";
 
-	$ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
+	$ps_url = preseed_create_guest($gho, $arch, '',
+				       Suite=>$suite,
+				       PvMenuLst=>($bl eq "pvgrub"));
 
 	$extra_disk = "";
     }
@@ -180,7 +182,7 @@ END
 
 	($method_cfg,$extra_disk) = setup_netinst($tmpdir, $arch, $cd);
 
-	$ps_url = preseed_create_guest($gho, '', CDROM=>1);
+	$ps_url = preseed_create_guest($gho, $arch, '', CDROM=>1);
     }
     else
     {
-- 
2.1.4

[PATCH OSSTEST v4 17/25] standalone: propagate result of command from with_logging

[Ian Campbell]

Such that the overall script gets the return code of the test/job
which was run.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 standalone | 1 +
 1 file changed, 1 insertion(+)

diff --git a/standalone b/standalone
index caf3fd5..3189e84 100755
--- a/standalone
+++ b/standalone
@@ -156,6 +156,7 @@ with_logging() {
     if [ $rc -ne 0 ] ; then
 	echo "FAILED rc=${rc}" >&2
     fi
+    return $rc
 }
 
 # other potential ops:
-- 
2.1.4

[PATCH OSSTEST v4 18/25] ts-debian-di-install: Refactor root_disk specification

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: new patch
---
 ts-debian-di-install | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/ts-debian-di-install b/ts-debian-di-install
index 56b6ad8..281e7cb 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -205,12 +205,14 @@ END
 	OnPowerOff => "preserve"
     );
 
+    my $root_disk = "'phy:$gho->{Lvdev},xvda,w'";
+
     prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
 $method_cfg
 extra       = "$cmdline"
 #
 disk        = [
-            $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+            $extra_disk $root_disk
             ]
 END
 
@@ -234,7 +236,7 @@ END
 $blcfg
 #
 disk        = [
-            'phy:$gho->{Lvdev},xvda,w'
+            $root_disk
             ]
 END
     return;
-- 
2.1.4

[PATCH OSSTEST v4 19/25] make-flight: refactor PV debian tests

[Ian Campbell]

No functional change, standalong-generate-dump-flight-runvars confirms
no change to the runvars.

Includes a hook which is not used yet, $recipe_sfx.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: new patch
---
 make-flight | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/make-flight b/make-flight
index ca48b81..909cebe 100755
--- a/make-flight
+++ b/make-flight
@@ -339,6 +339,17 @@ do_pvgrub_tests () {
       all_hostflags=$most_hostflags
 }
 
+do_pv_debian_test_one () {
+  testname=$1; shift
+  recipe_sfx=$1; shift
+  toolstack=$1; shift
+
+  job_create_test test-$xenarch$kern-$dom0arch-$testname        \
+     test-debian$recipe_sfx $toolstack                          \
+            $xenarch $dom0arch                                  \
+            $debian_runvars all_hostflags=$most_hostflags $@
+}
+
 do_pv_debian_tests () {
   xsms=$(xenbranch_xsm_variants)
 
@@ -354,20 +365,13 @@ do_pv_debian_tests () {
       suffix=${platform:+-$platform}
       hostflags=${most_hostflags}${platform:+,platform-$platform}
 
-      job_create_test test-$xenarch$kern-$dom0arch-xl$suffix   \
-                      test-debian xl                           \
-                      $xenarch $dom0arch                       \
-                      enable_xsm=$xsm                          \
-                      $debian_runvars all_hostflags=$hostflags
+      do_pv_debian_test_one xl$suffix '' xl enable_xsm=$xsm
+
     done
   done
 
   for xsm in $xsms ; do
-    job_create_test test-$xenarch$kern-$dom0arch-libvirt         \
-                    test-debian libvirt                          \
-                    $xenarch $dom0arch                           \
-                    enable_xsm=$xsm                              \
-                    $debian_runvars all_hostflags=$most_hostflags
+    do_pv_debian_test_one libvirt '' libvirt enable_xsm=$xsm
   done
 }
 
-- 
2.1.4

[PATCH OSSTEST v4 20/25] Add testing of non-LVM/phy disk backends.

[Ian Campbell]

xen-create-image makes this tricky to do since it is rather LVM
centric. Now that we have the ability to install from d-i it's
possible to arrange fairly easily that they use non-LVM disk backend
arrangements.

Here we add support to the test script and infra and create a bunch of
new jobs testing the cross product of {xl,libvirt} x {raw,qcow2,vhd}.

The test scripts are modified such that when constructing a domain
with a non-LVM diskfmt runvar:
 - the LVM device is slightly enlarged to account for file format
   headers (1M should be plenty).
 - the LVM device will have an ext3 filesystem created on it instead
   of being used as a phy device for the guest. Reusing the LVM volume
   in this way means we don't need to do more storage management in
   dom0 (i.e. arranging for / to be large enough, or managing a
   special "images" LV)
 - the relevant type of container is created within the filesystem
   using the appropriate tool.
 - New properties Disk{fmt,spec} are added to all $gho, containing
   the format used for the root disk and the xl diskspec to load it.
     - lvm backed guests use a xend/xm compatible spec, everything
       else uses the improved xl syntax which libvirt also supports.
       We won't test non-LVM on xend.
 - New properties Disk{mnt,img} are added to $gho which are not using
   LVM. These contain the mount point to use (configurable via
   OSSTEST_CONFIG and runvars) and the full path (including mount
   point) to the image itself.
 - When starting or stopping a guest we arrange for the filesystem to
   be (u)mounted.
     - The prepearation when starting a guest copes gracefully with
       the disk already being prepared.
     - Hooks are called from guest_create() and guest_destroy() to
       manipulate the disk as needed.

Using standalong-generate-dump-flight-runvars a representative set of
runvars is:

xen-unstable               test-amd64-amd64-xl-qcow2                 all_hostflags               arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
xen-unstable               test-amd64-amd64-xl-qcow2                 arch                        amd64
xen-unstable               test-amd64-amd64-xl-qcow2                 buildjob                    build-amd64
xen-unstable               test-amd64-amd64-xl-qcow2                 debian_arch                 amd64
xen-unstable               test-amd64-amd64-xl-qcow2                 debian_bootloader           pygrub
xen-unstable               test-amd64-amd64-xl-qcow2                 debian_diskfmt              qcow2
xen-unstable               test-amd64-amd64-xl-qcow2                 debian_dist                 wheezy
xen-unstable               test-amd64-amd64-xl-qcow2                 debian_method               netboot
xen-unstable               test-amd64-amd64-xl-qcow2                 kernbuildjob                build-amd64-pvops
xen-unstable               test-amd64-amd64-xl-qcow2                 kernkind                    pvops
xen-unstable               test-amd64-amd64-xl-qcow2                 toolstack                   xl
xen-unstable               test-amd64-amd64-xl-qcow2                 xenbuildjob                 build-amd64

Compared to test-amd64-amd64-pygrub (which is the most similar job) and
normalising the test name the difference is:
 xen-unstable               test-amd64-amd64-SUFFIX                   all_hostflags               arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
 xen-unstable               test-amd64-amd64-SUFFIX                   arch                        amd64
 xen-unstable               test-amd64-amd64-SUFFIX                   buildjob                    build-amd64
 xen-unstable               test-amd64-amd64-SUFFIX                   debian_arch                 amd64
 xen-unstable               test-amd64-amd64-SUFFIX                   debian_bootloader           pygrub
+xen-unstable               test-amd64-amd64-SUFFIX                   debian_diskfmt              qcow2
 xen-unstable               test-amd64-amd64-SUFFIX                   debian_dist                 wheezy
 xen-unstable               test-amd64-amd64-SUFFIX                   debian_method               netboot
 xen-unstable               test-amd64-amd64-SUFFIX                   kernbuildjob                build-amd64-pvops
 xen-unstable               test-amd64-amd64-SUFFIX                   kernkind                    pvops
 xen-unstable               test-amd64-amd64-SUFFIX                   toolstack                   xl
 xen-unstable               test-amd64-amd64-SUFFIX                   xenbuildjob                 build-amd64

These are added to the following flights:
  +libvirt
  +linux-3.0
  +linux-3.10
  +linux-3.14
  +linux-3.4
  +linux-arm-xen
  +linux-linus
  +linux-mingo-tip-master
  +linux-next
  +osstest
  +qemu-mainline
  +qemu-upstream-4.3-testing
  +qemu-upstream-4.4-testing
  +qemu-upstream-unstable
  +seabios
  +xen-4.0-testing
  +xen-4.1-testing
  +xen-4.2-testing
  +xen-4.3-testing
  +xen-4.4-testing
  +xen-unstable

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: new patch
---
 Osstest/TestSupport.pm | 95 ++++++++++++++++++++++++++++++++++++++++++++++++--
 make-flight            | 16 +++++++++
 ts-debian-di-install   | 10 ++----
 ts-guest-start         |  1 -
 4 files changed, 112 insertions(+), 10 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 3cd8d22..61fd7bc 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -89,7 +89,8 @@ BEGIN {
                       target_var target_var_prefix
                       selectguest prepareguest more_prepareguest_hvm
                       guest_var guest_var_commalist
-                      prepareguest_part_lvmdisk prepareguest_part_xencfg
+                      prepareguest_part_lvmdisk prepareguest_part_diskimg
+                      prepareguest_part_xencfg
                       guest_umount_lv guest_await guest_await_dhcp_tcp
                       guest_checkrunning guest_check_ip guest_find_ether
                       guest_find_domid guest_check_up guest_check_up_quick
@@ -97,6 +98,7 @@ BEGIN {
                       guest_await_shutdown guest_await_destroy guest_destroy
                       guest_vncsnapshot_begin guest_vncsnapshot_stash
 		      guest_check_remus_ok guest_editconfig
+                      guest_prepare_disk guest_unprepare_disk
                       host_involves_pcipassthrough host_get_pcipassthrough_devs
                       toolstack guest_create
 
@@ -1319,6 +1321,7 @@ sub selectguest ($$) {
     }
     logm("guest: using $gn on $gho->{Host}{Name}");
     guest_find_lv($gho);
+    guest_find_diskimg($gho);
     guest_find_ether($gho);
     guest_find_tcpcheckport($gho);
     dhcp_watch_setup($ho,$gho);
@@ -1334,6 +1337,22 @@ sub guest_find_lv ($) {
         ? '/dev/'.$gho->{Vg}.'/'.$gho->{Lv} : undef;
 }
 
+sub guest_find_diskimg($)
+{
+    my ($gho) = @_;
+    $gho->{Diskfmt} = $r{"$gho->{Guest}_diskfmt"} // "lvm";
+    $gho->{Diskspec} = "phy:$gho->{Lvdev},xvda,w";
+
+    return if $gho->{Diskfmt} eq "lvm";
+
+    my $mntroot = get_host_property($gho->{Host}, "DiskImageMount",
+			    $c{DiskImageMount} // "/var/lib/xen/images");
+
+    $gho->{Diskmnt} = "$mntroot/$gho->{Guest}";
+    $gho->{Diskimg} = "$gho->{Diskmnt}/disk.$gho->{Diskfmt}";
+    $gho->{Diskspec} = "format=$gho->{Diskfmt},vdev=xvda,target=$gho->{Diskimg}";
+}
+
 sub guest_find_ether ($) {
     my ($gho) = @_;
     $gho->{Ether}= $r{"$gho->{Guest}_ether"};
@@ -1373,6 +1392,7 @@ sub guest_destroy ($) {
     my ($gho) = @_;
     my $ho = $gho->{Host};
     toolstack($ho)->destroy($gho);
+    guest_unprepare_disk($gho);
 }
 
 sub guest_await_destroy ($$) {
@@ -1384,9 +1404,32 @@ sub guest_await_destroy ($$) {
 sub guest_create ($) {
     my ($gho) = @_;
     my $ho = $gho->{Host};
+    guest_prepare_disk($gho);
     toolstack($ho)->create($gho);
 }
 
+sub guest_prepare_disk ($) {
+    my ($gho) = @_;
+
+    guest_umount_lv($gho->{Host}, $gho);
+
+    return if $gho->{Diskfmt} eq "lvm";
+
+    target_cmd_root($gho->{Host}, <<END);
+mkdir -p $gho->{Diskmnt}
+mount $gho->{Lvdev} $gho->{Diskmnt};
+END
+}
+
+sub guest_unprepare_disk ($) {
+    my ($gho) = @_;
+    return if $gho->{Diskfmt} eq "lvm";
+    target_cmd_root($gho->{Host}, <<END);
+umount $gho->{Lvdev} || :
+END
+}
+
+
 
 sub target_choose_vg ($$) {
     my ($ho, $mbneeded) = @_;
@@ -1518,6 +1561,7 @@ sub prepareguest ($$$$$$) {
     }
 
     guest_find_lv($gho);
+    guest_find_diskimg($gho);
     guest_find_ether($gho);
     guest_find_tcpcheckport($gho);
     return $gho;
@@ -1528,7 +1572,54 @@ sub prepareguest_part_lvmdisk ($$$) {
     target_cmd_root($ho, "lvremove -f $gho->{Lvdev} ||:");
     target_cmd_root($ho, "lvcreate -L ${disk_mb}M -n $gho->{Lv} $gho->{Vg}");
     target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Lvdev} count=10");
-}    
+}
+
+sub make_vhd ($$$) {
+    my ($ho, $gho, $disk_mb) = @_;
+    target_cmd_root($ho, "vhd-util create -n $gho->{Rootimg} -s $disk_mb");
+}
+sub make_qcow2 ($$$) {
+    my ($ho, $gho, $disk_mb) = @_;
+    # upstream qemu's version. Seems preferable to qemu-xen-img from qemu-trad.
+    my $qemu_img = "/usr/local/lib/xen/bin/qemu-img";
+    target_cmd_root($ho, "$qemu_img create -f qcow2 $gho->{Rootimg} ${disk_mb}M");
+}
+sub make_raw ($$$) {
+    my ($ho, $gho, $disk_mb) = @_;
+    target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Rootimg} bs=1M count=${disk_mb}");
+}
+
+sub prepareguest_part_diskimg ($$$) {
+    my ($ho, $gho, $disk_mb) = @_;
+
+    my $diskfmt = $gho->{Diskfmt};
+    # Allow an extra megabyte for image format headers
+    my $disk_overhead = $diskfmt eq "lvm" ? 0 : 1;
+
+    logm("preparing guest disks in $diskfmt format");
+
+    target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+
+    prepareguest_part_lvmdisk($ho, $gho, $disk_mb + $disk_overhead);
+
+    if ($diskfmt ne "lvm") {
+
+	$gho->{Rootimg} = "$gho->{Diskmnt}/disk.$diskfmt";
+	$gho->{Rootcfg} = "format=$diskfmt,vdev=xvda,target=$gho->{Rootimg}";
+
+	target_cmd_root($ho, <<END);
+mkfs.ext3 $gho->{Lvdev}
+mkdir -p $gho->{Diskmnt}
+mount $gho->{Lvdev} $gho->{Diskmnt}
+END
+        no strict qw(refs);
+        &{"make_$diskfmt"}($ho, $gho, $disk_mb);
+
+	target_cmd_root($ho, <<END);
+umount $gho->{Lvdev}
+END
+    }
+}
 
 sub prepareguest_part_xencfg ($$$$$) {
     my ($ho, $gho, $ram_mb, $xopts, $cfgrest) = @_;
diff --git a/make-flight b/make-flight
index 909cebe..c923f21 100755
--- a/make-flight
+++ b/make-flight
@@ -373,6 +373,22 @@ do_pv_debian_tests () {
   for xsm in $xsms ; do
     do_pv_debian_test_one libvirt '' libvirt enable_xsm=$xsm
   done
+
+  for ts in xl libvirt ; do
+
+    for fmt in raw vhd qcow2 ; do
+
+      fmt_runvar="debian_diskfmt=$fmt"
+
+      do_pv_debian_test_one $ts-$fmt '-di' $ts  \
+          debian_arch=amd64                     \
+          debian_dist=$guestsuite               \
+          debian_method=netboot                 \
+          debian_bootloader=pygrub              \
+          $fmt_runvar
+
+    done
+  done
 }
 
 test_matrix_do_one () {
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 281e7cb..839cc27 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -41,9 +41,7 @@ sub prep () {
     $gho= prepareguest($ho, $gn, $guesthost, 22,
                        $disk_mb, 40);
 
-    prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
-
-    target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+    prepareguest_part_diskimg($ho, $gho, $disk_mb);
 }
 
 sub setup_netinst($$$)
@@ -205,14 +203,12 @@ END
 	OnPowerOff => "preserve"
     );
 
-    my $root_disk = "'phy:$gho->{Lvdev},xvda,w'";
-
     prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
 $method_cfg
 extra       = "$cmdline"
 #
 disk        = [
-            $extra_disk $root_disk
+            $extra_disk '$gho->{Diskspec}'
             ]
 END
 
@@ -236,7 +232,7 @@ END
 $blcfg
 #
 disk        = [
-            $root_disk
+            '$gho->{Diskspec}'
             ]
 END
     return;
diff --git a/ts-guest-start b/ts-guest-start
index 1aa9e69..a434720 100755
--- a/ts-guest-start
+++ b/ts-guest-start
@@ -25,7 +25,6 @@ tsreadconfig();
 our ($ho,$gho) = ts_get_host_guest(@ARGV);
 
 sub start () {
-    guest_umount_lv($ho, $gho);
     guest_create($gho);
 }
 
-- 
2.1.4

[PATCH OSSTEST v4 21/25] mfi-common: Allow make-*flight to filter the set of build jobs to include

[Ian Campbell]

By using the same job_create_build(_filter_callback) scheme used for
the test jobs.

Will be used in make-distros-flight.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-distros-flight |  4 ++++
 make-flight         |  4 ++++
 mfi-common          | 21 +++++++++++++++------
 3 files changed, 23 insertions(+), 6 deletions(-)

diff --git a/make-distros-flight b/make-distros-flight
index e3ec8f7..2724dd2 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -33,6 +33,10 @@ flight=`./cs-flight-create $blessing $branch`
 defsuite=`getconfig DebianSuite`
 defguestsuite=`getconfig GuestDebianSuite`
 
+job_create_build_filter_callback () {
+    :
+}
+
 if [ x$buildflight = x ]; then
 
   WANT_XEND=false REVISION_LINUX_OLD=disable
diff --git a/make-flight b/make-flight
index c923f21..ef1e422 100755
--- a/make-flight
+++ b/make-flight
@@ -34,6 +34,10 @@ flight=`./cs-flight-create $blessing $branch`
 defsuite=`getconfig DebianSuite`
 defguestsuite=`getconfig GuestDebianSuite`
 
+job_create_build_filter_callback () {
+    :
+}
+
 if [ x$buildflight = x ]; then
 
   create_build_jobs
diff --git a/mfi-common b/mfi-common
index a9e966f..a100afb 100644
--- a/mfi-common
+++ b/mfi-common
@@ -54,6 +54,15 @@ xenbranch_xsm_variants () {
     esac
 }
 
+job_create_build () {
+  job_create_build_filter_callback "$@" || return 0
+
+  local job=$1; shift
+  local recipe=$1; shift
+
+  ./cs-job-create $flight $job $recipe "$@"
+}
+
 create_build_jobs () {
 
   local arch
@@ -164,7 +173,7 @@ create_build_jobs () {
       else
         xsm_suffix=""
       fi
-      ./cs-job-create $flight build-$arch$xsm_suffix build                   \
+      job_create_build build-$arch$xsm_suffix build                          \
                 arch=$arch enable_xend=$build_defxend enable_ovmf=$enable_ovmf\
                 enable_xsm=$enable_xsm                                       \
         tree_qemu=$TREE_QEMU                                                 \
@@ -183,7 +192,7 @@ create_build_jobs () {
     done
 
     if [ $build_extraxend = "true" ] ; then
-    ./cs-job-create $flight build-$arch-xend build                           \
+    job_create_build build-$arch-xend build                                  \
                 arch=$arch enable_xend=true enable_ovmf=$enable_ovmf         \
         tree_qemu=$TREE_QEMU                                                 \
         tree_qemuu=$TREE_QEMU_UPSTREAM                                       \
@@ -196,7 +205,7 @@ create_build_jobs () {
                 revision_qemuu=$REVISION_QEMU_UPSTREAM
     fi
 
-    ./cs-job-create $flight build-$arch-pvops build-kern                     \
+    job_create_build build-$arch-pvops build-kern                            \
                 arch=$arch kconfighow=xen-enable-xen-config                  \
                 $RUNVARS $BUILD_RUNVARS $BUILD_LINUX_RUNVARS $arch_runvars   \
                 $suite_runvars                                               \
@@ -208,7 +217,7 @@ create_build_jobs () {
 
     if [ "x$REVISION_LIBVIRT" != xdisable ]; then
 
-    ./cs-job-create $flight build-$arch-libvirt build-libvirt                \
+    job_create_build build-$arch-libvirt build-libvirt                       \
                 arch=$arch                                                   \
         tree_xen=$TREE_XEN                                                   \
                 $RUNVARS $BUILD_RUNVARS $BUILD_LIBVIRT_RUNVARS $arch_runvars \
@@ -223,7 +232,7 @@ create_build_jobs () {
 
     case $arch in
     i386|amd64)
-    ./cs-job-create $flight build-$arch-rumpuserxen build-rumpuserxen        \
+    job_create_build build-$arch-rumpuserxen build-rumpuserxen               \
                 arch=$arch                                                   \
         tree_xen=$TREE_XEN                                                   \
                 $RUNVARS $BUILD_RUNVARS $BUILD_RUMPUSERXEN_RUNVARS $arch_runvars \
@@ -252,7 +261,7 @@ create_build_jobs () {
 
     if [ "x$REVISION_LINUX_OLD" != xdisable ]; then
 
-      ./cs-job-create $flight build-$arch-oldkern build-kern            \
+      job_create_build build-$arch-oldkern build-kern                   \
                 arch=$arch kconfighow=create-config-sh                  \
                 kimagefile=vmlinux                                      \
                 $RUNVARS $BUILD_RUNVARS $BUILD_LINUX_OLD_RUNVARS        \
-- 
2.1.4

[PATCH OSSTEST v4 22/25] make-distros-flight: don't bother building for XSM.

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-distros-flight | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/make-distros-flight b/make-distros-flight
index 2724dd2..62c527b 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -34,7 +34,10 @@ defsuite=`getconfig DebianSuite`
 defguestsuite=`getconfig GuestDebianSuite`
 
 job_create_build_filter_callback () {
-    :
+  case " $* " in
+    *" enable_xsm=true "*) return 1;;
+  esac
+  return 0
 }
 
 if [ x$buildflight = x ]; then
-- 
2.1.4

[PATCH OSSTEST v4 23/25] Debian.pm: Assume 100MB/s dd from /dev/zero when creating a raw disk image

[Ian Campbell]

In my local test:
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 80.8332 s, 130 MB/s

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/TestSupport.pm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 61fd7bc..391c9b2 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1586,7 +1586,9 @@ sub make_qcow2 ($$$) {
 }
 sub make_raw ($$$) {
     my ($ho, $gho, $disk_mb) = @_;
-    target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Rootimg} bs=1M count=${disk_mb}");
+    # In local tests this reported 130MB/s, so calculate a timeout assuming 100MB/s.
+    target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Rootimg} bs=1M count=${disk_mb}",
+	${disk_mb} / 100);
 }
 
 sub prepareguest_part_diskimg ($$$) {
-- 
2.1.4

[PATCH OSSTEST v4 24/25] distros: email only me on play flights

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 daily-cron-email-play--distros | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 daily-cron-email-play--distros

diff --git a/daily-cron-email-play--distros b/daily-cron-email-play--distros
new file mode 100644
index 0000000..0d82e65
--- /dev/null
+++ b/daily-cron-email-play--distros
@@ -0,0 +1 @@
+To: ian.campbell@citrix.com
-- 
2.1.4

[PATCH OSSTEST v4 25/25] ts-debian-di-install: Use ftp.debian.org directly

[Ian Campbell]

The local proxy seems to serve stale packages for Jessie etc, I blame
the intercepting cache on the way out of our network, similar to
b5f15136900d "mg-debian-installer-update: workaround caching proxies",
except it is between the apt-cache and the world not the osstest vm
and the world.

Since the netboot kernel+initrd are reasonably small, these flights
are infrequent and they are intended to test the current upstream
version I think this is tollerable.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-debian-di-install | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/ts-debian-di-install b/ts-debian-di-install
index 839cc27..469c1b6 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -119,7 +119,9 @@ sub setup_netboot($$$)
 	target_putfile_root($ho, 60, $initrd, "$didir/initrd_${suite}_${arch}");
 
     } else {
-	my $mirror = "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
+	#my $mirror = "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
+        # XXX local mirror seems to serve up stale files.
+	my $mirror = "http://ftp.debian.org/debian";
 
 	my $di_url = $suite eq "daily" ?
 	    "http://d-i.debian.org/daily-images/$arch/daily/netboot" :
-- 
2.1.4

Re: [PATCH OSSTEST v4 00/25] add distro domU testing flight

[Ian Jackson]

Ian Campbell writes ("[Xen-devel] [PATCH OSSTEST v4 00/25] add distro domU testing flight"):
> The failures mainly seem to be some sort of proxy issue returning old
> data (similar to what used to happen to mg-debian-installer-update),

Maybe we can set up an explicit proxy which actually works.  I guess
that might need negotiation with Citrix corporate IT.

Luckily the new colo is not affected by anything like this.

> On a small number I see "The installer failed to download a file from
> the mirror." messages, which might be the cache/proxy or might be an
> actual Debian issue.

Hrm.  I find these vague error messages exceptionally frustrating
(debootstrap has this problem too).

> The -raw tests all failed due to a timeout doing:
>         dd if=/dev/zero of=/var/lib/xen/images/debian/disk.raw bs=1M count=10000
>         
> I'm unclear if it timed out after 30 (as the code seems to say) or 60
> (as the logs seem to say) but either way something obviously needs
> adjusting. I just tried it on my local test box and it took 1m20s and
> reported 130 MB/s. I've appended an as yet untested patch which assumed
> 100 MB/s and calculates a timeout from that.

Do you intend to fold that patch in ?

> I could switch to using "count=1 seek=10000" to create a sparse file,
> which should be pretty much instantaneous, but allocating the whole
> backing file seemed like a good idea.

Right.

Thanks,
Ian.

Re: [PATCH OSSTEST v4 00/25] add distro domU testing flight

[Ian Campbell]

On Thu, 2015-04-02 at 12:21 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[Xen-devel] [PATCH OSSTEST v4 00/25] add distro domU testing flight"):
> > The failures mainly seem to be some sort of proxy issue returning old
> > data (similar to what used to happen to mg-debian-installer-update),
> 
> Maybe we can set up an explicit proxy which actually works.  I guess
> that might need negotiation with Citrix corporate IT.

I think it would, in order for the proxy to get past the intercepting
thing. I suspect the best answer we could expect would be "here is how
to use the intercepting thing directly", which is unlikely to help.

> Luckily the new colo is not affected by anything like this.

Indeed, and I was thinking that this flight would normally run there,
unless capacity concerns mean not for now.

> > On a small number I see "The installer failed to download a file from
> > the mirror." messages, which might be the cache/proxy or might be an
> > actual Debian issue.
> 
> Hrm.  I find these vague error messages exceptionally frustrating
> (debootstrap has this problem too).

If it happened in an interactive install you could get to a more
comprehensive logs, but that's not that useful for automation.

Perhaps a wishlist against d-i such that if it is preseeding and
DEBIAN_FRONTEND=text it will dump /var/log/syslog on fail? Or at least
an option to do so.

> > The -raw tests all failed due to a timeout doing:
> >         dd if=/dev/zero of=/var/lib/xen/images/debian/disk.raw bs=1M count=10000
> >         
> > I'm unclear if it timed out after 30 (as the code seems to say) or 60
> > (as the logs seem to say) but either way something obviously needs
> > adjusting. I just tried it on my local test box and it took 1m20s and
> > reported 130 MB/s. I've appended an as yet untested patch which assumed
> > 100 MB/s and calculates a timeout from that.
> 
> Do you intend to fold that patch in ?

Yes, I just left it separate on this iteration because it was new and
untested, even if somewhat obvious.

> > I could switch to using "count=1 seek=10000" to create a sparse file,
> > which should be pretty much instantaneous, but allocating the whole
> > backing file seemed like a good idea.
> 
> Right.
> 
> Thanks,
> Ian.