* [PATCH OSSTEST v3 00/19] add distro domU testing flight
@ 2014-12-01 12:56 Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 01/19] TestSupport: Add helper to fetch a URL on a host Ian Campbell
` (19 more replies)
0 siblings, 20 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:56 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
Hi,
The following series adds a new flight type intended to test distro domU
support, i.e. running distro installers and kernels on top of current
Xen to ensure things work. I intend for the flight to be run on a weekly
basis, and I've scheduled it here to run at the weekend when in theory
things might be quieter.
I've initially just done Debian PV domU support since Debian is what I
know, but I hope others will add other distros as time goes on.
The series introduces support for running the Debian Installer in a PV
domU booting from netboot (PXE-ish) and netinst (ISO media) and adding
jobs to test all of those for Squeeze onwards up to Sid and the
daily/weekly builds (CDs are built daily, DVDs weekly).
The flight is a bit unusual since distro installer builds are not really
versioned in a way which is useful to osstest, so instead each flight
will be compared against the previous run only.
Compared with the previous version I've addressed the review comments
and I've also reworked the cr-daily-branch changes after discussion with
Ian. I've run this a few times in play mode on the production system and
things are looking pretty good. See
http://www.chiark.greenend.org.uk/~xensrcts/logs/31967/ for the latest
results.
Most tests passed modulo a few host install failures and the fact that
Debian Jessie+Sid don't install right now for tedious Debian reasons
(the nightlies are OK though, so these will be fixed next time d-i is
uploaded). The armhf failures should be addressed by the patch at the
end of the series.
The exception to that good news is that sg-report-flight
--that-flight=NNN MMM doesn't seem to be considering failures in MMM
regressions vs NNN. e.g. test-amd64-i386-i386-daily-netboot-pvgrub
should appear as a regression from 31961 to 31964 but isn't. I'm going
to have a poke around that.
The new Debian install functionality here also gives us the opportunity
to test pygrub and pvgrub, so a couple of jobs are added to the regular
tests using installer images versioned via TftpDiVersion in the usual
way. This requires an early patch "mg-debian-installer-update: grab Xen
PV domU capable images too" to go in and a patch to update TftpDiVersion
to a refreshed version to be inserted before "Test pygrub and pvgrub on
the regular flights" is committed.
I've also pushed a branch:
The following changes since commit 9fa3df2d9838f2aecb0d0cb6d1a565d824828d65:
cs-adjust-flight: runvar-perlop: Do not report non-changes (2014-11-17 17:32:11 +0000)
are available in the git repository at:
git://xenbits.xen.org/people/ianc/osstest.git distro-flight-v3
for you to fetch changes up to aa67780314f5d9c88858dddde0c6f2b6b149d697:
Debian: Handle lack of bootloader support in d-i on ARM. (2014-12-01 12:43:08 +0000)
----------------------------------------------------------------
Ian Campbell (19):
TestSupport: Add helper to fetch a URL on a host
TestSupport: Add helper to wait for a guest to shutdown
TestSupport: allow overring of on_* in prepareguest_part_xencfg
TestSupport: allow caller of prepareguest_part_xencfg to specify viftype
create_webfile: Support use with guests as well as hosts.
Debian: refactor code to add preseed commands to the preseed file
Debian: refactor preseeding of .ssh directories
Debian: ensure preseed base comment remains at actual end of base.
Debian: Refactor installation of overlays, so it can be used for guests too
Debian: add preseed_create_guest helper
make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs()
distros: add support for installing Debian PV guests via d-i, flight and jobs
distros: support booting Debian PV (d-i installed) guests with pvgrub.
distros: Support pvgrub for Wheezy too.
distros: support PV guest install from Debian netinst media.
Test pygrub and pvgrub on the regular flights
distros: add branch infrastructure
distros: Run a flight over the weekend.
Debian: Handle lack of bootloader support in d-i on ARM.
Osstest.pm | 7 ++
Osstest/Debian.pm | 247 +++++++++++++++++++++++++++++++++++--------------
Osstest/TestSupport.pm | 47 ++++++++--
cr-daily-branch | 39 ++++++--
cri-common | 1 +
crontab | 1 +
make-distros-flight | 140 ++++++++++++++++++++++++++++
make-flight | 43 ++++++++-
mfi-common | 4 +
sg-run-job | 11 +++
ts-debian-di-install | 236 ++++++++++++++++++++++++++++++++++++++++++++++
ts-debian-hvm-install | 8 +-
12 files changed, 689 insertions(+), 95 deletions(-)
create mode 100755 make-distros-flight
create mode 100755 ts-debian-di-install
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 01/19] TestSupport: Add helper to fetch a URL on a host
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 02/19] TestSupport: Add helper to wait for a guest to shutdown Ian Campbell
` (18 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Make sure wget is installed
---
Osstest/Debian.pm | 2 +-
Osstest/TestSupport.pm | 8 ++++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index c8db601..89cd205 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -492,7 +492,7 @@ d-i apt-setup/another boolean false
d-i apt-setup/non-free boolean false
d-i apt-setup/contrib boolean false
-d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, $extra_packages
+d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, wget, $extra_packages
$xopts{ExtraPreseed}
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 46b6720..477ad18 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -55,6 +55,7 @@ BEGIN {
target_putfilecontents_stash
target_putfilecontents_root_stash
target_put_guest_image target_editfile
+ target_fetchurl
target_editfile_root target_file_exists
target_run_apt
target_install_packages target_install_packages_norec
@@ -1472,6 +1473,13 @@ END
return $cfgpath;
}
+sub target_fetchurl($$$;$) {
+ my ($ho, $url, $path, $timeo) = @_;
+ $timeo ||= 2000;
+ target_cmd_root($ho, "wget --progress=dot:mega -O $path $url", $timeo);
+}
+
+
sub target_put_guest_image ($$$) {
my ($ho, $gho, $default) = @_;
my $specimage = $r{"$gho->{Guest}_image"};
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 02/19] TestSupport: Add helper to wait for a guest to shutdown
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 01/19] TestSupport: Add helper to fetch a URL on a host Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 03/19] TestSupport: allow overring of on_* in prepareguest_part_xencfg Ian Campbell
` (17 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Refactor the guts of guest_await_reboot into a helper and use for both
shutdown and reboot handling.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Refactor common code with guest_await_reboot.
---
Osstest/TestSupport.pm | 24 ++++++++++++++++++------
1 file changed, 18 insertions(+), 6 deletions(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 477ad18..c0325c4 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -93,7 +93,8 @@ BEGIN {
guest_umount_lv guest_await guest_await_dhcp_tcp
guest_checkrunning guest_check_ip guest_find_ether
guest_find_domid guest_check_up guest_check_up_quick
- guest_get_state guest_await_reboot guest_destroy
+ guest_get_state guest_await_reboot
+ guest_await_shutdown guest_destroy
guest_vncsnapshot_begin guest_vncsnapshot_stash
guest_check_remus_ok guest_editconfig
host_involves_pcipassthrough host_get_pcipassthrough_devs
@@ -1278,17 +1279,28 @@ sub report_once ($$$) {
$ho->{$k}= $msg;
}
-sub guest_await_reboot ($$$) {
- my ($ho,$gho, $timeout) = @_;
- poll_loop($timeout, 30, "await reboot request from $gho->{Guest}", sub {
+sub guest_await_state ($$$$$) {
+ my ($ho,$gho, $what,$wait_st,$timeout) = @_;
+
+ poll_loop($timeout, 30, "await $what request from $gho->{Guest}", sub {
my $st= guest_get_state($ho,$gho);
- return undef if $st eq 'sr';
+ return undef if $st eq $wait_st;
fail("guest unexpectedly shutdown; state is '$st'")
if $st =~ m/^s/ || $st eq '';
- return "guest state is $st";
+ return "guest state is \"$st\"";
});
}
+sub guest_await_reboot ($$$) {
+ my ($ho,$gho, $timeout) = @_;
+ return guest_await_state($ho,$gho, "reboot", "sr", $timeout);
+}
+
+sub guest_await_shutdown ($$$) {
+ my ($ho,$gho, $timeout) = @_;
+ return guest_await_state($ho,$gho, "shutdown", "s", $timeout);
+}
+
sub guest_destroy ($$) {
my ($ho,$gho) = @_;
target_cmd_root($ho, toolstack()->{Command}." destroy $gho->{Name}", 40);
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 03/19] TestSupport: allow overring of on_* in prepareguest_part_xencfg
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 01/19] TestSupport: Add helper to fetch a URL on a host Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 02/19] TestSupport: Add helper to wait for a guest to shutdown Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 04/19] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype Ian Campbell
` (16 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Currently only on_reboot can be overridden
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Osstest/TestSupport.pm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index c0325c4..3074568 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1457,6 +1457,8 @@ sub prepareguest_part_lvmdisk ($$$) {
sub prepareguest_part_xencfg ($$$$$) {
my ($ho, $gho, $ram_mb, $xopts, $cfgrest) = @_;
my $onreboot= $xopts->{OnReboot} || 'restart';
+ my $onpoweroff= $xopts->{OnPowerOff} || 'destroy';
+ my $oncrash= $xopts->{OnCrash} || 'preserve';
my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
my $xoptcfg= $xopts->{ExtraConfig};
$xoptcfg='' unless defined $xoptcfg;
@@ -1465,9 +1467,9 @@ name = '$gho->{Name}'
memory = ${ram_mb}
vif = [ 'type=ioemu,mac=$gho->{Ether}' ]
#
-on_poweroff = 'destroy'
+on_poweroff = '$onpoweroff'
on_reboot = '$onreboot'
-on_crash = 'preserve'
+on_crash = '$oncrash'
#
vcpus = $vcpus
#
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 04/19] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (2 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 03/19] TestSupport: allow overring of on_* in prepareguest_part_xencfg Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 05/19] create_webfile: Support use with guests as well as hosts Ian Campbell
` (15 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
--
v3: Use CamelCase for xopts, use the correct condition
---
Osstest/TestSupport.pm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 3074568..5edb2fd 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1460,12 +1460,13 @@ sub prepareguest_part_xencfg ($$$$$) {
my $onpoweroff= $xopts->{OnPowerOff} || 'destroy';
my $oncrash= $xopts->{OnCrash} || 'preserve';
my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
+ my $viftype= $xopts->{VifType} ? "type=$xopts->{VifType}," : "";
my $xoptcfg= $xopts->{ExtraConfig};
$xoptcfg='' unless defined $xoptcfg;
my $xencfg= <<END;
name = '$gho->{Name}'
memory = ${ram_mb}
-vif = [ 'type=ioemu,mac=$gho->{Ether}' ]
+vif = [ '${viftype}mac=$gho->{Ether}' ]
#
on_poweroff = '$onpoweroff'
on_reboot = '$onreboot'
@@ -1560,6 +1561,7 @@ END
$cfg .= "bios='$bios'\n";
}
+ $xopts{VifType} ||= "ioemu";
my $cfgpath= prepareguest_part_xencfg($ho, $gho, $ram_mb, \%xopts, $cfg);
target_cmd_root($ho, <<END);
(echo $passwd; echo $passwd) | vncpasswd $gho->{Guest}.vncpw
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 05/19] create_webfile: Support use with guests as well as hosts.
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (3 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 04/19] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 06/19] Debian: refactor code to add preseed commands to the preseed file Ian Campbell
` (14 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
In particular make the path unique by ensuring it includes the host
and guest name in the guest case.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: New patch
---
Osstest/TestSupport.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 5edb2fd..2f23fd4 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1932,6 +1932,8 @@ sub await_webspace_fetch_byleaf ($$$$$) {
sub create_webfile ($$$) {
my ($ho, $tail, $contents) = @_; # $contents as for file_link_contents
my $wf_rhs= $ho->{Name}."_".$tail;
+ # $ho->{Host} is set if $ho is a guest.
+ $wf_rhs= $ho->{Host}->{Name}."_${wf_rhs}" if $ho->{Host};
my $wf_common= $c{WebspaceCommon}.$wf_rhs;
my $wf_url= $c{WebspaceUrl}.$wf_common;
my $wf_file= $c{WebspaceFile}.$wf_common;
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 06/19] Debian: refactor code to add preseed commands to the preseed file
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (4 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 05/19] create_webfile: Support use with guests as well as hosts Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 07/19] Debian: refactor preseeding of .ssh directories Ian Campbell
` (13 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Call it from ts-debian-hvm-install.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: New patch
---
Osstest/Debian.pm | 16 +++++++++++-----
ts-debian-hvm-install | 3 +++
2 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 89cd205..d1adb02 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -35,7 +35,7 @@ BEGIN {
%preseed_cmds
preseed_base
preseed_create
- preseed_hook_command preseed_hook_installscript
+ preseed_hook_command preseed_hook_installscript preseed_hook_cmds
di_installcmdline_core
);
%EXPORT_TAGS = ( );
@@ -726,10 +726,7 @@ d-i partman-auto/expert_recipe string \\
END
- foreach my $di_key (keys %preseed_cmds) {
- $preseed_file .= "d-i preseed/$di_key string ".
- (join ' && ', @{ $preseed_cmds{$di_key} }). "\n";
- }
+ $preseed_file .= preseed_hook_cmds();
if ($ho->{Flags}{'no-di-kernel'}) {
$preseed_file .= <<END;
@@ -773,4 +770,13 @@ chmod +x '$installer_pathname'
END
}
+sub preseed_hook_cmds () {
+ my $preseed;
+ foreach my $di_key (keys %preseed_cmds) {
+ $preseed .= "d-i preseed/$di_key string ".
+ (join ' && ', @{ $preseed_cmds{$di_key} }). "\n";
+ }
+ return $preseed;
+}
+
1;
diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 37eade2..878083f 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -80,6 +80,9 @@ d-i preseed/late_command string \\
in-target mkdir -p /root/.ssh; \\
in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
END
+
+ $preseed_file .= preseed_hook_cmds();
+
return $preseed_file;
}
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 07/19] Debian: refactor preseeding of .ssh directories
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (5 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 06/19] Debian: refactor code to add preseed commands to the preseed file Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 08/19] Debian: ensure preseed base comment remains at actual end of base Ian Campbell
` (12 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Causes known_hosts to be consistently created as well as ~osstest/.ssh
to be consistently populated (it previsouly wasn't for HVM guests).
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: New patch
---
Osstest/Debian.pm | 99 +++++++++++++++++++++++++++++----------------------
ts-debian-hvm-install | 5 ++-
2 files changed, 59 insertions(+), 45 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index d1adb02..bdbe2f3 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -35,6 +35,7 @@ BEGIN {
%preseed_cmds
preseed_base
preseed_create
+ preseed_ssh
preseed_hook_command preseed_hook_installscript preseed_hook_cmds
di_installcmdline_core
);
@@ -427,6 +428,60 @@ sub di_installcmdline_core ($$;@) {
return @cl;
}
+sub preseed_ssh ($$) {
+ my ($ho,$sfx) = @_;
+
+ my $authkeys_url= create_webfile($ho, "authkeys$sfx", authorized_keys());
+
+ my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key.pub";
+ my $hostkey= get_filecontents($hostkeyfile);
+ chomp($hostkey); $hostkey.="\n";
+ my $knownhosts= '';
+
+ my $hostsq= $dbh_tests->prepare(<<END);
+ SELECT val FROM runvars
+ WHERE flight=? AND name LIKE '%host'
+ GROUP BY val
+END
+ $hostsq->execute($flight);
+ while (my ($node) = $hostsq->fetchrow_array()) {
+ my $longname= "$node.$c{TestHostDomain}";
+ my (@hostent)= gethostbyname($longname);
+ if (!@hostent) {
+ logm("skipping host key for nonexistent host $longname");
+ next;
+ }
+ my $specs= join ',', $longname, $node, map {
+ join '.', unpack 'W4', $_;
+ } @hostent[4..$#hostent];
+ logm("adding host key for $specs");
+ $knownhosts.= "$specs ".$hostkey;
+ }
+ $hostsq->finish();
+
+ $knownhosts.= "localhost,127.0.0.1 ".$hostkey;
+ my $knownhosts_url= create_webfile($ho, "known_hosts$sfx", $knownhosts);
+
+ preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+r=/target/root
+cd \$r
+
+umask 022
+mkdir .ssh
+wget -O .ssh/authorized_keys '$authkeys_url'
+wget -O .ssh/known_hosts '$knownhosts_url'
+
+u=osstest
+h=/home/\$u
+mkdir /target\$h/.ssh
+cp .ssh/authorized_keys /target\$h/.ssh
+chroot /target chown -R \$u.\$u \$h/.ssh
+END
+}
+
sub preseed_base ($$;@) {
my ($suite,$extra_packages,%xopts) = @_;
@@ -504,40 +559,9 @@ END
sub preseed_create ($$;@) {
my ($ho, $sfx, %xopts) = @_;
- my $authkeys_url= create_webfile($ho, "authkeys$sfx", authorized_keys());
-
- my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key.pub";
- my $hostkey= get_filecontents($hostkeyfile);
- chomp($hostkey); $hostkey.="\n";
- my $knownhosts= '';
-
my $disk= $xopts{DiskDevice} || '/dev/sda';
my $suite= $xopts{Suite} || $c{DebianSuite};
- my $hostsq= $dbh_tests->prepare(<<END);
- SELECT val FROM runvars
- WHERE flight=? AND name LIKE '%host'
- GROUP BY val
-END
- $hostsq->execute($flight);
- while (my ($node) = $hostsq->fetchrow_array()) {
- my $longname= "$node.$c{TestHostDomain}";
- my (@hostent)= gethostbyname($longname);
- if (!@hostent) {
- logm("skipping host key for nonexistent host $longname");
- next;
- }
- my $specs= join ',', $longname, $node, map {
- join '.', unpack 'W4', $_;
- } @hostent[4..$#hostent];
- logm("adding host key for $specs");
- $knownhosts.= "$specs ".$hostkey;
- }
- $hostsq->finish();
-
- $knownhosts.= "localhost,127.0.0.1 ".$hostkey;
- my $knownhosts_url= create_webfile($ho, "known_hosts$sfx", $knownhosts);
-
my $overlays= '';
my $create_overlay= sub {
my ($srcdir, $tfilename) = @_;
@@ -602,6 +626,8 @@ ls -l /dev/sd*
true
END
+ preseed_ssh($ho, $sfx);
+
preseed_hook_command($ho, 'late_command', $sfx, <<END);
#!/bin/sh
set -ex
@@ -609,17 +635,6 @@ set -ex
r=/target/root
cd \$r
-umask 022
-mkdir .ssh
-wget -O .ssh/authorized_keys '$authkeys_url'
-wget -O .ssh/known_hosts '$knownhosts_url'
-
-u=osstest
-h=/home/\$u
-mkdir /target\$h/.ssh
-cp .ssh/authorized_keys /target\$h/.ssh
-chroot /target chown -R \$u.\$u \$h/.ssh
-
echo FANCYTTY=0 >> /target/etc/lsb-base-logging.sh
$overlays
diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 878083f..d2bb6f8 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -47,7 +47,6 @@ our $toolstack= toolstack()->{Command};
sub preseed () {
my $preseed_file = preseed_base('wheezy','',());
- my $authkeys = join('\\n', split(/\n/, authorized_keys()));
$preseed_file .= (<<END);
d-i netcfg/get_hostname string $gn
@@ -77,10 +76,10 @@ d-i apt-setup/cdrom/set-first boolean false
d-i preseed/late_command string \\
in-target mkdir -p /boot/efi/EFI/boot; \\
in-target cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/boot/bootx64.efi ;\\
- in-target mkdir -p /root/.ssh; \\
- in-target sh -c "echo -e '$authkeys'> /root/.ssh/authorized_keys";
END
+ preseed_ssh($ho,'');
+
$preseed_file .= preseed_hook_cmds();
return $preseed_file;
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 08/19] Debian: ensure preseed base comment remains at actual end of base.
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (6 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 07/19] Debian: refactor preseeding of .ssh directories Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 09/19] Debian: Refactor installation of overlays, so it can be used for guests too Ian Campbell
` (11 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
By making it a separate stanza. This is to help avoid people
accidentally adding stuff after it, such as in the following patches.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: New patch
---
Osstest/Debian.pm | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index bdbe2f3..c985913 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -550,11 +550,14 @@ d-i apt-setup/contrib boolean false
d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, wget, $extra_packages
$xopts{ExtraPreseed}
+END
+ $preseed .= <<END;
### END OF DEBIAN PRESEED BASE
-
END
-}
+
+ return $preseed;
+}
sub preseed_create ($$;@) {
my ($ho, $sfx, %xopts) = @_;
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 09/19] Debian: Refactor installation of overlays, so it can be used for guests too
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (7 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 08/19] Debian: ensure preseed base comment remains at actual end of base Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 10/19] Debian: add preseed_create_guest helper Ian Campbell
` (10 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: New patch
---
Osstest/Debian.pm | 57 ++++++++++++++++++++++++++++---------------------------
1 file changed, 29 insertions(+), 28 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index c985913..8ec4a3b 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -36,7 +36,9 @@ BEGIN {
preseed_base
preseed_create
preseed_ssh
- preseed_hook_command preseed_hook_installscript preseed_hook_cmds
+ preseed_hook_command preseed_hook_installscript
+ preseed_hook_overlay
+ preseed_hook_cmds
di_installcmdline_core
);
%EXPORT_TAGS = ( );
@@ -565,26 +567,6 @@ sub preseed_create ($$;@) {
my $disk= $xopts{DiskDevice} || '/dev/sda';
my $suite= $xopts{Suite} || $c{DebianSuite};
- my $overlays= '';
- my $create_overlay= sub {
- my ($srcdir, $tfilename) = @_;
- my $url= create_webfile($ho, "$tfilename$sfx", sub {
- my ($fh) = @_;
- contents_make_cpio($fh, 'ustar', $srcdir);
- });
- $overlays .= <<END;
-wget -O overlay.tar '$url'
-cd /target
-tar xf \$r/overlay.tar
-cd \$r
-rm overlay.tar
-
-END
- };
-
- $create_overlay->('overlay', 'overlay.tar');
- $create_overlay->($c{OverlayLocal}, 'overlay-local.tar');
-
preseed_hook_installscript($ho, $sfx,
'/lib/partman/init.d', '000override-parted-devices', <<END);
#!/bin/sh
@@ -630,19 +612,14 @@ true
END
preseed_ssh($ho, $sfx);
+ preseed_hook_overlay($ho, $sfx, 'overlay', 'overlay.tar');
+ preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
preseed_hook_command($ho, 'late_command', $sfx, <<END);
#!/bin/sh
set -ex
-r=/target/root
-cd \$r
-
echo FANCYTTY=0 >> /target/etc/lsb-base-logging.sh
-
-$overlays
-
-echo latecmd done.
END
foreach my $kp (keys %{ $ho->{Flags} }) {
@@ -788,6 +765,30 @@ chmod +x '$installer_pathname'
END
}
+sub preseed_hook_overlay ($$$$) {
+ my ($ho, $sfx, $srcdir, $tfilename) = @_;
+ my $url= create_webfile($ho, "$tfilename$sfx", sub {
+ my ($fh) = @_;
+ contents_make_cpio($fh, 'ustar', $srcdir);
+ });
+ preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+r=/target/root
+cd \$r
+
+umask 022
+
+wget -O overlay.tar '$url'
+cd /target
+tar xf \$r/overlay.tar
+cd \$r
+rm overlay.tar
+
+END
+}
+
sub preseed_hook_cmds () {
my $preseed;
foreach my $di_key (keys %preseed_cmds) {
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 10/19] Debian: add preseed_create_guest helper
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (8 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 09/19] Debian: Refactor installation of overlays, so it can be used for guests too Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 11/19] make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs() Ian Campbell
` (9 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Creates a preseed file suitable for use in a PV guest
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Handle $xopts{ExtraPreseed} undefined in preseed_base
---
Osstest/Debian.pm | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 8ec4a3b..c4afde1 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -35,6 +35,7 @@ BEGIN {
%preseed_cmds
preseed_base
preseed_create
+ preseed_create_guest
preseed_ssh
preseed_hook_command preseed_hook_installscript
preseed_hook_overlay
@@ -487,6 +488,9 @@ END
sub preseed_base ($$;@) {
my ($suite,$extra_packages,%xopts) = @_;
+ $extra_packages ||= '';
+ $xopts{ExtraPreseed} ||= '';
+
return <<"END";
d-i mirror/suite string $suite
@@ -561,6 +565,31 @@ END
return $preseed;
}
+sub preseed_create_guest ($$;@) {
+ my ($ho, $sfx, %xopts) = @_;
+
+ my $suite= $xopts{Suite} || $c{DebianSuite};
+
+ my $extra_packages;
+
+ my $preseed_file= preseed_base($suite, $extra_packages, %xopts);
+ $preseed_file.= (<<END);
+d-i partman-auto/method string regular
+d-i partman-auto/choose_recipe \\
+ select All files in one partition (recommended for new users)
+
+d-i grub-installer/bootdev string /dev/xvda
+
+END
+
+ preseed_ssh($ho, $sfx);
+ preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
+
+ $preseed_file .= preseed_hook_cmds();
+
+ return create_webfile($ho, "preseed$sfx", $preseed_file);
+}
+
sub preseed_create ($$;@) {
my ($ho, $sfx, %xopts) = @_;
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 11/19] make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs()
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (9 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 10/19] Debian: add preseed_create_guest helper Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 12/19] distros: add support for installing Debian PV guests via d-i, flight and jobs Ian Campbell
` (8 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: New patch
---
make-flight | 4 ----
mfi-common | 4 ++++
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/make-flight b/make-flight
index 9963a46..b641683 100755
--- a/make-flight
+++ b/make-flight
@@ -35,10 +35,6 @@ defguestsuite=`getconfig GuestDebianSuite`
if [ x$buildflight = x ]; then
- if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
- BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
- fi
-
create_build_jobs
else
diff --git a/mfi-common b/mfi-common
index 5c4f5d5..86901fe 100644
--- a/mfi-common
+++ b/mfi-common
@@ -50,6 +50,10 @@ create_build_jobs () {
local enable_ovmf
local build_hostflags
+ if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
+ BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
+ fi
+
for arch in ${BUILD_ARCHES- i386 amd64 armhf }; do
if [ "x$arch" = xdisable ]; then continue; fi
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 12/19] distros: add support for installing Debian PV guests via d-i, flight and jobs
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (10 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 11/19] make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs() Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 13/19] distros: support booting Debian PV (d-i installed) guests with pvgrub Ian Campbell
` (7 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
This patch introduces ts-debian-di-install which can install Debian
from a netboot (PXE) debian installer image. By default it installs
from the d-i image used by osstest (using the special Xen PV guest
enabled flavour where necessary) but it can also install the current
release versions from Squeeze onwards (up to and including Sid) and
the d-i daily builds. This is controlled by runvars {Guest}_diver =
osstest|current and {Guest}_dist = squeeze|...|sid. The resulting
guests boot the distro kernel using pygrub (pvgrub will follow).
The distros flights differ substantially from the existing flights.
Introduce make-distros-flight using the functionality previously
refactored into mfi-common. The new flight tests all versions of
Debian from Squeeze onward as an amd64, i386 and armhf guests (armhf
from Jessie onwards only) using the usual smoke tests.
Test names are suffixed -pygrub pending the addition of pvgrub
variants in a future commit.
Add the new cases to sg-run-job
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: $BUILD_LVEXTEND_MAX now handled in mfi-common
Consolidate setting of ruvars
Include $flight and $job in tmpdir name
Use Osstest::Debian::di_installcmdline_core
Document the usage of get_host_property on a guest object
Correct ARM netboot paths
Include bootloader in test name
Should include -pv too?
console= repetition for Jessie onwards.
Wait for up to an hour for the install. I'd seen timeouts right at
the end of the install with the previous value
---
Osstest/TestSupport.pm | 3 +
make-distros-flight | 98 ++++++++++++++++++++++++++++
sg-run-job | 11 ++++
ts-debian-di-install | 170 +++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 282 insertions(+)
create mode 100755 make-distros-flight
create mode 100755 ts-debian-di-install
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 2f23fd4..c1ecd8d 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -861,8 +861,11 @@ sub propname_massage ($) {
return $prop;
}
+# It is fine to call this on a guest object too, in which case it will
+# always return $defval.
sub get_host_property ($$;$) {
my ($ho, $prop, $defval) = @_;
+ return $defval unless $ho->{Properties};
my $val = $ho->{Properties}{propname_massage($prop)};
return defined($val) ? $val : $defval;
}
diff --git a/make-distros-flight b/make-distros-flight
new file mode 100755
index 0000000..5067342
--- /dev/null
+++ b/make-distros-flight
@@ -0,0 +1,98 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. cri-common
+. ap-common
+. mfi-common
+
+defsuite=`getconfig DebianSuite`
+defguestsuite=`getconfig GuestDebianSuite`
+
+if [ x$buildflight = x ]; then
+
+ WANT_XEND=false REVISION_LINUX_OLD=disable
+
+ create_build_jobs
+
+else
+
+ bfi=$buildflight.
+
+fi
+
+job_create_test_filter_callback () {
+ if [ "$xenarch" = "i386" ]; then return 1; fi
+ return 0
+}
+
+test_matrix_branch_filter_callback () {
+ :
+}
+
+test_do_one_netboot () {
+ job_create_test \
+ test-$xenarch$kern-$dom0arch-$domU-$dist-netboot-pygrub \
+ test-debian-di xl $xenarch $dom0arch \
+ kernbuildjob=${bfi}build-$dom0arch-$kernbuild \
+ debian_arch=$domU \
+ debian_dist=$dist \
+ debian_method=netboot \
+ debian_diver=current \
+ all_hostflags=$most_hostflags
+}
+
+test_matrix_do_one () {
+ case ${xenarch} in
+ amd64) domUarches="amd64 i386";;
+ armhf) domUarches="armhf";;
+ esac
+
+ for domU in $domUarches ; do
+ for dist in squeeze wheezy jessie sid daily ; do
+ case ${domU}_${dist} in
+ armhf_squeeze) continue;; # No armhf in Squeeze
+ armhf_wheezy) continue;; # No armhf guest support in Wheezy
+ *) ;;
+ esac
+
+ test_do_one_netboot
+
+ done
+
+ done
+}
+
+test_matrix_iterate
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:
diff --git a/sg-run-job b/sg-run-job
index 2cf810a..9943ee4 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -247,6 +247,17 @@ proc run-job/test-debian {} {
test-guest debian
}
+proc install-guest-debian-di {} {
+ run-ts . = ts-debian-di-install
+ run-ts . = ts-guest-start + debian
+}
+
+proc need-hosts/test-debian-di {} { return host }
+proc run-job/test-debian-di {} {
+ install-guest-debian-di
+ test-guest debian
+}
+
proc need-hosts/test-freebsd {} { return host }
proc run-job/test-freebsd {} {
run-ts . = ts-freebsd-install
diff --git a/ts-debian-di-install b/ts-debian-di-install
new file mode 100755
index 0000000..edccf52
--- /dev/null
+++ b/ts-debian-di-install
@@ -0,0 +1,170 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+$gn ||= 'debian';
+
+our $ho= selecthost($whhost);
+
+our $ram_mb= 512;
+our $disk_mb= 10000;
+
+our $guesthost= "$gn.guest.osstest";
+our $gho;
+
+sub prep () {
+ target_install_packages_norec($ho, qw(lvm2));
+
+ $gho= prepareguest($ho, $gn, $guesthost, 22,
+ $disk_mb, 40);
+
+ prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
+
+ target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+}
+
+sub setup_netboot($$$)
+{
+ my ($didir, $arch, $suite) = @_;
+
+ my $di_ver= $r{"$gho->{Guest}_diver"} || "osstest";
+
+ my ($kernel,$initrd);
+
+ if ( $di_ver eq "osstest" ) {
+ my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.$c{TftpDiVersion}.'-'.$ho->{Suite};
+
+ $kernel = "$di_path/vmlinuz-xen";
+ $initrd = "$di_path/initrd.gz-xen";
+
+ target_putfile_root($ho, 60, $kernel, "$didir/kernel_${suite}_${arch}");
+ target_putfile_root($ho, 60, $initrd, "$didir/initrd_${suite}_${arch}");
+
+ } else {
+ my $mirror = "http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath}";
+
+ my $di_url = $suite eq "daily" ?
+ "http://d-i.debian.org/daily-images/$arch/daily/netboot" :
+ "$mirror/dists/$suite/main/installer-$arch/$di_ver/images/netboot";
+
+ $di_url .= "/xen" if $arch =~ m/amd64|i386/;
+ $di_url .= "/debian-installer/arm64" if $arch =~ /arm64/;
+
+ $kernel = "$di_url/vmlinuz";
+ $initrd = "$di_url/initrd.gz";
+
+ target_fetchurl($ho, $kernel, "$didir/kernel_${suite}_${arch}");
+ target_fetchurl($ho, $initrd, "$didir/initrd_${suite}_${arch}");
+ }
+
+ store_runvar("$gho->{Guest}_netboot_kernel", $kernel);
+ store_runvar("$gho->{Guest}_netboot_initrd", $initrd);
+
+ return <<END;
+kernel = "$didir/kernel_${suite}_${arch}"
+ramdisk = "$didir/initrd_${suite}_${arch}"
+END
+}
+sub ginstall () {
+ my $arch= $r{"$gho->{Guest}_arch"};
+ my $method= $r{"$gho->{Guest}_method"};
+
+ my $tmpdir= "/root/$flight-$job-di";
+ target_cmd_root($ho, <<END);
+rm -rf $tmpdir
+mkdir $tmpdir
+END
+
+ my ($method_cfg, $ps_url, $extra_disk);
+
+ if ( $method eq "netboot" )
+ {
+ my $suite= $r{"$gho->{Guest}_dist"};
+ logm("$method $suite/$arch");
+
+ $method_cfg = setup_netboot($tmpdir, $arch, $suite);
+
+ $suite = "sid" if $suite eq "daily";
+
+ $ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+
+ $extra_disk = "";
+ }
+ else
+ {
+ die "$method";
+ }
+
+ my @cmdline = ();
+ push @cmdline, "debian-installer/exit/always_halt=true";
+ push @cmdline, "domain=$c{TestHostDomain}";
+ push @cmdline, "console=hvc0";
+ push @cmdline, di_installcmdline_core($gho, $ps_url);
+ push @cmdline, "--";
+ # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007 for
+ # why this is repeated.
+ push @cmdline, "console=hvc0";
+
+ my $cmdline = join(" ", @cmdline);
+
+ my %install_xopts = (
+ OnPowerOff => "preserve"
+ );
+
+ prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
+$method_cfg
+extra = "$cmdline"
+#
+disk = [
+ $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+ ]
+END
+
+ my $cmd= toolstack()->{Command}." create ".
+ $r{ $gho->{Guest}.'_'. toolstack()->{CfgPathVar} };
+ target_cmd_root($ho, $cmd, 300);
+
+ guest_checkrunning($ho, $gho) or die "$gho->{Name} not running";
+
+ guest_await_shutdown($ho,$gho,3600);
+ guest_destroy($ho,$gho);
+
+ my $blcfg = <<END;
+bootloader = "pygrub"
+END
+
+ prepareguest_part_xencfg($ho, $gho, $ram_mb, {}, <<END);
+$blcfg
+#
+disk = [
+ 'phy:$gho->{Lvdev},xvda,w'
+ ]
+END
+ return;
+}
+
+prep();
+ginstall();
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 13/19] distros: support booting Debian PV (d-i installed) guests with pvgrub.
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (11 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 12/19] distros: add support for installing Debian PV guests via d-i, flight and jobs Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 14/19] distros: Support pvgrub for Wheezy too Ian Campbell
` (6 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
This requires the use of the pv-grub-menu package which is in Jessie
onwards. (it is in wheezy-backports which is the subject of a
subsequent flight).
The bootloader to use is specified via a runvar {Guest}_bootloader.
Adjust make-distros-flight to use pvgrub for some subset of i386 and
amd64 guests to get coverage.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Define and use arch_debian2xen and arch_xen2debian
Avoid pv-grub-x86_64.gz on i386 dom0, we don't built it there.
Fiddle with py vs pv grub stripy a bit.
---
Osstest.pm | 7 +++++++
Osstest/Debian.pm | 2 +-
make-distros-flight | 21 ++++++++++++++++++++-
ts-debian-di-install | 11 +++++++++--
4 files changed, 37 insertions(+), 4 deletions(-)
diff --git a/Osstest.pm b/Osstest.pm
index fc9698b..31088ba 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -36,6 +36,7 @@ BEGIN {
db_begin_work
ensuredir get_filecontents_core_quiet system_checked
nonempty visible_undef show_abs_time
+ %arch_debian2xen %arch_xen2debian
);
%EXPORT_TAGS = ( );
@@ -47,6 +48,12 @@ our $mjobdb;
our $dbh_tests;
+our %arch_debian2xen = qw(i386 x86_32
+ amd64 x86_64
+ armhf armhf);
+our %arch_xen2debian;
+$arch_xen2debian{$arch_debian2xen{$_}} = $_ foreach keys %arch_debian2xen;
+
#---------- static default config settings ----------
our %c = qw(
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index c4afde1..a15768f 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -570,7 +570,7 @@ sub preseed_create_guest ($$;@) {
my $suite= $xopts{Suite} || $c{DebianSuite};
- my $extra_packages;
+ my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
my $preseed_file= preseed_base($suite, $extra_packages, %xopts);
$preseed_file.= (<<END);
diff --git a/make-distros-flight b/make-distros-flight
index 5067342..80df21b 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -55,14 +55,33 @@ test_matrix_branch_filter_callback () {
}
test_do_one_netboot () {
+ stripy bootloader pvgrub pygrub \
+ "$xenarch" "amd64" \
+ "$dom0arch" "i386" \
+ "$domU" "amd64" \
+ "$dist" "sid"
+
+ case ${dom0arch}_${domU}_${dist} in
+ arm*_arm*_*) bootloader="pygrub";; # no pvgrub for arm
+
+ # Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
+ # workedaround in Jessie+ with pv-grub-menu package.
+ *_squeeze) bootloader="pygrub";;
+ *_wheezy) bootloader="pygrub";;
+
+ # pv-grub-x86_64.gz is not built by 32-bit dom0 userspace build.
+ i386_amd64_*) bootloader="pygrub";;
+ esac
+
job_create_test \
- test-$xenarch$kern-$dom0arch-$domU-$dist-netboot-pygrub \
+ test-$xenarch$kern-$dom0arch-$domU-$dist-netboot-$bootloader \
test-debian-di xl $xenarch $dom0arch \
kernbuildjob=${bfi}build-$dom0arch-$kernbuild \
debian_arch=$domU \
debian_dist=$dist \
debian_method=netboot \
debian_diver=current \
+ debian_bootloader=$bootloader \
all_hostflags=$most_hostflags
}
diff --git a/ts-debian-di-install b/ts-debian-di-install
index edccf52..f3fe4cf 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -93,6 +93,8 @@ sub ginstall () {
my $method= $r{"$gho->{Guest}_method"};
my $tmpdir= "/root/$flight-$job-di";
+ my $bl= $r{"$gho->{Guest}_bootloader"};
+
target_cmd_root($ho, <<END);
rm -rf $tmpdir
mkdir $tmpdir
@@ -109,7 +111,7 @@ END
$suite = "sid" if $suite eq "daily";
- $ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+ $ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
$extra_disk = "";
}
@@ -152,7 +154,12 @@ END
guest_await_shutdown($ho,$gho,3600);
guest_destroy($ho,$gho);
- my $blcfg = <<END;
+ my $xenarch = $arch_debian2xen{$arch};
+ my $pvgrub = "/usr/local/lib/xen/boot/pv-grub-$xenarch.gz";
+ my $blcfg = $bl eq "pvgrub" ? <<END : <<END;
+kernel = "$pvgrub"
+extra = "(hd0,0)/boot/grub/menu.lst"
+END
bootloader = "pygrub"
END
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 14/19] distros: Support pvgrub for Wheezy too.
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (12 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 13/19] distros: support booting Debian PV (d-i installed) guests with pvgrub Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 15/19] distros: support PV guest install from Debian netinst media Ian Campbell
` (5 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
This requires us to install pv-grub-menu from backports, which we do
using a late_command.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3:
- Remove spurious () from <<(END) (and the prexisting "" too)
- Remove $xopts{EnableBackports} and automatically handle the need to add
backports in preseed_base.
- Install via late_command not apt-setup, since the former has
issues, hence subject drops "attempt to..."
---
Osstest/Debian.pm | 46 ++++++++++++++++++++++++++++++++++++++++++++--
make-distros-flight | 4 ++--
2 files changed, 46 insertions(+), 4 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index a15768f..dc01be5 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -491,7 +491,7 @@ sub preseed_base ($$;@) {
$extra_packages ||= '';
$xopts{ExtraPreseed} ||= '';
- return <<"END";
+ my $preseed= <<END;
d-i mirror/suite string $suite
d-i debian-installer/locale string en_GB
@@ -558,6 +558,17 @@ d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, wget, $extra_pa
$xopts{ExtraPreseed}
END
+ # deb http://ftp.debian.org/debian/ wheezy-backports main
+ $preseed .= <<END if $extra_packages =~ m#\b/$suite-backports\b#;
+d-i apt-setup/local0/repository string \\
+ http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
+d-i apt-setup/local0/comment string $suite backports
+END
+# Jessie onwards...
+# $preseed .= <<END if $extra_packages =~ m#\b/$suite-backports\b#;
+#d-i apt-setup/services-select multiselect security, updates, backports
+#END
+
$preseed .= <<END;
### END OF DEBIAN PRESEED BASE
END
@@ -570,7 +581,38 @@ sub preseed_create_guest ($$;@) {
my $suite= $xopts{Suite} || $c{DebianSuite};
- my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
+ my $extra_packages = "";
+ if ($xopts{PvMenuLst}) {
+ if ($suite =~ m/wheezy/) {
+ # pv-grub-menu/wheezy-backports + using apt-setup to add
+ # backports results in iproute, ifupdown and
+ # isc-dhcp-client getting removed because tasksel's
+ # invocation of apt-get install somehow decides the
+ # iproute2 from wheezy-backports is a thing it wants to
+ # install. So instead lets fake it with a late command...
+ #
+ # This also has the bonus of working round an issue with
+ # 1.2.1~bpo70+1 which created an invalid menu.lst using
+ # "root(/dev/xvda,0)" which pvgrub cannot parse because
+ # the Grub device.map isn't present at pkgsel/include time
+ # but it is by late_command time. This was fixed by
+ # version 1.3 which is in Jessie onwards.
+ preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+(
+ echo
+ echo \\\# $suite backports
+ echo deb http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
+) >> /target/etc/apt/sources.list
+in-target apt-get update
+in-target apt-get install -y -t wheezy-backports pv-grub-menu
+END
+ } else {
+ $extra_packages = "pv-grub-menu";
+ }
+ }
my $preseed_file= preseed_base($suite, $extra_packages, %xopts);
$preseed_file.= (<<END);
diff --git a/make-distros-flight b/make-distros-flight
index 80df21b..97df89a 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -65,9 +65,9 @@ test_do_one_netboot () {
arm*_arm*_*) bootloader="pygrub";; # no pvgrub for arm
# Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
- # workedaround in Jessie+ with pv-grub-menu package.
+ # workedaround in Wheezy+ with pv-grub-menu package (backports in Wheezy,
+ # in Jessie+ main).
*_squeeze) bootloader="pygrub";;
- *_wheezy) bootloader="pygrub";;
# pv-grub-x86_64.gz is not built by 32-bit dom0 userspace build.
i386_amd64_*) bootloader="pygrub";;
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 15/19] distros: support PV guest install from Debian netinst media.
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (13 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 14/19] distros: Support pvgrub for Wheezy too Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 16/19] Test pygrub and pvgrub on the regular flights Ian Campbell
` (4 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
The netinst media are iso images containing a base Debian install and
some (image size dependent) additional tasks.
On x86 the "multiarch" iso flavour contains a Xen capable kernel for
both i386 and amd64 so use that.
This adds support for two classes of ISO, the CD sized ones which are
built nightly and the DVD sized ones which are built weekly.
The images are downloaded using jigdo which sources the majority of
the data from a local Debian mirror, for this reason I have not
worried about the fact that the i386 and amd64 tests are downloading
the same thing (adding a specific download job would require finding
up to 4GB of scratch space for each flight).
The ISOs booted using pygrub which can extract the kernel and initrd
from a ISO image. The resulting guests are also booted with pygrub
since the pv-grub-menu package is not available on the ISO images and
we have pvgrub coverage from the netboot tests.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3:
- Indent jigdo-lite script line to improve readability
- Wrap bootloader_args
- Fetch URL on target, to get a timeout. Use wget since that is arranged to
be present.
- include bootloader in test name.
---
Osstest/Debian.pm | 7 +++++--
make-distros-flight | 23 +++++++++++++++++++++
ts-debian-di-install | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 85 insertions(+), 2 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index dc01be5..25cacd2 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -492,8 +492,6 @@ sub preseed_base ($$;@) {
$xopts{ExtraPreseed} ||= '';
my $preseed= <<END;
-d-i mirror/suite string $suite
-
d-i debian-installer/locale string en_GB
d-i console-keymaps-at/keymap select gb
d-i keyboard-configuration/xkb-keymap string en_GB
@@ -558,6 +556,11 @@ d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, wget, $extra_pa
$xopts{ExtraPreseed}
END
+ # For CDROM the suite is part of the image
+ $preseed .= <<END unless $xopts{CDROM};
+d-i mirror/suite string $suite
+END
+
# deb http://ftp.debian.org/debian/ wheezy-backports main
$preseed .= <<END if $extra_packages =~ m#\b/$suite-backports\b#;
d-i apt-setup/local0/repository string \\
diff --git a/make-distros-flight b/make-distros-flight
index 97df89a..e3ec8f7 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -85,6 +85,20 @@ test_do_one_netboot () {
all_hostflags=$most_hostflags
}
+test_do_one_netinst () {
+ # Always pygrub since no pv-grub-menu on CD
+ job_create_test \
+ test-$xenarch$kern-$dom0arch-$domU-$cd-netinst-pygrub \
+ test-debian-di xl $xenarch $dom0arch \
+ kernbuildjob=${bfi}build-$dom0arch-$kernbuild \
+ debian_arch=$domU \
+ debian_cd=$cd \
+ debian_method=netinst \
+ debian_bootloader=pygrub \
+ all_hostflags=$most_hostflags
+
+}
+
test_matrix_do_one () {
case ${xenarch} in
amd64) domUarches="amd64 i386";;
@@ -103,6 +117,15 @@ test_matrix_do_one () {
done
+ for cd in current weekly ; do
+ case ${domU}_${dist} in
+ armhf_*) continue;; # No iso targets for armhf
+ *) ;;
+ esac
+
+ test_do_one_netinst
+
+ done
done
}
diff --git a/ts-debian-di-install b/ts-debian-di-install
index f3fe4cf..3878f93 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -46,6 +46,53 @@ sub prep () {
target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
}
+sub setup_netinst($$$)
+{
+ my ($didir, $arch, $cd) = @_;
+ my %arch_props = (
+ amd64 => { PathArch => "multi-arch", FileArch => "amd64-i386", IsoPath => "/install.amd/xen" },
+ i386 => { PathArch => "multi-arch", FileArch => "amd64-i386", IsoPath => "/install.386/xen" },
+ armhf => { PathArch => "armhf", FileArch => "armhf", IsoPath => "/install.armhf" }
+ );
+ my $props = $arch_props{$arch} or die "Unknown arch $arch";
+
+ target_install_packages($ho, qw(jigdo-file));
+
+ my $baseurl = $cd eq "current" ?
+ "http://cdimage.debian.org/debian-cd/current/$props->{PathArch}/jigdo-cd" :
+ "http://cdimage.debian.org/cdimage/weekly-builds/$props->{PathArch}/jigdo-cd";
+
+ my $filebase;
+
+ # Use the MD5SUMs file as an index
+ logm("Fetch index from $baseurl/MD5SUMS");
+ my $idx = target_cmd_output_root($ho, "wget --quiet -O - $baseurl/MD5SUMS");
+ foreach (split /\n/, $idx) {
+ m/^[0-9a-f]{32} (debian-.*-$props->{FileArch}-netinst)\.iso$/ or next;
+ $filebase = $1;
+ last;
+ }
+
+ die unless $filebase;
+
+ logm("Downloading $baseurl/$filebase.jigdo");
+ # Jigdo seems to use /etc/apt/sources.list or something, so this
+ # just works using the already configured mirror without
+ # additional configuration, which is good because there doesn't
+ # seem to be any support for such things, at least in Squeeze.
+ my $netinst_jigdo = "$baseurl/$filebase.jigdo";
+ target_cmd_root($ho, <<END, 3600);
+ cd $didir && jigdo-lite --noask $netinst_jigdo
+END
+ store_runvar("$gho->{Guest}_netinst_jigdo", $netinst_jigdo);
+
+ return (<<END, "\"file:$didir/$filebase.iso,xvdd:cdrom,r\",");
+bootloader = "pygrub"
+bootloader_args = ["--kernel=$props->{IsoPath}/vmlinuz",
+ "--ramdisk=$props->{IsoPath}/initrd.gz"]
+END
+}
+
sub setup_netboot($$$)
{
my ($didir, $arch, $suite) = @_;
@@ -115,6 +162,16 @@ END
$extra_disk = "";
}
+ elsif ($method eq "netinst" )
+ {
+ my $cd = $r{"$gho->{Guest}_cd"};
+
+ logm("$method $cd/$arch");
+
+ ($method_cfg,$extra_disk) = setup_netinst($tmpdir, $arch, $cd);
+
+ $ps_url = preseed_create_guest($gho, '', CDROM=>1);
+ }
else
{
die "$method";
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 16/19] Test pygrub and pvgrub on the regular flights
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (14 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 15/19] distros: support PV guest install from Debian netinst media Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 17/19] distros: add branch infrastructure Ian Campbell
` (3 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Since we now have the ability to test these drop one of each of
pygrub, pvgrub-32 and pvgrub-64 into the standard flights. Omitting
the {Guest}_diver runvar causes ts-debian-di-install to use the d-i
images in the location configured via TftpDiVersion, so they are
Version Controlled along with the d-i version used for the host.
This adds three new jobs:
test-amd64-amd64-amd64-pvgrub:
all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
arch amd64
buildjob build-amd64
debian_arch amd64
debian_bootloader pvgrub
debian_dist wheezy
debian_method netboot
kernbuildjob build-amd64-pvops
kernkind pvops
toolstack xl
xenbuildjob build-amd64
test-amd64-amd64-i386-pvgrub:
all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
arch amd64
buildjob build-amd64
debian_arch i386
debian_bootloader pvgrub
debian_dist wheezy
debian_method netboot
kernbuildjob build-amd64-pvops
kernkind pvops
toolstack xl
xenbuildjob build-amd64
test-amd64-amd64-pygrub:
all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
arch amd64
buildjob build-amd64
debian_arch amd64
debian_bootloader pygrub
debian_dist wheezy
debian_method netboot
kernbuildjob build-amd64-pvops
kernkind pvops
toolstack xl
xenbuildjob build-amd64
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: added runvar details
---
make-flight | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/make-flight b/make-flight
index b641683..3e3c513 100755
--- a/make-flight
+++ b/make-flight
@@ -277,6 +277,42 @@ do_passthrough_tests () {
done
}
+do_pygrub_tests () {
+ if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+ return
+ fi
+
+ job_create_test test-$xenarch$kern-$dom0arch-pygrub \
+ test-debian-di xl $xenarch $dom0arch \
+ debian_arch=amd64 \
+ debian_dist=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pygrub \
+ all_hostflags=$most_hostflags
+}
+
+do_pvgrub_tests () {
+ if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+ return
+ fi
+
+ job_create_test test-$xenarch$kern-$dom0arch-amd64-pvgrub \
+ test-debian-di xl $xenarch $dom0arch \
+ debian_arch=amd64 \
+ debian_dist=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pvgrub \
+ all_hostflags=$most_hostflags \
+
+ job_create_test test-$xenarch$kern-$dom0arch-i386-pvgrub \
+ test-debian-di xl $xenarch $dom0arch \
+ debian_arch=i386 \
+ debian_dist=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pvgrub \
+ all_hostflags=$most_hostflags
+}
+
test_matrix_do_one () {
# Basic PV Linux test with xl
@@ -361,6 +397,9 @@ test_matrix_do_one () {
fi
do_passthrough_tests
+
+ do_pygrub_tests
+ do_pvgrub_tests
}
test_matrix_iterate
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 17/19] distros: add branch infrastructure
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (15 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 16/19] Test pygrub and pvgrub on the regular flights Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 18/19] distros: Run a flight over the weekend Ian Campbell
` (2 subsequent siblings)
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Since the distro nightlies are not version controlled we cannot use
the usual mechanisms for detecting regressions. Special case things
appropriately. We use an OLD_REVISION of "flight-NNN" to signify that
the old revision is another flight and not a tree revision.
A grep over $NEW_REVISION needed adjusting since NEW_REVISION is empty
in this mode, leading to "grep <filename>" which hangs waiting for
stdin.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3:
Handle within cr-daily-branch, since ap-fetch-version* don't make sense for
a branch such as this.
---
cr-daily-branch | 39 ++++++++++++++++++++++++++++++---------
cri-common | 1 +
2 files changed, 31 insertions(+), 9 deletions(-)
diff --git a/cr-daily-branch b/cr-daily-branch
index d00ecbb..5e8e51e 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -68,24 +68,36 @@ fetch_version () {
printf '%s\n' "$fetch_version_result"
}
-treeurl=`./ap-print-url $branch`
+case $branch in
+ distros)
+ treeurl=none;;
+ *)
+ treeurl=`./ap-print-url $branch`;;
+esac
force_baseline=false
skipidentical=true
wantpush=$OSSTEST_PUSH
-if [ "x$OLD_REVISION" = x ]; then
- OLD_REVISION="`./ap-fetch-version-old $branch`"
- export OLD_REVISION
-fi
-
check_tested () {
./sg-check-tested --debug --branch=$branch \
--blessings=${DAILY_BRANCH_TESTED_BLESSING:-$OSSTEST_BLESSING} \
"$@"
}
-if [ "x$OSSTEST_NO_BASELINE" != xy ] ; then
+if [ "x$OLD_REVISION" = x ]; then
+ case $branch in
+ distros)
+ OSSTEST_NO_BASELINE=y
+ OLD_REVISION=flight-`check_tested`
+ ;;
+ *) OLD_REVISION="`./ap-fetch-version-old $branch`";;
+ esac
+ export OLD_REVISION
+fi
+
+
+if [ "x$OSSTEST_NO_BASELINE" != xy ]; then
testedflight=`check_tested --revision-$tree="$OLD_REVISION"`
if [ "x$testedflight" = x ]; then
@@ -217,6 +229,11 @@ if [ "x$OLD_REVISION" = xdetermine-late ]; then
OLD_REVISION="`./ap-fetch-version-baseline-late $branch $NEW_REVISION`"
fi
+case $branch in
+distros) makeflight=./make-distros-flight ;;
+*) makeflight=./make-flight ;;
+esac
+
if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
wantpush=false
for checkbranch in x $BRANCHES_ALWAYS; do
@@ -231,7 +248,7 @@ if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
fi
$DAILY_BRANCH_PREMAKE_HOOK
-flight=`./make-flight $branch $xenbranch $OSSTEST_BLESSING "$@"`
+flight=`$makeflight $branch $xenbranch $OSSTEST_BLESSING "$@"`
$DAILY_BRANCH_POSTMAKE_HOOK
heading=tmp/$flight.heading-info
@@ -251,6 +268,10 @@ fi
revlog=tmp/$flight.revision-log
case "$NEW_REVISION/$OLD_REVISION" in
+/flight-[0-9]*)
+ echo >&2 "SGR COMPARISON AGAINST ${OLD_REVISION}"
+ sgr_args+=" --that-flight=${OLD_REVISION#flight-}"
+ ;;
*/*[^0-9a-f]* | *[^0-9a-f]*/*)
echo >&2 "NO SGR COMPARISON badchar $NEW_REVISION/$OLD_REVISION"
;;
@@ -309,7 +330,7 @@ start_email $flight $branch "$sgr_args" "$subject_prefix"
push=false
if grep '^tolerable$' $mrof >/dev/null 2>&1; then push=$wantpush; fi
if test -f $branch.force; then push=$OSSTEST_PUSH; fi
-if grep -xF $NEW_REVISION $branch.force-rev; then push=$OSSTEST_PUSH; fi
+if test -n "$NEW_REVISION" && grep -xF $NEW_REVISION $branch.force-rev; then push=$OSSTEST_PUSH; fi
if test -f $branch.block; then push=false; fi
if test -e $mrof && test -e $tree_bisect && ! grep '^broken' $mrof; then
diff --git a/cri-common b/cri-common
index 06a8d67..e107cd7 100644
--- a/cri-common
+++ b/cri-common
@@ -71,6 +71,7 @@ select_xenbranch () {
libvirt) tree=libvirt; xenbranch=xen-unstable ;;
rumpuserxen) tree=rumpuserxen; xenbranch=xen-unstable ;;
seabios) tree=seabios; xenbranch=xen-unstable ;;
+ distros) tree=none; xenbranch=xen-unstable ;;
osstest) tree=osstest; xenbranch=xen-unstable ;;
esac
if [ "x$tree" = xlinux ]; then
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 18/19] distros: Run a flight over the weekend.
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (16 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 17/19] distros: add branch infrastructure Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 19/19] Debian: Handle lack of bootloader support in d-i on ARM Ian Campbell
2014-12-01 13:24 ` [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Once a week should be sufficient for this test. It involves quite a
large number of jobs so schedule it to start early on Saturday so it
can run over the weekend when, in theory, things should be quieter.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
crontab | 1 +
1 file changed, 1 insertion(+)
diff --git a/crontab b/crontab
index 5dc8675..1917ec4 100644
--- a/crontab
+++ b/crontab
@@ -6,5 +6,6 @@ MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
18 9 * * 1,3,5 cd testing.git && BRANCHES=linux-next ./cr-for-branches branches -w "./cr-daily-branch --real"
18 4 * * * cd testing.git && BRANCHES='linux-linus linux-mingo-tip-master linux-3.0 libvirt rumpuserxen' ./cr-for-branches branches -w "./cr-daily-branch --real"
6-59/15 * * * * cd testing.git && EXTRA_BRANCHES='linux-linus linux-3.0 rumpuserxen libvirt' ./cr-for-branches bisects -w "./cr-try-bisect --real"
+46 7 * * 6 cd testing.git && BRANCHES=distros ./cr-for-branches branches -w "./cr-daily-branch --real"
#8-59/5 * * * * cd bisects/adhoc.git && with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
3 4 * * * savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [PATCH OSSTEST v3 19/19] Debian: Handle lack of bootloader support in d-i on ARM.
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (17 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 18/19] distros: Run a flight over the weekend Ian Campbell
@ 2014-12-01 12:57 ` Ian Campbell
2014-12-01 13:24 ` [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 12:57 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Debian doesn't currently know what bootloader to install in a Xen
guest on ARM. We install pv-grub-menu above which actually does what
we need, but the installer doesn't treat that as a "bootloader".
Most ARM platforms end up installing a u-boot boot.scr, based on a
platform whitelist. This doesn't seem appropriate for us. Grub is not
available for arm32. For arm64 we will eventually end up with in-guest
UEFI and therefore grub-efi and things will work normally. I'm not
sure what the answer is going to be for arm32.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: New
---
Osstest/Debian.pm | 12 ++++++++++--
ts-debian-di-install | 6 ++++--
2 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 25cacd2..9bb0a59 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -579,8 +579,8 @@ END
return $preseed;
}
-sub preseed_create_guest ($$;@) {
- my ($ho, $sfx, %xopts) = @_;
+sub preseed_create_guest ($$$;@) {
+ my ($ho, $arch, $sfx, %xopts) = @_;
my $suite= $xopts{Suite} || $c{DebianSuite};
@@ -627,6 +627,14 @@ d-i grub-installer/bootdev string /dev/xvda
END
+ # Debian doesn't currently know what bootloader to install in a
+ # Xen guest on ARM. We install pv-grub-menu above which actually
+ # does what we need, but the installer doesn't treat that as a
+ # "bootloader".
+ $preseed_file.= (<<END) if $arch =~ /^arm/ and $suite =~ /wheezy|jessie/;
+d-i nobootloader/confirmation_common note
+END
+
preseed_ssh($ho, $sfx);
preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 3878f93..c93b9ec 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -158,7 +158,9 @@ END
$suite = "sid" if $suite eq "daily";
- $ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
+ $ps_url = preseed_create_guest($gho, $arch, '',
+ Suite=>$suite,
+ PvMenuLst=>($bl eq "pvgrub"));
$extra_disk = "";
}
@@ -170,7 +172,7 @@ END
($method_cfg,$extra_disk) = setup_netinst($tmpdir, $arch, $cd);
- $ps_url = preseed_create_guest($gho, '', CDROM=>1);
+ $ps_url = preseed_create_guest($gho, $arch, '', CDROM=>1);
}
else
{
--
2.1.1
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH OSSTEST v3 00/19] add distro domU testing flight
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
` (18 preceding siblings ...)
2014-12-01 12:57 ` [PATCH OSSTEST v3 19/19] Debian: Handle lack of bootloader support in d-i on ARM Ian Campbell
@ 2014-12-01 13:24 ` Ian Campbell
19 siblings, 0 replies; 21+ messages in thread
From: Ian Campbell @ 2014-12-01 13:24 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Mon, 2014-12-01 at 12:56 +0000, Ian Campbell wrote:
> The exception to that good news is that sg-report-flight
> --that-flight=NNN MMM doesn't seem to be considering failures in MMM
> regressions vs NNN. e.g. test-amd64-i386-i386-daily-netboot-pvgrub
> should appear as a regression from 31961 to 31964 but isn't. I'm going
> to have a poke around that.
This turned out to be as easy as passing --blessings=play to
sg-report-flight (the default is ["real", "real-bisect"]), which
produced the output below (previously everything was "Tests which did
not succeed, but are not blocking")
I have a feeling this is deliberate and that only real+bisects are
supposed to be considered here on purpose.
If not then I think the right answer is just to add this to sgr_args in
the relevant case:
--blessings=${DAILY_BRANCH_TESTED_BLESSING:-$OSSTEST_BLESSING}
Ian.
------
31964: regressions - FAIL
flight 31964 distros play [play]
http://www.chiark.greenend.org.uk/~xensrcts/logs/31964/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-amd64-i386-i386-current-netinst-pygrub 5 xen-boot fail REGR. vs. 31961
test-amd64-i386-i386-daily-netboot-pvgrub 7 debian-di-install fail REGR. vs. 31961
test-amd64-i386-i386-jessie-netboot-pvgrub 5 xen-boot fail REGR. vs. 31961
Regressions which are regarded as allowable (not blocking):
test-amd64-i386-i386-weekly-netinst-pygrub 7 debian-di-install fail like 31961
Tests which did not succeed, but are not blocking:
test-amd64-amd64-i386-sid-netboot-pvgrub 7 debian-di-install fail never pass
test-amd64-amd64-amd64-jessie-netboot-pvgrub 7 debian-di-install fail never pass
test-armhf-armhf-armhf-sid-netboot-pygrub 7 debian-di-install fail never pass
test-amd64-i386-i386-sid-netboot-pygrub 7 debian-di-install fail never pass
test-armhf-armhf-armhf-jessie-netboot-pygrub 7 debian-di-install fail never pass
test-armhf-armhf-armhf-daily-netboot-pygrub 7 debian-di-install fail never pass
test-amd64-amd64-i386-jessie-netboot-pygrub 7 debian-di-install fail never pass
test-amd64-amd64-amd64-sid-netboot-pygrub 7 debian-di-install fail never pass
test-amd64-i386-amd64-jessie-netboot-pygrub 7 debian-di-install fail never pass
test-amd64-i386-amd64-sid-netboot-pygrub 7 debian-di-install fail never pass
baseline version:
flight 31961
jobs:
build-amd64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
test-amd64-amd64-amd64-daily-netboot-pvgrub pass
test-amd64-i386-i386-daily-netboot-pvgrub fail
test-amd64-amd64-amd64-jessie-netboot-pvgrub fail
test-amd64-i386-i386-jessie-netboot-pvgrub fail
test-amd64-amd64-i386-sid-netboot-pvgrub fail
test-amd64-amd64-amd64-wheezy-netboot-pvgrub pass
test-amd64-i386-i386-wheezy-netboot-pvgrub pass
test-amd64-i386-amd64-daily-netboot-pygrub pass
test-armhf-armhf-armhf-daily-netboot-pygrub fail
test-amd64-amd64-i386-daily-netboot-pygrub pass
test-amd64-i386-amd64-jessie-netboot-pygrub fail
test-armhf-armhf-armhf-jessie-netboot-pygrub fail
test-amd64-amd64-i386-jessie-netboot-pygrub fail
test-amd64-amd64-amd64-sid-netboot-pygrub fail
test-amd64-i386-amd64-sid-netboot-pygrub fail
test-armhf-armhf-armhf-sid-netboot-pygrub fail
test-amd64-i386-i386-sid-netboot-pygrub fail
test-amd64-amd64-amd64-squeeze-netboot-pygrub pass
test-amd64-i386-amd64-squeeze-netboot-pygrub pass
test-amd64-amd64-i386-squeeze-netboot-pygrub pass
test-amd64-i386-i386-squeeze-netboot-pygrub pass
test-amd64-i386-amd64-wheezy-netboot-pygrub pass
test-amd64-amd64-i386-wheezy-netboot-pygrub pass
test-amd64-amd64-amd64-current-netinst-pygrub pass
test-amd64-i386-amd64-current-netinst-pygrub pass
test-amd64-amd64-i386-current-netinst-pygrub pass
test-amd64-i386-i386-current-netinst-pygrub fail
test-amd64-amd64-amd64-weekly-netinst-pygrub pass
test-amd64-i386-amd64-weekly-netinst-pygrub pass
test-amd64-amd64-i386-weekly-netinst-pygrub pass
test-amd64-i386-i386-weekly-netinst-pygrub fail
------------------------------------------------------------
sg-report-flight on kazak.uk.xensource.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images
Logs, config files, etc. are available at
http://www.chiark.greenend.org.uk/~xensrcts/logs
Test harness code can be found at
http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2014-12-01 13:24 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-12-01 12:56 [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 01/19] TestSupport: Add helper to fetch a URL on a host Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 02/19] TestSupport: Add helper to wait for a guest to shutdown Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 03/19] TestSupport: allow overring of on_* in prepareguest_part_xencfg Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 04/19] TestSupport: allow caller of prepareguest_part_xencfg to specify viftype Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 05/19] create_webfile: Support use with guests as well as hosts Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 06/19] Debian: refactor code to add preseed commands to the preseed file Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 07/19] Debian: refactor preseeding of .ssh directories Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 08/19] Debian: ensure preseed base comment remains at actual end of base Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 09/19] Debian: Refactor installation of overlays, so it can be used for guests too Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 10/19] Debian: add preseed_create_guest helper Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 11/19] make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs() Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 12/19] distros: add support for installing Debian PV guests via d-i, flight and jobs Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 13/19] distros: support booting Debian PV (d-i installed) guests with pvgrub Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 14/19] distros: Support pvgrub for Wheezy too Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 15/19] distros: support PV guest install from Debian netinst media Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 16/19] Test pygrub and pvgrub on the regular flights Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 17/19] distros: add branch infrastructure Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 18/19] distros: Run a flight over the weekend Ian Campbell
2014-12-01 12:57 ` [PATCH OSSTEST v3 19/19] Debian: Handle lack of bootloader support in d-i on ARM Ian Campbell
2014-12-01 13:24 ` [PATCH OSSTEST v3 00/19] add distro domU testing flight Ian Campbell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.