* [PATCH OSSTEST v7 00/16] add distro domU testing flight
@ 2015-05-27 13:44 Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host Ian Campbell
` (14 more replies)
0 siblings, 15 replies; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:44 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
Hi,
Since v6 I've refactored into multiple flights (one per Debian release)
as well as moving all the configuration of URLs etc into make flight, so
the test scripts are now generic. I also defaulted to running on the
Cambridge instance as discussed.
I think I've dropped the acks where appropriate, since some of those
involved some bigger refactoring.
More details in the individual patches.
An initial portion of cleanup/refactoring has been applied so it's a
shorter series this time.
Summary of (A)cks, (M)odified and (N)ew (NM==Replaced something):
M TestSupport: Add helper to fetch a URL on a host
M distros: add support for installing Debian PV guests via d-i, flight and jobs
AM distros: support booting Debian PV (d-i installed) guests with pvgrub.
M distros: Support pvgrub for Wheezy too.
M distros: support PV guest install from Debian netinst media.
AM Test pygrub and pvgrub on the regular flights
M distros: add branch infrastructure
MN distros: Run one suite per day on a weekly basis
Debian: Handle lack of bootloader support in d-i on ARM.
ts-debian-di-install: Refactor root_disk specification
make-flight: refactor PV debian tests
M Add testing of non-LVM/phy disk backends.
mfi-common: Allow make-*flight to filter the set of build jobs to include
make-distros-flight: don't bother building for XSM or libvirt.
make-distros-flight: Use ftp.debian.org directly
Ian.
adhoc xen-unstable test result, followed by one for
distros-debian-wheezy as an example (which reused a build job due to the
way in invoked it, by default it would build too, but I wanted to save
time).
------8>-----------------------------
37589: regressions - FAIL
flight 37589 xen-unstable play [play]
http://osstest.xs.citrite.net/~osstest/testlogs/logs/37589/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop fail REGR. vs. 37590
test-amd64-amd64-xl-qemut-winxpsp3 9 windows-install fail REGR. vs. 37590
test-amd64-amd64-xl-qemuu-winxpsp3 9 windows-install fail REGR. vs. 37590
Tests which did not succeed, but are not blocking:
test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm 9 debian-hvm-install fail never pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm 9 debian-hvm-install fail never pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm 9 debian-hvm-install fail never pass
test-amd64-amd64-rumpuserxen-amd64 15 rumpuserxen-demo-xenstorels/xenstorels.repeat fail never pass
test-amd64-amd64-libvirt-xsm 11 guest-start fail never pass
test-armhf-armhf-libvirt-xsm 11 guest-start fail never pass
test-armhf-armhf-xl-xsm 11 guest-start fail never pass
test-amd64-i386-xl-qemut-debianhvm-amd64-xsm 9 debian-hvm-install fail never pass
test-amd64-i386-libvirt 12 migrate-support-check fail never pass
test-amd64-amd64-xl-pvh-amd 11 guest-start fail never pass
test-amd64-amd64-xl-pvh-intel 11 guest-start fail never pass
test-amd64-i386-libvirt-xsm 11 guest-start fail never pass
test-amd64-amd64-xl-xsm 11 guest-start fail never pass
test-amd64-i386-xl-xsm 11 guest-start fail never pass
test-armhf-armhf-xl-sedf 12 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 12 migrate-support-check fail never pass
test-armhf-armhf-xl-sedf-pin 12 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 12 migrate-support-check fail never pass
test-armhf-armhf-xl-midway 12 migrate-support-check fail never pass
test-armhf-armhf-xl 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt 12 migrate-support-check fail never pass
test-amd64-amd64-libvirt 12 migrate-support-check fail never pass
test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop fail never pass
test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop fail never pass
test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop fail never pass
test-armhf-armhf-xl-raw 9 debian-di-install fail never pass
test-amd64-amd64-xl-raw 16 guest-localmigrate/x10 fail never pass
test-amd64-i386-libvirt-qcow2 11 migrate-support-check fail never pass
test-amd64-amd64-pygrub 10 guest-start fail never pass
test-armhf-armhf-libvirt-qcow2 9 debian-di-install fail never pass
test-amd64-i386-libvirt-raw 9 debian-di-install fail never pass
test-armhf-armhf-libvirt-raw 9 debian-di-install fail never pass
test-amd64-i386-libvirt-vhd 11 migrate-support-check fail never pass
test-amd64-amd64-libvirt-vhd 11 migrate-support-check fail never pass
test-amd64-amd64-libvirt-raw 11 migrate-support-check fail never pass
test-amd64-amd64-libvirt-qcow2 11 migrate-support-check fail never pass
test-armhf-armhf-libvirt-vhd 9 debian-di-install fail never pass
test-armhf-armhf-xl-qcow2 9 debian-di-install fail never pass
test-armhf-armhf-xl-vhd 9 debian-di-install fail never pass
baseline version:
flight 37590
jobs:
build-amd64-xsm pass
build-armhf-xsm pass
build-i386-xsm pass
build-amd64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-oldkern pass
build-i386-oldkern pass
build-amd64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
build-amd64-rumpuserxen pass
build-i386-rumpuserxen pass
test-amd64-amd64-xl pass
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm fail
test-amd64-i386-xl-qemut-debianhvm-amd64-xsm fail
test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm fail
test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm fail
test-amd64-amd64-libvirt-xsm fail
test-armhf-armhf-libvirt-xsm fail
test-amd64-i386-libvirt-xsm fail
test-amd64-amd64-xl-xsm fail
test-armhf-armhf-xl-xsm fail
test-amd64-i386-xl-xsm fail
test-amd64-amd64-xl-pvh-amd fail
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-rumpuserxen-amd64 fail
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-amd64-amd64-xl-credit2 pass
test-armhf-armhf-xl-credit2 pass
test-amd64-i386-freebsd10-i386 pass
test-amd64-i386-rumpuserxen-i386 pass
test-amd64-amd64-xl-pvh-intel fail
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-armhf-armhf-libvirt pass
test-amd64-i386-libvirt pass
test-armhf-armhf-xl-midway pass
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-xl-sedf-pin pass
test-armhf-armhf-xl-sedf-pin pass
test-amd64-amd64-amd64-pvgrub pass
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-pygrub fail
test-amd64-amd64-libvirt-qcow2 pass
test-armhf-armhf-libvirt-qcow2 fail
test-amd64-i386-libvirt-qcow2 pass
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-xl-qcow2 fail
test-amd64-i386-xl-qcow2 pass
test-amd64-amd64-libvirt-raw pass
test-armhf-armhf-libvirt-raw fail
test-amd64-i386-libvirt-raw fail
test-amd64-amd64-xl-raw fail
test-armhf-armhf-xl-raw fail
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-sedf pass
test-armhf-armhf-xl-sedf pass
test-amd64-i386-xl-qemut-winxpsp3-vcpus1 pass
test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-libvirt-vhd fail
test-amd64-i386-libvirt-vhd pass
test-amd64-amd64-xl-vhd pass
test-armhf-armhf-xl-vhd fail
test-amd64-i386-xl-vhd pass
test-amd64-amd64-xl-qemut-winxpsp3 fail
test-amd64-i386-xl-qemut-winxpsp3 pass
test-amd64-amd64-xl-qemuu-winxpsp3 fail
test-amd64-i386-xl-qemuu-winxpsp3 pass
------------------------------------------------------------
sg-report-flight on osstest.xs.citrite.net
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images
Logs, config files, etc. are available at
http://osstest.xs.citrite.net/~osstest/testlogs/logs
Test harness code can be found at
http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
-----8>-----------
flight 37595 distros-debian-wheezy play [play]
http://osstest.xs.citrite.net/~osstest/testlogs/logs/37595/
Failures :-/ but no regressions.
Tests which did not succeed, but are not blocking:
test-amd64-i386-i386-wheezy-netboot-pvgrub 10 guest-start fail never pass
baseline version:
flight 37567
jobs:
build-amd64 pass
build-armhf pass
build-i386 pass
build-amd64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
test-amd64-amd64-amd64-wheezy-netboot-pvgrub pass
test-amd64-i386-i386-wheezy-netboot-pvgrub fail
test-amd64-i386-amd64-wheezy-netboot-pygrub pass
test-amd64-amd64-i386-wheezy-netboot-pygrub pass
------------------------------------------------------------
sg-report-flight on osstest.xs.citrite.net
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images
Logs, config files, etc. are available at
http://osstest.xs.citrite.net/~osstest/testlogs/logs
Test harness code can be found at
http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
Push not applicable.
^ permalink raw reply [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 14:48 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 02/15] distros: add support for installing Debian PV guests via d-i, flight and jobs Ian Campbell
` (13 subsequent siblings)
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v7: Quote $url and $path, switch to a heredoc to avoid resulting over
long line
v5: Support http_proxy via $c{HttpProxy}
v3: Make sure wget is installed
---
Osstest/Debian.pm | 2 +-
Osstest/TestSupport.pm | 12 +++++++++++-
2 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 537ccbe..673ceba 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -685,7 +685,7 @@ d-i apt-setup/another boolean false
d-i apt-setup/non-free boolean false
d-i apt-setup/contrib boolean false
-d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, $extra_packages
+d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bin, wget, $extra_packages
$xopts{ExtraPreseed}
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index abb3195..d84ca51 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -55,7 +55,7 @@ BEGIN {
target_putfilecontents_stash
target_putfilecontents_root_stash
target_put_guest_image target_editfile
- target_editfile_cancel
+ target_editfile_cancel target_fetchurl
target_editfile_root target_file_exists
target_editfile_kvp_replace
target_run_apt
@@ -1589,6 +1589,16 @@ END
return $cfgpath;
}
+sub target_fetchurl($$$;$) {
+ my ($ho, $url, $path, $timeo) = @_;
+ $timeo ||= 2000;
+ my $useproxy = "export http_proxy=$c{HttpProxy};" if $c{HttpProxy};
+ target_cmd_root($ho, <<END, $timeo);
+ $useproxy wget --progress=dot:mega -O \"$path\" \"$url\"
+END
+}
+
+
sub target_put_guest_image ($$;$) {
my ($ho, $gho, $default) = @_;
my $specimage = $r{"$gho->{Guest}_image"};
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 02/15] distros: add support for installing Debian PV guests via d-i, flight and jobs
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 14:51 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 03/15] distros: support booting Debian PV (d-i installed) guests with pvgrub Ian Campbell
` (12 subsequent siblings)
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
This patch introduces ts-debian-di-install which can install Debian
from a netboot (PXE) debian installer image. By default it installs
from the d-i image used by osstest (using the special Xen PV guest
enabled flavour where necessary) but it can also fetch the kernel and
ramdisk from URLs specified in runvars. The resulting guests boot the
distro kernel using pygrub (pvgrub will follow).
The distros flights differ substantially from the existing flights.
Introduce make-distros-flight using the functionality previously
refactored into mfi-common. The new flight tests all versions of
Debian from Squeeze onward as an amd64, i386 and armhf guests (armhf
from Jessie onwards only) using the usual smoke tests.
Test names are suffixed -pygrub pending the addition of pvgrub
variants in a future commit.
Add the new cases to sg-run-job
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v7: Use {Guest}_suite as runvar. Also use $suite not $dist in
make-distros-flight for consistency.
Switch to a flight per Debian suite model rather than one
enourmous flight.
Switch to constructing the URLs in make-distros-flight
v6: Only apply -xen suffix to x86 images when doing a netboot using
the osstest version of d-i, since that is the only arch where we
create such files, other arches can use the bare names.
Use the guest $arch not the host $r{arch} when finding the
kernel+initrd to use for d-i install using the osstest d-i.
v4: use guest create
v3: $BUILD_LVEXTEND_MAX now handled in mfi-common
Consolidate setting of ruvars
Include $flight and $job in tmpdir name
Use Osstest::Debian::di_installcmdline_core
Document the usage of get_host_property on a guest object
Correct ARM netboot paths
Include bootloader in test name
Should include -pv too?
console= repetition for Jessie onwards.
Wait for up to an hour for the install. I'd seen timeouts right at
the end of the install with the previous value
---
Osstest/TestSupport.pm | 3 +
make-distros-flight | 126 ++++++++++++++++++++++++++++++++++++++
sg-run-job | 11 ++++
ts-debian-di-install | 161 +++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 301 insertions(+)
create mode 100755 make-distros-flight
create mode 100755 ts-debian-di-install
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index d84ca51..1b29445 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -934,8 +934,11 @@ sub propname_massage ($) {
return $prop;
}
+# It is fine to call this on a guest object too, in which case it will
+# always return $defval.
sub get_host_property ($$;$) {
my ($ho, $prop, $defval) = @_;
+ return $defval unless $ho->{Properties};
my $val = $ho->{Properties}{propname_massage($prop)};
return defined($val) ? $val : $defval;
}
diff --git a/make-distros-flight b/make-distros-flight
new file mode 100755
index 0000000..586737c
--- /dev/null
+++ b/make-distros-flight
@@ -0,0 +1,126 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. cri-common
+. ap-common
+. mfi-common
+
+defsuite=`getconfig DebianSuite`
+defguestsuite=`getconfig GuestDebianSuite`
+
+case $branch in
+ distros-debian-*) debian_suite=${branch#distros-debian-} ;;
+ *) echo $branch >&2; exit 1 ;;
+esac
+
+if [ x$buildflight = x ]; then
+
+ WANT_XEND=false REVISION_LINUX_OLD=disable
+
+ create_build_jobs
+
+else
+
+ bfi=$buildflight.
+
+fi
+
+job_create_test_filter_callback () {
+ if [ "$xenarch" = "i386" ]; then return 1; fi
+ return 0
+}
+
+test_matrix_branch_filter_callback () {
+ :
+}
+
+test_do_one_netboot () {
+ local diurl=
+
+ if [ x$gsuite = xsnapshot ] ; then
+ diurl="http://d-i.debian.org/daily-images/$domU/daily/netboot"
+ gsuite=sid
+ gver=daily
+ else
+ local mirror="http://`getconfig DebianMirrorHost`/`getconfig DebianMirrorSubpath`"
+ diurl="$mirror/dists/$gsuite/main/installer-$domU/current/images/netboot"
+ gver=$gsuite
+ fi
+
+ case ${domU}_${gsuite} in
+ armhf_squeeze) return;; # No armhf in Squeeze
+ armhf_wheezy) return;; # No armhf guest support in Wheezy
+ *) ;;
+ esac
+
+ case $domU in
+ i386|amd64)
+ diurl="$diurl/xen";;
+ arm64)
+ diurl="$diurl/debian-installer/arm64";;
+ esac
+
+ job_create_test \
+ test-$xenarch$kern-$dom0arch-$domU-$gver-netboot-pygrub \
+ test-debian-di xl $xenarch $dom0arch \
+ kernbuildjob=${bfi}build-$dom0arch-$kernbuild \
+ debian_arch=$domU \
+ debian_suite=$gsuite \
+ debian_method=netboot \
+ debian_netboot_kernel="$diurl/vmlinuz" \
+ debian_netboot_ramdisk="$diurl/initrd.gz" \
+ all_hostflags=$most_hostflags
+}
+
+test_matrix_do_one () {
+ case ${xenarch} in
+ amd64) domUarches="amd64 i386";;
+ armhf) domUarches="armhf";;
+ esac
+
+ for domU in $domUarches ; do
+
+ if [ -n $debian_suite ] ; then
+ gsuite=$debian_suite
+
+ test_do_one_netboot
+
+ fi
+
+ done
+}
+
+test_matrix_iterate
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:
diff --git a/sg-run-job b/sg-run-job
index d53fd83..bba52ed 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -258,6 +258,17 @@ proc run-job/test-debian {} {
test-guest debian
}
+proc install-guest-debian-di {} {
+ run-ts . = ts-debian-di-install
+ run-ts . = ts-guest-start + debian
+}
+
+proc need-hosts/test-debian-di {} { return host }
+proc run-job/test-debian-di {} {
+ install-guest-debian-di
+ test-guest debian
+}
+
proc need-hosts/test-freebsd {} { return host }
proc run-job/test-freebsd {} {
run-ts . = ts-freebsd-install
diff --git a/ts-debian-di-install b/ts-debian-di-install
new file mode 100755
index 0000000..ed3c348
--- /dev/null
+++ b/ts-debian-di-install
@@ -0,0 +1,161 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost,$gn) = @ARGV;
+$whhost ||= 'host';
+$gn ||= 'debian';
+
+our $ho= selecthost($whhost);
+
+our $ram_mb= 512;
+our $disk_mb= 10000;
+
+our $guesthost= "$gn.guest.osstest";
+our $gho;
+
+sub prep () {
+ target_install_packages_norec($ho, qw(lvm2));
+
+ $gho= prepareguest($ho, $gn, $guesthost, 22,
+ $disk_mb, 40);
+
+ prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
+
+ target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+}
+
+sub setup_netboot($$$)
+{
+ my ($didir, $arch, $suite) = @_;
+
+ my ($kernel,$ramdisk);
+
+ if ( $r{ "$gho->{Guest}_netboot_kernel" } &&
+ $r{ "$gho->{Guest}_netboot_ramdisk" } ) {
+ target_fetchurl($ho, $r{ "$gho->{Guest}_netboot_kernel" },
+ "$didir/kernel_${suite}_${arch}");
+ target_fetchurl($ho, $r{ "$gho->{Guest}_netboot_ramdisk" },
+ "$didir/ramdisk_${suite}_${arch}");
+ } else {
+ # Both or neither must be specified
+ die if $r{ "$gho->{Guest}_netboot_kernel" }
+ || $r{ "$gho->{Guest}_netboot_ramdisk" };
+
+ my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.${arch}.'/'.$c{TftpDiVersion}.'-'.$ho->{Suite};
+
+ my $suffix = '';
+ $suffix .= "-xen" if ${arch} =~ m/amd64|i386/;
+ $kernel = "$di_path/vmlinuz$suffix";
+ $ramdisk = "$di_path/initrd.gz$suffix";
+
+ target_putfile_root($ho, 60, $kernel, "$didir/kernel_${suite}_${arch}");
+ target_putfile_root($ho, 60, $ramdisk, "$didir/ramdisk_${suite}_${arch}");
+
+ store_runvar("$gho->{Guest}_netboot_kernel", $kernel);
+ store_runvar("$gho->{Guest}_netboot_ramdisk", $ramdisk);
+ }
+
+ return <<END;
+kernel = "$didir/kernel_${suite}_${arch}"
+ramdisk = "$didir/ramdisk_${suite}_${arch}"
+END
+}
+
+sub ginstall () {
+ my $arch= $r{"$gho->{Guest}_arch"};
+ my $method= $r{"$gho->{Guest}_method"};
+
+ my $tmpdir= "/root/$flight-$job-di";
+ target_cmd_root($ho, <<END);
+rm -rf $tmpdir
+mkdir $tmpdir
+END
+
+ my ($method_cfg, $ps_url, $extra_disk);
+
+ if ( $method eq "netboot" )
+ {
+ my $suite= $r{"$gho->{Guest}_suite"};
+ logm("$method $suite/$arch");
+
+ $method_cfg = setup_netboot($tmpdir, $arch, $suite);
+
+ $ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+
+ $extra_disk = "";
+ }
+ else
+ {
+ die "$method";
+ }
+
+ my @cmdline = ();
+ push @cmdline, "debian-installer/exit/always_halt=true";
+ push @cmdline, "domain=$c{TestHostDomain}";
+ push @cmdline, "console=hvc0";
+ push @cmdline, di_installcmdline_core($gho, $ps_url);
+ push @cmdline, "--";
+ # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007 for
+ # why this is repeated.
+ push @cmdline, "console=hvc0";
+
+ my $cmdline = join(" ", @cmdline);
+
+ my %install_xopts = (
+ OnPowerOff => "preserve"
+ );
+
+ prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
+$method_cfg
+extra = "$cmdline"
+#
+disk = [
+ $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+ ]
+END
+
+ guest_create($gho);
+
+ guest_checkrunning($ho, $gho) or die "$gho->{Name} not running";
+
+ guest_await_shutdown($ho,$gho,3600);
+ guest_destroy($gho);
+
+ my $blcfg = <<END;
+bootloader = "pygrub"
+END
+
+ prepareguest_part_xencfg($ho, $gho, $ram_mb, {}, <<END);
+$blcfg
+#
+disk = [
+ 'phy:$gho->{Lvdev},xvda,w'
+ ]
+END
+ return;
+}
+
+prep();
+ginstall();
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 03/15] distros: support booting Debian PV (d-i installed) guests with pvgrub.
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 02/15] distros: add support for installing Debian PV guests via d-i, flight and jobs Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too Ian Campbell
` (11 subsequent siblings)
14 siblings, 0 replies; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
This requires the use of the pv-grub-menu package which is in Jessie
onwards. (it is in wheezy-backports which is the subject of a
subsequent patch).
The bootloader to use is specified via a runvar {Guest}_bootloader.
Adjust make-distros-flight to use pvgrub for some subset of i386 and
amd64 guests to get coverage.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v7: Move definition of $extra_packages variable to here which is its
first usage.
Use {Guest}_suite not {Guest}_dist as runvar to choose version.
v3: Define and use arch_debian2xen and arch_xen2debian
Avoid pv-grub-x86_64.gz on i386 dom0, we don't built it there.
Fiddle with py vs pv grub stripy a bit.
---
Osstest.pm | 7 +++++++
Osstest/Debian.pm | 4 +++-
make-distros-flight | 20 +++++++++++++++++++-
ts-debian-di-install | 11 +++++++++--
4 files changed, 38 insertions(+), 4 deletions(-)
diff --git a/Osstest.pm b/Osstest.pm
index e8bd77b..e189017 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -38,6 +38,7 @@ BEGIN {
db_begin_work
ensuredir get_filecontents_core_quiet system_checked
nonempty visible_undef show_abs_time
+ %arch_debian2xen %arch_xen2debian
);
%EXPORT_TAGS = ( );
@@ -49,6 +50,12 @@ our $mjobdb;
our $dbh_tests;
+our %arch_debian2xen = qw(i386 x86_32
+ amd64 x86_64
+ armhf armhf);
+our %arch_xen2debian;
+$arch_xen2debian{$arch_debian2xen{$_}} = $_ foreach keys %arch_debian2xen;
+
#---------- static default config settings ----------
our %c = qw(
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 673ceba..a3c5244 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -709,7 +709,9 @@ sub preseed_create_guest ($$;@) {
my $suite= $xopts{Suite} || $c{DebianSuite};
- my $preseed_file= preseed_base($ho, $suite, $sfx, '', %xopts);
+ my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
+
+ my $preseed_file= preseed_base($ho, $suite, $sfx, $extra_packages, %xopts);
$preseed_file.= (<<END);
d-i partman-auto/method string regular
d-i partman-auto/choose_recipe \\
diff --git a/make-distros-flight b/make-distros-flight
index 586737c..5fc8552 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -78,6 +78,11 @@ test_do_one_netboot () {
*) ;;
esac
+ stripy bootloader pvgrub pygrub \
+ "$xenarch" "amd64" \
+ "$dom0arch" "i386" \
+ "$domU" "amd64" \
+
case $domU in
i386|amd64)
diurl="$diurl/xen";;
@@ -85,8 +90,20 @@ test_do_one_netboot () {
diurl="$diurl/debian-installer/arm64";;
esac
+ case ${dom0arch}_${domU}_${gsuite} in
+ arm*_arm*_*) bootloader="pygrub";; # no pvgrub for arm
+
+ # Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
+ # workedaround in Jessie+ with pv-grub-menu package.
+ *_squeeze) bootloader="pygrub";;
+ *_wheezy) bootloader="pygrub";;
+
+ # pv-grub-x86_64.gz is not built by 32-bit dom0 userspace build.
+ i386_amd64_*) bootloader="pygrub";;
+ esac
+
job_create_test \
- test-$xenarch$kern-$dom0arch-$domU-$gver-netboot-pygrub \
+ test-$xenarch$kern-$dom0arch-$domU-$gver-netboot-$bootloader \
test-debian-di xl $xenarch $dom0arch \
kernbuildjob=${bfi}build-$dom0arch-$kernbuild \
debian_arch=$domU \
@@ -94,6 +111,7 @@ test_do_one_netboot () {
debian_method=netboot \
debian_netboot_kernel="$diurl/vmlinuz" \
debian_netboot_ramdisk="$diurl/initrd.gz" \
+ debian_bootloader=$bootloader \
all_hostflags=$most_hostflags
}
diff --git a/ts-debian-di-install b/ts-debian-di-install
index ed3c348..14167c8 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -88,6 +88,8 @@ sub ginstall () {
my $method= $r{"$gho->{Guest}_method"};
my $tmpdir= "/root/$flight-$job-di";
+ my $bl= $r{"$gho->{Guest}_bootloader"};
+
target_cmd_root($ho, <<END);
rm -rf $tmpdir
mkdir $tmpdir
@@ -102,7 +104,7 @@ END
$method_cfg = setup_netboot($tmpdir, $arch, $suite);
- $ps_url = preseed_create_guest($gho, '', Suite=>$suite);
+ $ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
$extra_disk = "";
}
@@ -143,7 +145,12 @@ END
guest_await_shutdown($ho,$gho,3600);
guest_destroy($gho);
- my $blcfg = <<END;
+ my $xenarch = $arch_debian2xen{$arch};
+ my $pvgrub = "/usr/local/lib/xen/boot/pv-grub-$xenarch.gz";
+ my $blcfg = $bl eq "pvgrub" ? <<END : <<END;
+kernel = "$pvgrub"
+extra = "(hd0,0)/boot/grub/menu.lst"
+END
bootloader = "pygrub"
END
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too.
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (2 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 03/15] distros: support booting Debian PV (d-i installed) guests with pvgrub Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 14:55 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 05/15] distros: support PV guest install from Debian netinst media Ian Campbell
` (10 subsequent siblings)
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
This requires us to install pv-grub-menu from backports, which we do
using a late_command.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v7:
- Remove vestigial attempts to enable -backports via d-i preseeding.
v3:
- Remove spurious () from <<(END) (and the prexisting "" too)
- Remove $xopts{EnableBackports} and automatically handle the need to add
backports in preseed_base.
- Install via late_command not apt-setup, since the former has
issues, hence subject drops "attempt to..."
---
Osstest/Debian.pm | 33 ++++++++++++++++++++++++++++++++-
make-distros-flight | 4 ++--
2 files changed, 34 insertions(+), 3 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index a3c5244..0975e73 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -709,7 +709,38 @@ sub preseed_create_guest ($$;@) {
my $suite= $xopts{Suite} || $c{DebianSuite};
- my $extra_packages = "pv-grub-menu" if $xopts{PvMenuLst};
+ my $extra_packages = "";
+ if ($xopts{PvMenuLst}) {
+ if ($suite =~ m/wheezy/) {
+ # pv-grub-menu/wheezy-backports + using apt-setup to add
+ # backports results in iproute, ifupdown and
+ # isc-dhcp-client getting removed because tasksel's
+ # invocation of apt-get install somehow decides the
+ # iproute2 from wheezy-backports is a thing it wants to
+ # install. So instead lets fake it with a late command...
+ #
+ # This also has the bonus of working round an issue with
+ # 1.2.1~bpo70+1 which created an invalid menu.lst using
+ # "root(/dev/xvda,0)" which pvgrub cannot parse because
+ # the Grub device.map isn't present at pkgsel/include time
+ # but it is by late_command time. This was fixed by
+ # version 1.3 which is in Jessie onwards.
+ preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+(
+ echo
+ echo \\\# $suite backports
+ echo deb http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
+) >> /target/etc/apt/sources.list
+in-target apt-get update
+in-target apt-get install -y -t wheezy-backports pv-grub-menu
+END
+ } else {
+ $extra_packages = "pv-grub-menu";
+ }
+ }
my $preseed_file= preseed_base($ho, $suite, $sfx, $extra_packages, %xopts);
$preseed_file.= (<<END);
diff --git a/make-distros-flight b/make-distros-flight
index 5fc8552..06b14f2 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -94,9 +94,9 @@ test_do_one_netboot () {
arm*_arm*_*) bootloader="pygrub";; # no pvgrub for arm
# Needs a menu.lst, not present in Squeeze+ due to switch to grub2,
- # workedaround in Jessie+ with pv-grub-menu package.
+ # workedaround in Wheezy+ with pv-grub-menu package (backports in Wheezy,
+ # in Jessie+ main).
*_squeeze) bootloader="pygrub";;
- *_wheezy) bootloader="pygrub";;
# pv-grub-x86_64.gz is not built by 32-bit dom0 userspace build.
i386_amd64_*) bootloader="pygrub";;
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 05/15] distros: support PV guest install from Debian netinst media.
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (3 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 15:01 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 06/15] Test pygrub and pvgrub on the regular flights Ian Campbell
` (9 subsequent siblings)
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
The netinst media are iso images containing a base Debian install and
some (image size dependent) additional tasks.
On x86 the "multiarch" iso flavour contains a Xen capable kernel for
both i386 and amd64 so use that.
This adds support for two classes of ISO, the CD sized ones which are
built nightly and the DVD sized ones which are built weekly.
The images are downloaded using jigdo which sources the majority of
the data from a local Debian mirror, for this reason I have not
worried about the fact that the i386 and amd64 tests are downloading
the same thing (adding a specific download job would require finding
up to 4GB of scratch space for each flight).
The ISOs booted using pygrub which can extract the kernel and initrd
from a ISO image. The resulting guests are also booted with pygrub
since the pv-grub-menu package is not available on the ISO images and
we have pvgrub coverage from the netboot tests.
This is configured from make*flight using runvars:
- <gident>_netinst_baseurl:
Base URL of directory containing the netinst jigdo images.
- <gident>_netinst_filere:
Regular expression to match actual file within baseurl to use
(without .jigdo suffix)
<gident>_netinst_kernel:
Path to kernel within the netinst image
<gident>_netinst_ramdisk
Path to ramdisk within the netinst image
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v7: Use {Guest}_suite not {Guest}_dist as runvar to choose version.
Use shorter lined method for $arch_props definition
Switch to flight-per-suite model
Handle lack of both $c{DebianMirrorProxy} and $c{HttpProxy}
Construct URLs in make-distros-flight
v4: Support http_proxy in jigdo download
v3:
- Indent jigdo-lite script line to improve readability
- Wrap bootloader_args
- Fetch URL on target, to get a timeout. Use wget since that is arranged to
be present.
- include bootloader in test name.
---
Osstest/Debian.pm | 7 ++++--
make-distros-flight | 54 ++++++++++++++++++++++++++++++++++++++++++++
ts-debian-di-install | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 123 insertions(+), 2 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 0975e73..30b1b3a 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -623,8 +623,6 @@ sub preseed_base ($$$$;@) {
preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
my $preseed = <<"END";
-d-i mirror/suite string $suite
-
d-i debian-installer/locale string en_GB
d-i console-keymaps-at/keymap select gb
d-i keyboard-configuration/xkb-keymap string en_GB
@@ -696,6 +694,11 @@ END
d-i clock-setup/ntp-server string $ntpserver
END
+ # For CDROM the suite is part of the image
+ $preseed .= <<END unless $xopts{CDROM};
+d-i mirror/suite string $suite
+END
+
$preseed .= <<"END";
### END OF DEBIAN PRESEED BASE
diff --git a/make-distros-flight b/make-distros-flight
index 06b14f2..4b9f100 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -115,6 +115,48 @@ test_do_one_netboot () {
all_hostflags=$most_hostflags
}
+test_do_one_netinst () {
+ local path_arch
+ case $domU in
+ amd64|i386) path_arch="multi-arch"; file_arch="amd64-i386";;
+ *) path_arch="$domU"; file_arch="$domU";;
+ esac
+ case $domU in
+ amd64) iso_path="/install.amd/xen";;
+ i386) iso_path="/install.386/xen";;
+ *) iso_path="/install.$domU";;
+ esac
+
+ local cdurl=
+ case $cd in
+ current)
+ cdurl="http://cdimage.debian.org/debian-cd/current/${path_arch}/jigdo-cd"
+ ;;
+ weekly)
+ cdurl="http://cdimage.debian.org/cdimage/weekly-builds/${path_arch}/jigdo-cd"
+ ;;
+ *)
+ echo "cd $cd?"
+ exit 1
+ ;;
+ esac
+
+ # Always pygrub since no pv-grub-menu on CD
+ job_create_test \
+ test-$xenarch$kern-$dom0arch-$domU-$cd-netinst-pygrub \
+ test-debian-di xl $xenarch $dom0arch \
+ kernbuildjob=${bfi}build-$dom0arch-$kernbuild \
+ debian_arch=$domU \
+ debian_method=netinst \
+ debian_netinst_baseurl=$cdurl \
+ debian_netinst_filere="debian-.*-${file_arch}-netinst" \
+ debian_netinst_kernel="${iso_path}/vmlinuz" \
+ debian_netinst_ramdisk="${iso_path}/initrd.gz" \
+ debian_bootloader=pygrub \
+ all_hostflags=$most_hostflags
+
+}
+
test_matrix_do_one () {
case ${xenarch} in
amd64) domUarches="amd64 i386";;
@@ -130,6 +172,18 @@ test_matrix_do_one () {
fi
+ if [ $debian_suite = "snapshot" ]; then
+ gsuite=sid
+ for cd in current weekly ; do
+ case ${domU}_${gsuite} in
+ armhf_*) continue;; # No iso targets for armhf
+ *) ;;
+ esac
+
+ test_do_one_netinst
+
+ done
+ fi
done
}
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 14167c8..22c1930 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -46,6 +46,60 @@ sub prep () {
target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
}
+sub setup_netinst($$$)
+{
+ my ($didir, $arch, $cd) = @_;
+
+ target_install_packages($ho, qw(jigdo-file));
+
+ my $baseurl = $r{ "$gho->{Guest}_netinst_baseurl" };
+ my $filere = $r{ "$gho->{Guest}_netinst_filere" };
+ my $kernel = $r{ "$gho->{Guest}_netinst_kernel" };
+ my $ramdisk = $r{ "$gho->{Guest}_netinst_ramdisk" };
+
+ die "params" unless $baseurl && $filere && $kernel && $ramdisk;
+
+ my $filebase;
+
+ # Use the MD5SUMs file as an index
+ logm("Fetch index from $baseurl/MD5SUMS");
+ my $idx = target_cmd_output_root($ho, "wget --quiet -O - $baseurl/MD5SUMS");
+ foreach (split /\n/, $idx) {
+ m/^[0-9a-f]{32} ($filere)\.iso$/ or next;
+ $filebase = $1;
+ last;
+ }
+
+ die unless $filebase;
+
+ logm("Downloading $baseurl/$filebase.jigdo");
+ my $netinst_jigdo = "$baseurl/$filebase.jigdo";
+ # Jigdo uses wget internally, and so obeys $http_proxy. This seems
+ # simpler than the advice at
+ # https://www.debian.org/CD/jigdo-cd/#faq which is to edit
+ # ~/.jigdo-lite and change the wgetOpts setting.
+ my $useproxy = $c{DebianMirrorProxy} // $c{HttpProxy} // '';
+ my $cmd = '';
+ $cmd .= <<END if $useproxy;
+ export http_proxy=$useproxy
+END
+ $cmd .= <<END;
+ cd $didir && jigdo-lite --noask $netinst_jigdo
+END
+ # Jigdo seems to use /etc/apt/sources.list or something, so this
+ # just works using the already configured mirror without
+ # additional configuration, which is good because there doesn't
+ # seem to be any support for such things, at least in Squeeze.
+ target_cmd_root($ho, $cmd, 3600);
+ store_runvar("$gho->{Guest}_netinst_jigdo", $netinst_jigdo);
+
+ return (<<END, "\"file:$didir/$filebase.iso,xvdd:cdrom,r\",");
+bootloader = "pygrub"
+bootloader_args = ["--kernel=$kernel",
+ "--ramdisk=$ramdisk"]
+END
+}
+
sub setup_netboot($$$)
{
my ($didir, $arch, $suite) = @_;
@@ -108,6 +162,16 @@ END
$extra_disk = "";
}
+ elsif ($method eq "netinst" )
+ {
+ my $cd = $r{"$gho->{Guest}_cd"};
+
+ logm("$method $cd/$arch");
+
+ ($method_cfg,$extra_disk) = setup_netinst($tmpdir, $arch, $cd);
+
+ $ps_url = preseed_create_guest($gho, '', CDROM=>1);
+ }
else
{
die "$method";
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 06/15] Test pygrub and pvgrub on the regular flights
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (4 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 05/15] distros: support PV guest install from Debian netinst media Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 07/15] distros: add branch infrastructure Ian Campbell
` (8 subsequent siblings)
14 siblings, 0 replies; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Since we now have the ability to test these drop one of each of
pygrub, pvgrub-32 and pvgrub-64 into the standard flights. Omitting
the {Guest}_diver runvar causes ts-debian-di-install to use the d-i
images in the location configured via TftpDiVersion, so they are
Version Controlled along with the d-i version used for the host.
This adds three new jobs:
+test-amd64-amd64-amd64-pvgrub all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+test-amd64-amd64-amd64-pvgrub arch amd64
+test-amd64-amd64-amd64-pvgrub buildjob build-amd64
+test-amd64-amd64-amd64-pvgrub debian_arch amd64
+test-amd64-amd64-amd64-pvgrub debian_bootloader pvgrub
+test-amd64-amd64-amd64-pvgrub debian_method netboot
+test-amd64-amd64-amd64-pvgrub debian_suite wheezy
+test-amd64-amd64-amd64-pvgrub kernbuildjob build-amd64-pvops
+test-amd64-amd64-amd64-pvgrub kernkind pvops
+test-amd64-amd64-amd64-pvgrub toolstack xl
+test-amd64-amd64-amd64-pvgrub xenbuildjob build-amd64
+test-amd64-amd64-i386-pvgrub all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+test-amd64-amd64-i386-pvgrub arch amd64
+test-amd64-amd64-i386-pvgrub buildjob build-amd64
+test-amd64-amd64-i386-pvgrub debian_arch i386
+test-amd64-amd64-i386-pvgrub debian_bootloader pvgrub
+test-amd64-amd64-i386-pvgrub debian_method netboot
+test-amd64-amd64-i386-pvgrub debian_suite wheezy
+test-amd64-amd64-i386-pvgrub kernbuildjob build-amd64-pvops
+test-amd64-amd64-i386-pvgrub kernkind pvops
+test-amd64-amd64-i386-pvgrub toolstack xl
+test-amd64-amd64-i386-pvgrub xenbuildjob build-amd64
+test-amd64-amd64-pygrub all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+test-amd64-amd64-pygrub arch amd64
+test-amd64-amd64-pygrub buildjob build-amd64
+test-amd64-amd64-pygrub debian_arch amd64
+test-amd64-amd64-pygrub debian_bootloader pygrub
+test-amd64-amd64-pygrub debian_method netboot
+test-amd64-amd64-pygrub debian_suite wheezy
+test-amd64-amd64-pygrub kernbuildjob build-amd64-pvops
+test-amd64-amd64-pygrub kernkind pvops
+test-amd64-amd64-pygrub toolstack xl
+test-amd64-amd64-pygrub xenbuildjob build-amd64
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v7: Use {Guest}_suite not {Guest}_dist as runvar to choose version.
Refreshed runvars i ncommit message.
v3: added runvar details
---
make-flight | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/make-flight b/make-flight
index 8a1fceb..99fb1ee 100755
--- a/make-flight
+++ b/make-flight
@@ -313,6 +313,42 @@ do_passthrough_tests () {
done
}
+do_pygrub_tests () {
+ if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+ return
+ fi
+
+ job_create_test test-$xenarch$kern-$dom0arch-pygrub \
+ test-debian-di xl $xenarch $dom0arch \
+ debian_arch=amd64 \
+ debian_suite=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pygrub \
+ all_hostflags=$most_hostflags
+}
+
+do_pvgrub_tests () {
+ if [ $xenarch != amd64 -o $dom0arch != amd64 -o "$kern" != "" ]; then
+ return
+ fi
+
+ job_create_test test-$xenarch$kern-$dom0arch-amd64-pvgrub \
+ test-debian-di xl $xenarch $dom0arch \
+ debian_arch=amd64 \
+ debian_suite=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pvgrub \
+ all_hostflags=$most_hostflags \
+
+ job_create_test test-$xenarch$kern-$dom0arch-i386-pvgrub \
+ test-debian-di xl $xenarch $dom0arch \
+ debian_arch=i386 \
+ debian_suite=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pvgrub \
+ all_hostflags=$most_hostflags
+}
+
do_pv_debian_tests () {
xsms=$(xenbranch_xsm_variants)
@@ -448,6 +484,9 @@ test_matrix_do_one () {
fi
#do_passthrough_tests
+
+ do_pygrub_tests
+ do_pvgrub_tests
}
test_matrix_iterate
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 07/15] distros: add branch infrastructure
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (5 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 06/15] Test pygrub and pvgrub on the regular flights Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 15:04 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 08/15] distros: Run one suite per day on a weekly basis Ian Campbell
` (7 subsequent siblings)
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Since the distro nightlies are not version controlled we cannot use
the usual mechanisms for detecting regressions. Special case things
appropriately. We use an OLD_REVISION of "flight-NNN" to signify that
the old revision is another flight and not a tree revision.
A grep over $NEW_REVISION needed adjusting since NEW_REVISION is empty
in this mode, leading to "grep <filename>" which hangs waiting for
stdin.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v7:
Handle empty $NEW_REVISION by quoting it instead of a needless test -n
Switch to flight-per-suite model
v3:
Handle within cr-daily-branch, since ap-fetch-version* don't make sense for
a branch such as this.
---
cr-daily-branch | 36 ++++++++++++++++++++++++++++--------
cri-common | 1 +
2 files changed, 29 insertions(+), 8 deletions(-)
diff --git a/cr-daily-branch b/cr-daily-branch
index c7d1071..99a2315 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -68,23 +68,34 @@ fetch_version () {
printf '%s\n' "$fetch_version_result"
}
-treeurl=`./ap-print-url $branch`
+case $branch in
+ distros-*)
+ treeurl=none;;
+ *)
+ treeurl=`./ap-print-url $branch`;;
+esac
force_baseline=false
skipidentical=true
wantpush=$OSSTEST_PUSH
-if [ "x$OLD_REVISION" = x ]; then
- OLD_REVISION="`./ap-fetch-version-old $branch`"
- export OLD_REVISION
-fi
-
check_tested () {
./sg-check-tested --debug --branch=$branch \
--blessings=${DAILY_BRANCH_TESTED_BLESSING:-$OSSTEST_BLESSING} \
"$@"
}
+if [ "x$OLD_REVISION" = x ]; then
+ case $branch in
+ distros-*)
+ OSSTEST_NO_BASELINE=y
+ OLD_REVISION=flight-`check_tested`
+ ;;
+ *) OLD_REVISION="`./ap-fetch-version-old $branch`";;
+ esac
+ export OLD_REVISION
+fi
+
if [ "x$OSSTEST_NO_BASELINE" != xy ] ; then
testedflight=`check_tested --revision-$tree="$OLD_REVISION"`
@@ -227,6 +238,11 @@ if [ "x$OLD_REVISION" = xdetermine-late ]; then
OLD_REVISION="`./ap-fetch-version-baseline-late $branch $NEW_REVISION`"
fi
+case $branch in
+distros-*) makeflight=./make-distros-flight ;;
+*) makeflight=./make-flight ;;
+esac
+
if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
wantpush=false
for checkbranch in x $BRANCHES_ALWAYS; do
@@ -241,7 +257,7 @@ if [ "x$NEW_REVISION" = "x$OLD_REVISION" ]; then
fi
$DAILY_BRANCH_PREMAKE_HOOK
-flight=`./make-flight $branch $xenbranch $OSSTEST_BLESSING "$@"`
+flight=`$makeflight $branch $xenbranch $OSSTEST_BLESSING "$@"`
$DAILY_BRANCH_POSTMAKE_HOOK
heading=tmp/$flight.heading-info
@@ -261,6 +277,10 @@ fi
revlog=tmp/$flight.revision-log
case "$NEW_REVISION/$OLD_REVISION" in
+/flight-[0-9]*)
+ echo >&2 "SGR COMPARISON AGAINST ${OLD_REVISION}"
+ sgr_args+=" --that-flight=${OLD_REVISION#flight-}"
+ ;;
*/*[^0-9a-f]* | *[^0-9a-f]*/*)
echo >&2 "NO SGR COMPARISON badchar $NEW_REVISION/$OLD_REVISION"
;;
@@ -321,7 +341,7 @@ start_email $flight $branch "$sgr_args" "$subject_prefix"
push=false
if grep '^tolerable$' $mrof >/dev/null 2>&1; then push=$wantpush; fi
if test -f $branch.force; then push=$OSSTEST_PUSH; fi
-if grep -xF $NEW_REVISION $branch.force-rev; then push=$OSSTEST_PUSH; fi
+if grep -xF "$NEW_REVISION" $branch.force-rev; then push=$OSSTEST_PUSH; fi
if test -f $branch.block; then push=false; fi
if test -e $mrof && test -e $tree_bisect && ! grep '^broken' $mrof; then
diff --git a/cri-common b/cri-common
index ad44546..58b08f2 100644
--- a/cri-common
+++ b/cri-common
@@ -72,6 +72,7 @@ select_xenbranch () {
rumpuserxen) tree=rumpuserxen; xenbranch=xen-unstable ;;
seabios) tree=seabios; xenbranch=xen-unstable ;;
ovmf) tree=ovmf; xenbranch=xen-unstable ;;
+ distros-*) tree=none; xenbranch=xen-unstable ;;
osstest) tree=osstest; xenbranch=xen-unstable ;;
esac
if [ "x$tree" = xlinux ]; then
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 08/15] distros: Run one suite per day on a weekly basis
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (6 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 07/15] distros: add branch infrastructure Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 15:05 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 09/15] Debian: Handle lack of bootloader support in d-i on ARM Ian Campbell
` (6 subsequent siblings)
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Once a week should be sufficient for these tests. Perhaps in the
future we will want to increase the frequency for the suites under
active development (testing, unstable)
For now run this on the Citrix Cambridge instance until the XenProject
instance has sufficient capacity.
Switch to tabs for alignment.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v7: Replaces "distros: Run a flight over the weekend."
Now run in Cambridge
Run separate flight per-suite
Dropped Ack
---
crontab-cambridge | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/crontab-cambridge b/crontab-cambridge
index 60bb4fd..7d3ed57 100644
--- a/crontab-cambridge
+++ b/crontab-cambridge
@@ -1,5 +1,10 @@
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=ian.jackson@citrix.com,ian.campbell@eu.citrix.com
-# m h dom mon dow command
-4-59/30 * * * * cd testing.git && BRANCHES=osstest ./cr-for-branches branches -q "./cr-daily-branch --real"
-3 4 * * * savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
+# m h dom mon dow command
+4-59/30 * * * * cd testing.git && BRANCHES='osstest' ./cr-for-branches branches -q "./cr-daily-branch --real"
+46 7 * * 6 cd testing.git && BRANCHES='distros-debian-snapshot' ./cr-for-branches branches -w "./cr-daily-branch --real"
+46 7 * * 5 cd testing.git && BRANCHES='distros-debian-sid' ./cr-for-branches branches -w "./cr-daily-branch --real"
+46 7 * * 4 cd testing.git && BRANCHES='distros-debian-jessie' ./cr-for-branches branches -w "./cr-daily-branch --real"
+46 7 * * 3 cd testing.git && BRANCHES='distros-debian-wheezy' ./cr-for-branches branches -w "./cr-daily-branch --real"
+46 7 * * 2 cd testing.git && BRANCHES='distros-debian-squeeze' ./cr-for-branches branches -w "./cr-daily-branch --real"
+3 4 * * * savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 09/15] Debian: Handle lack of bootloader support in d-i on ARM.
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (7 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 08/15] distros: Run one suite per day on a weekly basis Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 15:07 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 10/15] ts-debian-di-install: Refactor root_disk specification Ian Campbell
` (5 subsequent siblings)
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Debian doesn't currently know what bootloader to install in a Xen
guest on ARM. We install pv-grub-menu above which actually does what
we need, but the installer doesn't treat that as a "bootloader".
Most ARM platforms end up installing a u-boot boot.scr, based on a
platform whitelist. This doesn't seem appropriate for us. Grub is not
available for arm32. For arm64 we will eventually end up with in-guest
UEFI and therefore grub-efi and things will work normally. I'm not
sure what the answer is going to be for arm32.
This patch enables the workaround for Wheezy, Jessie and Sid,
post-Jessie should be enabled as we add them. (Pre-wheezy does not
support running as a Xen guest on ARM so we don't test them at all).
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Handle sid too
v3: New
---
Osstest/Debian.pm | 14 ++++++++++++--
ts-debian-di-install | 6 ++++--
2 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 30b1b3a..ee24e4e 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -707,8 +707,8 @@ END
return $preseed;
}
-sub preseed_create_guest ($$;@) {
- my ($ho, $sfx, %xopts) = @_;
+sub preseed_create_guest ($$$;@) {
+ my ($ho, $arch, $sfx, %xopts) = @_;
my $suite= $xopts{Suite} || $c{DebianSuite};
@@ -755,6 +755,16 @@ d-i grub-installer/bootdev string /dev/xvda
END
+ # Debian doesn't currently know what bootloader to install in a
+ # Xen guest on ARM. We install pv-grub-menu above which actually
+ # does what we need, but the installer doesn't treat that as a
+ # "bootloader".
+ logm("\$arch is $arch, \$suite is $suite");
+ $preseed_file.= (<<END) if $arch =~ /^arm/ && $suite =~ /wheezy|jessie|sid/;
+d-i nobootloader/confirmation_common boolean true
+
+END
+
$preseed_file .= preseed_hook_cmds();
return create_webfile($ho, "preseed$sfx", $preseed_file);
diff --git a/ts-debian-di-install b/ts-debian-di-install
index 22c1930..da6ab73 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -158,7 +158,9 @@ END
$method_cfg = setup_netboot($tmpdir, $arch, $suite);
- $ps_url = preseed_create_guest($gho, '', Suite=>$suite, PvMenuLst=>($bl eq "pvgrub"));
+ $ps_url = preseed_create_guest($gho, $arch, '',
+ Suite=>$suite,
+ PvMenuLst=>($bl eq "pvgrub"));
$extra_disk = "";
}
@@ -170,7 +172,7 @@ END
($method_cfg,$extra_disk) = setup_netinst($tmpdir, $arch, $cd);
- $ps_url = preseed_create_guest($gho, '', CDROM=>1);
+ $ps_url = preseed_create_guest($gho, $arch, '', CDROM=>1);
}
else
{
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 10/15] ts-debian-di-install: Refactor root_disk specification
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (8 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 09/15] Debian: Handle lack of bootloader support in d-i on ARM Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 15:07 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 11/15] make-flight: refactor PV debian tests Ian Campbell
` (4 subsequent siblings)
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: new patch
---
ts-debian-di-install | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/ts-debian-di-install b/ts-debian-di-install
index da6ab73..bb2d2d2 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -195,12 +195,14 @@ END
OnPowerOff => "preserve"
);
+ my $root_disk = "'phy:$gho->{Lvdev},xvda,w'";
+
prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
$method_cfg
extra = "$cmdline"
#
disk = [
- $extra_disk 'phy:$gho->{Lvdev},xvda,w'
+ $extra_disk $root_disk
]
END
@@ -224,7 +226,7 @@ END
$blcfg
#
disk = [
- 'phy:$gho->{Lvdev},xvda,w'
+ $root_disk
]
END
return;
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 11/15] make-flight: refactor PV debian tests
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (9 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 10/15] ts-debian-di-install: Refactor root_disk specification Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 15:08 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends Ian Campbell
` (3 subsequent siblings)
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
No functional change, standalone-generate-dump-flight-runvars confirms
no change to the runvars.
Includes a hook which is not used yet, $recipe_sfx.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: new patch
---
make-flight | 24 ++++++++++++++----------
1 file changed, 14 insertions(+), 10 deletions(-)
diff --git a/make-flight b/make-flight
index 99fb1ee..8206508 100755
--- a/make-flight
+++ b/make-flight
@@ -349,6 +349,17 @@ do_pvgrub_tests () {
all_hostflags=$most_hostflags
}
+do_pv_debian_test_one () {
+ testname=$1; shift
+ recipe_sfx=$1; shift
+ toolstack=$1; shift
+
+ job_create_test test-$xenarch$kern-$dom0arch-$testname \
+ test-debian$recipe_sfx $toolstack \
+ $xenarch $dom0arch \
+ $debian_runvars all_hostflags=$most_hostflags $@
+}
+
do_pv_debian_tests () {
xsms=$(xenbranch_xsm_variants)
@@ -364,20 +375,13 @@ do_pv_debian_tests () {
suffix=${platform:+-$platform}
hostflags=${most_hostflags}${platform:+,platform-$platform}
- job_create_test test-$xenarch$kern-$dom0arch-xl$suffix \
- test-debian xl \
- $xenarch $dom0arch \
- enable_xsm=$xsm \
- $debian_runvars all_hostflags=$hostflags
+ do_pv_debian_test_one xl$suffix '' xl enable_xsm=$xsm
+
done
done
for xsm in $xsms ; do
- job_create_test test-$xenarch$kern-$dom0arch-libvirt \
- test-debian libvirt \
- $xenarch $dom0arch \
- enable_xsm=$xsm \
- $debian_runvars all_hostflags=$most_hostflags
+ do_pv_debian_test_one libvirt '' libvirt enable_xsm=$xsm
done
}
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends.
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (10 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 11/15] make-flight: refactor PV debian tests Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 15:21 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include Ian Campbell
` (2 subsequent siblings)
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
xen-create-image makes this tricky to do since it is rather LVM
centric. Now that we have the ability to install from d-i it's
possible to arrange fairly easily that they use non-LVM disk backend
arrangements.
Here we add support to the test script and infra and create a bunch of
new jobs testing the cross product of {xl,libvirt} x {raw,qcow2,vhd}.
The test scripts are modified such that when constructing a domain
with a non-LVM diskfmt runvar:
- the LVM device is slightly enlarged to account for file format
headers (1M should be plenty).
- the LVM device will have an ext3 filesystem created on it instead
of being used as a phy device for the guest. Reusing the LVM volume
in this way means we don't need to do more storage management in
dom0 (i.e. arranging for / to be large enough, or managing a
special "images" LV)
- the relevant type of container is created within the filesystem
using the appropriate tool.
- New properties Disk{fmt,spec} are added to all $gho, containing
the format used for the root disk and the xl diskspec to load it.
- lvm backed guests use a xend/xm compatible spec, everything
else uses the improved xl syntax which libvirt also supports.
We won't test non-LVM on xend.
- New properties Disk{mnt,img} are added to $gho which are not using
LVM. These contain the mount point to use (configurable via
OSSTEST_CONFIG and runvars) and the full path (including mount
point) to the image itself.
- When starting or stopping a guest we arrange for the filesystem to
be (u)mounted.
- The prepearation when starting a guest copes gracefully with
the disk already being prepared.
- Hooks are called from guest_create() and guest_destroy() to
manipulate the disk as needed.
Using standalong-generate-dump-flight-runvars a representative set of
runvars is:
+test-amd64-amd64-xl-qcow2 all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+test-amd64-amd64-xl-qcow2 arch amd64
+test-amd64-amd64-xl-qcow2 buildjob build-amd64
+test-amd64-amd64-xl-qcow2 debian_arch amd64
+test-amd64-amd64-xl-qcow2 debian_bootloader pygrub
+test-amd64-amd64-xl-qcow2 debian_diskfmt qcow2
+test-amd64-amd64-xl-qcow2 debian_kernkind pvops
+test-amd64-amd64-xl-qcow2 debian_method netboot
+test-amd64-amd64-xl-qcow2 debian_suite wheezy
+test-amd64-amd64-xl-qcow2 kernbuildjob build-amd64-pvops
+test-amd64-amd64-xl-qcow2 kernkind pvops
+test-amd64-amd64-xl-qcow2 toolstack xl
+test-amd64-amd64-xl-qcow2 xenbuildjob build-amd64
Compared to test-amd64-amd64-pygrub (which is the most similar job) and
normalising the test name the difference is:
test-amd64-amd64-SUFFIX all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
test-amd64-amd64-SUFFIX arch amd64
test-amd64-amd64-SUFFIX buildjob build-amd64
test-amd64-amd64-SUFFIX debian_arch amd64
test-amd64-amd64-SUFFIX debian_bootloader pygrub
+test-amd64-amd64-SUFFIX debian_diskfmt qcow2
+test-amd64-amd64-SUFFIX debian_kernkind pvops
test-amd64-amd64-SUFFIX debian_method netboot
test-amd64-amd64-SUFFIX debian_suite wheezy
test-amd64-amd64-SUFFIX kernbuildjob build-amd64-pvops
test-amd64-amd64-SUFFIX kernkind pvops
test-amd64-amd64-SUFFIX toolstack xl
test-amd64-amd64-SUFFIX xenbuildjob build-amd64
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v7: Use the right arch for tests, not always amd64 (doesn't work well
on arm!)
Defer guest_find_diskimg until _vg runvar and thence Lvdev are
setup:
selectguest calls guest_find_lv then guest_find_diskimg, using
preexisting runvars.
But prepare_guest calls selectguest before setting disk_lv, so
Lvdev ends up undefined, after setting disk_lv prepare_guest
calls guest_find_lv+guest_find_diskimg again and things get
configured.
This follows how guest_find_lv only sets Lvdev iff Vg and Lv
are both set.
Use {Guest}_suite not {Guest}_dist as runvar to choose version.
Assume slower dd for raw population, since I was still seeing
timeouts, assume at worst 1/2 the speed I happened to see in a
local test
Refresh the runvars in the commit log and drop the list of flights.
v6: Use bs=1MB (=1*1000) when creating the raw images instead of bs=1M
(=1*1024), this matches the units used by lvcreate's -L option and
therefore arranges that the imaage actually fits.
v5: Assume 100MB/s dd from /dev/zero when creating a raw disk image
Allow 10M of slack on filesystem for raw, qcow and vhd. 1M wasn't
enough in practice for raw.
v4: new patch
---
Osstest/TestSupport.pm | 100 ++++++++++++++++++++++++++++++++++++++++++++++++-
make-flight | 16 ++++++++
ts-debian-di-install | 10 ++---
ts-guest-start | 1 -
4 files changed, 117 insertions(+), 10 deletions(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 1b29445..df48338 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -91,7 +91,8 @@ BEGIN {
target_var target_var_prefix
selectguest prepareguest more_prepareguest_hvm
guest_var guest_var_commalist
- prepareguest_part_lvmdisk prepareguest_part_xencfg
+ prepareguest_part_lvmdisk prepareguest_part_diskimg
+ prepareguest_part_xencfg
guest_umount_lv guest_await guest_await_dhcp_tcp
guest_checkrunning guest_check_ip guest_find_ether
guest_find_domid guest_check_up guest_check_up_quick
@@ -99,6 +100,7 @@ BEGIN {
guest_await_shutdown guest_await_destroy guest_destroy
guest_vncsnapshot_begin guest_vncsnapshot_stash
guest_check_remus_ok guest_editconfig
+ guest_prepare_disk guest_unprepare_disk
host_involves_pcipassthrough host_get_pcipassthrough_devs
toolstack guest_create
@@ -1347,6 +1349,7 @@ sub selectguest ($$) {
}
logm("guest: using $gn on $gho->{Host}{Name}");
guest_find_lv($gho);
+ guest_find_diskimg($gho);
guest_find_ether($gho);
guest_find_tcpcheckport($gho);
dhcp_watch_setup($ho,$gho);
@@ -1362,6 +1365,25 @@ sub guest_find_lv ($) {
? '/dev/'.$gho->{Vg}.'/'.$gho->{Lv} : undef;
}
+sub guest_find_diskimg($)
+{
+ my ($gho) = @_;
+
+ return unless $gho->{Lvdev};
+
+ $gho->{Diskfmt} = $r{"$gho->{Guest}_diskfmt"} // "lvm";
+ $gho->{Diskspec} = "phy:$gho->{Lvdev},xvda,w";
+
+ return if $gho->{Diskfmt} eq "lvm";
+
+ my $mntroot = get_host_property($gho->{Host}, "DiskImageMount",
+ $c{DiskImageMount} // "/var/lib/xen/images");
+
+ $gho->{Diskmnt} = "$mntroot/$gho->{Guest}";
+ $gho->{Diskimg} = "$gho->{Diskmnt}/disk.$gho->{Diskfmt}";
+ $gho->{Diskspec} = "format=$gho->{Diskfmt},vdev=xvda,target=$gho->{Diskimg}";
+}
+
sub guest_find_ether ($) {
my ($gho) = @_;
$gho->{Ether}= $r{"$gho->{Guest}_ether"};
@@ -1401,6 +1423,7 @@ sub guest_destroy ($) {
my ($gho) = @_;
my $ho = $gho->{Host};
toolstack($ho)->destroy($gho);
+ guest_unprepare_disk($gho);
}
sub guest_await_destroy ($$) {
@@ -1412,9 +1435,32 @@ sub guest_await_destroy ($$) {
sub guest_create ($) {
my ($gho) = @_;
my $ho = $gho->{Host};
+ guest_prepare_disk($gho);
toolstack($ho)->create($gho);
}
+sub guest_prepare_disk ($) {
+ my ($gho) = @_;
+
+ guest_umount_lv($gho->{Host}, $gho);
+
+ return if $gho->{Diskfmt} eq "lvm";
+
+ target_cmd_root($gho->{Host}, <<END);
+mkdir -p $gho->{Diskmnt}
+mount $gho->{Lvdev} $gho->{Diskmnt};
+END
+}
+
+sub guest_unprepare_disk ($) {
+ my ($gho) = @_;
+ return if $gho->{Diskfmt} eq "lvm";
+ target_cmd_root($gho->{Host}, <<END);
+umount $gho->{Lvdev} || :
+END
+}
+
+
sub target_choose_vg ($$) {
my ($ho, $mbneeded) = @_;
@@ -1546,6 +1592,7 @@ sub prepareguest ($$$$$$) {
}
guest_find_lv($gho);
+ guest_find_diskimg($gho);
guest_find_ether($gho);
guest_find_tcpcheckport($gho);
return $gho;
@@ -1556,7 +1603,56 @@ sub prepareguest_part_lvmdisk ($$$) {
target_cmd_root($ho, "lvremove -f $gho->{Lvdev} ||:");
target_cmd_root($ho, "lvcreate -L ${disk_mb}M -n $gho->{Lv} $gho->{Vg}");
target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Lvdev} count=10");
-}
+}
+
+sub make_vhd ($$$) {
+ my ($ho, $gho, $disk_mb) = @_;
+ target_cmd_root($ho, "vhd-util create -n $gho->{Rootimg} -s $disk_mb");
+}
+sub make_qcow2 ($$$) {
+ my ($ho, $gho, $disk_mb) = @_;
+ # upstream qemu's version. Seems preferable to qemu-xen-img from qemu-trad.
+ my $qemu_img = "/usr/local/lib/xen/bin/qemu-img";
+ target_cmd_root($ho, "$qemu_img create -f qcow2 $gho->{Rootimg} ${disk_mb}M");
+}
+sub make_raw ($$$) {
+ my ($ho, $gho, $disk_mb) = @_;
+ # In local tests this reported 130MB/s, so calculate a timeout assuming 65MB/s.
+ target_cmd_root($ho, "dd if=/dev/zero of=$gho->{Rootimg} bs=1MB count=${disk_mb}",
+ ${disk_mb} / 65);
+}
+
+sub prepareguest_part_diskimg ($$$) {
+ my ($ho, $gho, $disk_mb) = @_;
+
+ my $diskfmt = $gho->{Diskfmt};
+ # Allow an extra 10 megabytes for image format headers
+ my $disk_overhead = $diskfmt eq "lvm" ? 0 : 10;
+
+ logm("preparing guest disks in $diskfmt format");
+
+ target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+
+ prepareguest_part_lvmdisk($ho, $gho, $disk_mb + $disk_overhead);
+
+ if ($diskfmt ne "lvm") {
+
+ $gho->{Rootimg} = "$gho->{Diskmnt}/disk.$diskfmt";
+ $gho->{Rootcfg} = "format=$diskfmt,vdev=xvda,target=$gho->{Rootimg}";
+
+ target_cmd_root($ho, <<END);
+mkfs.ext3 $gho->{Lvdev}
+mkdir -p $gho->{Diskmnt}
+mount $gho->{Lvdev} $gho->{Diskmnt}
+END
+ no strict qw(refs);
+ &{"make_$diskfmt"}($ho, $gho, $disk_mb);
+
+ target_cmd_root($ho, <<END);
+umount $gho->{Lvdev}
+END
+ }
+}
sub prepareguest_part_xencfg ($$$$$) {
my ($ho, $gho, $ram_mb, $xopts, $cfgrest) = @_;
diff --git a/make-flight b/make-flight
index 8206508..0313696 100755
--- a/make-flight
+++ b/make-flight
@@ -383,6 +383,22 @@ do_pv_debian_tests () {
for xsm in $xsms ; do
do_pv_debian_test_one libvirt '' libvirt enable_xsm=$xsm
done
+
+ for ts in xl libvirt ; do
+
+ for fmt in raw vhd qcow2 ; do
+
+ fmt_runvar="debian_diskfmt=$fmt"
+
+ do_pv_debian_test_one $ts-$fmt '-di' $ts \
+ debian_arch=$dom0arch \
+ debian_suite=$guestsuite \
+ debian_method=netboot \
+ debian_bootloader=pygrub \
+ $fmt_runvar
+
+ done
+ done
}
test_matrix_do_one () {
diff --git a/ts-debian-di-install b/ts-debian-di-install
index bb2d2d2..ab833e4 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -41,9 +41,7 @@ sub prep () {
$gho= prepareguest($ho, $gn, $guesthost, 22,
$disk_mb, 40);
- prepareguest_part_lvmdisk($ho, $gho, $disk_mb);
-
- target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
+ prepareguest_part_diskimg($ho, $gho, $disk_mb);
}
sub setup_netinst($$$)
@@ -195,14 +193,12 @@ END
OnPowerOff => "preserve"
);
- my $root_disk = "'phy:$gho->{Lvdev},xvda,w'";
-
prepareguest_part_xencfg($ho, $gho, $ram_mb, \%install_xopts, <<END);
$method_cfg
extra = "$cmdline"
#
disk = [
- $extra_disk $root_disk
+ $extra_disk '$gho->{Diskspec}'
]
END
@@ -226,7 +222,7 @@ END
$blcfg
#
disk = [
- $root_disk
+ '$gho->{Diskspec}'
]
END
return;
diff --git a/ts-guest-start b/ts-guest-start
index 1aa9e69..a434720 100755
--- a/ts-guest-start
+++ b/ts-guest-start
@@ -25,7 +25,6 @@ tsreadconfig();
our ($ho,$gho) = ts_get_host_guest(@ARGV);
sub start () {
- guest_umount_lv($ho, $gho);
guest_create($gho);
}
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (11 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 15:22 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 14/15] make-distros-flight: don't bother building for XSM or libvirt Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 15/15] make-distros-flight: Use ftp.debian.org directly Ian Campbell
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
By using the same job_create_build(_filter_callback) scheme used for
the test jobs.
Will be used in make-distros-flight.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
make-distros-flight | 4 ++++
make-flight | 4 ++++
mfi-common | 21 +++++++++++++++------
3 files changed, 23 insertions(+), 6 deletions(-)
diff --git a/make-distros-flight b/make-distros-flight
index 4b9f100..c61f5c8 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -38,6 +38,10 @@ case $branch in
*) echo $branch >&2; exit 1 ;;
esac
+job_create_build_filter_callback () {
+ :
+}
+
if [ x$buildflight = x ]; then
WANT_XEND=false REVISION_LINUX_OLD=disable
diff --git a/make-flight b/make-flight
index 0313696..f97fb73 100755
--- a/make-flight
+++ b/make-flight
@@ -34,6 +34,10 @@ flight=`./cs-flight-create $blessing $branch`
defsuite=`getconfig DebianSuite`
defguestsuite=`getconfig GuestDebianSuite`
+job_create_build_filter_callback () {
+ :
+}
+
if [ x$buildflight = x ]; then
create_build_jobs
diff --git a/mfi-common b/mfi-common
index a9e966f..a100afb 100644
--- a/mfi-common
+++ b/mfi-common
@@ -54,6 +54,15 @@ xenbranch_xsm_variants () {
esac
}
+job_create_build () {
+ job_create_build_filter_callback "$@" || return 0
+
+ local job=$1; shift
+ local recipe=$1; shift
+
+ ./cs-job-create $flight $job $recipe "$@"
+}
+
create_build_jobs () {
local arch
@@ -164,7 +173,7 @@ create_build_jobs () {
else
xsm_suffix=""
fi
- ./cs-job-create $flight build-$arch$xsm_suffix build \
+ job_create_build build-$arch$xsm_suffix build \
arch=$arch enable_xend=$build_defxend enable_ovmf=$enable_ovmf\
enable_xsm=$enable_xsm \
tree_qemu=$TREE_QEMU \
@@ -183,7 +192,7 @@ create_build_jobs () {
done
if [ $build_extraxend = "true" ] ; then
- ./cs-job-create $flight build-$arch-xend build \
+ job_create_build build-$arch-xend build \
arch=$arch enable_xend=true enable_ovmf=$enable_ovmf \
tree_qemu=$TREE_QEMU \
tree_qemuu=$TREE_QEMU_UPSTREAM \
@@ -196,7 +205,7 @@ create_build_jobs () {
revision_qemuu=$REVISION_QEMU_UPSTREAM
fi
- ./cs-job-create $flight build-$arch-pvops build-kern \
+ job_create_build build-$arch-pvops build-kern \
arch=$arch kconfighow=xen-enable-xen-config \
$RUNVARS $BUILD_RUNVARS $BUILD_LINUX_RUNVARS $arch_runvars \
$suite_runvars \
@@ -208,7 +217,7 @@ create_build_jobs () {
if [ "x$REVISION_LIBVIRT" != xdisable ]; then
- ./cs-job-create $flight build-$arch-libvirt build-libvirt \
+ job_create_build build-$arch-libvirt build-libvirt \
arch=$arch \
tree_xen=$TREE_XEN \
$RUNVARS $BUILD_RUNVARS $BUILD_LIBVIRT_RUNVARS $arch_runvars \
@@ -223,7 +232,7 @@ create_build_jobs () {
case $arch in
i386|amd64)
- ./cs-job-create $flight build-$arch-rumpuserxen build-rumpuserxen \
+ job_create_build build-$arch-rumpuserxen build-rumpuserxen \
arch=$arch \
tree_xen=$TREE_XEN \
$RUNVARS $BUILD_RUNVARS $BUILD_RUMPUSERXEN_RUNVARS $arch_runvars \
@@ -252,7 +261,7 @@ create_build_jobs () {
if [ "x$REVISION_LINUX_OLD" != xdisable ]; then
- ./cs-job-create $flight build-$arch-oldkern build-kern \
+ job_create_build build-$arch-oldkern build-kern \
arch=$arch kconfighow=create-config-sh \
kimagefile=vmlinux \
$RUNVARS $BUILD_RUNVARS $BUILD_LINUX_OLD_RUNVARS \
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 14/15] make-distros-flight: don't bother building for XSM or libvirt.
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (12 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 15:22 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 15/15] make-distros-flight: Use ftp.debian.org directly Ian Campbell
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
make-distros-flight | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/make-distros-flight b/make-distros-flight
index c61f5c8..49f4b60 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -39,7 +39,15 @@ case $branch in
esac
job_create_build_filter_callback () {
- :
+ local job=$1; shift
+
+ case "$job" in
+ build-*-libvirt) return 1;;
+ esac
+ case " $* " in
+ *" enable_xsm=true "*) return 1;;
+ esac
+ return 0
}
if [ x$buildflight = x ]; then
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [PATCH OSSTEST v7 15/15] make-distros-flight: Use ftp.debian.org directly
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
` (13 preceding siblings ...)
2015-05-27 13:45 ` [PATCH OSSTEST v7 14/15] make-distros-flight: don't bother building for XSM or libvirt Ian Campbell
@ 2015-05-27 13:45 ` Ian Campbell
2015-07-06 15:23 ` Ian Jackson
14 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-05-27 13:45 UTC (permalink / raw)
To: ian.jackson; +Cc: Ian Campbell, xen-devel
The local proxy seems to serve stale packages for Jessie etc, I blame
the intercepting cache on the way out of our network, similar to
b5f15136900d "mg-debian-installer-update: workaround caching proxies",
except it is between the apt-cache and the world not the osstest vm
and the world.
Since the netboot kernel+initrd are reasonably small, these flights
are infrequent and they are intended to test the current upstream
version I think this is tollerable.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
make-distros-flight | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/make-distros-flight b/make-distros-flight
index 49f4b60..d407fcb 100755
--- a/make-distros-flight
+++ b/make-distros-flight
@@ -79,7 +79,9 @@ test_do_one_netboot () {
gsuite=sid
gver=daily
else
- local mirror="http://`getconfig DebianMirrorHost`/`getconfig DebianMirrorSubpath`"
+ #local mirror="http://`getconfig DebianMirrorHost`/`getconfig DebianMirrorSubpath`"
+ # XXX local mirror seems to serve up stale files.
+ local mirror="http://ftp.debian.org/debian"
diurl="$mirror/dists/$gsuite/main/installer-$domU/current/images/netboot"
gver=$gsuite
fi
--
2.1.4
^ permalink raw reply related [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host
2015-05-27 13:45 ` [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host Ian Campbell
@ 2015-07-06 14:48 ` Ian Jackson
2015-07-06 15:03 ` Ian Campbell
0 siblings, 1 reply; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 14:48 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
> v7: Quote $url and $path, switch to a heredoc to avoid resulting over
> long line
...
Last time I wrote:
Do we care that this will break badly if the url contains shell
metacharacters ? I think we may do.
but:
> + $useproxy wget --progress=dot:mega -O \"$path\" \"$url\"
Did you try this with $path or $url containing $ or or " or \ or ` or
starting with - ?
There are a fair few places in osstest where we're quite lax with this
kind of thing, but (hopefully) only where the information definitely
comes from the configuration (or some other trusted source). A
general helper like this ought to be robust against that kind of input
(which may well mean failing, but it should not include potentially
executing bits of the input or misinterpreting it as command line
options to wget.
\Q may be of some help.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 02/15] distros: add support for installing Debian PV guests via d-i, flight and jobs
2015-05-27 13:45 ` [PATCH OSSTEST v7 02/15] distros: add support for installing Debian PV guests via d-i, flight and jobs Ian Campbell
@ 2015-07-06 14:51 ` Ian Jackson
0 siblings, 0 replies; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 14:51 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 02/15] distros: add support for installing Debian PV guests via d-i, flight and jobs"):
> This patch introduces ts-debian-di-install which can install Debian
> from a netboot (PXE) debian installer image. By default it installs
> from the d-i image used by osstest (using the special Xen PV guest
> enabled flavour where necessary) but it can also fetch the kernel and
> ramdisk from URLs specified in runvars. The resulting guests boot the
> distro kernel using pygrub (pvgrub will follow).
>
> The distros flights differ substantially from the existing flights.
> Introduce make-distros-flight using the functionality previously
> refactored into mfi-common. The new flight tests all versions of
> Debian from Squeeze onward as an amd64, i386 and armhf guests (armhf
> from Jessie onwards only) using the usual smoke tests.
>
> Test names are suffixed -pygrub pending the addition of pvgrub
> variants in a future commit.
>
> Add the new cases to sg-run-job
>
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too.
2015-05-27 13:45 ` [PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too Ian Campbell
@ 2015-07-06 14:55 ` Ian Jackson
2015-07-06 15:06 ` Ian Campbell
0 siblings, 1 reply; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 14:55 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too."):
> This requires us to install pv-grub-menu from backports, which we do
> using a late_command.
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
However:
> + preseed_hook_command($ho, 'late_command', $sfx, <<END);
> +#!/bin/sh
> +set -ex
> +
> +(
> + echo
> + echo \\\# $suite backports
> + echo deb http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
> +) >> /target/etc/apt/sources.list
> +in-target apt-get update
> +in-target apt-get install -y -t wheezy-backports pv-grub-menu
> +END
This way of saying this requires quite a lot of quoting! Have you
considered using a nested here document ? In shell you can say
<<'END2' or something and the content will not be subject to
$-expansion.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 05/15] distros: support PV guest install from Debian netinst media.
2015-05-27 13:45 ` [PATCH OSSTEST v7 05/15] distros: support PV guest install from Debian netinst media Ian Campbell
@ 2015-07-06 15:01 ` Ian Jackson
2015-07-07 9:29 ` Ian Campbell
0 siblings, 1 reply; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:01 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 05/15] distros: support PV guest install from Debian netinst media."):
> The netinst media are iso images containing a base Debian install and
> some (image size dependent) additional tasks.
...
> This is configured from make*flight using runvars:
>
> - <gident>_netinst_baseurl:
> Base URL of directory containing the netinst jigdo images.
> - <gident>_netinst_filere:
> Regular expression to match actual file within baseurl to use
> (without .jigdo suffix)
> <gident>_netinst_kernel:
> Path to kernel within the netinst image
> <gident>_netinst_ramdisk
> Path to ramdisk within the netinst image
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
This documentation could profitably be placed in the script itself.
I've acked this anyway because existing practice is very sloppy.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host
2015-07-06 14:48 ` Ian Jackson
@ 2015-07-06 15:03 ` Ian Campbell
2015-07-06 15:59 ` Ian Jackson
0 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-07-06 15:03 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Mon, 2015-07-06 at 15:48 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host"):
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> > ---
> > v7: Quote $url and $path, switch to a heredoc to avoid resulting over
> > long line
> ...
>
> Last time I wrote:
>
> Do we care that this will break badly if the url contains shell
> metacharacters ? I think we may do.
>
> but:
>
> > + $useproxy wget --progress=dot:mega -O \"$path\" \"$url\"
>
> Did you try this with $path or $url containing $ or or " or \ or ` or
> starting with - ?
No, I think I just hadn't thought very clearly enough about what you
meant.
> There are a fair few places in osstest where we're quite lax with this
> kind of thing, but (hopefully) only where the information definitely
> comes from the configuration (or some other trusted source). A
> general helper like this ought to be robust against that kind of input
> (which may well mean failing, but it should not include potentially
> executing bits of the input or misinterpreting it as command line
> options to wget.
>
> \Q may be of some help.
Really? I thought that escaped things at the Perl level, is that
sufficient for the shell stuff within the Perl too?
I suspect I just don't understand clearly enough what \Q actually does.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 07/15] distros: add branch infrastructure
2015-05-27 13:45 ` [PATCH OSSTEST v7 07/15] distros: add branch infrastructure Ian Campbell
@ 2015-07-06 15:04 ` Ian Jackson
0 siblings, 0 replies; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:04 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 07/15] distros: add branch infrastructure"):
> Since the distro nightlies are not version controlled we cannot use
> the usual mechanisms for detecting regressions. Special case things
> appropriately. We use an OLD_REVISION of "flight-NNN" to signify that
> the old revision is another flight and not a tree revision.
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 08/15] distros: Run one suite per day on a weekly basis
2015-05-27 13:45 ` [PATCH OSSTEST v7 08/15] distros: Run one suite per day on a weekly basis Ian Campbell
@ 2015-07-06 15:05 ` Ian Jackson
0 siblings, 0 replies; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:05 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 08/15] distros: Run one suite per day on a weekly basis"):
> Once a week should be sufficient for these tests. Perhaps in the
> future we will want to increase the frequency for the suites under
> active development (testing, unstable)
>
> For now run this on the Citrix Cambridge instance until the XenProject
> instance has sufficient capacity.
>
> Switch to tabs for alignment.
Please can you split this whitespace change out ? It makes the actual
diff hard to spot.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too.
2015-07-06 14:55 ` Ian Jackson
@ 2015-07-06 15:06 ` Ian Campbell
2015-07-06 15:54 ` Ian Jackson
0 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-07-06 15:06 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Mon, 2015-07-06 at 15:55 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too."):
> > This requires us to install pv-grub-menu from backports, which we do
> > using a late_command.
>
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
>
> However:
>
> > + preseed_hook_command($ho, 'late_command', $sfx, <<END);
> > +#!/bin/sh
> > +set -ex
> > +
> > +(
> > + echo
> > + echo \\\# $suite backports
> > + echo deb http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
> > +) >> /target/etc/apt/sources.list
> > +in-target apt-get update
> > +in-target apt-get install -y -t wheezy-backports pv-grub-menu
> > +END
>
> This way of saying this requires quite a lot of quoting!
I don't follow, the only thing above which I think you might be
referring to is the "\\\#". There isn't any other quoting.
Or do you mean that this needs quoting and is lacking it and that adding
it would make the here document useful? (I don't think you did)
> Have you
> considered using a nested here document ? In shell you can say
> <<'END2' or something and the content will not be subject to
> $-expansion.
All of the $ here are intended to be expanded at the Perl level.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 09/15] Debian: Handle lack of bootloader support in d-i on ARM.
2015-05-27 13:45 ` [PATCH OSSTEST v7 09/15] Debian: Handle lack of bootloader support in d-i on ARM Ian Campbell
@ 2015-07-06 15:07 ` Ian Jackson
0 siblings, 0 replies; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:07 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 09/15] Debian: Handle lack of bootloader support in d-i on ARM."):
> Debian doesn't currently know what bootloader to install in a Xen
> guest on ARM. We install pv-grub-menu above which actually does what
> we need, but the installer doesn't treat that as a "bootloader".
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 10/15] ts-debian-di-install: Refactor root_disk specification
2015-05-27 13:45 ` [PATCH OSSTEST v7 10/15] ts-debian-di-install: Refactor root_disk specification Ian Campbell
@ 2015-07-06 15:07 ` Ian Jackson
0 siblings, 0 replies; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:07 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 10/15] ts-debian-di-install: Refactor root_disk specification"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 11/15] make-flight: refactor PV debian tests
2015-05-27 13:45 ` [PATCH OSSTEST v7 11/15] make-flight: refactor PV debian tests Ian Campbell
@ 2015-07-06 15:08 ` Ian Jackson
0 siblings, 0 replies; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:08 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 11/15] make-flight: refactor PV debian tests"):
> No functional change, standalone-generate-dump-flight-runvars confirms
> no change to the runvars.
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends.
2015-05-27 13:45 ` [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends Ian Campbell
@ 2015-07-06 15:21 ` Ian Jackson
2015-07-06 15:53 ` Ian Campbell
0 siblings, 1 reply; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:21 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends."):
> xen-create-image makes this tricky to do since it is rather LVM
> centric. Now that we have the ability to install from d-i it's
> possible to arrange fairly easily that they use non-LVM disk backend
> arrangements.
>
> Here we add support to the test script and infra and create a bunch of
> new jobs testing the cross product of {xl,libvirt} x {raw,qcow2,vhd}.
This looks pretty good.
I have just two minor questions:
> +sub prepareguest_part_diskimg ($$$) {
> + my ($ho, $gho, $disk_mb) = @_;
> +
> + my $diskfmt = $gho->{Diskfmt};
> + # Allow an extra 10 megabytes for image format headers
> + my $disk_overhead = $diskfmt eq "lvm" ? 0 : 10;
> +
> + logm("preparing guest disks in $diskfmt format");
> +
> + target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
Does this not sometimes produce an undefined variable warning ?
> diff --git a/make-flight b/make-flight
> index 8206508..0313696 100755
> --- a/make-flight
> +++ b/make-flight
> @@ -383,6 +383,22 @@ do_pv_debian_tests () {
> for xsm in $xsms ; do
> do_pv_debian_test_one libvirt '' libvirt enable_xsm=$xsm
> done
> +
> + for ts in xl libvirt ; do
> +
> + for fmt in raw vhd qcow2 ; do
> +
> + fmt_runvar="debian_diskfmt=$fmt"
> +
> + do_pv_debian_test_one $ts-$fmt '-di' $ts \
> + debian_arch=$dom0arch \
> + debian_suite=$guestsuite \
> + debian_method=netboot \
> + debian_bootloader=pygrub \
> + $fmt_runvar
Are some of these jobs not very similar to existing ones ?
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include
2015-05-27 13:45 ` [PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include Ian Campbell
@ 2015-07-06 15:22 ` Ian Jackson
2015-07-07 9:40 ` Ian Campbell
0 siblings, 1 reply; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:22 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include"):
> By using the same job_create_build(_filter_callback) scheme used for
> the test jobs.
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
I would like this for my xen-unstable-smoke series. Can you put it at
the front of your series ?
Thanks,
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 14/15] make-distros-flight: don't bother building for XSM or libvirt.
2015-05-27 13:45 ` [PATCH OSSTEST v7 14/15] make-distros-flight: don't bother building for XSM or libvirt Ian Campbell
@ 2015-07-06 15:22 ` Ian Jackson
0 siblings, 0 replies; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:22 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 14/15] make-distros-flight: don't bother building for XSM or libvirt."):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 15/15] make-distros-flight: Use ftp.debian.org directly
2015-05-27 13:45 ` [PATCH OSSTEST v7 15/15] make-distros-flight: Use ftp.debian.org directly Ian Campbell
@ 2015-07-06 15:23 ` Ian Jackson
2015-07-07 9:47 ` Ian Campbell
0 siblings, 1 reply; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:23 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("[PATCH OSSTEST v7 15/15] make-distros-flight: Use ftp.debian.org directly"):
> The local proxy seems to serve stale packages for Jessie etc, I blame
> the intercepting cache on the way out of our network, similar to
> b5f15136900d "mg-debian-installer-update: workaround caching proxies",
> except it is between the apt-cache and the world not the osstest vm
> and the world.
>
> Since the netboot kernel+initrd are reasonably small, these flights
> are infrequent and they are intended to test the current upstream
> version I think this is tollerable.
I bet we don't need this in the new colo.
Can we make this conditional on a config var set in
production-config-cambridge ?
I'm not sure what the precise scope ought to be.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends.
2015-07-06 15:21 ` Ian Jackson
@ 2015-07-06 15:53 ` Ian Campbell
2015-07-06 16:01 ` Ian Jackson
0 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-07-06 15:53 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Mon, 2015-07-06 at 16:21 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends."):
> > xen-create-image makes this tricky to do since it is rather LVM
> > centric. Now that we have the ability to install from d-i it's
> > possible to arrange fairly easily that they use non-LVM disk backend
> > arrangements.
> >
> > Here we add support to the test script and infra and create a bunch of
> > new jobs testing the cross product of {xl,libvirt} x {raw,qcow2,vhd}.
>
> This looks pretty good.
>
> I have just two minor questions:
>
> > +sub prepareguest_part_diskimg ($$$) {
> > + my ($ho, $gho, $disk_mb) = @_;
> > +
> > + my $diskfmt = $gho->{Diskfmt};
> > + # Allow an extra 10 megabytes for image format headers
> > + my $disk_overhead = $diskfmt eq "lvm" ? 0 : 10;
> > +
> > + logm("preparing guest disks in $diskfmt format");
> > +
> > + target_cmd_root($ho, "umount $gho->{Lvdev} ||:");
>
> Does this not sometimes produce an undefined variable warning ?
I don't think so, but I'll keep a look out next time I run a flight.
> > diff --git a/make-flight b/make-flight
> > index 8206508..0313696 100755
> > --- a/make-flight
> > +++ b/make-flight
> > @@ -383,6 +383,22 @@ do_pv_debian_tests () {
> > for xsm in $xsms ; do
> > do_pv_debian_test_one libvirt '' libvirt enable_xsm=$xsm
> > done
> > +
> > + for ts in xl libvirt ; do
> > +
> > + for fmt in raw vhd qcow2 ; do
> > +
> > + fmt_runvar="debian_diskfmt=$fmt"
> > +
> > + do_pv_debian_test_one $ts-$fmt '-di' $ts \
> > + debian_arch=$dom0arch \
> > + debian_suite=$guestsuite \
> > + debian_method=netboot \
> > + debian_bootloader=pygrub \
> > + $fmt_runvar
>
> Are some of these jobs not very similar to existing ones ?
They differ in the backing store used for the guest's disks, but
otherwise they are quite similar to the existing flight, except they are
installed with d-i not xen-create-image.
The use of d-i is simply because it was easier to reuse
ts-debian-di-install than to convince xen-create-image to using non-LVM,
rather than a particular desire to test d-i in this context.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too.
2015-07-06 15:06 ` Ian Campbell
@ 2015-07-06 15:54 ` Ian Jackson
0 siblings, 0 replies; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:54 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("Re: [PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too."):
> On Mon, 2015-07-06 at 15:55 +0100, Ian Jackson wrote:
> > Ian Campbell writes ("[PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too."):
...
> > > + echo \\\# $suite backports
> > > + echo deb http://$c{DebianMirrorHost}/$c{DebianMirrorSubpath} $suite-backports main
...
> > This way of saying this requires quite a lot of quoting!
>
> I don't follow, the only thing above which I think you might be
> referring to is the "\\\#". There isn't any other quoting.
>
> Or do you mean that this needs quoting and is lacking it and that adding
> it would make the here document useful? (I don't think you did)
I meant the \\\# (which seems like `a lot' of quoting all by itself),
but of course also the unnecessary ( ) and the multiple `echo' (which
in this context is a kind of quoting of new lines).
But if you prefer it this way, fine.
> > Have you
> > considered using a nested here document ? In shell you can say
> > <<'END2' or something and the content will not be subject to
> > $-expansion.
>
> All of the $ here are intended to be expanded at the Perl level.
Indeed. So <<'END2' is exactly what you want.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host
2015-07-06 15:03 ` Ian Campbell
@ 2015-07-06 15:59 ` Ian Jackson
2015-07-06 16:24 ` Ian Campbell
0 siblings, 1 reply; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 15:59 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("Re: [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host"):
> On Mon, 2015-07-06 at 15:48 +0100, Ian Jackson wrote:
> > Ian Campbell writes ("[PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host"):
> > > + $useproxy wget --progress=dot:mega -O \"$path\" \"$url\"
...
> > \Q may be of some help.
>
> Really? I thought that escaped things at the Perl level, is that
> sufficient for the shell stuff within the Perl too?
If you wrote
+ target_cmd_root($ho, <<END, $timeo);
+ $useproxy wget --progress=dot:mega -O \Q$path\E \Q$url\E
and $url contained
http://chars $"'\ considered harmful.iso
then target_cmd_root would get
http_proxy=something wget --progress=dot:mega -O \/some\/path \
http\:\/\/chars\ \$\"\'\\\ considered\ harmful\.iso
And:
mariner:~> echo http\:\/\/chars\ \$\"\'\\\ considered\ harmful\.iso
http://chars $"'\ considered harmful.iso
mariner:~>
But maybe this is too ugly, in which case you could use ''
and some regexp like s/['\\]/'\\$&'/g (not tested).
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends.
2015-07-06 15:53 ` Ian Campbell
@ 2015-07-06 16:01 ` Ian Jackson
2015-07-06 16:26 ` Ian Campbell
0 siblings, 1 reply; 42+ messages in thread
From: Ian Jackson @ 2015-07-06 16:01 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("Re: [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends."):
> On Mon, 2015-07-06 at 16:21 +0100, Ian Jackson wrote:
> > Are some of these jobs not very similar to existing ones ?
>
> They differ in the backing store used for the guest's disks, but
> otherwise they are quite similar to the existing flight, except they are
> installed with d-i not xen-create-image.
I meant the lvm ones.
> The use of d-i is simply because it was easier to reuse
> ts-debian-di-install than to convince xen-create-image to using non-LVM,
> rather than a particular desire to test d-i in this context.
So we are add d-i pv storage-on-lvm tests here, as well as
storage-not-on-lvm tests. That's good.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host
2015-07-06 15:59 ` Ian Jackson
@ 2015-07-06 16:24 ` Ian Campbell
0 siblings, 0 replies; 42+ messages in thread
From: Ian Campbell @ 2015-07-06 16:24 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Mon, 2015-07-06 at 16:59 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host"):
> > On Mon, 2015-07-06 at 15:48 +0100, Ian Jackson wrote:
> > > Ian Campbell writes ("[PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host"):
> > > > + $useproxy wget --progress=dot:mega -O \"$path\" \"$url\"
> ...
> > > \Q may be of some help.
> >
> > Really? I thought that escaped things at the Perl level, is that
> > sufficient for the shell stuff within the Perl too?
>
> If you wrote
>
> + target_cmd_root($ho, <<END, $timeo);
> + $useproxy wget --progress=dot:mega -O \Q$path\E \Q$url\E
>
> and $url contained
>
> http://chars $"'\ considered harmful.iso
>
> then target_cmd_root would get
>
> http_proxy=something wget --progress=dot:mega -O \/some\/path \
> http\:\/\/chars\ \$\"\'\\\ considered\ harmful\.iso
Interesting, I had assumed \Q...\E was some sort of lexer/parser level
thing, when actually it "just" rewrites what is between it.
And as it happens Perl and shell need a very similar set of things
quoting.
>
> And:
>
> mariner:~> echo http\:\/\/chars\ \$\"\'\\\ considered\ harmful\.iso
> http://chars $"'\ considered harmful.iso
> mariner:~>
>
> But maybe this is too ugly,
I think it is ok actually.
> in which case you could use ''
> and some regexp like s/['\\]/'\\$&'/g (not tested).
>
> Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends.
2015-07-06 16:01 ` Ian Jackson
@ 2015-07-06 16:26 ` Ian Campbell
2015-07-06 16:40 ` Ian Campbell
0 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-07-06 16:26 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Mon, 2015-07-06 at 17:01 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends."):
> > On Mon, 2015-07-06 at 16:21 +0100, Ian Jackson wrote:
> > > Are some of these jobs not very similar to existing ones ?
> >
> > They differ in the backing store used for the guest's disks, but
> > otherwise they are quite similar to the existing flight, except they are
> > installed with d-i not xen-create-image.
>
> I meant the lvm ones.
The list here was:
for fmt in raw vhd qcow2 ; do
i.e. no LVM.
(as an implementation detail each of the above will in its own LVM LV,
but that's not what I think you were getting at).
>
> > The use of d-i is simply because it was easier to reuse
> > ts-debian-di-install than to convince xen-create-image to using non-LVM,
> > rather than a particular desire to test d-i in this context.
>
> So we are add d-i pv storage-on-lvm tests here, as well as
> storage-not-on-lvm tests. That's good.
We aren't adding that, but we could if you wanted.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends.
2015-07-06 16:26 ` Ian Campbell
@ 2015-07-06 16:40 ` Ian Campbell
0 siblings, 0 replies; 42+ messages in thread
From: Ian Campbell @ 2015-07-06 16:40 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Mon, 2015-07-06 at 17:26 +0100, Ian Campbell wrote:
> On Mon, 2015-07-06 at 17:01 +0100, Ian Jackson wrote:
> > Ian Campbell writes ("Re: [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends."):
> > > On Mon, 2015-07-06 at 16:21 +0100, Ian Jackson wrote:
> > > > Are some of these jobs not very similar to existing ones ?
> > >
> > > They differ in the backing store used for the guest's disks, but
> > > otherwise they are quite similar to the existing flight, except they are
> > > installed with d-i not xen-create-image.
> >
> > I meant the lvm ones.
>
> The list here was:
> for fmt in raw vhd qcow2 ; do
> i.e. no LVM.
After f2f discussion the upshot is I should change the default diskfmt
to be "none" rather than "lvm", i.e. in
+ return if $gho->{Diskfmt} eq "lvm";
so that there is less confusion between "raw" and "lvm".
I should also update the commit message to talk less about "non-LVM
backends" and more about "file formats" or some such.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 05/15] distros: support PV guest install from Debian netinst media.
2015-07-06 15:01 ` Ian Jackson
@ 2015-07-07 9:29 ` Ian Campbell
0 siblings, 0 replies; 42+ messages in thread
From: Ian Campbell @ 2015-07-07 9:29 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Mon, 2015-07-06 at 16:01 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v7 05/15] distros: support PV guest install from Debian netinst media."):
> > The netinst media are iso images containing a base Debian install and
> > some (image size dependent) additional tasks.
> ...
> > This is configured from make*flight using runvars:
> >
> > - <gident>_netinst_baseurl:
> > Base URL of directory containing the netinst jigdo images.
> > - <gident>_netinst_filere:
> > Regular expression to match actual file within baseurl to use
> > (without .jigdo suffix)
> > <gident>_netinst_kernel:
> > Path to kernel within the netinst image
> > <gident>_netinst_ramdisk
> > Path to ramdisk within the netinst image
>
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
>
> This documentation could profitably be placed in the script itself.
> I've acked this anyway because existing practice is very sloppy.
I did this, which implied changes to previous patches which added the
script and other runvars, I have retained your ack where those were the
only changes.
For this specific patch I noticed an unused <gident>_cd thing while
documenting, which I have removed, hence I didn't take the ack above.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include
2015-07-06 15:22 ` Ian Jackson
@ 2015-07-07 9:40 ` Ian Campbell
2015-07-07 10:38 ` Ian Jackson
0 siblings, 1 reply; 42+ messages in thread
From: Ian Campbell @ 2015-07-07 9:40 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Mon, 2015-07-06 at 16:22 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include"):
> > By using the same job_create_build(_filter_callback) scheme used for
> > the test jobs.
>
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
>
> I would like this for my xen-unstable-smoke series. Can you put it at
> the front of your series ?
I've done this. Would you like me to resend sooner rather than later
(i.e. without having run a flight after the mods associated with all the
feedback you gave on this posting, not just the one change suggested
here).
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 15/15] make-distros-flight: Use ftp.debian.org directly
2015-07-06 15:23 ` Ian Jackson
@ 2015-07-07 9:47 ` Ian Campbell
0 siblings, 0 replies; 42+ messages in thread
From: Ian Campbell @ 2015-07-07 9:47 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On Mon, 2015-07-06 at 16:23 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v7 15/15] make-distros-flight: Use ftp.debian.org directly"):
> > The local proxy seems to serve stale packages for Jessie etc, I blame
> > the intercepting cache on the way out of our network, similar to
> > b5f15136900d "mg-debian-installer-update: workaround caching proxies",
> > except it is between the apt-cache and the world not the osstest vm
> > and the world.
> >
> > Since the netboot kernel+initrd are reasonably small, these flights
> > are infrequent and they are intended to test the current upstream
> > version I think this is tollerable.
>
> I bet we don't need this in the new colo.
>
> Can we make this conditional on a config var set in
> production-config-cambridge ?
>
> I'm not sure what the precise scope ought to be.
It's hard to say, since we don't want to do this for all uses of
DebianMirrorHost (i.e. host install, regular use of stable).
So the wording of the name probably wants to somehow imply that it wants
to avoid the caching mirror only for things which are more "volatile"
than Debian stable or something?
Alternatively, perhaps target_fetchurl should gain an option to bypas
the proxy (as mg-debian-installer-update does), the argument there might
be that we definitely want the very latest thing from upstream for these
cases, and even a well managed cache timeout from the new colo's http
proxy should be avoided.
I'm not sure. I'll not let this hold up a potential v8 while we think
about the best approach.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: [PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include
2015-07-07 9:40 ` Ian Campbell
@ 2015-07-07 10:38 ` Ian Jackson
0 siblings, 0 replies; 42+ messages in thread
From: Ian Jackson @ 2015-07-07 10:38 UTC (permalink / raw)
To: Ian Campbell; +Cc: xen-devel
Ian Campbell writes ("Re: [PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include"):
> On Mon, 2015-07-06 at 16:22 +0100, Ian Jackson wrote:
> > I would like this for my xen-unstable-smoke series. Can you put it at
> > the front of your series ?
>
> I've done this. Would you like me to resend sooner rather than later
> (i.e. without having run a flight after the mods associated with all the
> feedback you gave on this posting, not just the one change suggested
> here).
Or, just a repost of that one patch. Whatever is easiest.
Ian.
^ permalink raw reply [flat|nested] 42+ messages in thread
end of thread, other threads:[~2015-07-07 10:38 UTC | newest]
Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-05-27 13:44 [PATCH OSSTEST v7 00/16] add distro domU testing flight Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 01/15] TestSupport: Add helper to fetch a URL on a host Ian Campbell
2015-07-06 14:48 ` Ian Jackson
2015-07-06 15:03 ` Ian Campbell
2015-07-06 15:59 ` Ian Jackson
2015-07-06 16:24 ` Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 02/15] distros: add support for installing Debian PV guests via d-i, flight and jobs Ian Campbell
2015-07-06 14:51 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 03/15] distros: support booting Debian PV (d-i installed) guests with pvgrub Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 04/15] distros: Support pvgrub for Wheezy too Ian Campbell
2015-07-06 14:55 ` Ian Jackson
2015-07-06 15:06 ` Ian Campbell
2015-07-06 15:54 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 05/15] distros: support PV guest install from Debian netinst media Ian Campbell
2015-07-06 15:01 ` Ian Jackson
2015-07-07 9:29 ` Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 06/15] Test pygrub and pvgrub on the regular flights Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 07/15] distros: add branch infrastructure Ian Campbell
2015-07-06 15:04 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 08/15] distros: Run one suite per day on a weekly basis Ian Campbell
2015-07-06 15:05 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 09/15] Debian: Handle lack of bootloader support in d-i on ARM Ian Campbell
2015-07-06 15:07 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 10/15] ts-debian-di-install: Refactor root_disk specification Ian Campbell
2015-07-06 15:07 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 11/15] make-flight: refactor PV debian tests Ian Campbell
2015-07-06 15:08 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 12/15] Add testing of non-LVM/phy disk backends Ian Campbell
2015-07-06 15:21 ` Ian Jackson
2015-07-06 15:53 ` Ian Campbell
2015-07-06 16:01 ` Ian Jackson
2015-07-06 16:26 ` Ian Campbell
2015-07-06 16:40 ` Ian Campbell
2015-05-27 13:45 ` [PATCH OSSTEST v7 13/15] mfi-common: Allow make-*flight to filter the set of build jobs to include Ian Campbell
2015-07-06 15:22 ` Ian Jackson
2015-07-07 9:40 ` Ian Campbell
2015-07-07 10:38 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 14/15] make-distros-flight: don't bother building for XSM or libvirt Ian Campbell
2015-07-06 15:22 ` Ian Jackson
2015-05-27 13:45 ` [PATCH OSSTEST v7 15/15] make-distros-flight: Use ftp.debian.org directly Ian Campbell
2015-07-06 15:23 ` Ian Jackson
2015-07-07 9:47 ` Ian Campbell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.