[PATCH OSSTEST RFC 00/19] support for ARM32 arndale and cubietruck platforms

[PATCH OSSTEST RFC 00/19] support for ARM32 arndale and cubietruck platforms

[Ian Campbell]

I'm preparing a bunch of each of these boards to be racked for inclusion
in osstest. This series adds the necessary osstest support:

      * Support for platforms which require us to supply a Device Tree
      * Removing various hardcoded midway assumptions
      * Some new features needed to workaround h/w quirks (of arndale in
        particular)
      * Support for the power relay switch board which these are
        attached to
      * Some new convenience features

I have run a full (including host reinstall) build-* job on each
platform.

I have successfully run test-armhf-armhf-xl on cubietruck, but not yet
on arndale (struggling to find a pvops config which boots, even
natively!). I thought I'd send the bulk out to get started with.

For reference the relevant bits of my ~/.xen-osstest/config are below.

Ian.

#HostProp_metrocentre_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 1
HostProp_westfield_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 2
HostProp_lakeside_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 3
HostProp_bluewater_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 4

HostProp_braque_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 5
HostProp_picaso_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 6
HostProp_metzinger_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 7
HostProp_gleizes_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 8

HostGroupProp_arndale_LinuxSerialConsole ttySAC2
HostGroupProp_arndale_Build_Make_Flags -j4
HostGroupProp_arndale_XenSerialConsole dtuart
HostGroupProp_arndale_XenDTUARTPath /serial@12C20000
HostGroupProp_arndale_Interface_Force eth0
HostGroupProp_arndale_ExtraInitramfsModules clk-s2mps11 s5m8767 i2c-s3c2410 phy-exynos5250-sata
HostGroupProp_arndale_Rootdelay 3
HostGroupProp_arndale_UBootScriptEarlyCommands setenv xen_addr_r 0x41000000

HostGroupFlags_arndale suite-wheezy,equiv-arndale,need-kernel-deb-armmp,no-di-kernel,force-mac-address,need-uboot-bootscr

HostGroup_metrocentre arndale
HostProp_metrocentre_Fqdn metrocentre.uk.xensource.com
HostProp_metrocentre_Ether 9e:04:00:59:64:5a

HostGroup_westfield arndale
HostProp_westfield_Fqdn westfield.uk.xensource.com
HostProp_westfield_Ether a6:46:13:77:e8:2f

HostGroup_lakeside arndale
HostProp_lakeside_Fqdn lakeside.uk.xensource.com
HostProp_lakeside_Ether e2:e7:75:3f:df:4c

HostGroup_bluewater arndale
HostProp_bluewater_Fqdn bluewater.uk.xensource.com
HostProp_bluewater_Ether f2:dc:22:d7:e9:e9

HostGroupProp_cubietruck_LinuxSerialConsole ttyS0
HostGroupProp_cubietruck_Build_Make_Flags -j4
HostGroupProp_cubietruck_XenSerialConsole dtuart
HostGroupProp_cubietruck_XenDTUARTPath /soc@01c00000/serial@01c28000 
HostGroupProp_cubietruck_UBootScriptEarlyCommands setenv xen_addr_r 0x41000000
HostGroupFlags_cubietruck suite-wheezy,equiv-cubietruck,need-kernel-deb-armmp,no-di-kernel,need-uboot-bootscr

HostGroup_braque cubietruck
HostProp_braque_Fqdn braque.uk.xensource.com

HostGroup_picaso cubietruck
HostProp_picaso_Fqdn picaso.uk.xensource.com

HostGroup_metzinger cubietruck
HostProp_metzinger metzinger.uk.xensource.com

HostGroup_gleizes cubietruck
HostProp_gleizes_Fqdn gleizes.uk.xensource.com

[PATCH RFC OSSTEST 01/19] standalone: Provide a helper to clear the host runvar

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 standalone | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/standalone b/standalone
index daab7ae..c1139f8 100755
--- a/standalone
+++ b/standalone
@@ -29,6 +29,12 @@ Operations:
 
   Required options: -h HOST
 
+* reset-host [cf] [JOB]
+
+  Allow a job which has been run before to run on a different host
+  next time. Otherwise osstest will complain if you change the host
+  which a job is running on on successive runs.
+
 Options:
 
 -c FILE, --config=FILE        Use FILE as configuration file
@@ -200,6 +206,18 @@ case $op in
 	fi
 	;;
 
+    reset-host)
+	need_flight;
+
+	if [ $# -lt 1 ] ; then
+	    echo "run-job: Need job" >&2
+	    exit 1
+	fi
+
+	job=$1; shift
+	./cs-adjust-flight -v $flight runvar-del $job host
+	;;
+
     run-job)
 	need_flight; need_host
 
-- 
2.1.0

[PATCH RFC OSSTEST 02/19] Osstest/PDU: Add eth008.pm method to control the ARM rack PDU

[Ian Campbell]

This controls the eth008 relay board: http://www.robot-electronics.co.uk/htm/eth008tech.htm

Due to the use of the CGI interface this requires firmware version 4+.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/PDU/eth008.pm | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
 create mode 100644 Osstest/PDU/eth008.pm

diff --git a/Osstest/PDU/eth008.pm b/Osstest/PDU/eth008.pm
new file mode 100644
index 0000000..da6af78
--- /dev/null
+++ b/Osstest/PDU/eth008.pm
@@ -0,0 +1,68 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package Osstest::PDU::eth008;
+
+use strict;
+use warnings;
+
+use Osstest;
+use Osstest::TestSupport;
+use IO::File;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho, $methname, $pdu, $user, $pass, $port, @opts) = @_;
+    return bless { Host => $ho,
+		   PDU => $pdu,
+		   User => $user,
+		   Pass => $pass,
+		   Port => $port,
+		   Opts => \@opts }, $class;
+}
+
+sub pdu_power_state {
+    my ($mo, $on) = @_;
+    my $op= $on ? "DOA" : "DOI"; # Digital Output (In)Active
+
+    # Use the CGI interface since it is less prone to being firewalled
+    # off, unlike the standard interface on port 17494. This is only
+    # available from firmware v4 onwards.
+
+    my $url = "http://$mo->{PDU}/io.cgi?$op$mo->{Port}=0";
+    my $cmd = "curl --silent --user $mo->{User}:$mo->{Pass} $url";
+
+    logm("$cmd");
+    open CURL, "$cmd |" or die "fork curl: $!";
+    
+    my $result = <CURL>;
+    close CURL or die "result curl: $!";
+
+    logm("$result");
+    die "curl failed" unless $result =~ /^Success! \d+/;
+}
+
+1;
-- 
2.1.0

[PATCH RFC OSSTEST 03/19] mg-debian-installer-update: Include any device tree blobs in the tftp dir

[Ian Campbell]

We will need to supply these for some arm boards.

We provide both an unpacked version, which can be referenced via u-boot's
support for the "fdtdir" property in pxelinux.cfg and a tarball which is more
convenient for installing into the target filesystem on host install

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 mg-debian-installer-update | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index b964f95..4e1da8a 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -130,6 +130,10 @@ if [ $arch = armhf ]; then
           |pax -x sv4cpio -s '%lib%/lib%' -d -w >../cpio; cd ..
     gzip -9f cpio
     mv cpio.gz armmp.cpio.gz
+    rm -rf dtbs/
+    mkdir dtbs/
+    mv x/usr/lib/linux-image-*-armmp/*.dtb dtbs/
+    tar -caf dtbs.tar.gz dtbs
     rm -rf x
 fi
 
-- 
2.1.0

[PATCH RFC OSSTEST 04/19] mg-debian-installer-update: Add more modules when creating initrd overlay

[Ian Campbell]

The arndale platform has a usb network device. In addition both the SATA and
NIC/USB hardware on this platform requires additional regulator (power), clk
and phy modules, which in turn require i2c support.

Add everything which is needed on this platform.

Sort the list of drivers while here.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 mg-debian-installer-update | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index 4e1da8a..4af4a42 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -123,10 +123,18 @@ if [ $arch = armhf ]; then
                          -path \*/kernel/fs/mbcache.ko -o \
                          -path \*/kernel/fs/ext\* -o \
                          -path \*/kernel/fs/jbd\* -o \
-                         -path \*/kernel/drivers/net/\* -o \
+                         -path \*/kernel/drivers/clk/\* -o \
+                         -path \*/kernel/drivers/mmc/\* -o \
+                         -path \*/kernel/drivers/phy/\* -o \
+                         -path \*/kernel/drivers/usb/misc/\* -o \
                          -path \*/kernel/drivers/ata/\* -o \
+                         -path \*/kernel/drivers/i2c/busses/\* -o \
+                         -path \*/kernel/drivers/md/\* -o \
+                         -path \*/kernel/drivers/net/\* -o \
+                         -path \*/kernel/drivers/regulator/\* -o \
                          -path \*/kernel/drivers/scsi/\* -o \
-                         -path \*/kernel/drivers/md/\* \) \
+                         -path \*/kernel/drivers/usb/dwc3/\* -o \
+                         -path \*/kernel/drivers/usb/host/\* \) \
           |pax -x sv4cpio -s '%lib%/lib%' -d -w >../cpio; cd ..
     gzip -9f cpio
     mv cpio.gz armmp.cpio.gz
-- 
2.1.0

[PATCH RFC OSSTEST 05/19] ts-host-install: Add fdtdir to PXE stanza if dtbs directory exists

[Ian Campbell]

u-boot's pxe client uses this together with $fdtfile in its local environment
to automatically load the fdt when pxe booting.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-host-install | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ts-host-install b/ts-host-install
index 95ce845..ca26f95 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -188,6 +188,9 @@ SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="$ho->{Ether}", A
 END
     }
 
+    my $dtbs = "fdtdir /$d_i/dtbs" if
+	-e "$ho->{Tftp}{Path}/$d_i/dtbs";
+
     file_simple_write_contents("$initrd_overlay.cpio", sub {
         contents_make_cpio($_[0], 'newc', "$initrd_overlay.d");
     });
@@ -222,6 +225,7 @@ label overwrite
 	kernel $kernel
 	append $installcmdline
 	ipappend $ipappend
+	$dtbs
 default overwrite
 END
 }
-- 
2.1.0

[PATCH RFC OSSTEST 06/19] ts-host-install: Refactor code to construct a Linux console= for a host

[Ian Campbell]

And reuse it in when writing the u-boot script for initial boot. This was
previously hardcoded to ttyAMA which matches the production host DB for the
current set of hosts which use this script.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm      |  4 +++-
 Osstest/TestSupport.pm | 10 ++++++++++
 ts-host-install        |  6 +++---
 3 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index ab09abb..29dec3b 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -660,6 +660,8 @@ END
 
     if ( $ho->{Flags}{'need-uboot-bootscr'} ) {
 	my $root=target_guest_lv_name($ho,"root");
+	my $console = get_host_native_linux_console($ho);
+	my $consolecmd = "console=$console" unless $console eq "NONE";
 
 	preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
@@ -671,7 +673,7 @@ kernel=`readlink \$r/vmlinuz | sed -e 's|boot/||'`
 initrd=`readlink \$r/initrd.img | sed -e 's|boot/||'`
 
 cat >\$r/boot/boot <<EOF
-setenv bootargs console=ttyAMA0 root=$root
+setenv bootargs $consolecmd root=$root
 mw.l 800000 0 10000
 scsi scan
 ext2load scsi 0 \\\${kernel_addr_r} \$kernel
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 1d77933..d66708e 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -65,6 +65,7 @@ BEGIN {
                       contents_make_cpio file_simple_write_contents
 
                       selecthost get_hostflags get_host_property
+                      get_host_native_linux_console
                       power_state power_cycle power_cycle_time
                       serial_fetch_logs
                       propname_massage
@@ -863,6 +864,15 @@ sub get_host_property ($$;$) {
     return defined($val) ? $val : $defval;
 }
 
+sub get_host_native_linux_console ($) {
+    my ($ho) = @_;
+
+    my $console = get_host_property($ho, "LinuxSerialConsole", "ttyS0");
+    return $console if $console eq 'NONE';
+
+    return "$console,$c{Baud}n8";
+}
+
 sub get_host_method_object ($$$) {
     my ($ho, $kind, $meth) = @_;
     my (@meth) = split /\s+/, $meth;
diff --git a/ts-host-install b/ts-host-install
index ca26f95..cf9506d 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -210,9 +210,9 @@ END
 
     push @installcmdline, qw(--);
 
-    my $console = get_host_property($ho, "LinuxSerialConsole", "ttyS0");
-    push @installcmdline, "console=$console,$c{Baud}n8"
-        unless $console eq "NONE";
+    my $console = get_host_native_linux_console($ho);
+
+    push @installcmdline, "console=$console" unless $console eq "NONE";
 
     my $installcmdline= join ' ', @installcmdline;
 
-- 
2.1.0

[PATCH RFC OSSTEST 07/19] ts-host-install: include console before *and* after the -- marker.

[Ian Campbell]

As of v3.15 Linux now stop parsing its command line at the -- mark and passes
the remainder to init. This has broken Debian Installer's feature where
anything after the -- is propagated to the installed system.

See also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007

To work around this we need to pass console= twice, once before the -- for the
current kernel's use and then again after the -- which is propagated to the
installed system.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-host-install | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/ts-host-install b/ts-host-install
index cf9506d..cdbd774 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -208,12 +208,16 @@ END
     push @installcmdline,
         get_host_property($ho, "install-append $ho->{Suite}", '');
 
-    push @installcmdline, qw(--);
-
     my $console = get_host_native_linux_console($ho);
 
     push @installcmdline, "console=$console" unless $console eq "NONE";
 
+    push @installcmdline, qw(--);
+
+    # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007 for
+    # why this is repeated.
+    push @installcmdline, "console=$console" unless $console eq "NONE";
+
     my $installcmdline= join ' ', @installcmdline;
 
     setup_pxeboot($ho, <<END);
-- 
2.1.0

[PATCH RFC OSSTEST 08/19] ts-host-install: Add option to boot debian-installer in rescue mode

[Ian Campbell]

This is useful when trying to bootstrap onto a new platform, since you can log
in to an otherwise unbootable rootfs to figure out what is going wrong.

The procedure is the same as installing except "rescue/enabled=true" is added
to the command line, hence part of ts-host-install rather than separate.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm | 1 +
 ts-host-install   | 9 +++++++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 29dec3b..8f80eb4 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -422,6 +422,7 @@ sub di_installcmdline_core ($$;@) {
     my $debconf_priority= $xopts{DebconfPriority};
     push @cl, "debconf/priority=$debconf_priority"
         if defined $debconf_priority;
+    push @cl, "rescue/enable=true" if $xopts{RescueMode};
 
     return @cl;
 }
diff --git a/ts-host-install b/ts-host-install
index cdbd774..7c8d693 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -30,9 +30,14 @@ my $debconf_priority;
 
 our %xopts;
 
-if (@ARGV && $ARGV[0] =~ m/^--priority(?:=(.*))?$/) {
+while (@ARGV and $ARGV[0] =~ m/^-/) {
+    $_= shift @ARGV;
+    last if m/^--$/;
+    if (m/^--priority(?:=(.*))?$/) {
     $xopts{DebconfPriority}= defined($1) ? $1 : 'low';
-    shift @ARGV;
+    } elsif  (m/^--rescue$/) {
+	$xopts{RescueMode}= 1;
+    }
 }
 
 our ($whhost) = @ARGV;
-- 
2.1.0

[PATCH RFC OSSTEST 09/19] ts-host-install: Add force-mac-address host flag

[Ian Campbell]

This causes us to emit udev runes which force the device named in the host's
Interface_Force property to have the MAC address given in the host's Ether
property. This is the reverse of the behaviour of the Interface_Force property
without this flag, which is to rename the device with MAC address Ether to have
the requested name.

This is necessary because the arndale's USB Ethernet device has no burned in
MAC address and there is no mechanism to configure this e.g. via the bootloader
in a way which Linux will then notice.

Without this change Linux will pick a random mac address when loading the
driver, which breaks osstest's attempts to observe when the test host has
picked up its preseed file by watching the webserver logs, since DHCP will not
map a random mac address to the IP we are expecting.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-host-install | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/ts-host-install b/ts-host-install
index 7c8d693..e874b98 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -181,6 +181,7 @@ sub setup_pxeboot_firstboot($) {
 
     my $ipappend = 2;
     my $wantphysif= get_host_property($ho,'interface force','auto');
+    logm("Forcing interface $wantphysif");
     if ($wantphysif ne 'auto') {
 	$ipappend = 0;
 	die "need Ether for $ho->{Name} ($wantphysif)"
@@ -188,7 +189,9 @@ sub setup_pxeboot_firstboot($) {
         system_checked(qw(mkdir -p --), "$initrd_overlay.d/etc/udev/rules.d");
         file_simple_write_contents
             ("$initrd_overlay.d/etc/udev/rules.d/70-persistent-net.rules",
-             <<END);
+             $ho->{Flags}{'force-mac-address'} ? <<END : <<END);
+SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNEL=="$wantphysif", RUN += "/bin/ip link set $wantphysif address $ho->{Ether}"
+END
 SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="$ho->{Ether}", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="$wantphysif"
 END
     }
-- 
2.1.0

[PATCH RFC OSSTEST 10/19] make-flight: Run a basic test on each arm platform

[Ian Campbell]

Unlike x86 there is enough variation in the ARM platforms that it is worth
having a basic test on each as part of a standard run. This relies on each host
having an appropriate equiv-$platform host flag.

Currently the existing arm midway boxes are "equiv-marilith", this turns out to
be a bad choice at least for this purpose, since marilith is the host name, not
the platform, we should either switch to or add equiv-midway. ISTR discussing
this at the time and picking equiv-marilith pretty much arbitrarily, AFAIK
nothing is using it explicitly today so I think removing and adding
equiv-midway is the right choice.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 make-flight | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/make-flight b/make-flight
index 9963a46..0e8e26a 100755
--- a/make-flight
+++ b/make-flight
@@ -284,10 +284,19 @@ do_passthrough_tests () {
 test_matrix_do_one () {
 
   # Basic PV Linux test with xl
+  case $xenarch in
+    armhf) platforms='midway cubietruck arndale' ;;
+    *)     platforms='' ;;
+  esac
+
+  for platform in ${platforms:-''} ; do
+    suffix=${platform:+-$platform}
+    hostflags=${most_hostflags}${platform:+,equiv-$platform}
 
-  job_create_test test-$xenarch$kern-$dom0arch-xl test-debian xl \
+    job_create_test test-$xenarch$kern-$dom0arch-xl$suffix test-debian xl \
             $xenarch $dom0arch                                   \
-            $debian_runvars all_hostflags=$most_hostflags
+            $debian_runvars all_hostflags=$hostflags
+  done
 
   job_create_test test-$xenarch$kern-$dom0arch-libvirt test-debian libvirt \
             $xenarch $dom0arch                                       \
-- 
2.1.0

[PATCH RFC OSSTEST 11/19] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG

[Ian Campbell]

This saves repeating identical HostProp and HostFlags for sets of identical
machines. e.g.

    HostGroupProp_cubietruck_LinuxSerialConsole ttyS0
    HostGroupProp_cubietruck_Build_Make_Flags -j12
    HostGroupProp_cubietruck_XenSerialConsole dtuart
    HostGroupProp_cubietruck_XenDTUARTPath /soc@01c00000/serial@01c28000
    HostGroupFlags_cubietruck suite-wheezy,equiv-cubietruck,need-kernel-deb-armmp,no-di-kernel,need-uboot-bootscr

    HostGroup_braque cubietruck
    HostProp_braque_Fqdn braque.uk.xensource.com

    HostGroup_picaso cubietruck
    HostProp_picaso_Fqdn picaso.uk.xensource.com

    HostGroup_metzinger cubietruck
    HostProp_metzinger metzinger.uk.xensource.com

    HostGroup_gleizes cubietruck
    HostProp_gleizes_Fqdn gleizes.uk.xensource.com

I was unsure whether HostGroupProp's ought to be processed before or after the
HostDB properties, but since the latter is a NOP I've arbitrarily put them
before.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/HostDB/Static.pm |  2 ++
 Osstest/TestSupport.pm   | 19 ++++++++++++++++++-
 README                   | 20 ++++++++++++++++++++
 3 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/Osstest/HostDB/Static.pm b/Osstest/HostDB/Static.pm
index d0c13a1..ad18395 100644
--- a/Osstest/HostDB/Static.pm
+++ b/Osstest/HostDB/Static.pm
@@ -58,6 +58,8 @@ sub get_flags ($$) { #method
     };
 
     $process->('HostFlags');
+    $process->("HostGroupFlags_$ho->{Properties}{HostGroup}")
+	if $ho->{Properties}{HostGroup};
     $process->("HostFlags_$ho->{Name}");
 
     return $flags;
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index d66708e..6901563 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -766,12 +766,29 @@ sub selecthost ($) {
 	$ho->{Properties}{$pn} = $val;
     };
 
-    # First, we use the config file's general properites as defaults
+    # First, set the prop group if any.
+    foreach my $k (keys %c) {
+	next unless $k =~ m/^HostGroup_([-a-z0-9]+)$/;
+	next unless $1 eq $name;
+	$setprop->("HostGroup", $c{$k});
+	logm("Host $name is in HostGroup $ho->{Properties}{HostGroup}");
+    }
+
+    # Next, we use the config file's general properites as defaults
     foreach my $k (keys %c) {
 	next unless $k =~ m/^HostProp_([A-Z].*)$/;
 	$setprop->($1, $c{$k});
     }
 
+    # Next, we set any HostGroup based properties
+    if ( $ho->{Properties}{HostGroup} ) {
+	foreach my $k (keys %c) {
+	    next unless $k =~ m/^HostGroupProp_([-a-z0-9]+)_(.*)$/;
+	    next unless $1 eq $ho->{Properties}{HostGroup};
+	    $setprop->($2, $c{$k});
+	}
+    }
+
     # Then we read in the HostDB's properties
     $mhostdb->get_properties($name, $ho->{Properties});
 
diff --git a/README b/README
index 9a85549..b4f9cfb 100644
--- a/README
+++ b/README
@@ -333,6 +333,26 @@ HostProp_<testbox>_TftpScope
    Defines the Tftp scope (i.e. subnet) where this host resides. See
    "TftpFoo_<scope> and TftpFoo" below.
 
+HostFlags_<testbox>
+   Defines a set of flags for the host. Flags is a list separated by
+   whitespace, comma or semi-colon. A flag can be unset by prepending
+   a !. Only used in standalone mode.
+
+HostGroup_<testbox> <group>
+   Defines a group of similar hosts of which <testbox> is a
+   member. This can then be used with HostGroupProp and HostGroupFlags
+
+HostGroupProps_<group>_<prop>
+   Equivalent to writing HostProp_<testbox>_<prop> for every testbox
+   which declares HostGroup_<testbox>_<group>. Allows setting a set of
+   common properties for a group of similar machines.
+
+HostGroupFlags_<group>
+   Equivalent to writing HostFlags_<testbox> for every testbox which
+   declares HostGroup_<testbox>_<group>. Allows setting a set of
+   common flags for a group of similar machines. Only used in
+   standalone mode.
+
 DebianPreseed
    Text to add to the debian-installer preseed file.  Optional
    but you will need to set some NTP servers here if your firewall
-- 
2.1.0

[PATCH RFC OSSTEST 12/19] Osstest/Debian: Refactor code to set bootargs in u-boot script

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 8f80eb4..3ffbc50 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -660,9 +660,15 @@ END
     }
 
     if ( $ho->{Flags}{'need-uboot-bootscr'} ) {
+	my @bootargs;
+
 	my $root=target_guest_lv_name($ho,"root");
-	my $console = get_host_native_linux_console($ho);
-	my $consolecmd = "console=$console" unless $console eq "NONE";
+	my $console=get_host_native_linux_console($ho);
+
+	push @bootargs, "root=$root";
+	push @bootargs, "console=$console" unless $console eq "NONE";
+
+	my $bootargs = join ' ', @bootargs;
 
 	preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
@@ -674,7 +680,7 @@ kernel=`readlink \$r/vmlinuz | sed -e 's|boot/||'`
 initrd=`readlink \$r/initrd.img | sed -e 's|boot/||'`
 
 cat >\$r/boot/boot <<EOF
-setenv bootargs $consolecmd root=$root
+setenv bootargs $bootargs
 mw.l 800000 0 10000
 scsi scan
 ext2load scsi 0 \\\${kernel_addr_r} \$kernel
-- 
2.1.0

[PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs

[Ian Campbell]

This is done unconditionally for every platform which has a
need-kernel-deb-* flag, which is currently all ARM platforms. Since
mg-debian-installer-update produces the necessary inputs for the only such
kernel (armmp) this seems tolerable for now.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 3ffbc50..9171c72 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -643,6 +643,11 @@ END
 		or die "Copy initramfs-tools failed: $!";
         });
 
+	my $durl = create_webfile($ho, "dtbs", sub {
+	    copy("$d_i/dtbs.tar.gz", $_[0])
+		or die "Copy dtbs failed: $!";
+	});
+
 	preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
 set -ex
@@ -651,6 +656,9 @@ r=/target
 
 wget -O \$r/tmp/kern.deb $kurl
 wget -O \$r/tmp/initramfs-tools.deb $iurl
+wget -O \$r/tmp/dtbs.tar.gz $durl
+
+in-target tar -C /boot -xaf /tmp/dtbs.tar.gz
 
 # This will fail due to dependencies...
 in-target dpkg -i /tmp/kern.deb /tmp/initramfs-tools.deb || true
-- 
2.1.0

[PATCH RFC OSSTEST 14/19] Osstest/Debian: Support for loading an FDT from u-boot script

[Ian Campbell]

The currently supported platform provides an FDT preloaded at 0x1000. Replace
this with ${fdt_addr} (which the current platform exposes) and for platforms
which do not provide an fdt arrange to load the relevant file as named in the
${fdtfile} (which is conventionally provided by u-boot for platforms which need
this).

Drop some random memory clearing rune, I've no idea what the intended purpose
was, 0x800000 doesn't correspond to any $foo_addr_r on the midway systems for
example.

Also get rid of the scsi scan which must necessarily have already happened
(since the boot.scr itselfs lives on a scsi drive).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 9171c72..19d6a22 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -141,9 +141,11 @@ cp -n /boot/boot.scr /boot/boot.scr.bak
 xen=`readlink /boot/$xen`
 
 cat >/boot/boot <<EOF
-
-mw.l 800000 0 10000
-scsi scan
+if test -z "\\\${fdt_addr}" && test -n "\\\${fdtfile}" ; then
+    echo Loading dtbs/\\\${fdtfile}
+    ext2load scsi 0 \\\${fdt_addr_r} dtbs/\\\${fdtfile}
+    setenv fdt_addr \\\${fdt_addr_r}
+fi
 
 fdt addr \\\${fdt_addr}
 fdt resize
@@ -689,11 +691,17 @@ initrd=`readlink \$r/initrd.img | sed -e 's|boot/||'`
 
 cat >\$r/boot/boot <<EOF
 setenv bootargs $bootargs
-mw.l 800000 0 10000
-scsi scan
+if test -z "\\\${fdt_addr}" && test -n "\\\${fdtfile}" ; then
+    echo Loading dtbs/\\\${fdtfile}
+    ext2load scsi 0 \\\${fdt_addr_r} dtbs/\\\${fdtfile}
+    setenv fdt_addr \\\${fdt_addr_r}
+fi
+echo Loading \$kernel
 ext2load scsi 0 \\\${kernel_addr_r} \$kernel
+echo Loading \$initrd
 ext2load scsi 0 \\\${ramdisk_addr_r} \$initrd
-bootz \\\${kernel_addr_r} \\\${ramdisk_addr_r}:\\\${filesize} 0x1000
+echo Booting
+bootz \\\${kernel_addr_r} \\\${ramdisk_addr_r}:\\\${filesize} \\\${fdt_addr}
 EOF
 
 in-target mkimage -A arm -T script -d /boot/boot /boot/boot.scr
-- 
2.1.0

[PATCH RFC OSSTEST 15/19] Osstest/Debian: Add support for "ExtraInitramfsModules" host property

[Ian Campbell]

The arndale platform needs a bunch of clk, phy and regulator stuff in order to
access its root filesystem. However mkinitramfs is not (currently?) able to
figure this out and therefore doesn't include them in the initrd.

Add a new host prop which can list these required additional module and
arranges for a suitable /etc/initramfs-tools/modules to be created on install.

Using the new HostGroupProp syntax the required modules are:

HostGroupProp_arndale_ExtraInitramfsModules clk-s2mps11 s5m8767 i2c-s3c2410 phy-exynos5250-sata

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 19d6a22..97a6cda 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -708,6 +708,20 @@ in-target mkimage -A arm -T script -d /boot/boot /boot/boot.scr
 END
     }
 
+    my $modules = get_host_property($ho, "ExtraInitramfsModules", "NONE");
+    if ( $modules ne "NONE" )
+    {
+        preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+for i in $modules ; do
+    echo \$i >> /target/etc/initramfs-tools/modules
+done
+in-target update-initramfs -u -k all
+END
+    }
+
     my @extra_packages = ();
     push(@extra_packages, "u-boot-tools") if $ho->{Flags}{'need-uboot-bootscr'};
 
-- 
2.1.0

[PATCH RFC OSSTEST 16/19] Osstest/Debian: support adding a rootdelay property to bootargs

[Ian Campbell]

arndale appears to be quite slow (and asynchronous) at finding it's scsi
controller. rootdelay=3 seems to do the trick.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 97a6cda..eca4d7f 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -673,9 +673,11 @@ END
 	my @bootargs;
 
 	my $root=target_guest_lv_name($ho,"root");
+	my $rootdelay=get_host_property($ho, "rootdelay");
 	my $console=get_host_native_linux_console($ho);
 
 	push @bootargs, "root=$root";
+	push @bootargs, "rootdelay=$rootdelay";
 	push @bootargs, "console=$console" unless $console eq "NONE";
 
 	my $bootargs = join ' ', @bootargs;
-- 
2.1.0

[PATCH RFC OSSTEST 17/19] Osstest/Debian: Remove hardcoded midway specific addresses from boot.scr

[Ian Campbell]

This needs to go along with a change to the hostdb to add "setenv xen_addr_r
0x01000000" to the UBootScriptEarlyCommands property of the midway machines.

Most platforms will need something similar. For both cubietruck and arndale
"setenv xen_addr_r 0x41000000" is appropriate.

This is going to be a pretty common requirement on ARM/uboot systems, should I
make it a specific property instead of reusing the generic hook?

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index eca4d7f..4539452 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -155,10 +155,6 @@ ${early_commands}
 fdt set /chosen \\\#address-cells <1>
 fdt set /chosen \\\#size-cells <1>
 
-setenv xen_addr_r 0x01000000
-#   kernel_addr_r=0x02000000
-#  ramdisk_addr_r=0x04000000
-
 ext2load scsi 0 \\\${xen_addr_r} \$xen
 setenv bootargs "$xenhopt"
 echo Loaded \$xen to \\\${xen_addr_r} (\\\${filesize})
-- 
2.1.0

[PATCH RFC OSSTEST 18/19] ts-kernel-build: Enable CONFIG_PHY_EXYNOS5250_SATA

[Ian Campbell]

Enables the disk on an arndale, which isn't enabled by default in v3.16 for
some reason.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ts-kernel-build | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/ts-kernel-build b/ts-kernel-build
index 3b48920..bae118c 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -171,6 +171,8 @@ setopt CONFIG_BLK_DEV_NBD y
 # At least with Linux 3.4.77 on wheezy, the nbd module is
 # not loaded automatically.
 
+setopt CONFIG_PHY_EXYNOS5250_SATA y
+
 END
 
 our $config_features= <<END;
-- 
2.1.0

[PATCH RFC OSSTEST 19/19] Osstest/Debian: Workaround oddities in the u-boot script parser.

[Ian Campbell]

${filesize} is a bare hex number (i.e. no 0x prefix), but apparently 'fdt set'
needs the 0x prefix to make sense of it.

Furthermore it is confused by "0xABCD>" (and "ABCD>") and complains because ">"
isn't a valid digit. So put a space after the number.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 4539452..2fe4728 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -163,16 +163,16 @@ echo command line: \\\${bootargs}
 ext2load scsi 0 \\\${kernel_addr_r} $kern
 fdt mknod /chosen module\@0
 fdt set /chosen/module\@0 compatible "xen,linux-zimage" "xen,multiboot-module"
-fdt set /chosen/module\@0 reg <\\\${kernel_addr_r} \\\${filesize}>
-fdt set /chosen/module\@0 bootargs "$xenkopt ro root=$root"
-echo Loaded $kern to \\\${kernel_addr_r} (\\\${filesize})
+fdt set /chosen/module\@0 reg <\\\${kernel_addr_r} 0x\\\${filesize} >
+fdt set /chosen/module\@0 bootargs "$xenkopt ro root=$root clk_ignore_unused"
+echo Loaded $kern to \\\${kernel_addr_r} (0x\\\${filesize})
 echo command line: $xenkopt ro root=$root
 
 ext2load scsi 0 \\\${ramdisk_addr_r} $initrd
 fdt mknod /chosen module\@1
 fdt set /chosen/module\@1 compatible "xen,linux-initrd" "xen,multiboot-module"
-fdt set /chosen/module\@1 reg <\\\${ramdisk_addr_r} \\\${filesize}>
-echo Loaded $initrd to \\\${ramdisk_addr_r} (\\\${filesize})
+fdt set /chosen/module\@1 reg <\\\${ramdisk_addr_r} 0x\\\${filesize} >
+echo Loaded $initrd to \\\${ramdisk_addr_r} (0x\\\${filesize})
 
 fdt print /chosen
 
-- 
2.1.0

Re: [PATCH RFC OSSTEST 01/19] standalone: Provide a helper to clear the host runvar

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 01/19] standalone: Provide a helper to clear the host runvar"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Of course this only works for jobs with a single host whose runvar is
`host' rather than (say) `src_host' or whatever.  But:

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH RFC OSSTEST 01/19] standalone: Provide a helper to clear the host runvar

[Ian Campbell]

On Fri, 2014-10-10 at 14:51 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 01/19] standalone: Provide a helper to clear the host runvar"):
> > Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> 
> Of course this only works for jobs with a single host whose runvar is
> `host' rather than (say) `src_host' or whatever.

Ah yes, I suppose I never run such tests in standalone mode.

Is there some way to specify multiple hosts? OSSTEST_HOST_SRC_HOST= and
OSSTEST_HOST_DST_HOST= perhaps?

>   But:
> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH RFC OSSTEST 02/19] Osstest/PDU: Add eth008.pm method to control the ARM rack PDU

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 02/19] Osstest/PDU: Add eth008.pm method to control the ARM rack PDU"):
> This controls the eth008 relay board: http://www.robot-electronics.co.uk/htm/eth008tech.htm
> 
> Due to the use of the CGI interface this requires firmware version 4+.

In the future we are likely to want to have more community members
working with osstest and that probably means we want to have a less
juicy `inside' to our network.

That probably means nontrivial passwords, but it also probably means
we'll want to stop passing passwords on command lines.

> +    my $url = "http://$mo->{PDU}/io.cgi?$op$mo->{Port}=0";
> +    my $cmd = "curl --silent --user $mo->{User}:$mo->{Pass} $url";

Is there a way to do this where the password goes in a file or via an
fd ?

I'm tempted to suggest that you should use the non-CGI interface and
get the firewall fixed...

Ian.

Re: [PATCH RFC OSSTEST 03/19] mg-debian-installer-update: Include any device tree blobs in the tftp dir

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 03/19] mg-debian-installer-update: Include any device tree blobs in the tftp dir"):
> We will need to supply these for some arm boards.
> 
> We provide both an unpacked version, which can be referenced via u-boot's
> support for the "fdtdir" property in pxelinux.cfg and a tarball which is more
> convenient for installing into the target filesystem on host install

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH RFC OSSTEST 04/19] mg-debian-installer-update: Add more modules when creating initrd overlay

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 04/19] mg-debian-installer-update: Add more modules when creating initrd overlay"):
> The arndale platform has a usb network device. In addition both the SATA and
> NIC/USB hardware on this platform requires additional regulator (power), clk
> and phy modules, which in turn require i2c support.

This is quite a list.  I assume we need to do this filtering to make
the initrd not too big.  So:

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Ian.

Re: [PATCH RFC OSSTEST 05/19] ts-host-install: Add fdtdir to PXE stanza if dtbs directory exists

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 05/19] ts-host-install: Add fdtdir to PXE stanza if dtbs directory exists"):
> u-boot's pxe client uses this together with $fdtfile in its local environment
> to automatically load the fdt when pxe booting.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Although, bikeshedding,

> +    my $dtbs = "fdtdir /$d_i/dtbs" if
> +	-e "$ho->{Tftp}{Path}/$d_i/dtbs";
> +

the if at the end of the line strikes me as a slightly odd style.
There are many fewer of those in the codebase right now than places
where it's on the start and the next line.

Ian.

Re: [PATCH RFC OSSTEST 06/19] ts-host-install: Refactor code to construct a Linux console= for a host

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 06/19] ts-host-install: Refactor code to construct a Linux console= for a host"):
> And reuse it in when writing the u-boot script for initial boot. This was
> previously hardcoded to ttyAMA which matches the production host DB for the
> current set of hosts which use this script.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH RFC OSSTEST 07/19] ts-host-install: include console before *and* after the -- marker.

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 07/19] ts-host-install: include console before *and* after the -- marker."):
> As of v3.15 Linux now stop parsing its command line at the -- mark and passes
> the remainder to init. This has broken Debian Installer's feature where
> anything after the -- is propagated to the installed system.
> 
> See also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007

Urgh.  Thanks, systemd.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH RFC OSSTEST 02/19] Osstest/PDU: Add eth008.pm method to control the ARM rack PDU

[Ian Campbell]

On Fri, 2014-10-10 at 14:56 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 02/19] Osstest/PDU: Add eth008.pm method to control the ARM rack PDU"):
> > This controls the eth008 relay board: http://www.robot-electronics.co.uk/htm/eth008tech.htm
> > 
> > Due to the use of the CGI interface this requires firmware version 4+.
> 
> In the future we are likely to want to have more community members
> working with osstest and that probably means we want to have a less
> juicy `inside' to our network.
> 
> That probably means nontrivial passwords, but it also probably means
> we'll want to stop passing passwords on command lines.
> 
> > +    my $url = "http://$mo->{PDU}/io.cgi?$op$mo->{Port}=0";
> > +    my $cmd = "curl --silent --user $mo->{User}:$mo->{Pass} $url";
> 
> Is there a way to do this where the password goes in a file or via an
> fd ?

There's --netrc which looks in ~/.netrc (or use --netrc-file).

> I'm tempted to suggest that you should use the non-CGI interface and
> get the firewall fixed...

Probably easiest to do it as part of the move, when we (I pressume) gain
control over the f/w, if any, between control host and test machines?

I could code it as an option for now if you like.

Ian.

Re: [PATCH RFC OSSTEST 08/19] ts-host-install: Add option to boot debian-installer in rescue mode

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 08/19] ts-host-install: Add option to boot debian-installer in rescue mode"):
> This is useful when trying to bootstrap onto a new platform, since
> you can log in to an otherwise unbootable rootfs to figure out what
> is going wrong.
> 
> The procedure is the same as installing except "rescue/enabled=true" is added
> to the command line, hence part of ts-host-install rather than separate.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH RFC OSSTEST 04/19] mg-debian-installer-update: Add more modules when creating initrd overlay

[Ian Campbell]

On Fri, 2014-10-10 at 14:58 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 04/19] mg-debian-installer-update: Add more modules when creating initrd overlay"):
> > The arndale platform has a usb network device. In addition both the SATA and
> > NIC/USB hardware on this platform requires additional regulator (power), clk
> > and phy modules, which in turn require i2c support.
> 
> This is quite a list.  I assume we need to do this filtering to make
> the initrd not too big.

That was the original idea. Totally unscientifically:
$ du -shc armmp.deb,initrd.gz
23M	armmp.deb
5.2M	initrd.gz

Where the vast majority of the .deb is modules. So it is reducing the
size pretty considerably.

>   So:
> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
> 
> Ian.

Re: [PATCH RFC OSSTEST 05/19] ts-host-install: Add fdtdir to PXE stanza if dtbs directory exists

[Ian Campbell]

On Fri, 2014-10-10 at 15:01 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 05/19] ts-host-install: Add fdtdir to PXE stanza if dtbs directory exists"):
> > u-boot's pxe client uses this together with $fdtfile in its local environment
> > to automatically load the fdt when pxe booting.
> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
> 
> Although, bikeshedding,
> 
> > +    my $dtbs = "fdtdir /$d_i/dtbs" if
> > +	-e "$ho->{Tftp}{Path}/$d_i/dtbs";
> > +
> 
> the if at the end of the line strikes me as a slightly odd style.
> There are many fewer of those in the codebase right now than places
> where it's on the start and the next line.

Yes, in fact I would naturally have done it the other way, I suppose I
wasn't paying close enough attention to my fingers...

> 
> Ian.

Re: [PATCH RFC OSSTEST 09/19] ts-host-install: Add force-mac-address host flag

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 09/19] ts-host-install: Add force-mac-address host flag"):
> This causes us to emit udev runes which force the device named in
> the host's Interface_Force property to have the MAC address given in
> the host's Ether property. This is the reverse of the behaviour of
> the Interface_Force property without this flag, which is to rename
> the device with MAC address Ether to have the requested name.

There is a potential problem with this, which is that Linux doctrine
is that devices do not have stable names.   So if you don't identify
the device by mac address, and your system has more than one network
interface, osstest can't identify the device reliably at all.

Do your actual boards have more than one eth* ?

Ian.

Re: [PATCH RFC OSSTEST 09/19] ts-host-install: Add force-mac-address host flag

[Ian Campbell]

On Fri, 2014-10-10 at 15:09 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 09/19] ts-host-install: Add force-mac-address host flag"):
> > This causes us to emit udev runes which force the device named in
> > the host's Interface_Force property to have the MAC address given in
> > the host's Ether property. This is the reverse of the behaviour of
> > the Interface_Force property without this flag, which is to rename
> > the device with MAC address Ether to have the requested name.
> 
> There is a potential problem with this, which is that Linux doctrine
> is that devices do not have stable names.   So if you don't identify
> the device by mac address, and your system has more than one network
> interface, osstest can't identify the device reliably at all.
> 
> Do your actual boards have more than one eth* ?

Just the one.

Ian.

Re: [PATCH RFC OSSTEST 10/19] make-flight: Run a basic test on each arm platform

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 10/19] make-flight: Run a basic test on each arm platform"):
> Unlike x86 there is enough variation in the ARM platforms that it is
> worth having a basic test on each as part of a standard run. This
> relies on each host having an appropriate equiv-$platform host flag.

I think using equiv-* for this is not correct.

Hosts named equiv-* are supposed to be completely identical, but we
might end up with hosts which are similar enough that they qualify for
this cross-architecture testing, and but not completely identical.

Furthermore, using equiv-* makes it difficult to automatically search
for a list of the different board classes we would want to use.

I would invent a new prefix.  Perhaps flags like
  platform-midway
etc.

Then you could search the database

  SELECT UNIQUE platform
  WHERE EXISTS host
     SUCHTHAT (host,"arch-$arch") in hostflags
          AND (host,"blessed-$whatever") in hostflags
          etc.

rather than hardcoding the list in make-flight.


> Currently the existing arm midway boxes are "equiv-marilith", this
> turns out to be a bad choice at least for this purpose, since
> marilith is the host name, not the platform, we should either switch
> to or add equiv-midway. ISTR discussing this at the time and picking
> equiv-marilith pretty much arbitrarily, AFAIK nothing is using it
> explicitly today so I think removing and adding equiv-midway is the
> right choice.

See above.  `equiv-marilith' would be correct if we had another
identical Calxeda box.

Ian.

Re: [PATCH RFC OSSTEST 10/19] make-flight: Run a basic test on each arm platform

[Ian Campbell]

On Fri, 2014-10-10 at 15:20 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 10/19] make-flight: Run a basic test on each arm platform"):
> > Unlike x86 there is enough variation in the ARM platforms that it is
> > worth having a basic test on each as part of a standard run. This
> > relies on each host having an appropriate equiv-$platform host flag.
> 
> I think using equiv-* for this is not correct.
> 
> Hosts named equiv-* are supposed to be completely identical, but we
> might end up with hosts which are similar enough that they qualify for
> this cross-architecture testing, and but not completely identical.
> 
> Furthermore, using equiv-* makes it difficult to automatically search
> for a list of the different board classes we would want to use.
> 
> I would invent a new prefix.  Perhaps flags like
>   platform-midway
> etc.

Would I have to add some code somewhere to process this? (I'm often
surprised to find that these just work via some generic magic, e.g. the
buildjob stuff etc, but I don't think so in this case?)

Perhaps in ts-hosts-allocate-Executive alongside the equiv handling?

> Then you could search the database
> 
>   SELECT UNIQUE platform
>   WHERE EXISTS host
>      SUCHTHAT (host,"arch-$arch") in hostflags
>           AND (host,"blessed-$whatever") in hostflags
>           etc.
> 
> rather than hardcoding the list in make-flight.

I think this wouldn't work for standalone mode, would it?

> > Currently the existing arm midway boxes are "equiv-marilith", this
> > turns out to be a bad choice at least for this purpose, since
> > marilith is the host name, not the platform, we should either switch
> > to or add equiv-midway. ISTR discussing this at the time and picking
> > equiv-marilith pretty much arbitrarily, AFAIK nothing is using it
> > explicitly today so I think removing and adding equiv-midway is the
> > right choice.
> 
> See above.  `equiv-marilith' would be correct if we had another
> identical Calxeda box.

Given "marilith" is the host name and not any sort of name which would
identify "similar" machines -- really?

Re: [PATCH RFC OSSTEST 14/19] Osstest/Debian: Support for loading an FDT from u-boot script

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 14/19] Osstest/Debian: Support for loading an FDT from u-boot script"):
> The currently supported platform provides an FDT preloaded at
> 0x1000. Replace this with ${fdt_addr} (which the current platform
> exposes) and for platforms which do not provide an fdt arrange to
> load the relevant file as named in the ${fdtfile} (which is
> conventionally provided by u-boot for platforms which need this).

You should mention the extra `echo's in the commit message.

You have two copies of the same dtb-loading code.  (You had two copies
of the `scsi scan' and `mw.l' previously, but this is now worse
because it's bigger.)

I confess I don't really understand some of this.  `Platforms which do
not provide an fdt' are ones where ${fdt_addr} is empty ?  And on
those platforms `fdt addr \${fdt_addr}' is a no-op ?

You might find the code clearer (less toothpick-counting) if you used
<<'delim' for some of it, at one or other of the levels.

Thanks,
Ian.

Re: [PATCH RFC OSSTEST 17/19] Osstest/Debian: Remove hardcoded midway specific addresses from boot.scr

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 17/19] Osstest/Debian: Remove hardcoded midway specific addresses from boot.scr"):
> This needs to go along with a change to the hostdb to add "setenv xen_addr_r
> 0x01000000" to the UBootScriptEarlyCommands property of the midway machines.

`Go along with' is not possible.  Do you mean `this depends on' ?

> This is going to be a pretty common requirement on ARM/uboot
> systems, should I make it a specific property instead of reusing the
> generic hook?

Yes.

Ian.

Re: [PATCH RFC OSSTEST 12/19] Osstest/Debian: Refactor code to set bootargs in u-boot script

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 12/19] Osstest/Debian: Refactor code to set bootargs in u-boot script"):
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH RFC OSSTEST 18/19] ts-kernel-build: Enable CONFIG_PHY_EXYNOS5250_SATA

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 18/19] ts-kernel-build: Enable CONFIG_PHY_EXYNOS5250_SATA"):
> Enables the disk on an arndale, which isn't enabled by default in v3.16 for
> some reason.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH RFC OSSTEST 17/19] Osstest/Debian: Remove hardcoded midway specific addresses from boot.scr

[Ian Campbell]

On Fri, 2014-10-10 at 15:30 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 17/19] Osstest/Debian: Remove hardcoded midway specific addresses from boot.scr"):
> > This needs to go along with a change to the hostdb to add "setenv xen_addr_r
> > 0x01000000" to the UBootScriptEarlyCommands property of the midway machines.
> 
> `Go along with' is not possible.  Do you mean `this depends on' ?

Yes. The hostdb change can happen first without breaking anything.

> > This is going to be a pretty common requirement on ARM/uboot
> > systems, should I make it a specific property instead of reusing the
> > generic hook?
> 
> Yes.

Ack

Re: [PATCH RFC OSSTEST 14/19] Osstest/Debian: Support for loading an FDT from u-boot script

[Ian Campbell]

On Fri, 2014-10-10 at 15:29 +0100, Ian Jackson wrote:
> I confess I don't really understand some of this.  `Platforms which do
> not provide an fdt' are ones where ${fdt_addr} is empty ?

Yes.

Midway (marilith) provides a dtb in the firmware, and it is located at
$fdt_addr. This is how device tree is supposed to work, but is actually
vanishingly rare on ARM+u-boot in real life (I've never seen another
such platform...)

In practice most ARM+u-boot platforms require you to load the dtb from
somewhere by hand, and provide fdt_addr_r as the place you should put
it.

(EFI on arm64 is a bit better here...)

>   And on
> those platforms `fdt addr \${fdt_addr}' is a no-op ?

fdt_addr is set by the new code on those platforms.

> You might find the code clearer (less toothpick-counting) if you used
> <<'delim' for some of it, at one or other of the levels.

You mean as opposed to <<delim because it changes the quoting behaviour?

That would break places where I wanted to expand a Perl var, I think?
Perhaps those uses can be broken out in some convenient way and
concatenated back again. I'll give it a go and see how it looks.

Ian.

Re: [PATCH RFC OSSTEST 11/19] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 11/19] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG"):
> This saves repeating identical HostProp and HostFlags for sets of identical
> machines. e.g.

The existing code is rather weird in that host properties from the
configuration are dealt with in TestSupport (so apply to both
standalone and executive), but host flags in the config are dealt with
in HostDB/Static.pm (so apply only to standalone).

> I was unsure whether HostGroupProp's ought to be processed before or
> after the HostDB properties, but since the latter is a NOP I've
> arbitrarily put them before.

The HostDB properties are not a NOP if the HostDB isn't Static!

In general the configuration should override the database.  So I think
this is the wrong way round.

> diff --git a/README b/README
> index 9a85549..b4f9cfb 100644
> --- a/README
> +++ b/README
> @@ -333,6 +333,26 @@ HostProp_<testbox>_TftpScope
>     Defines the Tftp scope (i.e. subnet) where this host resides. See
>     "TftpFoo_<scope> and TftpFoo" below.
>  
> +HostFlags_<testbox>
> +   Defines a set of flags for the host. Flags is a list separated by
> +   whitespace, comma or semi-colon. A flag can be unset by prepending
> +   a !. Only used in standalone mode.
> +
> +HostGroup_<testbox> <group>
> +   Defines a group of similar hosts of which <testbox> is a
> +   member. This can then be used with HostGroupProp and HostGroupFlags

Thanks for documenting this.

> +HostGroupProps_<group>_<prop>
> +   Equivalent to writing HostProp_<testbox>_<prop> for every testbox
> +   which declares HostGroup_<testbox>_<group>. Allows setting a set of
> +   common properties for a group of similar machines.

You should specify that this overrides the hostdb but is overridden by
host-specific properties.

> +HostGroupFlags_<group>
> +   Equivalent to writing HostFlags_<testbox> for every testbox which
> +   declares HostGroup_<testbox>_<group>. Allows setting a set of
> +   common flags for a group of similar machines. Only used in
> +   standalone mode.

You should say that this is merged with the host-specific properties
and applies only to standlone mode.

Ian.

Re: [PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs"):
> This is done unconditionally for every platform which has a
> need-kernel-deb-* flag, which is currently all ARM platforms. Since
> mg-debian-installer-update produces the necessary inputs for the only such
> kernel (armmp) this seems tolerable for now.

Surely it would be better to check whether the source file exists and
use it iff it does ?

Ian.

Re: [PATCH RFC OSSTEST 15/19] Osstest/Debian: Add support for "ExtraInitramfsModules" host property

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 15/19] Osstest/Debian: Add support for "ExtraInitramfsModules" host property"):
> The arndale platform needs a bunch of clk, phy and regulator stuff
> in order to access its root filesystem. However mkinitramfs is not
> (currently?) able to figure this out and therefore doesn't include
> them in the initrd.

Is this not a bug in mkinitrams or something ?  If so I think we
should provide a reference to the bug here (eg bugs.debian.org URL or
something).

I'm surprised that there isn't a less-raw way of doing this with d-i,
than using a late-command.

Ian.

Re: [PATCH RFC OSSTEST 11/19] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG

[Ian Campbell]

On Fri, 2014-10-10 at 15:44 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 11/19] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG"):
> > This saves repeating identical HostProp and HostFlags for sets of identical
> > machines. e.g.
> 
> The existing code is rather weird in that host properties from the
> configuration are dealt with in TestSupport (so apply to both
> standalone and executive), but host flags in the config are dealt with
> in HostDB/Static.pm (so apply only to standalone).

Yes, I noticed that... No reason for it, just hysterical raisins?

> > I was unsure whether HostGroupProp's ought to be processed before or
> > after the HostDB properties, but since the latter is a NOP I've
> > arbitrarily put them before.
> 
> The HostDB properties are not a NOP if the HostDB isn't Static!

Sorry, I meant for the Static HostDB, on the basis that I don't think
you would use the config file with the proper DB, would you?

(which might argue for moving a bunch of code from TestSupport to
HostDB/Static.pm?)

> In general the configuration should override the database.  So I think
> this is the wrong way round.

Ack.

Ian.

Re: [PATCH RFC OSSTEST 19/19] Osstest/Debian: Workaround oddities in the u-boot script parser.

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 19/19] Osstest/Debian: Workaround oddities in the u-boot script parser."):
> ${filesize} is a bare hex number (i.e. no 0x prefix), but apparently
> 'fdt set' needs the 0x prefix to make sense of it.

WTF.

> Furthermore it is confused by "0xABCD>" (and "ABCD>") and complains
> because ">" isn't a valid digit. So put a space after the number.

Surely these are bugs, so can we have the ref to the upstream bug
tracker ?

Thanks,
Ian.

Re: [PATCH RFC OSSTEST 16/19] Osstest/Debian: support adding a rootdelay property to bootargs

[Ian Jackson]

Ian Campbell writes ("[PATCH RFC OSSTEST 16/19] Osstest/Debian: support adding a rootdelay property to bootargs"):
> arndale appears to be quite slow (and asynchronous) at finding it's scsi
> controller. rootdelay=3 seems to do the trick.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs

[Ian Campbell]

On Fri, 2014-10-10 at 15:45 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs"):
> > This is done unconditionally for every platform which has a
> > need-kernel-deb-* flag, which is currently all ARM platforms. Since
> > mg-debian-installer-update produces the necessary inputs for the only such
> > kernel (armmp) this seems tolerable for now.
> 
> Surely it would be better to check whether the source file exists and
> use it iff it does ?

I think I was just being lazy trying to reuse the existing hook script
etc.

I could throw the script text into a variable and include it that way
though.

Re: [PATCH RFC OSSTEST 15/19] Osstest/Debian: Add support for "ExtraInitramfsModules" host property

[Ian Campbell]

On Fri, 2014-10-10 at 15:47 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 15/19] Osstest/Debian: Add support for "ExtraInitramfsModules" host property"):
> > The arndale platform needs a bunch of clk, phy and regulator stuff
> > in order to access its root filesystem. However mkinitramfs is not
> > (currently?) able to figure this out and therefore doesn't include
> > them in the initrd.
> 
> Is this not a bug in mkinitrams or something ?  If so I think we
> should provide a reference to the bug here (eg bugs.debian.org URL or
> something).

Yes, I should file one.

> I'm surprised that there isn't a less-raw way of doing this with d-i,
> than using a late-command.

So was I.

Ian.

Re: [PATCH RFC OSSTEST 09/19] ts-host-install: Add force-mac-address host flag

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH RFC OSSTEST 09/19] ts-host-install: Add force-mac-address host flag"):
> On Fri, 2014-10-10 at 15:09 +0100, Ian Jackson wrote:
> > Ian Campbell writes ("[PATCH RFC OSSTEST 09/19] ts-host-install: Add force-mac-address host flag"):
> > > This causes us to emit udev runes which force the device named in
> > > the host's Interface_Force property to have the MAC address given in
> > > the host's Ether property. This is the reverse of the behaviour of
> > > the Interface_Force property without this flag, which is to rename
> > > the device with MAC address Ether to have the requested name.
> > 
> > There is a potential problem with this, which is that Linux doctrine
> > is that devices do not have stable names.   So if you don't identify
> > the device by mac address, and your system has more than one network
> > interface, osstest can't identify the device reliably at all.
> > 
> > Do your actual boards have more than one eth* ?
> 
> Just the one.

OK then,

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH RFC OSSTEST 10/19] make-flight: Run a basic test on each arm platform

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH RFC OSSTEST 10/19] make-flight: Run a basic test on each arm platform"):
> Would I have to add some code somewhere to process this? (I'm often
> surprised to find that these just work via some generic magic, e.g. the
> buildjob stuff etc, but I don't think so in this case?)

(We discussed this in person.)

> > See above.  `equiv-marilith' would be correct if we had another
> > identical Calxeda box.
> 
> Given "marilith" is the host name and not any sort of name which would
> identify "similar" machines -- really?

Modulo that `marilith' is the wrong identifier and that I complained
about that at the time, yes :-).

Ian.

Re: [PATCH RFC OSSTEST 14/19] Osstest/Debian: Support for loading an FDT from u-boot script

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH RFC OSSTEST 14/19] Osstest/Debian: Support for loading an FDT from u-boot script"):
> [stuff]

Thanks for the explanation.

> > You might find the code clearer (less toothpick-counting) if you used
> > <<'delim' for some of it, at one or other of the levels.
> 
> You mean as opposed to <<delim because it changes the quoting behaviour?
> 
> That would break places where I wanted to expand a Perl var, I think?
> Perhaps those uses can be broken out in some convenient way and
> concatenated back again. I'll give it a go and see how it looks.

Right.  In Perl you can write
    blah <<END.<<'END';
for example.

Ian.

Re: [PATCH RFC OSSTEST 11/19] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH RFC OSSTEST 11/19] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG"):
> On Fri, 2014-10-10 at 15:44 +0100, Ian Jackson wrote:
> > The HostDB properties are not a NOP if the HostDB isn't Static!
> 
> Sorry, I meant for the Static HostDB, on the basis that I don't think
> you would use the config file with the proper DB, would you?

You might set some properties from the config and some from the DB.  I
agree that this would be unusual but I wouldn't want to rule it out
and I think the precedence ought to be in favour of the config file.

> (which might argue for moving a bunch of code from TestSupport to
> HostDB/Static.pm?)

Let's not tackle that now :-).

Ian.

Re: [PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs"):
> On Fri, 2014-10-10 at 15:45 +0100, Ian Jackson wrote:
> > Surely it would be better to check whether the source file exists and
> > use it iff it does ?
> 
> I think I was just being lazy trying to reuse the existing hook script
> etc.

Uh ?  You could reuse the existing hook script but just put the call
to add the hook inside a conditional.

Ian.

Re: [PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs

[Ian Campbell]

On Fri, 2014-10-10 at 16:10 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs"):
> > On Fri, 2014-10-10 at 15:45 +0100, Ian Jackson wrote:
> > > Surely it would be better to check whether the source file exists and
> > > use it iff it does ?
> > 
> > I think I was just being lazy trying to reuse the existing hook script
> > etc.
> 
> Uh ?  You could reuse the existing hook script but just put the call
> to add the hook inside a conditional.

I meant reuse existing <<here doc with a script in it, rather than
introduce a second call to the helper etc. As I say, lazy ;-)

Ian.

Re: [PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH RFC OSSTEST 13/19] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs"):
> On Fri, 2014-10-10 at 16:10 +0100, Ian Jackson wrote:
> > Uh ?  You could reuse the existing hook script but just put the call
> > to add the hook inside a conditional.
> 
> I meant reuse existing <<here doc with a script in it, rather than
> introduce a second call to the helper etc. As I say, lazy ;-)

OIC.  The whole point about this helper is that you can call it
multiple times...

Ian.

Re: [PATCH RFC OSSTEST 10/19] make-flight: Run a basic test on each arm platform

[Ian Campbell]

On Fri, 2014-10-10 at 15:20 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH RFC OSSTEST 10/19] make-flight: Run a basic test on each arm platform"):
> > Unlike x86 there is enough variation in the ARM platforms that it is
> > worth having a basic test on each as part of a standard run. This
> > relies on each host having an appropriate equiv-$platform host flag.
> 
> I think using equiv-* for this is not correct.
> 
> Hosts named equiv-* are supposed to be completely identical, but we
> might end up with hosts which are similar enough that they qualify for
> this cross-architecture testing, and but not completely identical.
> 
> Furthermore, using equiv-* makes it difficult to automatically search
> for a list of the different board classes we would want to use.
> 
> I would invent a new prefix.  Perhaps flags like
>   platform-midway
> etc.
> 
> Then you could search the database
> 
>   SELECT UNIQUE platform
>   WHERE EXISTS host
>      SUCHTHAT (host,"arch-$arch") in hostflags
>           AND (host,"blessed-$whatever") in hostflags
>           etc.

This was just a pseudo idea, right? Because the Internet has never heard
of this EXISTS ... SUCHTHAT syntax that appeared so tantalizingly
useful.

I think the real answer is going to involve some exciting JOIN-s?
Something like (using equiv-% as a surrogate for the currently
non-existent platform-*):

SELECT DISTINCT h4.hostflag
FROM hostflags h1
INNER JOIN hostflags h2 ON h1.hostname = h2.hostname AND h2.hostflag = 'arch-amd64'
INNER JOIN hostflags h3 ON h1.hostname = h3.hostname AND h3.hostflag = 'blessed-real'
INNER JOIN hostflags h4 ON h1.hostname = h4.hostname AND h4.hostflag LIKE 'equiv-%';

which lists:
 equiv-rackservers-s40670
 equiv-r310-moth
 equiv-dell-r310
 equiv-rackservers-s40680
 equiv-rackservers-s40679
 equiv-rackservers-s40663
 equiv-rackservers-q21011

Was it your intention that every machine was given a platform-*
property? Or do I need to create code to handle the "everything else"
case? i.e. on x86 where there is no platform do we want to retain the
exact same set of jobs?

Ian.

[PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms

[Ian Campbell]

I'm preparing a bunch of each of these boards to be racked for inclusion
in osstest. This series adds the necessary osstest support:

      * Support for platforms which require us to supply a Device Tree
      * Removing various hardcoded midway assumptions
      * Some new features needed to workaround h/w quirks (of arndale in
        particular)
      * Support for the power relay switch board which these are
        attached to
      * Some new convenience features

I have run a full (including host reinstall) build-* job on each
platform.

I have successfully run test-armhf-armhf-xl on cubietruck, but not yet
on arndale (struggling to find a pvops config which boots, even
natively!). I thought I'd send the bulk out to get started with.

For reference the relevant bits of my ~/.xen-osstest/config are below.

Since v1 I've reordered to pulled the stuff which relates to updating
mg-debian-installer-update and TftpDiVersion to the front, in the hopes
of getting that in sooner. This includes everything up to at least
"ts-host-install: include console before *and* after the -- marker."
which is needed because rerunning mg-debian-installer-update today will
pull in a kernel which is affected by that issue. In reality the initial
set of acked patches here is larger.

Ian.

#HostProp_metrocentre_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 1
HostProp_westfield_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 2
HostProp_lakeside_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 3
HostProp_bluewater_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 4

HostProp_braque_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 5
HostProp_picaso_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 6
HostProp_metzinger_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 7
HostProp_gleizes_PowerMethod eth008 arm-pdu-01.uk.xensource.com admin password 8

HostGroupProp_arndale_LinuxSerialConsole ttySAC2
HostGroupProp_arndale_Build_Make_Flags -j4
HostGroupProp_arndale_XenSerialConsole dtuart
HostGroupProp_arndale_XenDTUARTPath /serial@12C20000
HostGroupProp_arndale_Interface_Force eth0
HostGroupProp_arndale_ExtraInitramfsModules clk-s2mps11 s5m8767 i2c-s3c2410 phy-exynos5250-sata
HostGroupProp_arndale_Rootdelay 3
HostGroupProp_arndale_UBootScriptEarlyCommands setenv xen_addr_r 0x41000000

HostGroupFlags_arndale suite-wheezy,equiv-arndale,need-kernel-deb-armmp,no-di-kernel,force-mac-address,need-uboot-bootscr

HostGroup_metrocentre arndale
HostProp_metrocentre_Fqdn metrocentre.uk.xensource.com
HostProp_metrocentre_Ether 9e:04:00:59:64:5a

HostGroup_westfield arndale
HostProp_westfield_Fqdn westfield.uk.xensource.com
HostProp_westfield_Ether a6:46:13:77:e8:2f

HostGroup_lakeside arndale
HostProp_lakeside_Fqdn lakeside.uk.xensource.com
HostProp_lakeside_Ether e2:e7:75:3f:df:4c

HostGroup_bluewater arndale
HostProp_bluewater_Fqdn bluewater.uk.xensource.com
HostProp_bluewater_Ether f2:dc:22:d7:e9:e9

HostGroupProp_cubietruck_LinuxSerialConsole ttyS0
HostGroupProp_cubietruck_Build_Make_Flags -j4
HostGroupProp_cubietruck_XenSerialConsole dtuart
HostGroupProp_cubietruck_XenDTUARTPath /soc@01c00000/serial@01c28000 
HostGroupProp_cubietruck_UBootScriptEarlyCommands setenv xen_addr_r 0x41000000
HostGroupFlags_cubietruck suite-wheezy,equiv-cubietruck,need-kernel-deb-armmp,no-di-kernel,need-uboot-bootscr

HostGroup_braque cubietruck
HostProp_braque_Fqdn braque.uk.xensource.com

HostGroup_picaso cubietruck
HostProp_picaso_Fqdn picaso.uk.xensource.com

HostGroup_metzinger cubietruck
HostProp_metzinger metzinger.uk.xensource.com

HostGroup_gleizes cubietruck
HostProp_gleizes_Fqdn gleizes.uk.xensource.com

[PATCH OSSTEST v2 01/20] standalone: Provide a helper to clear the host runvar

[Ian Campbell]

This clears the host flag used for single-host test cases.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 standalone | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/standalone b/standalone
index daab7ae..c1139f8 100755
--- a/standalone
+++ b/standalone
@@ -29,6 +29,12 @@ Operations:
 
   Required options: -h HOST
 
+* reset-host [cf] [JOB]
+
+  Allow a job which has been run before to run on a different host
+  next time. Otherwise osstest will complain if you change the host
+  which a job is running on on successive runs.
+
 Options:
 
 -c FILE, --config=FILE        Use FILE as configuration file
@@ -200,6 +206,18 @@ case $op in
 	fi
 	;;
 
+    reset-host)
+	need_flight;
+
+	if [ $# -lt 1 ] ; then
+	    echo "run-job: Need job" >&2
+	    exit 1
+	fi
+
+	job=$1; shift
+	./cs-adjust-flight -v $flight runvar-del $job host
+	;;
+
     run-job)
 	need_flight; need_host
 
-- 
2.1.1

[PATCH OSSTEST v2 02/20] mg-debian-installer-update: Include any device tree blobs in the tftp dir

[Ian Campbell]

We will need to supply these for some arm boards.

We provide both an unpacked version, which can be referenced via u-boot's
support for the "fdtdir" property in pxelinux.cfg and a tarball which is more
convenient for installing into the target filesystem on host install

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-debian-installer-update | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index b964f95..4e1da8a 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -130,6 +130,10 @@ if [ $arch = armhf ]; then
           |pax -x sv4cpio -s '%lib%/lib%' -d -w >../cpio; cd ..
     gzip -9f cpio
     mv cpio.gz armmp.cpio.gz
+    rm -rf dtbs/
+    mkdir dtbs/
+    mv x/usr/lib/linux-image-*-armmp/*.dtb dtbs/
+    tar -caf dtbs.tar.gz dtbs
     rm -rf x
 fi
 
-- 
2.1.1

[PATCH OSSTEST v2 03/20] mg-debian-installer-update: Add more modules when creating initrd overlay

[Ian Campbell]

The arndale platform has a usb network device. In addition both the SATA and
NIC/USB hardware on this platform requires additional regulator (power), clk
and phy modules, which in turn require i2c support.

Add everything which is needed on this platform.

Sort the list of drivers while here.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-debian-installer-update | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index 4e1da8a..4af4a42 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -123,10 +123,18 @@ if [ $arch = armhf ]; then
                          -path \*/kernel/fs/mbcache.ko -o \
                          -path \*/kernel/fs/ext\* -o \
                          -path \*/kernel/fs/jbd\* -o \
-                         -path \*/kernel/drivers/net/\* -o \
+                         -path \*/kernel/drivers/clk/\* -o \
+                         -path \*/kernel/drivers/mmc/\* -o \
+                         -path \*/kernel/drivers/phy/\* -o \
+                         -path \*/kernel/drivers/usb/misc/\* -o \
                          -path \*/kernel/drivers/ata/\* -o \
+                         -path \*/kernel/drivers/i2c/busses/\* -o \
+                         -path \*/kernel/drivers/md/\* -o \
+                         -path \*/kernel/drivers/net/\* -o \
+                         -path \*/kernel/drivers/regulator/\* -o \
                          -path \*/kernel/drivers/scsi/\* -o \
-                         -path \*/kernel/drivers/md/\* \) \
+                         -path \*/kernel/drivers/usb/dwc3/\* -o \
+                         -path \*/kernel/drivers/usb/host/\* \) \
           |pax -x sv4cpio -s '%lib%/lib%' -d -w >../cpio; cd ..
     gzip -9f cpio
     mv cpio.gz armmp.cpio.gz
-- 
2.1.1

[PATCH OSSTEST v2 04/20] ts-host-install: Add fdtdir to PXE stanza if dtbs directory exists

[Ian Campbell]

u-boot's pxe client uses this together with $fdtfile in its local environment
to automatically load the fdt when pxe booting.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Place if on the next line
---
 ts-host-install | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ts-host-install b/ts-host-install
index 95ce845..90058e3 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -188,6 +188,9 @@ SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="$ho->{Ether}", A
 END
     }
 
+    my $dtbs = "fdtdir /$d_i/dtbs"
+	if -e "$ho->{Tftp}{Path}/$d_i/dtbs";
+
     file_simple_write_contents("$initrd_overlay.cpio", sub {
         contents_make_cpio($_[0], 'newc', "$initrd_overlay.d");
     });
@@ -222,6 +225,7 @@ label overwrite
 	kernel $kernel
 	append $installcmdline
 	ipappend $ipappend
+	$dtbs
 default overwrite
 END
 }
-- 
2.1.1

[PATCH OSSTEST v2 05/20] ts-host-install: Refactor code to construct a Linux console= for a host

[Ian Campbell]

And reuse it in when writing the u-boot script for initial boot. This was
previously hardcoded to ttyAMA which matches the production host DB for the
current set of hosts which use this script.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm      |  4 +++-
 Osstest/TestSupport.pm | 10 ++++++++++
 ts-host-install        |  6 +++---
 3 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index ab09abb..29dec3b 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -660,6 +660,8 @@ END
 
     if ( $ho->{Flags}{'need-uboot-bootscr'} ) {
 	my $root=target_guest_lv_name($ho,"root");
+	my $console = get_host_native_linux_console($ho);
+	my $consolecmd = "console=$console" unless $console eq "NONE";
 
 	preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
@@ -671,7 +673,7 @@ kernel=`readlink \$r/vmlinuz | sed -e 's|boot/||'`
 initrd=`readlink \$r/initrd.img | sed -e 's|boot/||'`
 
 cat >\$r/boot/boot <<EOF
-setenv bootargs console=ttyAMA0 root=$root
+setenv bootargs $consolecmd root=$root
 mw.l 800000 0 10000
 scsi scan
 ext2load scsi 0 \\\${kernel_addr_r} \$kernel
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 1d77933..d66708e 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -65,6 +65,7 @@ BEGIN {
                       contents_make_cpio file_simple_write_contents
 
                       selecthost get_hostflags get_host_property
+                      get_host_native_linux_console
                       power_state power_cycle power_cycle_time
                       serial_fetch_logs
                       propname_massage
@@ -863,6 +864,15 @@ sub get_host_property ($$;$) {
     return defined($val) ? $val : $defval;
 }
 
+sub get_host_native_linux_console ($) {
+    my ($ho) = @_;
+
+    my $console = get_host_property($ho, "LinuxSerialConsole", "ttyS0");
+    return $console if $console eq 'NONE';
+
+    return "$console,$c{Baud}n8";
+}
+
 sub get_host_method_object ($$$) {
     my ($ho, $kind, $meth) = @_;
     my (@meth) = split /\s+/, $meth;
diff --git a/ts-host-install b/ts-host-install
index 90058e3..881c2fd 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -210,9 +210,9 @@ END
 
     push @installcmdline, qw(--);
 
-    my $console = get_host_property($ho, "LinuxSerialConsole", "ttyS0");
-    push @installcmdline, "console=$console,$c{Baud}n8"
-        unless $console eq "NONE";
+    my $console = get_host_native_linux_console($ho);
+
+    push @installcmdline, "console=$console" unless $console eq "NONE";
 
     my $installcmdline= join ' ', @installcmdline;
 
-- 
2.1.1

[PATCH OSSTEST v2 06/20] ts-host-install: include console before *and* after the -- marker.

[Ian Campbell]

As of v3.15 Linux now stop parsing its command line at the -- mark and passes
the remainder to init. This has broken Debian Installer's feature where
anything after the -- is propagated to the installed system.

See also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007

To work around this we need to pass console= twice, once before the -- for the
current kernel's use and then again after the -- which is propagated to the
installed system.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-host-install | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/ts-host-install b/ts-host-install
index 881c2fd..c611ed1 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -208,12 +208,16 @@ END
     push @installcmdline,
         get_host_property($ho, "install-append $ho->{Suite}", '');
 
-    push @installcmdline, qw(--);
-
     my $console = get_host_native_linux_console($ho);
 
     push @installcmdline, "console=$console" unless $console eq "NONE";
 
+    push @installcmdline, qw(--);
+
+    # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762007 for
+    # why this is repeated.
+    push @installcmdline, "console=$console" unless $console eq "NONE";
+
     my $installcmdline= join ' ', @installcmdline;
 
     setup_pxeboot($ho, <<END);
-- 
2.1.1

[PATCH OSSTEST v2 07/20] ts-host-install: Add option to boot debian-installer in rescue mode

[Ian Campbell]

This is useful when trying to bootstrap onto a new platform, since you can log
in to an otherwise unbootable rootfs to figure out what is going wrong.

The procedure is the same as installing except "rescue/enabled=true" is added
to the command line, hence part of ts-host-install rather than separate.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 1 +
 ts-host-install   | 9 +++++++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 29dec3b..8f80eb4 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -422,6 +422,7 @@ sub di_installcmdline_core ($$;@) {
     my $debconf_priority= $xopts{DebconfPriority};
     push @cl, "debconf/priority=$debconf_priority"
         if defined $debconf_priority;
+    push @cl, "rescue/enable=true" if $xopts{RescueMode};
 
     return @cl;
 }
diff --git a/ts-host-install b/ts-host-install
index c611ed1..2f72b5f 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -30,9 +30,14 @@ my $debconf_priority;
 
 our %xopts;
 
-if (@ARGV && $ARGV[0] =~ m/^--priority(?:=(.*))?$/) {
+while (@ARGV and $ARGV[0] =~ m/^-/) {
+    $_= shift @ARGV;
+    last if m/^--$/;
+    if (m/^--priority(?:=(.*))?$/) {
     $xopts{DebconfPriority}= defined($1) ? $1 : 'low';
-    shift @ARGV;
+    } elsif  (m/^--rescue$/) {
+	$xopts{RescueMode}= 1;
+    }
 }
 
 our ($whhost) = @ARGV;
-- 
2.1.1

[PATCH OSSTEST v2 08/20] ts-host-install: Add force-mac-address host flag

[Ian Campbell]

This causes us to emit udev runes which force the device named in the host's
Interface_Force property to have the MAC address given in the host's Ether
property. This is the reverse of the behaviour of the Interface_Force property
without this flag, which is to rename the device with MAC address Ether to have
the requested name.

This is necessary because the arndale's USB Ethernet device has no burned in
MAC address and there is no mechanism to configure this e.g. via the bootloader
in a way which Linux will then notice.

Without this change Linux will pick a random mac address when loading the
driver, which breaks osstest's attempts to observe when the test host has
picked up its preseed file by watching the webserver logs, since DHCP will not
map a random mac address to the IP we are expecting.

This is only useful on systems which have exactly one Ethernet device,
otherwise there is no reliable way to determine which device is which.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2; Added final para to commit log.
---
 ts-host-install | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/ts-host-install b/ts-host-install
index 2f72b5f..b954e53 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -181,6 +181,7 @@ sub setup_pxeboot_firstboot($) {
 
     my $ipappend = 2;
     my $wantphysif= get_host_property($ho,'interface force','auto');
+    logm("Forcing interface $wantphysif");
     if ($wantphysif ne 'auto') {
 	$ipappend = 0;
 	die "need Ether for $ho->{Name} ($wantphysif)"
@@ -188,7 +189,9 @@ sub setup_pxeboot_firstboot($) {
         system_checked(qw(mkdir -p --), "$initrd_overlay.d/etc/udev/rules.d");
         file_simple_write_contents
             ("$initrd_overlay.d/etc/udev/rules.d/70-persistent-net.rules",
-             <<END);
+             $ho->{Flags}{'force-mac-address'} ? <<END : <<END);
+SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNEL=="$wantphysif", RUN += "/bin/ip link set $wantphysif address $ho->{Ether}"
+END
 SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="$ho->{Ether}", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="$wantphysif"
 END
     }
-- 
2.1.1

[PATCH OSSTEST v2 09/20] make-flight: Run a basic test on each arm platform

[Ian Campbell]

Unlike x86 there is enough variation in the ARM platforms that it is worth
having a basic test on each as part of a standard run. This relies on each host
having an appropriate platform-$platform host flag.

The existing test-ARCH-ARCH-xl test is retained as a floating test, while a new
variant is added for each distinct platform present in the hostdb which is tied
to that platform type. The intention is that only arm platforms will have
platforms at first, although perhaps platform-intel and platform-amd could be
added in the future too.

There are currently no platform-* flags in the hostdb, so tested with
s/platform-/equiv-/ and:

  ( set -ex ;
    source ./cri-getplatforms ;
    blessing=real ;
    export OSSTEST_CONFIG=production-config ;
    for p in '' `getplatforms "armhf"` ; do
      set +x ;
      echo PLATFORM: $p ;
    done
  )
which prints:
  PLATFORM:
  PLATFORM: marilith
and with s/armhf/amd64/:
  PLATFORM:
  PLATFORM: rackservers-s40670
  PLATFORM: r310-moth
  PLATFORM: rackservers-s40680
  PLATFORM: dell-r310
  PLATFORM: rackservers-s40679
  PLATFORM: rackservers-s40663
  PLATFORM: rackservers-q21011

Also tested in standalone mode with a ~/.xen-osstest/config containing:

  PlatformsArmhf midway cubietruck arndale

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2:
    Use platform-* prop not equiv-*
    Select platforms from host db
---
 Osstest/HostDB/Executive.pm | 28 ++++++++++++++++++++++++++++
 Osstest/HostDB/Static.pm    |  9 +++++++++
 README                      |  3 +++
 cri-getplatforms            | 25 +++++++++++++++++++++++++
 make-flight                 |  9 +++++++--
 5 files changed, 72 insertions(+), 2 deletions(-)
 create mode 100755 cri-getplatforms

diff --git a/Osstest/HostDB/Executive.pm b/Osstest/HostDB/Executive.pm
index b2c8dc9..6257829 100644
--- a/Osstest/HostDB/Executive.pm
+++ b/Osstest/HostDB/Executive.pm
@@ -67,6 +67,34 @@ END
     return $flags;
 }
 
+sub get_arch_platforms ($$) {
+    my ($hd, $blessing, $arch) = @_;
+
+    my @plats = ( );
+    my $platsq = $dbh_tests->prepare(<<END);
+SELECT DISTINCT hostflag
+           FROM hostflags h0
+   WHERE EXISTS (
+       SELECT *
+         FROM hostflags h1, hostflags h2
+        WHERE h0.hostname = h1.hostname AND h1.hostname = h2.hostname
+          AND h1.hostflag = ?
+          AND h2.hostflag = ?
+   )
+   AND hostflag like 'platform-%';
+END
+
+    $platsq->execute("blessed-$blessing", "arch-$arch");
+
+    while (my ($plat) = $platsq->fetchrow_array()) {
+	$plat =~ s/^platform-//g or die;
+	push @plats, $plat;
+    }
+
+    $platsq->finish();
+    return @plats;
+}
+
 sub default_methods ($$) {
     my ($hd, $ho) = @_;
 
diff --git a/Osstest/HostDB/Static.pm b/Osstest/HostDB/Static.pm
index d0c13a1..e786761 100644
--- a/Osstest/HostDB/Static.pm
+++ b/Osstest/HostDB/Static.pm
@@ -63,6 +63,15 @@ sub get_flags ($$) { #method
     return $flags;
 }
 
+sub get_arch_platforms ($$) {
+    my ($hd, $blessing, $arch) = @_;
+
+    my $prop = "Platforms".ucfirst($arch);
+
+    return split /[, ]/, $c{$prop} if $c{$prop};
+    return () unless $c{$prop};
+}
+
 sub default_methods ($$) { #method
     my ($hd, $ho) = @_;
 
diff --git a/README b/README
index 9a85549..5c5f347 100644
--- a/README
+++ b/README
@@ -381,6 +381,9 @@ The keys in ~/.ssh/id_{rsa,dsa}.pub and ~/.ssh/authorized_keys
 
 TestHostKeypairPath
 
+Platforms<Arch>
+   List of platforms (i.e. distinct host types) to run a basic test on.
+
 HostProp_GenEtherPrefixBase 5e:36:0e:f5
 # 	                               :00:01 guest number in job appended
 #            in standalone jobdb, ^^^^^ xor'd with low 16 bits of your uid
diff --git a/cri-getplatforms b/cri-getplatforms
new file mode 100755
index 0000000..067730c
--- /dev/null
+++ b/cri-getplatforms
@@ -0,0 +1,25 @@
+# -*- bash -*-
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2014 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+getplatforms () {
+        perl -e '
+                use Osstest;
+                csreadconfig();
+                print join " ", $mhostdb->get_arch_platforms("'$blessing'", "'$1'") or die $!;
+        '
+}
diff --git a/make-flight b/make-flight
index 9963a46..0814eba 100755
--- a/make-flight
+++ b/make-flight
@@ -27,6 +27,7 @@ buildflight=$4
 flight=`./cs-flight-create $blessing $branch`
 
 . cri-common
+. cri-getplatforms
 . ap-common
 . mfi-common
 
@@ -284,10 +285,14 @@ do_passthrough_tests () {
 test_matrix_do_one () {
 
   # Basic PV Linux test with xl
+  for platform in '' `getplatforms $xenarch` ; do
+    suffix=${platform:+-$platform}
+    hostflags=${most_hostflags}${platform:+,platform-$platform}
 
-  job_create_test test-$xenarch$kern-$dom0arch-xl test-debian xl \
+    job_create_test test-$xenarch$kern-$dom0arch-xl$suffix test-debian xl \
             $xenarch $dom0arch                                   \
-            $debian_runvars all_hostflags=$most_hostflags
+            $debian_runvars all_hostflags=$hostflags
+  done
 
   job_create_test test-$xenarch$kern-$dom0arch-libvirt test-debian libvirt \
             $xenarch $dom0arch                                       \
-- 
2.1.1

[PATCH OSSTEST v2 10/20] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG

[Ian Campbell]

This saves repeating identical HostProp and HostFlags for sets of identical
machines. e.g.

    HostGroupProp_cubietruck_LinuxSerialConsole ttyS0
    HostGroupProp_cubietruck_Build_Make_Flags -j12
    HostGroupProp_cubietruck_XenSerialConsole dtuart
    HostGroupProp_cubietruck_XenDTUARTPath /soc@01c00000/serial@01c28000
    HostGroupFlags_cubietruck suite-wheezy,equiv-cubietruck,need-kernel-deb-armmp,no-di-kernel,need-uboot-bootscr

    HostGroup_braque cubietruck
    HostProp_braque_Fqdn braque.uk.xensource.com

    HostGroup_picaso cubietruck
    HostProp_picaso_Fqdn picaso.uk.xensource.com

    HostGroup_metzinger cubietruck
    HostProp_metzinger metzinger.uk.xensource.com

    HostGroup_gleizes cubietruck
    HostProp_gleizes_Fqdn gleizes.uk.xensource.com

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Set HostGroup props after DB
    Clarify docs
---
 Osstest/HostDB/Static.pm |  2 ++
 Osstest/TestSupport.pm   | 19 ++++++++++++++++++-
 README                   | 22 ++++++++++++++++++++++
 3 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/Osstest/HostDB/Static.pm b/Osstest/HostDB/Static.pm
index e786761..076da62 100644
--- a/Osstest/HostDB/Static.pm
+++ b/Osstest/HostDB/Static.pm
@@ -58,6 +58,8 @@ sub get_flags ($$) { #method
     };
 
     $process->('HostFlags');
+    $process->("HostGroupFlags_$ho->{Properties}{HostGroup}")
+	if $ho->{Properties}{HostGroup};
     $process->("HostFlags_$ho->{Name}");
 
     return $flags;
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index d66708e..a1c2947 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -766,7 +766,15 @@ sub selecthost ($) {
 	$ho->{Properties}{$pn} = $val;
     };
 
-    # First, we use the config file's general properites as defaults
+    # First, set the prop group if any.
+    foreach my $k (keys %c) {
+	next unless $k =~ m/^HostGroup_([-a-z0-9]+)$/;
+	next unless $1 eq $name;
+	$setprop->("HostGroup", $c{$k});
+	logm("Host $name is in HostGroup $ho->{Properties}{HostGroup}");
+    }
+
+    # Next, we use the config file's general properites as defaults
     foreach my $k (keys %c) {
 	next unless $k =~ m/^HostProp_([A-Z].*)$/;
 	$setprop->($1, $c{$k});
@@ -775,6 +783,15 @@ sub selecthost ($) {
     # Then we read in the HostDB's properties
     $mhostdb->get_properties($name, $ho->{Properties});
 
+    # Next, we set any HostGroup based properties
+    if ( $ho->{Properties}{HostGroup} ) {
+	foreach my $k (keys %c) {
+	    next unless $k =~ m/^HostGroupProp_([-a-z0-9]+)_(.*)$/;
+	    next unless $1 eq $ho->{Properties}{HostGroup};
+	    $setprop->($2, $c{$k});
+	}
+    }
+
     # Finally, we override any host-specific properties from the config
     foreach my $k (keys %c) {
 	next unless $k =~ m/^HostProp_([-a-z0-9]+)_(.*)$/;
diff --git a/README b/README
index 5c5f347..5ba695d 100644
--- a/README
+++ b/README
@@ -333,6 +333,28 @@ HostProp_<testbox>_TftpScope
    Defines the Tftp scope (i.e. subnet) where this host resides. See
    "TftpFoo_<scope> and TftpFoo" below.
 
+HostFlags_<testbox>
+   Defines a set of flags for the host. Flags is a list separated by
+   whitespace, comma or semi-colon. A flag can be unset by prepending
+   a !. Only used in standalone mode.
+
+HostGroup_<testbox> <group>
+   Defines a group of similar hosts of which <testbox> is a
+   member. This can then be used with HostGroupProp and HostGroupFlags
+
+HostGroupProps_<group>_<prop>
+   Equivalent to writing HostProp_<testbox>_<prop> for every testbox
+   which declares HostGroup_<testbox>_<group>. Allows setting a set of
+   common properties for a group of similar machines. These settings
+   take precedence over the database provided settings, but are
+   themselves overridden by host-specific properties.
+
+HostGroupFlags_<group>
+   Equivalent to writing HostFlags_<testbox> for every testbox which
+   declares HostGroup_<testbox>_<group>. Allows setting a set of
+   common flags for a group of similar machines. These flags are
+   merged with the host specific flags. Only used in standalone mode.
+
 DebianPreseed
    Text to add to the debian-installer preseed file.  Optional
    but you will need to set some NTP servers here if your firewall
-- 
2.1.1

[PATCH OSSTEST v2 11/20] Osstest/PDU: Add eth008.pm method to control the ARM rack PDU

[Ian Campbell]

This controls the eth008 relay board: http://www.robot-electronics.co.uk/htm/eth008tech.htm

Due to the use of the CGI interface this requires firmware version 4+.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Pass username and password via a netrc file.
---
 Osstest/PDU/eth008.pm | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
 create mode 100644 Osstest/PDU/eth008.pm

diff --git a/Osstest/PDU/eth008.pm b/Osstest/PDU/eth008.pm
new file mode 100644
index 0000000..f56cfa0
--- /dev/null
+++ b/Osstest/PDU/eth008.pm
@@ -0,0 +1,73 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2014 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package Osstest::PDU::eth008;
+
+use strict;
+use warnings;
+
+use Osstest;
+use Osstest::TestSupport;
+use IO::File;
+use File::Temp qw/ tempfile /;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho, $methname, $pdu, $user, $pass, $port, @opts) = @_;
+    return bless { Host => $ho,
+		   PDU => $pdu,
+		   User => $user,
+		   Pass => $pass,
+		   Port => $port,
+		   Opts => \@opts }, $class;
+}
+
+sub pdu_power_state {
+    my ($mo, $on) = @_;
+    my $op= $on ? "DOA" : "DOI"; # Digital Output (In)Active
+
+    # Use the CGI interface since it is less prone to being firewalled
+    # off, unlike the standard interface on port 17494. This is only
+    # available from firmware v4 onwards.
+
+    # Create a netrc file to avoid putting l/p on command line
+    my ($passfh, $passfilename) = tempfile( UNLINK => 1 );
+    print $passfh "machine $mo->{PDU} login $mo->{User} password $mo->{Pass}\n";
+
+    my $url = "http://$mo->{PDU}/io.cgi?$op$mo->{Port}=0";
+    my $cmd = "curl --silent --netrc --netrc-file $passfilename $url";
+
+    logm("$cmd");
+    open CURL, "$cmd |" or die "fork curl: $!";
+    
+    my $result = <CURL>;
+    close CURL or die "result curl: $!";
+
+    logm("$result");
+    die "curl failed" unless $result =~ /^Success! \d+/;
+}
+
+1;
-- 
2.1.1

[PATCH OSSTEST v2 12/20] Osstest/Debian: Refactor code to set bootargs in u-boot script

[Ian Campbell]

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 8f80eb4..3ffbc50 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -660,9 +660,15 @@ END
     }
 
     if ( $ho->{Flags}{'need-uboot-bootscr'} ) {
+	my @bootargs;
+
 	my $root=target_guest_lv_name($ho,"root");
-	my $console = get_host_native_linux_console($ho);
-	my $consolecmd = "console=$console" unless $console eq "NONE";
+	my $console=get_host_native_linux_console($ho);
+
+	push @bootargs, "root=$root";
+	push @bootargs, "console=$console" unless $console eq "NONE";
+
+	my $bootargs = join ' ', @bootargs;
 
 	preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
@@ -674,7 +680,7 @@ kernel=`readlink \$r/vmlinuz | sed -e 's|boot/||'`
 initrd=`readlink \$r/initrd.img | sed -e 's|boot/||'`
 
 cat >\$r/boot/boot <<EOF
-setenv bootargs $consolecmd root=$root
+setenv bootargs $bootargs
 mw.l 800000 0 10000
 scsi scan
 ext2load scsi 0 \\\${kernel_addr_r} \$kernel
-- 
2.1.1

[PATCH OSSTEST v2 13/20] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs

[Ian Campbell]

This is done whenever dtbs.tar.gz exists. mg-debian-installer-update produces
the necessary inputs on the relevant platform (armhf).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Install dtbs iff dtbs.tar.gz exists.
---
 Osstest/Debian.pm | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 3ffbc50..6bb14f7 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -514,6 +514,9 @@ sub preseed_create ($$;@) {
     my $disk= $xopts{DiskDevice} || '/dev/sda';
     my $suite= $xopts{Suite} || $c{DebianSuite};
 
+    my $d_i= $ho->{Tftp}{Path}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.
+	$c{TftpDiVersion}.'-'.$ho->{Suite};
+
     my $hostsq= $dbh_tests->prepare(<<END);
         SELECT val FROM runvars
          WHERE flight=? AND name LIKE '%host'
@@ -627,12 +630,26 @@ $overlays
 echo latecmd done.
 END
 
+    if ( -e "$d_i/dtbs.tar.gz" ) {
+	my $durl = create_webfile($ho, "dtbs", sub {
+	    copy("$d_i/dtbs.tar.gz", $_[0])
+		or die "Copy dtbs failed: $!";
+	});
+	preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+r=/target
+
+wget -O \$r/tmp/dtbs.tar.gz $durl
+
+in-target tar -C /boot -xaf /tmp/dtbs.tar.gz
+END
+    }
+
     foreach my $kp (keys %{ $ho->{Flags} }) {
 	$kp =~ s/need-kernel-deb-// or next;
 
-	my $d_i= $ho->{Tftp}{Path}.'/'.$ho->{Tftp}{DiBase}.'/'.$r{arch}.'/'.
-	    $c{TftpDiVersion}.'-'.$ho->{Suite};
-
 	my $kurl = create_webfile($ho, "kernel", sub {
 	    copy("$d_i/$kp.deb", $_[0])
 		or die "Copy kernel failed: $!";
-- 
2.1.1

[PATCH OSSTEST v2 14/20] Osstest/Debian: Support for loading an FDT from u-boot script

[Ian Campbell]

The currently supported platform provides an FDT preloaded at 0x1000. Replace
this with ${fdt_addr} (which the current platform exposes) and for platforms
which do not provide an fdt arrange to load the relevant file as named in the
${fdtfile} (which is conventionally provided by u-boot for platforms which need
this).

Drop some random memory clearing rune, I've no idea what the intended purpose
was, 0x800000 doesn't correspond to any $foo_addr_r on the midway systems for
example.

Also get rid of the scsi scan which must necessarily have already happened
(since the boot.scr itselfs lives on a scsi drive).

Lastly, add some echos to show progress through the script, as an aid to
debugging.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Refactor uboot dtb loading scriptlet, which previously appeared twice
---
 Osstest/Debian.pm | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 6bb14f7..e390ec9 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -112,6 +112,15 @@ sub bl_getmenu_open ($$$) {
     return $f;
 }
 
+sub uboot_scr_load_dtb () {
+    return <<'END';
+if test -z "\${fdt_addr}" && test -n "\${fdtfile}" ; then
+    echo Loading dtbs/\${fdtfile}
+    ext2load scsi 0 \${fdt_addr_r} dtbs/\${fdtfile}
+    setenv fdt_addr \${fdt_addr_r}
+fi
+END
+}
 sub setupboot_uboot ($$$) {
     my ($ho,$want_kernver,$xenhopt,$xenkopt) = @_;
     my $bl= { };
@@ -130,6 +139,8 @@ sub setupboot_uboot ($$$) {
 
 	my $early_commands = get_host_property($ho, 'UBootScriptEarlyCommands', '');
 
+	my $load_dtb = uboot_scr_load_dtb();
+
 	target_cmd_root($ho, <<END);
 if test ! -f /boot/$kern ; then
     exit 1
@@ -141,9 +152,7 @@ cp -n /boot/boot.scr /boot/boot.scr.bak
 xen=`readlink /boot/$xen`
 
 cat >/boot/boot <<EOF
-
-mw.l 800000 0 10000
-scsi scan
+${load_dtb}
 
 fdt addr \\\${fdt_addr}
 fdt resize
@@ -687,6 +696,8 @@ END
 
 	my $bootargs = join ' ', @bootargs;
 
+	my $load_dtb = uboot_scr_load_dtb();
+
 	preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
 set -ex
@@ -698,11 +709,13 @@ initrd=`readlink \$r/initrd.img | sed -e 's|boot/||'`
 
 cat >\$r/boot/boot <<EOF
 setenv bootargs $bootargs
-mw.l 800000 0 10000
-scsi scan
+${load_dtb}
+echo Loading \$kernel
 ext2load scsi 0 \\\${kernel_addr_r} \$kernel
+echo Loading \$initrd
 ext2load scsi 0 \\\${ramdisk_addr_r} \$initrd
-bootz \\\${kernel_addr_r} \\\${ramdisk_addr_r}:\\\${filesize} 0x1000
+echo Booting
+bootz \\\${kernel_addr_r} \\\${ramdisk_addr_r}:\\\${filesize} \\\${fdt_addr}
 EOF
 
 in-target mkimage -A arm -T script -d /boot/boot /boot/boot.scr
-- 
2.1.1

[PATCH OSSTEST v2 15/20] Osstest/Debian: Add support for "ExtraInitramfsModules" host property

[Ian Campbell]

The arndale platform needs a bunch of clk, phy and regulator stuff in order to
access its root filesystem. However mkinitramfs is not (currently?) able to
figure this out and therefore doesn't include them in the initrd. See
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762042

Add a new host prop which can list these required additional module and
arranges for a suitable /etc/initramfs-tools/modules to be created on install.

Using the new HostGroupProp syntax the required modules are:

HostGroupProp_arndale_ExtraInitramfsModules clk-s2mps11 s5m8767 i2c-s3c2410 phy-exynos5250-sata

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: References to bugs.
---
 Osstest/Debian.pm | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index e390ec9..5435bdf 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -722,6 +722,23 @@ in-target mkimage -A arm -T script -d /boot/boot /boot/boot.scr
 END
     }
 
+    my $modules = get_host_property($ho, "ExtraInitramfsModules", "NONE");
+    if ( $modules ne "NONE" )
+    {
+	# This is currently the best available way to add modules to
+	# the installed initramfs. See
+	# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=764805
+        preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+for i in $modules ; do
+    echo \$i >> /target/etc/initramfs-tools/modules
+done
+in-target update-initramfs -u -k all
+END
+    }
+
     my @extra_packages = ();
     push(@extra_packages, "u-boot-tools") if $ho->{Flags}{'need-uboot-bootscr'};
 
-- 
2.1.1

[PATCH OSSTEST v2 16/20] Osstest/Debian: support adding a rootdelay property to bootargs

[Ian Campbell]

arndale appears to be quite slow (and asynchronous) at finding it's scsi
controller. rootdelay=3 seems to do the trick.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 5435bdf..8a4ad02 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -689,9 +689,11 @@ END
 	my @bootargs;
 
 	my $root=target_guest_lv_name($ho,"root");
+	my $rootdelay=get_host_property($ho, "rootdelay");
 	my $console=get_host_native_linux_console($ho);
 
 	push @bootargs, "root=$root";
+	push @bootargs, "rootdelay=$rootdelay";
 	push @bootargs, "console=$console" unless $console eq "NONE";
 
 	my $bootargs = join ' ', @bootargs;
-- 
2.1.1

[PATCH OSSTEST v2 17/20] Osstest/Debian: Remove hardcoded midway specific addresses from boot.scr

[Ian Campbell]

This depends on a change to the hostdb to add "setenv xen_addr_r 0x01000000" to
the UBootScriptEarlyCommands property of the midway machines.

Most platforms will need something similar. For both cubietruck and arndale
"setenv xen_addr_r 0x41000000" is appropriate.

This is going to be a pretty common requirement on ARM/uboot systems, should I
make it a specific property instead of reusing the generic hook?

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: s/go along with/depends on/
---
 Osstest/Debian.pm | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 8a4ad02..0e808a0 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -162,10 +162,6 @@ ${early_commands}
 fdt set /chosen \\\#address-cells <1>
 fdt set /chosen \\\#size-cells <1>
 
-setenv xen_addr_r 0x01000000
-#   kernel_addr_r=0x02000000
-#  ramdisk_addr_r=0x04000000
-
 ext2load scsi 0 \\\${xen_addr_r} \$xen
 setenv bootargs "$xenhopt"
 echo Loaded \$xen to \\\${xen_addr_r} (\\\${filesize})
-- 
2.1.1

[PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line

[Ian Campbell]

This stops dom0 from messing with clocks which it should and is required on
some platforms. It's harmless even when not needed.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch, previously incorrectly included in "Osstest/Debian: Workaround
    oddities in the u-boot script parser."
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 0e808a0..eb7464c 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -171,7 +171,7 @@ ext2load scsi 0 \\\${kernel_addr_r} $kern
 fdt mknod /chosen module\@0
 fdt set /chosen/module\@0 compatible "xen,linux-zimage" "xen,multiboot-module"
 fdt set /chosen/module\@0 reg <\\\${kernel_addr_r} \\\${filesize}>
-fdt set /chosen/module\@0 bootargs "$xenkopt ro root=$root"
+fdt set /chosen/module\@0 bootargs "$xenkopt ro root=$root clk_ignore_unused"
 echo Loaded $kern to \\\${kernel_addr_r} (\\\${filesize})
 echo command line: $xenkopt ro root=$root
 
-- 
2.1.1

[PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize

[Ian Campbell]

$filesize is an unprefixed hex number, but fdt set requires the 0x to interpret
it properly.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch, previously included in "Osstest/Debian: Workaround oddities in
    the u-boot script parser.". Note that the oddities with requiring a space
    before the ">" is no longer reproducible with the latest u-boot.
---
 Osstest/Debian.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index eb7464c..9d105b1 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -170,7 +170,7 @@ echo command line: \\\${bootargs}
 ext2load scsi 0 \\\${kernel_addr_r} $kern
 fdt mknod /chosen module\@0
 fdt set /chosen/module\@0 compatible "xen,linux-zimage" "xen,multiboot-module"
-fdt set /chosen/module\@0 reg <\\\${kernel_addr_r} \\\${filesize}>
+fdt set /chosen/module\@0 reg <\\\${kernel_addr_r} 0x\\\${filesize}>
 fdt set /chosen/module\@0 bootargs "$xenkopt ro root=$root clk_ignore_unused"
 echo Loaded $kern to \\\${kernel_addr_r} (\\\${filesize})
 echo command line: $xenkopt ro root=$root
@@ -178,7 +178,7 @@ echo command line: $xenkopt ro root=$root
 ext2load scsi 0 \\\${ramdisk_addr_r} $initrd
 fdt mknod /chosen module\@1
 fdt set /chosen/module\@1 compatible "xen,linux-initrd" "xen,multiboot-module"
-fdt set /chosen/module\@1 reg <\\\${ramdisk_addr_r} \\\${filesize}>
+fdt set /chosen/module\@1 reg <\\\${ramdisk_addr_r} 0x\\\${filesize}>
 echo Loaded $initrd to \\\${ramdisk_addr_r} (\\\${filesize})
 
 fdt print /chosen
-- 
2.1.1

[PATCH OSSTEST v2 20/20] ts-kernel-build: Enable CONFIG_PHY_EXYNOS5250_SATA

[Ian Campbell]

Enables the disk on an arndale, which isn't enabled by default in v3.16 for
some reason.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-kernel-build | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/ts-kernel-build b/ts-kernel-build
index 3b48920..bae118c 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -171,6 +171,8 @@ setopt CONFIG_BLK_DEV_NBD y
 # At least with Linux 3.4.77 on wheezy, the nbd module is
 # not loaded automatically.
 
+setopt CONFIG_PHY_EXYNOS5250_SATA y
+
 END
 
 our $config_features= <<END;
-- 
2.1.1

Re: [PATCH OSSTEST v2 09/20] make-flight: Run a basic test on each arm platform

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 09/20] make-flight: Run a basic test on each arm platform"):
> Unlike x86 there is enough variation in the ARM platforms that it is worth
> having a basic test on each as part of a standard run. This relies on each host
> having an appropriate platform-$platform host flag.
> 
> The existing test-ARCH-ARCH-xl test is retained as a floating test, while a new
> variant is added for each distinct platform present in the hostdb which is tied
> to that platform type. The intention is that only arm platforms will have
> platforms at first, although perhaps platform-intel and platform-amd could be
> added in the future too.

The sql looks good.

> +sub get_arch_platforms ($$) {
> +    my ($hd, $blessing, $arch) = @_;
> +
> +    my $prop = "Platforms".ucfirst($arch);
> +
> +    return split /[, ]/, $c{$prop} if $c{$prop};

I think you probably want   split /[, \t]+/   or maybe  split /\s+/

Thanks,
Ian.

Re: [PATCH OSSTEST v2 10/20] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 10/20] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG"):
> This saves repeating identical HostProp and HostFlags for sets of identical
> machines. e.g.

Thanks.

> -    # First, we use the config file's general properites as defaults
> +    # First, set the prop group if any.
> +    foreach my $k (keys %c) {
> +	next unless $k =~ m/^HostGroup_([-a-z0-9]+)$/;
> +	next unless $1 eq $name;

Am I being dim or could this loop be replaced with a single lookup of
$c{"HostGroup_${name}"} ?

Apart from that this looks fine to me.

Thanks,
Ian.

Re: [PATCH OSSTEST v2 11/20] Osstest/PDU: Add eth008.pm method to control the ARM rack PDU

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 11/20] Osstest/PDU: Add eth008.pm method to control the ARM rack PDU"):
> This controls the eth008 relay board: http://www.robot-electronics.co.uk/htm/eth008tech.htm
> 
> Due to the use of the CGI interface this requires firmware version 4+.
...
> +    # Create a netrc file to avoid putting l/p on command line
> +    my ($passfh, $passfilename) = tempfile( UNLINK => 1 );
> +    print $passfh "machine $mo->{PDU} login $mo->{User} password $mo->{Pass}\n";

Where does tempfile(UNLINK=>1) create its files and what deletes it if
the script crashes ?  It might be better to use something in tmp/
(presumably with the host name in the filename).  See for example
ts-host-install which uses tmp/t.<host>.initrd.

Or of course you could use LWP::UserAgent.

Ian.

Re: [PATCH OSSTEST v2 13/20] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 13/20] Osstest/Debian: Install dtbs into target filesystem in /boot/dtbs"):
> This is done whenever dtbs.tar.gz exists. mg-debian-installer-update produces
> the necessary inputs on the relevant platform (armhf).

This patch looks good except for one quibble:

> +    if ( -e "$d_i/dtbs.tar.gz" ) {

If stat fails with EIO or something, this will proceed as if the dtb
doesn't exist.  This is generally a problem with perl's -e things.  I
would use stat and check $! if stat gives false.

See eg the lstat in TestSupport.pm:file_link_contents.  (You probably
want stat rather than lstat.)

Thanks,
Ian.

Re: [PATCH OSSTEST v2 14/20] Osstest/Debian: Support for loading an FDT from u-boot script

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 14/20] Osstest/Debian: Support for loading an FDT from u-boot script"):
> The currently supported platform provides an FDT preloaded at 0x1000. Replace
> this with ${fdt_addr} (which the current platform exposes) and for platforms
> which do not provide an fdt arrange to load the relevant file as named in the
> ${fdtfile} (which is conventionally provided by u-boot for platforms which need
> this).

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST v2 15/20] Osstest/Debian: Add support for "ExtraInitramfsModules" host property

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 15/20] Osstest/Debian: Add support for "ExtraInitramfsModules" host property"):
> The arndale platform needs a bunch of clk, phy and regulator stuff in order to
> access its root filesystem. However mkinitramfs is not (currently?) able to
> figure this out and therefore doesn't include them in the initrd. See
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762042

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST v2 10/20] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG

[Ian Campbell]

On Wed, 2014-10-29 at 16:22 +0000, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 10/20] standalone: Introduce "HostGroups" for use in OSSTEST_CONFIG"):
> > This saves repeating identical HostProp and HostFlags for sets of identical
> > machines. e.g.
> 
> Thanks.
> 
> > -    # First, we use the config file's general properites as defaults
> > +    # First, set the prop group if any.
> > +    foreach my $k (keys %c) {
> > +	next unless $k =~ m/^HostGroup_([-a-z0-9]+)$/;
> > +	next unless $1 eq $name;
> 
> Am I being dim or could this loop be replaced with a single lookup of
> $c{"HostGroup_${name}"} ?

Uh, no, I think it was me who was being dim.

> Apart from that this looks fine to me.

Thanks.

Ian.

Re: [PATCH RFC OSSTEST 17/19] Osstest/Debian: Remove hardcoded midway specific addresses from boot.scr [and 1 more messages]

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 17/20] Osstest/Debian: Remove hardcoded midway specific addresses from boot.scr"):
> This depends on a change to the hostdb to add "setenv xen_addr_r
> 0x01000000" to the UBootScriptEarlyCommands property of the midway
> machines.
> 
> Most platforms will need something similar. For both cubietruck and arndale
> "setenv xen_addr_r 0x41000000" is appropriate.
 > 
> This is going to be a pretty common requirement on ARM/uboot
> systems, should I make it a specific property instead of reusing the
> generic hook?

I don't understand why this question is still in the commit message.
I already answered it with "yes" and you said "ack" ...

Ian.

Re: [PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line"):
> This stops dom0 from messing with clocks which it should and is required on
> some platforms. It's harmless even when not needed.

This is pretty odd.  Doesn't this correspond to some kind of bug, in
the dom0 kernel perhaps ?

Ian.

Re: [PATCH OSSTEST v2 11/20] Osstest/PDU: Add eth008.pm method to control the ARM rack PDU

[Ian Campbell]

On Wed, 2014-10-29 at 16:27 +0000, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 11/20] Osstest/PDU: Add eth008.pm method to control the ARM rack PDU"):
> > This controls the eth008 relay board: http://www.robot-electronics.co.uk/htm/eth008tech.htm
> > 
> > Due to the use of the CGI interface this requires firmware version 4+.
> ...
> > +    # Create a netrc file to avoid putting l/p on command line
> > +    my ($passfh, $passfilename) = tempfile( UNLINK => 1 );
> > +    print $passfh "machine $mo->{PDU} login $mo->{User} password $mo->{Pass}\n";
> 
> Where does tempfile(UNLINK=>1) create its files

I had assumed it was in /tmp (perhaps with handling for $TMP and
synonyms). TFD doesn't really say anything about it though.

> and what deletes it if the script crashes ?

Nothing (it just does it when the thing goes out of scope).

>   It might be better to use something in tmp/
> (presumably with the host name in the filename).  See for example
> ts-host-install which uses tmp/t.<host>.initrd.

Good idea.

> Or of course you could use LWP::UserAgent.

I'll check that out.

Ian/

Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize"):
> $filesize is an unprefixed hex number, but fdt set requires the 0x
> to interpret it properly.

What will happen if they fix u-boot ?

Ian.

Re: [PATCH RFC OSSTEST 17/19] Osstest/Debian: Remove hardcoded midway specific addresses from boot.scr [and 1 more messages]

[Ian Campbell]

On Wed, 2014-10-29 at 16:37 +0000, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 17/20] Osstest/Debian: Remove hardcoded midway specific addresses from boot.scr"):
> > This depends on a change to the hostdb to add "setenv xen_addr_r
> > 0x01000000" to the UBootScriptEarlyCommands property of the midway
> > machines.
> > 
> > Most platforms will need something similar. For both cubietruck and arndale
> > "setenv xen_addr_r 0x41000000" is appropriate.
>  > 
> > This is going to be a pretty common requirement on ARM/uboot
> > systems, should I make it a specific property instead of reusing the
> > generic hook?
> 
> I don't understand why this question is still in the commit message.
> I already answered it with "yes" and you said "ack" ...

Looks like I overlooked that when refreshing my memory of the last time
around, sorry.

Ian.

Re: [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms

[Ian Campbell]

On Wed, 2014-10-29 at 10:41 +0000, Ian Campbell wrote:
> Since v1 I've reordered to pulled the stuff which relates to updating
> mg-debian-installer-update and TftpDiVersion to the front, in the hopes
> of getting that in sooner. This includes everything up to at least
> "ts-host-install: include console before *and* after the -- marker."
> which is needed because rerunning mg-debian-installer-update today will
> pull in a kernel which is affected by that issue. In reality the initial
> set of acked patches here is larger.

I've pushed the first 8 which were acked:
50d9107 standalone: Provide a helper to clear the host runvar
81f6b3b mg-debian-installer-update: Include any device tree blobs in the tftp dir
5c94792 mg-debian-installer-update: Add more modules when creating initrd overlay
54fdfe0 ts-host-install: Add fdtdir to PXE stanza if dtbs directory exists
cd903c2 ts-host-install: Refactor code to construct a Linux console= for a host
db26f58 ts-host-install: include console before *and* after the -- marker.
d11e246 ts-host-install: Add option to boot debian-installer in rescue mode
ff9e0d8 ts-host-install: Add force-mac-address host flag

Once this set has passed osstest's own push gate I intend to followup
with a mass rerun of ./mg-debian-installer-update and an update to
production-config.

Ian.

Re: [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms"):
> Once this set has passed osstest's own push gate I intend to followup
> with a mass rerun of ./mg-debian-installer-update and an update to
> production-config.

Jolly good.  Assuming the config update is what I expect (just a
change to the d-i generation date) you have my (pre-emptive) ack for
it.

Ian.

Re: [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms

[Ian Campbell]

On Wed, 2014-10-29 at 17:13 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms"):
> > Once this set has passed osstest's own push gate I intend to followup
> > with a mass rerun of ./mg-debian-installer-update and an update to
> > production-config.
> 
> Jolly good.  Assuming the config update is what I expect (just a
> change to the d-i generation date) you have my (pre-emptive) ack for
> it.

Yes, it will be exactly that, thanks.

BTW, is there some helper which will call mg-debian-installer-update the
appropriate number of times with the right args for each arch for a
production system update?

AFAIK there is only a helper for standalone mode.

Ian.

Re: [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms"):
> BTW, is there some helper which will call mg-debian-installer-update the
> appropriate number of times with the right args for each arch for a
> production system update?

No.

> AFAIK there is only a helper for standalone mode.

Feel free to write one :-).

Ian.

Re: [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms

[Ian Campbell]

On Thu, 2014-10-30 at 11:30 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms"):
> > BTW, is there some helper which will call mg-debian-installer-update the
> > appropriate number of times with the right args for each arch for a
> > production system update?
> 
> No.
> 
> > AFAIK there is only a helper for standalone mode.
> 
> Feel free to write one :-).

It's a bit noddy, but how about:

----8<----------------

>From 3ec8b49595849d19e4dfd6ed786f540fba9b3ebd Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Thu, 30 Oct 2014 11:39:17 +0000
Subject: [PATCH] Add simple helper to update DI for all architectures.

Uses DebianNonfreeFirmware, even (especially) for production, so move the
README stanza out of standalone only section. The current default matches what
is in the current production versions of DI.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 README                         | 16 ++++++++--------
 mg-debian-installer-update-all | 31 +++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+), 8 deletions(-)
 create mode 100755 mg-debian-installer-update-all

diff --git a/README b/README
index 5ba695d..35532b7 100644
--- a/README
+++ b/README
@@ -360,14 +360,6 @@ DebianPreseed
    but you will need to set some NTP servers here if your firewall
    doesn't permit NTP to Debian's pool.ntp.org servers.
 
-DebianNonfreeFirmware
-  List of debs of non-free firmware to include in the massaged
-  debian-installer.  You will need this if you want to use network
-  card which requires non-free firmware.  The default is just
-  "firmware-bnx2'.  If your host operating system doesn't have
-  grep-dctrl (for example because it's not Debian) then you must set
-  this to the empty string, by writing  DebianNonfreeFirmware=''
-
 ========================================
 
 Config settings relevant only to standalone mode
@@ -423,6 +415,14 @@ GuestDebianSuite   defaults to DebianSuite
 
 DebianPreseed      added to existing preseed file
 
+DebianNonfreeFirmware
+  List of debs of non-free firmware to include in the massaged
+  debian-installer.  You will need this if you want to use network
+  card which requires non-free firmware.  The default is just
+  "firmware-bnx2'.  If your host operating system doesn't have
+  grep-dctrl (for example because it's not Debian) then you must set
+  this to the empty string, by writing  DebianNonfreeFirmware=''
+
 TftpFoo_<scope> and TftpFoo
 
     Describes various properties relating to Tftp in a given <scope>,
diff --git a/mg-debian-installer-update-all b/mg-debian-installer-update-all
new file mode 100755
index 0000000..eca4a5f
--- /dev/null
+++ b/mg-debian-installer-update-all
@@ -0,0 +1,31 @@
+#!/bin/bash
+# usage
+#   ./mg-debian-installer-update-all
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+set -e
+
+. cri-getconfig
+
+suite=`getconfig DebianSuite`
+fws=`getconfig DebianNonfreeFirmware`
+arches="armhf amd64 i386"
+
+for arch in $arches ; do
+    ./mg-debian-installer-update $suite $arch $fws
+done
-- 
2.1.1

Re: [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms"):
> It's a bit noddy, but how about:

LGTM.  Consider it acked and push it whenever convenient.

Ian.

Re: [PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line

[Ian Campbell]

create !
title it arm: domain 0 disables clocks which are in fact being used
thanks

On Wed, 2014-10-29 at 16:39 +0000, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line"):
> > This stops dom0 from messing with clocks which it should and is required on
> > some platforms. It's harmless even when not needed.
> 
> This is pretty odd.  Doesn't this correspond to some kind of bug, in
> the dom0 kernel perhaps ?

dom0 is not aware that some clocks are actually in use (e.g. by the
hypervisor), so the bug is probably in the lack of some interface to
communicate this from Xen to dom0, or some other mechanism to gate this.

I think some or all of the above ought to be added to the commit
message, and perhaps in a comment too. Along with a reference to the bug
which this mail should be about to create. Would you like me to do
anything else?

Ian.

Processed: Re: [PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line

[xen]

Processing commands for xen@bugs.xenproject.org:

> create !
Created new bug #45 rooted at `<1414672390.2064.31.camel@citrix.com>'
Title: `Re: [PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line'
> title it arm: domain 0 disables clocks which are in fact being used
Set title for #45 to `arm: domain 0 disables clocks which are in fact being used'
> thanks
Finished processing.

Modified/created Bugs:
 - 45: http://bugs.xenproject.org/xen/bug/45 (new)

---
Xen Hypervisor Bug Tracker
See http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen for information on reporting bugs
Contact xen-bugs-owner@bugs.xenproject.org with any infrastructure issues

Re: [PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line"):
> create !
> title it arm: domain 0 disables clocks which are in fact being used
> thanks
...
> dom0 is not aware that some clocks are actually in use (e.g. by the
> hypervisor), so the bug is probably in the lack of some interface to
> communicate this from Xen to dom0, or some other mechanism to gate this.

Right.

> I think some or all of the above ought to be added to the commit
> message, and perhaps in a comment too. Along with a reference to the bug
> which this mail should be about to create. Would you like me to do
> anything else?

I think that's enough, thanks.

Ian.

Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize

[Ian Campbell]

On Wed, 2014-10-29 at 16:40 +0000, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize"):
> > $filesize is an unprefixed hex number, but fdt set requires the 0x
> > to interpret it properly.
> 
> What will happen if they fix u-boot ?

I've sent http://lists.denx.de/pipermail/u-boot/2014-October/193622.html

Lets see where they consider the bug to be (if anywhere)

Ian.

Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize

[Ian Campbell]

On Thu, 2014-10-30 at 14:01 +0000, Ian Campbell wrote:
> On Wed, 2014-10-29 at 16:40 +0000, Ian Jackson wrote:
> > Ian Campbell writes ("[PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize"):
> > > $filesize is an unprefixed hex number, but fdt set requires the 0x
> > > to interpret it properly.
> > 
> > What will happen if they fix u-boot ?
> 
> I've sent http://lists.denx.de/pipermail/u-boot/2014-October/193622.html
> 
> Lets see where they consider the bug to be (if anywhere)

In the meantime I've added this URL to the commit message.

Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize"):
> On Wed, 2014-10-29 at 16:40 +0000, Ian Jackson wrote:
> > Ian Campbell writes ("[PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize"):
> > > $filesize is an unprefixed hex number, but fdt set requires the 0x
> > > to interpret it properly.
> > 
> > What will happen if they fix u-boot ?
> 
> I've sent http://lists.denx.de/pipermail/u-boot/2014-October/193622.html
> 
> Lets see where they consider the bug to be (if anywhere)

Right.

I'm a bit worried that if they fix this by making the value contain a
0x at the start, it will break your workaround.

Ian.

Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize

[Ian Campbell]

On Thu, 2014-10-30 at 14:49 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize"):
> > On Wed, 2014-10-29 at 16:40 +0000, Ian Jackson wrote:
> > > Ian Campbell writes ("[PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize"):
> > > > $filesize is an unprefixed hex number, but fdt set requires the 0x
> > > > to interpret it properly.
> > > 
> > > What will happen if they fix u-boot ?
> > 
> > I've sent http://lists.denx.de/pipermail/u-boot/2014-October/193622.html
> > 
> > Lets see where they consider the bug to be (if anywhere)
> 
> Right.
> 
> I'm a bit worried that if they fix this by making the value contain a
> 0x at the start, it will break your workaround.

Right.

Note that updating u-boot on these boards requires someone to actually
go and reflash them, we don't do it as part of each test run for
example.

Ian.

Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize"):
> Note that updating u-boot on these boards requires someone to actually
> go and reflash them, we don't do it as part of each test run for
> example.

Ah, OK, that makes it less of a potential annoyance.

Ian.

Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize

[Ian Campbell]

On Thu, 2014-10-30 at 14:54 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v2 19/20] Osstest/Debian: Add 0x prefix to $filesize"):
> > Note that updating u-boot on these boards requires someone to actually
> > go and reflash them, we don't do it as part of each test run for
> > example.
> 
> Ah, OK, that makes it less of a potential annoyance.

Or at least a controlled annoyance which we'll have brought onto
ourselves at the time of our choosing ;-)

Re: [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms

[Ian Campbell]

On Wed, 2014-10-29 at 17:15 +0000, Ian Campbell wrote:
> On Wed, 2014-10-29 at 17:13 +0000, Ian Jackson wrote:
> > Ian Campbell writes ("Re: [Xen-devel] [PATCH OSSTEST v2 00/20] support for ARM32 arndale and cubietruck platforms"):
> > > Once this set has passed osstest's own push gate I intend to followup
> > > with a mass rerun of ./mg-debian-installer-update and an update to
> > > production-config.
> > 
> > Jolly good.  Assuming the config update is what I expect (just a
> > change to the d-i generation date) you have my (pre-emptive) ack for
> > it.
> 
> Yes, it will be exactly that, thanks.

i.e. the below which I've just now pushed.

> BTW, is there some helper which will call mg-debian-installer-update the
> appropriate number of times with the right args for each arch for a
> production system update?
> 
> AFAIK there is only a helper for standalone mode.

I stuck this helper in there too.

Ian.


commit 4eed3c718ac02dd81dfbee09e8c779fb9f32ec52
Author: Ian Campbell <ian.campbell@citrix.com>
Date:   Mon Nov 3 10:20:55 2014 +0000

    TftpDiVersion 2014-11-03
    
    Picks up changes on armhf to include dtbs and more modules in the initrd
    overlay.
    
    Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
    Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

diff --git a/production-config b/production-config
index 2b3ec10..8b2c3e1 100644
--- a/production-config
+++ b/production-config
@@ -68,7 +68,7 @@ TftpPxeDir /
 TftpPxeTemplates %ipaddrhex%/pxelinux.cfg
 
 TftpPxeGroup osstest
-TftpDiVersion 2014-10-20
+TftpDiVersion 2014-11-03
 
 XenUsePath /usr/groups/xencore/systems/bin/xenuse
 XenUseUser osstest

Re: [PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line

[Julien Grall]

Hi,

Somehow I missed this email.

On 30/10/2014 13:33, Ian Campbell wrote:
> create !
> title it arm: domain 0 disables clocks which are in fact being used
> thanks
>
> On Wed, 2014-10-29 at 16:39 +0000, Ian Jackson wrote:
>> Ian Campbell writes ("[PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line"):
>>> This stops dom0 from messing with clocks which it should and is required on
>>> some platforms. It's harmless even when not needed.
>>
>> This is pretty odd.  Doesn't this correspond to some kind of bug, in
>> the dom0 kernel perhaps ?
>
> dom0 is not aware that some clocks are actually in use (e.g. by the
> hypervisor), so the bug is probably in the lack of some interface to
> communicate this from Xen to dom0, or some other mechanism to gate this.

In reality, Xen is only using the clock of the UART. Even though, I 
think this would also happen with platform device passthrough.

For the latter, it would be fairly easy via a PV drivers. But for 
UART... gating the clock could be a nightmare because every platform 
have their own way to enable/disable the clock. Futhermore, the clock 
could be shared with multiple device...

I'm wondering if we could expose the UART to DOM0 without marking as 
disabled. It would avoid DOM0 to mess up the clock while everything 
would be catch via the vuart implementation in Xen.

Regards,

-- 
Julien Grall

Re: [PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line

[Ian Campbell]

On Tue, 2014-11-11 at 17:50 +0100, Julien Grall wrote:
> Hi,
> 
> Somehow I missed this email.
> 
> On 30/10/2014 13:33, Ian Campbell wrote:
> > create !
> > title it arm: domain 0 disables clocks which are in fact being used
> > thanks
> >
> > On Wed, 2014-10-29 at 16:39 +0000, Ian Jackson wrote:
> >> Ian Campbell writes ("[PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line"):
> >>> This stops dom0 from messing with clocks which it should and is required on
> >>> some platforms. It's harmless even when not needed.
> >>
> >> This is pretty odd.  Doesn't this correspond to some kind of bug, in
> >> the dom0 kernel perhaps ?
> >
> > dom0 is not aware that some clocks are actually in use (e.g. by the
> > hypervisor), so the bug is probably in the lack of some interface to
> > communicate this from Xen to dom0, or some other mechanism to gate this.
> 
> In reality, Xen is only using the clock of the UART. Even though, I 
> think this would also happen with platform device passthrough.
> 
> For the latter, it would be fairly easy via a PV drivers. But for 
> UART... gating the clock could be a nightmare because every platform 
> have their own way to enable/disable the clock. Futhermore, the clock 
> could be shared with multiple device...
> 
> I'm wondering if we could expose the UART to DOM0 without marking as 
> disabled. It would avoid DOM0 to mess up the clock while everything 
> would be catch via the vuart implementation in Xen.

Perhaps we could propagate any clock related properties from the UART
node into the Xen hypervisor node (or a subnode)? The Xen Linux code
would then consume those and mark the relevant clocks as in use. I've
not looked into the clock bindings to know how plausible this might be.

Ian.

Re: [PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line

[Julien Grall]

On 11/12/2014 10:07 AM, Ian Campbell wrote:
> On Tue, 2014-11-11 at 17:50 +0100, Julien Grall wrote:
>> Hi,
>>
>> Somehow I missed this email.
>>
>> On 30/10/2014 13:33, Ian Campbell wrote:
>>> create !
>>> title it arm: domain 0 disables clocks which are in fact being used
>>> thanks
>>>
>>> On Wed, 2014-10-29 at 16:39 +0000, Ian Jackson wrote:
>>>> Ian Campbell writes ("[PATCH OSSTEST v2 18/20] Osstest/Debian: Add "clk_ignore_unused" to default command line"):
>>>>> This stops dom0 from messing with clocks which it should and is required on
>>>>> some platforms. It's harmless even when not needed.
>>>>
>>>> This is pretty odd.  Doesn't this correspond to some kind of bug, in
>>>> the dom0 kernel perhaps ?
>>>
>>> dom0 is not aware that some clocks are actually in use (e.g. by the
>>> hypervisor), so the bug is probably in the lack of some interface to
>>> communicate this from Xen to dom0, or some other mechanism to gate this.
>>
>> In reality, Xen is only using the clock of the UART. Even though, I 
>> think this would also happen with platform device passthrough.
>>
>> For the latter, it would be fairly easy via a PV drivers. But for 
>> UART... gating the clock could be a nightmare because every platform 
>> have their own way to enable/disable the clock. Futhermore, the clock 
>> could be shared with multiple device...
>>
>> I'm wondering if we could expose the UART to DOM0 without marking as 
>> disabled. It would avoid DOM0 to mess up the clock while everything 
>> would be catch via the vuart implementation in Xen.
> 
> Perhaps we could propagate any clock related properties from the UART
> node into the Xen hypervisor node (or a subnode)? The Xen Linux code
> would then consume those and mark the relevant clocks as in use. I've
> not looked into the clock bindings to know how plausible this might be.

It looks like the bindings for clock is well defined and fairly common
(Documentation/devicetree/bindins/clock/clock-bindings.txt). So your
solution sounds plausible.

Regards,

-- 
Julien Grall

[OSSTEST PATCH v2 00/19] Upgrade to Stretch

[Wei Liu]

First version of this series can be found at [0].

This version contains workaround for Arndale boards. They are now functional.

A bunch of test cases failed:

1. Rumpkernel tests -- I've sent an email to Antti for advice.
2. Windows tests -- They don't look different from normal flights. 
3. memdisk-try-append -- Osstest couldn't find some file. I don't think it is
   related to the code I modified.
4. guest-localmigrate/x10 for xl-qcow2 test -- Guest kernel bug.
5. nested hvm amd, pvhv2 -- Expected failure.

Example flight:
http://logs.test-lab.xenproject.org/osstest/logs/115404/

The armhf d-i failure is fixed with an additional patch ("Skip bootloader
installaion for arm32 on Stretch) on top of the code for 15404, in:

http://logs.test-lab.xenproject.org/osstest/logs/115404/

Cc: Julien Grall <julien.grall@arm.com>

[0] <20171020103840.32762-1-wei.liu2@citrix.com>

Wei Liu (19):
  gitignore: ignore vim swap file
  ts-xen-build-prep: only install w3c-dtd-xhtml for suites <Stretch
  ts-xen-build-prep: install packages for suites >jessie
  ts-xen-install: install some packages on stretch
  Debian.pm: use sysvinit-core instead of systemd
  ts-leak-check: suppress systemd-shim, which leaks in stretch
  ts-host-install: don't use the new nic naming scheme
  ts-guests-nbd-mirror: use target_{get,put}file_root to transfter cfg
  ts-debian-fixup: merge origin extra= to our own
  ts-debian-fixup: use correct resume device
  ts-debian-hvm-install: disable new nic naming scheme
  ts-xen-build-prep: install e2fslibs-dev
  TestSupport: add dpkg option when installing packages
  ts-guests-nbd-mirror: make it work with stretch
  Add clk_ignore_unused for stretch for arm hosts
  Set mac address in interfaces(5) if force-mac-address is set
  Skip bootloader installation for arm32 in Stretch
  make-flight: don't test pvgrub for Xen XXX
  Switch to Debian Stretch

 .gitignore             |  1 +
 Osstest.pm             |  2 +-
 Osstest/Debian.pm      | 19 ++++++++++++++++--
 Osstest/TestSupport.pm |  3 ++-
 make-flight            | 17 +++++++++++++++-
 production-config      |  2 ++
 ts-debian-fixup        | 14 ++++++++++++-
 ts-debian-hvm-install  | 13 ++++++++++++
 ts-guests-nbd-mirror   | 54 ++++++++++++++++++++++++++++++++++++++++++++------
 ts-host-install        |  4 ++++
 ts-leak-check          |  1 +
 ts-xen-build-prep      | 15 +++++++++++++-
 ts-xen-install         |  3 +++
 13 files changed, 135 insertions(+), 13 deletions(-)

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 01/19] gitignore: ignore vim swap file

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 425506b..f7e5b77 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,6 @@
 *~
 *.bak
+*.swp
 tmp
 *.tmp
 bisection.ps
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 02/19] ts-xen-build-prep: only install w3c-dtd-xhtml for suites <Stretch

[Wei Liu]

That package is not included in Stretch.

That package was installed because libvirt build needed it. However
libvirt builds fine without it in Stretch.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build-prep | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 3e98364..7718cff 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -206,9 +206,12 @@ sub prep () {
                       libglib2.0-dev liblzma-dev pkg-config
                       autoconf automake libtool xsltproc
                       libxml2-utils libxml2-dev
-                      libdevmapper-dev w3c-dtd-xhtml libxml-xpath-perl
+                      libdevmapper-dev libxml-xpath-perl
                       ccache nasm checkpolicy ebtables);
 
+    if ($ho->{Suite} =~ m/squeeze|wheezy|jessie/) {
+	push(@packages, "w3c-dtd-xhtml");
+    }
     if ($ho->{Suite} !~ m/squeeze|wheezy/) {
 	push(@packages, qw(ocaml-nox ocaml-findlib));
     }
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 03/19] ts-xen-build-prep: install packages for suites >jessie

[Wei Liu]

Stubdom build needs texinfo.

Libvirt build needs autopoint.

QEMU build needs libpciaccess-dev.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build-prep | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 7718cff..80bfb30 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -221,6 +221,10 @@ sub prep () {
 	# jessie (>jessie?)
 	push(@packages, "libnl-route-3-dev");
     }
+    if ($ho->{Suite} !~ m/squeeze|wheezy|jessie/) {
+        push(@packages, qw(texinfo autopoint libpciaccess-dev));
+    }
+
     target_install_packages($ho, @packages);
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 04/19] ts-xen-install: install some packages on stretch

[Wei Liu]

The "route" command is now in that package.

libnl is needed when running xl.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-install | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/ts-xen-install b/ts-xen-install
index ec907c5..d4c25c7 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -57,6 +57,9 @@ sub packages () {
                                libsdl1.2debian libglib2.0-0 liblzma5
 			       qemu-utils
                                netcat-openbsd));
+    if ($ho->{Suite} =~ m/stretch/) {
+        target_install_packages($ho, 'net-tools libnl-route-3-200');
+    }
     if ($ho->{Suite} =~ m/jessie/) {
         target_install_packages($ho, 'libnl-route-3-200');
     }
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 05/19] Debian.pm: use sysvinit-core instead of systemd

[Wei Liu]

Install that packages for suites >wheezy, because they use systemd as
the default init.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 845027a..e7fb020 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -827,7 +827,7 @@ sub preseed_base ($$$;@) {
 
     # Systemd doesn't honor osstest-confirm-booted service, which
     # breaks ts-leak-check.  Fall back to SysV init for now.
-    if ( $suite =~ /jessie/ ) {
+    if ( $suite !~ /squeeze|wheezy/ ) {
        preseed_hook_command($ho, 'late_command', $sfx, <<END)
 in-target apt-get install -y sysvinit-core
 END
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 06/19] ts-leak-check: suppress systemd-shim, which leaks in stretch

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-leak-check | 1 +
 1 file changed, 1 insertion(+)

diff --git a/ts-leak-check b/ts-leak-check
index 678d069..41e6245 100755
--- a/ts-leak-check
+++ b/ts-leak-check
@@ -202,6 +202,7 @@ xenstore /vm
 xenstore /libxl
 
 process .* udevd
+process .* /.+/systemd-shim
 
 file /var/run/xenstored/db
 file /var/run/xenstored/db.debug
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-host-install | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ts-host-install b/ts-host-install
index 11c14a7..7339858 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -271,6 +271,10 @@ END
     # why this is repeated.
     push @hocmdline, "console=$console" unless $console eq "NONE";
 
+    # Don't use "Predictable Network Interface Names"
+    # https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
+    push @hocmdline, "net.ifnames=0" if $ho->{Suite} =~ m/stretch/;
+
     push @hocmdline,
         get_host_property($ho, "linux-boot-append $ho->{Suite}", ''),
         get_host_property($ho, "linux-boot-append $ho->{Suite} $r{arch}", '');
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 08/19] ts-guests-nbd-mirror: use target_{get, put}file_root to transfter cfg

[Wei Liu]

The original code used target_cmd_output_root which caused a trailing
new line to be deleted, which caused libvirt converter to fail.

It wasn't discovered until now because we appended too many "\n".

Use target_{get,put}file_root to do the job.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-guests-nbd-mirror | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/ts-guests-nbd-mirror b/ts-guests-nbd-mirror
index ca8300d..3032204 100755
--- a/ts-guests-nbd-mirror
+++ b/ts-guests-nbd-mirror
@@ -115,8 +115,11 @@ sub shuffleconfigs () {
 	my $gn= $gns[$i];
 	my $gho= $ghos[$i];
 	my $cfgpath= $r{ "$gho->{Guest}_cfgpath" };
-	my $cfgdata= target_cmd_output_root($sho,"cat $cfgpath");
-	target_putfilecontents_root_stash($cho,10,$cfgdata,$cfgpath);
+	my $file= $cfgpath;
+	$file=~ s,/,-,g;
+	$file= "$stash/".hostnamepath($cho)."--$file";
+	target_getfile_root($sho, 60, $cfgpath, $file);
+	target_putfile_root($cho, 60, $file, $cfgpath);
     }
 }
 
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 09/19] ts-debian-fixup: merge origin extra= to our own

[Wei Liu]

The original extra= was not removed, so there were two extra= in the
resulting config file.

It wasn't a problem for xl because the second extra= took precedence.
However libvirt tests would only pick up the first extra= --  they
worked by chance.

Fix this issue by merging the original.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-fixup | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-debian-fixup b/ts-debian-fixup
index f29971d..288766d 100755
--- a/ts-debian-fixup
+++ b/ts-debian-fixup
@@ -175,7 +175,7 @@ sub otherfixupcfg () {
         $extra .= " iommu=soft";
     }
 
-    $cfg .= "\nextra='$extra'\n";
+    $cfg =~ s/^extra\s*=\s*['"](.*)['"]/extra = '$1 $extra'/mg;
 };
 
 sub writecfg () {
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 10/19] ts-debian-fixup: use correct resume device

[Wei Liu]

See code comment for explanation.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-fixup | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/ts-debian-fixup b/ts-debian-fixup
index 288766d..9a63c61 100755
--- a/ts-debian-fixup
+++ b/ts-debian-fixup
@@ -175,6 +175,18 @@ sub otherfixupcfg () {
         $extra .= " iommu=soft";
     }
 
+    # There might be stale entries in /etc/initramfs-tools/conf.d/resume which
+    # get stored in the initramfs. That introduces delay in guest booting which
+    # might cause tests to fail.
+    #
+    # This is particularly prominent in stretch when it tries to scan for the
+    # inexistent device(s) for a long time. See also Debian bug #784810.
+    #
+    # Override that in kernel command line with the correct swap partition.
+    $cfg =~ m/'phy:.+-swap,(xvda\d+),.*'/;
+    $extra .= " resume=/dev/$1";
+    logm("change resume device to $1");
+
     $cfg =~ s/^extra\s*=\s*['"](.*)['"]/extra = '$1 $extra'/mg;
 };
 
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 11/19] ts-debian-hvm-install: disable new nic naming scheme

[Wei Liu]

This is required to fix nested hvm test. The L1 host is installed by
this script. We want the L1 host to not use the new nic naming scheme.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-hvm-install | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 54d5d1c..afe8dc9 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -114,6 +114,19 @@ set -ex
 in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list
 END
 
+    # Do not use "Predictable Network Interface Names" -- this can break
+    # nested HVM tests.
+    # https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
+    #
+    # See also
+    # https://www.debian.org/releases/stretch/example-preseed.txt
+    my $netifnames = "";
+    $netifnames = <<END if $ho->{Suite} =~ m/stretch/;
+d-i debian-installer/add-kernel-opts string net.ifnames=0
+END
+
+    $preseed_file .= "$netifnames";
+
     $preseed_file .= preseed_hook_cmds();
 
     return $preseed_file;
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 12/19] ts-xen-build-prep: install e2fslibs-dev

[Wei Liu]

The in-tree libfsimage ext2fs implementation can't handle 64bit
enabled ext4, which is the default in stretch.

Installing e2fslibs-dev causes libfsimage to pick up the packaged
ext2fs implementation.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build-prep | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 80bfb30..8b4b08a 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -225,6 +225,12 @@ sub prep () {
         push(@packages, qw(texinfo autopoint libpciaccess-dev));
     }
 
+    # The in-tree ext4 support in libfsimage can't cope with 64bit ext4 on
+    # 32bit build. Use the packaged library.
+    if ($ho->{Suite} !~ m/squeeze|wheezy|jessie/) {
+        push(@packages, qw(e2fslibs-dev));
+    }
+
     target_install_packages($ho, @packages);
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 13/19] TestSupport: add dpkg option when installing packages

[Wei Liu]

Upgrading configuration file of nbd-client is controlled by dpkg in
stretch. Add dpkg option to keep old configuration file(s).

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/TestSupport.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 6d5e667..238be9e 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -527,7 +527,8 @@ sub target_run_apt {
     my ($ho, @aptopts) = @_;
     target_cmd_root($ho,
         "DEBIAN_PRIORITY=critical UCF_FORCE_CONFFOLD=y \\
-            with-lock-ex -w /var/lock/osstest-apt apt-get @aptopts", 3000);
+            with-lock-ex -w /var/lock/osstest-apt \\
+            apt-get -o Dpkg::Options::=\"--force-confold\" @aptopts", 3000);
 }
 sub target_install_packages {
     my ($ho, @packages) = @_;
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 14/19] ts-guests-nbd-mirror: make it work with stretch

[Wei Liu]

On the server side, only add oldstyle= and port= on Wheezy and Jessie.
Stretch doesn't support or need those anymore.

On the client side, generate new style configuration file.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-guests-nbd-mirror | 47 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 43 insertions(+), 4 deletions(-)

diff --git a/ts-guests-nbd-mirror b/ts-guests-nbd-mirror
index 3032204..7bcc02c 100755
--- a/ts-guests-nbd-mirror
+++ b/ts-guests-nbd-mirror
@@ -60,15 +60,19 @@ sub configserver () {
 [generic]
     user = root
 END
-    $scfg .= <<END unless $sho->{Suite} =~ m/sarge|lenny|squeeze/;
+
+    $scfg .= <<END if $sho->{Suite} =~ m/wheezy|jessie/;
     oldstyle = true
 END
+
     foreach my $v (@vols) {
 	$v->{Port}= unique_incrementing_runvar("${srvhost}_nextport",4000);
 	$v->{Path}= "/dev/$v->{Gho}{Vg}/$v->{Lv}";
 	$scfg.=<<END;
 [export$v->{Ix}]
     exportname = $v->{Path}
+END
+	$scfg.=<<END if $sho->{Suite} =~ m/wheezy|jessie/;
     port = $v->{Port}
 END
     }
@@ -79,9 +83,7 @@ END
     target_install_packages($sho, qw(nbd-server));
 }
 
-sub configclient () {
-    target_cmd_root($cho, "dpkg --purge nbd-client ||:");
-
+sub configclient_pre_stretch () {
     my $mydaemon= '/root/nbd-client-async';
     target_putfilecontents_root_stash($cho,10,<<'END',$mydaemon);
 #!/bin/sh
@@ -107,7 +109,44 @@ NBD_PORT[$v->{Ix}]=$v->{Port}
 END
     }
     target_putfilecontents_root_stash($cho,10,$ccfg,"/etc/nbd-client");
+}
+
+sub configclient_stretch () {
+    my $ccfg = <<END;
+# generated by $0
+END
+
+    foreach my $v (@vols) {
+	my $nbddev = "nbd$v->{Ix}";
+	$ccfg .= <<END;
+$nbddev $sho->{Name} export$v->{Ix}
+END
+    }
+
+    target_putfilecontents_root_stash($cho,10,$ccfg,"/etc/nbdtab");
+}
+
+sub configclient () {
+    target_cmd_root($cho, "dpkg --purge nbd-client ||:");
+
+    if ($cho->{Suite} !~ m/stretch/) {
+        configclient_pre_stretch();
+    } else {
+        configclient_stretch();
+    }
+
     target_install_packages($cho, qw(nbd-client));
+
+    if ($cho->{Suite} !~ m/squeeze|wheezy|jessie/) {
+	foreach my $v (@vols) {
+	    my $nbddev = "nbd$v->{Ix}";
+	    target_cmd_root($cho, <<END);
+mkdir -p /dev/$v->{Gho}{Vg}
+if ! test -L $v->{Path}; then ln -s /dev/$nbddev $v->{Path}; fi
+END
+	}
+	target_cmd_root($cho, "/etc/init.d/nbd-client start");
+    }
 }
 
 sub shuffleconfigs () {
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 15/19] Add clk_ignore_unused for stretch for arm hosts

[Wei Liu]

Without that parameter we lose uart output.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index e7fb020..2bfd5ae 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -240,7 +240,7 @@ END
 	push @xenkopt, $xenkopt;
 	# http://bugs.xenproject.org/xen/bug/45
 	push @xenkopt, "clk_ignore_unused"
-	    if $ho->{Suite} =~ m/wheezy|jessie/;
+	    if $ho->{Suite} =~ m/wheezy|jessie|stretch/;
 
 	$xenkopt = join ' ', @xenkopt;
 	logm("Dom0 Linux options: $xenkopt");
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 16/19] Set mac address in interfaces(5) if force-mac-address is set

[Wei Liu]

ff9e0d8cbd generated a udev rule for setting the mac address. But that
udev rule is not copied into the target so reboot after installation
will fail.

We can copy the udev rule to target system so the reboot after
installation works, but then the generated udev rules will end up in
initramfs, which means the guest (which uses host's initrd)  will use
the same rune to set conflicting mac address.

Put the mac address in interfaces(5). We still need to keep the udev
rule for the initrd overlay otherwise host installation will fail.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/Debian.pm | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 2bfd5ae..2c3bcf4 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -1207,6 +1207,16 @@ END
 	preseed_hook_command($ho, 'late_command', $sfx, $cmds);
     }
 
+    my $wantphysif= get_host_property($ho,'interface force','auto');
+    if ($wantphysif ne 'auto' && $ho->{Flags}{'force-mac-address'}) {
+	preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+
+sed -i 's/\\(iface\\s*$wantphysif\\s*inet.*\\)/\\1\\n    hwaddress $ho->{Ether}/' /target/etc/network/interfaces
+END
+    }
+
     if ( $ho->{Flags}{'need-uboot-bootscr'} ) {
 	my @bootargs = uboot_common_kernel_bootargs($ho);
 
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 17/19] Skip bootloader installation for arm32 in Stretch

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/Debian.pm | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 2c3bcf4..b2d5007 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -1027,6 +1027,11 @@ END
 	$preseed_file.= (<<END);
 d-i     nobootloader/confirmation_common boolean true
 END
+        # Stretch has a different method to skip bootloader installation
+        $preseed_file.= (<<END);
+d-i     grub-installer/skip boolean true
+d-i     lilo-installer/skip boolean true
+END
 
         # Debian Bug #771949 means that update-menu-list always
         # generates a full absolute path to the kernel + initrd, while
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 18/19] make-flight: don't test pvgrub for Xen XXX

[Wei Liu]

XXX Need to pin down the version of Xen when the upgrade to stretch is
complete because osstest configuration is branched for each version.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 make-flight | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/make-flight b/make-flight
index 76620c1..b0f1e69 100755
--- a/make-flight
+++ b/make-flight
@@ -827,7 +827,22 @@ test_matrix_do_one () {
   #do_passthrough_tests
 
   do_pygrub_tests
-  do_pvgrub_tests
+
+  # pvgrub1 tests for version < XXX only
+  case "$xenbranch" in
+  xen-4.3-testing) test_pvgrub=y ;;
+  xen-4.4-testing) test_pvgrub=y ;;
+  xen-4.5-testing) test_pvgrub=y ;;
+  xen-4.6-testing) test_pvgrub=y ;;
+  xen-4.7-testing) test_pvgrub=y ;;
+  xen-4.8-testing) test_pvgrub=y ;;
+  xen-4.9-testing) test_pvgrub=y ;;
+  *)               test_pvgrub=n ;;
+  esac
+
+  if [ x$test_pvgrub = xy ]; then
+      do_pvgrub_tests
+  fi
 
   do_xtf_tests
   do_livepatch_tests
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v2 19/19] Switch to Debian Stretch

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 Osstest.pm        | 2 +-
 production-config | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/Osstest.pm b/Osstest.pm
index ceb62ca..f2e4dfa 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -86,7 +86,7 @@ our %c = qw(
 
     Images images
 
-    DebianSuite jessie
+    DebianSuite stretch
     DebianMirrorSubpath debian
 
     TestHostKeypairPath id_rsa_osstest
diff --git a/production-config b/production-config
index 0e6a51e..251d82f 100644
--- a/production-config
+++ b/production-config
@@ -93,10 +93,12 @@ TftpNetbootGroup osstest
 # Update with ./mg-debian-installer-update(-all)
 TftpDiVersion_wheezy 2016-06-08
 TftpDiVersion_jessie 2017-04-06
+TftpDiVersion_stretch 2017-10-11
 
 # For ISO installs
 DebianImageVersion_wheezy 7.2.0
 DebianImageVersion_jessie 8.2.0
+DebianImageVersion_stretch 9.2.1
 
 # These should normally be the same.
 # Update with ./mg-cpu-microcode-update
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH v2 00/19] Upgrade to Stretch

[Wei Liu]

On Tue, Oct 31, 2017 at 01:51:44PM +0000, Wei Liu wrote:
> First version of this series can be found at [0].
> 
> This version contains workaround for Arndale boards. They are now functional.
> 
> A bunch of test cases failed:
> 
> 1. Rumpkernel tests -- I've sent an email to Antti for advice.
> 2. Windows tests -- They don't look different from normal flights. 
> 3. memdisk-try-append -- Osstest couldn't find some file. I don't think it is
>    related to the code I modified.
> 4. guest-localmigrate/x10 for xl-qcow2 test -- Guest kernel bug.
> 5. nested hvm amd, pvhv2 -- Expected failure.
> 
> Example flight:
> http://logs.test-lab.xenproject.org/osstest/logs/115404/
> 
> The armhf d-i failure is fixed with an additional patch ("Skip bootloader
> installaion for arm32 on Stretch) on top of the code for 15404, in:
> 
> http://logs.test-lab.xenproject.org/osstest/logs/115404/

This should be 115433.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH v2 00/19] Upgrade to Stretch

[Wei Liu]

On Tue, Oct 31, 2017 at 06:42:28PM +0000, Wei Liu wrote:
> On Tue, Oct 31, 2017 at 01:51:44PM +0000, Wei Liu wrote:
> > First version of this series can be found at [0].
> > 
> > This version contains workaround for Arndale boards. They are now functional.
> > 
> > A bunch of test cases failed:
> > 
> > 1. Rumpkernel tests -- I've sent an email to Antti for advice.
> > 2. Windows tests -- They don't look different from normal flights. 
> > 3. memdisk-try-append -- Osstest couldn't find some file. I don't think it is
> >    related to the code I modified.
> > 4. guest-localmigrate/x10 for xl-qcow2 test -- Guest kernel bug.
> > 5. nested hvm amd, pvhv2 -- Expected failure.
> > 
> > Example flight:
> > http://logs.test-lab.xenproject.org/osstest/logs/115404/
> > 
> > The armhf d-i failure is fixed with an additional patch ("Skip bootloader
> > installaion for arm32 on Stretch) on top of the code for 15404, in:
> > 
> > http://logs.test-lab.xenproject.org/osstest/logs/115404/
> 
> This should be 115433.

And a complete run of the series:

http://logs.test-lab.xenproject.org/osstest/logs/115436/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 0/2] Keep pvgrub tests

[Wei Liu]

These are extra patches to replace

    make-flight: don't test pvgrub for Xen XXX

so that we can keep pvgrub tests.

See
http://logs.test-lab.xenproject.org/osstest/logs/115518/


Wei Liu (2):
  ts-debian-di-install: use gho to pick d-i
  make-flight: guest should use jessie to test pvgrub

 make-flight          | 8 ++++----
 ts-debian-di-install | 4 ++--
 2 files changed, 6 insertions(+), 6 deletions(-)

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 1/2] ts-debian-di-install: use gho to pick d-i

[Wei Liu]

The original code used ho which gave us the host suite, but we wanted
the guest suite.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-di-install | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ts-debian-di-install b/ts-debian-di-install
index 6007971..361a171 100755
--- a/ts-debian-di-install
+++ b/ts-debian-di-install
@@ -152,8 +152,8 @@ sub setup_netboot($$$)
 	die if $r{ "$gho->{Guest}_netboot_kernel" }
 	||     $r{ "$gho->{Guest}_netboot_ramdisk" };
 
-	my $di_path = $c{TftpPath}.'/'.$ho->{Tftp}{DiBase}.'/'.${arch}.'/'.
-	    debian_guest_di_version($ho).'-'.$ho->{Suite};
+	my $di_path = $c{TftpPath}.'/'.$gho->{Tftp}{DiBase}.'/'.${arch}.'/'.
+	    debian_guest_di_version($gho).'-'.$gho->{Suite};
 
         if (${arch} =~ m/amd64|i386/) {
 	    $kernel = "$di_path/vmlinuz-xen";
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 2/2] make-flight: guest should use jessie to test pvgrub

[Wei Liu]

Stretch has 64bit feature enabled for ext4, which pvgrub can't cope.
We want to continue to test pvgrub, so specify jessie in the guest
suite field.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 make-flight | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/make-flight b/make-flight
index 76620c1..9c5399b 100755
--- a/make-flight
+++ b/make-flight
@@ -578,8 +578,8 @@ do_pvgrub_tests () {
   job_create_test test-$xenarch$kern-$dom0arch-amd64-pvgrub     \
     test-debian-di xl $xenarch $dom0arch                        \
       debian_arch=amd64                                         \
-      debian_suite=$guestsuite                                  \
-      debian_di_version=$guest_di_version                       \
+      debian_suite=jessie                                       \
+      debian_di_version=`getconfig_TftpDiVersion_suite jessie`  \
       debian_method=netboot                                     \
       debian_bootloader=pvgrub                                  \
       all_hostflags=$most_hostflags                             \
@@ -587,8 +587,8 @@ do_pvgrub_tests () {
   job_create_test test-$xenarch$kern-$dom0arch-i386-pvgrub      \
     test-debian-di xl $xenarch $dom0arch                        \
       debian_arch=i386                                          \
-      debian_suite=$guestsuite                                  \
-      debian_di_version=$guest_di_version                       \
+      debian_suite=jessie                                       \
+      debian_di_version=`getconfig_TftpDiVersion_suite jessie`  \
       debian_method=netboot                                     \
       debian_bootloader=pvgrub                                  \
       all_hostflags=$most_hostflags
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 1/2] ts-debian-di-install: use gho to pick d-i

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 1/2] ts-debian-di-install: use gho to pick d-i"):
> The original code used ho which gave us the host suite, but we wanted
> the guest suite.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 2/2] make-flight: guest should use jessie to test pvgrub

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 2/2] make-flight: guest should use jessie to test pvgrub"):
> Stretch has 64bit feature enabled for ext4, which pvgrub can't cope.
> We want to continue to test pvgrub, so specify jessie in the guest
> suite field.

I'm not entirely comfortable with the hardcoding here, but I guess few
people are still using osstest configured to use squeeze so I think it
will do.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme

[Julien Grall]

On 12/12/17 15:15, Julien Grall wrote:
> Hi Wei/Ian,

Hi,

> I have tried this series on Arm64 hardware. I am able to boot and 
> install Debian on AMD Seattle (laxton{0,1}). But I don't get network 
> when using Cavium Thunder-X (rochester{0,1}) after reboot.
> 
> Looking into more details, the interface naming is not the same between 
> the installer and after reboot. In the former, eth4 is used whilst the 
> latter uses eth5 for the same interface (USB network dongle).
> 
> I have tried to use 'interface force' property to impose eth4, without 
> much luck. The interface naming is still eth5 after reboot.
> 
> I was wondering if you have any idea how to solve the naming scheme issue?

I dug a bit more, so 'force interface' will create the 
/etc/udev/rules.d/70-persistent-net.rules for the installer initrd.

However, it does not get copied to the installed Debian. For the moment, 
I workaround this by copying the file in overlay/ so it get installed.

With that, I manage to get the network in the installed Debian. I guess 
this is not the right way. What would be the right way?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme

[Wei Liu]

On Fri, Dec 15, 2017 at 03:45:06PM +0000, Julien Grall wrote:
> 
> 
> On 12/12/17 15:15, Julien Grall wrote:
> > Hi Wei/Ian,
> 
> Hi,
> 
> > I have tried this series on Arm64 hardware. I am able to boot and
> > install Debian on AMD Seattle (laxton{0,1}). But I don't get network
> > when using Cavium Thunder-X (rochester{0,1}) after reboot.
> > 
> > Looking into more details, the interface naming is not the same between
> > the installer and after reboot. In the former, eth4 is used whilst the
> > latter uses eth5 for the same interface (USB network dongle).
> > 
> > I have tried to use 'interface force' property to impose eth4, without
> > much luck. The interface naming is still eth5 after reboot.
> > 
> > I was wondering if you have any idea how to solve the naming scheme issue?
> 
> I dug a bit more, so 'force interface' will create the
> /etc/udev/rules.d/70-persistent-net.rules for the installer initrd.
> 
> However, it does not get copied to the installed Debian. For the moment, I
> workaround this by copying the file in overlay/ so it get installed.
> 

I tried that as well, but that's wrong.

That would make the generated rule get copied in to the host initrd,
which ends up being used as the guests initrd (at least on x86), then
the guest will hijack the host's IP address (!).

> With that, I manage to get the network in the installed Debian. I guess this
> is not the right way. What would be the right way?
> 

See v3 of this series, which has a patch named:

Set mac address in interfaces(5) if force-mac-address is set

That should work on ARM, too. 

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme

[Ian Jackson]

Wei Liu writes ("Re: [Xen-devel] [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme"):
> On Fri, Dec 15, 2017 at 03:45:06PM +0000, Julien Grall wrote:
> > I dug a bit more, so 'force interface' will create the
> > /etc/udev/rules.d/70-persistent-net.rules for the installer initrd.
> > 
> > However, it does not get copied to the installed Debian. For the moment, I
> > workaround this by copying the file in overlay/ so it get installed.
> 
> I tried that as well, but that's wrong.
> 
> That would make the generated rule get copied in to the host initrd,
> which ends up being used as the guests initrd (at least on x86), then
> the guest will hijack the host's IP address (!).

???

1. persistent-net.rules contains information saying "if nic with
address aa:bb:cc:dd:ee:ff shows up, call it eth2; also, do not call
anything else eth2".  So it has nothing to do with ip addresses.

2. initrds don't need to bring up the network so I'm not sure why this
would end up there.  If it does, we could add something to the
initramfs-tools config to stop it.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme

[Wei Liu]

On Fri, Jan 19, 2018 at 05:08:09PM +0000, Ian Jackson wrote:
> Wei Liu writes ("Re: [Xen-devel] [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme"):
> > On Fri, Dec 15, 2017 at 03:45:06PM +0000, Julien Grall wrote:
> > > I dug a bit more, so 'force interface' will create the
> > > /etc/udev/rules.d/70-persistent-net.rules for the installer initrd.
> > > 
> > > However, it does not get copied to the installed Debian. For the moment, I
> > > workaround this by copying the file in overlay/ so it get installed.
> > 
> > I tried that as well, but that's wrong.
> > 
> > That would make the generated rule get copied in to the host initrd,
> > which ends up being used as the guests initrd (at least on x86), then
> > the guest will hijack the host's IP address (!).
> 
> ???
> 
> 1. persistent-net.rules contains information saying "if nic with
> address aa:bb:cc:dd:ee:ff shows up, call it eth2; also, do not call
> anything else eth2".  So it has nothing to do with ip addresses.

The rule is generated by osstest, which says "please make this nic have
mac address XXXX". That would cause the guest to have the same mac
address as the host. When the guest is up and asks DHCP server for an
IP, it gets the host's IP.

> 
> 2. initrds don't need to bring up the network so I'm not sure why this
> would end up there.  If it does, we could add something to the
> initramfs-tools config to stop it.
> 

initramfs-tools does that. I'm not aware of an option to stop it from
doing that.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme

[Ian Jackson]

Wei Liu writes ("Re: [Xen-devel] [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme"):
> On Fri, Jan 19, 2018 at 05:08:09PM +0000, Ian Jackson wrote:
> > 1. persistent-net.rules contains information saying "if nic with
> > address aa:bb:cc:dd:ee:ff shows up, call it eth2; also, do not call
> > anything else eth2".  So it has nothing to do with ip addresses.
> 
> The rule is generated by osstest, which says "please make this nic have
> mac address XXXX". That would cause the guest to have the same mac
> address as the host. When the guest is up and asks DHCP server for an
> IP, it gets the host's IP.

persistent-net.rules (at least the parts of it generated by
persistent-net-generator) does not change the mac addresses of nics.
It matches according to the mac address and changes their "names".

I hope the osstest-generated one is no different.  I'm not even sure
it's possible to change the mac address from a udev rule and if it is
certainly nothing near us should do it.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme

[Wei Liu]

On Fri, Jan 19, 2018 at 05:56:15PM +0000, Ian Jackson wrote:
> Wei Liu writes ("Re: [Xen-devel] [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme"):
> > On Fri, Jan 19, 2018 at 05:08:09PM +0000, Ian Jackson wrote:
> > > 1. persistent-net.rules contains information saying "if nic with
> > > address aa:bb:cc:dd:ee:ff shows up, call it eth2; also, do not call
> > > anything else eth2".  So it has nothing to do with ip addresses.
> > 
> > The rule is generated by osstest, which says "please make this nic have
> > mac address XXXX". That would cause the guest to have the same mac
> > address as the host. When the guest is up and asks DHCP server for an
> > IP, it gets the host's IP.
> 
> persistent-net.rules (at least the parts of it generated by
> persistent-net-generator) does not change the mac addresses of nics.
> It matches according to the mac address and changes their "names".
> 
> I hope the osstest-generated one is no different.  I'm not even sure
> it's possible to change the mac address from a udev rule and if it is
> certainly nothing near us should do it.
> 

It is rather different. It is a hack put in by IanC to deal with
unreliable USB nic. IIRC the rule invokes ip to set the mac address.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme

[Ian Jackson]

Wei Liu writes ("Re: [Xen-devel] [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme"):
> It is rather different. It is a hack put in by IanC to deal with
> unreliable USB nic. IIRC the rule invokes ip to set the mac address.

Yikes.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 00/19] Upgrade to Stretch

[Julien Grall]

Hi Wei,

On 31/10/17 13:51, Wei Liu wrote:
> First version of this series can be found at [0].
> 
> This version contains workaround for Arndale boards. They are now functional.

I need the following patch [1] to use that branch on Arm64 boxes.

Also, on Thunder-X I needed a hack to keep the interface name
consistent between the installer and Debian (see discussion on
patch #7). Looking at the discussion I am not sure what is the
way forward to fix it.

Cheers,

[1]

commit 26badf528e343d3fc540b61490011d391f2360a8
Author: Julien Grall <julien.grall@arm.com>
Date:   Tue Feb 20 11:54:51 2018 +0000

    stretch: Use chainloading when booting using GRUB on Arm64
    
    The GRUB package in Stretch is not able to boot Xen on Arm64.
    Use chainloading as we did for Jessie for the time being.
    
    Note that a bug has been filled on Debian to integrate Xen
    pactches for the next release (see [1]).
    
    [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=884770
    
    Signed-off-by: Julien Grall <julien.grall@arm.com>

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index b2d5007..2a2efa0 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -437,7 +437,7 @@ sub setupboot_grub2 ($$$$) {
     # Grub2 on Jessie/arm* doesn't do multiboot, so we must chainload.
     my $need_uefi_chainload =
         get_host_property($ho, "firmware", "") eq "uefi" &&
-        $ho->{Suite} =~ m/jessie/ && $r{arch} =~ m/^arm/;
+        $ho->{Suite} =~ m/jessie|stretch/ && $r{arch} =~ m/^arm/;
 
     my $parsemenu= sub {
         my $f= bl_getmenu_open($ho, $rmenu, "$stash/$ho->{Name}--grub.cfg.1");

> 
> A bunch of test cases failed:
> 
> 1. Rumpkernel tests -- I've sent an email to Antti for advice.
> 2. Windows tests -- They don't look different from normal flights.
> 3. memdisk-try-append -- Osstest couldn't find some file. I don't think it is
>     related to the code I modified.
> 4. guest-localmigrate/x10 for xl-qcow2 test -- Guest kernel bug.
> 5. nested hvm amd, pvhv2 -- Expected failure.
> 
> Example flight:
> http://logs.test-lab.xenproject.org/osstest/logs/115404/
> 
> The armhf d-i failure is fixed with an additional patch ("Skip bootloader
> installaion for arm32 on Stretch) on top of the code for 15404, in:
> 
> http://logs.test-lab.xenproject.org/osstest/logs/115404/
> 
> Cc: Julien Grall <julien.grall@arm.com>
> 
> [0] <20171020103840.32762-1-wei.liu2@citrix.com>
> 
> Wei Liu (19):
>    gitignore: ignore vim swap file
>    ts-xen-build-prep: only install w3c-dtd-xhtml for suites <Stretch
>    ts-xen-build-prep: install packages for suites >jessie
>    ts-xen-install: install some packages on stretch
>    Debian.pm: use sysvinit-core instead of systemd
>    ts-leak-check: suppress systemd-shim, which leaks in stretch
>    ts-host-install: don't use the new nic naming scheme
>    ts-guests-nbd-mirror: use target_{get,put}file_root to transfter cfg
>    ts-debian-fixup: merge origin extra= to our own
>    ts-debian-fixup: use correct resume device
>    ts-debian-hvm-install: disable new nic naming scheme
>    ts-xen-build-prep: install e2fslibs-dev
>    TestSupport: add dpkg option when installing packages
>    ts-guests-nbd-mirror: make it work with stretch
>    Add clk_ignore_unused for stretch for arm hosts
>    Set mac address in interfaces(5) if force-mac-address is set
>    Skip bootloader installation for arm32 in Stretch
>    make-flight: don't test pvgrub for Xen XXX
>    Switch to Debian Stretch
> 
>   .gitignore             |  1 +
>   Osstest.pm             |  2 +-
>   Osstest/Debian.pm      | 19 ++++++++++++++++--
>   Osstest/TestSupport.pm |  3 ++-
>   make-flight            | 17 +++++++++++++++-
>   production-config      |  2 ++
>   ts-debian-fixup        | 14 ++++++++++++-
>   ts-debian-hvm-install  | 13 ++++++++++++
>   ts-guests-nbd-mirror   | 54 ++++++++++++++++++++++++++++++++++++++++++++------
>   ts-host-install        |  4 ++++
>   ts-leak-check          |  1 +
>   ts-xen-build-prep      | 15 +++++++++++++-
>   ts-xen-install         |  3 +++
>   13 files changed, 135 insertions(+), 13 deletions(-)
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 00/19] Upgrade to Stretch

[Wei Liu]

On Tue, Feb 20, 2018 at 01:38:12PM +0000, Julien Grall wrote:
> Hi Wei,
> 
> On 31/10/17 13:51, Wei Liu wrote:
> > First version of this series can be found at [0].
> > 
> > This version contains workaround for Arndale boards. They are now functional.
> 
> I need the following patch [1] to use that branch on Arm64 boxes.
> 
> Also, on Thunder-X I needed a hack to keep the interface name
> consistent between the installer and Debian (see discussion on
> patch #7). Looking at the discussion I am not sure what is the
> way forward to fix it.

IIRC Ian suggested you copy one of the udev rules from Jessie's d-i to
the initrd osstest generated for stretch, and perhaps copy it to the
installed OS as well. I can't remember which files and don't have a
jessie system to hand.

Ian?

> 
> Cheers,
> 
> [1]
> 
> commit 26badf528e343d3fc540b61490011d391f2360a8
> Author: Julien Grall <julien.grall@arm.com>
> Date:   Tue Feb 20 11:54:51 2018 +0000
> 
>     stretch: Use chainloading when booting using GRUB on Arm64
>     
>     The GRUB package in Stretch is not able to boot Xen on Arm64.
>     Use chainloading as we did for Jessie for the time being.
>     
>     Note that a bug has been filled on Debian to integrate Xen
>     pactches for the next release (see [1]).
>     
>     [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=884770
>     
>     Signed-off-by: Julien Grall <julien.grall@arm.com>
> 
> diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
> index b2d5007..2a2efa0 100644
> --- a/Osstest/Debian.pm
> +++ b/Osstest/Debian.pm
> @@ -437,7 +437,7 @@ sub setupboot_grub2 ($$$$) {
>      # Grub2 on Jessie/arm* doesn't do multiboot, so we must chainload.

Please also update the comment here.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 00/19] Upgrade to Stretch

[Ian Jackson]

Wei Liu writes ("Re: [OSSTEST PATCH v2 00/19] Upgrade to Stretch"):
> IIRC Ian suggested you copy one of the udev rules from Jessie's d-i to
> the initrd osstest generated for stretch, and perhaps copy it to the
> installed OS as well. I can't remember which files and don't have a
> jessie system to hand.

75-persistent-net-generator.rules
which is in /lib/udev/rules.d on jessie but would go in /etc
on a stretch system.  It will make 70-persistent-net.rules.

IDK if stretch installer will naturally copy 70-persistent-net.rules
to the installed system.  If not then a hook will have to be used to
do that too, so that the device name(s) assigned during the install
are replicated on the installed system.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 02/19] ts-xen-build-prep: only install w3c-dtd-xhtml for suites <Stretch

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 02/19] ts-xen-build-prep: only install w3c-dtd-xhtml for suites <Stretch"):
> That package is not included in Stretch.
> 
> That package was installed because libvirt build needed it. However
> libvirt builds fine without it in Stretch.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 04/19] ts-xen-install: install some packages on stretch

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 04/19] ts-xen-install: install some packages on stretch"):
> The "route" command is now in that package.
> 
> libnl is needed when running xl.
...
> +    if ($ho->{Suite} =~ m/stretch/) {
> +        target_install_packages($ho, 'net-tools libnl-route-3-200');
> +    }

We want to do this indefinitely, don't we ?  So it should be
  !~ m/squeeze|wheezy|jessie/
?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 14/19] ts-guests-nbd-mirror: make it work with stretch

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 14/19] ts-guests-nbd-mirror: make it work with stretch"):
> On the server side, only add oldstyle= and port= on Wheezy and Jessie.
> Stretch doesn't support or need those anymore.
...
> +    if ($cho->{Suite} !~ m/stretch/) {
> +        configclient_pre_stretch();

This will go wrong in buster.  Your match needs to be inverted and the
set of suites too, so that unknown suites get the new behaviour.

It's probably easier to swap the limbs of the if.

> +
> +    if ($cho->{Suite} !~ m/squeeze|wheezy|jessie/) {
> +	foreach my $v (@vols) {
> +	    my $nbddev = "nbd$v->{Ix}";
> +	    target_cmd_root($cho, <<END);
> +mkdir -p /dev/$v->{Gho}{Vg}
> +if ! test -L $v->{Path}; then ln -s /dev/$nbddev $v->{Path}; fi
> +END

This seems to duplicate code in what is now configclient_pre_stretch.

I also don't understand the logic that says:
 - on stretch, do the post-stretch thing, and make this symlink
 - on squeeze..jessie, do the pre-stretch thing, and make this symlink
 - on sarge, do the pre-stretch thing, and make the symlink twice
?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 17/19] Skip bootloader installation for arm32 in Stretch

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 17/19] Skip bootloader installation for arm32 in Stretch"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

But some minor style questions:

> diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
> index 2c3bcf4..b2d5007 100644
> --- a/Osstest/Debian.pm
> +++ b/Osstest/Debian.pm
> @@ -1027,6 +1027,11 @@ END
>  	$preseed_file.= (<<END);
>  d-i     nobootloader/confirmation_common boolean true
>  END
> +        # Stretch has a different method to skip bootloader installation
> +        $preseed_file.= (<<END);
> +d-i     grub-installer/skip boolean true
> +d-i     lilo-installer/skip boolean true

Why not fold this into the same here document ?

I would put the comments in the preseed file, for both versions, eg:
 
  +# jessie and earlier:
   d-i     nobootloader/confirmation_common boolean true
  +# stretch and later:
  +d-i     grub-installer/skip boolean true
  +d-i     lilo-installer/skip boolean true

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 15/19] Add clk_ignore_unused for stretch for arm hosts

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 15/19] Add clk_ignore_unused for stretch for arm hosts"):
> Without that parameter we lose uart output.

In this patch we are propagating a workaroound further into the
future.  This is an opportunity to prod the people responsible for the
bug.

Julien, is this workaround still needed ?
If so, what are the plans for a proper fix ?

But, the osstest patch is

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Thanks,
Ian.

> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  Osstest/Debian.pm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
> index e7fb020..2bfd5ae 100644
> --- a/Osstest/Debian.pm
> +++ b/Osstest/Debian.pm
> @@ -240,7 +240,7 @@ END
>  	push @xenkopt, $xenkopt;
>  	# http://bugs.xenproject.org/xen/bug/45
>  	push @xenkopt, "clk_ignore_unused"
> -	    if $ho->{Suite} =~ m/wheezy|jessie/;
> +	    if $ho->{Suite} =~ m/wheezy|jessie|stretch/;
>  
>  	$xenkopt = join ' ', @xenkopt;
>  	logm("Dom0 Linux options: $xenkopt");
> -- 
> 2.11.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 19/19] Switch to Debian Stretch

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 19/19] Switch to Debian Stretch"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 05/19] Debian.pm: use sysvinit-core instead of systemd

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 05/19] Debian.pm: use sysvinit-core instead of systemd"):
> Install that packages for suites >wheezy, because they use systemd as
> the default init.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 08/19] ts-guests-nbd-mirror: use target_{get, put}file_root to transfter cfg

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 08/19] ts-guests-nbd-mirror: use target_{get,put}file_root to transfter cfg"):
> The original code used target_cmd_output_root which caused a trailing
> new line to be deleted, which caused libvirt converter to fail.
> 
> It wasn't discovered until now because we appended too many "\n".
> 
> Use target_{get,put}file_root to do the job.

Maybe target_getfilecontents and stashfilecontents and
target_putfilecontents would be easier and also better ?

It is usually better to use the existing facilities for inventing
stash file names because they check for uniqueness.

Or maybe you want to invent `target_getfile_stash' which uses
open_unique_stashfile and target_getfile, and returns the filename.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 09/19] ts-debian-fixup: merge origin extra= to our own

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 09/19] ts-debian-fixup: merge origin extra= to our own"):
> The original extra= was not removed, so there were two extra= in the
> resulting config file.
> 
> It wasn't a problem for xl because the second extra= took precedence.
> However libvirt tests would only pick up the first extra= --  they
> worked by chance.
...
> -    $cfg .= "\nextra='$extra'\n";
> +    $cfg =~ s/^extra\s*=\s*['"](.*)['"]/extra = '$1 $extra'/mg;

Isn't this a no-op if there is no extra= already ?

Also you use s///g but AFAICT multiple extra would already be an error
of some kind and editing them all doesn't seem to make sense...

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 10/19] ts-debian-fixup: use correct resume device

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 10/19] ts-debian-fixup: use correct resume device"):
> See code comment for explanation.
...
> +    $cfg =~ m/'phy:.+-swap,(xvda\d+),.*'/;

This is quite a careless regexp!

> +    $extra .= " resume=/dev/$1";
> +    logm("change resume device to $1");

Can't we just say "noresume" or something ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 11/19] ts-debian-hvm-install: disable new nic naming scheme

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 11/19] ts-debian-hvm-install: disable new nic naming scheme"):
> This is required to fix nested hvm test. The L1 host is installed by
> this script. We want the L1 host to not use the new nic naming scheme.

Without this, and without the scheme that was in jessie and earlier,
machines with multiple nics can find that the nic names change
unpredictably, breaking stuff.

AIUI Julien has patches to drop in the network-*-generator things from
jessie.  We should have those.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 12/19] ts-xen-build-prep: install e2fslibs-dev

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 12/19] ts-xen-build-prep: install e2fslibs-dev"):
> The in-tree libfsimage ext2fs implementation can't handle 64bit
> enabled ext4, which is the default in stretch.
> 
> Installing e2fslibs-dev causes libfsimage to pick up the packaged
> ext2fs implementation.

This should be documented in xen.git:INSTALL I think ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 13/19] TestSupport: add dpkg option when installing packages

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 13/19] TestSupport: add dpkg option when installing packages"):
> Upgrading configuration file of nbd-client is controlled by dpkg in
> stretch. Add dpkg option to keep old configuration file(s).

I don't think w just want to suppress all these conffile conflicts.

Instead, perhaps it is possible to install nbd-client first and then
configure it, avoiding this problem.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 16/19] Set mac address in interfaces(5) if force-mac-address is set

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 16/19] Set mac address in interfaces(5) if force-mac-address is set"):
> ff9e0d8cbd generated a udev rule for setting the mac address. But that
> udev rule is not copied into the target so reboot after installation
> will fail.
> 
> We can copy the udev rule to target system so the reboot after
> installation works, but then the generated udev rules will end up in
> initramfs, which means the guest (which uses host's initrd)  will use
> the same rune to set conflicting mac address.
> 
> Put the mac address in interfaces(5). We still need to keep the udev
> rule for the initrd overlay otherwise host installation will fail.
...
> +sed -i 's/\\(iface\\s*$wantphysif\\s*inet.*\\)/\\1\\n    hwaddress $ho->{Ether}/' /target/etc/network/interfaces
> +END

Lovely!

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 18/19] make-flight: don't test pvgrub for Xen XXX

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH v2 18/19] make-flight: don't test pvgrub for Xen XXX"):
> XXX Need to pin down the version of Xen when the upgrade to stretch is
> complete because osstest configuration is branched for each version.

I don't understand this.

> +  # pvgrub1 tests for version < XXX only
> +  case "$xenbranch" in
> +  xen-4.3-testing) test_pvgrub=y ;;
> +  xen-4.4-testing) test_pvgrub=y ;;
> +  xen-4.5-testing) test_pvgrub=y ;;
> +  xen-4.6-testing) test_pvgrub=y ;;
> +  xen-4.7-testing) test_pvgrub=y ;;
> +  xen-4.8-testing) test_pvgrub=y ;;
> +  xen-4.9-testing) test_pvgrub=y ;;
> +  *)               test_pvgrub=n ;;
> +  esac

I don't understand what the relationship is between pvgrub1 and
stretch, nor how this relates to Xen releases.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme

[Ian Jackson]

Ian Jackson writes ("Re: [Xen-devel] [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme"):
> Wei Liu writes ("Re: [Xen-devel] [OSSTEST PATCH v2 07/19] ts-host-install: don't use the new nic naming scheme"):
> > It is rather different. It is a hack put in by IanC to deal with
> > unreliable USB nic. IIRC the rule invokes ip to set the mac address.
> 
> Yikes.

I have looked at this again.  Part of the difficulty is the confusion
between these two things:

1. Making the same NICs show up with the same names in each of
   the installer and every boot.  Without this, the network
   configuration in /etc/network/interfaces (written by the installer)
   does not work, and also osstest is not able to properly configure
   the bridging (in /etc/network/interfaces, again).

2. On some broken NICs the MAC address has to be set manually.
   This is the function of
     ff9e0d8cbd9e "ts-host-install: Add force-mac-address host flag"
   Obviously this can only possibly work if there is only one NIC.

In Debian, in jessie and earlier, (1) is achieved by
/lib/udev/rules.d/75-persistent-net-generator.rules.  That maintains a
persistent database of MAC address <-> name mappings, in
/etc/udev/rules.d/70-persistent-net.rules.  When a previously seen NIC
shows up, it is renamed by 70-persistent-net.rules.  When a
not-previously-seen NIC shows up, 75-persistent-net-generator.rules
selects a new unique eth* name for it, adds a rule to
70-persistent-net.rules, and renames the NIC.

For guests which share the host's initrd this works fine because the
guest has a new MAC address.  Its NIC therefore gets a new name.  It
does not matter that that name is probably eth1 or something.  (Or,
maybe, if the host has many NICs, eth7 or something.)

In Debian upstream in stretch and later, (1) is achieved by the new
kernel netname scheme (which this patch disables).  I don't know
exactly how it is implemented, but the basic idea is to (mostly) use
the MAC address in the name.  These names are inconvenient to work
with because they are opaque, and they break some of osstest's (IMO
reasonable) assumptions.

I think that we should fix (1) by copying jessie's
/lib/udev/rules.d/75-persistent-net-generator.rules into installs of
stretch and later.  We will have to also copy the
70-persistent-net.rules generated in the installer into the target
system.  I think Julien has patches to do that ?

(2) is a different kettle of fish entirely.  It is slightly
unfortunate that ff9e0d8cbd9e edits 70-persistent-net.rules rather
than providing its own rules file - that is confusing.

I think ff9e0d8cbd9e avoids breaking guests which share the hosts
initrd, on jessie, simply because 70-persistent-net.rules is not in
the initrd at all.

Discussion surrounding
  "[16/19] Set mac address in interfaces(5) if force -mac-address is set"
suggests to me that in stretch, 70-persistent-net.rules *is* in the
initrd if it exists.  This is odd because I thought stretch had
dropped official support for this scheme.

Overall I think the right solution for stretch is:

 * Set net.ifnames=0 as this patch does.

 * Use the installer early command hook to install
   75-persistent-net-generator.rules in the initramfs.

 * If the installer does not do this already, use an installer
   late command hook to copy 70-persistent-net.rules from the
   installer to the target.

The above will fix (1) by reverting everything to the way it works on
jessie.  To fix (2) (which maybe worked as it was on jessie by
chance),

 * Use an early command hook to install something like the
   udev rule in ff9e0d8cbd9e but not into 70-persistent-net.rules.
   Instead, put it in 69-fixup-broken-nic.rules or something.
   That rule will not make it into the installed system so
   will cause no trouble for any guests.

 * Use a late command hook to fix the mac address in /e/n/i,
   as done in "[16/19] Set mac address in interfaces(5)..."

Does everyone agree ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 09/19] ts-debian-fixup: merge origin extra= to our own

[Wei Liu]

On Wed, Mar 07, 2018 at 03:06:19PM +0000, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH v2 09/19] ts-debian-fixup: merge origin extra= to our own"):
> > The original extra= was not removed, so there were two extra= in the
> > resulting config file.
> > 
> > It wasn't a problem for xl because the second extra= took precedence.
> > However libvirt tests would only pick up the first extra= --  they
> > worked by chance.
> ...
> > -    $cfg .= "\nextra='$extra'\n";
> > +    $cfg =~ s/^extra\s*=\s*['"](.*)['"]/extra = '$1 $extra'/mg;
> 
> Isn't this a no-op if there is no extra= already ?

Right. We will need to test if there is already extra=.

> 
> Also you use s///g but AFAICT multiple extra would already be an error
> of some kind and editing them all doesn't seem to make sense...
> 

Yeah, removing the 'g' should be fine.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 08/19] ts-guests-nbd-mirror: use target_{get, put}file_root to transfter cfg

[Wei Liu]

On Wed, Mar 07, 2018 at 03:04:41PM +0000, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH v2 08/19] ts-guests-nbd-mirror: use target_{get,put}file_root to transfter cfg"):
> > The original code used target_cmd_output_root which caused a trailing
> > new line to be deleted, which caused libvirt converter to fail.
> > 
> > It wasn't discovered until now because we appended too many "\n".
> > 
> > Use target_{get,put}file_root to do the job.
> 
> Maybe target_getfilecontents and stashfilecontents and
> target_putfilecontents would be easier and also better ?
> 
> It is usually better to use the existing facilities for inventing
> stash file names because they check for uniqueness.
> 

There is no target_getfilecontents, and the method I used is used
throughout osstest code.

> Or maybe you want to invent `target_getfile_stash' which uses
> open_unique_stashfile and target_getfile, and returns the filename.
> 

Let me see if I can invent something after fixing those more important
issues.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 13/19] TestSupport: add dpkg option when installing packages

[Wei Liu]

On Wed, Mar 07, 2018 at 03:11:47PM +0000, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH v2 13/19] TestSupport: add dpkg option when installing packages"):
> > Upgrading configuration file of nbd-client is controlled by dpkg in
> > stretch. Add dpkg option to keep old configuration file(s).
> 
> I don't think w just want to suppress all these conffile conflicts.
> 
> Instead, perhaps it is possible to install nbd-client first and then
> configure it, avoiding this problem.

AIUI the confi file is installed first because we want nbd-client to
get configured and starts automatically after installation.

It might work if we install nbd-client first, install config file and
then restart the service. I will need to try.

Wei.

> 
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 14/19] ts-guests-nbd-mirror: make it work with stretch

[Wei Liu]

On Wed, Mar 07, 2018 at 02:45:50PM +0000, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH v2 14/19] ts-guests-nbd-mirror: make it work with stretch"):
> > On the server side, only add oldstyle= and port= on Wheezy and Jessie.
> > Stretch doesn't support or need those anymore.
> ...
> > +    if ($cho->{Suite} !~ m/stretch/) {
> > +        configclient_pre_stretch();
> 
> This will go wrong in buster.  Your match needs to be inverted and the
> set of suites too, so that unknown suites get the new behaviour.
> 
> It's probably easier to swap the limbs of the if.
> 
> > +
> > +    if ($cho->{Suite} !~ m/squeeze|wheezy|jessie/) {
> > +	foreach my $v (@vols) {
> > +	    my $nbddev = "nbd$v->{Ix}";
> > +	    target_cmd_root($cho, <<END);
> > +mkdir -p /dev/$v->{Gho}{Vg}
> > +if ! test -L $v->{Path}; then ln -s /dev/$nbddev $v->{Path}; fi
> > +END
> 
> This seems to duplicate code in what is now configclient_pre_stretch.

The snippet is duplicate but there isn't a better way to do it.

In pre stretch function, the snippet is put into the client config file
and nbd-client will run it.

New ndb-client config file is different. I don't think it can run shell
script anymore.

> 
> I also don't understand the logic that says:
>  - on stretch, do the post-stretch thing, and make this symlink
>  - on squeeze..jessie, do the pre-stretch thing, and make this symlink

They are both needed by osstest.

>  - on sarge, do the pre-stretch thing, and make the symlink twice

I don't follow. There is no sarge here. I don't think that's relevant
anymore. If you like, I can list squeeze and sarge in the pre stretch
listing.

Wei.

> ?
> 
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [OSSTEST PATCH v2 14/19] ts-guests-nbd-mirror: make it work with stretch

[Wei Liu]

On Tue, May 08, 2018 at 10:20:20AM +0100, Wei Liu wrote:
> On Wed, Mar 07, 2018 at 02:45:50PM +0000, Ian Jackson wrote:
> > Wei Liu writes ("[OSSTEST PATCH v2 14/19] ts-guests-nbd-mirror: make it work with stretch"):
> > > On the server side, only add oldstyle= and port= on Wheezy and Jessie.
> > > Stretch doesn't support or need those anymore.
> > ...
> > > +    if ($cho->{Suite} !~ m/stretch/) {
> > > +        configclient_pre_stretch();
> > 
> > This will go wrong in buster.  Your match needs to be inverted and the
> > set of suites too, so that unknown suites get the new behaviour.
> > 
> > It's probably easier to swap the limbs of the if.
> > 
> > > +
> > > +    if ($cho->{Suite} !~ m/squeeze|wheezy|jessie/) {
> > > +	foreach my $v (@vols) {
> > > +	    my $nbddev = "nbd$v->{Ix}";
> > > +	    target_cmd_root($cho, <<END);
> > > +mkdir -p /dev/$v->{Gho}{Vg}
> > > +if ! test -L $v->{Path}; then ln -s /dev/$nbddev $v->{Path}; fi
> > > +END
> > 
> > This seems to duplicate code in what is now configclient_pre_stretch.
> 
> The snippet is duplicate but there isn't a better way to do it.
> 
> In pre stretch function, the snippet is put into the client config file
> and nbd-client will run it.
> 
> New ndb-client config file is different. I don't think it can run shell
> script anymore.
> 
> > 
> > I also don't understand the logic that says:
> >  - on stretch, do the post-stretch thing, and make this symlink
> >  - on squeeze..jessie, do the pre-stretch thing, and make this symlink
> 
> They are both needed by osstest.
> 
> >  - on sarge, do the pre-stretch thing, and make the symlink twice
> 
> I don't follow. There is no sarge here. I don't think that's relevant
> anymore. If you like, I can list squeeze and sarge in the pre stretch
> listing.
> 

BTW my assumption is anyone who runs osstest in the wild will have at
least wheezy at this point. If you think otherwise, please let me know.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel