[OSSTEST PATCH 00/32] Changes to support the new colo

[OSSTEST PATCH 00/32] Changes to support the new colo

[Ian Jackson]

This is the current patchset I am using in the new colo, tidied up.

I think 01-29 are in principle suitable for pushing to the current
live instance in Cambridge right away.

30-32 are changes to the config which will go wrong on the Cambridge
instance.

Thanks,
Ian.

[OSSTEST PATCH 01/32] PDU: msw: support arbitrary arguments

[Ian Jackson]

No functional change with existing host properties.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/PDU/msw.pm |    7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/Osstest/PDU/msw.pm b/Osstest/PDU/msw.pm
index 40a2970..c159cc3 100644
--- a/Osstest/PDU/msw.pm
+++ b/Osstest/PDU/msw.pm
@@ -34,14 +34,15 @@ BEGIN {
 }
 
 sub new {
-    my ($class, $ho, $methname,$pdu,$port) = @_;
-    return bless { Pdu => $pdu, Port => $port }, $class;
+    my ($class, $ho, $methname, @args) = @_;
+
+    return bless { Args => \@args }, $class;
 }
 
 sub pdu_power_state {
     my ($mo, $on) = @_;
     my $onoff= $on ? "on" : "off";
-    system_checked("./pdu-msw $mo->{Pdu} $mo->{Port} $onoff");
+    system_checked("./pdu-msw @{ $mo->{Args} } $onoff");
 }
 
 1;
-- 
1.7.10.4

[OSSTEST PATCH 02/32] PDU: pdu-msw: Split $read_oid and $write_oid

[Ian Jackson]

Some PDUs have a different OID space for turning ports on and off, to
the one for querying.  To make this easier to handle, split the
variable $oid into $read_oid and $write_oid.

Also move $baseoid settings earlier so that we can modify them with
command-line arguments.

No functional change in this patch.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 pdu-msw |   18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/pdu-msw b/pdu-msw
index 075f9b8..fc4d314 100755
--- a/pdu-msw
+++ b/pdu-msw
@@ -26,13 +26,14 @@ use strict qw(refs vars);
 use Net::SNMP;
 use Data::Dumper;
 
+my $community= 'private';
+my $baseoid= '.1.3.6.1.4.1.318.1.1.4.4.2.1';
+my $baseoid_write= "$baseoid.3";
+
 if (@ARGV<2 || @ARGV>3 || $ARGV[0] =~ m/^-/) { die "bad usage\n$usagemsg"; }
 
 our ($dnsname,$outlet,$action) = @ARGV;
 
-my $community= 'private';
-my $baseoid= '.1.3.6.1.4.1.318.1.1.4.4.2.1';
-
 my ($session,$error) = Net::SNMP->session(
                                           -hostname => $dnsname,
                                           -community => $community,
@@ -83,7 +84,8 @@ if ($outlet =~ m/^\d+$/) {
     ($useport,$usename)= @{ $found[0] };
 }
 
-my $oid= "$baseoid.3.$useport";
+my $read_oid= "$baseoid.3.$useport";
+my $write_oid= "$baseoid_write.$useport";
 
 my @map= (undef, qw(
                     on
@@ -94,9 +96,9 @@ my @map= (undef, qw(
                     delayed-reboot));
 
 sub show () {
-    my $got= $session->get_request($oid);
-    die "SNMP error reading $oid ".$session->error()." " unless $got;
-    my $val= $got->{$oid};
+    my $got= $session->get_request($read_oid);
+    die "SNMP error reading $read_oid ".$session->error()." " unless $got;
+    my $val= $got->{$read_oid};
     die unless $val;
     my $mean= $map[$val];
     die "$val ?" unless defined $mean;
@@ -110,7 +112,7 @@ sub set () {
                  $action =~ m/^(?:reboot)$/ ? 3 :
                  die "unknown action $action\n$usagemsg");
         
-    my $res= $session->set_request(-varbindlist => [ $oid, INTEGER, $valset ]);
+    my $res= $session->set_request(-varbindlist => [ $write_oid, INTEGER, $valset ]);
     die "SNMP set ".$session->error()." " unless $res;
 }
 
-- 
1.7.10.4

[OSSTEST PATCH 03/32] PDU: pdu-msw: Support APC v6 firmware

[Ian Jackson]

APC PDUs with firmware 6.x have a different OID space for turning
ports on and off, to the one for querying.  The old namespace still
works to turn the port on and off but returns a genErr error response!

Support a new command-line option --apc6 to use this other OID.

Information about OIDs etc. obtained here:
  https://git.fedorahosted.org/cgit/fence-agents.git/commit/?id=55ccdd79f530092af06eea5b4ce6a24bd82c0875

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 pdu-msw |    9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/pdu-msw b/pdu-msw
index fc4d314..6c25690 100755
--- a/pdu-msw
+++ b/pdu-msw
@@ -30,6 +30,15 @@ my $community= 'private';
 my $baseoid= '.1.3.6.1.4.1.318.1.1.4.4.2.1';
 my $baseoid_write= "$baseoid.3";
 
+while (@ARGV && $ARGV[0] =~ m/^-/) {
+    $_ = shift @ARGV;
+    if (m/^--apc6$/) {
+	$baseoid_write= '.1.3.6.1.4.1.318.1.1.12.3.3.1.1.4';
+    } else {
+	die "$_ ?";
+    }
+}
+
 if (@ARGV<2 || @ARGV>3 || $ARGV[0] =~ m/^-/) { die "bad usage\n$usagemsg"; }
 
 our ($dnsname,$outlet,$action) = @ARGV;
-- 
1.7.10.4

[OSSTEST PATCH 04/32] pxe setup: Define %name% in templates

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm |    1 +
 README                 |    1 +
 2 files changed, 2 insertions(+)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 8754e22..5677946 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -2017,6 +2017,7 @@ sub host_pxefile ($) {
 	$eth =~ s/\-$//;
 	$v{'etherhyph'} = $eth;
     }
+    $v{'name'} = $ho->{Name};
     if (defined $ho->{IpStatic}) {
 	my $ip = $ho->{IpStatic};
 	$ip =~ s/\b0+//g;
diff --git a/README b/README
index 0a0242c..fe0abb2 100644
--- a/README
+++ b/README
@@ -453,6 +453,7 @@ TftpPxeTemplates
     List (space-separated) of template filenames for writing
     The templates contain variable substitutions %var%
     The variables are the runvars plus
+        name          hostname (short name as in resource databases etc.)
         ether         host/guest mac address
         etherhyp      ... in lowercase hyphen-separated form (for pxelinux)
 	ipaddr        host/guest static ip address
-- 
1.7.10.4

[OSSTEST PATCH 05/32] mg-hosts: Provide new "power" subcommand

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-hosts |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/mg-hosts b/mg-hosts
index a6f6e6e..6ec8d56 100755
--- a/mg-hosts
+++ b/mg-hosts
@@ -122,6 +122,19 @@ END
     }
 }
 
+sub cmd_power () {
+    die unless @ARGV==2;
+    my ($host,$power) = @ARGV;
+    $_ = $power;
+    $power = m/^1|^on/ ? 1 : m/^0|^off/ ? 0 : m/^r|^c/ ? -1 : die;
+    my $ho= selecthost("host=$host");
+    if ($power >= 0) {
+	power_state($ho, $power);
+    } else {
+	power_cycle($ho);
+    }
+}
+
 sub cmd_create_like () {
     die unless @ARGV==2;
     my ($src,$dsts) = @ARGV;
-- 
1.7.10.4

[OSSTEST PATCH 06/32] Osstest: Default config: Use foreach for empty defaults

[Ian Jackson]

No functional change, but makes adding new empty defaults shorter.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest.pm |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Osstest.pm b/Osstest.pm
index 6d7c083..7a49591 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -74,7 +74,9 @@ our %c = qw(
     DebianNonfreeFirmware firmware-bnx2
 );
 
-$c{DebianPreseed} = '';
+$c{$_}='' foreach qw(
+    DebianPreseed
+);
 
 #---------- general setup and config reading ----------
 
-- 
1.7.10.4

[OSSTEST PATCH 07/32] pxe setup: New TftpPxeTemplatesReal feature

[Ian Jackson]

Rather than having mg-hosts mkpxedir hardcode the strange thing done
in the XenClient test lab in the Citrix Cambridge office, provide a
somewhat more general and correct approach:

* Generalise host_pxefile to support [Tftp]PxeTemplatesReal as well
  as [Tftp]PxeTemplates.
* Default [Tftp]PxeTemplatesReal to ''

mg-hosts mkpxedir now uses these templates, as follow:
* Create the host's PxeTemplates-based pxe file's parent
  directories and make the parent directory be owned by PxeGroup.
* If the PxeTemplatesReal is specified and different, make a symlink
  named according to PxeTemplatesReal pointing at the previous file.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest.pm             |    1 +
 Osstest/TestSupport.pm |   13 ++++++++-----
 README                 |    5 +++++
 mg-hosts               |   28 ++++++++++++++++++++--------
 4 files changed, 34 insertions(+), 13 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 7a49591..22dbc8b 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -76,6 +76,7 @@ our %c = qw(
 
 $c{$_}='' foreach qw(
     DebianPreseed
+    TftpPxeTemplatesReal
 );
 
 #---------- general setup and config reading ----------
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 5677946..2d1ba9e 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -860,7 +860,8 @@ sub selecthost ($) {
     logm("TftpScope is $tftpscope");
     $ho->{Tftp} = { };
     $ho->{Tftp}{$_} = $c{"Tftp${_}_${tftpscope}"} || $c{"Tftp${_}"}
-        foreach qw(Path TmpDir PxeDir PxeGroup PxeTemplates DiBase);
+        foreach qw(Path TmpDir PxeDir PxeGroup PxeTemplates PxeTemplatesReal
+                   DiBase);
 
     #----- finalise -----
 
@@ -2005,9 +2006,11 @@ sub file_link_contents ($$$) {
     logm("wrote $fn". (defined $stash ? " (stashed as $stash)" : ""));
 }
 
-sub host_pxefile ($) {
-    my ($ho) = @_;
+sub host_pxefile ($;$) {
+    my ($ho, $templatekey) = @_;
     my %v = %r;
+    $templatekey //= 'PxeTemplates';
+    my $templates = $ho->{Tftp}{$templatekey} || $ho->{Tftp}{PxeTemplates};
     if (defined $ho->{Ether}) {
 	my $eth = $v{'ether'} = $ho->{Ether};
 	$eth =~ y/A-Z/a-z/;
@@ -2024,7 +2027,7 @@ sub host_pxefile ($) {
 	$v{'ipaddr'} = $ip;
 	$v{'ipaddrhex'} = sprintf "%02X%02X%02X%02X", split /\./, $ip;
     }
-    foreach my $pat (split /\s+/, $ho->{Tftp}{PxeTemplates}) {
+    foreach my $pat (split /\s+/, $templates) {
 	# we skip patterns that contain any references to undefined %var%s
 	$pat =~ s{\%(\w*)\%}{
 		    $1 eq '' ? '%' :
@@ -2034,7 +2037,7 @@ sub host_pxefile ($) {
 	# and return the first pattern we managed to completely substitute
         return $pat;
     }
-    die "no pxe template matched $ho->{Tftp}{PxeTemplates} ".
+    die "no pxe template ($templatekey) matched $templates ".
         (join ",", sort keys %v)." ?";
 }
 
diff --git a/README b/README
index fe0abb2..e780762 100644
--- a/README
+++ b/README
@@ -441,6 +441,7 @@ TftpFoo_<scope> and TftpFoo
                       (e.g. pxelinux.cfg)
         PxeGroup      The Unix group which should own files under `PxeDir'.
         PxeTemplates  See TftpPxeTemplates
+        PxeTemplatesReal
 
         DiBase        The path under `Path' to the root of the debian installer
                       images.
@@ -463,6 +464,10 @@ TftpPxeTemplates
     ip address - are skipped.  The first template all of whose ingredients
     are known is used, with TftpPath and TftpPxeDir prepended.
 
+TftpPxeTemplatesReal
+    Template filename which mg-hosts mkpxedir should make be a symlink to
+    the TftpPxeTemplates.  Not used otherwise.
+
 ========================================
 
 Host-specific config settigs
diff --git a/mg-hosts b/mg-hosts
index 6ec8d56..f695f98 100755
--- a/mg-hosts
+++ b/mg-hosts
@@ -87,6 +87,7 @@ use strict qw(vars refs);
 use DBI;
 use Osstest;
 use Osstest::TestSupport;
+use File::Basename;
 
 csreadconfig();
 
@@ -108,17 +109,28 @@ sub cmd_mkpxedir () {
     my $sudo = $ENV{'OSSTEST_SUDO'} // 'sudo';
     foreach my $hn (@ARGV) {
         my $ho= selecthost("host=$hn");
-        my $macdir= $ho->{Ether};
-        $macdir =~ s/\:/-/g;
-        system_checked(<<END);
+	my $pxefile = host_pxefile($ho);
+	my $pxerealfile = host_pxefile($ho, 'PxeTemplatesReal');
+	my $dirname = dirname $pxefile;
+	my $cmd = <<END;
             set -e
 	    cd $ho->{Tftp}{Path}$ho->{Tftp}{PxeDir}
-            $sudo chown root.$ho->{Tftp}{PxeGroup} $macdir
-            $sudo chmod 2775 $macdir
-            $sudo rm -f $hn
-            $sudo ln -s $macdir $hn
-            ls -ald $hn $macdir
 END
+        $cmd .= <<END if $dirname ne '.';
+	    $sudo mkdir -p $dirname
+END
+        $cmd .= <<END;
+            $sudo chown root.$ho->{Tftp}{PxeGroup} $dirname
+            $sudo chmod 2775 $dirname
+END
+        if ($pxefile ne $pxerealfile) {
+	    my $subdirs = "../" x $pxefile =~ m#/#;
+            $cmd .= <<END;
+	    $sudo ln -sf $subdirs$pxefile $pxerealfile
+END
+        }
+        print $cmd;
+        system_checked($cmd) unless $dryrun;
     }
 }
 
-- 
1.7.10.4

[OSSTEST PATCH 08/32] pxe setup: mg-hosts mkpxedir has -n (dry run) option

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-hosts |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/mg-hosts b/mg-hosts
index f695f98..497139a 100755
--- a/mg-hosts
+++ b/mg-hosts
@@ -105,6 +105,8 @@ sub checkhost ($) {
 sub l ($) { return split /,/, $_[0]; }
 
 sub cmd_mkpxedir () {
+    my $dryrun = 0;
+    if (@ARGV && $ARGV[0] eq '-n') { shift @ARGV; $dryrun= 1; }
     die unless @ARGV>=1;
     my $sudo = $ENV{'OSSTEST_SUDO'} // 'sudo';
     foreach my $hn (@ARGV) {
-- 
1.7.10.4

[OSSTEST PATCH 09/32] Debian mirrors: Support proxy, and default to ftp.debian.org if specified

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest.pm        |    3 +++
 Osstest/Debian.pm |    1 +
 README            |    7 ++++++-
 3 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/Osstest.pm b/Osstest.pm
index 22dbc8b..aaaaf7a 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -76,6 +76,7 @@ our %c = qw(
 
 $c{$_}='' foreach qw(
     DebianPreseed
+    DebianMirrorProxy
     TftpPxeTemplatesReal
 );
 
@@ -186,6 +187,8 @@ sub readglobalconfig () {
     $c{GuestDebianSuite} ||= $c{DebianSuite};
 
     $c{DefaultBranch} ||= 'xen-unstable';
+
+    $c{DebianMirrorHost} ||= 'ftp.debian.org' if $c{DebianMirrorProxy};
 }
 
 sub augmentconfigdefaults {
diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 7633b51..f419f22 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -562,6 +562,7 @@ d-i finish-install/reboot_in_progress note
 d-i cdrom-detect/eject boolean false
 
 d-i mirror/http/hostname string $c{DebianMirrorHost}
+d-i mirror/http/proxy string $c{DebianMirrorProxy}
 d-i mirror/http/directory string /$c{DebianMirrorSubpath}
 d-i apt-setup/use_mirror boolean yes
 d-i apt-setup/another boolean false
diff --git a/README b/README
index e780762..9634185 100644
--- a/README
+++ b/README
@@ -291,7 +291,12 @@ NetNameservers
 DebianMirrorHost
    Domain name or address of the Debian mirror to use.
    Set DebianMirrorSubpath to the path inside the mirror if
-   your mirror isn't at http://<DebianMirrorHost>/debian/
+   your mirror isn't at http://<DebianMirrorHost>/debian/ .
+   Defaults to ftp.debian.org if DebianMirrorProxy is set.
+
+DebianMirrorProxy
+   The apt proxy to specify for Debian (and derivatives),
+   eg http://apt-cacher:3142/ .
 
 TestHost <hostname>
 TestHost_<ident> <hostname>
-- 
1.7.10.4

[OSSTEST PATCH 10/32] readglobalconfig: Break out $readcfg as a sub

[Ian Jackson]

And make it use anonymous filehandles, and be able to not tolerate
ENOENT.  This is preparation for making it recursive (`Include').

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest.pm |   75 +++++++++++++++++++++++++++++++++---------------------------
 1 file changed, 41 insertions(+), 34 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index aaaaf7a..8e63389 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -106,43 +106,50 @@ sub readglobalconfig () {
 
     my $cfgfiles = $ENV{'OSSTEST_CONFIG'} || "$ENV{'HOME'}/.xen-osstest/config";
 
-    foreach my $cfgfile (split /\:/, $cfgfiles) {
-	if (!open C, '<', "$cfgfile") {
-	    die "$cfgfile $!" unless $!==&ENOENT;
-	} else {
-	    while (<C>) {
-		die "missing newline" unless chomp;
-		s/^\s*//;
-		s/\s+$//;
-		next if m/^\#/;
-		next unless m/\S/;
-		if (m/^($cfgvar_re)\s+(\S.*)$/) {
-		    $c{$1} = $2;
-		} elsif (m/^($cfgvar_re)=\s*\<\<(\'?)(.*)\2\s*$/) {
-		    my ($vn,$qu,$delim) = ($1,$2,$3);
-		    my $val = '';
-		    $!=0; while (<C>) {
-			last if $_ eq "$delim\n";
-			$val .= $_;
-		    }
-		    die $! unless length $_;
-		    die unless !length $val || $val =~ m/\n$/;
-		    if ($qu eq '') {
-			my $reconstruct =
-			    "\$val = <<${qu}${delim}${qu}; 1;\n".
-			    "${val}${delim}\n";
-			eval $reconstruct or
-			    die "$1 here doc ($reconstruct) $@";
-		    }
-		    $c{$vn} = $val;
-		} elsif (m/^($cfgvar_re)=(.*)$/) {
-		    eval "\$c{$1} = ( $2 ); 1;" or die "$1 parsed val ($2) $@";
-		} else {
-		    die "bad syntax";
+    my $readcfg;
+    $readcfg = sub ($$) {
+	my ($cfgfile,$enoent_ok) = @_;
+	my $fh = new IO::File $cfgfile, '<';
+	if (!$fh) {
+	    die "$cfgfile $!" unless $enoent_ok && $!==&ENOENT;
+	    return;
+	}
+	while (<$fh>) {
+	    die "missing newline" unless chomp;
+	    s/^\s*//;
+	    s/\s+$//;
+	    next if m/^\#/;
+	    next unless m/\S/;
+	    if (m/^($cfgvar_re)\s+(\S.*)$/) {
+		$c{$1} = $2;
+	    } elsif (m/^($cfgvar_re)=\s*\<\<(\'?)(.*)\2\s*$/) {
+		my ($vn,$qu,$delim) = ($1,$2,$3);
+		my $val = '';
+		$!=0; while (<$fh>) {
+		    last if $_ eq "$delim\n";
+		    $val .= $_;
+		}
+		die $! unless length $_;
+		die unless !length $val || $val =~ m/\n$/;
+		if ($qu eq '') {
+		    my $reconstruct =
+			"\$val = <<${qu}${delim}${qu}; 1;\n".
+			"${val}${delim}\n";
+		    eval $reconstruct or
+			die "$1 here doc ($reconstruct) $@";
 		}
+		$c{$vn} = $val;
+	    } elsif (m/^($cfgvar_re)=(.*)$/) {
+		eval "\$c{$1} = ( $2 ); 1;" or die "$1 parsed val ($2) $@";
+	    } else {
+		die "bad syntax";
 	    }
-	    close C or die "$cfgfile $!";
 	}
+	close $fh or die "$cfgfile $!";
+    };
+
+    foreach my $cfgfile (split /\:/, $cfgfiles) {
+	$readcfg->($cfgfile, 1);
     }
 
     # dynamic default config settings
-- 
1.7.10.4

[OSSTEST PATCH 11/32] readglobalconfig: Support Include

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest.pm |    9 ++++++++-
 README     |    3 +++
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/Osstest.pm b/Osstest.pm
index 8e63389..9fbadbd 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -20,6 +20,7 @@ use strict;
 use warnings;
 
 use POSIX;
+use File::Basename;
 
 BEGIN {
     use Exporter ();
@@ -120,7 +121,13 @@ sub readglobalconfig () {
 	    s/\s+$//;
 	    next if m/^\#/;
 	    next unless m/\S/;
-	    if (m/^($cfgvar_re)\s+(\S.*)$/) {
+	    if (m/^include\s+(\S.*)$/i) {
+		my $newfn = $1;
+		if ($newfn !~ m#^/|^\./#) {
+		    $newfn = dirname($cfgfile)."/".$newfn;
+		}
+		$readcfg->($newfn, 0);
+	    } elsif (m/^($cfgvar_re)\s+(\S.*)$/) {
 		$c{$1} = $2;
 	    } elsif (m/^($cfgvar_re)=\s*\<\<(\'?)(.*)\2\s*$/) {
 		my ($vn,$qu,$delim) = ($1,$2,$3);
diff --git a/README b/README
index 9634185..0aaba2c 100644
--- a/README
+++ b/README
@@ -276,6 +276,9 @@ To set a value to the empty string, say
 For an example see  ./standalone-config-example
 Values later in the file, or in a later file, take precedence.
 
+You can say `Include <filename>'.  <filename> is relative to the file
+doing the including, unless it starts with `/' or `./'.
+
 ========================================
 
 Important config settings for standalone mode:
-- 
1.7.10.4

[OSSTEST PATCH 12/32] HostDB Executive: Option to suppress "configdb"

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/HostDB/Executive.pm |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/Osstest/HostDB/Executive.pm b/Osstest/HostDB/Executive.pm
index 6257829..3fa9982 100644
--- a/Osstest/HostDB/Executive.pm
+++ b/Osstest/HostDB/Executive.pm
@@ -101,6 +101,8 @@ sub default_methods ($$) {
     return if $ho->{Flags}{'no-reinstall'};
     return if $ho->{Ether} && $ho->{Power};
 
+    return if $c{HostDB_Executive_NoConfigDB};
+
     my $dbh_config= opendb('configdb');
     my $selname= $ho->{Fqdn};
     my $sth= $dbh_config->prepare(<<END);
-- 
1.7.10.4

[OSSTEST PATCH 13/32] target_cmd_build: Honour $c{HttpProxy}

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 2d1ba9e..1053016 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -559,7 +559,12 @@ sub target_cmd_build ($$$$) {
         export CCACHE_PREFIX DISTCC_FALLBACK DISTCC_HOSTS
 END
 
-    target_cmd($ho, <<END.$distcc.<<END.$script, $timeout);
+    my $httpproxy = defined($c{HttpProxy}) ? <<END : "";
+        http_proxy=$c{HttpProxy}
+        export http_proxy
+END
+
+    target_cmd($ho, <<END.$distcc.$httpproxy.<<END.$script, $timeout);
 	set -xe
         LC_ALL=C; export LC_ALL
         PATH=/usr/lib/ccache:\$PATH:/usr/lib/git-core
-- 
1.7.10.4

[OSSTEST PATCH 14/32] daemons: Allow QueueDaemon and OwnerDaemon to run on different hosts

[Ian Jackson]

We want the OwnerDaemon to run on the same host as the database (for
fate-sharing reasons).  OTOH the QueueDaemon is less critical if it
fails, and it generates reports etc., and wants to be more frequently
updated, so it should run on the osstest VM.

Permit this by:
 * Providing OwnerDaemonHost and QueueDaemonHost config settings
   which default to the value of ControlDaemonHost.
 * Using those everywhere.
 * In the daemons' Tcl code, have main-daemon take the string `Owner'
   or `Queue' so that it can look up both the host and port.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Executive.pm    |    7 ++++++-
 ms-ownerdaemon          |    2 +-
 ms-queuedaemon          |    4 ++--
 ms-reportuptime         |    2 +-
 tcl/JobDB-Executive.tcl |    2 +-
 tcl/daemonlib.tcl       |    5 +++--
 6 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index 3dc37d1..118a91a 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -100,6 +100,11 @@ augmentconfigdefaults(
     BisectionRevisonGraphSize => '600x300',
 );
 
+augmentconfigdefaults(
+    OwnerDaemonHost => $c{ControlDaemonHost},
+    QueueDaemonHost => $c{ControlDaemonHost},
+);
+
 #---------- configuration reader etc. ----------
 
 sub opendb_tests () {
@@ -241,7 +246,7 @@ sub alloc_resources_rollback_begin_work () {
 our $alloc_resources_waitstart;
 
 sub tcpconnect_queuedaemon () {
-    my $qserv= tcpconnect($c{ControlDaemonHost}, $c{QueueDaemonPort});
+    my $qserv= tcpconnect($c{QueueDaemonHost}, $c{QueueDaemonPort});
     $qserv->autoflush(1);
 
     $_= <$qserv>;  defined && m/^OK ms-queuedaemon\s/ or die "$_?";
diff --git a/ms-ownerdaemon b/ms-ownerdaemon
index 61b5ade..33ee238 100755
--- a/ms-ownerdaemon
+++ b/ms-ownerdaemon
@@ -89,7 +89,7 @@ proc banner {chan} {
     return "OK ms-ownerdaemon"
 }
 
-main-daemon $c(OwnerDaemonPort) {
+main-daemon Owner {
     jobdb::db-open
     
     jobdb::transaction resources {
diff --git a/ms-queuedaemon b/ms-queuedaemon
index 26d83e2..e15bc79 100755
--- a/ms-queuedaemon
+++ b/ms-queuedaemon
@@ -470,7 +470,7 @@ proc add-inhibit {why seconds} {
     log "inhibition $why: starts, ${seconds}s, now $inhibit"
 }
 
-main-daemon $c(QueueDaemonPort) {
+main-daemon Queue {
     global owndchan chandesc inhibit plan
     
     jobdb::db-open
@@ -484,7 +484,7 @@ main-daemon $c(QueueDaemonPort) {
     # 1: run queue only if new resources available
     # 2: definitely run queue
 
-    set owndchan [socket $c(ControlDaemonHost) $c(OwnerDaemonPort)]
+    set owndchan [socket $c(OwnerDaemonHost) $c(OwnerDaemonPort)]
     fconfigure $owndchan -buffering line -translation lf
     set chandesc($owndchan) ownd
     
diff --git a/ms-reportuptime b/ms-reportuptime
index 9b376a7..aeac483 100755
--- a/ms-reportuptime
+++ b/ms-reportuptime
@@ -23,7 +23,7 @@ source daemonlib.tcl
 
 readconfig
 
-set s [socket $c(ControlDaemonHost) $c(QueueDaemonPort)]
+set s [socket $c(QueueDaemonHost) $c(QueueDaemonPort)]
 fconfigure $s -buffering line
 set chandesc($s) qd
 
diff --git a/tcl/JobDB-Executive.tcl b/tcl/JobDB-Executive.tcl
index ad13cbd..430ea84 100644
--- a/tcl/JobDB-Executive.tcl
+++ b/tcl/JobDB-Executive.tcl
@@ -253,7 +253,7 @@ proc become-task {comment} {
     global env c
     if {[info exists env(OSSTEST_TASK)]} return
 
-    set ownerqueue [socket $c(ControlDaemonHost) $c(OwnerDaemonPort)]
+    set ownerqueue [socket $c(OwnerDaemonHost) $c(OwnerDaemonPort)]
     fconfigure $ownerqueue -buffering line -translation lf
     must-gets $ownerqueue {^OK ms-ownerdaemon\M}
     puts $ownerqueue create-task
diff --git a/tcl/daemonlib.tcl b/tcl/daemonlib.tcl
index c0e703d..d097624 100644
--- a/tcl/daemonlib.tcl
+++ b/tcl/daemonlib.tcl
@@ -184,10 +184,11 @@ proc newconn {chan addr port} {
     }
 }
 
-proc main-daemon {port setup} {
+proc main-daemon {which setup} {
     global c argv
 
-    set host $c(ControlDaemonHost)
+    set host $c(${which}DaemonHost)
+    set port $c(${which}DaemonPort)
 
     foreach arg $argv {
         switch -glob -- $arg {
-- 
1.7.10.4

[OSSTEST PATCH 15/32] cr-ensure-disk-space: Emacs perl-mode syntax workaround

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cr-ensure-disk-space |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 1c66ea7..aeaf572 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -84,7 +84,7 @@ for (;;) {
     die "age $age" if $age < $c{LogsMinExpireAge};
 
     $c{LogsPublish} =~ m/\:/ or die "$c{LogsPublish} ?";
-    my ($pubhost, $pubdir) = ($`,$');
+    my ($pubhost, $pubdir) = ($`,$'); #';
 
     printf "...";
 
-- 
1.7.10.4

[OSSTEST PATCH 16/32] cr-ensure-disk-space: Move LogsPublish decomposition up

[Ian Jackson]

We are going to want this for the df.
No functional change yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cr-ensure-disk-space |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index aeaf572..ff07b76 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -50,6 +50,9 @@ my $chkq= $dbh_tests->prepare("SELECT * FROM flights WHERE flight=?");
 our @flights;
 our @sshopts= qw(-o batchmode=yes);
 
+$c{LogsPublish} =~ m/\:/ or die "$c{LogsPublish} ?";
+my ($pubhost, $pubdir) = ($`,$'); #';
+
 for (;;) {
     open P, "df --block-size=1M -P . |" or die $!;
     $_= <P>;
@@ -83,9 +86,6 @@ for (;;) {
 
     die "age $age" if $age < $c{LogsMinExpireAge};
 
-    $c{LogsPublish} =~ m/\:/ or die "$c{LogsPublish} ?";
-    my ($pubhost, $pubdir) = ($`,$'); #';
-
     printf "...";
 
     my $spawn= sub {
-- 
1.7.10.4

[OSSTEST PATCH 17/32] cr-ensure-disk-space: Break out ontarget

[Ian Jackson]

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cr-ensure-disk-space |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index ff07b76..31880f7 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -53,6 +53,12 @@ our @sshopts= qw(-o batchmode=yes);
 $c{LogsPublish} =~ m/\:/ or die "$c{LogsPublish} ?";
 my ($pubhost, $pubdir) = ($`,$'); #';
 
+sub ontarget ($) {
+    my ($shellcmd) = @_;
+    # returns list to run that command
+    return qw(ssh -n), @sshopts, $pubhost, $shellcmd;
+}
+
 for (;;) {
     open P, "df --block-size=1M -P . |" or die $!;
     $_= <P>;
@@ -91,7 +97,7 @@ for (;;) {
     my $spawn= sub {
         $!=0; my $r= system @_; die "@_ $r $!" if $r;
     };
-    $spawn->(qw(ssh -n), @sshopts, $pubhost, <<END);
+    $spawn->(ontarget <<END);
              set -e
              cd $pubdir
              test -e $flight || exit 0
-- 
1.7.10.4

[OSSTEST PATCH 18/32] cr-ensure-disk-space: Do df check on target.

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cr-ensure-disk-space |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 31880f7..6f129bd 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -60,7 +60,7 @@ sub ontarget ($) {
 }
 
 for (;;) {
-    open P, "df --block-size=1M -P . |" or die $!;
+    open P, "-|", ontarget "df --block-size=1M -P $pubdir" or die $!;
     $_= <P>;
     m/^filesystem/i or die "$_ ?";
     $_= <P>;
-- 
1.7.10.4

[OSSTEST PATCH 19/32] cr-ensure-disk-space: Do listing of flights on target.

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cr-ensure-disk-space |   12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 6f129bd..af8efe6 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -71,11 +71,13 @@ for (;;) {
     last if $space >= $c{LogsMinSpaceMby};
 
     if (!@flights) {
-        opendir D, "." or die $!;
-        $!=0;
-        @flights = sort { $b <=> $a } grep { m/^\d+$/ } readdir D;
-        die if $!;
-        closedir D or die $!;
+	open P, "-|", ontarget "ls -1 $pubdir" or die $!;
+	while (<P>) {
+	    next unless m/^(\d+)\n$/;
+	    push @flights, $1;
+	}
+	$!=$?=0; close P or die "ls: $? $!";
+	@flights = sort { $b <=> $a } @flights;
         printf "(%d flights) ", scalar @flights;
         die unless @flights;
     }
-- 
1.7.10.4

[OSSTEST PATCH 20/32] cr-ensure-disk-space: Take argument for what to clean

[Ian Jackson]

Currently always "LogsPublish" (so no functional change).

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cr-ensure-disk-space |    7 +++++--
 cri-args-hostlists   |    2 +-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index af8efe6..884be26 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -36,7 +36,9 @@ while (@ARGV && $ARGV[0] =~ m/^\-/) {
     }
 }
 
-die if @ARGV;
+die unless @ARGV==1;
+
+our ($cfgbase) = @ARGV;
 
 csreadconfig();
 
@@ -50,7 +52,8 @@ my $chkq= $dbh_tests->prepare("SELECT * FROM flights WHERE flight=?");
 our @flights;
 our @sshopts= qw(-o batchmode=yes);
 
-$c{LogsPublish} =~ m/\:/ or die "$c{LogsPublish} ?";
+my $cfgvalue = $c{$cfgbase};
+$cgvvalue =~ m/\:/ or die "$c{LogsPublish} ?";
 my ($pubhost, $pubdir) = ($`,$'); #';
 
 sub ontarget ($) {
diff --git a/cri-args-hostlists b/cri-args-hostlists
index 2ee0b7b..11f01bd 100644
--- a/cri-args-hostlists
+++ b/cri-args-hostlists
@@ -74,7 +74,7 @@ execute_flight () {
         rm -f abort
         check_stop_core ''
 
-        ./cr-ensure-disk-space
+        ./cr-ensure-disk-space LogsPublish
 
 	export OSSTEST_RESOURCE_PREINFO="[$branch $2]"
 
-- 
1.7.10.4

[OSSTEST PATCH 21/32] cr-ensure-disk-space: Fix -n option

[Ian Jackson]

This would not really do a dry run - instead it would stop after
deleting one flight's logs!

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cr-ensure-disk-space |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 884be26..1c665b3 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -99,6 +99,8 @@ for (;;) {
 
     printf "...";
 
+    die if $dryrun;
+
     my $spawn= sub {
         $!=0; my $r= system @_; die "@_ $r $!" if $r;
     };
@@ -114,7 +116,6 @@ END
     $spawn->(qw(rm -rf), $flight);
 
     printf "done.\n";
-    exit 0 if $dryrun;
 }
 
 printf "ok.\n";
-- 
1.7.10.4

[OSSTEST PATCH 22/32] cr-ensure-disk-space: Permit argument to specify local directory.

[Ian Jackson]

If the argument is Logs rather than LogsPublish (ie, refers to a local
directory (without `:') rather than a remote one (with `:'), do things
locally (by invoking sh -ec so that we have identical quoting rules to
ssh).

No effectively functional change with the current configuration.

We still always do a local deletion.  This is anomalous and will
disappear shortly.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cr-ensure-disk-space |   15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 1c665b3..1defa6a 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -53,13 +53,22 @@ our @flights;
 our @sshopts= qw(-o batchmode=yes);
 
 my $cfgvalue = $c{$cfgbase};
-$cgvvalue =~ m/\:/ or die "$c{LogsPublish} ?";
-my ($pubhost, $pubdir) = ($`,$'); #';
+
+my ($pubhost, $pubdir);
+if ($cfgvalue =~ m/\:/) {
+    ($pubhost, $pubdir) = ($`,$'); #');
+} else {
+    ($pubhost, $pubdir) = (undef, $cfgvalue);
+}
 
 sub ontarget ($) {
     my ($shellcmd) = @_;
     # returns list to run that command
-    return qw(ssh -n), @sshopts, $pubhost, $shellcmd;
+    if (defined $pubhost) {
+	return qw(ssh -n), @sshopts, $pubhost, $shellcmd;
+    } else {
+	return qw(sh -ec), $shellcmd;
+    }
 }
 
 for (;;) {
-- 
1.7.10.4

[OSSTEST PATCH 23/32] cr-ensure-disk-space: Operate locally entirely separately.

[Ian Jackson]

Run cr-ensure-disk-space separately for local and remote directories.

Ie, do the local deletion in its own run (so removing the anomalous
separate invocation).

This makes little difference if the two disks are of roughly similar
sizes.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cr-ensure-disk-space |    4 ----
 cri-args-hostlists   |    1 +
 2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 1defa6a..1064590 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -45,7 +45,6 @@ csreadconfig();
 open LOCK, "> $c{GlobalLockDir}/publish-lock" or die $!;
 flock LOCK, LOCK_EX or die $!;
 
-chdir $c{Logs} or die $c{Logs};
 $|=1;
 
 my $chkq= $dbh_tests->prepare("SELECT * FROM flights WHERE flight=?");
@@ -121,9 +120,6 @@ for (;;) {
              rm -rf $flight
 END
 
-    printf "local..";
-    $spawn->(qw(rm -rf), $flight);
-
     printf "done.\n";
 }
 
diff --git a/cri-args-hostlists b/cri-args-hostlists
index 11f01bd..04cd6be 100644
--- a/cri-args-hostlists
+++ b/cri-args-hostlists
@@ -74,6 +74,7 @@ execute_flight () {
         rm -f abort
         check_stop_core ''
 
+        ./cr-ensure-disk-space Logs
         ./cr-ensure-disk-space LogsPublish
 
 	export OSSTEST_RESOURCE_PREINFO="[$branch $2]"
-- 
1.7.10.4

[OSSTEST PATCH 24/32] cr-ensure-disk-space: Honour different config for published log expiry

[Ian Jackson]

Break out references to $c{Logs...} into new dircfg() function, and
look up $c{${cfgbase}...} first.  No functional change with existing
config.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 cr-ensure-disk-space |   11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space
index 1064590..0ee84c7 100755
--- a/cr-ensure-disk-space
+++ b/cr-ensure-disk-space
@@ -70,6 +70,11 @@ sub ontarget ($) {
     }
 }
 
+sub dircfg ($) {
+    my ($k) = @_;
+    return $c{"${cfgbase}${k}"} // $c{"Logs${k}"};
+}
+
 for (;;) {
     open P, "-|", ontarget "df --block-size=1M -P $pubdir" or die $!;
     $_= <P>;
@@ -78,8 +83,8 @@ for (;;) {
     m,^\S+\s+\d+\s+\d+\s+(\d+)\s+, or die "$_ ?";
     $!=0; $?=0; close P or die "$! $?";
     my $space= $1;
-    printf "space: %8d, wanted: %8d ", $space, $c{LogsMinSpaceMby};
-    last if $space >= $c{LogsMinSpaceMby};
+    printf "space: %8d, wanted: %8d ", $space, dircfg('MinSpaceMby');
+    last if $space >= dircfg('MinSpaceMby');
 
     if (!@flights) {
 	open P, "-|", ontarget "ls -1 $pubdir" or die $!;
@@ -103,7 +108,7 @@ for (;;) {
 
     printf "(age %dd) ", $age / 86400;
 
-    die "age $age" if $age < $c{LogsMinExpireAge};
+    die "age $age" if $age < dircfg('MinExpireAge');
 
     printf "...";
 
-- 
1.7.10.4

[OSSTEST PATCH 25/32] mg-execute-flight: New -P (publish) option

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-execute-flight |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/mg-execute-flight b/mg-execute-flight
index 287caa0..7f3d969 100755
--- a/mg-execute-flight
+++ b/mg-execute-flight
@@ -24,12 +24,15 @@ usage () { cat <<END
     -B<blessing>
     -E<email>
     -f<basisflight>
+    -P                [run cr-publish-flight-logs]
 
 END
 }
 
 set -e
 
+publish=false
+
 badusage () { echo >&2 "bad usage"; usage >&2; exit 126; }
 
 while [ $# != 0 ]; do
@@ -38,6 +41,7 @@ while [ $# != 0 ]; do
         -B?*)	blessing=${arg#-B}			;;
         -E?*)	email=${arg#-E}				;;
 	-f?*)	basisflightopt=--that-flight=${arg#-f}	;;
+	-P)	publish=true				;;
 	[0-9]*)	if [ "x$flight" != x ]; then badusage; fi
 		flight=$arg				;;
 	*)	badusage				;;
@@ -90,5 +94,9 @@ cat <tmp/$flight.transcript
 
 exec >&2
 
+if $publish; then
+	./cr-publish-flight-logs $flight >/dev/null
+fi
+
 /usr/sbin/sendmail -odi -oee -oi -t <tmp/$flight.email
 mv tmp/$flight.email tmp/$flight.email.sent
-- 
1.7.10.4

[OSSTEST PATCH 26/32] HostnameSortSwapWords: Make name order mangling configurable

[Ian Jackson]

We still default to having the mangling enabled.  Arguably this is
wrong I'm am minimising the number of things that will be wrong for
the existing Cambridge instance.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest.pm |    2 ++
 mg-hosts   |    5 +++--
 ms-planner |    4 +++-
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 9fbadbd..5854257 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -73,6 +73,8 @@ our %c = qw(
     Baud  115200
 
     DebianNonfreeFirmware firmware-bnx2
+
+    HostnameSortSwapWords 1
 );
 
 $c{$_}='' foreach qw(
diff --git a/mg-hosts b/mg-hosts
index 497139a..a911099 100755
--- a/mg-hosts
+++ b/mg-hosts
@@ -439,9 +439,10 @@ END
 
 sub cmd_showflags () {
     die if @ARGV;
+    my $sortkey = !$c{HostnameSortSwapWords} ? "hostname"
+	: "split_part(hostname, '-', 2), hostname";
     my $hostsq= $dbh_tests->prepare(<<END);
-        SELECT hostname FROM hostflags ORDER BY
-            split_part(hostname, '-', 2), hostname
+        SELECT hostname FROM hostflags ORDER BY $sortkey
 END
     my $findq= $dbh_tests->prepare(<<END);
         SELECT * FROM hostflags ORDER BY hostflag
diff --git a/ms-planner b/ms-planner
index f045bbf..1dd01a9 100755
--- a/ms-planner
+++ b/ms-planner
@@ -563,7 +563,9 @@ sub cmd_show_html () {
 	} @{ $plan->{Events}{$reso} } ];
     }
 
-    my $colheadsortkey= sub { ($_[0] =~ m/-/ ? "$' $`" : "").$_[0]; };
+    my $colheadsortkey=
+	!$c{HostnameSortSwapWords} ? sub { $_[0]; }
+        : sub { ($_[0] =~ m/-/ ? "$' $`" : "").$_[0]; };
     my @colheads = sort {
         $colheadsortkey->($a) cmp $colheadsortkey->($b)
     } keys %{ $plan->{Events} };
-- 
1.7.10.4

[OSSTEST PATCH 27/32] ts-kernel-build: Document a hacky way to get CONFIG_... options

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-kernel-build |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/ts-kernel-build b/ts-kernel-build
index b2f19cf..93903d9 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -52,6 +52,19 @@ END
     }
 }
 
+# Some runes for finding out what might be missing
+#
+# On working boot (eg Debian kernel eg before ts-xen-install):
+#   lsmod | perl -ne 'm/^\w+/ or die; push @e, "+= $&\\.o" if $& ne 
+#         "Module"; END { print "git-grep '\''"; print join "\\|", @e; 
+#         print "'\'' | perl -ne '\'' print \"# \",\$&,\" is not set\\n\" 
+#         if m/\\bCONFIG_[0-9A-Z_]+\\b/ '\''\n" }'
+# This prints a big git-grep | perl rune
+# Run said rune in a clone of linux.git
+# Output of git-grep |perl rune is list of "# CONFIG_... is not set"
+# Copy to a file on relevant host, and grep the osstest kernel's config
+#   grep -f t /boot/config-3.14.34+ |less
+
 our $config_hardware= <<END;
 setopt CONFIG_DM_CRYPT m
 setopt CONFIG_DM_SNAPSHOT m
-- 
1.7.10.4

[OSSTEST PATCH 28/32] ts-kernel-build: enable CONFIG_SCSI_ISCI

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-kernel-build |    1 +
 1 file changed, 1 insertion(+)

diff --git a/ts-kernel-build b/ts-kernel-build
index 93903d9..4cbed44 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -101,6 +101,7 @@ setopt CONFIG_SCSI_AACRAID m
 setopt CONFIG_SCSI_AIC7XXX m
 setopt CONFIG_SCSI_AIC79XX m
 setopt CONFIG_SCSI_AIC94XX m
+setopt CONFIG_SCSI_ISCI m
 setopt CONFIG_SCSI_MVSAS m
 setopt CONFIG_SCSI_MVSAS_DEBUG=y
 setopt CONFIG_SCSI_DPT_I2O m
-- 
1.7.10.4

[OSSTEST PATCH 29/32] ts-kernel-build: enable CONFIG_SCSI_HPSA

[Ian Jackson]

---
 ts-kernel-build |    1 +
 1 file changed, 1 insertion(+)

diff --git a/ts-kernel-build b/ts-kernel-build
index 4cbed44..8966336 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -159,6 +159,7 @@ setopt CONFIG_SCSI_NSP32 m
 setopt CONFIG_SCSI_PMCRAID m
 setopt CONFIG_SCSI_SRP m
 setopt CONFIG_SCSI_BFA_FC n
+setopt CONFIG_SCSI_HPSA n
 
 setopt CONFIG_MEGARAID_NEWGEN y
 setopt CONFIG_MEGARAID_MM m
-- 
1.7.10.4

[OSSTEST PATCH 30/32] config defaults: Do not use woking for dhcp leases

[Ian Jackson]

This will probably break the Cambridge instance, where it will need to
be overridden.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest.pm |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest.pm b/Osstest.pm
index 5854257..449f312 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -101,7 +101,7 @@ sub readglobalconfig () {
     return if $readglobalconfig_done;
     $readglobalconfig_done=1;
 
-    $c{HostProp_DhcpWatchMethod} = 'leases dhcp3 woking.cam.xci-test.com:5556';
+    $c{HostProp_DhcpWatchMethod} = 'leases dhcp3 infra:5556';
     $c{AuthorizedKeysFiles} = '';
     $c{AuthorizedKeysAppend} = '';
 
-- 
1.7.10.4

[OSSTEST PATCH 31/32] Branches: Rename "incoming" branch to "production"

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 README.dev           |    2 +-
 ap-fetch-version-old |    2 +-
 ap-push              |    2 +-
 cr-for-branches      |    4 ++--
 4 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/README.dev b/README.dev
index aae4f17..1987b73 100644
--- a/README.dev
+++ b/README.dev
@@ -113,7 +113,7 @@ Creating a new branch
   to production pretest branch), rewind so that osstest can fast
   forward it to the actual production version.  Ie in bisects and
   branches for-$branch.git,
-     git fetch origin incoming
+     git fetch origin production
      git reset --hard FETCH_HEAD
 
 $ Add to BRANCHES setting in cr-for-branches
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 7b7f50e..0b4739b 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -99,7 +99,7 @@ ovmf)
 	;;
 osstest)
 	if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
-	    git fetch -f $HOME/testing.git incoming:ap-fetch
+	    git fetch -f $HOME/testing.git production:ap-fetch
             git rev-parse ap-fetch^0
 	else
 	    git rev-parse HEAD^0
diff --git a/ap-push b/ap-push
index 01089f3..c141464 100755
--- a/ap-push
+++ b/ap-push
@@ -98,7 +98,7 @@ ovmf)
 	git push $TREE_OVMF $revision:refs/heads/xen-tested-master
 	;;
 osstest)
-	git push $HOME/testing.git $revision:incoming
+	git push $HOME/testing.git $revision:production
 	git push $XENBITS:/home/xen/git/osstest.git $revision:master
 	;;
 *)
diff --git a/cr-for-branches b/cr-for-branches
index cbd1c74..7a93a62 100755
--- a/cr-for-branches
+++ b/cr-for-branches
@@ -43,7 +43,7 @@ with-lock-ex $fetchwlem data-tree-lock bash -ec '
 	exec >>$LOGFILE
 	date
         printf "%s\n" "$FOR_LOGFILE"
-	git pull . incoming:master 2>&1 ||:
+	git pull . production:master 2>&1 ||:
 	git checkout HEAD
 '
 
@@ -94,7 +94,7 @@ for branch in $BRANCHES; do
 		find tmp -type f -mtime +60 -print0 | xargs -0r rm -f --
 
 		git fetch $OSSTEST_TEST_PULLFROM master:incoming 2>&1 ||:
-		git fetch $OSSTEST_TEST_PULLFROM incoming:incoming 2>&1 ||:
+		git fetch $OSSTEST_TEST_PULLFROM production:incoming 2>&1 ||:
 		git pull --no-commit . incoming:master 2>&1 ||:
 		git checkout HEAD
 
-- 
1.7.10.4

[OSSTEST PATCH 32/32] production-config: Configuration for new production colo

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 production-config |   52 +++++++++++++++++++++++++++++++++-------------------
 1 file changed, 33 insertions(+), 19 deletions(-)

diff --git a/production-config b/production-config
index 5fbca50..5e8a477 100644
--- a/production-config
+++ b/production-config
@@ -19,31 +19,43 @@
 JobDB Executive
 HostDB Executive
 
-DnsDomain cam.xci-test.com
-NetNameservers 10.80.248.2 10.80.16.28 10.80.16.67
+HostDB_Executive_NoConfigDB 1
+Include /root/osstest-hosts-config
 
-Stash /home/xc_osstest/logs
-Images /home/xc_osstest/images
-Logs /home/xc_osstest/logs
+DnsDomain test-lab.xenproject.org
+NetNameservers 172.16.144.4
 
-Results /home/xc_osstest/results
-PubBaseDir /home/xc_osstest
+OwnerDaemonHost db
+QueueDaemonHost osstest
 
-OverlayLocal /export/home/osstest/overlay-local
+ExecutiveDbnamePat dbname=osstestdb;host=db
 
-LogsMinSpaceMby= 10*1e3
+Stash /home/osstest/pub/logs
+Images /home/osstest/pub/images
+Logs /home/osstest/pub/logs
+
+Results /home/osstest/pub/results
+PubBaseDir /home/osstest/pub
+
+OverlayLocal /home/osstest/overlay-local
+
+LogsMinSpaceMby= 50*1e3
 LogsMinExpireAge= 86400*4
+LogsPublishMinSpaceMby= 10*1e3
+LogsPublishMinExpireAge= 86400*7
+
+TestHostKeypairPath /home/osstest/.ssh/id_rsa_osstest
 
-TestHostKeypairPath /export/home/osstest/.ssh/id_rsa_osstest
+GitCacheProxy git://cache:9419/
 
-GitCacheProxy git://drall.uk.xensource.com:9419/
+HttpProxy http://cache:3128/
 
-PubBaseUrl http://www.chiark.greenend.org.uk/~xensrcts
+PubBaseUrl http://logs.test-lab.xenproject.org/osstest
 ReportHtmlPubBaseUrl="$c{PubBaseUrl}/logs"
 ResultsHtmlPubBaseUrl="$c{PubBaseUrl}/results"
 
-Publish xensrcts@login.chiark.greenend.org.uk:/home/ian/work/xc_osstest
-GlobalLockDir /export/home/osstest/testing.git
+Publish osstest@www:/var/www/osstest
+GlobalLockDir /home/osstest/testing.git
 
 LogsPublish= "$c{Publish}/logs"
 ResultsPublish= "$c{Publish}/results"
@@ -61,11 +73,12 @@ END
 
 PlanRogueAllocationDuration= 86400*7
 
-TftpPath /home/xc_tftpboot/pxe/
+TftpPath /home/tftp/
 TftpPlayDir osstest/
 TftpTmpDir osstest/tmp/
 TftpPxeDir /
-TftpPxeTemplates %ipaddrhex%/pxelinux.cfg
+TftpPxeTemplates %name%/pxelinux.cfg
+TftpPxeTemplatesReal pxelinux.cfg/%ipaddrhex%
 
 TftpPxeGroup osstest
 TftpDiVersion 2015-01-10
@@ -75,14 +88,15 @@ XenUseUser osstest
 
 # We use the IP address because Citrix can't manage reliable nameservice
 #DebianMirrorHost debian.uk.xensource.com
-DebianMirrorHost 10.80.16.196
+#DebianMirrorHost 10.80.16.196
+DebianMirrorProxy http://cache:3142/
 
-HostProp_NtpServer ntp.uk.xensource.com
+HostProp_NtpServer infra.test-lab.xenproject.org
 
 DebianPreseed= <<'END'
 END
 
-HostProp_Serial sympathy woking
+HostProp_Serial sympathy serial0
 HostProp_GenEtherPrefixBase 5a:36:0e:00
 #                                      :00:01 guest number in job appended
 #                                    ^^ xor'd with low 8 bits of flight
-- 
1.7.10.4

Re: [OSSTEST PATCH 29/32] ts-kernel-build: enable CONFIG_SCSI_HPSA

[Ian Campbell]

On Tue, 2015-03-24 at 19:02 +0000, Ian Jackson wrote:
> ---
>  ts-kernel-build |    1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/ts-kernel-build b/ts-kernel-build
> index 4cbed44..8966336 100755
> --- a/ts-kernel-build
> +++ b/ts-kernel-build
> @@ -159,6 +159,7 @@ setopt CONFIG_SCSI_NSP32 m
>  setopt CONFIG_SCSI_PMCRAID m
>  setopt CONFIG_SCSI_SRP m
>  setopt CONFIG_SCSI_BFA_FC n
> +setopt CONFIG_SCSI_HPSA n

This is disabling the option, not enabling it per the commit message.

>  
>  setopt CONFIG_MEGARAID_NEWGEN y
>  setopt CONFIG_MEGARAID_MM m

Re: [OSSTEST PATCH 00/32] Changes to support the new colo

[Ian Campbell]

On Tue, 2015-03-24 at 19:02 +0000, Ian Jackson wrote:
> This is the current patchset I am using in the new colo, tidied up.
> 
> I think 01-29 are in principle suitable for pushing to the current
> live instance in Cambridge right away.

I glanced through and apart from an inconsistency in #29 I agree.

> 30-32 are changes to the config which will go wrong on the Cambridge
> instance.
> 
> Thanks,
> Ian.

Re: [OSSTEST PATCH 29/32] ts-kernel-build: enable CONFIG_SCSI_HPSA

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 29/32] ts-kernel-build: enable CONFIG_SCSI_HPSA"):
> On Tue, 2015-03-24 at 19:02 +0000, Ian Jackson wrote:
> > +setopt CONFIG_SCSI_HPSA n
> 
> This is disabling the option, not enabling it per the commit message.

Yes.  I noticed that when I executed it :-/.  I intended to enable it
(setting it to `m').

Ian.