[OSSTEST PATCH 0/9] Fix NTP unreliability, and other fixes

[OSSTEST PATCH 0/9] Fix NTP unreliability, and other fixes

[Ian Jackson]

We are having persistent spurious "xen-boot" failures.  Cursory
investigation revealed that this was due to problems with NTP startup,
so we put it down to infrastructure problems.

However, it turns out that the cause is that we are using the
Debian-default NTP servers from pool.ntp.org.  We might expect some of
these servers don't work some of the time.

The underlying cause is a bug in Debian: the NTP servers specified via
preseeding are used during the installer environment to set the clock,
but the information is not used to configure the actual ntp server
package in the install.  I have reported this as a bug in Debian but
we need to work around it by editing ntp.conf after installation.

There are also two unrelated fixes in this series.

One of the changes (`Provide get_target_property') is likely to be
useful for nested virt.

In the absence of objections I am going to push this series soon.

[OSSTEST PATCH 1/9] cs-adjust-flight: Add missing `use Data::Dumper'

[Ian Jackson]

Dumper is used in some of the error reporting so we need to `use' it.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 cs-adjust-flight |    1 +
 1 file changed, 1 insertion(+)

diff --git a/cs-adjust-flight b/cs-adjust-flight
index d6cab1a..9bd8757 100755
--- a/cs-adjust-flight
+++ b/cs-adjust-flight
@@ -40,6 +40,7 @@
 use strict qw(vars);
 use DBI;
 use Osstest;
+use Data::Dumper;
 
 csreadconfig();
 
-- 
1.7.10.4

[OSSTEST PATCH 2/9] sg-report-job-history: Increase default history depth to 500

[Ian Jackson]

We are putting through a lot more jobs and branches.  Older history is
still interesting.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 sg-report-job-history |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sg-report-job-history b/sg-report-job-history
index 3e4b0ef..e7052a3 100755
--- a/sg-report-job-history
+++ b/sg-report-job-history
@@ -27,7 +27,7 @@ use HTML::Entities;
 use Osstest::TestSupport;
 
 our (@blessings,@branches);
-our $limit= 100;
+our $limit= 500;
 our $htmlout;
 
 open DEBUG, ">/dev/null";
-- 
1.7.10.4

[OSSTEST PATCH 3/9] config: Tiny fixes

[Ian Jackson]

* Default $c{DebianPreseed} to '' (previously, if it wasn't specified,
  there would be undefined variable warnings).

* Cope with empty <<END-notated config items in the configuration file
  parser.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest.pm |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Osstest.pm b/Osstest.pm
index fc9698b..6d7c083 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -74,6 +74,8 @@ our %c = qw(
     DebianNonfreeFirmware firmware-bnx2
 );
 
+$c{DebianPreseed} = '';
+
 #---------- general setup and config reading ----------
 
 sub getmethod {
@@ -120,7 +122,7 @@ sub readglobalconfig () {
 			$val .= $_;
 		    }
 		    die $! unless length $_;
-		    die unless $val =~ m/\n$/;
+		    die unless !length $val || $val =~ m/\n$/;
 		    if ($qu eq '') {
 			my $reconstruct =
 			    "\$val = <<${qu}${delim}${qu}; 1;\n".
-- 
1.7.10.4

[OSSTEST PATCH 4/9] TestSupport: Provide get_target_property

[Ian Jackson]

This looks in the supplied $ho, but if that's a $gho (ie it has a
$gho->{Host}) it also looks in its host.

This is going to be useful for a fair variety of host-specific or
infrastructure-determined properties.

It seems to me that whether a property ought to be looked up in the
host if not found in a particular guest depends mostly on the
property, and not on the way the guest is configured.  The easiest way
to represent that in the osstest codebase is probably to call
get_target_property instead of get_host_property in the appropriate
places.

The use of recursion will make this look through a series of nested
hosts if we have nested virtualisation going on.  Indeed, nested
virtualisation may benefit from replacement of get_host_property by
get_target_property in a number of cases.  (At the time of writing
there is no nested virt in osstest mainline, but it's on the way.)

Currently there are no callers of get_target_property.  One will
appear shortly.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Robert Hu <robert.hu@intel.com>
CC: LongtaoX Pang <longtaox.pang@intel.com>
---
 Osstest/TestSupport.pm |   13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 8aed285..8754e22 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -66,7 +66,7 @@ BEGIN {
                       contents_make_cpio file_simple_write_contents
 
                       selecthost get_hostflags get_host_property
-                      get_host_native_linux_console
+                      get_target_property get_host_native_linux_console
                       power_state power_cycle power_cycle_sleep
                       serial_fetch_logs
                       propname_massage
@@ -906,6 +906,17 @@ sub get_host_property ($$;$) {
     return defined($val) ? $val : $defval;
 }
 
+sub get_target_property ($$;$);
+sub get_target_property ($$;$) {
+    my ($ho, $prop, $defval) = @_;
+    # $ho may be a guest; if so we look for the property in the
+    # guest and failing that the same property in the host.
+    return
+	get_host_property($ho, $prop) //
+	($ho->{Host} && get_target_property($ho->{Host}, $prop)) //
+	$defval;
+}
+
 sub get_host_native_linux_console ($) {
     my ($ho) = @_;
 
-- 
1.7.10.4

[OSSTEST PATCH 5/9] Debian.pm: Pass $ho/$gho to preseed_base

[Ian Jackson]

This is going to be useful soon.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/Debian.pm     |    6 +++---
 ts-debian-hvm-install |    2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index e3e1c90..a044f8f 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -503,8 +503,8 @@ sub di_installcmdline_core ($$;@) {
     return @cl;
 }
 
-sub preseed_base ($$;@) {
-    my ($suite,$extra_packages,%xopts) = @_;
+sub preseed_base ($$$;@) {
+    my ($ho,$suite,$extra_packages,%xopts) = @_;
 
     return <<"END";
 d-i mirror/suite string $suite
@@ -813,7 +813,7 @@ END
 
     my $extra_packages = join(",",@extra_packages);
 
-    my $preseed_file= preseed_base($suite,$extra_packages,%xopts);
+    my $preseed_file= preseed_base($ho,$suite,$extra_packages,%xopts);
 
     $preseed_file .= (<<END);
 d-i partman-auto/method string lvm
diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 449b96c..cfd5144 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -43,7 +43,7 @@ our $gho;
 
 sub preseed () {
 
-    my $preseed_file = preseed_base('wheezy','',());
+    my $preseed_file = preseed_base($gho,'wheezy','',());
     my $authkeys = join('\\n', split(/\n/, authorized_keys()));
 
     $preseed_file .= (<<END);
-- 
1.7.10.4

[OSSTEST PATCH 6/9] Debian.pm: Slightly refactor preseed_base

[Ian Jackson]

Arrange for preseed_base to accumulate its result in a variable.  This
is going to make it easier to add entries which are determined
programatically.

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/Debian.pm |   11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index a044f8f..315d25c 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -506,7 +506,7 @@ sub di_installcmdline_core ($$;@) {
 sub preseed_base ($$$;@) {
     my ($ho,$suite,$extra_packages,%xopts) = @_;
 
-    return <<"END";
+    my $preseed = <<"END";
 d-i mirror/suite string $suite
 
 d-i debian-installer/locale string en_GB
@@ -572,10 +572,15 @@ d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bi
 
 $xopts{ExtraPreseed}
 
-### END OF DEBIAN PRESEED BASE
+END
+
+    $preseed .= <<"END";
 
+### END OF DEBIAN PRESEED BASE
 END
-}          
+
+    return $preseed;
+}
 
 sub preseed_create ($$;@) {
     my ($ho, $sfx, %xopts) = @_;
-- 
1.7.10.4

[OSSTEST PATCH 7/9] NTP servers: Introduce new NtpServer host property

[Ian Jackson]

Allow the specification of NTP servers in a dedicated NtpServer host
property, which we honour in preseed_create.

Change the documentation to recommend this, rather than use of
ad-hoc text in DebianPreseed.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index a044f8f..cc075a4 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -506,7 +506,7 @@ sub di_installcmdline_core ($$;@) {
 sub preseed_base ($$$;@) {
     my ($ho,$suite,$extra_packages,%xopts) = @_;

-    return <<"END";
+    my $preseed = <<"END";
 d-i mirror/suite string $suite

 d-i debian-installer/locale string en_GB
@@ -572,10 +572,22 @@ d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bi

 $xopts{ExtraPreseed}

-### END OF DEBIAN PRESEED BASE
+END
+
+    my $ntpserver = get_target_property($ho,'NtpServer');
+use Data::Dumper;
+print STDERR "PRESEED NTP ", Dumper($ntpserver);
+    $preseed .= <<"END" if $ntpserver;
+d-i clock-setup/ntp-server string $ntpserver
+END

+    $preseed .= <<"END";
+
+### END OF DEBIAN PRESEED BASE
 END
-}
+
+    return $preseed;
+}

 sub preseed_create ($$;@) {
     my ($ho, $sfx, %xopts) = @_;
diff --git a/README b/README
index 6e63e97..0a0242c 100644
--- a/README
+++ b/README
@@ -334,6 +334,11 @@ HostProp_<testbox>_TftpScope
    Defines the Tftp scope (i.e. subnet) where this host resides. See
    "TftpFoo_<scope> and TftpFoo" below.

+HostProp_<testbox>_NtpServer
+   NTP server to use.  You should probably have your own local
+   NTP server for production use; the default is to use the operating
+   system's default (normally, Debian's pool.ntp.org servers).
+
 HostFlags_<testbox>
    Defines a set of flags for the host. Flags is a list separated by
    whitespace, comma or semi-colon. A flag can be unset by prepending
@@ -357,9 +362,7 @@ HostGroupFlags_<group>
    merged with the host specific flags. Only used in standalone mode.

 DebianPreseed
-   Text to add to the debian-installer preseed file.  Optional
-   but you will need to set some NTP servers here if your firewall
-   doesn't permit NTP to Debian's pool.ntp.org servers.
+   Text to add to the debian-installer preseed file.  Optional.

 ========================================

diff --git a/production-config b/production-config
index 515bd98..5fbca50 100644
--- a/production-config
+++ b/production-config
@@ -77,8 +77,9 @@ XenUseUser osstest
 #DebianMirrorHost debian.uk.xensource.com
 DebianMirrorHost 10.80.16.196

+HostProp_NtpServer ntp.uk.xensource.com
+
 DebianPreseed= <<'END'
-d-i clock-setup/ntp-server string ntp.uk.xensource.com
 END

 HostProp_Serial sympathy woking
---
 Osstest/Debian.pm |    5 +++++
 README            |    9 ++++++---
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 315d25c..7633b51 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -574,6 +574,11 @@ $xopts{ExtraPreseed}
 
 END
 
+    my $ntpserver = get_target_property($ho,'NtpServer');
+    $preseed .= <<"END" if $ntpserver;
+d-i clock-setup/ntp-server string $ntpserver
+END
+
     $preseed .= <<"END";
 
 ### END OF DEBIAN PRESEED BASE
diff --git a/README b/README
index 6e63e97..0a0242c 100644
--- a/README
+++ b/README
@@ -334,6 +334,11 @@ HostProp_<testbox>_TftpScope
    Defines the Tftp scope (i.e. subnet) where this host resides. See
    "TftpFoo_<scope> and TftpFoo" below.
 
+HostProp_<testbox>_NtpServer
+   NTP server to use.  You should probably have your own local
+   NTP server for production use; the default is to use the operating
+   system's default (normally, Debian's pool.ntp.org servers).
+
 HostFlags_<testbox>
    Defines a set of flags for the host. Flags is a list separated by
    whitespace, comma or semi-colon. A flag can be unset by prepending
@@ -357,9 +362,7 @@ HostGroupFlags_<group>
    merged with the host specific flags. Only used in standalone mode.
 
 DebianPreseed
-   Text to add to the debian-installer preseed file.  Optional
-   but you will need to set some NTP servers here if your firewall
-   doesn't permit NTP to Debian's pool.ntp.org servers.
+   Text to add to the debian-installer preseed file.  Optional.
 
 ========================================
 
-- 
1.7.10.4

[OSSTEST PATCH 8/9] NTP servers: Use NtpServer host property

[Ian Jackson]

Move the specification of NTP servers in the production-config from
the ad-hoc entry in DebianPreseed to a dedicated NtpServer host
property.

This results in no overall functional change, except that the order of
elements in the preseed file is slightly different.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index a044f8f..cc075a4 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -506,7 +506,7 @@ sub di_installcmdline_core ($$;@) {
 sub preseed_base ($$$;@) {
     my ($ho,$suite,$extra_packages,%xopts) = @_;

-    return <<"END";
+    my $preseed = <<"END";
 d-i mirror/suite string $suite

 d-i debian-installer/locale string en_GB
@@ -572,10 +572,22 @@ d-i pkgsel/include string openssh-server, ntp, ntpdate, ethtool, chiark-utils-bi

 $xopts{ExtraPreseed}

-### END OF DEBIAN PRESEED BASE
+END
+
+    my $ntpserver = get_target_property($ho,'NtpServer');
+use Data::Dumper;
+print STDERR "PRESEED NTP ", Dumper($ntpserver);
+    $preseed .= <<"END" if $ntpserver;
+d-i clock-setup/ntp-server string $ntpserver
+END

+    $preseed .= <<"END";
+
+### END OF DEBIAN PRESEED BASE
 END
-}
+
+    return $preseed;
+}

 sub preseed_create ($$;@) {
     my ($ho, $sfx, %xopts) = @_;
diff --git a/README b/README
index 6e63e97..0a0242c 100644
--- a/README
+++ b/README
@@ -334,6 +334,11 @@ HostProp_<testbox>_TftpScope
    Defines the Tftp scope (i.e. subnet) where this host resides. See
    "TftpFoo_<scope> and TftpFoo" below.

+HostProp_<testbox>_NtpServer
+   NTP server to use.  You should probably have your own local
+   NTP server for production use; the default is to use the operating
+   system's default (normally, Debian's pool.ntp.org servers).
+
 HostFlags_<testbox>
    Defines a set of flags for the host. Flags is a list separated by
    whitespace, comma or semi-colon. A flag can be unset by prepending
@@ -357,9 +362,7 @@ HostGroupFlags_<group>
    merged with the host specific flags. Only used in standalone mode.

 DebianPreseed
-   Text to add to the debian-installer preseed file.  Optional
-   but you will need to set some NTP servers here if your firewall
-   doesn't permit NTP to Debian's pool.ntp.org servers.
+   Text to add to the debian-installer preseed file.  Optional.

 ========================================

diff --git a/production-config b/production-config
index 515bd98..5fbca50 100644
--- a/production-config
+++ b/production-config
@@ -77,8 +77,9 @@ XenUseUser osstest
 #DebianMirrorHost debian.uk.xensource.com
 DebianMirrorHost 10.80.16.196

+HostProp_NtpServer ntp.uk.xensource.com
+
 DebianPreseed= <<'END'
-d-i clock-setup/ntp-server string ntp.uk.xensource.com
 END

 HostProp_Serial sympathy woking
---
 production-config |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/production-config b/production-config
index 515bd98..5fbca50 100644
--- a/production-config
+++ b/production-config
@@ -77,8 +77,9 @@ XenUseUser osstest
 #DebianMirrorHost debian.uk.xensource.com
 DebianMirrorHost 10.80.16.196
 
+HostProp_NtpServer ntp.uk.xensource.com
+
 DebianPreseed= <<'END'
-d-i clock-setup/ntp-server string ntp.uk.xensource.com
 END
 
 HostProp_Serial sympathy woking
-- 
1.7.10.4

[OSSTEST PATCH 9/9] NTP servers: Work around Debian's failure to honour preseed

[Ian Jackson]

Setting clock-setup/ntp-server is not sufficient: it only takes effect
in the installer (!)

I have reported this as Debian #778564.  In the meantime we should
work around it for current releases (including jessie, which is
frozen).

For later releases, the new ntp.conf editing code arranges to bomb out
if we have an NTP server configured and find it hasn't been honoured
during the install.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 ts-host-install |   20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/ts-host-install b/ts-host-install
index ae1d228..9656079 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -119,6 +119,26 @@ END
 
     target_install_packages($ho, qw(ed));
 
+    my $ntpserver = get_target_property($ho, 'NtpServer');
+    if ($ntpserver) {
+	target_editfile_root($ho, '/etc/ntp.conf', sub {
+	    my $done= 0;
+	    while (<EI>) {
+		if (m/^server\s/) {
+		    if ($ho->{Suite} =~ m/lenny|squeeze|wheezy|jessie/) {
+			$_= $done ? "" : "server $ntpserver\n";
+		    } else {
+			m/^server \Q$ntpserver\E\s/ or
+			    die "NTP server not honoured, Debian #778564 ";
+		    }
+		    $done= 1;
+		}
+		print EO or die $!;
+	    }
+	    $done or die;
+	});
+    }
+
     target_cmd_root($ho, "update-rc.d osstest-confirm-booted start 99 2 .");
 
     logm('OK: install completed');
-- 
1.7.10.4