[OSSTEST PATCH 0/3] stretch fallout: run 32-bit builds on 64-bit Linux

[OSSTEST PATCH 0/3] stretch fallout: run 32-bit builds on 64-bit Linux

[Ian Jackson]

I am going to push this to osstest pretest right away.  This problem
is currently blocking osstest's own self-push-gate and preventing many
other fixes from making it in...

Ian Jackson (3):
  ts-syslog-server: --no-stdin option
  sg-run-job, ts-host-install: New --build option
  builds: Run i386 builds on amd64 kernels

 Osstest/Debian.pm | 35 +++++++++++++++++++++++++++++++++++
 sg-run-job        |  2 +-
 ts-host-install   |  9 +++++++++
 ts-syslog-server  | 10 +++++++++-
 4 files changed, 54 insertions(+), 2 deletions(-)

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH 0/3] stretch fallout: run 32-bit builds on 64-bit Linux

[Ian Jackson]

I am going to push this to osstest pretest right away.  This problem
is currently blocking osstest's own self-push-gate and preventing many
other fixes from making it in...

Ian Jackson (3):
  ts-syslog-server: --no-stdin option
  sg-run-job, ts-host-install: New --build option
  builds: Run i386 builds on amd64 kernels

 Osstest/Debian.pm | 35 +++++++++++++++++++++++++++++++++++
 sg-run-job        |  2 +-
 ts-host-install   |  9 +++++++++
 ts-syslog-server  | 10 +++++++++-
 4 files changed, 54 insertions(+), 2 deletions(-)

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 1/3] ts-syslog-server: --no-stdin option

[Ian Jackson]

Useful when running on a tty interactively.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-syslog-server | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/ts-syslog-server b/ts-syslog-server
index 1234bb7d..06a07adf 100755
--- a/ts-syslog-server
+++ b/ts-syslog-server
@@ -28,6 +28,13 @@ use Socket qw(PF_INET SOCK_DGRAM INADDR_ANY
 
 tsreadconfig();
 
+our $use_stdin = 1;
+
+if (@ARGV && $ARGV[0] eq '--no-stdin') {
+    $use_stdin = 0;
+    shift @ARGV;
+}
+
 die if @ARGV && $ARGV[0] =~ m/^-/;
 
 logm("starting syslog server ...");
@@ -42,7 +49,8 @@ $myaddr = inet_ntoa($myaddr);
 
 store_runvar('syslog_server',"$myaddr:$port");
 
-my @fhs = qw(S STDIN);
+my @fhs = qw(S);
+push @fhs, qw(STDIN) if $use_stdin;
 
 foreach my $fh (@fhs) {
     my $fl = fcntl($fh, F_GETFL, 0) // die $!;
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH 1/3] ts-syslog-server: --no-stdin option

[Ian Jackson]

Useful when running on a tty interactively.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-syslog-server | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/ts-syslog-server b/ts-syslog-server
index 1234bb7d..06a07adf 100755
--- a/ts-syslog-server
+++ b/ts-syslog-server
@@ -28,6 +28,13 @@ use Socket qw(PF_INET SOCK_DGRAM INADDR_ANY
 
 tsreadconfig();
 
+our $use_stdin = 1;
+
+if (@ARGV && $ARGV[0] eq '--no-stdin') {
+    $use_stdin = 0;
+    shift @ARGV;
+}
+
 die if @ARGV && $ARGV[0] =~ m/^-/;
 
 logm("starting syslog server ...");
@@ -42,7 +49,8 @@ $myaddr = inet_ntoa($myaddr);
 
 store_runvar('syslog_server',"$myaddr:$port");
 
-my @fhs = qw(S STDIN);
+my @fhs = qw(S);
+push @fhs, qw(STDIN) if $use_stdin;
 
 foreach my $fh (@fhs) {
     my $fl = fcntl($fh, F_GETFL, 0) // die $!;
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 2/3] sg-run-job, ts-host-install: New --build option

[Ian Jackson]

Used to specify that the host will be used for building.  Currently
has no effect.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-run-job      | 2 +-
 ts-host-install | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/sg-run-job b/sg-run-job
index 56b6384a..7c58d4ba 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -734,7 +734,7 @@ proc allocate-build-host {ostype} {
 }
 proc prepare-build-host-linux {} {
     global jobinfo
-    run-ts broken host-install(*) ts-host-install-twice
+    run-ts broken host-install(*) ts-host-install-twice + --build
     run-ts . host-build-prep ts-xen-build-prep
 }
 
diff --git a/ts-host-install b/ts-host-install
index 4213c3a2..7094559f 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -41,6 +41,7 @@ while (@ARGV and $ARGV[0] =~ m/^-/) {
 	$poweron_test_only= 1;
     } elsif  (m/^--rescue$/) {
 	$xopts{RescueMode}= 1;
+    } elsif  (m/^--build$/) {
     } else {
 	die "$_ $!";
     }
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH 2/3] sg-run-job, ts-host-install: New --build option

[Ian Jackson]

Used to specify that the host will be used for building.  Currently
has no effect.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 sg-run-job      | 2 +-
 ts-host-install | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/sg-run-job b/sg-run-job
index 56b6384a..7c58d4ba 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -734,7 +734,7 @@ proc allocate-build-host {ostype} {
 }
 proc prepare-build-host-linux {} {
     global jobinfo
-    run-ts broken host-install(*) ts-host-install-twice
+    run-ts broken host-install(*) ts-host-install-twice + --build
     run-ts . host-build-prep ts-xen-build-prep
 }
 
diff --git a/ts-host-install b/ts-host-install
index 4213c3a2..7094559f 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -41,6 +41,7 @@ while (@ARGV and $ARGV[0] =~ m/^-/) {
 	$poweron_test_only= 1;
     } elsif  (m/^--rescue$/) {
 	$xopts{RescueMode}= 1;
+    } elsif  (m/^--build$/) {
     } else {
 	die "$_ $!";
     }
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 3/3] builds: Run i386 builds on amd64 kernels

[Ian Jackson]

Most hardware that supports i386 supports amd64 too.  When doing
builds we do need the right userland, but we don't actually care what
the kernel is doing.  With Linux 32-on-64 is good for that.

Especially, there is a kernel regression (evident in the Debian
stretch kernel, but not present in jessie's) where 32-bit Linux
mismanages the memory on hosts with moderately large amounts of
RAM (ie, significantly more RAM than can be addressed at once),
resulting in what amounts to a near-stall of the paging system.  Since
the paging system is used for filesystem writes too, the effect is
that commands run by builds can take totally unreasonable amounts of
time.  Ie, this version of Linux is broken when i386 PAE is needed.

In practice this is causing significant trouble in the Xen Project CI.

This kernel bug probably won't affect our test jobs because
(i) we use our own kernels, so we would probably detect this
regression when switching kernel branches etc. (ii) test jobs
run with a dom0_mem setting which avoids the preconditions for the
particular bug.

CC: Juergen Gross <jgross@suse.com>
CC: Boris Ostrovsky <boris.ostrovsky@oracle.com>
CC: Wei Liu <wei.liu2@citrix.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Paul Durrant <paul.durrant@citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 35 +++++++++++++++++++++++++++++++++++
 ts-host-install   |  8 ++++++++
 2 files changed, 43 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 600f18b1..0f10a9c3 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -1334,6 +1334,41 @@ END
 	preseed_hook_command($ho, 'late_command', $sfx, $cmds);
     });
 
+    my $kern_arch = $xopts{LinuxKernArchMap}{$r{arch}};
+    if ($kern_arch &&
+	$ho->{Flags}{"arch-$r{arch}"} && !$ho->{Flags}{"arch-$kern_arch"}) {
+	# We check the flag for $r{arch} because for various reasons
+	# it may not be set.  For example, in standalone mode it is
+	# currently not normally necessary to set any arch hostflags.
+	# In that situation we bet that the kern arch can be used;
+	# this is a good bet because currently this is used for
+	# running amd64 kernels with i386 userland and hosts that can
+	# do only i386 are fairly rare, especially server hosts.
+	# If this causes a problem, setting the i386 host flag will
+	# (somewhat counterintuitively) cause this check to miss
+	# the amd64 host flag
+	logm("NOT using $kern_arch kernel (flags say not supported by host)");
+	$kern_arch = undef;
+    }
+    if ($kern_arch) {
+	logm("Using $kern_arch kernel for $r{arch} userland");
+	preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+in-target sh -xec '
+    # the $r{arch} kernels end up being the default so shuffle them
+    # where update-grub will not find them
+    for f in /boot/vmlinu*; do
+        dpkg-divert --rename --divert "/boot/~disable~\${f##*/}" "\$f"
+    done
+    dpkg --add-architecture $kern_arch
+    apt-get update
+    apt-get install -f -y linux-image-$kern_arch
+    update-grub
+'
+END
+    }
+
     if ( $ho->{Flags}{'need-uboot-bootscr'} ) {
 	my @bootargs = uboot_common_kernel_bootargs($ho);
 
diff --git a/ts-host-install b/ts-host-install
index 7094559f..35601a42 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -31,6 +31,7 @@ my $debconf_priority;
 my $poweron_test_only;
 
 our %xopts;
+our %linux_kern_arch_map;
 
 while (@ARGV and $ARGV[0] =~ m/^-/) {
     $_= shift @ARGV;
@@ -42,6 +43,12 @@ while (@ARGV and $ARGV[0] =~ m/^-/) {
     } elsif  (m/^--rescue$/) {
 	$xopts{RescueMode}= 1;
     } elsif  (m/^--build$/) {
+	# When installing to do builds rather than tests, we can use an
+	# amd64 kernel on i386 hosts, and this is generally faster.
+	# Also, it avoids a kernel regression in Debian stretch -
+	# a regression which Linux upstream do not consider a bug.  Eg
+	#  https://lkml.org/lkml/2017/8/2/394
+	$linux_kern_arch_map{i386} = 'amd64';
     } else {
 	die "$_ $!";
     }
@@ -61,6 +68,7 @@ sub install () {
         ($ho, '',
          DiskDevice => $ho->{DiskDevice},
          Properties => $ho->{Properties},
+	 LinuxKernArchMap => \%linux_kern_arch_map,
          ExtraPreseed => <<END );
 d-i netcfg/get_hostname string $ho->{Name}
 d-i netcfg/get_ipaddress string $ho->{Ip}
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH 3/3] builds: Run i386 builds on amd64 kernels

[Ian Jackson]

Most hardware that supports i386 supports amd64 too.  When doing
builds we do need the right userland, but we don't actually care what
the kernel is doing.  With Linux 32-on-64 is good for that.

Especially, there is a kernel regression (evident in the Debian
stretch kernel, but not present in jessie's) where 32-bit Linux
mismanages the memory on hosts with moderately large amounts of
RAM (ie, significantly more RAM than can be addressed at once),
resulting in what amounts to a near-stall of the paging system.  Since
the paging system is used for filesystem writes too, the effect is
that commands run by builds can take totally unreasonable amounts of
time.  Ie, this version of Linux is broken when i386 PAE is needed.

In practice this is causing significant trouble in the Xen Project CI.

This kernel bug probably won't affect our test jobs because
(i) we use our own kernels, so we would probably detect this
regression when switching kernel branches etc. (ii) test jobs
run with a dom0_mem setting which avoids the preconditions for the
particular bug.

CC: Juergen Gross <jgross@suse.com>
CC: Boris Ostrovsky <boris.ostrovsky@oracle.com>
CC: Wei Liu <wei.liu2@citrix.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Paul Durrant <paul.durrant@citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm | 35 +++++++++++++++++++++++++++++++++++
 ts-host-install   |  8 ++++++++
 2 files changed, 43 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 600f18b1..0f10a9c3 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -1334,6 +1334,41 @@ END
 	preseed_hook_command($ho, 'late_command', $sfx, $cmds);
     });
 
+    my $kern_arch = $xopts{LinuxKernArchMap}{$r{arch}};
+    if ($kern_arch &&
+	$ho->{Flags}{"arch-$r{arch}"} && !$ho->{Flags}{"arch-$kern_arch"}) {
+	# We check the flag for $r{arch} because for various reasons
+	# it may not be set.  For example, in standalone mode it is
+	# currently not normally necessary to set any arch hostflags.
+	# In that situation we bet that the kern arch can be used;
+	# this is a good bet because currently this is used for
+	# running amd64 kernels with i386 userland and hosts that can
+	# do only i386 are fairly rare, especially server hosts.
+	# If this causes a problem, setting the i386 host flag will
+	# (somewhat counterintuitively) cause this check to miss
+	# the amd64 host flag
+	logm("NOT using $kern_arch kernel (flags say not supported by host)");
+	$kern_arch = undef;
+    }
+    if ($kern_arch) {
+	logm("Using $kern_arch kernel for $r{arch} userland");
+	preseed_hook_command($ho, 'late_command', $sfx, <<END);
+#!/bin/sh
+set -ex
+in-target sh -xec '
+    # the $r{arch} kernels end up being the default so shuffle them
+    # where update-grub will not find them
+    for f in /boot/vmlinu*; do
+        dpkg-divert --rename --divert "/boot/~disable~\${f##*/}" "\$f"
+    done
+    dpkg --add-architecture $kern_arch
+    apt-get update
+    apt-get install -f -y linux-image-$kern_arch
+    update-grub
+'
+END
+    }
+
     if ( $ho->{Flags}{'need-uboot-bootscr'} ) {
 	my @bootargs = uboot_common_kernel_bootargs($ho);
 
diff --git a/ts-host-install b/ts-host-install
index 7094559f..35601a42 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -31,6 +31,7 @@ my $debconf_priority;
 my $poweron_test_only;
 
 our %xopts;
+our %linux_kern_arch_map;
 
 while (@ARGV and $ARGV[0] =~ m/^-/) {
     $_= shift @ARGV;
@@ -42,6 +43,12 @@ while (@ARGV and $ARGV[0] =~ m/^-/) {
     } elsif  (m/^--rescue$/) {
 	$xopts{RescueMode}= 1;
     } elsif  (m/^--build$/) {
+	# When installing to do builds rather than tests, we can use an
+	# amd64 kernel on i386 hosts, and this is generally faster.
+	# Also, it avoids a kernel regression in Debian stretch -
+	# a regression which Linux upstream do not consider a bug.  Eg
+	#  https://lkml.org/lkml/2017/8/2/394
+	$linux_kern_arch_map{i386} = 'amd64';
     } else {
 	die "$_ $!";
     }
@@ -61,6 +68,7 @@ sub install () {
         ($ho, '',
          DiskDevice => $ho->{DiskDevice},
          Properties => $ho->{Properties},
+	 LinuxKernArchMap => \%linux_kern_arch_map,
          ExtraPreseed => <<END );
 d-i netcfg/get_hostname string $ho->{Name}
 d-i netcfg/get_ipaddress string $ho->{Ip}
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel