[PATCH OSSTEST v2 00/11] Upgrade to Debian Jessie

[PATCH OSSTEST v2 00/11] Upgrade to Debian Jessie

[Wei Liu]

This is a collection of patches from Ian and me to upgrade osstest to Debian
Jessie.

I've run

  build-amd64
  build-amd64-pvops
  test-amd64-amd64-xl
  test-amd64-amd64-xl-qemuu-debianhvm-amd64

all worked fine.

I can't test the arm patches though.

Wei.

Ian Campbell (5):
  More flexible handling of need-kernel-deb-$flavour host flag
  mg-debian-installer-update: updates to better handle Jessie onwards.
  mg-debian-installer-update: Also collect DTBs for standard kernels
  Force sysvinit with Jessie
  Debian: grub2: Allow submenu and menuentry items to be indented.

Wei Liu (6):
  Debian.pm: install grub to default device
  Cope with Jessie's d-i vg name
  ts-xen-build-prep: install libnl-route-3-dev on Jessie
  ts-xen-build-prep: install libc6-dev-i386 for jessie
  ts-xen-install: install libnl-route-3-200 for jessie
  ts-kernel-build: enable CONFIG_FHANDLE

 Osstest/Debian.pm              | 22 ++++++++++--
 mg-debian-installer-update     | 82 +++++++++++++++++++++++++++---------------
 mg-debian-installer-update-all |  2 +-
 ts-host-install                | 21 ++++++++---
 ts-kernel-build                |  2 ++
 ts-xen-build-prep              | 40 ++++++++++++---------
 ts-xen-install                 |  3 ++
 7 files changed, 119 insertions(+), 53 deletions(-)

-- 
1.9.1

[PATCH OSSTEST v2 01/11] More flexible handling of need-kernel-deb-$flavour host flag

[Wei Liu]

From: Ian Campbell <ian.campbell@citrix.com>

The current behaviour does not account for the need to apply this
workaround only for particular Debian releases, i.e. Midway will be
supported just fine in Jessie.

It also does not allow for kernels from alternative sources other than
backports (e.g. experimental, although that is hypothetical right
now).

The flavour name is actually irrelevant in practice (none of our
supported arch's need multiple kernel flavours at install time, which
is all we care about). So omit it from the new scheme.

The new scheme is "need-kernel-deb-$suite-$altsource", i.e.
need-kernel-deb-wheezy-backports. Possibly in the future e.g.
need-kernel-deb-jessie-experimental indicating that the experimental
kernel is needed on jessie.

mg-debian-installer-update is modified to download to the new name.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm          |  4 +++-
 mg-debian-installer-update | 22 +++++++++++-----------
 ts-host-install            | 14 +++++++++++---
 3 files changed, 25 insertions(+), 15 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index f09cad6..9b733d7 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -818,7 +818,9 @@ END
     }
 
     foreach my $kp (keys %{ $ho->{Flags} }) {
-	$kp =~ s/need-kernel-deb-// or next;
+	# Backwards compatibility
+	$kp = "need-kernel-deb-wheezy-backports" if $kp eq "need-kernel-deb-armmp";
+	$kp =~ s/need-kernel-deb-$ho->{Suite}// or next;
 
 	my $kern = "$d_i/$kp.deb";
 	my $kurl = create_webfile($ho, "kernel", sub {
diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index 4ff346f..f21f216 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -107,16 +107,16 @@ if [ $arch = armhf ]; then
     fetch "$site/$pkgfile" >initramfs-tools.deb
 
     set +e
-    echo >&2 "collecting armmp kernel"
+    echo >&2 "collecting backports kernel"
     # Be careful to pickup the actual kernel package from the 'linux'
     # source and not a meta package from 'linux-latest'
     pkgfile=`zcat Packages.gz | grep-dctrl -S linux | grep-dctrl -Pe ^linux-image-.*-armmp$ -nsFilename | sort -n -r | head -n1`
     rc=$?
     set -e
-    if [ $rc != 0 ]; then fail "armmp kernel package not found"; fi
-    fetch "$site/$pkgfile" >armmp.deb
-    dpkg-deb -x armmp.deb x
-    cp x/boot/vmlinuz-* linux.armmp
+    if [ $rc != 0 ]; then fail "backports kernel package not found"; fi
+    fetch "$site/$pkgfile" >backports.deb
+    dpkg-deb -x backports.deb x
+    cp x/boot/vmlinuz-* linux.backports
     # The full set of modules is pretty large and not that
     # useful. Just pull in the ones we need for our particular
     # preseed, which is basically disk, net, filesystems and LVM.
@@ -142,12 +142,12 @@ if [ $arch = armhf ]; then
                          -path \*/kernel/drivers/usb/host/\* \) \
           |pax -x sv4cpio -s '%lib%/lib%' -d -w -M norm >../cpio; cd ..
     gzip -9nf cpio
-    mv cpio.gz armmp.cpio.gz
-    rm -rf dtbs/
-    mkdir dtbs/
-    mv x/usr/lib/linux-image-*-armmp/*.dtb dtbs/
-    tar --mtime=./x/usr/lib -cf dtbs.tar dtbs
-    gzip -9nf dtbs.tar
+    mv cpio.gz backports.cpio.gz
+    rm -rf backports-dtbs/
+    mkdir backports-dtbs/
+    mv x/usr/lib/linux-image-*-armmp/*.dtb backports-dtbs/
+    tar --mtime=./x/usr/lib -cf backports-dtbs.tar backports-dtbs
+    gzip -9nf backports-dtbs.tar
     rm -rf x
 fi
 
diff --git a/ts-host-install b/ts-host-install
index b73c7ff..2fea451 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -198,7 +198,9 @@ sub setup_pxeboot_firstboot($) {
     }
 
     foreach my $kp (keys %{ $ho->{Flags} }) {
-        $kp =~ s/need-kernel-deb-// or next;
+        # Backwards compatibility
+        $kp = "need-kernel-deb-wheezy-backports" if $kp eq "need-kernel-deb-armmp";
+        $kp =~ s/need-kernel-deb-$ho->{Suite}// or next;
         my $kern= "$ho->{Tftp}{Path}/$d_i/linux.$kp";
         if (stat $kern) {
             logm("using kernel from: $kern");
@@ -244,8 +246,14 @@ END
     }
 
     my $dtbs = "";
-    $dtbs = "fdtdir /$d_i/dtbs"
-	if -e "$ho->{Tftp}{Path}/$d_i/dtbs";
+    foreach my $kp (keys %{ $ho->{Flags} }) {
+	# Backwards compatibility
+	$kp = "need-kernel-deb-wheezy-backports" if $kp eq "need-kernel-deb-armmp";
+	$kp =~ s/need-kernel-deb-$ho->{Suite}// or next;
+
+	$dtbs = "fdtdir /$d_i/$kp-dtbs"
+	    if -e "$ho->{Tftp}{Path}/$d_i/$kp-dtbs";
+    }
 
     file_simple_write_contents("$initrd_overlay.cpio", sub {
         contents_make_cpio($_[0], 'newc', "$initrd_overlay.d");
-- 
1.9.1

[PATCH OSSTEST v2 02/11] mg-debian-installer-update: updates to better handle Jessie onwards.

[Wei Liu]

From: Ian Campbell <ian.campbell@citrix.com>

In mg-debian-installer-update:

  - Expand the list of (suite,arch) combinations which don't exist and
    move it to the top.

  - Tweak the backports.org handling to allow it to be specified on a
    per (suite,arch) basis, and specify it only for (wheezy,armhf)
    since it is not currently needed for jessie. Make the
    initramfs-tools fetching conditional too.

  - Only extract backports-dtbs if they are present in the package.

  - Handle arm64.

  - Collect the correct armhf kernel for Jessie onwards (vexpress
    removed from path).

Have mg-debian-installer-update-all fetch arm64 too.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/Debian.pm              |  2 +-
 mg-debian-installer-update     | 59 ++++++++++++++++++++++++++----------------
 mg-debian-installer-update-all |  2 +-
 ts-host-install                |  4 +--
 4 files changed, 41 insertions(+), 26 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 9b733d7..c26f88e 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -820,7 +820,7 @@ END
     foreach my $kp (keys %{ $ho->{Flags} }) {
 	# Backwards compatibility
 	$kp = "need-kernel-deb-wheezy-backports" if $kp eq "need-kernel-deb-armmp";
-	$kp =~ s/need-kernel-deb-$ho->{Suite}// or next;
+	$kp =~ s/need-kernel-deb-$ho->{Suite}-// or next;
 
 	my $kern = "$d_i/$kp.deb";
 	my $kurl = create_webfile($ho, "kernel", sub {
diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index f21f216..f6bf962 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -38,20 +38,28 @@ fetch () {
 }
 
 case ${suite}_${arch} in
+    lenny_armhf|squeeze_armhf|lenny_arm64|squeeze_arm64|wheezy_arm64)
+        # No such thing.
+        exit 0
+        ;;
     *_i386|*_amd64)
         src=$sbase/main/installer-$arch/current/images/netboot/
         files="debian-installer/$arch/initrd.gz:initrd.gz debian-installer/$arch/linux:linux xen/vmlinuz:vmlinuz-xen xen/initrd.gz:initrd.gz-xen"
         ;;
-    lenny_armhf|squeeze_armhf)
-        # No such thing.
-        exit 0
+    *_arm64)
+        src=$sbase/main/installer-$arch/current/images/netboot/
+        files="debian-installer/$arch/initrd.gz:initrd.gz debian-installer/$arch/linux:linux"
         ;;
-    *_armhf)
+    wheezy_armhf)
         # Use the vexpress image, we are going to bash in our own
         # kernel+modules anyway.
         src=$sbase/main/installer-$arch/current/images/vexpress/netboot
         files='initrd.gz' # no kernel -- needs a newer kernel
         ;;
+    *_armhf)
+        src=$sbase/main/installer-$arch/current/images/netboot
+        files='vmlinuz initrd.gz'
+	;;
 esac
 
 pfile=$sbase/non-free/binary-$arch/Packages.gz
@@ -89,28 +97,33 @@ for p in $packages; do
         rm -rf x
 done
 
-# armhf requires a newer kernel than was in Wheezy for most
-# platforms. Construct something suitable from the latest kernel in
-# wheezy-backports.
-if [ $arch = armhf ]; then
+# Some platforms require a newer kernel than is in Debian. Construct
+# something suitable from the latest kernel ($bpok=flavour) in
+# backports.
+case $suite_$arch in
+    wheezy_armhf) bpok=armmp; need_initramfs=y;;
+esac
+if [ x$bpok != x ]; then
     bp="$sbase-backports"
-    pfile=$bp/main/binary-armhf/Packages.gz
+    pfile=$bp/main/binary-$arch/Packages.gz
 
     fetch $pfile >Packages.gz
 
-    # Newer kernel often needs a newer initramfs-tools. Make that available
-    echo >&2 "collecting backports initramfs-tools"
-    pkgfile=`zcat Packages.gz | grep-dctrl -PX initramfs-tools -nsFilename | sort -n -r | head -n1`
-    rc=$?
-    set -e
-    if [ $rc != 0 ]; then fail "initramfs-tools package not found"; fi
-    fetch "$site/$pkgfile" >initramfs-tools.deb
+    if [ x$need_initramfs = xy ]; then
+        # Newer kernel often needs a newer initramfs-tools. Make that available
+        echo >&2 "collecting backports initramfs-tools"
+        pkgfile=`zcat Packages.gz | grep-dctrl -PX initramfs-tools -nsFilename | sort -n -r | head -n1`
+        rc=$?
+        set -e
+        if [ $rc -ne 0 ] || [ x$pkgfile = x ]; then fail "initramfs-tools package not found"; fi
+        fetch "$site/$pkgfile" >initramfs-tools.deb
+    fi
 
     set +e
     echo >&2 "collecting backports kernel"
     # Be careful to pickup the actual kernel package from the 'linux'
     # source and not a meta package from 'linux-latest'
-    pkgfile=`zcat Packages.gz | grep-dctrl -S linux | grep-dctrl -Pe ^linux-image-.*-armmp$ -nsFilename | sort -n -r | head -n1`
+    pkgfile=`zcat Packages.gz | grep-dctrl -S linux | grep-dctrl -Pe ^linux-image-.*-${bpok}$ -nsFilename | sort -n -r | head -n1`
     rc=$?
     set -e
     if [ $rc != 0 ]; then fail "backports kernel package not found"; fi
@@ -143,11 +156,13 @@ if [ $arch = armhf ]; then
           |pax -x sv4cpio -s '%lib%/lib%' -d -w -M norm >../cpio; cd ..
     gzip -9nf cpio
     mv cpio.gz backports.cpio.gz
-    rm -rf backports-dtbs/
-    mkdir backports-dtbs/
-    mv x/usr/lib/linux-image-*-armmp/*.dtb backports-dtbs/
-    tar --mtime=./x/usr/lib -cf backports-dtbs.tar backports-dtbs
-    gzip -9nf backports-dtbs.tar
+    if [ -d x/usr/lib/linux-image-*-${bpok}/ ] ; then
+        rm -rf backports-dtbs/
+        mkdir backports-dtbs/
+        mv x/usr/lib/linux-image-*-${bpok}/*.dtb backports-dtbs/
+        tar --mtime=./x/usr/lib -cf backports-dtbs.tar backports-dtbs
+        gzip -9nf backports-dtbs.tar
+    fi
     rm -rf x
 fi
 
diff --git a/mg-debian-installer-update-all b/mg-debian-installer-update-all
index eca4a5f..241e2e3 100755
--- a/mg-debian-installer-update-all
+++ b/mg-debian-installer-update-all
@@ -24,7 +24,7 @@ set -e
 
 suite=`getconfig DebianSuite`
 fws=`getconfig DebianNonfreeFirmware`
-arches="armhf amd64 i386"
+arches="arm64 armhf amd64 i386"
 
 for arch in $arches ; do
     ./mg-debian-installer-update $suite $arch $fws
diff --git a/ts-host-install b/ts-host-install
index 2fea451..242bfb0 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -200,7 +200,7 @@ sub setup_pxeboot_firstboot($) {
     foreach my $kp (keys %{ $ho->{Flags} }) {
         # Backwards compatibility
         $kp = "need-kernel-deb-wheezy-backports" if $kp eq "need-kernel-deb-armmp";
-        $kp =~ s/need-kernel-deb-$ho->{Suite}// or next;
+        $kp =~ s/need-kernel-deb-$ho->{Suite}-// or next;
         my $kern= "$ho->{Tftp}{Path}/$d_i/linux.$kp";
         if (stat $kern) {
             logm("using kernel from: $kern");
@@ -249,7 +249,7 @@ END
     foreach my $kp (keys %{ $ho->{Flags} }) {
 	# Backwards compatibility
 	$kp = "need-kernel-deb-wheezy-backports" if $kp eq "need-kernel-deb-armmp";
-	$kp =~ s/need-kernel-deb-$ho->{Suite}// or next;
+	$kp =~ s/need-kernel-deb-$ho->{Suite}-// or next;
 
 	$dtbs = "fdtdir /$d_i/$kp-dtbs"
 	    if -e "$ho->{Tftp}{Path}/$d_i/$kp-dtbs";
-- 
1.9.1

[PATCH OSSTEST v2 03/11] mg-debian-installer-update: Also collect DTBs for standard kernels

[Wei Liu]

From: Ian Campbell <ian.campbell@citrix.com>

For armhf and arm64 for Jessie we will need these in the normal case
as well as in the backports case. Arrange to download.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 mg-debian-installer-update | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index f6bf962..3dd036a 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -49,6 +49,7 @@ case ${suite}_${arch} in
     *_arm64)
         src=$sbase/main/installer-$arch/current/images/netboot/
         files="debian-installer/$arch/initrd.gz:initrd.gz debian-installer/$arch/linux:linux"
+        dtbs=$sbase/main/installer-$arch/current/images/device-tree/
         ;;
     wheezy_armhf)
         # Use the vexpress image, we are going to bash in our own
@@ -59,7 +60,8 @@ case ${suite}_${arch} in
     *_armhf)
         src=$sbase/main/installer-$arch/current/images/netboot
         files='vmlinuz initrd.gz'
-	;;
+        dtbs=$sbase/main/installer-$arch/current/images/device-tree/
+        ;;
 esac
 
 pfile=$sbase/non-free/binary-$arch/Packages.gz
@@ -79,6 +81,15 @@ for f in $files; do
         fetch $src/$s >$d.new
 done
 
+if [ "x$dtbs" != "x" ] ; then
+    # Can't seem to get curl to globs.
+    rm -rf dtbs
+    mkdir dtbs
+    ( cd dtbs && wget -nv -A README,\*.dtb -nd -nH -np -m  $dtbs )
+    tar --mtime=./dtbs/README -cf dtbs.tar dtbs
+    gzip -9nf dtbs.tar
+fi
+
 fetch $pfile >Packages.gz
 
 for p in $packages; do
-- 
1.9.1

[PATCH OSSTEST v2 04/11] Force sysvinit with Jessie

[Wei Liu]

From: Ian Campbell <ian.campbell@citrix.com>

Systemd has some issues:

1. It doesn't honor osstest-confirm-booted service, which breaks
   ts-leak-check.
2. It kills the ssh connection before 'init 6' has a chance
   return, which breaks ts-host-reboot.

Fall back to SysV init for now.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
[wei: rewrite commit message and code comment]
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/Debian.pm | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index c26f88e..fa94210 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -903,6 +903,18 @@ in-target update-initramfs -u -k all
 END
     }
 
+    # Systemd has some issues:
+    #  1. it doesn't honor osstest-confirm-booted service,
+    #     which breaks ts-leak-check.
+    #  2. it kills the ssh connection before 'init 6' has a chance
+    #     return, which breaks ts-host-reboot.
+    # Fall back to SysV init for now.
+    if ( $ho->{Suite} =~ /jessie/ ) {
+       preseed_hook_command($ho, 'late_command', $sfx, <<END)
+in-target apt-get install -y sysvinit-core
+END
+    }
+
     my @extra_packages = ();
     push(@extra_packages, "u-boot-tools") if $ho->{Flags}{'need-uboot-bootscr'};
 
-- 
1.9.1

[PATCH OSSTEST v2 05/11] Debian.pm: install grub to default device

[Wei Liu]

Debian installer used to have the behaviour to install grub on the
default device. But it's changed in Jessie.

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=759737.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
>From Weis v1 Jessie series
---
 Osstest/Debian.pm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index fa94210..5463fd0 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -685,6 +685,8 @@ popularity-contest popularity-contest/participate boolean false
 tasksel tasksel/first multiselect standard, web-server
 
 d-i grub-installer/only_debian boolean true
+# see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=759737
+d-i grub-installer/bootdev string default
 
 d-i finish-install/keep-consoles boolean true
 d-i finish-install/reboot_in_progress note
-- 
1.9.1

[PATCH OSSTEST v2 06/11] Cope with Jessie's d-i vg name

[Wei Liu]

In Jessie the default vg name is changed to "$hostname-vg". Make that
default case and check for wheezy, squeeze and lenny for backward
compatibility.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
Changes in v2:
1. Make Jessie case default.
---
 ts-host-install   | 7 +++++--
 ts-xen-build-prep | 8 +++++---
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/ts-host-install b/ts-host-install
index 242bfb0..50b39a0 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -87,10 +87,13 @@ END
 
     await_tcp(get_timeout($ho,'reboot',$timeout{Sshd}), 14,$ho);
 
-    our $vgname= $ho->{Name};
+    our $vgname= $ho->{Suite} =~ m/wheezy|squeeze/
+                 ? $ho->{Name}
+                 : $ho->{Suite} =~ m/lenny/
+		     ? "$ho->{Name}.$c{TestHostDomain}"
+		     : "$ho->{Name}-vg";
 
     if ($ho->{Suite} =~ m/lenny/) {
-        $vgname .= ".$c{TestHostDomain}";
         target_editfile_root($ho, '/boot/grub/menu.lst', sub {
             while (<EI>) {
                 s/^terminal.*/terminal --timeout=10 serial console/;
diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 4dece65..befb7d4 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -53,9 +53,11 @@ sub overall_limit_pe ($) {
 
 sub determine_vg_lv () {
     $vg=
-        $ho->{Suite} =~ m/lenny/
-        ? "$ho->{Name}.$c{TestHostDomain}"
-        : $ho->{Name};
+        $ho->{Suite} =~ m/wheezy|squeeze/
+        ? $ho->{Name}
+        : $ho->{Suite} =~ m/lenny/
+	    ? "$ho->{Name}.$c{TestHostDomain}"
+	    : "$ho->{Name}-vg";
     $lv = "/dev/$vg/$lvleaf";
 }
 
-- 
1.9.1

[PATCH OSSTEST v2 07/11] Debian: grub2: Allow submenu and menuentry items to be indented.

[Wei Liu]

From: Ian Campbell <ian.campbell@citrix.com>

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest/Debian.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 5463fd0..50730f5 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -444,12 +444,12 @@ sub setupboot_grub2 ($$$$) {
             if (m/^function.*\{/) {
                 $entry= { StartLine => $. };
             }
-            if (m/^menuentry\s+[\'\"](.*)[\'\"].*\{\s*$/) {
+            if (m/^\s*menuentry\s+[\'\"](.*)[\'\"].*\{\s*$/) {
                 die $entry->{StartLine} if $entry;
                 $entry= { Title => $1, StartLine => $., MenuEntryPath => join ">", @offsets };
                 $offsets[$#offsets]++;
             }
-            if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
+            if (m/^\s*submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
                 $submenu={ StartLine =>$., MenuEntryPath => join ">", @offsets };
                 push @offsets,(0);
             }
-- 
1.9.1

[PATCH OSSTEST v2 08/11] ts-xen-build-prep: install libnl-route-3-dev on Jessie

[Wei Liu]

There is no package called libnl-dev on Jessie.  To build remus, use
libnl-route-3-dev instead.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build-prep | 30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index befb7d4..4fdc37e 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -200,19 +200,25 @@ END
 }
 
 sub prep () {
-    target_install_packages($ho,
-                            qw(mercurial
-                               build-essential bin86 bcc iasl bc
-                               flex bison cmake
-                               libpci-dev libncurses5-dev libssl-dev python-dev
-                               libx11-dev git-core uuid-dev gettext gawk
-                               libsdl-dev libyajl-dev libaio-dev libpixman-1-dev
-                               libglib2.0-dev liblzma-dev pkg-config
-                               autoconf automake libtool xsltproc
-                               libxml2-utils libxml2-dev libnl-dev
-                               libdevmapper-dev w3c-dtd-xhtml libxml-xpath-perl
-                               ccache nasm checkpolicy));
+    my @packages = qw(mercurial
+                      build-essential bin86 bcc iasl bc
+                      flex bison cmake
+                      libpci-dev libncurses5-dev libssl-dev python-dev
+                      libx11-dev git-core uuid-dev gettext gawk
+                      libsdl-dev libyajl-dev libaio-dev libpixman-1-dev
+                      libglib2.0-dev liblzma-dev pkg-config
+                      autoconf automake libtool xsltproc
+                      libxml2-utils libxml2-dev
+                      libdevmapper-dev w3c-dtd-xhtml libxml-xpath-perl
+                      ccache nasm checkpolicy);
 
+    if ($ho->{Suite} =~ m/wheezy|squeeze|lenny/) {
+	push(@packages, "libnl-dev");
+    } else {
+	# jessie (>jessie?)
+	push(@packages, "libnl-route-3-dev");
+    }
+    target_install_packages($ho, @packages);
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
 
-- 
1.9.1

[PATCH OSSTEST v2 09/11] ts-xen-build-prep: install libc6-dev-i386 for jessie

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build-prep | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 4fdc37e..c348dad 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -232,7 +232,7 @@ END
     if ($ho->{Suite} !~ m/lenny|squeeze/) {
         target_install_packages($ho, 'libfdt-dev');
     }
-    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy/) {
+    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy|jessie/) {
         target_install_packages($ho, 'libc6-dev-i386');
     }
 }
-- 
1.9.1

[PATCH OSSTEST v2 10/11] ts-xen-install: install libnl-route-3-200 for jessie

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-install | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/ts-xen-install b/ts-xen-install
index b2f667f..6dfab1b 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -54,6 +54,9 @@ sub packages () {
     target_install_packages($ho,
                             qw(bridge-utils vncsnapshot libaio1 libpixman-1-0
                                libsdl1.2debian libglib2.0-0 liblzma5));
+    if ($ho->{Suite} =~ m/jessie/) {
+        target_install_packages($ho, 'libnl-route-3-200');
+    }
     target_install_packages($ho,
 			    $ho->{Suite} =~ /squeeze/ ? "libyajl1" : "libyajl2");
     if ($ho->{Suite} !~ m/lenny|squeeze/) {
-- 
1.9.1

[PATCH OSSTEST v2 11/11] ts-kernel-build: enable CONFIG_FHANDLE

[Wei Liu]

According to Debian bug #764539, systemd needs this kernel feature to be
able to mount disks. Complete list of required kernel features can be
found in /usr/share/doc/systemd/README.gz.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-kernel-build | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/ts-kernel-build b/ts-kernel-build
index e7cbdc1..438be6d 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -525,6 +525,8 @@ setopt CONFIG_FUSION_LOGGING n
 
 setopt CONFIG_BLK_CPQ_CISS_DA m
 
+setopt CONFIG_FHANDLE y
+
 case ${XEN_TARGET_ARCH} in
     x86_32) setopt CONFIG_64BIT n ;;
     x86_64)
-- 
1.9.1

Re: [PATCH OSSTEST v2 04/11] Force sysvinit with Jessie

[Ian Campbell]

On Mon, 2015-06-15 at 14:36 +0100, Wei Liu wrote:
> From: Ian Campbell <ian.campbell@citrix.com>
> 
> Systemd has some issues:
> 
> 1. It doesn't honor osstest-confirm-booted service, which breaks
>    ts-leak-check.
> 2. It kills the ssh connection before 'init 6' has a chance
>    return, which breaks ts-host-reboot.
> 
> Fall back to SysV init for now.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> [wei: rewrite commit message and code comment]

Those are fine by me, thanks.

> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  Osstest/Debian.pm | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
> index c26f88e..fa94210 100644
> --- a/Osstest/Debian.pm
> +++ b/Osstest/Debian.pm
> @@ -903,6 +903,18 @@ in-target update-initramfs -u -k all
>  END
>      }
>  
> +    # Systemd has some issues:
> +    #  1. it doesn't honor osstest-confirm-booted service,
> +    #     which breaks ts-leak-check.
> +    #  2. it kills the ssh connection before 'init 6' has a chance
> +    #     return, which breaks ts-host-reboot.
> +    # Fall back to SysV init for now.
> +    if ( $ho->{Suite} =~ /jessie/ ) {
> +       preseed_hook_command($ho, 'late_command', $sfx, <<END)
> +in-target apt-get install -y sysvinit-core
> +END
> +    }
> +
>      my @extra_packages = ();
>      push(@extra_packages, "u-boot-tools") if $ho->{Flags}{'need-uboot-bootscr'};
>  

Re: [PATCH OSSTEST v2 04/11] Force sysvinit with Jessie

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH OSSTEST v2 04/11] Force sysvinit with Jessie"):
> On Mon, 2015-06-15 at 14:36 +0100, Wei Liu wrote:
...
> > 2. It kills the ssh connection before 'init 6' has a chance
> >    return, which breaks ts-host-reboot.

Arguably this isn't a bug in systemd, since sshd is racing with the
system teardown.  Perhaps we should simply ignore errors from this
command.

Ian.

Re: [PATCH OSSTEST v2 04/11] Force sysvinit with Jessie

[Wei Liu]

On Mon, Jun 15, 2015 at 04:48:52PM +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [PATCH OSSTEST v2 04/11] Force sysvinit with Jessie"):
> > On Mon, 2015-06-15 at 14:36 +0100, Wei Liu wrote:
> ...
> > > 2. It kills the ssh connection before 'init 6' has a chance
> > >    return, which breaks ts-host-reboot.
> 
> Arguably this isn't a bug in systemd, since sshd is racing with the
> system teardown.  Perhaps we should simply ignore errors from this
> command.
> 

That would do. But this point is moot since we're not using systemd now.
I can rewrite the commit message and comment in next version.

For the record I also had a patch to use "shutdown -r +1" to workaround
this problem, but since you're fine with ignoring error I think that's
better.

Wei.

> Ian.

Re: [PATCH OSSTEST v2 04/11] Force sysvinit with Jessie

[Ian Jackson]

Wei Liu writes ("Re: [PATCH OSSTEST v2 04/11] Force sysvinit with Jessie"):
> That would do. But this point is moot since we're not using systemd now.
> I can rewrite the commit message and comment in next version.

Right.

> For the record I also had a patch to use "shutdown -r +1" to workaround
> this problem, but since you're fine with ignoring error I think that's
> better.

`shutdown -r +1' waits 1 minute, which is rather too long.

Thanks,
Ian.

Re: [PATCH OSSTEST v2 01/11] More flexible handling of need-kernel-deb-$flavour host flag

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 01/11] More flexible handling of need-kernel-deb-$flavour host flag"):
> From: Ian Campbell <ian.campbell@citrix.com>
> 
> The current behaviour does not account for the need to apply this
> workaround only for particular Debian releases, i.e. Midway will be
> supported just fine in Jessie.
> 
> It also does not allow for kernels from alternative sources other than
> backports (e.g. experimental, although that is hypothetical right
> now).
> 
> The flavour name is actually irrelevant in practice (none of our
> supported arch's need multiple kernel flavours at install time, which
> is all we care about). So omit it from the new scheme.
> 
> The new scheme is "need-kernel-deb-$suite-$altsource", i.e.
> need-kernel-deb-wheezy-backports. Possibly in the future e.g.
> need-kernel-deb-jessie-experimental indicating that the experimental
> kernel is needed on jessie.
> 
> mg-debian-installer-update is modified to download to the new name.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

But AFAICT this needs the update to be run on the production instance,
and to accompanied by a di version change ?

Ian.

Re: [PATCH OSSTEST v2 02/11] mg-debian-installer-update: updates to better handle Jessie onwards.

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 02/11] mg-debian-installer-update: updates to better handle Jessie onwards."):
> From: Ian Campbell <ian.campbell@citrix.com>

I'm afraid I have got fed up of reading the wrap damage in this file.

Can you please rewrap at least all the new or changed lines to ~70,
and respin ?

Thanks,
Ian.

Re: [PATCH OSSTEST v2 03/11] mg-debian-installer-update: Also collect DTBs for standard kernels

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 03/11] mg-debian-installer-update: Also collect DTBs for standard kernels"):
> From: Ian Campbell <ian.campbell@citrix.com>
> 
> For armhf and arm64 for Jessie we will need these in the normal case
> as well as in the backports case. Arrange to download.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Ian.

Re: [PATCH OSSTEST v2 04/11] Force sysvinit with Jessie

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 04/11] Force sysvinit with Jessie"):
> From: Ian Campbell <ian.campbell@citrix.com>
> 
> Systemd has some issues:
> 
> 1. It doesn't honor osstest-confirm-booted service, which breaks
>    ts-leak-check.
...
> Fall back to SysV init for now.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST v2 05/11] Debian.pm: install grub to default device

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 05/11] Debian.pm: install grub to default device"):
> Debian installer used to have the behaviour to install grub on the
> default device. But it's changed in Jessie.
> 
> See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=759737.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST v2 06/11] Cope with Jessie's d-i vg name

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 06/11] Cope with Jessie's d-i vg name"):
> In Jessie the default vg name is changed to "$hostname-vg". Make that
> default case and check for wheezy, squeeze and lenny for backward
> compatibility.
...
> -    our $vgname= $ho->{Name};
> +    our $vgname= $ho->{Suite} =~ m/wheezy|squeeze/
> +                 ? $ho->{Name}
> +                 : $ho->{Suite} =~ m/lenny/
> +		     ? "$ho->{Name}.$c{TestHostDomain}"
> +		     : "$ho->{Name}-vg";

This nested indentation is not conventional (in any language, AFAICT)
for sequences of ? :.

There are various acceptable indentations but I don't think this is
one of them.  (I normally prefer versions which put the condition and
the result on the same line, but that may be too tedious here.)

>  sub determine_vg_lv () {
>      $vg=
> -        $ho->{Suite} =~ m/lenny/
> -        ? "$ho->{Name}.$c{TestHostDomain}"
> -        : $ho->{Name};
> +        $ho->{Suite} =~ m/wheezy|squeeze/
> +        ? $ho->{Name}
> +        : $ho->{Suite} =~ m/lenny/
> +	    ? "$ho->{Name}.$c{TestHostDomain}"
> +	    : "$ho->{Name}-vg";

Now, admittedly, this logic was duplicated before you started.  But
really I think you should abolish the duplication before extending it.

Thanks,
Ian.

Re: [PATCH OSSTEST v2 07/11] Debian: grub2: Allow submenu and menuentry items to be indented.

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 07/11] Debian: grub2: Allow submenu and menuentry items to be indented."):
> From: Ian Campbell <ian.campbell@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST v2 08/11] ts-xen-build-prep: install libnl-route-3-dev on Jessie

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 08/11] ts-xen-build-prep: install libnl-route-3-dev on Jessie"):
> There is no package called libnl-dev on Jessie.  To build remus, use
> libnl-route-3-dev instead.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST v2 09/11] ts-xen-build-prep: install libc6-dev-i386 for jessie

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 09/11] ts-xen-build-prep: install libc6-dev-i386 for jessie"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
...
> diff --git a/ts-xen-build-prep b/ts-xen-build-prep
> index 4fdc37e..c348dad 100755
> --- a/ts-xen-build-prep
> +++ b/ts-xen-build-prep
> @@ -232,7 +232,7 @@ END
>      if ($ho->{Suite} !~ m/lenny|squeeze/) {
>          target_install_packages($ho, 'libfdt-dev');
>      }
> -    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy/) {
> +    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy|jessie/) {
>          target_install_packages($ho, 'libc6-dev-i386');

If we are expecting this to be permanent, rather than it being a bug
in wheezy, I think the sense of the test here is wrong, and we should
be listing the older suites which don't need it.

Thanks,
Ian.

Re: [PATCH OSSTEST v2 10/11] ts-xen-install: install libnl-route-3-200 for jessie

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 10/11] ts-xen-install: install libnl-route-3-200 for jessie"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
...
>      target_install_packages($ho,
>                          qw(bridge-utils vncsnapshot libaio1 libpixman-1-0
>                             libsdl1.2debian libglib2.0-0 liblzma5));
> +    if ($ho->{Suite} =~ m/jessie/) {
> +        target_install_packages($ho, 'libnl-route-3-200');

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>


.oO{

I wonder if we should make target_install_packages have some kind of
magic metasyntax for this kind of thing.

   target_install_packages($ho,
                     qw(bridge-utils vncsnapshot libaio1 libpixman-1-0
-                       libsdl1.2debian libglib2.0-0 liblzma5));
+                       libsdl1.2debian libglib2.0-0 liblzma5
+                       {>=jessie}libnl-route-3-200));

}

Ian.

Re: [PATCH OSSTEST v2 11/11] ts-kernel-build: enable CONFIG_FHANDLE

[Ian Jackson]

Wei Liu writes ("[PATCH OSSTEST v2 11/11] ts-kernel-build: enable CONFIG_FHANDLE"):
> According to Debian bug #764539, systemd needs this kernel feature to be
> able to mount disks. Complete list of required kernel features can be
> found in /usr/share/doc/systemd/README.gz.

Uh, and it's not the default.  Brilliant.  I don't suppose any of the
systemd people felt like trying to fix upstream kernels.

> +setopt CONFIG_FHANDLE y
> +

Fine, whatever :-).

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Thanks,
Ian.

Re: [PATCH OSSTEST v2 09/11] ts-xen-build-prep: install libc6-dev-i386 for jessie

[Wei Liu]

On Mon, Jun 15, 2015 at 05:21:12PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[PATCH OSSTEST v2 09/11] ts-xen-build-prep: install libc6-dev-i386 for jessie"):
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ...
> > diff --git a/ts-xen-build-prep b/ts-xen-build-prep
> > index 4fdc37e..c348dad 100755
> > --- a/ts-xen-build-prep
> > +++ b/ts-xen-build-prep
> > @@ -232,7 +232,7 @@ END
> >      if ($ho->{Suite} !~ m/lenny|squeeze/) {
> >          target_install_packages($ho, 'libfdt-dev');
> >      }
> > -    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy/) {
> > +    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy|jessie/) {
> >          target_install_packages($ho, 'libc6-dev-i386');
> 
> If we are expecting this to be permanent, rather than it being a bug
> in wheezy, I think the sense of the test here is wrong, and we should
> be listing the older suites which don't need it.
> 

Yes, this is going to be permanent. The root cause is wheezy onwards
introduces multi-arch that separates amd64 and i386.

I will reverse logic: add that package in the initial set and remove it
for older suites.

Wei.

> Thanks,
> Ian.