[OSSTEST PATCH 00/16] Upgrade to Stretch

[OSSTEST PATCH 00/16] Upgrade to Stretch

[Wei Liu]

Hi all

This series upgrades osstest to Debian Strech, which aims to:

1. Keep osstest up-to-date.
2. Enable us to collect aggregate gcov data (depends on newer gcc in Stretch).
3. Enable us to run (yet to be implemented) hypervisor fuzzer in the (distant)
   future (depends on newer gcc in Stretch).

There are a few issues:

1. Debian stock kernel has a bug that causes local migration fails.
2. Hosts merlot* don't work well even without xen:
   2.1 Sometimes they don't boot after debian-install.
   2.2 Sometimes they stall when installing packages, causing tests to time out.
   These aren't really related to Stretch.
3. The unstability with Arndale boards' nic is more prominent. Or worse -- they
   have become completely unusable. I don't have enough data yet. We might
   need to work around this, but I'm not sure how to do that yet.
4. Pvgrub1 won't work due to it is bitrot and can't deal with ext4 fs with
   64bit enabled.
5. Osstest currently doesn't have any ARM64 boxes, so I'm not sure if they
   will require any fixes.
6. Rumprun doesn't build due to its build system can't cope. This should be
   fixed in rumprun.

I have taken the following measures:

1. Mark merlot* unsuitable for Stretch (!suite-stretch) to avoid false
   positives during development.
2. Drop pvgrub1 tests.

Example flight:
http://logs.test-lab.xenproject.org/osstest/logs/114769/

Wei.

Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Julien Grall <julien.grall@arm.com>

Wei Liu (16):
  XXX add a stretch config based on production-config
  gitignore: ignore vim swap file
  ts-xen-build-prep: don't install w3c-dtd-xhtml for Stretch onward
  ts-xen-build-prep: install packages for stretch
  mg-debian-installer-update-all: put quotes around arguments
  ts-xen-install: install some packages on stretch
  Debian.pm: use sysvinit-core on stretch
  ts-leak-check: suppress systemd-shim, which leaks in stretch
  ts-host-install: don't use the new nic naming scheme
  ts-debian-fixup: remove extra= before appending our own
  ts-debian-fixup: use correct resume device
  make-flight: don't test pvgrub for Xen XXX
  ts-debian-hvm-install: disable new nic naming scheme
  ts-xen-build-prep: install e2fslibs-dev
  TestSupport: add dpkg option when installing packages
  ts-guests-nbd-mirror: make it work with stretch

 .gitignore                     |   1 +
 Osstest/Debian.pm              |   2 +-
 Osstest/TestSupport.pm         |   3 +-
 make-flight                    |  17 +++-
 mg-debian-installer-update     |   2 +-
 mg-debian-installer-update-all |   2 +-
 production-config-stretch      | 173 +++++++++++++++++++++++++++++++++++++++++
 ts-debian-fixup                |  14 ++++
 ts-debian-hvm-install          |  12 +++
 ts-guests-nbd-mirror           |  47 ++++++++++-
 ts-host-install                |   4 +
 ts-leak-check                  |   1 +
 ts-xen-build-prep              |  15 +++-
 ts-xen-install                 |   3 +
 14 files changed, 286 insertions(+), 10 deletions(-)
 create mode 100644 production-config-stretch

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 01/16] XXX add a stretch config based on production-config

[Wei Liu]

diff -ub production-config production-config-stretch

--- production-config   2017-10-11 15:01:16.891770210 +0100
+++ production-config-stretch   2017-10-19 16:12:20.839743993 +0100
@@ -21,6 +21,9 @@
 JobDB Executive
 HostDB Executive

+DebianSuite stretch
+GuestDebianSuite stretch
+
 HostnameSortSwapWords 0

 HostDB_Executive_NoConfigDB 1
@@ -93,10 +96,12 @@
 # Update with ./mg-debian-installer-update(-all)
 TftpDiVersion_wheezy 2016-06-08
 TftpDiVersion_jessie 2017-04-06
+TftpDiVersion_stretch 2017-10-11

 # For ISO installs
 DebianImageVersion_wheezy 7.2.0
 DebianImageVersion_jessie 8.2.0
+DebianImageVersion_stretch 9.2.1

 # These should normally be the same.
 # Update with ./mg-cpu-microcode-update
---
 production-config-stretch | 173 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 173 insertions(+)
 create mode 100644 production-config-stretch

diff --git a/production-config-stretch b/production-config-stretch
new file mode 100644
index 0000000..5d48bfd
--- /dev/null
+++ b/production-config-stretch
@@ -0,0 +1,173 @@
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This is the master instance, therefore it has no upstream
+OsstestUpstream= ''
+
+JobDB Executive
+HostDB Executive
+
+DebianSuite stretch
+GuestDebianSuite stretch
+
+HostnameSortSwapWords 0
+
+HostDB_Executive_NoConfigDB 1
+Include /root/osstest-hosts-config
+
+DnsDomain test-lab.xenproject.org
+NetNameservers 172.16.144.4
+
+OwnerDaemonHost db
+QueueDaemonHost osstest
+
+ExecutiveDbnamePat dbname=osstestdb;host=db
+
+Images /home/logs/images
+
+PubBaseDir /home/logs
+
+OverlayLocal /home/osstest/overlay-local
+
+LogsMinSpaceMby= 10*1e3
+LogsMinExpireAge= 86400*4
+LogsPublishMinSpaceMby= 10*1e3
+LogsPublishMinExpireAge= 86400*7
+
+TestHostKeypairPath /home/osstest/.ssh/id_rsa_osstest
+
+GitCacheProxy git://cache:9419/
+
+HttpProxy http://cache:3128/
+
+PubBaseUrl http://logs.test-lab.xenproject.org/osstest
+ReportHtmlPubBaseUrl="$c{PubBaseUrl}/logs"
+ResultsHtmlPubBaseUrl="$c{PubBaseUrl}/results"
+ReportHtmlUnpubBaseUrl="http://osstest/~osstest/pub/logs/"
+
+Publish osstest@www:/var/www/osstest
+GlobalLockDir /home/osstest/testing.git
+
+LogsPublish= "$c{Publish}/logs"
+ResultsPublish= "$c{Publish}/results"
+
+HarnessPublishGitUserHost osstest@xenbits.xen.org
+HarnessPublishGitRepoDir ext/osstest-massachusetts.git
+
+ReportTrailer= <<END
+Logs, config files, etc. are available at
+    $c{ReportHtmlPubBaseUrl}
+
+Explanation of these reports, and of osstest in general, is at
+    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
+    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
+
+Test harness code can be found at
+    http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
+END
+
+PlanRogueAllocationDuration= 86400*7
+
+TftpPath /home/tftp/
+TftpPlayDir osstest/
+TftpTmpDir osstest/tmp/
+TftpPxeDir /
+TftpPxeTemplates %name%/pxelinux.cfg
+TftpPxeTemplatesReal pxelinux.cfg/%ipaddrhex%
+TftpNetGrubDir /
+TftpNetGrubTemplates %name%/netgrub.cfg
+TftpNetGrubTemplatesReal Netgrub.cfg/%ether%
+
+TftpNetbootGroup osstest
+# Update with ./mg-debian-installer-update(-all)
+TftpDiVersion_wheezy 2016-06-08
+TftpDiVersion_jessie 2017-04-06
+TftpDiVersion_stretch 2017-10-11
+
+# For ISO installs
+DebianImageVersion_wheezy 7.2.0
+DebianImageVersion_jessie 8.2.0
+DebianImageVersion_stretch 9.2.1
+
+# These should normally be the same.
+# Update with ./mg-cpu-microcode-update
+MicrocodeUpdateAmd64 microcode.x86.2015-06-12.cpio
+MicrocodeUpdateI386 microcode.x86.2015-06-12.cpio
+
+# Update with ./mg-netgrub-loader-update
+TftpGrubVersion XXXX-XX-XX
+
+XenUsePath /usr/groups/xencore/systems/bin/xenuse
+XenUseUser osstest
+
+# Results might include potential vulnerabilities.
+CoverityEmail security@xen.org
+# This is only read from daily-cron-settings-real, everything else
+# gets the default/dummy path
+CoverityUploadUrl https://scan.coverity.com/builds?project=XenProject
+CoverityTools cov-analysis-linux64-2017.07.tar.gz
+CoverityToolsStripComponents 2
+
+# We use the IP address because Citrix can't manage reliable nameservice
+#DebianMirrorHost debian.uk.xensource.com
+#DebianMirrorHost 10.80.16.196
+DebianMirrorProxy http://cache:3143/
+
+HostProp_NtpServer infra.test-lab.xenproject.org
+
+DebianPreseed= <<'END'
+END
+
+HostProp_Serial sympathy serial0
+HostProp_GenEtherPrefixBase 5a:36:0e:00
+#                                      :00:01 guest number in job appended
+#                                    ^^ xor'd with low 8 bits of flight
+
+AuthorizedKeysAppend= <<'END'
+ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq8eHHFJ+XHYgpHxfSdciq0b3tYPdMhHf9CgtwdKGSqCyDyocbn1jX6P0Z535K/JcVaxvaRQbGDl9FZ25neQw6lysE8pGf+G353mgLAE7Lw6xKqlTXDcR0GpKHiZUyY8Ck5AJlGF2MO0cDEzMBx+xkOahDBvAozikUcDHJsTNP+UUIGoRaPeQK0DfgprPkoaLzXFDiZvEoBtYcUUieuNygJt+QVM+ovyTXC68wg5Xb5Ou2PopmDaVMX6/A1HxziTWc3XdhOF5ocuRF/kfWpZL223Auuu/xvNQDly13DhuVlQiU3gRIP7BSCwCdsQC/K68Q6SgfBklKRiqHquYo/QyNQ== osstest@woking.xci-test.com
+ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAs6FF9nfzWIlLPeYdqNteJBoYJAcgGxQgeNi7FHYDgWNFhoYPlMPXWOuXhgNxA2/vkX9tUMVZaAh+4WTL1iRBW5B/AS/Ek2O7uM2Uq8v68D2aU9/XalLVnIxssr84pewUmKW8hZfjNnRm99RTQ2Knr2BvtwcHqXtdGYdTYCJkel+FPYQ51yXGRU7dS0D59WapkDFU1tH1Y8s+dRZcRZNRJ5f1w/KO1zx1tOrZRkO3fPlEGNZHVUYfpZLPxz0VX8tOeoaOXhKZO8vSp1pD0L/uaD6FOmugMZxbtq9wEjhZciNCq61ynRf2yt2v9DMu4EAzbW/Ws7OBvWtYj/RHcSxKbw== iwj@woking.xci-test.com
+ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2m8+FRm8zaCy4+L2ZLsINt3OiRzDu82JE67b4Xyt3O0+IEyflPgw5zgGH69ypOn2GqYTaiBoiYNoAn9bpUksMk71q+co4gsZJ17Acm0256A3NP46ByT6z6/AKTl58vwwNKSCEAzNru53sXTYw2TcCZUN8A4vXY76OeJNJmCmgBDHCNod9fW6+EOn8ZSU1YjFUBV2UmS2ekKmsGNP5ecLAF1bZ8I13KpKUIDIY+UiG0UMwTWDfQY59SNsz6bCxv9NsxSXL29RS2XHFeIQis7t6hJuyZTT4b9YzjEAxvk8kdGzzK6314kwILibm1O1Y8LLyrYsWK1AvnJQFIhcYXF0EQ== iwj@mariner
+ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEApAkFv1FwknjOoataWvq5SRN/eUHjfQ5gfWnGJpIa4qnT+zAqYuC10BAHu3pHPV6NiedMxud0KcYlu/giQBMVMnYBdb7gWKdK4AQTgxHgvMMWHufa8oTLONLRsvyp1wQADJBzjQSjmo6HHF9faUckZHfJTfRxqLuR/3ENIyl+CRV9G6KfN9fbABejBxdfsbuTHc5ew2JsYxhDJsDFHgMjtrUoHI/d6eBTQDx8GRj8uUor8W+riFpW3whTH9dqloOyrqIke2qGVQlMNmzx5Z04vB1+n95nu9c5SGOZTUT4BQ5FybEANWQsNfJ7b3aMcYgVCVkKuRHSbW8Q4Pyn1Nh31w== ian@liberator
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDY+wyHPeVjiqPzmKZxSxA0fxQ8r0zKMN1cYxJrryE68XzrocprAEqrGR8n3LN3JBApt4kf5gNn4DUdDo6BmCrnTuO4p43ydKJ2BDtWjQJAYdm0g5ttvF3C0A0wnog+jP3WZhTXu40LohKWO5a0If4/SBTkZvKBuSGV4v6wihbeA2Y2aEqwIlfvdSeq96jcbppNXlhWC4bB8VIVU1pa422nTQwpLdaD4qdLi31FEWSqPd2Ro/Z5i/w22M/5wvjYMkUXQcQIn6IsajM6BR56aBGgzIxWGwkxp7iQMPCCXJ4/wTpP1A5lU4k3B8FJCkM9nnSM2koPPr+HSnOKewBqwD1V iwj@osstest
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxArew0DKa3mGEx73adSfaWowq0At3+QbA/LYh/lIw1Wn4YkgxXh2OS1/7NUjNu0IGjnHLelG+HHhUv1fH5JunNJj+GLRsU09dag8LVmbmX0fkebJg6a4iKSXREgoM7NcDn3lA5HXTiNq1Q0To7fb09tZ5wG2inaDvt8PMhVF+CsJXbfPGqJcnKyYosNMCIj6HVwvUl4UAyfyil3CfhklV8Dk1Wd626R93Q39TRrnV3Cm+xVSX2MEfY5mIaqYGeHSPhybqI/h6J6fVTRGNc2Y4oEwWd0ElZbRpCJ5OQx5oV4qMcCh4MSdOC5ZcSv1j3KLaehkwbLjKzpm4hf+2+bkRvua/4nMRWlDNG2C/k05Lt363OJkmUUR2DT+2Cf5P44WmhZeo3ekTN2Z19x8TGGU60UznbscPn/ri0F0bfAtLjlXgxG8W/L7i885uxTrETin6jrY/BBj7G/Fqu6ZffIQvhrww53vv3a/0VYYKBJ9c7HvTI0C5UugbV7oyHbT/2l0= ianc@osstest-ansible-generated
+
+END
+
+HttpsProxyMITMCert= <<'END'
+-----BEGIN CERTIFICATE-----
+MIIDtzCCAp+gAwIBAgIJAKYOfDsTZAKAMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV
+BAYTAlVTMQswCQYDVQQIDAJNQTETMBEGA1UECgwKeGVucHJvamVjdDEZMBcGA1UE
+CwwQeGVucHJvamVjdF9zcXVpZDEmMCQGA1UEAwwdY2FjaGUudGVzdC1sYWIueGVu
+cHJvamVjdC5vcmcwHhcNMTcwMzA5MTI0NTI3WhcNMTkwMzA5MTI0NTI3WjByMQsw
+CQYDVQQGEwJVUzELMAkGA1UECAwCTUExEzARBgNVBAoMCnhlbnByb2plY3QxGTAX
+BgNVBAsMEHhlbnByb2plY3Rfc3F1aWQxJjAkBgNVBAMMHWNhY2hlLnRlc3QtbGFi
+LnhlbnByb2plY3Qub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
+9WoZTo/f0bLYz+BAWGVt5f1Wycj05Mil3DU875Sql80giqU2b80rzwB60Puf/0Sp
+5kx8twY8QvR+JthLS4mfSddDq+W+ZwMbrEpp5E8Le+mdrw4YDkdu9bx47NrWOKqF
+z7Q+WONCq75YAWaa0b1Jp6fqQichNbkCdZY3hvemB7ZI+CFWuirOhzcqrZlS88qu
+iF9W0Oj+0O31ZThNVaINUqfrYO3xIbOwJl6qEcp/cAGvFr7ux3KiAg/Tc69xRn+U
+Heh4kbqOlkE8W5wn+0wiei0sEbgi9zxPVnu1XQY1LBh9zh7gbFt7X9ZKka7YlM4H
++IzRHcZl33kY2aVCNF4c6QIDAQABo1AwTjAdBgNVHQ4EFgQUpkoT3BVZ14oKu6qe
+Tt6aMvZ3sKEwHwYDVR0jBBgwFoAUpkoT3BVZ14oKu6qeTt6aMvZ3sKEwDAYDVR0T
+BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAgKYcfH8ykXgnXJVSv1U7carxV/O/
+Io76jrXi3NtqXSugadzakFEOgjkNAfeOrtmQRAF6JuFhPfhweWvnfYGXV/QrxF7N
+nkCVhvNTsyNM2WDeuBtzj/CBoyk0jBkwxdcG3SLEx/sHKPkTPziFuXcPvITdaP2a
+DC+NgIdL3FmEfzaabbDdrgGRt+/vfvu877atvrX1/B5/NOL+7mXaTR6jzuJRQeXK
+ftnS+ZW/GlbttfovPgFN/FjUioenfS9IS1IBwCJ4d9uHDbgOgL6uhEQ0Iq8Ou68f
+L6qOW4qBJldqxl9VX2RqcJExP0YwrvqmdsdC+xsTmXnOtpUkAYKNrvh7dg==
+-----END CERTIFICATE-----
+END
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 02/16] gitignore: ignore vim swap file

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 425506b..f7e5b77 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,6 @@
 *~
 *.bak
+*.swp
 tmp
 *.tmp
 bisection.ps
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 03/16] ts-xen-build-prep: don't install w3c-dtd-xhtml for Stretch onward

[Wei Liu]

It is not included in Stretch.

That package was installed because libvirt build needed it. However
libvirt builds fine without it in Stretch. It should be fine to just not
install it on Stretch.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build-prep | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 3e98364..6152d0a 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -206,9 +206,12 @@ sub prep () {
                       libglib2.0-dev liblzma-dev pkg-config
                       autoconf automake libtool xsltproc
                       libxml2-utils libxml2-dev
-                      libdevmapper-dev w3c-dtd-xhtml libxml-xpath-perl
+                      libdevmapper-dev libxml-xpath-perl
                       ccache nasm checkpolicy ebtables);
 
+    if ($ho->{Suite} !~ m/stretch/) {
+	push(@packages, "w3c-dtd-xhtml");
+    }
     if ($ho->{Suite} !~ m/squeeze|wheezy/) {
 	push(@packages, qw(ocaml-nox ocaml-findlib));
     }
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 04/16] ts-xen-build-prep: install packages for stretch

[Wei Liu]

Stubdom build needs texinfo.

Libvirt build needs autopoint.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build-prep | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 6152d0a..70a53ca 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -221,6 +221,10 @@ sub prep () {
 	# jessie (>jessie?)
 	push(@packages, "libnl-route-3-dev");
     }
+    if ($ho->{Suite} =~ m/stretch/) {
+        push(@packages, qw(texinfo autopoint libpciaccess-dev));
+    }
+
     target_install_packages($ho, @packages);
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 05/16] mg-debian-installer-update-all: put quotes around arguments

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 mg-debian-installer-update     | 2 +-
 mg-debian-installer-update-all | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index 37cec58..dc807a2 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -26,7 +26,7 @@ set -e -o posix
 
 suite=$1
 arch=$2
-packages="$3"
+packages=$3
 
 site=http://ftp.debian.org/debian/
 sbase=$site/dists/$suite
diff --git a/mg-debian-installer-update-all b/mg-debian-installer-update-all
index d88ebf5..d590b2b 100755
--- a/mg-debian-installer-update-all
+++ b/mg-debian-installer-update-all
@@ -31,5 +31,5 @@ fws=`getconfig DebianNonfreeFirmware`
 arches="arm64 armhf amd64 i386"
 
 for arch in $arches ; do
-    ./mg-debian-installer-update $suite $arch $fws
+    ./mg-debian-installer-update "$suite" "$arch" "$fws"
 done
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 06/16] ts-xen-install: install some packages on stretch

[Wei Liu]

The "route" command is now in that package.

libnl is needed when running xl.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-install | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/ts-xen-install b/ts-xen-install
index ec907c5..d4c25c7 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -57,6 +57,9 @@ sub packages () {
                                libsdl1.2debian libglib2.0-0 liblzma5
 			       qemu-utils
                                netcat-openbsd));
+    if ($ho->{Suite} =~ m/stretch/) {
+        target_install_packages($ho, 'net-tools libnl-route-3-200');
+    }
     if ($ho->{Suite} =~ m/jessie/) {
         target_install_packages($ho, 'libnl-route-3-200');
     }
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 07/16] Debian.pm: use sysvinit-core on stretch

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/Debian.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 845027a..24bc260 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -827,7 +827,7 @@ sub preseed_base ($$$;@) {
 
     # Systemd doesn't honor osstest-confirm-booted service, which
     # breaks ts-leak-check.  Fall back to SysV init for now.
-    if ( $suite =~ /jessie/ ) {
+    if ( $suite =~ /jessie|stretch/ ) {
        preseed_hook_command($ho, 'late_command', $sfx, <<END)
 in-target apt-get install -y sysvinit-core
 END
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 08/16] ts-leak-check: suppress systemd-shim, which leaks in stretch

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-leak-check | 1 +
 1 file changed, 1 insertion(+)

diff --git a/ts-leak-check b/ts-leak-check
index 678d069..41e6245 100755
--- a/ts-leak-check
+++ b/ts-leak-check
@@ -202,6 +202,7 @@ xenstore /vm
 xenstore /libxl
 
 process .* udevd
+process .* /.+/systemd-shim
 
 file /var/run/xenstored/db
 file /var/run/xenstored/db.debug
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 09/16] ts-host-install: don't use the new nic naming scheme

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-host-install | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ts-host-install b/ts-host-install
index 20b8a60..530bc5c 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -271,6 +271,10 @@ END
     # why this is repeated.
     push @hocmdline, "console=$console" unless $console eq "NONE";
 
+    # Don't use "Predictable Network Interface Names"
+    # https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
+    push @hocmdline, "net.ifnames=0" if $ho->{Suite} =~ m/stretch/;
+
     push @hocmdline,
         get_host_property($ho, "linux-boot-append $ho->{Suite}", ''),
         get_host_property($ho, "linux-boot-append $ho->{Suite} $r{arch}", '');
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own

[Wei Liu]

The original extra= was not removed, so there were two extra= in the
resulting config file.

It wasn't a problem for xl because the second extra= took precedence.
However libvirt tests would only pick up the first extra= --  they
worked by chance.

Fix this issue by removing the first extra=.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-fixup | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/ts-debian-fixup b/ts-debian-fixup
index f29971d..d476467 100755
--- a/ts-debian-fixup
+++ b/ts-debian-fixup
@@ -175,6 +175,8 @@ sub otherfixupcfg () {
         $extra .= " iommu=soft";
     }
 
+
+    $cfg =~ s/^extra.*//mg;
     $cfg .= "\nextra='$extra'\n";
 };
 
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Wei Liu]

See code comment for explanation.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-fixup | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/ts-debian-fixup b/ts-debian-fixup
index d476467..fe37c30 100755
--- a/ts-debian-fixup
+++ b/ts-debian-fixup
@@ -176,6 +176,18 @@ sub otherfixupcfg () {
     }
 
 
+    # There might be stale entries in /etc/initramfs-tools/conf.d/resume
+    # which get stored in the initramfs. That introduces delay in guest booting
+    # which might cause tests to fail.
+    #
+    # This is particularly prominent in systemd enabled releases when it tries
+    # to scan for the inexistent device(s) for a long time.
+    #
+    # Override that in kernel command line with the correct swap partition.
+    $cfg =~ m/'phy:.+-swap,(xvda\d+),.*'/;
+    $extra .= " resume=/dev/$1";
+    logm("change resume device to $1");
+
     $cfg =~ s/^extra.*//mg;
     $cfg .= "\nextra='$extra'\n";
 };
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 12/16] make-flight: don't test pvgrub for Xen XXX

[Wei Liu]

XXX Need to pin down the version of Xen when the upgrade to stretch is
complete because osstest configuration is branched for each version.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 make-flight | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/make-flight b/make-flight
index ecce3c0..b95f7e3 100755
--- a/make-flight
+++ b/make-flight
@@ -823,7 +823,22 @@ test_matrix_do_one () {
   #do_passthrough_tests
 
   do_pygrub_tests
-  do_pvgrub_tests
+
+  # pvgrub1 tests for version < XXX only
+  case "$xenbranch" in
+  xen-4.3-testing) test_pvgrub=y ;;
+  xen-4.4-testing) test_pvgrub=y ;;
+  xen-4.5-testing) test_pvgrub=y ;;
+  xen-4.6-testing) test_pvgrub=y ;;
+  xen-4.7-testing) test_pvgrub=y ;;
+  xen-4.8-testing) test_pvgrub=y ;;
+  xen-4.9-testing) test_pvgrub=y ;;
+  *)               test_pvgrub=n ;;
+  esac
+
+  if [ x$test_pvgrub = xy ]; then
+      do_pvgrub_tests
+  fi
 
   do_xtf_tests
   do_livepatch_tests
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 13/16] ts-debian-hvm-install: disable new nic naming scheme

[Wei Liu]

This is required to fix nested hvm test. The L1 host is installed by
this script. We want the L1 host to not use the new nic naming scheme.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-hvm-install | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 54d5d1c..7f8251d 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -114,6 +114,18 @@ set -ex
 in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list
 END
 
+    # Do not use "Predictable Network Interface Names" -- this can break
+    # nested HVM tests.
+    # https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
+    #
+    # See also
+    # https://www.debian.org/releases/stretch/example-preseed.txt
+    my $netifnames = "";
+    $netifnames = "\nd-i debian-installer/add-kernel-opts string net.ifnames=0\n"
+        if $ho->{Suite} =~ m/stretch/;
+
+    $preseed_file .= "$netifnames";
+
     $preseed_file .= preseed_hook_cmds();
 
     return $preseed_file;
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 14/16] ts-xen-build-prep: install e2fslibs-dev

[Wei Liu]

The in-tree libfsimage ext2fs implementation can't handle 64bit
enabled ext4, which is the default in stretch.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build-prep | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 70a53ca..4224601 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -225,6 +225,12 @@ sub prep () {
         push(@packages, qw(texinfo autopoint libpciaccess-dev));
     }
 
+    # The in-tree ext4 support in libfsimage can't cope with 64bit ext4 on
+    # 32bit build. Use the packaged library.
+    if ($ho->{Suite} =~ m/stretch/) {
+        push(@packages, qw(e2fslibs-dev));
+    }
+
     target_install_packages($ho, @packages);
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 15/16] TestSupport: add dpkg option when installing packages

[Wei Liu]

Upgrading configuration file of nbd-client is controlled by dpkg in
stretch. Add dpkg option to keep old configuration file(s).

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/TestSupport.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 76730f1..363e888 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -524,7 +524,8 @@ sub target_run_apt {
     my ($ho, @aptopts) = @_;
     target_cmd_root($ho,
         "DEBIAN_PRIORITY=critical UCF_FORCE_CONFFOLD=y \\
-            with-lock-ex -w /var/lock/osstest-apt apt-get @aptopts", 3000);
+            with-lock-ex -w /var/lock/osstest-apt \\
+            apt-get -o Dpkg::Options::=\"--force-confold\" @aptopts", 3000);
 }
 sub target_install_packages {
     my ($ho, @packages) = @_;
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 16/16] ts-guests-nbd-mirror: make it work with stretch

[Wei Liu]

On the server side, only add oldstyle= and port= on jessie. Stretch
doesn't support or need those anymore.  Prune check for older versions
of Debian.

On the client side, generate new style configuration file.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-guests-nbd-mirror | 47 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 43 insertions(+), 4 deletions(-)

diff --git a/ts-guests-nbd-mirror b/ts-guests-nbd-mirror
index ca8300d..d7b07e2 100755
--- a/ts-guests-nbd-mirror
+++ b/ts-guests-nbd-mirror
@@ -60,15 +60,19 @@ sub configserver () {
 [generic]
     user = root
 END
-    $scfg .= <<END unless $sho->{Suite} =~ m/sarge|lenny|squeeze/;
+
+    $scfg .= <<END if $sho->{Suite} =~ m/jessie/;
     oldstyle = true
 END
+
     foreach my $v (@vols) {
 	$v->{Port}= unique_incrementing_runvar("${srvhost}_nextport",4000);
 	$v->{Path}= "/dev/$v->{Gho}{Vg}/$v->{Lv}";
 	$scfg.=<<END;
 [export$v->{Ix}]
     exportname = $v->{Path}
+END
+	$scfg.=<<END if $sho->{Suite} =~ m/jessie/;
     port = $v->{Port}
 END
     }
@@ -79,9 +83,7 @@ END
     target_install_packages($sho, qw(nbd-server));
 }
 
-sub configclient () {
-    target_cmd_root($cho, "dpkg --purge nbd-client ||:");
-
+sub configclient_pre_stretch () {
     my $mydaemon= '/root/nbd-client-async';
     target_putfilecontents_root_stash($cho,10,<<'END',$mydaemon);
 #!/bin/sh
@@ -107,7 +109,44 @@ NBD_PORT[$v->{Ix}]=$v->{Port}
 END
     }
     target_putfilecontents_root_stash($cho,10,$ccfg,"/etc/nbd-client");
+}
+
+sub configclient_stretch () {
+    my $ccfg = <<END;
+# generated by $0
+END
+
+    foreach my $v (@vols) {
+	my $nbddev = "nbd$v->{Ix}";
+	$ccfg .= <<END;
+$nbddev $sho->{Name} export$v->{Ix}
+END
+    }
+
+    target_putfilecontents_root_stash($cho,10,$ccfg,"/etc/nbdtab");
+}
+
+sub configclient () {
+    target_cmd_root($cho, "dpkg --purge nbd-client ||:");
+
+    if ($cho->{Suite} !~ m/stretch/) {
+        configclient_pre_stretch();
+    } else {
+        configclient_stretch();
+    }
+
     target_install_packages($cho, qw(nbd-client));
+
+    if ($cho->{Suite} =~ m/stretch/) {
+	foreach my $v (@vols) {
+	    my $nbddev = "nbd$v->{Ix}";
+	    target_cmd_root($cho, <<END);
+mkdir -p /dev/$v->{Gho}{Vg}
+if ! test -L $v->{Path}; then ln -s /dev/$nbddev $v->{Path}; fi
+END
+	}
+	target_cmd_root($cho, "/etc/init.d/nbd-client start");
+    }
 }
 
 sub shuffleconfigs () {
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 03/16] ts-xen-build-prep: don't install w3c-dtd-xhtml for Stretch onward

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 03/16] ts-xen-build-prep: don't install w3c-dtd-xhtml for Stretch onward"):
> It is not included in Stretch.
> 
> That package was installed because libvirt build needed it. However
> libvirt builds fine without it in Stretch. It should be fine to just not
> install it on Stretch.
...
> +    if ($ho->{Suite} !~ m/stretch/) {
> +	push(@packages, "w3c-dtd-xhtml");
> +    }

This is not correct because it will break when buster is released.
Instead you should list all still-supported old suites for which (you
think) it is required.

I suggest for `still-supported' we arbitrarily go back to `squeeze',
so you want   if  ... m/squeeze|jessie/

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 04/16] ts-xen-build-prep: install packages for stretch

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 04/16] ts-xen-build-prep: install packages for stretch"):
> Stubdom build needs texinfo.

Same comment as my previous patch.  You should only mention old
release names in these kind of tests, unless you know that the requirement
is specific to only stretch and not future releases.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 02/16] gitignore: ignore vim swap file

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 02/16] gitignore: ignore vim swap file"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Although, you may find your life improved by putting this in your
~/.config/git/ignore.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 01/16] XXX add a stretch config based on production-config

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 01/16] XXX add a stretch config based on production-config"):
> diff -ub production-config production-config-stretch

The changes LGTM but obviously this ought to go straight into
`production-config'.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 05/16] mg-debian-installer-update-all: put quotes around arguments

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 05/16] mg-debian-installer-update-all: put quotes around arguments"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
...
>  suite=$1
>  arch=$2
> -packages="$3"
> +packages=$3

Not sure why this needless style change, but if you did it
deliberately I don't really mind...

>  site=http://ftp.debian.org/debian/
>  sbase=$site/dists/$suite
> diff --git a/mg-debian-installer-update-all b/mg-debian-installer-update-all
> index d88ebf5..d590b2b 100755
> --- a/mg-debian-installer-update-all
> +++ b/mg-debian-installer-update-all
> @@ -31,5 +31,5 @@ fws=`getconfig DebianNonfreeFirmware`
>  arches="arm64 armhf amd64 i386"
>  
>  for arch in $arches ; do
> -    ./mg-debian-installer-update $suite $arch $fws
> +    ./mg-debian-installer-update "$suite" "$arch" "$fws"
>  done

This hunk LGTM.  Although the "" around $suite and $arch are
unnecessary I don't really mind them.

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 07/16] Debian.pm: use sysvinit-core on stretch

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 07/16] Debian.pm: use sysvinit-core on stretch"):
...
> diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
> index 845027a..24bc260 100644
> --- a/Osstest/Debian.pm
> +++ b/Osstest/Debian.pm
> @@ -827,7 +827,7 @@ sub preseed_base ($$$;@) {
>  
>      # Systemd doesn't honor osstest-confirm-booted service, which
>      # breaks ts-leak-check.  Fall back to SysV init for now.
> -    if ( $suite =~ /jessie/ ) {
> +    if ( $suite =~ /jessie|stretch/ ) {
>         preseed_hook_command($ho, 'late_command', $sfx, <<END)
>  in-target apt-get install -y sysvinit-core

I think we should probably recognise that this is not going to change,
and switch this from a workaround to a policy decision.  Ie,

  if ... !~ m/squeeze/

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 08/16] ts-leak-check: suppress systemd-shim, which leaks in stretch

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 08/16] ts-leak-check: suppress systemd-shim, which leaks in stretch"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 09/16] ts-host-install: don't use the new nic naming scheme

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 09/16] ts-host-install: don't use the new nic naming scheme"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 13/16] ts-debian-hvm-install: disable new nic naming scheme

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 13/16] ts-debian-hvm-install: disable new nic naming scheme"):
> This is required to fix nested hvm test. The L1 host is installed by
> this script. We want the L1 host to not use the new nic naming scheme.

The principle is fine.

> +    # Do not use "Predictable Network Interface Names" -- this can break
> +    # nested HVM tests.
> +    # https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/
> +    #
> +    # See also
> +    # https://www.debian.org/releases/stretch/example-preseed.txt
> +    my $netifnames = "";
> +    $netifnames = "\nd-i debian-installer/add-kernel-opts string net.ifnames=0\n"
> +        if $ho->{Suite} =~ m/stretch/;

Please use a <<END here document.  In fact, I think you should copy
the style seen elsewhere nearby, so something like

       $preseed_file .= <<END if $ho->{Suite}....
  d-i debian... etc. etc.
  END

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own"):
> The original extra= was not removed, so there were two extra= in the
> resulting config file.

What is the original extra= ?  Why should we not combine them ?

> It wasn't a problem for xl because the second extra= took precedence.
> However libvirt tests would only pick up the first extra= --  they
> worked by chance.

That's odd.  Is this to do with the xl -> libxl config converter ?
It seems to me that that converter should interpret xl config files
the same way xl does.  (Also xm did the same: last setting wins.)

>  
> +
> +    $cfg =~ s/^extra.*//mg;
>      $cfg .= "\nextra='$extra'\n";

Also, accidental whitespace change.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> See code comment for explanation.
...
> +    # There might be stale entries in /etc/initramfs-tools/conf.d/resume
> +    # which get stored in the initramfs. That introduces delay in guest booting
> +    # which might cause tests to fail.

Why might there be such stale entries ?

> +    # This is particularly prominent in systemd enabled releases when it tries
> +    # to scan for the inexistent device(s) for a long time.

Why is this not a bug in systemd ?  If there are workarounds in
osstest for Debian, I usually want a bug in the BTS, and a suite limit
on the workaround so that we poke Debian again in 2 years time...

> +    # Override that in kernel command line with the correct swap partition.
> +    $cfg =~ m/'phy:.+-swap,(xvda\d+),.*'/;
> +    $extra .= " resume=/dev/$1";
> +    logm("change resume device to $1");

Alternatively, if indeed this is necessary and not due to bugs,
perhaps it should be done by xen-tools ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 16/16] ts-guests-nbd-mirror: make it work with stretch

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH 16/16] ts-guests-nbd-mirror: make it work with stretch"):
> On the server side, only add oldstyle= and port= on jessie. Stretch
> doesn't support or need those anymore.

See my earlier comments about old vs new Debian suite names.

>  Prune check for older versions of Debian.

Why ?  Certainly I can see no justification for cutting off squeeze.
Perhaps someone is still trying to test with squeeze.  Also, doing
this in the same patch is quite confusing.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Wei Liu]

On Fri, Oct 20, 2017 at 12:05:55PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> > See code comment for explanation.
> ...
> > +    # There might be stale entries in /etc/initramfs-tools/conf.d/resume
> > +    # which get stored in the initramfs. That introduces delay in guest booting
> > +    # which might cause tests to fail.
> 
> Why might there be such stale entries ?

The ramdisk is taken from the host, which contains that file. The resume
device is going to point to the swap partition in the host.

> 
> > +    # This is particularly prominent in systemd enabled releases when it tries
> > +    # to scan for the inexistent device(s) for a long time.
> 
> Why is this not a bug in systemd ?  If there are workarounds in
> osstest for Debian, I usually want a bug in the BTS, and a suite limit
> on the workaround so that we poke Debian again in 2 years time...
> 

It has always been like that since forever. It appears that stretch will
try a lot harder than jessie (which only tries once). Now thinking about
it I can't say for sure whether that's due to the new kernel, or systemd
or both.

> > +    # Override that in kernel command line with the correct swap partition.
> > +    $cfg =~ m/'phy:.+-swap,(xvda\d+),.*'/;
> > +    $extra .= " resume=/dev/$1";
> > +    logm("change resume device to $1");
> 
> Alternatively, if indeed this is necessary and not due to bugs,
> perhaps it should be done by xen-tools ?
> 

No, see the first paragraph. xen-tools can't be expected to modify the
ramdisk.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own

[Wei Liu]

On Fri, Oct 20, 2017 at 12:03:24PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own"):
> > The original extra= was not removed, so there were two extra= in the
> > resulting config file.
> 
> What is the original extra= ?  Why should we not combine them ?

The original extra= is generated by xen-create-image. It has the content
"elevator=noop". It doesn't seem too useful to me. But I'm fine with
combination them.

> 
> > It wasn't a problem for xl because the second extra= took precedence.
> > However libvirt tests would only pick up the first extra= --  they
> > worked by chance.
> 
> That's odd.  Is this to do with the xl -> libxl config converter ?
> It seems to me that that converter should interpret xl config files
> the same way xl does.  (Also xm did the same: last setting wins.)
> 

Yes, the converter only picks up the first.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 01/16] XXX add a stretch config based on production-config

[Wei Liu]

On Fri, Oct 20, 2017 at 11:54:44AM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH 01/16] XXX add a stretch config based on production-config"):
> > diff -ub production-config production-config-stretch
> 
> The changes LGTM but obviously this ought to go straight into
> `production-config'.

I kept two files so that I can switch easily between them.

The final incarnation of this patch is going to be a bit different -- I
want to patch Osstest.pm to switch the default suite.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 16/16] ts-guests-nbd-mirror: make it work with stretch

[Wei Liu]

On Fri, Oct 20, 2017 at 12:09:35PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH 16/16] ts-guests-nbd-mirror: make it work with stretch"):
> > On the server side, only add oldstyle= and port= on jessie. Stretch
> > doesn't support or need those anymore.
> 
> See my earlier comments about old vs new Debian suite names.
> 

This is the correct way to handle this. Only Jessie will need it because
older suite doesn't under new style (from their PoV) and newer suite
doesn't understand old style (from their PoV).

> >  Prune check for older versions of Debian.
> 
> Why ?  Certainly I can see no justification for cutting off squeeze.
> Perhaps someone is still trying to test with squeeze.  Also, doing
> this in the same patch is quite confusing.
> 

It won't have any effect because the end result is the same. Squeeze
will still work.

Note the flip from "unless" to "if".

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 16/16] ts-guests-nbd-mirror: make it work with stretch

[Wei Liu]

On Fri, Oct 20, 2017 at 12:24:43PM +0100, Wei Liu wrote:
> On Fri, Oct 20, 2017 at 12:09:35PM +0100, Ian Jackson wrote:
> > Wei Liu writes ("[OSSTEST PATCH 16/16] ts-guests-nbd-mirror: make it work with stretch"):
> > > On the server side, only add oldstyle= and port= on jessie. Stretch
> > > doesn't support or need those anymore.
> > 
> > See my earlier comments about old vs new Debian suite names.
> > 
> 
> This is the correct way to handle this. Only Jessie will need it because
> older suite doesn't under new style (from their PoV) and newer suite
> doesn't understand old style (from their PoV).
> 

Correction: I forgot to count wheezy. Wheezy should be grouped together
with Jessie. The condition should be updated accordingly.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 03/16] ts-xen-build-prep: don't install w3c-dtd-xhtml for Stretch onward

[Wei Liu]

On Fri, Oct 20, 2017 at 11:50:11AM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH 03/16] ts-xen-build-prep: don't install w3c-dtd-xhtml for Stretch onward"):
> > It is not included in Stretch.
> > 
> > That package was installed because libvirt build needed it. However
> > libvirt builds fine without it in Stretch. It should be fine to just not
> > install it on Stretch.
> ...
> > +    if ($ho->{Suite} !~ m/stretch/) {
> > +	push(@packages, "w3c-dtd-xhtml");
> > +    }
> 
> This is not correct because it will break when buster is released.
> Instead you should list all still-supported old suites for which (you
> think) it is required.
> 
> I suggest for `still-supported' we arbitrarily go back to `squeeze',
> so you want   if  ... m/squeeze|jessie/
> 

No problem.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Ian Jackson]

Wei Liu writes ("Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> On Fri, Oct 20, 2017 at 12:05:55PM +0100, Ian Jackson wrote:
> > Wei Liu writes ("[OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> > > See code comment for explanation.
> > ...
> > > +    # There might be stale entries in /etc/initramfs-tools/conf.d/resume
> > > +    # which get stored in the initramfs. That introduces delay in guest booting
> > > +    # which might cause tests to fail.
> > 
> > Why might there be such stale entries ?
> 
> The ramdisk is taken from the host, which contains that file. The resume
> device is going to point to the swap partition in the host.

Ah.  Hrm.  I guess this was always a bit of a bodge.

> > > +    # Override that in kernel command line with the correct swap partition.
> > > +    $cfg =~ m/'phy:.+-swap,(xvda\d+),.*'/;
> > > +    $extra .= " resume=/dev/$1";
> > > +    logm("change resume device to $1");
> > 
> > Alternatively, if indeed this is necessary and not due to bugs,
> > perhaps it should be done by xen-tools ?
> 
> No, see the first paragraph. xen-tools can't be expected to modify the
> ramdisk.

Maybe it should set the appropriate "extra" though.  IIRC xen-tools
does implement this sort-of-trick of reusing the hosts initrd.  So why
does this bug not happen with other users of xen-tools ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own

[Ian Jackson]

Wei Liu writes ("Re: [OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own"):
> On Fri, Oct 20, 2017 at 12:03:24PM +0100, Ian Jackson wrote:
> > Wei Liu writes ("[OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own"):
> > > The original extra= was not removed, so there were two extra= in the
> > > resulting config file.
> > 
> > What is the original extra= ?  Why should we not combine them ?
> 
> The original extra= is generated by xen-create-image. It has the content
> "elevator=noop". It doesn't seem too useful to me. But I'm fine with
> combination them.

I think they should be combined.  And that "elevator=noop" doesn't
sound unreasonable.

> > > It wasn't a problem for xl because the second extra= took precedence.
> > > However libvirt tests would only pick up the first extra= --  they
> > > worked by chance.
> > 
> > That's odd.  Is this to do with the xl -> libxl config converter ?
> > It seems to me that that converter should interpret xl config files
> > the same way xl does.  (Also xm did the same: last setting wins.)
> 
> Yes, the converter only picks up the first.

This is a bug, then.  Where should we file it ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own

[Wei Liu]

On Fri, Oct 20, 2017 at 01:52:50PM +0100, Ian Jackson wrote:
> Wei Liu writes ("Re: [OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own"):
> > On Fri, Oct 20, 2017 at 12:03:24PM +0100, Ian Jackson wrote:
> > > Wei Liu writes ("[OSSTEST PATCH 10/16] ts-debian-fixup: remove extra= before appending our own"):
> > > > The original extra= was not removed, so there were two extra= in the
> > > > resulting config file.
> > > 
> > > What is the original extra= ?  Why should we not combine them ?
> > 
> > The original extra= is generated by xen-create-image. It has the content
> > "elevator=noop". It doesn't seem too useful to me. But I'm fine with
> > combination them.
> 
> I think they should be combined.  And that "elevator=noop" doesn't
> sound unreasonable.
> 
> > > > It wasn't a problem for xl because the second extra= took precedence.
> > > > However libvirt tests would only pick up the first extra= --  they
> > > > worked by chance.
> > > 
> > > That's odd.  Is this to do with the xl -> libxl config converter ?
> > > It seems to me that that converter should interpret xl config files
> > > the same way xl does.  (Also xm did the same: last setting wins.)
> > 
> > Yes, the converter only picks up the first.
> 
> This is a bug, then.  Where should we file it ?
> 

Not sure -- maybe email Jim and CC libvirt list?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Wei Liu]

On Fri, Oct 20, 2017 at 01:35:41PM +0100, Ian Jackson wrote:
> Wei Liu writes ("Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> > On Fri, Oct 20, 2017 at 12:05:55PM +0100, Ian Jackson wrote:
> > > Wei Liu writes ("[OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> > > > See code comment for explanation.
> > > ...
> > > > +    # There might be stale entries in /etc/initramfs-tools/conf.d/resume
> > > > +    # which get stored in the initramfs. That introduces delay in guest booting
> > > > +    # which might cause tests to fail.
> > > 
> > > Why might there be such stale entries ?
> > 
> > The ramdisk is taken from the host, which contains that file. The resume
> > device is going to point to the swap partition in the host.
> 
> Ah.  Hrm.  I guess this was always a bit of a bodge.
> 
> > > > +    # Override that in kernel command line with the correct swap partition.
> > > > +    $cfg =~ m/'phy:.+-swap,(xvda\d+),.*'/;
> > > > +    $extra .= " resume=/dev/$1";
> > > > +    logm("change resume device to $1");
> > > 
> > > Alternatively, if indeed this is necessary and not due to bugs,
> > > perhaps it should be done by xen-tools ?
> > 
> > No, see the first paragraph. xen-tools can't be expected to modify the
> > ramdisk.
> 
> Maybe it should set the appropriate "extra" though.  IIRC xen-tools
> does implement this sort-of-trick of reusing the hosts initrd.  So why
> does this bug not happen with other users of xen-tools ?
> 

I wouldn't call this behaviour a bug.  Most people won't notice it
because the guest will boot eventually. It is not like this will cause
the guest to crash.

Osstest sets a timeout. We can bump the timeout but that's worse than
the solution here.

There is no option in xen-create-image to manipulate extra=. The only
viable solution is to provide a new template. That seems overkill and
would require more code.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Ian Jackson]

Wei Liu writes ("Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> I wouldn't call this behaviour a bug.  Most people won't notice it
> because the guest will boot eventually. It is not like this will cause
> the guest to crash.

I don't think the guest booting "eventually" after some timeout is
correct.

> There is no option in xen-create-image to manipulate extra=. The only
> viable solution is to provide a new template. That seems overkill and
> would require more code.

What you're saying is that this infelicity in xen-tools is not
particularly easy to fix.  That doesn't mean it shouldn't be reported
as a bug.

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Wei Liu]

On Fri, Oct 20, 2017 at 02:20:29PM +0100, Ian Jackson wrote:
> Wei Liu writes ("Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> > I wouldn't call this behaviour a bug.  Most people won't notice it
> > because the guest will boot eventually. It is not like this will cause
> > the guest to crash.
> 
> I don't think the guest booting "eventually" after some timeout is
> correct.
>

The OS is honoring whatever is written in the resume file. That's the
correct behaviour to me.

Booting after trying is better than not booting at all.  That parameter
isn't vital to a functional system. This has always been the case since
forever. It is only the "timeout" in the past is shorter so osstest
never noticed it.

What is not correct is the content of the file. We can't blame the guest
OS for that.

> > There is no option in xen-create-image to manipulate extra=. The only
> > viable solution is to provide a new template. That seems overkill and
> > would require more code.
> 
> What you're saying is that this infelicity in xen-tools is not
> particularly easy to fix.  That doesn't mean it shouldn't be reported
> as a bug.

There is nothing to fix. System administrator should provide a proper
template -- that's what xen-tools developers most likely to say.

It would require a lot more code in osstest to fix the template, and the
essence of such patch would still be the same.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Ian Jackson]

Wei Liu writes ("Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> On Fri, Oct 20, 2017 at 02:20:29PM +0100, Ian Jackson wrote:
> > I don't think the guest booting "eventually" after some timeout is
> > correct.
> 
> The OS is honoring whatever is written in the resume file. That's the
> correct behaviour to me.
> 
> Booting after trying is better than not booting at all.  That parameter
> isn't vital to a functional system. This has always been the case since
> forever. It is only the "timeout" in the past is shorter so osstest
> never noticed it.
> 
> What is not correct is the content of the file. We can't blame the guest
> OS for that.

I agree with all of this.  But, the overall behaviour is not correct.
Even though all of the things that osstest specified to
xen-create-image were correct.  Therefore there is a bug.

> > > There is no option in xen-create-image to manipulate extra=. The only
> > > viable solution is to provide a new template. That seems overkill and
> > > would require more code.
> > 
> > What you're saying is that this infelicity in xen-tools is not
> > particularly easy to fix.  That doesn't mean it shouldn't be reported
> > as a bug.
> 
> There is nothing to fix. System administrator should provide a proper
> template -- that's what xen-tools developers most likely to say.

I'm not sure what you mean here.  We are using the config template
provided by xen-tools.  The reuse of the host's initrd is suggested by
the xen-create-image documentation.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Wei Liu]

On Fri, Oct 20, 2017 at 02:42:53PM +0100, Ian Jackson wrote:
> Wei Liu writes ("Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> > On Fri, Oct 20, 2017 at 02:20:29PM +0100, Ian Jackson wrote:
> > > I don't think the guest booting "eventually" after some timeout is
> > > correct.
> > 
> > The OS is honoring whatever is written in the resume file. That's the
> > correct behaviour to me.
> > 
> > Booting after trying is better than not booting at all.  That parameter
> > isn't vital to a functional system. This has always been the case since
> > forever. It is only the "timeout" in the past is shorter so osstest
> > never noticed it.
> > 
> > What is not correct is the content of the file. We can't blame the guest
> > OS for that.
> 
> I agree with all of this.  But, the overall behaviour is not correct.
> Even though all of the things that osstest specified to
> xen-create-image were correct.  Therefore there is a bug.
> 
> > > > There is no option in xen-create-image to manipulate extra=. The only
> > > > viable solution is to provide a new template. That seems overkill and
> > > > would require more code.
> > > 
> > > What you're saying is that this infelicity in xen-tools is not
> > > particularly easy to fix.  That doesn't mean it shouldn't be reported
> > > as a bug.
> > 
> > There is nothing to fix. System administrator should provide a proper
> > template -- that's what xen-tools developers most likely to say.
> 
> I'm not sure what you mean here.  We are using the config template
> provided by xen-tools.  The reuse of the host's initrd is suggested by
> the xen-create-image documentation.
> 

TBH I don't know what to write in a report. What is the expected
behaviour?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Ian Jackson]

Wei Liu writes ("Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> On Fri, Oct 20, 2017 at 02:42:53PM +0100, Ian Jackson wrote:
> > Wei Liu writes ("Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> > I'm not sure what you mean here.  We are using the config template
> > provided by xen-tools.  The reuse of the host's initrd is suggested by
> > the xen-create-image documentation.
> 
> TBH I don't know what to write in a report. What is the expected
> behaviour?

How about this ?  Adjust to taste, correcting any
lies/misunderstandings, maybe adding references, etc.


Steps to reproduce:

  Follow instructions in xen-create-image, including suggestion to
  re-use host's initrd, to make a guest running stretch.  Try to boot
  it.

Expected behaviour:

  Guest boots promptly.

Observed behaviour:

  Guest pauses for a long time during boot waiting for the "resume
  device" to become available.

Analysis:

  This is because the initrd has the host's swap device configured.

  With stretch, and systemd, the delay is longer; earler Debian
  releases had a similar issue but the shorter timeout meant it was a
  less serious problem.

Suggested remedy:

  It is unfortunate that the initrd has the resume device baked into
  it.  Normally this kind of thing (eg root=) comes via the command
  line.  So maybe the bug is in the initrd generation.

  Alternatively, maybe xen-create-image should write a setting "extra"
  into the domain config file "resume=<swap device>".


Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device

[Wei Liu]

On Fri, Oct 20, 2017 at 04:10:16PM +0100, Ian Jackson wrote:
> Wei Liu writes ("Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> > On Fri, Oct 20, 2017 at 02:42:53PM +0100, Ian Jackson wrote:
> > > Wei Liu writes ("Re: [OSSTEST PATCH 11/16] ts-debian-fixup: use correct resume device"):
> > > I'm not sure what you mean here.  We are using the config template
> > > provided by xen-tools.  The reuse of the host's initrd is suggested by
> > > the xen-create-image documentation.
> > 
> > TBH I don't know what to write in a report. What is the expected
> > behaviour?
> 
> How about this ?  Adjust to taste, correcting any
> lies/misunderstandings, maybe adding references, etc.

Actually, there is already a Debian bug. See #784810.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 00/16] Upgrade to Stretch

[Wei Liu]

On Fri, Oct 20, 2017 at 11:38:24AM +0100, Wei Liu wrote:
> 3. The unstability with Arndale boards' nic is more prominent. Or worse -- they
>    have become completely unusable. I don't have enough data yet. We might
>    need to work around this, but I'm not sure how to do that yet.

It appears arndale board network dongles are completely unusable with
stretch. Luckily it is not the first time we have this sort of problem.
There is already facility in osstest to force assigning a mac address.
I'm testing a patch now.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 00/16] Upgrade to Stretch

[Wei Liu]

On Fri, Oct 20, 2017 at 11:38:24AM +0100, Wei Liu wrote:
> 6. Rumprun doesn't build due to its build system can't cope. This should be
>    fixed in rumprun.

There are two bugs:

https://github.com/rumpkernel/rumprun/issues/99 (also numerous
duplicates)
https://github.com/rumpkernel/rumprun/issues/101

I have taken a stab at the first one; the second one really needs Antti
to fix -- the fix needs to be in netbsd first then gets pulled in to
src-netbsd used by rumprun.

No reply from Antti has been made in the past 4 months.  If we don't
hear back from Antti, there is no way we can fix that. I don't have
commit access to src-netbsd. I will try to ask on #rumpkernel.  But if
nothing is going to happen I propose we drop the rumprun tests.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel