[PATCH 0/2] osstest: update FreeBSD guest tests

[PATCH 0/2] osstest: update FreeBSD guest tests

[Roger Pau Monne]

Hello,

The following series adds FreeBSD 11 and 12 guests tests to osstest.
ATM this is only tested on amd64, since the i386 versions had a bug.

The result can be seen at:

http://logs.test-lab.xenproject.org/osstest/logs/150428/

Note this flight has been generated without using the freebsd-{11,12}
hostflags and with the following env variable set:

OSSTEST_JOBS_ONLY=^(.*)freebsd(.*)$,build-(amd64|i386),build-(amd64|i386)-pvops

In order to limit the number of tests run. The runvar difference can be
see in patch #2.

Roger Pau Monne (2):
  freebsd10: refactor code to generate jobs
  freebsd: add FreeBSD 11 and 12 guest jobs

 make-flight | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

-- 
2.26.2

[PATCH 1/2] freebsd10: refactor code to generate jobs

[Roger Pau Monne]

Slightly adjust the code to generate the freebsd10 jobs in order to
avoid exiting early from the function if the dom0 arch is different
than i386. No functional change expected, the freebsd10 jobs are still
limited to run on an i386 dom0. No runvar diff created as part of this
change.

This is a preparatory change for adding new FreeBSD 11 and 12 jobs
that will instead use an amd64 dom0.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
 make-flight | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/make-flight b/make-flight
index a361bcb1..af74bb4e 100755
--- a/make-flight
+++ b/make-flight
@@ -230,19 +230,18 @@ test_matrix_branch_filter_callback () {
 
 do_freebsd_tests () {
 
-  if [ $xenarch != amd64 -o $dom0arch != i386 -o "$kern" != "" ]; then
+  if [ $xenarch != amd64 -o "$kern" != "" ]; then
     return
   fi
 
-  for freebsdarch in amd64 i386; do
-
- job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-freebsd10-$freebsdarch \
-                        test-freebsd xl $xenarch $dom0arch \
-                        freebsd_arch=$freebsdarch \
+  if [ $dom0arch == i386 ]; then
+    for freebsdarch in amd64 i386; do
+      job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-freebsd10-$freebsdarch \
+                      test-freebsd xl $xenarch $dom0arch freebsd_arch=$freebsdarch \
  freebsd_image=${FREEBSD_IMAGE_PREFIX-FreeBSD-10.1-CUSTOM-}$freebsdarch${FREEBSD_IMAGE_SUFFIX--20150525.raw.xz} \
-                        all_hostflags=$most_hostflags,freebsd-10
-
-  done
+                      all_hostflags=$most_hostflags,freebsd-10
+    done
+  fi
 }
 
 do_hvm_winxp_tests () {
-- 
2.26.2

[PATCH 2/2] freebsd: add FreeBSD 11 and 12 guest jobs

[Roger Pau Monne]

Those are based on the upstream images and are run on an amd64 dom0.
The runvar difference is:

+test-amd64-amd64-qemuu-freebsd11-amd64 all_host_di_version 2020-02-10
+test-amd64-amd64-qemuu-freebsd12-amd64 all_host_di_version 2020-02-10
+test-amd64-amd64-qemuu-freebsd11-amd64 all_host_suite      stretch
+test-amd64-amd64-qemuu-freebsd12-amd64 all_host_suite      stretch
+test-amd64-amd64-qemuu-freebsd11-amd64 all_hostflags       arch-amd64,arch-xen-amd64,suite-stretch,purpose-test,freebsd-11
+test-amd64-amd64-qemuu-freebsd12-amd64 all_hostflags       arch-amd64,arch-xen-amd64,suite-stretch,purpose-test,freebsd-12
+test-amd64-amd64-qemuu-freebsd11-amd64 arch                amd64
+test-amd64-amd64-qemuu-freebsd12-amd64 arch                amd64
+test-amd64-amd64-qemuu-freebsd11-amd64 buildjob            build-amd64
+test-amd64-amd64-qemuu-freebsd12-amd64 buildjob            build-amd64
+test-amd64-amd64-qemuu-freebsd11-amd64 freebsd_arch        amd64
+test-amd64-amd64-qemuu-freebsd12-amd64 freebsd_arch        amd64
+test-amd64-amd64-qemuu-freebsd11-amd64 freebsd_image       FreeBSD-11.3-RELEASE-amd64.raw.xz
+test-amd64-amd64-qemuu-freebsd12-amd64 freebsd_image       FreeBSD-12.1-RELEASE-amd64.raw.xz
+test-amd64-amd64-qemuu-freebsd11-amd64 kernbuildjob        build-amd64-pvops
+test-amd64-amd64-qemuu-freebsd12-amd64 kernbuildjob        build-amd64-pvops
+test-amd64-amd64-qemuu-freebsd11-amd64 kernkind            pvops
+test-amd64-amd64-qemuu-freebsd12-amd64 kernkind            pvops
+test-amd64-amd64-qemuu-freebsd11-amd64 toolstack           xl
+test-amd64-amd64-qemuu-freebsd12-amd64 toolstack           xl
+test-amd64-amd64-qemuu-freebsd11-amd64 xenbuildjob         build-amd64
+test-amd64-amd64-qemuu-freebsd12-amd64 xenbuildjob         build-amd64

Note that only amd64 versions are tested at the moment, i386 had some
bugs that are being fixed so new releases can be tested.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Note this requires adding the freebsd-{11,12} hostflags to amd64
hosts.
---
 make-flight | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/make-flight b/make-flight
index af74bb4e..48dc927c 100755
--- a/make-flight
+++ b/make-flight
@@ -241,7 +241,20 @@ do_freebsd_tests () {
  freebsd_image=${FREEBSD_IMAGE_PREFIX-FreeBSD-10.1-CUSTOM-}$freebsdarch${FREEBSD_IMAGE_SUFFIX--20150525.raw.xz} \
                       all_hostflags=$most_hostflags,freebsd-10
     done
+    return
   fi
+
+  # NB: limit to amd64 ATM due to bugs with the i386 versions (11.3 and 12.1).
+  for freebsdarch in amd64; do
+    job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-freebsd11-$freebsdarch \
+                    test-freebsd xl $xenarch $dom0arch freebsd_arch=$freebsdarch \
+ freebsd_image=${FREEBSD_IMAGE_PREFIX-FreeBSD-11.3-RELEASE-}$freebsdarch${FREEBSD_IMAGE_SUFFIX-.raw.xz} \
+                    all_hostflags=$most_hostflags,freebsd-11
+    job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-freebsd12-$freebsdarch \
+                    test-freebsd xl $xenarch $dom0arch freebsd_arch=$freebsdarch \
+ freebsd_image=${FREEBSD_IMAGE_PREFIX-FreeBSD-12.1-RELEASE-}$freebsdarch${FREEBSD_IMAGE_SUFFIX-.raw.xz} \
+                    all_hostflags=$most_hostflags,freebsd-12
+  done
 }
 
 do_hvm_winxp_tests () {
-- 
2.26.2

Re: [PATCH 0/2] osstest: update FreeBSD guest tests

[Ian Jackson]

Roger Pau Monne writes ("[PATCH 0/2] osstest: update FreeBSD guest tests"):
> The following series adds FreeBSD 11 and 12 guests tests to osstest.
> ATM this is only tested on amd64, since the i386 versions had a bug.
> 
> The result can be seen at:
> 
> http://logs.test-lab.xenproject.org/osstest/logs/150428/

Thanks, Roger.

I think that for this change I ought to get an ack from Paul as the
RM.

Paul: how do you want to handle osstest changes during the freeze ?  I
already pushed on Monday - without asking you - a series to fix a
problem with bisection which was stopping osstest from bisecting the
libvirt failure in the smoke tests.  Please let me know if you think I
should have checked with you.

I think we should take this change from Roger.  Right now we are still
waiting for even the smoke tests from staging to pass.  I don't
think this would interfere with that nor will it get in the way of the
osstest buster upgrade.

Thanks,
Ian.

Re: [PATCH 0/2] osstest: update FreeBSD guest tests

[Ian Jackson]

Roger Pau Monne writes ("[PATCH 0/2] osstest: update FreeBSD guest tests"):
> The following series adds FreeBSD 11 and 12 guests tests to osstest.
> ATM this is only tested on amd64, since the i386 versions had a bug.
> 
> The result can be seen at:
> 
> http://logs.test-lab.xenproject.org/osstest/logs/150428/

Oh, I forgot to say, both patches

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

RE: [PATCH 0/2] osstest: update FreeBSD guest tests

[Paul Durrant]

> -----Original Message-----
> From: Ian Jackson <ian.jackson@citrix.com>
> Sent: 03 June 2020 10:51
> To: Roger Pau Monne <roger.pau@citrix.com>
> Cc: xen-devel@lists.xenproject.org; Paul Durrant <xadimgnik@gmail.com>
> Subject: Re: [PATCH 0/2] osstest: update FreeBSD guest tests
> 
> Roger Pau Monne writes ("[PATCH 0/2] osstest: update FreeBSD guest tests"):
> > The following series adds FreeBSD 11 and 12 guests tests to osstest.
> > ATM this is only tested on amd64, since the i386 versions had a bug.
> >
> > The result can be seen at:
> >
> > http://logs.test-lab.xenproject.org/osstest/logs/150428/
> 
> Thanks, Roger.
> 
> I think that for this change I ought to get an ack from Paul as the
> RM.
> 
> Paul: how do you want to handle osstest changes during the freeze ?  I
> already pushed on Monday - without asking you - a series to fix a
> problem with bisection which was stopping osstest from bisecting the
> libvirt failure in the smoke tests.  Please let me know if you think I
> should have checked with you.
> 

I'm no expert in osstest so I need to trust your judgement as to whether a patch is needed for 4.14 testing. That fix was clearly
necessary to help diagnose the libvirt issue so I don't think you need check with me for such things.

> I think we should take this change from Roger.  Right now we are still
> waiting for even the smoke tests from staging to pass.  I don't
> think this would interfere with that nor will it get in the way of the
> osstest buster upgrade.

We should certainly try to limit changes to only those where really are beneficial for 4.14 at this stage, but it looks like this
series is adding more relevant testing (i.e. on newer OS) so I think it qualifies.

  Paul

> 
> Thanks,
> Ian.

Re: [PATCH 0/2] osstest: update FreeBSD guest tests

[Ian Jackson]

Ian Jackson writes ("Re: [PATCH 0/2] osstest: update FreeBSD guest tests"):
> Roger Pau Monne writes ("[PATCH 0/2] osstest: update FreeBSD guest tests"):
> > The following series adds FreeBSD 11 and 12 guests tests to osstest.
> > ATM this is only tested on amd64, since the i386 versions had a bug.
> > 
> > The result can be seen at:
> > 
> > http://logs.test-lab.xenproject.org/osstest/logs/150428/
> 
> Oh, I forgot to say, both patches
> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Thanks, pushed.

Ian.