Re: [qemu-upstream-4.11-testing test] 136184: regressions - FAIL

From: Ian Jackson <ian.jackson@citrix.com>
To: Julien Grall <julien.grall@arm.com>
Cc: Anthony Perard <anthony.perard@citrix.com>,
	"xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
	Stefano Stabellini <sstabellini@kernel.org>
Subject: Re: [qemu-upstream-4.11-testing test] 136184: regressions - FAIL
Date: Fri, 17 May 2019 16:53:49 +0100	[thread overview]
Message-ID: <23774.55565.453626.345334@mariner.uk.xensource.com> (raw)
In-Reply-To: <c576ae9d-4a6e-1602-7f05-6fc2c7b26314@arm.com>

Julien Grall writes ("Re: [Xen-devel] [qemu-upstream-4.11-testing test] 136184: regressions - FAIL"):
> On 5/16/19 11:37 AM, Anthony PERARD wrote:
> >> Tests which did not succeed and are blocking,
> >> including tests which could not be run:
> >>   test-arm64-arm64-libvirt-xsm  7 xen-boot                 fail REGR. vs. 125575
> >>   test-arm64-arm64-xl           7 xen-boot                 fail REGR. vs. 125575
> >>   test-arm64-arm64-xl-xsm       7 xen-boot                 fail REGR. vs. 125575
> >>   test-arm64-arm64-xl-credit2   7 xen-boot                 fail REGR. vs. 125575
..
> > I can't figure out why Xen consistently fails to boot on rochester* in
> > the qemu-upstream-4.11-testing flights. The xen-4.11-testing seems to
> > pass.
> > 
> > At boot, the boot loader seems to load blobs, but when it's time to Xen
> > to shine, there are no output from Xen on the serial.
> 
> The serial console is initializing fairly late in the process. Any 
> useful message (such as memory setup or even part of the interrupts) 
> will be hide out. For getting them, you need earlyprintk. Unfortunately 
> they can't be configured at runtime today :(.

:-/.  Can we configure the earlyprintk at compile-time ?  We always
want it to be serial...

> > Do you know what could cause xen to fail to boot?
> 
> It is hard to say without the log. Looking at the different with a Xen 
> 4.11 flights on rochester0 [1], it seems the .config is slightly 
> different. 4.11 flight has CONFIG_LIVEPATCH set.

The osstest history shows this as a 100% repeatable boot failure but
only in the qemu flights.

Comparing 136231 (pass, xen-4.11-testing) with 136184 (fail,
qemu-upstream-4.11-testing), there are no differences in the test job
runvars.  Both used the same version of osstest.

But in the build-arm64 (Xen build) job runvars I see the following
differences:

                           136231               136184
                           pass                 fail
                           xen-4.11-testing     qemu-*4.11*
build-arm64 (Xen build)

 enable_livepatch          true                 (unset)
 [~built_]revision_qemuu   20c76f9a5fbf...      2871355a6957...
 [~built_]revision_xen     a6e07495c171...      3b062f5040a1...
 ~path_xenlptdist          build/xenlptdist.tar.gz  (unset)

build-arm64-pvops (kernel build)

 ~host                     rochester1           laxton1

 ~ indicates variable set by osstest during the test run.

The qemu revision is clearly not relevant.  I did this
   git-diff --stat a6e07495c171..3b062f5040a1
in xen.git and the differences really don't seem like they would be
relevant.

I think therefore that we need to blame the livepatch setting.  This
comes from osstest's flight construction code.  osstest is configured
to enable live patching, in the build, only on the xen-* branches.

Unfortunately due to the xen/cmdline regression, the osstest bisector
does not seem to have a useful enough baseline.  I have rm'd the stamp
files and it may manage to do better but I doubt it.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel