[xen-unstable-smoke test] 133030: trouble: blocked/broken/pass

[xen-unstable-smoke test] 133030: trouble: blocked/broken/pass

[osstest service owner]

flight 133030 xen-unstable-smoke real [real]
http://logs.test-lab.xenproject.org/osstest/logs/133030/

Failures and problems with tests :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 build-arm64-xsm                 <job status>                 broken
 build-arm64-xsm               4 host-install(4)        broken REGR. vs. 133005

Tests which did not succeed, but are not blocking:
 test-arm64-arm64-xl-xsm       1 build-check(1)               blocked  n/a
 test-amd64-amd64-libvirt     13 migrate-support-check        fail   never pass
 test-armhf-armhf-xl          13 migrate-support-check        fail   never pass
 test-armhf-armhf-xl          14 saverestore-support-check    fail   never pass

version targeted for testing:
 xen                  7b5278b28f8fbcd4402e4520d7a5d607d4a997a7
baseline version:
 xen                  455301716e1ff358cb79367213003fba771dd466

Last test of basis   133005  2019-02-07 14:00:32 Z    0 days
Failing since        133011  2019-02-07 18:00:36 Z    0 days    4 attempts
Testing same since   133017  2019-02-07 21:03:23 Z    0 days    3 attempts

------------------------------------------------------------
People who touched revisions under test:
  Ian Jackson <ian.jackson@eu.citrix.com>
  Stefano Stabellini <sstabellini@kernel.org>
  Stefano Stabellini <stefanos@xilinx.com>
  Wei Liu <wei.liu2@citrix.com>

jobs:
 build-arm64-xsm                                              broken  
 build-amd64                                                  pass    
 build-armhf                                                  pass    
 build-amd64-libvirt                                          pass    
 test-armhf-armhf-xl                                          pass    
 test-arm64-arm64-xl-xsm                                      blocked 
 test-amd64-amd64-xl-qemuu-debianhvm-i386                     pass    
 test-amd64-amd64-libvirt                                     pass    


------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
    http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
    http://xenbits.xen.org/gitweb?p=osstest.git;a=summary

broken-job build-arm64-xsm broken
broken-step build-arm64-xsm host-install(4)

Not pushing.

------------------------------------------------------------
commit 7b5278b28f8fbcd4402e4520d7a5d607d4a997a7
Author: Ian Jackson <ian.jackson@eu.citrix.com>
Date:   Thu Feb 7 15:02:27 2019 +0000

    tools: init scripts: make XEN_RUN_DIR and XEN_LOCK_DIR mode 700
    
    These directories ought not to be even world-readable.  If this script
    for some reason runs with a lax umask they might be created
    overly-writeable.  Avoid any such bug by setting the mode explicitly.
    
    Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
    Acked-by: Wei Liu <wei.liu2@citrix.com>
    Release-acked-by: Juergen Gross <jgross@suse.com>

commit 0c4a38c098f9bffeb33f8cf88abdea4b0f9a9070
Author: Ian Jackson <ian.jackson@eu.citrix.com>
Date:   Thu Feb 7 15:02:26 2019 +0000

    tools: init scripts: xencommons: Fixes to Description
    
    `neeeded' is a typo.  And xend is long gone.
    
    No functional change.
    
    Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
    Acked-by: Wei Liu <wei.liu2@citrix.com>
    Release-acked-by: Juergen Gross <jgross@suse.com>

commit 01097e0194321d27262513cf1291fddfea1606c3
Author: Ian Jackson <ian.jackson@eu.citrix.com>
Date:   Thu Feb 7 15:02:25 2019 +0000

    tools: init scripts: xencommons: Provides `xen'
    
    It is useful to have a single `xen' facility (in the LSB Provides
    namespace).  That allows other facilities to specify that they should
    go after `xen' without needing to know the implementation details.
    
    This service name is already Provide'd by the (fairly different) init
    scripts used in Debian.
    
    Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
    Acked-by: Wei Liu <wei.liu2@citrix.com>
    Release-acked-by: Juergen Gross <jgross@suse.com>

commit b4df73de493954c44f240f78779c9bd3782e1572
Author: Stefano Stabellini <sstabellini@kernel.org>
Date:   Tue Feb 5 13:38:53 2019 -0800

    xen/arm: gic-v2: deactivate interrupts during initialization
    
    Interrupts could be ACTIVE at boot. Make sure to deactivate them during
    initialization.
    
    Signed-off-by: Stefano Stabellini <stefanos@xilinx.com>
    Reviewed-by: Julien Grall <julien.grall@arm.com>
    CC: julien.grall@arm.com
    CC: peng.fan@nxp.com
    CC: jgross@suse.com
(qemu changes not included)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass

[Wei Liu]

On Fri, Feb 08, 2019 at 05:21:44AM +0000, osstest service owner wrote:
> flight 133030 xen-unstable-smoke real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/133030/
> 
> Failures and problems with tests :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  build-arm64-xsm                 <job status>                 broken
>  build-arm64-xsm               4 host-install(4)        broken REGR. vs. 133005

After some investigation, I think something is wrong with the linux-4.9
branch.

The issue to hand is:

    Feb  8 04:12:54.790904 
    Loading initial ramdisk ...
    Feb  8 04:12:55.114864 
    EFI stub: Booting Linux Kernel...
    Feb  8 04:12:55.354885 EFI stub: ERROR: Failed to alloc kernel memory
    Feb  8 04:12:55.354946 EFI stub: ERROR: Failed to relocate kernel
    Feb  8 04:12:55.354993 Feb  8 04:12:55.355016 
      Failed to boot both default and fallback entries.

The new 4.9 kernel can't be loaded _natively_ anymore.

But why did it pass its own pushgate in the first place?

That latest push to that branch is in 132748. There isn't any
serial-laxton*.log in its logs. So my conclusion is that that changeset
was built on a shared build host which ran a _previous_ version of Linux
4.9. And then, other test cases in the same flight loaded xen first,
which didn't cause any issue. The Linux changeset under test passed
pushgate.

But after a changeset passes pushgate, it becomes the new baseline. The
new baseline can't be booted _natively_ anymore.

I think someone more familiar with Arm / EFI will need to look into
fixing Linux 4.9 on laxton.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass

[Wei Liu]

CC Stefano as well

On Fri, Feb 08, 2019 at 01:14:16PM +0000, Wei Liu wrote:
> On Fri, Feb 08, 2019 at 05:21:44AM +0000, osstest service owner wrote:
> > flight 133030 xen-unstable-smoke real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/133030/
> > 
> > Failures and problems with tests :-(
> > 
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> >  build-arm64-xsm                 <job status>                 broken
> >  build-arm64-xsm               4 host-install(4)        broken REGR. vs. 133005
> 
> After some investigation, I think something is wrong with the linux-4.9
> branch.
> 
> The issue to hand is:
> 
>     Feb  8 04:12:54.790904 
>     Loading initial ramdisk ...
>     Feb  8 04:12:55.114864 
>     EFI stub: Booting Linux Kernel...
>     Feb  8 04:12:55.354885 EFI stub: ERROR: Failed to alloc kernel memory
>     Feb  8 04:12:55.354946 EFI stub: ERROR: Failed to relocate kernel
>     Feb  8 04:12:55.354993 Feb  8 04:12:55.355016 
>       Failed to boot both default and fallback entries.
> 
> The new 4.9 kernel can't be loaded _natively_ anymore.
> 
> But why did it pass its own pushgate in the first place?
> 
> That latest push to that branch is in 132748. There isn't any
> serial-laxton*.log in its logs. So my conclusion is that that changeset
> was built on a shared build host which ran a _previous_ version of Linux
> 4.9. And then, other test cases in the same flight loaded xen first,
> which didn't cause any issue. The Linux changeset under test passed
> pushgate.
> 
> But after a changeset passes pushgate, it becomes the new baseline. The
> new baseline can't be booted _natively_ anymore.
> 
> I think someone more familiar with Arm / EFI will need to look into
> fixing Linux 4.9 on laxton.
> 
> Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

arm64, laxton[01] (was Re: [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass)

[Ian Jackson]

On Fri, Feb 08, 2019 at 01:14:16PM +0000, Wei Liu wrote:
> On Fri, Feb 08, 2019 at 05:21:44AM +0000, osstest service owner wrote:
> > flight 133030 xen-unstable-smoke real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/133030/
> > 
> > Failures and problems with tests :-(

Thanks for looking at this.

> After some investigation, I think something is wrong with the linux-4.9
> branch.
> 
> The issue to hand is:
> 
>     Feb  8 04:12:54.790904 
>     Loading initial ramdisk ...
>     Feb  8 04:12:55.114864 
>     EFI stub: Booting Linux Kernel...
>     Feb  8 04:12:55.354885 EFI stub: ERROR: Failed to alloc kernel memory
>     Feb  8 04:12:55.354946 EFI stub: ERROR: Failed to relocate kernel
>     Feb  8 04:12:55.354993 Feb  8 04:12:55.355016 
>       Failed to boot both default and fallback entries.
> 
> The new 4.9 kernel can't be loaded _natively_ anymore.

This is not using our own-built kernel.

It is using (our copy of) the jessie arm64 debian-installer kernel
which has not changed since June.  I guess it is just about possible
that the file in /home/tftp has been corrupted somehow.  I have asked
Credativ to check them against our backups.

Looking at
  http://logs.test-lab.xenproject.org/osstest/results/host/laxton0.html
  http://logs.test-lab.xenproject.org/osstest/results/host/laxton1.html
it is evident that both boxes started failing at roughly the same
time.

On both boxes the first failing job was a test in flight 132973, the
linux-4.9 one.  But I think this is a red herring because (i) the
failure occurs during host installation, before the flight has
actually touched the kernel to be tested (ii) on laxton1 there were
two test jobs in 132973 which passed (iii) on both machines there were
earlier build jobs in 132973 which passed (which would have had a very
similar host installation step).

I have checked and there was no update to the osstest code.  I don't
think there have been updates to the infrastructure config but I
haven't searched all the infrastructure boxes etckeepers etc.

So something has caused both machines to fail simultaneously.

Possible explanations to me seem to be:

  * Some kind of common physical cause (power surge corrupting the
    firmware or something)

  * Some kind of common nonphysical cause external to the hosts or the
    tests: bad files on the infrastructure hosts; a change to the
    behaviour of the bootp/tftp servers; a new kind of broadcast
    network packet (perhaps from other tests) which causes the laxton
    firmware to malfunction; etc.

  * The new linux-4.9 kernel does something which has the effect of
    often (but not always) corrupting the laxtons' firmware.

  * A firmware bug triggered by the passage of time (eg
    clock-dependent)

  * Misunderstanding by me in my analysis of what ingredients are used
    by the host installation failures.

  * Some other common-mode failure that I haven't thought of.

In the meantime I have unblessed the laxtons to avoid osstest
repeatedly power cycling them to try to get them to work.  FTR this
will still not allow the push gate to pass.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

arm64, laxton[01] (was Re: [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass)

[Ian Jackson]

Ian Jackson writes ("arm64, laxton[01] (was Re: [Xen-devel] [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass)"):
> I have checked and there was no update to the osstest code.  I don't
> think there have been updates to the infrastructure config but I
> haven't searched all the infrastructure boxes etckeepers etc.

I have now checked the etckeeper and dpkg logs on the osstest (test
controller, tftp server) and infra (dhcp and dns server etc.) VMs and
there were no changes that could have caused this.

Credativ have checked the boot files being used against the backups
and they are file.

> So something has caused both machines to fail simultaneously.
> 
> Possible explanations to me seem to be:
> 
>   * Some kind of common physical cause (power surge corrupting the
>     firmware or something)
> 
>   * Some kind of common nonphysical cause external to the hosts or the
>     tests: bad files on the infrastructure hosts; a change to the
>     behaviour of the bootp/tftp servers; a new kind of broadcast
>     network packet (perhaps from other tests) which causes the laxton
>     firmware to malfunction; etc.
> 
>   * The new linux-4.9 kernel does something which has the effect of
>     often (but not always) corrupting the laxtons' firmware.
> 
>   * A firmware bug triggered by the passage of time (eg
>     clock-dependent)
> 
>   * Misunderstanding by me in my analysis of what ingredients are used
>     by the host installation failures.
> 
>   * Some other common-mode failure that I haven't thought of.
> 
> In the meantime I have unblessed the laxtons to avoid osstest
> repeatedly power cycling them to try to get them to work.  FTR this
> will still not allow the push gate to pass.

I don't have a good plan about what to do next.  I guess one thing we
could do would be to ask Yogesh to reflash the firmware on laxton0 and
see if that helps.

I think this issue is likely to be a protracted problem.  Any helpful
suggestions from people with more experience of these particular boxes
would be very welcome...

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: arm64, laxton[01] (was Re: [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass)

[Julien Grall]

Hi,

Sorry for the formatting.

On Tue, Feb 12, 2019 at 12:26 PM Ian Jackson <ian.jackson@citrix.com> wrote:
> I don't have a good plan about what to do next.  I guess one thing we
> could do would be to ask Yogesh to reflash the firmware on laxton0 and
> see if that helps.
>
> I think this issue is likely to be a protracted problem.  Any helpful
> suggestions from people with more experience of these particular boxes
> would be very welcome...

Looking at the first failure log [1], it seems we are trying to boot
Linux 3.16.  So the failure seems legit to me as I don't think 3.16
supports Seattle.
Furthermore, previous successful Debian install were using a backport
kernel (4.9).

So I think we can rule out a firmware bug.  Now, I am struggling to
understand why osstest is suddenly using 3.16.
Was there any change in osstest or the configuration?

[1] http://logs.test-lab.xenproject.org/osstest/logs/132973/test-arm64-arm64-xl/serial-laxton0.log




--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: arm64, laxton[01] (was Re: [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass)

[Ian Jackson]

Julien Grall writes ("Re: [Xen-devel] arm64, laxton[01] (was Re: [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass)"):
> On Tue, Feb 12, 2019 at 12:26 PM Ian Jackson <ian.jackson@citrix.com> wrote:
> > I don't have a good plan about what to do next.  I guess one thing we
> > could do would be to ask Yogesh to reflash the firmware on laxton0 and
> > see if that helps.
> >
> > I think this issue is likely to be a protracted problem.  Any helpful
> > suggestions from people with more experience of these particular boxes
> > would be very welcome...
> 
> Looking at the first failure log [1], it seems we are trying to boot
> Linux 3.16.  So the failure seems legit to me as I don't think 3.16
> supports Seattle.

This is odd.  You have definitely spotted something wrong.

In
  http://logs.test-lab.xenproject.org/osstest/logs/132973/test-arm64-arm64-xl/info.html
we see that the failing step started at 15:13:32.  In the log
  > [1] http://logs.test-lab.xenproject.org/osstest/logs/132973/test-arm64-arm64-xl/serial-laxton0.log
I see this:

  Feb 7 15:14:49.374771 [ 0.000000] Linux version 4.9.0-0.bpo.2-arm64
  (debian-kernel@lists.debian.org) (gcc version 4.9.2 (Debian/Linaro
  4.9.2-10) ) #1 SMP Debian 4.9.18-1~bpo8+1 (2017-04-10)

And that part works.  It runs through d-i and thinks it has succeeded.
But then when the host reboots it reboots into 3.16, not the backports
kernel.

Looking at the installer log:

  2019-02-07 15:18:34 Z 172.16.144.52:39843 <13>Feb  7 15:18:34
  base-installer: info: kernel linux-image-arm64 usable on arm64 

which is kind of weird.

> So I think we can rule out a firmware bug.  Now, I am struggling to
> understand why osstest is suddenly using 3.16.
> Was there any change in osstest or the configuration?

No.  I think something may be wrong with the way Debian is publishing
the backports kernels.  I will ask...

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: arm64, laxton[01] (was Re: [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass)

[Ian Jackson]

Ian Jackson writes ("Re: [Xen-devel] arm64, laxton[01] (was Re: [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass)"):
> And that part works.  It runs through d-i and thinks it has succeeded.
> But then when the host reboots it reboots into 3.16, not the backports
> kernel.
> 
> Looking at the installer log:
> 
>   2019-02-07 15:18:34 Z 172.16.144.52:39843 <13>Feb  7 15:18:34
>   base-installer: info: kernel linux-image-arm64 usable on arm64 
> 
> which is kind of weird.
> 
> > So I think we can rule out a firmware bug.  Now, I am struggling to
> > understand why osstest is suddenly using 3.16.
> > Was there any change in osstest or the configuration?
> 
> No.  I think something may be wrong with the way Debian is publishing
> the backports kernels.  I will ask...

Thanks, Julien, you have found enough for me to identify the problem.

According to snapshot.debian.org, at 2019-02-06T21:13:14, the file
  http://ftp.debian.org/debian/dists/jessie-backports/main/binary-arm64/Packages.xz
mentioned linux-base version 4.3~bpo8+1.  Now, it doesn't mention
it at all.

This is presumably part of cruft removal, since jessie-security does
contain an even newer version, 4.5~deb8u1.

linux-image-4.9.* require newer linux-support than is available in
plain jessie.  As a result of all this, the installer removes the
4.9.x kernel because its dependencies can't be satisfied.  The machine
then tries to reboot into some much older kernel.

The osstest installer system does not use jessie-security and I think
there was some reason for this which I have forgotten right now.
Maybe the security repositories were awkward to cache - but nowadays
we have a different approach to this caching.

I will investigate putting <suite>-security back.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: arm64, laxton[01] (was Re: [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass)

[Julien Grall]

[-- Attachment #1.1: Type: text/plain, Size: 2130 bytes --]

Sorry the formatting.

On Tue, 12 Feb 2019, 20:26 Ian Jackson, <ian.jackson@citrix.com> wrote:

> Ian Jackson writes ("Re: [Xen-devel] arm64, laxton[01] (was Re:
> [xen-unstable-smoke test] 133030: trouble: blocked/broken/pass)"):
> > And that part works.  It runs through d-i and thinks it has succeeded.
> > But then when the host reboots it reboots into 3.16, not the backports
> > kernel.
> >
> > Looking at the installer log:
> >
> >   2019-02-07 15:18:34 Z 172.16.144.52:39843 <13>Feb  7 15:18:34
> >   base-installer: info: kernel linux-image-arm64 usable on arm64
> >
> > which is kind of weird.
> >
> > > So I think we can rule out a firmware bug.  Now, I am struggling to
> > > understand why osstest is suddenly using 3.16.
> > > Was there any change in osstest or the configuration?
> >
> > No.  I think something may be wrong with the way Debian is publishing
> > the backports kernels.  I will ask...
>
> Thanks, Julien, you have found enough for me to identify the problem.
>
> According to snapshot.debian.org, at 2019-02-06T21:13:14, the file
>
> http://ftp.debian.org/debian/dists/jessie-backports/main/binary-arm64/Packages.xz
> mentioned linux-base version 4.3~bpo8+1.  Now, it doesn't mention
> it at all.
>
> This is presumably part of cruft removal, since jessie-security does
> contain an even newer version, 4.5~deb8u1.
>
> linux-image-4.9.* require newer linux-support than is available in
> plain jessie.  As a result of all this, the installer removes the
> 4.9.x kernel because its dependencies can't be satisfied.  The machine
> then tries to reboot into some much older kernel.
>
> The osstest installer system does not use jessie-security and I think
> there was some reason for this which I have forgotten right now.
> Maybe the security repositories were awkward to cache - but nowadays
> we have a different approach to this caching.
>

This may be related to [1]. I don't know whether the support status changed
since then.


> I will investigate putting <suite>-security back.


[1]
https://xenbits.xen.org/gitweb/?p=osstest.git;a=commit;h=c14980b543ba057dcdd7a6c65d74a12cb4192034



> Ian.
>

[-- Attachment #1.2: Type: text/html, Size: 3396 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel