[linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[osstest service owner]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 13065 bytes --]

flight 59041 linux-3.18 real [real]
http://logs.test-lab.xenproject.org/osstest/logs/59041/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-amd64-amd64-xl-pvh-intel 11 guest-start              fail REGR. vs. 58581
 build-armhf-xsm               4 host-build-prep           fail REGR. vs. 58581

Tests which are failing intermittently (not blocking):
 test-armhf-armhf-libvirt      6 xen-boot                    fail pass in 59001

Regressions which are regarded as allowable (not blocking):
 test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm 12 guest-localmigrate fail baseline untested
 test-armhf-armhf-xl-rtds     11 guest-start    fail in 59001 baseline untested
 test-armhf-armhf-libvirt-xsm  6 xen-boot              fail in 59001 like 58581
 test-armhf-armhf-xl-xsm       6 xen-boot              fail in 59001 like 58581
 test-amd64-i386-libvirt-xsm  11 guest-start                  fail   like 58558
 test-amd64-i386-libvirt      11 guest-start                  fail   like 58581
 test-armhf-armhf-xl-multivcpu  6 xen-boot                     fail  like 58581
 test-armhf-armhf-xl-credit2   6 xen-boot                     fail   like 58581
 test-armhf-armhf-xl           6 xen-boot                     fail   like 58581
 test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop              fail like 58581
 test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop             fail like 58581
 test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop             fail like 58581

Tests which did not succeed, but are not blocking:
 test-armhf-armhf-libvirt-xsm  1 build-check(1)               blocked  n/a
 test-armhf-armhf-xl-xsm       1 build-check(1)               blocked  n/a
 test-amd64-i386-libvirt-xsm  12 migrate-support-check fail in 59001 never pass
 test-armhf-armhf-libvirt     12 migrate-support-check fail in 59001 never pass
 test-amd64-amd64-xl-pvh-amd  11 guest-start                  fail   never pass
 test-amd64-i386-freebsd10-amd64  9 freebsd-install             fail never pass
 test-amd64-i386-freebsd10-i386  9 freebsd-install              fail never pass
 test-armhf-armhf-xl-cubietruck  6 xen-boot                     fail never pass
 test-amd64-amd64-libvirt-xsm 12 migrate-support-check        fail   never pass
 test-armhf-armhf-xl-rtds     14 guest-start.2                fail   never pass
 test-armhf-armhf-xl-rtds     12 migrate-support-check        fail   never pass
 test-amd64-amd64-libvirt     12 migrate-support-check        fail   never pass
 test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop              fail never pass
 test-armhf-armhf-xl-arndale  12 migrate-support-check        fail   never pass

version targeted for testing:
 linux                ea5dd38e93b3bec3427e5d3eef000bbf5d637e76
baseline version:
 linux                d048c068d00da7d4cfa5ea7651933b99026958cf

Last test of basis    58581  2015-06-15 09:42:22 Z   18 days
Testing same since    58976  2015-06-29 19:43:23 Z    4 days    6 attempts

------------------------------------------------------------
People who touched revisions under test:
  "Eric W. Biederman" <ebiederm@xmission.com>
  Aaron Lu <aaron.lu@intel.com>
  Alexander Duyck <alexander.h.duyck@redhat.com>
  Alexei Starovoitov <ast@plumgrid.com>
  Andrew de los Reyes <adlr@chromium.org>
  Andrew Morton <akpm@linux-foundation.org>
  Andy Lutomirski <luto@amacapital.net>
  Anna Schumaker <Anna.Schumaker@Netapp.com>
  Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>
  Ard Biesheuvel <ard.biesheuvel@linaro.org>
  Arnd Bergmann <arnd@arndb.de>
  Baruch Siach <baruch@tkos.co.il>
  Ben Serebrin <serebrin@google.com>
  Benjamin Tissoires <benjamin.tissoires@redhat.com>
  Bjørn Mork <bjorn@mork.no>
  Borislav Petkov <bp@suse.de>
  Chuck Lever <chuck.lever@oracle.com>
  Cong Wang <xiyou.wangcong@gmail.com>
  Daniel Borkmann <daniel@iogearbox.net>
  Darren Hart <dvhart@linux.intel.com>
  Darren Salt <devspam@moreofthesa.me.uk>
  David Daney <david.daney@cavium.com>
  David S. Miller <davem@davemloft.net>
  David Woodhouse <David.Woodhouse@intel.com>
  Devesh Sharma <Devesh.Sharma@Emulex.Com>
  Doug Ledford <dledford@redhat.com>
  Eric Dumazet <edumazet@google.com>
  Eric W. Biederman <ebiederm@xmission.com>
  Felipe Balbi <balbi@ti.com>
  Feng Kan <fkan@apm.com>
  Florent Fourcot <florent.fourcot@enst-bretagne.fr>
  Florian Fainelli <f.fainelli@gmail.com>
  Geert Uytterhoeven <geert+renesas@glider.be>
  Grant Likely <grant.likely@linaro.org>
  Greg Kroah-Hartman <gregkh@linuxfoundation.org>
  Hannes Frederic Sowa <hannes@stressinduktion.org>
  Henning Rogge <hrogge@gmail.com>
  Honggang Li <honli@redhat.com>
  Ian Campbell <ian.campbell@citrix.com>
  Ilya Dryomov <idryomov@gmail.com>
  Ingo Molnar <mingo@kernel.org>
  Jakub Sitnicki <jsitnicki@gmail.com>
  Jason Gunthorpe <jgunthorpe@obsidianresearch.com>
  Jiada Wang <jiada_wang@mentor.com>
  Jiri Kosina <jkosina@suse.cz>
  Jiri Pirko <jiri@resnulli.us>
  Joerg Roedel <jroedel@suse.de>
  Jovi Zhangwei <jovi@cloudflare.com>
  Ken Xue <Ken.Xue@amd.com>
  Kevin Hilman <khilman@linaro.org>
  Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
  Laura Abbott <labbott@fedoraproject.org>
  Laurent Pinchart <laurent.pinchart@ideasonboard.com>
  Linus Lüssing <linus.luessing@c0d3.blue>
  Linus Torvalds <torvalds@linux-foundation.org>
  Linus Walleij <linus.walleij@linaro.org>
  Mark Brown <broonie@kernel.org>
  Mark Salyzyn <salyzyn@android.com>
  Matt Fleming <matt.fleming@intel.com>
  Matthew Garrett <mjg59@coreos.com>
  Mauro Carvalho Chehab <mchehab@osg.samsung.com>
  Max Filippov <jcmvbkbc@gmail.com>
  Meghana Cheripady <Meghana.Cheripady@Emulex.Com>
  Mel Gorman <mgorman@suse.de>
  Mika Westerberg <mika.westerberg@linux.intel.com>
  Milan Plzik <milan.plzik@gmail.com>
  Neal Cardwell <ncardwell@google.com>
  Neil Horman <nhorman@tuxdriver.com>
  Nicolas Dichtel <nicolas.dichtel@6wind.com>
  Nicolas Iooss <nicolas.iooss_linux@m4x.org>
  Nicolas Pitre <nico@linaro.org>
  Nikolay Aleksandrov <nikolay@cumulusnetworks.com>
  Oliver Neukum <oliver@neukum.org>
  Oliver Neukum <oneukum@suse.de>
  oliver@neukum.org <oliver@neukum.org>
  Olof Johansson <olof@lixom.net>
  Pablo Neira Ayuso <pablo@netfilter.org>
  Paolo Bonzini <pbonzini@redhat.com>
  Pelle Nilsson <per.nilsson@xelmo.com>
  Peter Zijlstra (Intel) <peterz@infradead.org>
  Rafael J. Wysocki <rafael.j.wysocki@intel.com>
  Raphael Assenat <raph@raphnet.net>
  Richard Cochran <richardcochran@gmail.com>
  Rik Theys <Rik.Theys@esat.kuleuven.be>
  Ross Lagerwall <ross.lagerwall@citrix.com>
  Roy Franz <roy.franz@linaro.org>
  Russell King <rmk+kernel@arm.linux.org.uk>
  Sasha Levin <sasha.levin@oracle.com>
  Sean Young <sean@mess.org>
  Shawn Bohrer <sbohrer@rgmadvisors.com>
  Simon Horman <horms+renesas@verge.net.au>
  Sowmini Varadhan <sowmini.varadhan@oracle.com>
  Sriharsha Basavapatna <sriharsha.basavapatna@avagotech.com>
  Steffen Klassert <steffen.klassert@secunet.com>
  Thadeu Lima de Souza Cascardo <cascardo@redhat.com>
  Theodore Ts'o <tytso@mit.edu>
  Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
  Veaceslav Falico <vfalico@gmail.com>
  Veeresh U. Kokatnur <veereshuk@chelsio.com>
  Vijay Subramanian <subramanian.vijay@gmail.com>
  Vinod Koul <vinod.koul@intel.com>
  Vittorio Gambaletta <linuxbugs@vittgam.net>
  Vlad Yasevich <vyasevich@gmail.com>
  Vladislav Yasevich <vyasevic@redhat.com>
  WANG Cong <xiyou.wangcong@gmail.com>
  Wei Liu <wei.liu2@citrix.com>
  Yao Xiwei <xiwei.yao@6wind.com>
  Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
  Yuchung Cheng <ycheng@google.com>

jobs:
 build-amd64-xsm                                              pass
 build-armhf-xsm                                              broken
 build-i386-xsm                                               pass
 build-amd64                                                  pass
 build-armhf                                                  pass
 build-i386                                                   pass
 build-amd64-libvirt                                          pass
 build-armhf-libvirt                                          pass
 build-i386-libvirt                                           pass
 build-amd64-pvops                                            pass
 build-armhf-pvops                                            pass
 build-i386-pvops                                             pass
 build-amd64-rumpuserxen                                      pass
 build-i386-rumpuserxen                                       pass
 test-amd64-amd64-xl                                          pass
 test-armhf-armhf-xl                                          fail
 test-amd64-i386-xl                                           pass
 test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm                pass
 test-amd64-i386-xl-qemut-debianhvm-amd64-xsm                 pass
 test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm                pass
 test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm                 pass
 test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm        pass
 test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm         fail
 test-amd64-amd64-libvirt-xsm                                 pass
 test-armhf-armhf-libvirt-xsm                                 blocked
 test-amd64-i386-libvirt-xsm                                  fail
 test-amd64-amd64-xl-xsm                                      pass
 test-armhf-armhf-xl-xsm                                      blocked
 test-amd64-i386-xl-xsm                                       pass
 test-amd64-amd64-xl-pvh-amd                                  fail
 test-amd64-i386-qemut-rhel6hvm-amd                           pass
 test-amd64-i386-qemuu-rhel6hvm-amd                           pass
 test-amd64-amd64-xl-qemut-debianhvm-amd64                    pass
 test-amd64-i386-xl-qemut-debianhvm-amd64                     pass
 test-amd64-amd64-xl-qemuu-debianhvm-amd64                    pass
 test-amd64-i386-xl-qemuu-debianhvm-amd64                     pass
 test-amd64-i386-freebsd10-amd64                              fail
 test-amd64-amd64-xl-qemuu-ovmf-amd64                         pass
 test-amd64-i386-xl-qemuu-ovmf-amd64                          pass
 test-amd64-amd64-rumpuserxen-amd64                           pass
 test-amd64-amd64-xl-qemut-win7-amd64                         fail
 test-amd64-i386-xl-qemut-win7-amd64                          fail
 test-amd64-amd64-xl-qemuu-win7-amd64                         fail
 test-amd64-i386-xl-qemuu-win7-amd64                          fail
 test-armhf-armhf-xl-arndale                                  pass
 test-amd64-amd64-xl-credit2                                  pass
 test-armhf-armhf-xl-credit2                                  fail
 test-armhf-armhf-xl-cubietruck                               fail
 test-amd64-i386-freebsd10-i386                               fail
 test-amd64-i386-rumpuserxen-i386                             pass
 test-amd64-amd64-xl-pvh-intel                                fail
 test-amd64-i386-qemut-rhel6hvm-intel                         pass
 test-amd64-i386-qemuu-rhel6hvm-intel                         pass
 test-amd64-amd64-libvirt                                     pass
 test-armhf-armhf-libvirt                                     fail
 test-amd64-i386-libvirt                                      fail
 test-amd64-amd64-xl-multivcpu                                pass
 test-armhf-armhf-xl-multivcpu                                fail
 test-amd64-amd64-pair                                        pass
 test-amd64-i386-pair                                         pass
 test-amd64-amd64-xl-rtds                                     pass
 test-armhf-armhf-xl-rtds                                     fail
 test-amd64-i386-xl-qemut-winxpsp3-vcpus1                     pass
 test-amd64-i386-xl-qemuu-winxpsp3-vcpus1                     pass
 test-amd64-amd64-xl-qemut-winxpsp3                           pass
 test-amd64-i386-xl-qemut-winxpsp3                            pass
 test-amd64-amd64-xl-qemuu-winxpsp3                           pass
 test-amd64-i386-xl-qemuu-winxpsp3                            pass


------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
    http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
    http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Not pushing.

(No revision log; it would be 2209 lines long.)


[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Ian Campbell]

On Fri, 2015-07-03 at 22:21 +0000, osstest service owner wrote:

>  build-armhf-xsm               4 host-build-prep           fail REGR. vs. 58581

A strange one I think.

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Ian Campbell]

oops, that's not the button I wanted....

On Mon, 2015-07-06 at 09:27 +0100, Ian Campbell wrote:
> On Fri, 2015-07-03 at 22:21 +0000, osstest service owner wrote:
> 
> >  build-armhf-xsm               4 host-build-prep           fail REGR. vs. 58581
> 
> A strange one I think.

As I was going to say...

http://logs.test-lab.xenproject.org/osstest/logs/59041/build-armhf-xsm/4.ts-xen-build-prep.log ends with:

resource host arndale-metrocentre shared build-wheezy-armhf
3c464fbf8a05b24f4afb7e58bcf4ae77fbb7ad0c not build-wheezy-armhf
8600096139e58a283eb2c23c769673d451487e1c at Osstest/Executive.pm line
840.

I'm not sure how to interpret that...

Ian.

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Ian Campbell]

On Fri, 2015-07-03 at 22:21 +0000, osstest service owner wrote:
> flight 59041 linux-3.18 real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/59041/
> 
> Regressions :-(
> 
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-amd64-amd64-xl-pvh-intel 11 guest-start              fail REGR. vs. 58581

Not really sure who is on the hook for pvh fails these days, copying
some likely suspects + x86 maintainers. Looks like the guest is triple
faulting.

http://logs.test-lab.xenproject.org/osstest/logs/59041/test-amd64-amd64-xl-pvh-intel/info.html

http://logs.test-lab.xenproject.org/osstest/logs/59041/test-amd64-amd64-xl-pvh-intel/11.ts-guest-start.log
2015-07-03 19:50:21 Z executing ssh ... root@172.16.144.38 xl list
2015-07-03 19:50:21 Z guest debian.guest.osstest not present on this host
debian.guest.osstest not running at ./ts-guest-start line 33.

http://logs.test-lab.xenproject.org/osstest/logs/59041/test-amd64-amd64-xl-pvh-intel/serial-huxelrebe0.log
Jul  3 19:50:21.557035 (d1) mapping kernel into physical memory
Jul  3 19:50:21.661049 (d1) about to get started...
Jul  3 19:50:21.661065 (d1) [    0.000000] Initializing cgroup subsys cpuset
Jul  3 19:50:21.669059 (d1) [    0.000000] Initializing cgroup subsys cpu
Jul  3 19:50:21.669078 (d1) [    0.000000] Initializing cgroup subsys cpuacct
Jul  3 19:50:21.677045 (d1) [    0.000000] Linux version 3.18.17 (osstest@nocera0) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP Fri Jul 3 08:12:11 UTC 2015
Jul  3 19:50:21.685259 (d1) 
Jul  3 19:50:21.685271 (d1) [    0.000000] Command line: root=/dev/xvda2 ro console=hvc0 earlyprintk=xen
Jul  3 19:50:21.693061 (d1) [    0.000000] ACPI in unprivileged domain disabled
Jul  3 19:50:21.701043 (d1) [    0.000000] e820: BIOS-provided physical RAM map:
Jul  3 19:50:21.709048 (d1) [    0.000000] Xen: [mem 0x0000000000000000-0x000000001fffffff] usable
Jul  3 19:50:21.709072 (d1) [    0.000000] bootconsole [xenboot0] enabled
Jul  3 19:50:21.717050 (d1) [    0.000000] NX (Execute Disable) protection: active
Jul  3 19:50:21.725042 (d1) [    0.000000] DMI not present or invalid.
Jul  3 19:50:21.725062 (d1) [    0.000000] e820: last_pfn = 0x20000 max_arch_pfn = 0x400000000
Jul  3 19:50:21.733047 (d1) [    0.000000] Scanning 1 areas for low memory corruption
Jul  3 19:50:21.741040 (d1) [    0.000000] Using GB pages for direct mapping
Jul  3 19:50:21.741060 (d1) [    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
Jul  3 19:50:21.749053 (XEN) d1v0 Triple fault - invoking HVM shutdown action 0

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass"):
> On Fri, 2015-07-03 at 22:21 +0000, osstest service owner wrote:
> 
> >  build-armhf-xsm               4 host-build-prep           fail REGR. vs. 58581
> 
> A strange one I think.

resource host arndale-metrocentre shared build-wheezy-armhf
3c464fbf8a05b24f4afb7e58bcf4ae77fbb7ad0c not build-wheezy-armhf
8600096139e58a283eb2c23c769673d451487e1c at Osstest/Executive.pm line
840.

This is a safety catch - effectively, an assertion failure.  It is
trying to mark arndale-metrocentre suitable for other jobs to use,
with a sharing key of
  build-wheezy-armhf 8600096139e58a283eb2c23c769673d451487e1c

What it is asserting is that the host is currently in preparation,
with the same sharing key.  But the host's sharing key was
  build-wheezy-armhf 3c464fbf8a05b24f4afb7e58bcf4ae77fbb7ad0c

The hash is the osstest revision.  This is the result of me updating
the osstest revision under its feet.

Nice to see in
  http://logs.test-lab.xenproject.org/osstest/results/history/build-armhf-xsm/linux-3.18
that my display of multiple osstest revisions works :-).

Ian.

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Ian Campbell]

On Mon, 2015-07-06 at 11:16 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass"):
> > On Fri, 2015-07-03 at 22:21 +0000, osstest service owner wrote:
> > 
> > >  build-armhf-xsm               4 host-build-prep           fail REGR. vs. 58581
> > 
> > A strange one I think.
> 
> resource host arndale-metrocentre shared build-wheezy-armhf
> 3c464fbf8a05b24f4afb7e58bcf4ae77fbb7ad0c not build-wheezy-armhf
> 8600096139e58a283eb2c23c769673d451487e1c at Osstest/Executive.pm line
> 840.
> 
> This is a safety catch - effectively, an assertion failure.  It is
> trying to mark arndale-metrocentre suitable for other jobs to use,
> with a sharing key of
>   build-wheezy-armhf 8600096139e58a283eb2c23c769673d451487e1c
> 
> What it is asserting is that the host is currently in preparation,
> with the same sharing key.  But the host's sharing key was
>   build-wheezy-armhf 3c464fbf8a05b24f4afb7e58bcf4ae77fbb7ad0c
> 
> The hash is the osstest revision.  This is the result of me updating
> the osstest revision under its feet.

I somehow managed to miss that those massive numbers making up 2/3 of
the log message were different...

> Nice to see in
>   http://logs.test-lab.xenproject.org/osstest/results/history/build-armhf-xsm/linux-3.18
> that my display of multiple osstest revisions works :-).

Yes indeed!

Ian.

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Boris Ostrovsky]

On 07/06/2015 04:32 AM, Ian Campbell wrote:
> On Fri, 2015-07-03 at 22:21 +0000, osstest service owner wrote:
>> flight 59041 linux-3.18 real [real]
>> http://logs.test-lab.xenproject.org/osstest/logs/59041/
>>
>> Regressions :-(
>>
>> Tests which did not succeed and are blocking,
>> including tests which could not be run:
>>   test-amd64-amd64-xl-pvh-intel 11 guest-start              fail REGR. vs. 58581
> Not really sure who is on the hook for pvh fails these days, copying
> some likely suspects + x86 maintainers. Looks like the guest is triple
> faulting.


Looks like something happened between 2015-06-15 and 2015-06-29 (flights 
58581 and 58976) and it's only 3.18 kernel, right? (Sorry, I am not 
well-versed in osstest).

-boris



>
> http://logs.test-lab.xenproject.org/osstest/logs/59041/test-amd64-amd64-xl-pvh-intel/info.html
>
> http://logs.test-lab.xenproject.org/osstest/logs/59041/test-amd64-amd64-xl-pvh-intel/11.ts-guest-start.log
> 2015-07-03 19:50:21 Z executing ssh ... root@172.16.144.38 xl list
> 2015-07-03 19:50:21 Z guest debian.guest.osstest not present on this host
> debian.guest.osstest not running at ./ts-guest-start line 33.
>
> http://logs.test-lab.xenproject.org/osstest/logs/59041/test-amd64-amd64-xl-pvh-intel/serial-huxelrebe0.log
> Jul  3 19:50:21.557035 (d1) mapping kernel into physical memory
> Jul  3 19:50:21.661049 (d1) about to get started...
> Jul  3 19:50:21.661065 (d1) [    0.000000] Initializing cgroup subsys cpuset
> Jul  3 19:50:21.669059 (d1) [    0.000000] Initializing cgroup subsys cpu
> Jul  3 19:50:21.669078 (d1) [    0.000000] Initializing cgroup subsys cpuacct
> Jul  3 19:50:21.677045 (d1) [    0.000000] Linux version 3.18.17 (osstest@nocera0) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP Fri Jul 3 08:12:11 UTC 2015
> Jul  3 19:50:21.685259 (d1)
> Jul  3 19:50:21.685271 (d1) [    0.000000] Command line: root=/dev/xvda2 ro console=hvc0 earlyprintk=xen
> Jul  3 19:50:21.693061 (d1) [    0.000000] ACPI in unprivileged domain disabled
> Jul  3 19:50:21.701043 (d1) [    0.000000] e820: BIOS-provided physical RAM map:
> Jul  3 19:50:21.709048 (d1) [    0.000000] Xen: [mem 0x0000000000000000-0x000000001fffffff] usable
> Jul  3 19:50:21.709072 (d1) [    0.000000] bootconsole [xenboot0] enabled
> Jul  3 19:50:21.717050 (d1) [    0.000000] NX (Execute Disable) protection: active
> Jul  3 19:50:21.725042 (d1) [    0.000000] DMI not present or invalid.
> Jul  3 19:50:21.725062 (d1) [    0.000000] e820: last_pfn = 0x20000 max_arch_pfn = 0x400000000
> Jul  3 19:50:21.733047 (d1) [    0.000000] Scanning 1 areas for low memory corruption
> Jul  3 19:50:21.741040 (d1) [    0.000000] Using GB pages for direct mapping
> Jul  3 19:50:21.741060 (d1) [    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
> Jul  3 19:50:21.749053 (XEN) d1v0 Triple fault - invoking HVM shutdown action 0
>
>
>

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Ian Campbell]

On Mon, 2015-07-06 at 11:22 -0400, Boris Ostrovsky wrote:
> On 07/06/2015 04:32 AM, Ian Campbell wrote:
> > On Fri, 2015-07-03 at 22:21 +0000, osstest service owner wrote:
> >> flight 59041 linux-3.18 real [real]
> >> http://logs.test-lab.xenproject.org/osstest/logs/59041/
> >>
> >> Regressions :-(
> >>
> >> Tests which did not succeed and are blocking,
> >> including tests which could not be run:
> >>   test-amd64-amd64-xl-pvh-intel 11 guest-start              fail REGR. vs. 58581
> > Not really sure who is on the hook for pvh fails these days, copying
> > some likely suspects + x86 maintainers. Looks like the guest is triple
> > faulting.
> 
> 
> Looks like something happened between 2015-06-15 and 2015-06-29 (flights 
> 58581 and 58976)

Yes, and a bit of the original report I should have quoted is:

version targeted for testing:
 linux                ea5dd38e93b3bec3427e5d3eef000bbf5d637e76
baseline version:
 linux                d048c068d00da7d4cfa5ea7651933b99026958cf

See also the complete history of this test on this branch at
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-pvh-intel/linux-3.18.html

>  and it's only 3.18 kernel, right?

As far as I know. But: older linux-X.Y doesn't support PVH, so this test
has always failed at guest-start in those flights, before we would have
gotten to this failure.

The only newer stable branch which we test is linux-4.1, flight 59054
from didn't show this issue, but the next one might pick up something
newer, or 4.1.(y+1) might still be in the pipeline upstream and  the
issue will show up later.

We also test linux-linus and linux-next, so far neither of them seems to
exhibit this problem:
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-pvh-intel/linux-linus.html
http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-pvh-intel/linux-next.html

Which is interesting in itself I suppose.

Bisection was broken for linux-3.18 until this morning, Ian fixed the
config and it should pick up on this failure and start investigating
once the next flight completes (tonight some time).

Ian.

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Boris Ostrovsky]

On 07/06/2015 11:49 AM, Ian Campbell wrote:
> On Mon, 2015-07-06 at 11:22 -0400, Boris Ostrovsky wrote:
>> On 07/06/2015 04:32 AM, Ian Campbell wrote:
>>> On Fri, 2015-07-03 at 22:21 +0000, osstest service owner wrote:
>>>> flight 59041 linux-3.18 real [real]
>>>> http://logs.test-lab.xenproject.org/osstest/logs/59041/
>>>>
>>>> Regressions :-(
>>>>
>>>> Tests which did not succeed and are blocking,
>>>> including tests which could not be run:
>>>>    test-amd64-amd64-xl-pvh-intel 11 guest-start              fail REGR. vs. 58581
>>> Not really sure who is on the hook for pvh fails these days, copying
>>> some likely suspects + x86 maintainers. Looks like the guest is triple
>>> faulting.
>>
>> Looks like something happened between 2015-06-15 and 2015-06-29 (flights
>> 58581 and 58976)
> Yes, and a bit of the original report I should have quoted is:
>
> version targeted for testing:
>   linux                ea5dd38e93b3bec3427e5d3eef000bbf5d637e76
> baseline version:
>   linux                d048c068d00da7d4cfa5ea7651933b99026958cf
>
> See also the complete history of this test on this branch at
> http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-pvh-intel/linux-3.18.html
>
>>   and it's only 3.18 kernel, right?
> As far as I know. But: older linux-X.Y doesn't support PVH, so this test
> has always failed at guest-start in those flights, before we would have
> gotten to this failure.
>
> The only newer stable branch which we test is linux-4.1, flight 59054
> from didn't show this issue, but the next one might pick up something
> newer, or 4.1.(y+1) might still be in the pipeline upstream and  the
> issue will show up later.
>
> We also test linux-linus and linux-next, so far neither of them seems to
> exhibit this problem:
> http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-pvh-intel/linux-linus.html
> http://logs.test-lab.xenproject.org/osstest/results/history/test-amd64-amd64-xl-pvh-intel/linux-next.html
>
> Which is interesting in itself I suppose.
>
> Bisection was broken for linux-3.18 until this morning, Ian fixed the
> config and it should pick up on this failure and start investigating
> once the next flight completes (tonight some time).

OK, I'll wait until tomorrow then to see if it (the bisection) is 
successful.

BTW, our last test on Xen4.6/linux4.1+ was on July 1st and it passed too.

-boris

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Ian Campbell]

On Mon, 2015-07-06 at 12:06 -0400, Boris Ostrovsky wrote:
> > Bisection was broken for linux-3.18 until this morning, Ian fixed the
> > config and it should pick up on this failure and start investigating
> > once the next flight completes (tonight some time).
> 
> OK, I'll wait until tomorrow then to see if it (the bisection) is 
> successful.

After flight 59075 failed it has now started, progress is recorded at:

http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-3.18/test-amd64-amd64-xl-pvh-intel.guest-start.html

It's currently reproducing the baseline pass (the double line around a
cell indicates what it is currently testing). The hashes in each cell
are in the same order as the tree list at the top, i.e Linux is the
first entry and Xen is the last. 

Scheduling wise I don't think it is going to have made very much
progress by the time you get in today though.

> 
> BTW, our last test on Xen4.6/linux4.1+ was on July 1st and it passed too.
> 
> -boris
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Boris Ostrovsky]

On 07/07/2015 03:39 AM, Ian Campbell wrote:
> On Mon, 2015-07-06 at 12:06 -0400, Boris Ostrovsky wrote:
>>> Bisection was broken for linux-3.18 until this morning, Ian fixed the
>>> config and it should pick up on this failure and start investigating
>>> once the next flight completes (tonight some time).
>> OK, I'll wait until tomorrow then to see if it (the bisection) is
>> successful.
> After flight 59075 failed it has now started, progress is recorded at:
>
> http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-3.18/test-amd64-amd64-xl-pvh-intel.guest-start.html
>
> It's currently reproducing the baseline pass (the double line around a
> cell indicates what it is currently testing). The hashes in each cell
> are in the same order as the tree list at the top, i.e Linux is the
> first entry and Xen is the last.
>
> Scheduling wise I don't think it is going to have made very much
> progress by the time you get in today though.


So it is bisecting all five trees, right? (Well, four -- firmware hashes 
don't seem to be changing).

And if I assume that the issue is with kernel (which may not be a good 
assumption, but for the sake of my understanding of how the graph works) 
then d048c068d00d (green) was good and ea5dd38e93b3 (red) is bad? What 
about the one right below red (d24b9b8d95f0) --- is it bad or good?

-boris

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Ian Jackson]

Boris Ostrovsky writes ("Re: [Xen-devel] [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass"):
> So it is bisecting all five trees, right? (Well, four -- firmware hashes 
> don't seem to be changing).

Yes.  It has made the unified revision graph you see, where each node
is a version of each tree, and each edge changes the version of only
one of the trees.

> And if I assume that the issue is with kernel (which may not be a good 
> assumption, but for the sake of my understanding of how the graph works) 
> then d048c068d00d (green) was good and ea5dd38e93b3 (red) is bad?

Yes.

> What about the one right below red (d24b9b8d95f0) --- is it bad or
> good?

It has not been tested.  And, depending on what happens, it may not
ever be tested.

Ian.

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Ian Campbell]

On Tue, 2015-07-07 at 10:55 -0400, Boris Ostrovsky wrote:
> On 07/07/2015 03:39 AM, Ian Campbell wrote:
> > On Mon, 2015-07-06 at 12:06 -0400, Boris Ostrovsky wrote:
> >>> Bisection was broken for linux-3.18 until this morning, Ian fixed the
> >>> config and it should pick up on this failure and start investigating
> >>> once the next flight completes (tonight some time).
> >> OK, I'll wait until tomorrow then to see if it (the bisection) is
> >> successful.
> > After flight 59075 failed it has now started, progress is recorded at:
> >
> > http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-3.18/test-amd64-amd64-xl-pvh-intel.guest-start.html
> >
> > It's currently reproducing the baseline pass (the double line around a
> > cell indicates what it is currently testing). The hashes in each cell
> > are in the same order as the tree list at the top, i.e Linux is the
> > first entry and Xen is the last.
> >
> > Scheduling wise I don't think it is going to have made very much
> > progress by the time you get in today though.
> 
> 
> So it is bisecting all five trees, right? (Well, four -- firmware hashes 
> don't seem to be changing).

Correct.

> And if I assume that the issue is with kernel (which may not be a good 
> assumption, but for the sake of my understanding of how the graph works) 
> then d048c068d00d (green) was good and ea5dd38e93b3 (red) is bad?

Correct.

>  What 
> about the one right below red (d24b9b8d95f0) --- is it bad or good?

Gray means not yet tested, so we don't know.

Once it has established the baseline pass and fail by repeating each of
those three times then it will start testing things in the middle of the
graph and narrowing it down until it can identify the bad commit. I'd
expect this to take a few days at this point.

The other colours I think you might see are yellow, which means
unreliable (i.e. not failing of passing consistently) which will
probably result in the bisector giving up and blue which means blocked
which means we cannot test this revision because something failed prior
to the step which being bisected (i.e. failed to build or boot Xen, so
can't tell if the guest-start would work or not), in which case the
bisector will keep trying commits.

Ian.

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Boris Ostrovsky]

On 07/07/2015 11:08 AM, Ian Campbell wrote:
> On Tue, 2015-07-07 at 10:55 -0400, Boris Ostrovsky wrote:
>> On 07/07/2015 03:39 AM, Ian Campbell wrote:
>>> On Mon, 2015-07-06 at 12:06 -0400, Boris Ostrovsky wrote:
>>>>> Bisection was broken for linux-3.18 until this morning, Ian fixed the
>>>>> config and it should pick up on this failure and start investigating
>>>>> once the next flight completes (tonight some time).
>>>> OK, I'll wait until tomorrow then to see if it (the bisection) is
>>>> successful.
>>> After flight 59075 failed it has now started, progress is recorded at:
>>>
>>> http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-3.18/test-amd64-amd64-xl-pvh-intel.guest-start.html
>>>
>>> It's currently reproducing the baseline pass (the double line around a
>>> cell indicates what it is currently testing). The hashes in each cell
>>> are in the same order as the tree list at the top, i.e Linux is the
>>> first entry and Xen is the last.
>>>
>>> Scheduling wise I don't think it is going to have made very much
>>> progress by the time you get in today though.
>>
>> So it is bisecting all five trees, right? (Well, four -- firmware hashes
>> don't seem to be changing).
> Correct.
>
>> And if I assume that the issue is with kernel (which may not be a good
>> assumption, but for the sake of my understanding of how the graph works)
>> then d048c068d00d (green) was good and ea5dd38e93b3 (red) is bad?
> Correct.
>
>>   What
>> about the one right below red (d24b9b8d95f0) --- is it bad or good?
> Gray means not yet tested, so we don't know.
>
> Once it has established the baseline pass and fail by repeating each of
> those three times then it will start testing things in the middle of the
> graph and narrowing it down until it can identify the bad commit. I'd
> expect this to take a few days at this point.

OK, I think I'll try bisecting kernel myself then (if I can reproduce 
this locally).

Thanks (and to IanJ too).

-boris


>
> The other colours I think you might see are yellow, which means
> unreliable (i.e. not failing of passing consistently) which will
> probably result in the bisector giving up and blue which means blocked
> which means we cannot test this revision because something failed prior
> to the step which being bisected (i.e. failed to build or boot Xen, so
> can't tell if the guest-start would work or not), in which case the
> bisector will keep trying commits.
>
> Ian.
>

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Boris Ostrovsky]

On 07/07/2015 11:16 AM, Boris Ostrovsky wrote:
> On 07/07/2015 11:08 AM, Ian Campbell wrote:
>> On Tue, 2015-07-07 at 10:55 -0400, Boris Ostrovsky wrote:
>>> On 07/07/2015 03:39 AM, Ian Campbell wrote:
>>>> On Mon, 2015-07-06 at 12:06 -0400, Boris Ostrovsky wrote:
>>>>>> Bisection was broken for linux-3.18 until this morning, Ian fixed 
>>>>>> the
>>>>>> config and it should pick up on this failure and start investigating
>>>>>> once the next flight completes (tonight some time).
>>>>> OK, I'll wait until tomorrow then to see if it (the bisection) is
>>>>> successful.
>>>> After flight 59075 failed it has now started, progress is recorded at:
>>>>
>>>> http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-3.18/test-amd64-amd64-xl-pvh-intel.guest-start.html 
>>>>
>>>>
>>>> It's currently reproducing the baseline pass (the double line around a
>>>> cell indicates what it is currently testing). The hashes in each cell
>>>> are in the same order as the tree list at the top, i.e Linux is the
>>>> first entry and Xen is the last.
>>>>
>>>> Scheduling wise I don't think it is going to have made very much
>>>> progress by the time you get in today though.
>>>
>>> So it is bisecting all five trees, right? (Well, four -- firmware 
>>> hashes
>>> don't seem to be changing).
>> Correct.
>>
>>> And if I assume that the issue is with kernel (which may not be a good
>>> assumption, but for the sake of my understanding of how the graph 
>>> works)
>>> then d048c068d00d (green) was good and ea5dd38e93b3 (red) is bad?
>> Correct.
>>
>>>   What
>>> about the one right below red (d24b9b8d95f0) --- is it bad or good?
>> Gray means not yet tested, so we don't know.
>>
>> Once it has established the baseline pass and fail by repeating each of
>> those three times then it will start testing things in the middle of the
>> graph and narrowing it down until it can identify the bad commit. I'd
>> expect this to take a few days at this point.
>
> OK, I think I'll try bisecting kernel myself then (if I can reproduce 
> this locally).


I didn't need to bisect this, the bad commit is 63753fac67. I am 
actually surprised this went into a stable kernel since that patch was a 
performance improvement (a not a huge one at that, I suspect).

Anyway, this kernel needs 5054daa285 from upstream.


-boris

Re: [linux-3.18 test] 59041: regressions - trouble: blocked/broken/fail/pass

[Ian Campbell]

On Tue, 2015-07-07 at 15:22 -0400, Boris Ostrovsky wrote:
> I didn't need to bisect this, the bad commit is 63753fac67. I am 
> actually surprised this went into a stable kernel since that patch was a 
> performance improvement (a not a huge one at that, I suspect).
> 
> Anyway, this kernel needs 5054daa285 from upstream.

Great. I take it you will take care of requesting that to the
appropriate people?

Ian.