* [xen-unstable test] 88047: regressions - FAIL
@ 2016-04-01 6:12 osstest service owner
2016-04-01 9:43 ` Jan Beulich
0 siblings, 1 reply; 7+ messages in thread
From: osstest service owner @ 2016-04-01 6:12 UTC (permalink / raw)
To: xen-devel, osstest-admin
[-- Attachment #1: Type: text/plain, Size: 12782 bytes --]
flight 88047 xen-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/88047/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-amd64-i386-libvirt-pair 21 guest-migrate/src_host/dst_host fail REGR. vs. 86491
test-amd64-amd64-libvirt-pair 21 guest-migrate/src_host/dst_host fail REGR. vs. 86491
Regressions which are regarded as allowable (not blocking):
build-i386-rumpuserxen 6 xen-build fail like 86491
test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop fail like 86491
build-amd64-rumpuserxen 6 xen-build fail like 86491
test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop fail like 86491
test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop fail like 86491
test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop fail like 86491
Tests which did not succeed, but are not blocking:
test-amd64-i386-rumpuserxen-i386 1 build-check(1) blocked n/a
test-amd64-amd64-rumpuserxen-amd64 1 build-check(1) blocked n/a
test-amd64-amd64-xl-pvh-amd 11 guest-start fail never pass
test-amd64-amd64-xl-pvh-intel 11 guest-start fail never pass
test-armhf-armhf-libvirt-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt-xsm 14 guest-saverestore fail never pass
test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2 fail never pass
test-amd64-i386-libvirt-xsm 12 migrate-support-check fail never pass
test-amd64-amd64-libvirt-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 12 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 13 saverestore-support-check fail never pass
test-amd64-i386-libvirt 12 migrate-support-check fail never pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 12 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-multivcpu 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-multivcpu 12 migrate-support-check fail never pass
test-amd64-amd64-libvirt-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-xsm 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-xsm 12 migrate-support-check fail never pass
test-amd64-amd64-libvirt 12 migrate-support-check fail never pass
test-armhf-armhf-xl 12 migrate-support-check fail never pass
test-armhf-armhf-xl 13 saverestore-support-check fail never pass
test-armhf-armhf-libvirt 14 guest-saverestore fail never pass
test-armhf-armhf-libvirt 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt-raw 13 guest-saverestore fail never pass
test-armhf-armhf-libvirt-raw 11 migrate-support-check fail never pass
test-armhf-armhf-libvirt-qcow2 11 migrate-support-check fail never pass
test-armhf-armhf-libvirt-qcow2 13 guest-saverestore fail never pass
test-armhf-armhf-xl-rtds 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-rtds 12 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-credit2 12 migrate-support-check fail never pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 12 saverestore-support-check fail never pass
version targeted for testing:
xen 72a91f67714638ea664155ee253bcb02b1c6ae54
baseline version:
xen a6f2cdb633bf519244a16674031b8034b581ba7f
Last test of basis 86491 2016-03-17 15:24:59 Z 14 days
Failing since 86560 2016-03-18 10:56:34 Z 13 days 17 attempts
Testing same since 88047 2016-03-30 21:28:27 Z 1 days 1 attempts
------------------------------------------------------------
People who touched revisions under test:
Andrew Cooper <andrew.cooper3@citrix.com>
Anthony PERARD <anthony.perard@citrix.com>
Boris Ostrovsky <boris.ostrovsky@oracle.com>
Chunyan Liu <cyliu@suse.com>
Dagaen Golomb <dgolomb@seas.upenn.edu>
Daniel De Graaf <dgdegra@tycho.nsa.gov>
Dario Faggioli <dario.faggioli@citrix.com>
Dave Scott <dave@recoil.org>
David Scott <dave@recoil.org>
David Vrabel <david.vrabel@citrix.com>
Doug Goldstein <cardoe@cardoe.com>
George Dunlap <george.dunlap@citrix.com>
George Dunlap <george.dunlap@eu.citrix.com>
Ian Campbell <ian.campbell@citrix.com>
Ian Jackson <Ian.Jackson@eu.citrix.com>
Jan Beulich <jbeulich@suse.com>
Jim Fehlig <jfehlig@suse.com>
Jonathan Davies <jonathan.davies@citrix.com>
Juergen Gross <jgross@suse.com>
Julien Grall <julien.grall@arm.com>
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Meng Xu <mengxu@cis.upenn.edu>
Naresh Bhat <naresh.bhat@linaro.org>
Olaf Hering <olaf@aepfle.de>
Parth Dixit <parth.dixit@linaro.org>
Paul Durrant <paul.durrant@citrix.com>
Roger Pau Monne <roger.pau@citrix.com>
Roger Pau Monné <roger.pau@citrix.com>
Shannon Zhao <shannon.zhao@linaro.org>
Shuai Ruan <shuai.ruan@linux.intel.com>
Simon Cao <caobosimon@gmail.com>
Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Tianyang Chen <tiche@seas.upenn.edu>
Tim Deegan <tim@xen.org>
Wei Liu <wei.liu2@citrix.com>
Wen Congyang <wency@cn.fujitsu.com>
Zhigang Wang <zhigang.x.wang@oracle.com>
jobs:
build-amd64-xsm pass
build-armhf-xsm pass
build-i386-xsm pass
build-amd64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-oldkern pass
build-i386-oldkern pass
build-amd64-prev pass
build-i386-prev pass
build-amd64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
build-amd64-rumpuserxen fail
build-i386-rumpuserxen fail
test-amd64-amd64-xl pass
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-xsm pass
test-armhf-armhf-libvirt-xsm fail
test-amd64-i386-libvirt-xsm pass
test-amd64-amd64-xl-xsm pass
test-armhf-armhf-xl-xsm pass
test-amd64-i386-xl-xsm pass
test-amd64-amd64-qemuu-nested-amd fail
test-amd64-amd64-xl-pvh-amd fail
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-rumpuserxen-amd64 blocked
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-armhf-armhf-xl-arndale pass
test-amd64-amd64-xl-credit2 pass
test-armhf-armhf-xl-credit2 pass
test-armhf-armhf-xl-cubietruck pass
test-amd64-i386-freebsd10-i386 pass
test-amd64-i386-rumpuserxen-i386 blocked
test-amd64-amd64-qemuu-nested-intel pass
test-amd64-amd64-xl-pvh-intel fail
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-armhf-armhf-libvirt fail
test-amd64-i386-libvirt pass
test-amd64-amd64-migrupgrade pass
test-amd64-i386-migrupgrade pass
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-libvirt-pair fail
test-amd64-i386-libvirt-pair fail
test-amd64-amd64-amd64-pvgrub pass
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-pygrub pass
test-armhf-armhf-libvirt-qcow2 fail
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-libvirt-raw fail
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-rtds pass
test-armhf-armhf-xl-rtds pass
test-amd64-i386-xl-qemut-winxpsp3-vcpus1 pass
test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-xl-vhd pass
test-amd64-amd64-xl-qemut-winxpsp3 pass
test-amd64-i386-xl-qemut-winxpsp3 pass
test-amd64-amd64-xl-qemuu-winxpsp3 pass
test-amd64-i386-xl-qemuu-winxpsp3 pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
Not pushing.
(No revision log; it would be 1546 lines long.)
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 88047: regressions - FAIL
2016-04-01 6:12 [xen-unstable test] 88047: regressions - FAIL osstest service owner
@ 2016-04-01 9:43 ` Jan Beulich
2016-04-01 13:36 ` Ian Jackson
0 siblings, 1 reply; 7+ messages in thread
From: Jan Beulich @ 2016-04-01 9:43 UTC (permalink / raw)
To: osstest-admin; +Cc: xen-devel
>>> On 01.04.16 at 08:12, <osstest-admin@xenproject.org> wrote:
> flight 88047 xen-unstable real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/88047/
>
> Regressions :-(
>
> Tests which did not succeed and are blocking,
> including tests which could not be run:
> test-amd64-i386-libvirt-pair 21 guest-migrate/src_host/dst_host fail REGR. vs. 86491
> test-amd64-amd64-libvirt-pair 21 guest-migrate/src_host/dst_host fail REGR. vs. 86491
Both say
2016-04-01 02:00:56 Z executing ssh ... root@172.16.144.20 virsh migrate --live debian.guest.osstest xen+ssh://merlot1
error: Timed out during operation: cannot acquire state change lock
and I didn't spot anything interesting in the serial logs. I can't
find the above error message anywhere in the tools, so I suspect
this is an osstest one. Does anyone have any idea of what's
going on/wrong here?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 88047: regressions - FAIL
2016-04-01 9:43 ` Jan Beulich
@ 2016-04-01 13:36 ` Ian Jackson
2016-04-01 13:44 ` Jan Beulich
0 siblings, 1 reply; 7+ messages in thread
From: Ian Jackson @ 2016-04-01 13:36 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel, osstest-admin
Jan Beulich writes ("Re: [Xen-devel] [xen-unstable test] 88047: regressions - FAIL"):
> On 01.04.16 at 08:12, <osstest-admin@xenproject.org> wrote:
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> > test-amd64-i386-libvirt-pair 21 guest-migrate/src_host/dst_host fail REGR. vs. 86491
> > test-amd64-amd64-libvirt-pair 21 guest-migrate/src_host/dst_host fail REGR. vs. 86491
>
> Both say
>
> 2016-04-01 02:00:56 Z executing ssh ... root@172.16.144.20 virsh migrate --live debian.guest.osstest xen+ssh://merlot1
> error: Timed out during operation: cannot acquire state change lock
>
> and I didn't spot anything interesting in the serial logs. I can't
> find the above error message anywhere in the tools, so I suspect
> this is an osstest one. Does anyone have any idea of what's
> going on/wrong here?
That message is from libvirt. I don't know the exact situation.
But we have seen other timeouts with merlot[01]. They seem to
sometimes have weird stalls when running under Xen.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 88047: regressions - FAIL
2016-04-01 13:36 ` Ian Jackson
@ 2016-04-01 13:44 ` Jan Beulich
2016-04-01 14:40 ` Dario Faggioli
0 siblings, 1 reply; 7+ messages in thread
From: Jan Beulich @ 2016-04-01 13:44 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel, osstest-admin
>>> On 01.04.16 at 15:36, <Ian.Jackson@eu.citrix.com> wrote:
> Jan Beulich writes ("Re: [Xen-devel] [xen-unstable test] 88047: regressions -
> FAIL"):
>> On 01.04.16 at 08:12, <osstest-admin@xenproject.org> wrote:
>> > Tests which did not succeed and are blocking,
>> > including tests which could not be run:
>> > test-amd64-i386-libvirt-pair 21 guest-migrate/src_host/dst_host fail REGR. vs. 86491
>> > test-amd64-amd64-libvirt-pair 21 guest-migrate/src_host/dst_host fail REGR. vs. 86491
>>
>> Both say
>>
>> 2016-04-01 02:00:56 Z executing ssh ... root@172.16.144.20 virsh migrate --live debian.guest.osstest xen+ssh://merlot1
>> error: Timed out during operation: cannot acquire state change lock
>>
>> and I didn't spot anything interesting in the serial logs. I can't
>> find the above error message anywhere in the tools, so I suspect
>> this is an osstest one. Does anyone have any idea of what's
>> going on/wrong here?
>
> That message is from libvirt. I don't know the exact situation.
>
> But we have seen other timeouts with merlot[01]. They seem to
> sometimes have weird stalls when running under Xen.
Might it be worth trying whether that's a deep C-state issue (by
temporarily limiting their use of deep C states)?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 88047: regressions - FAIL
2016-04-01 13:44 ` Jan Beulich
@ 2016-04-01 14:40 ` Dario Faggioli
2016-04-01 14:56 ` Ian Jackson
0 siblings, 1 reply; 7+ messages in thread
From: Dario Faggioli @ 2016-04-01 14:40 UTC (permalink / raw)
To: Jan Beulich, Ian Jackson; +Cc: xen-devel, osstest-admin
[-- Attachment #1.1: Type: text/plain, Size: 1591 bytes --]
On Fri, 2016-04-01 at 07:44 -0600, Jan Beulich wrote:
> > On 01.04.16 at 15:36, <Ian.Jackson@eu.citrix.com> wrote:
> > That message is from libvirt. I don't know the exact situation.
> >
> > But we have seen other timeouts with merlot[01]. They seem to
> > sometimes have weird stalls when running under Xen.
> Might it be worth trying whether that's a deep C-state issue (by
> temporarily limiting their use of deep C states)?
>
IIRC, it's the box(es) with the weird NUMA setup... something about
nodes without any memory:
Apr 1 01:03:31.095559 (XEN) SRAT: Node 0 PXM 0 0-a0000
Apr 1 01:03:31.103546 (XEN) SRAT: Node 0 PXM 0 100000-c0000000
Apr 1 01:03:31.103565 (XEN) SRAT: Node 0 PXM 0 100000000-240000000
Apr 1 01:03:31.111600 (XEN) SRAT: Node 2 PXM 2 240000000-43f000000
Apr 1 01:03:31.111619 (XEN) NUMA: Allocated memnodemap from 43ee16000 - 43ee1b000
Apr 1 01:03:31.119615 (XEN) NUMA: Using 8 for the hash shift.
Apr 1 01:03:31.127613 (XEN) SRAT: Node 1 has no memory. BIOS Bug or mis-configured hardware?
Apr 1 01:03:31.135548 (XEN) SRAT: Node 3 has no memory. BIOS Bug or mis-configured hardware?
Not that I see how this could cause the long latencies / timeouts, though... :-/
In any case, I think Jan's C states suggestion is worth a shot.
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 88047: regressions - FAIL
2016-04-01 14:40 ` Dario Faggioli
@ 2016-04-01 14:56 ` Ian Jackson
2016-04-01 15:32 ` Jan Beulich
0 siblings, 1 reply; 7+ messages in thread
From: Ian Jackson @ 2016-04-01 14:56 UTC (permalink / raw)
To: Dario Faggioli; +Cc: xen-devel, osstest-admin, Jan Beulich
Dario Faggioli writes ("Re: [Xen-devel] [xen-unstable test] 88047: regressions - FAIL"):
> On Fri, 2016-04-01 at 07:44 -0600, Jan Beulich wrote:
> > Might it be worth trying whether that's a deep C-state issue (by
> > temporarily limiting their use of deep C states)?
I'm happy to try this.
> IIRC, it's the box(es) with the weird NUMA setup... something about
> nodes without any memory:
Right.
> Not that I see how this could cause the long latencies / timeouts, though... :-/
>
> In any case, I think Jan's C states suggestion is worth a shot.
Forgive my ignorance, but how would I do that ?
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [xen-unstable test] 88047: regressions - FAIL
2016-04-01 14:56 ` Ian Jackson
@ 2016-04-01 15:32 ` Jan Beulich
0 siblings, 0 replies; 7+ messages in thread
From: Jan Beulich @ 2016-04-01 15:32 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel, Dario Faggioli, osstest-admin
>>> On 01.04.16 at 16:56, <Ian.Jackson@eu.citrix.com> wrote:
> Dario Faggioli writes ("Re: [Xen-devel] [xen-unstable test] 88047: regressions -
> FAIL"):
>> On Fri, 2016-04-01 at 07:44 -0600, Jan Beulich wrote:
>> > Might it be worth trying whether that's a deep C-state issue (by
>> > temporarily limiting their use of deep C states)?
>
> I'm happy to try this.
>
>> IIRC, it's the box(es) with the weird NUMA setup... something about
>> nodes without any memory:
>
> Right.
>
>> Not that I see how this could cause the long latencies / timeouts, though...
> :-/
>>
>> In any case, I think Jan's C states suggestion is worth a shot.
>
> Forgive my ignorance, but how would I do that ?
As a first shot, disable them altogether by "max_cstate=1" on the
hypervisor command line. If that helps, we can bi-sect deeper
states as well as see whether there's a difference between the two
idle drivers.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-04-01 15:32 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-04-01 6:12 [xen-unstable test] 88047: regressions - FAIL osstest service owner
2016-04-01 9:43 ` Jan Beulich
2016-04-01 13:36 ` Ian Jackson
2016-04-01 13:44 ` Jan Beulich
2016-04-01 14:40 ` Dario Faggioli
2016-04-01 14:56 ` Ian Jackson
2016-04-01 15:32 ` Jan Beulich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).