* [xen-unstable test] 58821: tolerable FAIL
@ 2015-06-22 14:09 osstest service user
2015-06-22 15:17 ` Ian Campbell
0 siblings, 1 reply; 5+ messages in thread
From: osstest service user @ 2015-06-22 14:09 UTC (permalink / raw)
To: xen-devel; +Cc: ian.jackson
flight 58821 xen-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/58821/
Failures :-/ but no regressions.
Tests which are failing intermittently (not blocking):
test-armhf-armhf-xl-credit2 14 guest-start.2 fail in 58789 pass in 58821
test-amd64-i386-rumpuserxen-i386 15 rumpuserxen-demo-xenstorels/xenstorels.repeat fail pass in 58789
Regressions which are regarded as allowable (not blocking):
test-amd64-i386-libvirt-xsm 11 guest-start fail like 58789
test-amd64-i386-libvirt 11 guest-start fail like 58789
test-amd64-amd64-libvirt-xsm 11 guest-start fail like 58789
test-amd64-amd64-libvirt 11 guest-start fail like 58789
test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop fail like 58789
test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop fail like 58789
Tests which did not succeed, but are not blocking:
test-amd64-amd64-xl-pvh-intel 11 guest-start fail never pass
test-amd64-amd64-xl-pvh-amd 11 guest-start fail never pass
test-armhf-armhf-xl-arndale 12 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 12 migrate-support-check fail never pass
test-armhf-armhf-xl-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-xl 12 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 12 migrate-support-check fail never pass
test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop fail never pass
test-armhf-armhf-xl-cubietruck 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-xl-sedf-pin 12 migrate-support-check fail never pass
test-armhf-armhf-xl-sedf 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt 12 migrate-support-check fail never pass
test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop fail never pass
version targeted for testing:
xen e76ff6c156906b515c2a4300a81c95886ece5d5f
baseline version:
xen e76ff6c156906b515c2a4300a81c95886ece5d5f
jobs:
build-amd64-xsm pass
build-armhf-xsm pass
build-i386-xsm pass
build-amd64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-oldkern pass
build-i386-oldkern pass
build-amd64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
build-amd64-rumpuserxen pass
build-i386-rumpuserxen pass
test-amd64-amd64-xl pass
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-xsm fail
test-armhf-armhf-libvirt-xsm pass
test-amd64-i386-libvirt-xsm fail
test-amd64-amd64-xl-xsm pass
test-armhf-armhf-xl-xsm pass
test-amd64-i386-xl-xsm pass
test-amd64-amd64-xl-pvh-amd fail
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-rumpuserxen-amd64 pass
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-armhf-armhf-xl-arndale pass
test-amd64-amd64-xl-credit2 pass
test-armhf-armhf-xl-credit2 pass
test-armhf-armhf-xl-cubietruck pass
test-amd64-i386-freebsd10-i386 pass
test-amd64-i386-rumpuserxen-i386 fail
test-amd64-amd64-xl-pvh-intel fail
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt fail
test-armhf-armhf-libvirt pass
test-amd64-i386-libvirt fail
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-xl-sedf-pin pass
test-armhf-armhf-xl-sedf-pin pass
test-amd64-amd64-xl-sedf pass
test-armhf-armhf-xl-sedf pass
test-amd64-i386-xl-qemut-winxpsp3-vcpus1 pass
test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 pass
test-amd64-amd64-xl-qemut-winxpsp3 pass
test-amd64-i386-xl-qemut-winxpsp3 pass
test-amd64-amd64-xl-qemuu-winxpsp3 pass
test-amd64-i386-xl-qemuu-winxpsp3 pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
Published tested tree is already up to date.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [xen-unstable test] 58821: tolerable FAIL
2015-06-22 14:09 [xen-unstable test] 58821: tolerable FAIL osstest service user
@ 2015-06-22 15:17 ` Ian Campbell
2015-06-22 15:36 ` Jan Beulich
0 siblings, 1 reply; 5+ messages in thread
From: Ian Campbell @ 2015-06-22 15:17 UTC (permalink / raw)
To: Jan Beulich, Andrew Cooper; +Cc: xen-devel, ian.jackson
On Mon, 2015-06-22 at 14:09 +0000, osstest service user wrote:
> flight 58821 xen-unstable real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/58821/
>
[...]
> test-amd64-amd64-libvirt 11 guest-start fail like 58789
http://logs.test-lab.xenproject.org/osstest/logs/58821/test-amd64-amd64-libvirt/info.html
While investigating why libvirt hasn't been succeeding very well on
merlot* I came across some things in the serial log which initially
struck me as odd, but which I suspect are nothing (or at least not
terribly relevant), if someone could confirm that would be great.
Firstly is:
Jun 22 12:41:09.633294 (XEN) microcode: CPU2 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:09.665099 (XEN) microcode: CPU4 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:09.729089 (XEN) microcode: CPU6 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:09.793224 (XEN) microcode: CPU8 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:09.857118 (XEN) microcode: CPU10 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:09.921123 (XEN) microcode: CPU12 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:09.985563 (XEN) microcode: CPU14 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:10.049212 (XEN) microcode: CPU16 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:10.121106 (XEN) microcode: CPU18 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:10.185059 (XEN) microcode: CPU20 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:10.249070 (XEN) microcode: CPU22 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:10.313063 (XEN) microcode: CPU24 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:10.393217 (XEN) microcode: CPU26 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:10.457126 (XEN) microcode: CPU28 updated from revision 0x6000822 to 0x6000832
Jun 22 12:41:10.521228 (XEN) microcode: CPU30 updated from revision 0x6000822 to 0x6000832
i.e. only even numbered cpus are updated. (0 was done earlier in boot).
I suspect that the answer here is "hyperthreading", and the cpuinfo
shows all cpus have in fact been updated.
So I think that's just a red-herring.
The second thing is:
Jun 22 12:41:10.601103 (XEN) Brought up 32 CPUs
Jun 22 12:41:10.625270 (XEN) Testing NMI watchdog on all CPUs: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 stuck
i.e. at least one CPU has issues with NMI watchdog (looking at other
runs it seems to vary between 29-31). Is this just that the NMI watchdog
doesn't deal well with so many pCPUs? Or is it a real issue?
Lastly the eventual "'0' pressed -> dumping Dom0's registers" thing only
seems to dump cpus 0..9 (inclusive). It seems to vary a bit. I suspect
this is another "didn't wait long enough" thing, possibly on the osstest
end.
Ian.
[0] http://logs.test-lab.xenproject.org/osstest/logs/58821/test-amd64-amd64-libvirt/serial-merlot1.log
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [xen-unstable test] 58821: tolerable FAIL
2015-06-22 15:17 ` Ian Campbell
@ 2015-06-22 15:36 ` Jan Beulich
2015-06-22 16:00 ` Andrew Cooper
0 siblings, 1 reply; 5+ messages in thread
From: Jan Beulich @ 2015-06-22 15:36 UTC (permalink / raw)
To: Ian Campbell
Cc: Andrew Cooper, xen-devel, Aravind Gopalakrishnan, ian.jackson,
suravee.suthikulpanit
>>> On 22.06.15 at 17:17, <ian.campbell@citrix.com> wrote:
> On Mon, 2015-06-22 at 14:09 +0000, osstest service user wrote:
>> flight 58821 xen-unstable real [real]
>> http://logs.test-lab.xenproject.org/osstest/logs/58821/
>>
> [...]
>> test-amd64-amd64-libvirt 11 guest-start fail like
> 58789
>
> http://logs.test-lab.xenproject.org/osstest/logs/58821/test-amd64-amd64-libv
> irt/info.html
>
> While investigating why libvirt hasn't been succeeding very well on
> merlot* I came across some things in the serial log which initially
> struck me as odd, but which I suspect are nothing (or at least not
> terribly relevant), if someone could confirm that would be great.
>
> Firstly is:
>
> Jun 22 12:41:09.633294 (XEN) microcode: CPU2 updated from revision 0x6000822 to 0x6000832
> Jun 22 12:41:09.665099 (XEN) microcode: CPU4 updated from revision 0x6000822 to 0x6000832
> Jun 22 12:41:09.729089 (XEN) microcode: CPU6 updated from revision 0x6000822 to 0x6000832
> [...]
>
> i.e. only even numbered cpus are updated. (0 was done earlier in boot).
> I suspect that the answer here is "hyperthreading", and the cpuinfo
> shows all cpus have in fact been updated.
Yes (albeit hyperthreading is an Intel term, but iirc the same applies
to the two cores per compute unit).
> The second thing is:
> Jun 22 12:41:10.601103 (XEN) Brought up 32 CPUs
> Jun 22 12:41:10.625270 (XEN) Testing NMI watchdog on all CPUs: 0 1 2 3 4 5 6
> 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 stuck
>
> i.e. at least one CPU has issues with NMI watchdog (looking at other
> runs it seems to vary between 29-31). Is this just that the NMI watchdog
> doesn't deal well with so many pCPUs? Or is it a real issue?
Very few CPUs properly responding is certainly quite odd; one
would expect all or none of them to work. Perhaps our AMD
maintainers (now Cc-ed) could take a look...
> Lastly the eventual "'0' pressed -> dumping Dom0's registers" thing only
> seems to dump cpus 0..9 (inclusive). It seems to vary a bit. I suspect
> this is another "didn't wait long enough" thing, possibly on the osstest
> end.
This looks like an issue with the serial line not coping with the amount
of output (and hence dropping bits). Later on in the log there's also
quite large an unreadable part, which I would also attribute to such
a problem.
Jan
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [xen-unstable test] 58821: tolerable FAIL
2015-06-22 15:36 ` Jan Beulich
@ 2015-06-22 16:00 ` Andrew Cooper
2015-06-22 16:05 ` Ian Campbell
0 siblings, 1 reply; 5+ messages in thread
From: Andrew Cooper @ 2015-06-22 16:00 UTC (permalink / raw)
To: Jan Beulich, Ian Campbell
Cc: ian.jackson, xen-devel, Aravind Gopalakrishnan, suravee.suthikulpanit
On 22/06/15 16:36, Jan Beulich wrote:
>>>> On 22.06.15 at 17:17, <ian.campbell@citrix.com> wrote:
>> On Mon, 2015-06-22 at 14:09 +0000, osstest service user wrote:
>>> flight 58821 xen-unstable real [real]
>>> http://logs.test-lab.xenproject.org/osstest/logs/58821/
>>>
>> [...]
>>> test-amd64-amd64-libvirt 11 guest-start fail like
>> 58789
>>
>> http://logs.test-lab.xenproject.org/osstest/logs/58821/test-amd64-amd64-libv
>> irt/info.html
>>
>> While investigating why libvirt hasn't been succeeding very well on
>> merlot* I came across some things in the serial log which initially
>> struck me as odd, but which I suspect are nothing (or at least not
>> terribly relevant), if someone could confirm that would be great.
>>
>> Firstly is:
>>
>> Jun 22 12:41:09.633294 (XEN) microcode: CPU2 updated from revision 0x6000822 to 0x6000832
>> Jun 22 12:41:09.665099 (XEN) microcode: CPU4 updated from revision 0x6000822 to 0x6000832
>> Jun 22 12:41:09.729089 (XEN) microcode: CPU6 updated from revision 0x6000822 to 0x6000832
>> [...]
>>
>> i.e. only even numbered cpus are updated. (0 was done earlier in boot).
>> I suspect that the answer here is "hyperthreading", and the cpuinfo
>> shows all cpus have in fact been updated.
> Yes (albeit hyperthreading is an Intel term, but iirc the same applies
> to the two cores per compute unit).
Indeed. The "microcode: patch is already at required level or
greater.\n" message is helpfully unconditionally compiled out.
>
>> The second thing is:
>> Jun 22 12:41:10.601103 (XEN) Brought up 32 CPUs
>> Jun 22 12:41:10.625270 (XEN) Testing NMI watchdog on all CPUs: 0 1 2 3 4 5 6
>> 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 stuck
>>
>> i.e. at least one CPU has issues with NMI watchdog (looking at other
>> runs it seems to vary between 29-31). Is this just that the NMI watchdog
>> doesn't deal well with so many pCPUs? Or is it a real issue?
> Very few CPUs properly responding is certainly quite odd; one
> would expect all or none of them to work. Perhaps our AMD
> maintainers (now Cc-ed) could take a look...
There are several things wrong with the NMI testing in Xen atm,
following some recent investigation in XenServer. Time isn't accounted
properly for cores under bios/hardware power control, and Xen doesn't
wait for the requisite time even if the core were running at its
expected frequency.
I should see about making those patches appear, but for now, ignore this
line. It is more than likely wrong.
~Andrew
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [xen-unstable test] 58821: tolerable FAIL
2015-06-22 16:00 ` Andrew Cooper
@ 2015-06-22 16:05 ` Ian Campbell
0 siblings, 0 replies; 5+ messages in thread
From: Ian Campbell @ 2015-06-22 16:05 UTC (permalink / raw)
To: Andrew Cooper
Cc: ian.jackson, xen-devel, Aravind Gopalakrishnan,
suravee.suthikulpanit, Jan Beulich
On Mon, 2015-06-22 at 17:00 +0100, Andrew Cooper wrote:
> On 22/06/15 16:36, Jan Beulich wrote:
> >>>> On 22.06.15 at 17:17, <ian.campbell@citrix.com> wrote:
> >> On Mon, 2015-06-22 at 14:09 +0000, osstest service user wrote:
> >>> flight 58821 xen-unstable real [real]
> >>> http://logs.test-lab.xenproject.org/osstest/logs/58821/
> >>>
> >> [...]
> >>> test-amd64-amd64-libvirt 11 guest-start fail like
> >> 58789
> >>
> >> http://logs.test-lab.xenproject.org/osstest/logs/58821/test-amd64-amd64-libv
> >> irt/info.html
> >>
> >> While investigating why libvirt hasn't been succeeding very well on
> >> merlot* I came across some things in the serial log which initially
> >> struck me as odd, but which I suspect are nothing (or at least not
> >> terribly relevant), if someone could confirm that would be great.
> >>
> >> Firstly is:
> >>
> >> Jun 22 12:41:09.633294 (XEN) microcode: CPU2 updated from revision 0x6000822 to 0x6000832
> >> Jun 22 12:41:09.665099 (XEN) microcode: CPU4 updated from revision 0x6000822 to 0x6000832
> >> Jun 22 12:41:09.729089 (XEN) microcode: CPU6 updated from revision 0x6000822 to 0x6000832
> >> [...]
> >>
> >> i.e. only even numbered cpus are updated. (0 was done earlier in boot).
> >> I suspect that the answer here is "hyperthreading", and the cpuinfo
> >> shows all cpus have in fact been updated.
> > Yes (albeit hyperthreading is an Intel term, but iirc the same applies
> > to the two cores per compute unit).
>
> Indeed. The "microcode: patch is already at required level or
> greater.\n" message is helpfully unconditionally compiled out.
>
> >
> >> The second thing is:
> >> Jun 22 12:41:10.601103 (XEN) Brought up 32 CPUs
> >> Jun 22 12:41:10.625270 (XEN) Testing NMI watchdog on all CPUs: 0 1 2 3 4 5 6
> >> 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 stuck
> >>
> >> i.e. at least one CPU has issues with NMI watchdog (looking at other
> >> runs it seems to vary between 29-31). Is this just that the NMI watchdog
> >> doesn't deal well with so many pCPUs? Or is it a real issue?
> > Very few CPUs properly responding is certainly quite odd; one
> > would expect all or none of them to work. Perhaps our AMD
> > maintainers (now Cc-ed) could take a look...
>
> There are several things wrong with the NMI testing in Xen atm,
> following some recent investigation in XenServer. Time isn't accounted
> properly for cores under bios/hardware power control, and Xen doesn't
> wait for the requisite time even if the core were running at its
> expected frequency.
>
> I should see about making those patches appear, but for now, ignore this
> line. It is more than likely wrong.
Thanks. I think that means all three issues are not something wrong with
the host, so I shall ignore them
(which leaves me none the wiser about the actual failure, oh well...)
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-06-22 16:05 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-22 14:09 [xen-unstable test] 58821: tolerable FAIL osstest service user
2015-06-22 15:17 ` Ian Campbell
2015-06-22 15:36 ` Jan Beulich
2015-06-22 16:00 ` Andrew Cooper
2015-06-22 16:05 ` Ian Campbell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.