* [xen-unstable test] 106580: regressions - trouble: blocked/broken/fail/pass
@ 2017-03-10 7:20 osstest service owner
2017-03-10 8:37 ` Jan Beulich
0 siblings, 1 reply; 3+ messages in thread
From: osstest service owner @ 2017-03-10 7:20 UTC (permalink / raw)
To: xen-devel, osstest-admin
flight 106580 xen-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/106580/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-armhf-armhf-xl-arndale 3 host-install(3) broken REGR. vs. 106534
test-amd64-amd64-migrupgrade 10 xen-boot/dst_host fail REGR. vs. 106534
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm 15 guest-localmigrate/x10 fail REGR. vs. 106534
test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR. vs. 106534
Regressions which are regarded as allowable (not blocking):
test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop fail like 106534
test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop fail like 106534
test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop fail like 106534
test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop fail like 106534
test-armhf-armhf-libvirt 13 saverestore-support-check fail like 106534
test-armhf-armhf-libvirt-xsm 13 saverestore-support-check fail like 106534
test-armhf-armhf-libvirt-raw 12 saverestore-support-check fail like 106534
test-amd64-amd64-xl-rtds 9 debian-install fail like 106534
Tests which did not succeed, but are not blocking:
test-arm64-arm64-libvirt-xsm 1 build-check(1) blocked n/a
test-arm64-arm64-xl 1 build-check(1) blocked n/a
build-arm64-libvirt 1 build-check(1) blocked n/a
test-arm64-arm64-libvirt-qcow2 1 build-check(1) blocked n/a
test-arm64-arm64-libvirt 1 build-check(1) blocked n/a
test-arm64-arm64-xl-credit2 1 build-check(1) blocked n/a
test-arm64-arm64-xl-rtds 1 build-check(1) blocked n/a
test-arm64-arm64-xl-multivcpu 1 build-check(1) blocked n/a
test-arm64-arm64-xl-xsm 1 build-check(1) blocked n/a
test-amd64-amd64-xl-pvh-amd 11 guest-start fail never pass
test-amd64-amd64-xl-pvh-intel 11 guest-start fail never pass
test-amd64-i386-libvirt 12 migrate-support-check fail never pass
test-amd64-i386-libvirt-xsm 12 migrate-support-check fail never pass
test-amd64-amd64-libvirt 12 migrate-support-check fail never pass
build-arm64 5 xen-build fail never pass
test-amd64-amd64-libvirt-xsm 12 migrate-support-check fail never pass
build-arm64-xsm 5 xen-build fail never pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
build-arm64-pvops 5 kernel-build fail never pass
test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2 fail never pass
test-amd64-amd64-libvirt-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 12 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 13 saverestore-support-check fail never pass
test-armhf-armhf-xl 12 migrate-support-check fail never pass
test-armhf-armhf-xl 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-multivcpu 12 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-credit2 12 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-xl-xsm 13 saverestore-support-check fail never pass
test-armhf-armhf-libvirt 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-xl-rtds 12 migrate-support-check fail never pass
test-armhf-armhf-xl-rtds 13 saverestore-support-check fail never pass
test-armhf-armhf-libvirt-raw 11 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 12 saverestore-support-check fail never pass
version targeted for testing:
xen 9dc1e0cd81ee469d638d1962a92d9b4bd2972bfa
baseline version:
xen 4036e7c592905c2292cdeba8269e969959427237
Last test of basis 106534 2017-03-07 19:14:51 Z 2 days
Failing since 106547 2017-03-08 08:59:30 Z 1 days 4 attempts
Testing same since 106580 2017-03-09 23:15:02 Z 0 days 1 attempts
------------------------------------------------------------
People who touched revisions under test:
Andrew Cooper <andrew.cooper3@citrix.com>
Christoph Egger <chegger@amazon.de>
George Dunlap <george.dunlap@citrix.com>
Haozhong Zhang <haozhong.zhang@intel.com>
Jan Beulich <jbeulich@suse.com>
Julien Grall <julien.grall@arm.com>
Paul Durrant <paul.durrant@citrix.com>
jobs:
build-amd64-xsm pass
build-arm64-xsm fail
build-armhf-xsm pass
build-i386-xsm pass
build-amd64-xtf pass
build-amd64 pass
build-arm64 fail
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-arm64-libvirt blocked
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-oldkern pass
build-i386-oldkern pass
build-amd64-prev pass
build-i386-prev pass
build-amd64-pvops pass
build-arm64-pvops fail
build-armhf-pvops pass
build-i386-pvops pass
build-amd64-rumprun pass
build-i386-rumprun pass
test-xtf-amd64-amd64-1 pass
test-xtf-amd64-amd64-2 pass
test-xtf-amd64-amd64-3 pass
test-xtf-amd64-amd64-4 pass
test-xtf-amd64-amd64-5 pass
test-amd64-amd64-xl pass
test-arm64-arm64-xl blocked
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm fail
test-amd64-amd64-libvirt-xsm pass
test-arm64-arm64-libvirt-xsm blocked
test-armhf-armhf-libvirt-xsm pass
test-amd64-i386-libvirt-xsm pass
test-amd64-amd64-xl-xsm pass
test-arm64-arm64-xl-xsm blocked
test-armhf-armhf-xl-xsm pass
test-amd64-i386-xl-xsm pass
test-amd64-amd64-qemuu-nested-amd fail
test-amd64-amd64-xl-pvh-amd fail
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-rumprun-amd64 pass
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-armhf-armhf-xl-arndale broken
test-amd64-amd64-xl-credit2 pass
test-arm64-arm64-xl-credit2 blocked
test-armhf-armhf-xl-credit2 fail
test-armhf-armhf-xl-cubietruck pass
test-amd64-i386-freebsd10-i386 pass
test-amd64-i386-rumprun-i386 pass
test-amd64-amd64-qemuu-nested-intel pass
test-amd64-amd64-xl-pvh-intel fail
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-arm64-arm64-libvirt blocked
test-armhf-armhf-libvirt pass
test-amd64-i386-libvirt pass
test-amd64-amd64-migrupgrade fail
test-amd64-i386-migrupgrade pass
test-amd64-amd64-xl-multivcpu pass
test-arm64-arm64-xl-multivcpu blocked
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-libvirt-pair pass
test-amd64-i386-libvirt-pair pass
test-amd64-amd64-amd64-pvgrub pass
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-pygrub pass
test-arm64-arm64-libvirt-qcow2 blocked
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-libvirt-raw pass
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-rtds fail
test-arm64-arm64-xl-rtds blocked
test-armhf-armhf-xl-rtds pass
test-amd64-i386-xl-qemut-winxpsp3-vcpus1 pass
test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-xl-vhd pass
test-amd64-amd64-xl-qemut-winxpsp3 pass
test-amd64-i386-xl-qemut-winxpsp3 pass
test-amd64-amd64-xl-qemuu-winxpsp3 pass
test-amd64-i386-xl-qemuu-winxpsp3 pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
broken-step test-armhf-armhf-xl-arndale host-install(3)
Not pushing.
(No revision log; it would be 388 lines long.)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [xen-unstable test] 106580: regressions - trouble: blocked/broken/fail/pass
2017-03-10 7:20 [xen-unstable test] 106580: regressions - trouble: blocked/broken/fail/pass osstest service owner
@ 2017-03-10 8:37 ` Jan Beulich
2017-03-10 11:12 ` Andrew Cooper
0 siblings, 1 reply; 3+ messages in thread
From: Jan Beulich @ 2017-03-10 8:37 UTC (permalink / raw)
To: xen-devel; +Cc: Andrew Cooper, osstest-admin
>>> On 10.03.17 at 08:20, <osstest-admin@xenproject.org> wrote:
> flight 106580 xen-unstable real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/106580/
>
> Regressions :-(
>
> Tests which did not succeed and are blocking,
> including tests which could not be run:
> test-armhf-armhf-xl-arndale 3 host-install(3) broken REGR. vs. 106534
> test-amd64-amd64-migrupgrade 10 xen-boot/dst_host fail REGR. vs. 106534
The NMI watchdog has hit the EOI timer waiting to be able to send
an IPI on CPU1:
Mar 10 00:09:32.745677 (XEN) Xen call trace:
Mar 10 00:09:32.745727 (XEN) [<ffff82d080134083>] _spin_lock+0x2c/0x4f
Mar 10 00:09:32.745779 (XEN) [<ffff82d080133e34>] on_selected_cpus+0x2c/0xc6
Mar 10 00:09:32.753699 (XEN) [<ffff82d080177101>] irq.c#irq_guest_eoi_timer_fn+0x142/0x165
Mar 10 00:09:32.761711 (XEN) [<ffff82d080136ddc>] timer.c#execute_timer+0x47/0x62
Mar 10 00:09:32.769683 (XEN) [<ffff82d080136ed2>] timer.c#timer_softirq_action+0xdb/0x22c
Mar 10 00:09:32.769744 (XEN) [<ffff82d0801337e1>] softirq.c#__do_softirq+0x7f/0x8a
Mar 10 00:09:32.777697 (XEN) [<ffff82d080133836>] do_softirq+0x13/0x15
Mar 10 00:09:32.785792 (XEN) [<ffff82d080255081>] entry.o#process_softirqs+0x21/0x30
That lock is being held by CPU2:
Mar 10 00:15:25.133639 (XEN) Xen call trace:
Mar 10 00:15:25.133655 (XEN) [<ffff82d080102389>] __bitmap_empty+0x54/0x96
Mar 10 00:15:25.141636 (XEN) [<ffff82d080133eb5>] on_selected_cpus+0xad/0xc6
Mar 10 00:15:25.149635 (XEN) [<ffff82d0801ca640>] powernow.c#powernow_cpufreq_cpu_init+0x20d/0x372
Mar 10 00:15:25.157633 (XEN) [<ffff82d08014c476>] cpufreq_add_cpu+0x1d6/0x5d3
Mar 10 00:15:25.157654 (XEN) [<ffff82d0801ca173>] cpufreq_cpu_init+0x17/0x1a
Mar 10 00:15:25.165658 (XEN) [<ffff82d08014cd8d>] set_px_pminfo+0x2b6/0x2f7
Mar 10 00:15:25.165679 (XEN) [<ffff82d0801956dd>] do_platform_op+0xe69/0x1959
Mar 10 00:15:25.173667 (XEN) [<ffff82d080251485>] pv_hypercall+0x1ef/0x42d
Mar 10 00:15:25.181678 (XEN) [<ffff82d080254ff6>] entry.o#test_all_events+0/0x30
Register state tells us that it's CPU5 not responding. The only piece
of information we have about CPU5 is
Mar 10 00:09:32.809709 (XEN) CPU5 @ e008:ffff82d080134083 (0000000000000000)
which is the also in _spin_lock(), but which I'm afraid is too little to
diagnose the issue. I'm therefore wondering whether we wouldn't
better default "async-show-all" to true in debug builds.
What I'm also puzzled by is that the system is still partly alive after
the panic: There's Dom0 output, and it is also reacting to debug
key input. I would have expected a panic to bring down the system
right away...
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [xen-unstable test] 106580: regressions - trouble: blocked/broken/fail/pass
2017-03-10 8:37 ` Jan Beulich
@ 2017-03-10 11:12 ` Andrew Cooper
0 siblings, 0 replies; 3+ messages in thread
From: Andrew Cooper @ 2017-03-10 11:12 UTC (permalink / raw)
To: Jan Beulich, xen-devel; +Cc: osstest-admin
On 10/03/17 08:37, Jan Beulich wrote:
>>>> On 10.03.17 at 08:20, <osstest-admin@xenproject.org> wrote:
>> flight 106580 xen-unstable real [real]
>> http://logs.test-lab.xenproject.org/osstest/logs/106580/
>>
>> Regressions :-(
>>
>> Tests which did not succeed and are blocking,
>> including tests which could not be run:
>> test-armhf-armhf-xl-arndale 3 host-install(3) broken REGR. vs. 106534
>> test-amd64-amd64-migrupgrade 10 xen-boot/dst_host fail REGR. vs. 106534
> The NMI watchdog has hit the EOI timer waiting to be able to send
> an IPI on CPU1:
>
> Mar 10 00:09:32.745677 (XEN) Xen call trace:
> Mar 10 00:09:32.745727 (XEN) [<ffff82d080134083>] _spin_lock+0x2c/0x4f
> Mar 10 00:09:32.745779 (XEN) [<ffff82d080133e34>] on_selected_cpus+0x2c/0xc6
> Mar 10 00:09:32.753699 (XEN) [<ffff82d080177101>] irq.c#irq_guest_eoi_timer_fn+0x142/0x165
> Mar 10 00:09:32.761711 (XEN) [<ffff82d080136ddc>] timer.c#execute_timer+0x47/0x62
> Mar 10 00:09:32.769683 (XEN) [<ffff82d080136ed2>] timer.c#timer_softirq_action+0xdb/0x22c
> Mar 10 00:09:32.769744 (XEN) [<ffff82d0801337e1>] softirq.c#__do_softirq+0x7f/0x8a
> Mar 10 00:09:32.777697 (XEN) [<ffff82d080133836>] do_softirq+0x13/0x15
> Mar 10 00:09:32.785792 (XEN) [<ffff82d080255081>] entry.o#process_softirqs+0x21/0x30
>
> That lock is being held by CPU2:
>
> Mar 10 00:15:25.133639 (XEN) Xen call trace:
> Mar 10 00:15:25.133655 (XEN) [<ffff82d080102389>] __bitmap_empty+0x54/0x96
> Mar 10 00:15:25.141636 (XEN) [<ffff82d080133eb5>] on_selected_cpus+0xad/0xc6
> Mar 10 00:15:25.149635 (XEN) [<ffff82d0801ca640>] powernow.c#powernow_cpufreq_cpu_init+0x20d/0x372
> Mar 10 00:15:25.157633 (XEN) [<ffff82d08014c476>] cpufreq_add_cpu+0x1d6/0x5d3
> Mar 10 00:15:25.157654 (XEN) [<ffff82d0801ca173>] cpufreq_cpu_init+0x17/0x1a
> Mar 10 00:15:25.165658 (XEN) [<ffff82d08014cd8d>] set_px_pminfo+0x2b6/0x2f7
> Mar 10 00:15:25.165679 (XEN) [<ffff82d0801956dd>] do_platform_op+0xe69/0x1959
> Mar 10 00:15:25.173667 (XEN) [<ffff82d080251485>] pv_hypercall+0x1ef/0x42d
> Mar 10 00:15:25.181678 (XEN) [<ffff82d080254ff6>] entry.o#test_all_events+0/0x30
>
> Register state tells us that it's CPU5 not responding. The only piece
> of information we have about CPU5 is
>
> Mar 10 00:09:32.809709 (XEN) CPU5 @ e008:ffff82d080134083 (0000000000000000)
>
> which is the also in _spin_lock(), but which I'm afraid is too little to
> diagnose the issue. I'm therefore wondering whether we wouldn't
> better default "async-show-all" to true in debug builds.
>
> What I'm also puzzled by is that the system is still partly alive after
> the panic: There's Dom0 output, and it is also reacting to debug
> key input. I would have expected a panic to bring down the system
> right away...
Not very surprising. We crashed because the IPI lock was unavailable,
then disable the watchdog in machine_halt() and try to IPI again. CPU1
is almost certainly waiting trying to broadcast __machine_halt().
This is the second odd corner case we have seen around machine_halt().
The last one was because of being unsafe to use if you panic() from the
middle of context_switch(), as interrupts are re-enabled, and a guest
irq hits an assertion. The solution in both cases to make it more
reliable is to an NMI broadcast and leave interrupts disabled.
IMO, noreboot isn't a clever thing to be using at all. OSSTest should
be installing a crash kernel and collecting crash logs, which will be
far more useful to aid diagnosis.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2017-03-10 11:12 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-10 7:20 [xen-unstable test] 106580: regressions - trouble: blocked/broken/fail/pass osstest service owner
2017-03-10 8:37 ` Jan Beulich
2017-03-10 11:12 ` Andrew Cooper
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.