* [Xen-devel] [xen-unstable test] 140960: regressions - FAIL
@ 2019-09-03 20:00 osstest service owner
2019-09-04 8:36 ` Jan Beulich
0 siblings, 1 reply; 3+ messages in thread
From: osstest service owner @ 2019-09-03 20:00 UTC (permalink / raw)
To: xen-devel, osstest-admin
flight 140960 xen-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/140960/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-amd64-amd64-xl-pvshim 18 guest-localmigrate/x10 fail REGR. vs. 139876
build-amd64-xsm 6 xen-build fail REGR. vs. 139876
Tests which did not succeed, but are not blocking:
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm 1 build-check(1) blocked n/a
test-amd64-i386-xl-qemuu-debianhvm-i386-xsm 1 build-check(1) blocked n/a
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 1 build-check(1) blocked n/a
test-amd64-amd64-libvirt-xsm 1 build-check(1) blocked n/a
test-amd64-i386-xl-xsm 1 build-check(1) blocked n/a
test-amd64-i386-libvirt-xsm 1 build-check(1) blocked n/a
test-amd64-i386-xl-qemut-debianhvm-i386-xsm 1 build-check(1) blocked n/a
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm 1 build-check(1) blocked n/a
test-amd64-amd64-xl-qemuu-debianhvm-i386-xsm 1 build-check(1) blocked n/a
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 1 build-check(1) blocked n/a
test-amd64-amd64-xl-qemut-debianhvm-i386-xsm 1 build-check(1) blocked n/a
test-amd64-amd64-xl-xsm 1 build-check(1) blocked n/a
test-amd64-amd64-xl-qemut-win7-amd64 17 guest-stop fail like 139876
test-amd64-i386-xl-qemuu-win7-amd64 17 guest-stop fail like 139876
test-amd64-amd64-xl-qemuu-win7-amd64 17 guest-stop fail like 139876
test-amd64-amd64-xl-qemuu-ws16-amd64 17 guest-stop fail like 139876
test-armhf-armhf-libvirt 14 saverestore-support-check fail like 139876
test-amd64-i386-xl-qemut-win7-amd64 17 guest-stop fail like 139876
test-armhf-armhf-libvirt-raw 13 saverestore-support-check fail like 139876
test-amd64-amd64-xl-qemut-ws16-amd64 17 guest-stop fail like 139876
test-amd64-i386-xl-qemuu-ws16-amd64 17 guest-stop fail like 139876
test-amd64-amd64-libvirt 13 migrate-support-check fail never pass
test-amd64-i386-libvirt 13 migrate-support-check fail never pass
test-arm64-arm64-xl-seattle 13 migrate-support-check fail never pass
test-arm64-arm64-xl-seattle 14 saverestore-support-check fail never pass
test-amd64-amd64-qemuu-nested-amd 17 debian-hvm-install/l1/l2 fail never pass
test-arm64-arm64-xl-xsm 13 migrate-support-check fail never pass
test-arm64-arm64-xl-xsm 14 saverestore-support-check fail never pass
test-arm64-arm64-xl 13 migrate-support-check fail never pass
test-arm64-arm64-xl 14 saverestore-support-check fail never pass
test-arm64-arm64-xl-credit1 13 migrate-support-check fail never pass
test-arm64-arm64-xl-credit1 14 saverestore-support-check fail never pass
test-arm64-arm64-libvirt-xsm 13 migrate-support-check fail never pass
test-arm64-arm64-libvirt-xsm 14 saverestore-support-check fail never pass
test-arm64-arm64-xl-thunderx 13 migrate-support-check fail never pass
test-arm64-arm64-xl-thunderx 14 saverestore-support-check fail never pass
test-amd64-amd64-libvirt-vhd 12 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 13 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-rtds 13 migrate-support-check fail never pass
test-armhf-armhf-xl-rtds 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-cubietruck 13 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-multivcpu 13 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-credit2 13 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-credit1 13 migrate-support-check fail never pass
test-armhf-armhf-xl-credit1 14 saverestore-support-check fail never pass
test-armhf-armhf-xl 13 migrate-support-check fail never pass
test-armhf-armhf-xl 14 saverestore-support-check fail never pass
test-armhf-armhf-libvirt 13 migrate-support-check fail never pass
test-amd64-i386-xl-pvshim 12 guest-start fail never pass
test-arm64-arm64-xl-credit2 13 migrate-support-check fail never pass
test-arm64-arm64-xl-credit2 14 saverestore-support-check fail never pass
test-armhf-armhf-libvirt-raw 12 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 12 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 13 saverestore-support-check fail never pass
test-amd64-i386-xl-qemut-ws16-amd64 17 guest-stop fail never pass
test-amd64-amd64-xl-qemuu-win10-i386 10 windows-install fail never pass
test-amd64-i386-xl-qemuu-win10-i386 10 windows-install fail never pass
test-amd64-amd64-xl-qemut-win10-i386 10 windows-install fail never pass
test-amd64-i386-xl-qemut-win10-i386 10 windows-install fail never pass
version targeted for testing:
xen 8cc4476eb8176663b4a495c983daf02e885d4cf3
baseline version:
xen 6c9639a72f0ca3a9430ef75f375877182281fdef
Last test of basis 139876 2019-08-09 18:46:56 Z 25 days
Failing since 139915 2019-08-10 20:44:23 Z 23 days 28 attempts
Testing same since 140960 2019-09-03 01:07:05 Z 0 days 1 attempts
------------------------------------------------------------
People who touched revisions under test:
Alexandru Isaila <aisaila@bitdefender.com>
Alexandru Stefan ISAILA <aisaila@bitdefender.com>
Andrew Cooper <andrew.cooper3@citrix.com>
Brian Woods <brian.woods@amd.com>
Chao Gao <chao.gao@intel.com>
Christian Lindig <christian.lindig@citrix.com>
Daniel De Graaf <dgdegra@tycho.nsa.gov>
David Woodhouse <dwmw@amazon.co.uk>
George Dunlap <george.dunlap@citrix.com>
George Dunlap <George.Dunlap@eu.citrix.com>
Igor Druzhinin <igor.druzhinin@citrix.com>
Jan Beulich <jbeulich@suse.com>
Julien Grall <julien.grall@arm.com>
Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Michael Young <m.a.young@durham.ac.uk>
Michał Kowalczyk <mkow@invisiblethingslab.com>
Paul Durrant <paul.durrant@citrix.com>
Pawel Wieczorkiewicz <wipawel@amazon.de>
Roger Pau Monne <roger.pau@citrix.com>
Roger Pau Monné <roger.pau@citrix.com>
Roman Shaposhnik <roman@zededa.com>
Stefano Stabellini <sstabellini@kernel.org>
Stefano Stabellini <stefanos@xilinx.com>
Viktor Mitin <viktor.mitin.19@gmail.com>
Viktor Mitin <viktor_mitin@epam.com>
Wei Liu <wei.liu2@citrix.com>
Wei Liu <wl@xen.org>
jobs:
build-amd64-xsm fail
build-arm64-xsm pass
build-i386-xsm pass
build-amd64-xtf pass
build-amd64 pass
build-arm64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-arm64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-prev pass
build-i386-prev pass
build-amd64-pvops pass
build-arm64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
test-xtf-amd64-amd64-1 pass
test-xtf-amd64-amd64-2 pass
test-xtf-amd64-amd64-3 pass
test-xtf-amd64-amd64-4 pass
test-xtf-amd64-amd64-5 pass
test-amd64-amd64-xl pass
test-arm64-arm64-xl pass
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm blocked
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm blocked
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm blocked
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm blocked
test-amd64-amd64-xl-qemut-debianhvm-i386-xsm blocked
test-amd64-i386-xl-qemut-debianhvm-i386-xsm blocked
test-amd64-amd64-xl-qemuu-debianhvm-i386-xsm blocked
test-amd64-i386-xl-qemuu-debianhvm-i386-xsm blocked
test-amd64-amd64-libvirt-xsm blocked
test-arm64-arm64-libvirt-xsm pass
test-amd64-i386-libvirt-xsm blocked
test-amd64-amd64-xl-xsm blocked
test-arm64-arm64-xl-xsm pass
test-amd64-i386-xl-xsm blocked
test-amd64-amd64-qemuu-nested-amd fail
test-amd64-amd64-xl-pvhv2-amd pass
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-amd64-amd64-xl-qemut-ws16-amd64 fail
test-amd64-i386-xl-qemut-ws16-amd64 fail
test-amd64-amd64-xl-qemuu-ws16-amd64 fail
test-amd64-i386-xl-qemuu-ws16-amd64 fail
test-armhf-armhf-xl-arndale pass
test-amd64-amd64-xl-credit1 pass
test-arm64-arm64-xl-credit1 pass
test-armhf-armhf-xl-credit1 pass
test-amd64-amd64-xl-credit2 pass
test-arm64-arm64-xl-credit2 pass
test-armhf-armhf-xl-credit2 pass
test-armhf-armhf-xl-cubietruck pass
test-amd64-amd64-xl-qemuu-dmrestrict-amd64-dmrestrict pass
test-amd64-i386-xl-qemuu-dmrestrict-amd64-dmrestrict pass
test-amd64-amd64-examine pass
test-arm64-arm64-examine pass
test-armhf-armhf-examine pass
test-amd64-i386-examine pass
test-amd64-i386-freebsd10-i386 pass
test-amd64-amd64-xl-qemut-win10-i386 fail
test-amd64-i386-xl-qemut-win10-i386 fail
test-amd64-amd64-xl-qemuu-win10-i386 fail
test-amd64-i386-xl-qemuu-win10-i386 fail
test-amd64-amd64-qemuu-nested-intel pass
test-amd64-amd64-xl-pvhv2-intel pass
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-armhf-armhf-libvirt pass
test-amd64-i386-libvirt pass
test-amd64-amd64-livepatch pass
test-amd64-i386-livepatch pass
test-amd64-amd64-migrupgrade pass
test-amd64-i386-migrupgrade pass
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-libvirt-pair pass
test-amd64-i386-libvirt-pair pass
test-amd64-amd64-amd64-pvgrub pass
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-xl-pvshim fail
test-amd64-i386-xl-pvshim fail
test-amd64-amd64-pygrub pass
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-libvirt-raw pass
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-rtds pass
test-armhf-armhf-xl-rtds pass
test-arm64-arm64-xl-seattle pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-shadow pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-shadow pass
test-amd64-amd64-xl-shadow pass
test-amd64-i386-xl-shadow pass
test-arm64-arm64-xl-thunderx pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-xl-vhd pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
Not pushing.
(No revision log; it would be 1471 lines long.)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Xen-devel] [xen-unstable test] 140960: regressions - FAIL
2019-09-03 20:00 [Xen-devel] [xen-unstable test] 140960: regressions - FAIL osstest service owner
@ 2019-09-04 8:36 ` Jan Beulich
2019-09-04 8:50 ` Andrew Cooper
0 siblings, 1 reply; 3+ messages in thread
From: Jan Beulich @ 2019-09-04 8:36 UTC (permalink / raw)
To: osstest service owner, xen-devel
On 03.09.2019 22:00, osstest service owner wrote:
> flight 140960 xen-unstable real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/140960/
>
> Regressions :-(
>
> Tests which did not succeed and are blocking,
> including tests which could not be run:
> test-amd64-amd64-xl-pvshim 18 guest-localmigrate/x10 fail REGR. vs. 139876
This looks to be recurring, so I've taken another look. The three
migrations leave this abbreviated pattern in the log:
Sep 3 14:20:42.446667 (XEN) HVM d1v0 save: CPU_MSR
...
Sep 3 14:20:57.850670 (XEN) HVM2 restore: CPU 0
...
Sep 3 14:21:37.062840 (XEN) HVM d2v0 save: CPU_MSR
Sep 3 14:21:37.062888 (XEN) HVM3 restore: CPU 0
...
Sep 3 14:21:56.438552 (XEN) HVM d3v0 save: CPU_MSR
...
Sep 3 14:22:11.506508 (XEN) HVM4 restore: CPU 0
Therefore I wonder whether the first one got lucky and finished
barely ahead of timing out, while the 2nd worked instantly and the
3rd then ended up exceeding the timeout. What is curious are the
intermediate log entries (between the last "save" and the first
corresponding "restore" log entries): Many ones of the form
(XEN) emul-priv-op.c:1113:d0v2 Domain attempted WRMSR c0011020 from 0x0000000000000000 to 0x0040000000000000
with a 15s gap between the first and many subsequent ones) and
finally one of the form
[ 451.267669] systemd-logind[2766]: New session 39 of user root.
And finally, at around the time of the failed migration
INIT: Id "T0" respawning too fast: disabled for 5 minutes
While it's not clear that this parallel activity is causing the
migration to progress too slowly, it looks to be a possibility at
least. Can anyone explain what these are?
> build-amd64-xsm 6 xen-build fail REGR. vs. 139876
I take it that this is supposed to be taken care of by a342900d48
("tools/shim: Apply more duct tape to the linkfarm logic").
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Xen-devel] [xen-unstable test] 140960: regressions - FAIL
2019-09-04 8:36 ` Jan Beulich
@ 2019-09-04 8:50 ` Andrew Cooper
0 siblings, 0 replies; 3+ messages in thread
From: Andrew Cooper @ 2019-09-04 8:50 UTC (permalink / raw)
To: Jan Beulich, osstest service owner, xen-devel
On 04/09/2019 09:36, Jan Beulich wrote:
> On 03.09.2019 22:00, osstest service owner wrote:
>> flight 140960 xen-unstable real [real]
>> http://logs.test-lab.xenproject.org/osstest/logs/140960/
>>
>> Regressions :-(
>>
>> Tests which did not succeed and are blocking,
>> including tests which could not be run:
>> test-amd64-amd64-xl-pvshim 18 guest-localmigrate/x10 fail REGR. vs. 139876
> This looks to be recurring, so I've taken another look.
I had a suspicion as well, but fixing the intermittent build problems
was the first priority.
A major change in shim in the range under test is switching from Credit1
to NULL as a scheduler, following Dario's fixing of what we thought was
the final outstanding bug. Perhaps it wasn't the final bug...
> The three
> migrations leave this abbreviated pattern in the log:
>
> Sep 3 14:20:42.446667 (XEN) HVM d1v0 save: CPU_MSR
> ...
> Sep 3 14:20:57.850670 (XEN) HVM2 restore: CPU 0
> ...
> Sep 3 14:21:37.062840 (XEN) HVM d2v0 save: CPU_MSR
> Sep 3 14:21:37.062888 (XEN) HVM3 restore: CPU 0
> ...
> Sep 3 14:21:56.438552 (XEN) HVM d3v0 save: CPU_MSR
> ...
> Sep 3 14:22:11.506508 (XEN) HVM4 restore: CPU 0
>
> Therefore I wonder whether the first one got lucky and finished
> barely ahead of timing out, while the 2nd worked instantly and the
> 3rd then ended up exceeding the timeout. What is curious are the
> intermediate log entries (between the last "save" and the first
> corresponding "restore" log entries): Many ones of the form
>
> (XEN) emul-priv-op.c:1113:d0v2 Domain attempted WRMSR c0011020 from 0x0000000000000000 to 0x0040000000000000
This is due to a lack of MSR_VIRT_SPEC_CTRL. It is sshd (or systemd on
its behalf - unclear which) using the SSBD prctl to protect itself, and
Xen, having no support, is causing Linux to fall back to native methods
and falling fowl of Xens write/discard policy on MSRs.
> with a 15s gap between the first and many subsequent ones) and
> finally one of the form
>
> [ 451.267669] systemd-logind[2766]: New session 39 of user root.
>
> And finally, at around the time of the failed migration
>
> INIT: Id "T0" respawning too fast: disabled for 5 minutes
Googling around suggests it is an inittab misconfiguration.
>
> While it's not clear that this parallel activity is causing the
> migration to progress too slowly, it looks to be a possibility at
> least. Can anyone explain what these are?
>
>> build-amd64-xsm 6 xen-build fail REGR. vs. 139876
> I take it that this is supposed to be taken care of by a342900d48
> ("tools/shim: Apply more duct tape to the linkfarm logic").
Yes - it should do.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-09-04 8:51 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-03 20:00 [Xen-devel] [xen-unstable test] 140960: regressions - FAIL osstest service owner
2019-09-04 8:36 ` Jan Beulich
2019-09-04 8:50 ` Andrew Cooper
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.