* [Xen-devel] [xen-unstable test] 142973: regressions - FAIL
@ 2019-10-21 8:23 osstest service owner
2019-10-21 8:34 ` Jürgen Groß
0 siblings, 1 reply; 5+ messages in thread
From: osstest service owner @ 2019-10-21 8:23 UTC (permalink / raw)
To: xen-devel, osstest-admin
flight 142973 xen-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/142973/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-amd64-amd64-xl-pvshim 18 guest-localmigrate/x10 fail REGR. vs. 142750
Tests which are failing intermittently (not blocking):
test-amd64-amd64-xl-pvshim 16 guest-localmigrate fail in 142907 pass in 142973
test-arm64-arm64-examine 11 examine-serial/bootloader fail pass in 142907
Tests which did not succeed, but are not blocking:
test-amd64-amd64-xl-rtds 18 guest-localmigrate/x10 fail like 142750
test-amd64-amd64-xl-qemut-win7-amd64 17 guest-stop fail like 142750
test-amd64-i386-xl-qemuu-win7-amd64 17 guest-stop fail like 142750
test-amd64-amd64-xl-qemuu-win7-amd64 17 guest-stop fail like 142750
test-armhf-armhf-libvirt 14 saverestore-support-check fail like 142750
test-amd64-i386-xl-qemut-win7-amd64 17 guest-stop fail like 142750
test-armhf-armhf-xl-rtds 16 guest-start/debian.repeat fail like 142750
test-armhf-armhf-libvirt-raw 13 saverestore-support-check fail like 142750
test-amd64-amd64-xl-qemuu-ws16-amd64 17 guest-stop fail like 142750
test-amd64-amd64-xl-qemut-ws16-amd64 17 guest-stop fail like 142750
test-amd64-i386-xl-qemuu-ws16-amd64 17 guest-stop fail like 142750
test-amd64-amd64-libvirt 13 migrate-support-check fail never pass
test-amd64-i386-xl-pvshim 12 guest-start fail never pass
test-amd64-i386-libvirt 13 migrate-support-check fail never pass
test-amd64-i386-libvirt-xsm 13 migrate-support-check fail never pass
test-amd64-amd64-libvirt-xsm 13 migrate-support-check fail never pass
test-arm64-arm64-xl-seattle 13 migrate-support-check fail never pass
test-arm64-arm64-xl-seattle 14 saverestore-support-check fail never pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 11 migrate-support-check fail never pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 11 migrate-support-check fail never pass
test-amd64-amd64-qemuu-nested-amd 17 debian-hvm-install/l1/l2 fail never pass
test-arm64-arm64-xl-credit1 13 migrate-support-check fail never pass
test-arm64-arm64-xl-credit1 14 saverestore-support-check fail never pass
test-arm64-arm64-libvirt-xsm 13 migrate-support-check fail never pass
test-arm64-arm64-libvirt-xsm 14 saverestore-support-check fail never pass
test-arm64-arm64-xl-thunderx 13 migrate-support-check fail never pass
test-arm64-arm64-xl-thunderx 14 saverestore-support-check fail never pass
test-arm64-arm64-xl 13 migrate-support-check fail never pass
test-arm64-arm64-xl 14 saverestore-support-check fail never pass
test-arm64-arm64-xl-credit2 13 migrate-support-check fail never pass
test-arm64-arm64-xl-credit2 14 saverestore-support-check fail never pass
test-amd64-amd64-libvirt-vhd 12 migrate-support-check fail never pass
test-arm64-arm64-xl-xsm 13 migrate-support-check fail never pass
test-arm64-arm64-xl-xsm 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-arndale 13 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 14 saverestore-support-check fail never pass
test-armhf-armhf-xl 13 migrate-support-check fail never pass
test-armhf-armhf-xl 14 saverestore-support-check fail never pass
test-armhf-armhf-libvirt 13 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 13 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-rtds 13 migrate-support-check fail never pass
test-armhf-armhf-xl-rtds 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-credit2 13 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-credit1 13 migrate-support-check fail never pass
test-armhf-armhf-xl-credit1 14 saverestore-support-check fail never pass
test-armhf-armhf-libvirt-raw 12 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 12 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 13 saverestore-support-check fail never pass
test-amd64-i386-xl-qemut-ws16-amd64 17 guest-stop fail never pass
test-amd64-amd64-xl-qemuu-win10-i386 10 windows-install fail never pass
test-amd64-i386-xl-qemuu-win10-i386 10 windows-install fail never pass
test-amd64-amd64-xl-qemut-win10-i386 10 windows-install fail never pass
test-amd64-i386-xl-qemut-win10-i386 10 windows-install fail never pass
test-armhf-armhf-xl-multivcpu 13 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 14 saverestore-support-check fail never pass
version targeted for testing:
xen 59d03d28b2bb2fb819db2c1fa2e8dc4f650d3894
baseline version:
xen 518c935fac4d30b3ec35d4b6add82b17b7d7aca3
Last test of basis 142750 2019-10-14 15:08:18 Z 6 days
Failing since 142777 2019-10-15 13:39:06 Z 5 days 5 attempts
Testing same since 142907 2019-10-19 08:51:01 Z 1 days 2 attempts
------------------------------------------------------------
People who touched revisions under test:
Andrew Cooper <andrew.cooper3@citrix.com>
Daniel De Graaf <dgdegra@tycho.nsa.gov>
Jan Beulich <jbeulich@suse.com>
Julien Grall <julien.grall@arm.com>
Olaf Hering <olaf@aepfle.de>
Tim Deegan <tim@xen.org>
jobs:
build-amd64-xsm pass
build-arm64-xsm pass
build-i386-xsm pass
build-amd64-xtf pass
build-amd64 pass
build-arm64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-arm64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-prev pass
build-i386-prev pass
build-amd64-pvops pass
build-arm64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
test-xtf-amd64-amd64-1 pass
test-xtf-amd64-amd64-2 pass
test-xtf-amd64-amd64-3 pass
test-xtf-amd64-amd64-4 pass
test-xtf-amd64-amd64-5 pass
test-amd64-amd64-xl pass
test-arm64-arm64-xl pass
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemut-debianhvm-i386-xsm pass
test-amd64-i386-xl-qemut-debianhvm-i386-xsm pass
test-amd64-amd64-xl-qemuu-debianhvm-i386-xsm pass
test-amd64-i386-xl-qemuu-debianhvm-i386-xsm pass
test-amd64-amd64-libvirt-xsm pass
test-arm64-arm64-libvirt-xsm pass
test-amd64-i386-libvirt-xsm pass
test-amd64-amd64-xl-xsm pass
test-arm64-arm64-xl-xsm pass
test-amd64-i386-xl-xsm pass
test-amd64-amd64-qemuu-nested-amd fail
test-amd64-amd64-xl-pvhv2-amd pass
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-amd64-amd64-xl-qemut-ws16-amd64 fail
test-amd64-i386-xl-qemut-ws16-amd64 fail
test-amd64-amd64-xl-qemuu-ws16-amd64 fail
test-amd64-i386-xl-qemuu-ws16-amd64 fail
test-armhf-armhf-xl-arndale pass
test-amd64-amd64-xl-credit1 pass
test-arm64-arm64-xl-credit1 pass
test-armhf-armhf-xl-credit1 pass
test-amd64-amd64-xl-credit2 pass
test-arm64-arm64-xl-credit2 pass
test-armhf-armhf-xl-credit2 pass
test-armhf-armhf-xl-cubietruck pass
test-amd64-amd64-xl-qemuu-dmrestrict-amd64-dmrestrict pass
test-amd64-i386-xl-qemuu-dmrestrict-amd64-dmrestrict pass
test-amd64-amd64-examine pass
test-arm64-arm64-examine pass
test-armhf-armhf-examine pass
test-amd64-i386-examine pass
test-amd64-i386-freebsd10-i386 pass
test-amd64-amd64-xl-qemut-win10-i386 fail
test-amd64-i386-xl-qemut-win10-i386 fail
test-amd64-amd64-xl-qemuu-win10-i386 fail
test-amd64-i386-xl-qemuu-win10-i386 fail
test-amd64-amd64-qemuu-nested-intel pass
test-amd64-amd64-xl-pvhv2-intel pass
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-armhf-armhf-libvirt pass
test-amd64-i386-libvirt pass
test-amd64-amd64-livepatch pass
test-amd64-i386-livepatch pass
test-amd64-amd64-migrupgrade pass
test-amd64-i386-migrupgrade pass
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-libvirt-pair pass
test-amd64-i386-libvirt-pair pass
test-amd64-amd64-amd64-pvgrub pass
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-xl-pvshim fail
test-amd64-i386-xl-pvshim fail
test-amd64-amd64-pygrub pass
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-libvirt-raw pass
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-rtds fail
test-armhf-armhf-xl-rtds fail
test-arm64-arm64-xl-seattle pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-shadow pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-shadow pass
test-amd64-amd64-xl-shadow pass
test-amd64-i386-xl-shadow pass
test-arm64-arm64-xl-thunderx pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-xl-vhd pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
Not pushing.
------------------------------------------------------------
commit 59d03d28b2bb2fb819db2c1fa2e8dc4f650d3894
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date: Fri Jul 19 08:57:50 2019 +0100
docs/sphinx: Introduction
Put together an introduction page for the Sphinx/RST docs, along with a
glossary which will accumulate over time.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Lars Kurth <lars.kurth@citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
commit 6da80b22d3713ee15cb2aca87e09b1cd3c8d2a6d
Author: Tim Deegan <tim@xen.org>
Date: Thu Oct 17 07:18:16 2019 +0100
MAINTAINERS: drop Tim Deegan from 'The Rest'
I have not been active in this role for a while now.
Signed-off-by: Tim Deegan <tim@xen.org>
commit 00fc9004be169a065c10a5fb699e353e430190c2
Author: Julien Grall <julien.grall@arm.com>
Date: Thu Jun 13 18:11:45 2019 +0100
xen/arm: mm: Clear boot pagetables before bringing-up each secondary CPU
At the moment, boot pagetables are only cleared once at boot. This means
when booting CPU2 (and onwards) then boot pagetables will not be
cleared.
To keep the interface exactly the same for all secondary CPU, the boot
pagetables are now cleared before bringing-up each secondary CPU.
Signed-off-by: Julien Grall <julien.grall@arm.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Release-acked-by: Juergen Gross <jgross@suse.com>
commit 4c555ec758e829d424a85df02821b2293ca03702
Author: Julien Grall <julien.grall@arm.com>
Date: Tue Aug 13 19:11:28 2019 +0100
xen/arm: domain_build: Print the correct domain in dtb_load()
dtb_load() can be called by other domain than dom0. To avoid confusion
in the log, print the correct domain.
Signed-off-by: Julien Grall <julien.grall@arm.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Release-acked-by: Juergen Gross <jgross@suse.com>
commit 55ab292c42db41b05cfdba012680bf1e0ea02f7a
Author: Olaf Hering <olaf@aepfle.de>
Date: Wed Oct 2 19:05:36 2019 +0200
stubdom/vtpm: include stdio.h for declaration of printf
The function read_vtpmblk uses printf(3), but stdio.h is not included
in this file. This results in a warning from gcc-7:
vtpmblk.c: In function 'read_vtpmblk':
vtpmblk.c:322:7: warning: implicit declaration of function 'printf' [-Wimplicit-function-declaration]
printf("Expected: ");
vtpmblk.c:322:7: warning: incompatible implicit declaration of built-in function 'printf'
vtpmblk.c:322:7: note: include '<stdio.h>' or provide a declaration of 'printf'
Signed-off-by: Olaf Hering <olaf@aepfle.de>
Acked-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
Reviewed-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
Release-acked-by: Juergen Gross <jgross@suse.com>
commit e37058267e33e5e4525f920e3126d15891ee8dbb
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date: Fri Jul 19 08:57:50 2019 +0100
docs/sphinx: Indent cleanup
Sphinx, its linters, and RST modes in common editors, expect 3 spaces of
indentation. Some bits already conform to this expectation. Update the
rest to match.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Lars Kurth <lars.kurth@citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
commit 951ab406511e8b752f7c9275af9d26e4c34b619d
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date: Tue Oct 8 20:23:26 2019 +0100
x86/microcode: Drop trailing whitespace in printk()
This has actually been present since c/s bd7c09c0 in 2008, and survived
through all of the recent microcode refactoring.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
(qemu changes not included)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Xen-devel] [xen-unstable test] 142973: regressions - FAIL
2019-10-21 8:23 [Xen-devel] [xen-unstable test] 142973: regressions - FAIL osstest service owner
@ 2019-10-21 8:34 ` Jürgen Groß
2019-10-21 11:06 ` Ian Jackson
0 siblings, 1 reply; 5+ messages in thread
From: Jürgen Groß @ 2019-10-21 8:34 UTC (permalink / raw)
To: xen-devel, Roger Pau Monné, Ian Jackson
On 21.10.19 10:23, osstest service owner wrote:
> flight 142973 xen-unstable real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/142973/
>
> Regressions :-(
>
> Tests which did not succeed and are blocking,
> including tests which could not be run:
> test-amd64-amd64-xl-pvshim 18 guest-localmigrate/x10 fail REGR. vs. 142750
Roger, I believe you have looked into that one?
I guess the conversation via IRC with Ian regarding the race between
blkback and OSStest was related to the issue?
If this is the case, could you, Ian, please add the workaround you were
thinking of to OSStest (unconditional by now, maybe make it condtitional
later)?
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Xen-devel] [xen-unstable test] 142973: regressions - FAIL
2019-10-21 8:34 ` Jürgen Groß
@ 2019-10-21 11:06 ` Ian Jackson
2019-10-21 11:18 ` Roger Pau Monné
2019-10-21 11:25 ` Jürgen Groß
0 siblings, 2 replies; 5+ messages in thread
From: Ian Jackson @ 2019-10-21 11:06 UTC (permalink / raw)
To: Jürgen Groß; +Cc: xen-devel, Roger Pau Monne
Jürgen Groß writes ("Re: [Xen-devel] [xen-unstable test] 142973: regressions - FAIL"):
> On 21.10.19 10:23, osstest service owner wrote:
> > flight 142973 xen-unstable real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/142973/
> >
> > Regressions :-(
> >
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> > test-amd64-amd64-xl-pvshim 18 guest-localmigrate/x10 fail REGR. vs. 142750
>
> Roger, I believe you have looked into that one?
>
> I guess the conversation via IRC with Ian regarding the race between
> blkback and OSStest was related to the issue?
I think this failure is something else.
What happens here is this:
2019-10-21 02:58:32 Z executing ssh ... -v root@172.16.145.205 date
[bounch of output from ssh]
status (timed out) at Osstest/TestSupport.pm line 550.
2019-10-21 02:58:42 Z exit status 4
172.16.145.205 is the guest here. Ie, `ssh date guest' took longer
than 10s.
We can see that the guest networking is working soon after the
migration because we got most of the way through the ssh protocol
exchange. On the previous repetition the next message from ssh was
debug1: SSH2_MSG_SERVICE_ACCEPT received
Looking at
http://logs.test-lab.xenproject.org/osstest/logs/142973/test-amd64-amd64-xl-pvshim/rimava1---var-log-xen-console-guest-debian.guest.osstest--incoming.log
which is, I think, the log of the "new" instance of guest, after
migration, there are messages about killing various services. Eg
[1918064738.820550] systemd[1]: systemd-udevd.service: Main process
exited, code=killed, status=6/ABRT
They don't seem to be normal. For example:
http://logs.test-lab.xenproject.org/osstest/logs/142865/test-amd64-amd64-xl-pvshim/rimava1---var-log-xen-console-guest-debian.guest.osstest--incoming.log
is the previous xen-unstable flight and it doesn't have them. I
looked in
http://logs.test-lab.xenproject.org/osstest/logs/142865/test-amd64-amd64-xl-pvshim/rimava1---var-log-xen-console-guest-debian.guest.osstest.log.gz
too and that has some alarming messages from the kernel like
[ 686.692660] rcu_sched kthread starved for 1918092123128 jiffies!
g18446744073709551359 c18446744073709551358 f0x0 RCU_GP_WAIT_FQS(3)
->state=0x0 ->cpu=0
and accompanying stack traces. But the test passed there. I think
that is probably something else ?
ABRT suggests guest memory corruption.
> If this is the case, could you, Ian, please add the workaround you were
> thinking of to OSStest (unconditional by now, maybe make it condtitional
> later)?
I can add the block race workaround but I don't think it will help
with migration anyway. The case where things go wrong is destroy.
Roger, am I right that a normal guest shutdown is race-free ? I think
we tear things down in a slower manner and will therefore end up
waiting for blkback ? Or is that not true ?
Maybe the right workaround is to disable the code in osstest which
tries to clean up a previous failed run. I think the kernel doesn't
mind multiple blkfronts (or indeed multiple other tasks) using the
same device at once.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Xen-devel] [xen-unstable test] 142973: regressions - FAIL
2019-10-21 11:06 ` Ian Jackson
@ 2019-10-21 11:18 ` Roger Pau Monné
2019-10-21 11:25 ` Jürgen Groß
1 sibling, 0 replies; 5+ messages in thread
From: Roger Pau Monné @ 2019-10-21 11:18 UTC (permalink / raw)
To: Ian Jackson; +Cc: Jürgen Groß, xen-devel
On Mon, Oct 21, 2019 at 12:06:32PM +0100, Ian Jackson wrote:
> Jürgen Groß writes ("Re: [Xen-devel] [xen-unstable test] 142973: regressions - FAIL"):
> > On 21.10.19 10:23, osstest service owner wrote:
> > > flight 142973 xen-unstable real [real]
> > > http://logs.test-lab.xenproject.org/osstest/logs/142973/
> > >
> > > Regressions :-(
> > >
> > > Tests which did not succeed and are blocking,
> > > including tests which could not be run:
> > > test-amd64-amd64-xl-pvshim 18 guest-localmigrate/x10 fail REGR. vs. 142750
> >
> > Roger, I believe you have looked into that one?
> >
> > I guess the conversation via IRC with Ian regarding the race between
> > blkback and OSStest was related to the issue?
>
> I think this failure is something else.
I agree.
> What happens here is this:
>
> 2019-10-21 02:58:32 Z executing ssh ... -v root@172.16.145.205 date
> [bounch of output from ssh]
> status (timed out) at Osstest/TestSupport.pm line 550.
> 2019-10-21 02:58:42 Z exit status 4
>
> 172.16.145.205 is the guest here. Ie, `ssh date guest' took longer
> than 10s.
>
> We can see that the guest networking is working soon after the
> migration because we got most of the way through the ssh protocol
> exchange. On the previous repetition the next message from ssh was
> debug1: SSH2_MSG_SERVICE_ACCEPT received
>
> Looking at
> http://logs.test-lab.xenproject.org/osstest/logs/142973/test-amd64-amd64-xl-pvshim/rimava1---var-log-xen-console-guest-debian.guest.osstest--incoming.log
> which is, I think, the log of the "new" instance of guest, after
> migration, there are messages about killing various services. Eg
> [1918064738.820550] systemd[1]: systemd-udevd.service: Main process
> exited, code=killed, status=6/ABRT
> They don't seem to be normal. For example:
> http://logs.test-lab.xenproject.org/osstest/logs/142865/test-amd64-amd64-xl-pvshim/rimava1---var-log-xen-console-guest-debian.guest.osstest--incoming.log
> is the previous xen-unstable flight and it doesn't have them. I
> looked in
> http://logs.test-lab.xenproject.org/osstest/logs/142865/test-amd64-amd64-xl-pvshim/rimava1---var-log-xen-console-guest-debian.guest.osstest.log.gz
> too and that has some alarming messages from the kernel like
> [ 686.692660] rcu_sched kthread starved for 1918092123128 jiffies!
> g18446744073709551359 c18446744073709551358 f0x0 RCU_GP_WAIT_FQS(3)
> ->state=0x0 ->cpu=0
> and accompanying stack traces. But the test passed there. I think
> that is probably something else ?
AFAICT there's corruption when migrating and also some kind of
lockup, not sure if those are related or not yet.
> ABRT suggests guest memory corruption.
>
> > If this is the case, could you, Ian, please add the workaround you were
> > thinking of to OSStest (unconditional by now, maybe make it condtitional
> > later)?
>
> I can add the block race workaround but I don't think it will help
> with migration anyway. The case where things go wrong is destroy.
>
> Roger, am I right that a normal guest shutdown is race-free ? I think
> we tear things down in a slower manner and will therefore end up
> waiting for blkback ? Or is that not true ?
It doesn't really matter whether shutdown or destroy is used, the
issue is that blkback switches to state 6 (Closed) before the disk is
closed, and hence there's no way for the toolstack to detect when the
disk has actually been released.
>
> Maybe the right workaround is to disable the code in osstest which
> tries to clean up a previous failed run. I think the kernel doesn't
> mind multiple blkfronts (or indeed multiple other tasks) using the
> same device at once.
Since the action when the disk is found to be in use is to try to
unmount it, maybe osstest should make sure the disk is actually
mounted first by parsing the output of mount? (or maybe there's a
better way to do it)
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Xen-devel] [xen-unstable test] 142973: regressions - FAIL
2019-10-21 11:06 ` Ian Jackson
2019-10-21 11:18 ` Roger Pau Monné
@ 2019-10-21 11:25 ` Jürgen Groß
1 sibling, 0 replies; 5+ messages in thread
From: Jürgen Groß @ 2019-10-21 11:25 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel, Roger Pau Monne
On 21.10.19 13:06, Ian Jackson wrote:
> Jürgen Groß writes ("Re: [Xen-devel] [xen-unstable test] 142973: regressions - FAIL"):
>> On 21.10.19 10:23, osstest service owner wrote:
>>> flight 142973 xen-unstable real [real]
>>> http://logs.test-lab.xenproject.org/osstest/logs/142973/
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking,
>>> including tests which could not be run:
>>> test-amd64-amd64-xl-pvshim 18 guest-localmigrate/x10 fail REGR. vs. 142750
>>
>> Roger, I believe you have looked into that one?
>>
>> I guess the conversation via IRC with Ian regarding the race between
>> blkback and OSStest was related to the issue?
>
> I think this failure is something else.
>
> What happens here is this:
>
> 2019-10-21 02:58:32 Z executing ssh ... -v root@172.16.145.205 date
> [bounch of output from ssh]
> status (timed out) at Osstest/TestSupport.pm line 550.
> 2019-10-21 02:58:42 Z exit status 4
>
> 172.16.145.205 is the guest here. Ie, `ssh date guest' took longer
> than 10s.
>
> We can see that the guest networking is working soon after the
> migration because we got most of the way through the ssh protocol
> exchange. On the previous repetition the next message from ssh was
> debug1: SSH2_MSG_SERVICE_ACCEPT received
>
> Looking at
> http://logs.test-lab.xenproject.org/osstest/logs/142973/test-amd64-amd64-xl-pvshim/rimava1---var-log-xen-console-guest-debian.guest.osstest--incoming.log
> which is, I think, the log of the "new" instance of guest, after
> migration, there are messages about killing various services. Eg
> [1918064738.820550] systemd[1]: systemd-udevd.service: Main process
> exited, code=killed, status=6/ABRT
> They don't seem to be normal. For example:
> http://logs.test-lab.xenproject.org/osstest/logs/142865/test-amd64-amd64-xl-pvshim/rimava1---var-log-xen-console-guest-debian.guest.osstest--incoming.log
> is the previous xen-unstable flight and it doesn't have them. I
> looked in
> http://logs.test-lab.xenproject.org/osstest/logs/142865/test-amd64-amd64-xl-pvshim/rimava1---var-log-xen-console-guest-debian.guest.osstest.log.gz
> too and that has some alarming messages from the kernel like
> [ 686.692660] rcu_sched kthread starved for 1918092123128 jiffies!
> g18446744073709551359 c18446744073709551358 f0x0 RCU_GP_WAIT_FQS(3)
> ->state=0x0 ->cpu=0
> and accompanying stack traces. But the test passed there. I think
> that is probably something else ?
This seems to be the issue Sergey is seeing, too.
>
> ABRT suggests guest memory corruption.
Sure? I'd think of an abort() call.
>
>> If this is the case, could you, Ian, please add the workaround you were
>> thinking of to OSStest (unconditional by now, maybe make it condtitional
>> later)?
>
> I can add the block race workaround but I don't think it will help
> with migration anyway. The case where things go wrong is destroy.
Okay, no hurry then.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-10-21 11:26 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-21 8:23 [Xen-devel] [xen-unstable test] 142973: regressions - FAIL osstest service owner
2019-10-21 8:34 ` Jürgen Groß
2019-10-21 11:06 ` Ian Jackson
2019-10-21 11:18 ` Roger Pau Monné
2019-10-21 11:25 ` Jürgen Groß
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.