* [linux-4.1 test] 79008: regressions - FAIL
@ 2016-01-26 13:11 osstest service owner
2016-01-27 11:18 ` Ian Campbell
0 siblings, 1 reply; 5+ messages in thread
From: osstest service owner @ 2016-01-26 13:11 UTC (permalink / raw)
To: xen-devel, osstest-admin
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 13345 bytes --]
flight 79008 linux-4.1 real [real]
http://logs.test-lab.xenproject.org/osstest/logs/79008/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR. vs. 66399
test-armhf-armhf-xl-xsm 15 guest-start/debian.repeat fail REGR. vs. 66399
test-armhf-armhf-xl-cubietruck 15 guest-start/debian.repeat fail in 78925 REGR. vs. 66399
Tests which are failing intermittently (not blocking):
test-amd64-amd64-xl-qemuu-debianhvm-amd64 3 host-install(3) broken in 78925 pass in 79008
test-amd64-i386-qemuu-rhel6hvm-amd 3 host-install(3) broken in 78925 pass in 79008
test-amd64-i386-xl-qemut-debianhvm-amd64 9 debian-hvm-install fail in 78925 pass in 79008
test-armhf-armhf-xl-credit2 11 guest-start fail in 78925 pass in 79008
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm 15 guest-localmigrate/x10 fail in 78925 pass in 79008
test-armhf-armhf-xl-cubietruck 11 guest-start fail pass in 78925
Regressions which are regarded as allowable (not blocking):
test-amd64-i386-rumpuserxen-i386 10 guest-start fail like 66399
test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop fail like 66399
test-amd64-amd64-xl-qemuu-win7-amd64 16 guest-stop fail like 66399
test-armhf-armhf-xl-rtds 15 guest-start/debian.repeat fail like 66399
Tests which did not succeed, but are not blocking:
test-armhf-armhf-xl-cubietruck 12 migrate-support-check fail in 78925 never pass
test-armhf-armhf-xl-cubietruck 13 saverestore-support-check fail in 78925 never pass
test-amd64-amd64-xl-pvh-amd 11 guest-start fail never pass
test-amd64-amd64-xl-pvh-intel 14 guest-saverestore fail never pass
test-armhf-armhf-libvirt-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt-xsm 14 guest-saverestore fail never pass
test-amd64-amd64-libvirt 12 migrate-support-check fail never pass
test-amd64-amd64-libvirt-xsm 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt 14 guest-saverestore fail never pass
test-armhf-armhf-libvirt 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt-qcow2 11 migrate-support-check fail never pass
test-armhf-armhf-libvirt-qcow2 13 guest-saverestore fail never pass
test-amd64-i386-libvirt-xsm 12 migrate-support-check fail never pass
test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2 fail never pass
test-armhf-armhf-xl-credit2 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-credit2 12 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 12 migrate-support-check fail never pass
test-armhf-armhf-xl-arndale 13 saverestore-support-check fail never pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
test-armhf-armhf-xl 12 migrate-support-check fail never pass
test-armhf-armhf-xl 13 saverestore-support-check fail never pass
test-amd64-i386-libvirt 12 migrate-support-check fail never pass
test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop fail never pass
test-amd64-amd64-libvirt-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-xsm 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-xsm 12 migrate-support-check fail never pass
test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop fail never pass
test-armhf-armhf-xl-rtds 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-rtds 12 migrate-support-check fail never pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 10 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-multivcpu 12 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 12 saverestore-support-check fail never pass
test-armhf-armhf-libvirt-raw 9 debian-di-install fail never pass
version targeted for testing:
linux 99c5a856dcee7658ec7e250aa477a9afaab8cfc6
baseline version:
linux 07cc49f66973f49a391c91bf4b158fa0f2562ca8
Last test of basis 66399 2015-12-15 18:20:39 Z 41 days
Testing same since 78925 2016-01-24 13:50:39 Z 1 days 2 attempts
------------------------------------------------------------
People who touched revisions under test:
Alan Stern <stern@rowland.harvard.edu>
Alexey Khoroshilov <khoroshilov@ispras.ru>
Arnd Bergmann <arnd@arndb.de>
Azael Avalos <coproscefalo@gmail.com>
Ben Hutchings <ben@decadent.org.uk>
Bjørn Mork <bjorn@mork.no>
Cong Wang <xiyou.wangcong@gmail.com>
Daniele Palmas <dnlplm@gmail.com>
Darren Hart <dvhart@linux.intel.com>
David Howells <dhowells@redhat.com>
David J. Wilder <dwilder@us.ibm.com>
David S. Miller <davem@davemloft.net>
Dmitry Katsubo <dmitry.katsubo@gmail.com>
Dmitry Vyukov <dvyukov@google.com>
Don Zickus <dzickus@redhat.com>
Eric Dumazet <edumazet@google.com>
Felipe Balbi <balbi@ti.com>
Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Guillaume Nault <g.nault@alphalink.fr>
Hannes Frederic Sowa <hannes@stressinduktion.org>
Hans Yang <hansy@nvidia.com>
Hayes Wang <hayeswang@realtek.com>
Herbert Xu <herbert@gondor.apana.org.au>
James Morris <james.l.morris@oracle.com>
Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Jarod Wilson <jarod@redhat.com>
Jiri Benc <jbenc@redhat.com>
Johan Hovold <johan@kernel.org>
Jonas Jonsson <jonas@ludd.ltu.se>
Konstantin Shkolnyy <konstantin.shkolnyy@gmail.com>
Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>
Michael Marley <michael@michaelmarley.com>
Michal Hocko <mhocko@suse.com>
Mimi Zohar <zohar@linux.vnet.ibm.com> (on TPM 1.2)
Nicolas Dichtel <nicolas.dichtel@6wind.com>
Pavel Machek <pavel@ucw.cz>
Peter Huewe <peterhuewe@gmx.de>
Peter Wu <peter@lekensteyn.nl>
Prarit Bhargava <prarit@redhat.com>
Rainer Weikusat <rweikusat@mobileactivedefense.com>
Robert Jarzmik <robert.jarzmik@free.fr>
Sergei Shtylyov <sergei.shtylyov@cogentembedded.com>
Stas Nichiporovich <stasn77@gmail.com>
Stefan Wahren <stefan.wahren@i2se.com>
Tadeusz Struk <tadeusz.struk@intel.com>
tadeusz.struk@intel.com <tadeusz.struk@intel.com>
Tom Herbert <tom@herbertland.com>
Vlad Yasevich <vyasevich@gmail.com>
Vladislav Yasevich <vyasevic@redhat.com>
Vladislav Yasevich <vyasevich@gmail.com>
WANG Cong <xiyou.wangcong@gmail.com>
Willem de Bruijn <willemb@google.com>
Yevgeny Pats <yevgeny@perception-point.io>
Yuchung Cheng <ycheng@google.com>
jobs:
build-amd64-xsm pass
build-armhf-xsm pass
build-i386-xsm pass
build-amd64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
build-amd64-rumpuserxen pass
build-i386-rumpuserxen pass
test-amd64-amd64-xl pass
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-xsm pass
test-armhf-armhf-libvirt-xsm fail
test-amd64-i386-libvirt-xsm pass
test-amd64-amd64-xl-xsm pass
test-armhf-armhf-xl-xsm fail
test-amd64-i386-xl-xsm pass
test-amd64-amd64-qemuu-nested-amd fail
test-amd64-amd64-xl-pvh-amd fail
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-rumpuserxen-amd64 pass
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-armhf-armhf-xl-arndale pass
test-amd64-amd64-xl-credit2 pass
test-armhf-armhf-xl-credit2 fail
test-armhf-armhf-xl-cubietruck fail
test-amd64-i386-freebsd10-i386 pass
test-amd64-i386-rumpuserxen-i386 fail
test-amd64-amd64-qemuu-nested-intel pass
test-amd64-amd64-xl-pvh-intel fail
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-armhf-armhf-libvirt fail
test-amd64-i386-libvirt pass
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-libvirt-pair pass
test-amd64-i386-libvirt-pair pass
test-amd64-amd64-amd64-pvgrub pass
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-pygrub pass
test-armhf-armhf-libvirt-qcow2 fail
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-libvirt-raw fail
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-rtds pass
test-armhf-armhf-xl-rtds fail
test-amd64-i386-xl-qemut-winxpsp3-vcpus1 pass
test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-xl-vhd pass
test-amd64-amd64-xl-qemut-winxpsp3 pass
test-amd64-i386-xl-qemut-winxpsp3 pass
test-amd64-amd64-xl-qemuu-winxpsp3 pass
test-amd64-i386-xl-qemuu-winxpsp3 pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
Not pushing.
(No revision log; it would be 1259 lines long.)
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-4.1 test] 79008: regressions - FAIL
2016-01-26 13:11 [linux-4.1 test] 79008: regressions - FAIL osstest service owner
@ 2016-01-27 11:18 ` Ian Campbell
2016-01-27 12:05 ` Ian Campbell
0 siblings, 1 reply; 5+ messages in thread
From: Ian Campbell @ 2016-01-27 11:18 UTC (permalink / raw)
To: osstest service owner, xen-devel, Ian Jackson
On Tue, 2016-01-26 at 13:11 +0000, osstest service owner wrote:
> flight 79008 linux-4.1 real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/79008/
>
> Regressions :-(
>
> Tests which did not succeed and are blocking,
> including tests which could not be run:
> test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR. vs. 66399
> test-armhf-armhf-xl-xsm 15 guest-start/debian.repeat fail REGR. vs. 66399
These were both:
2016-01-26 01:20:33 Z executing ssh ... root@172.16.147.101 echo guest debian.guest.osstest: ok
Warning: Permanently added '172.16.147.101' (ECDSA) to the list of known hosts.
key_verify failed for server_host_key
Searching for that message online shows a variety of results for the last
15 years or so, although it's not clear that any of them are actually the
same, many of them also involve a "hash mismatch" message, which is not
present here.
As far as our own tests go:
$ echo ~osstest/pub/logs/*/*/[0-9]*.log | xargs grep -l "key_verify failed for server_host_key"
/home/osstest/pub/logs/78033/test-armhf-armhf-xl-cubietruck/15.ts-repeat-test.log
/home/osstest/pub/logs/78925/test-armhf-armhf-xl-credit2/11.ts-guest-start.log
/home/osstest/pub/logs/78925/test-armhf-armhf-xl-cubietruck/15.ts-repeat-test.log
/home/osstest/pub/logs/78925/test-armhf-armhf-xl-xsm/15.ts-repeat-test.log
/home/osstest/pub/logs/79008/test-armhf-armhf-xl-credit2/15.ts-repeat-test.log
/home/osstest/pub/logs/79008/test-armhf-armhf-xl-cubietruck/11.ts-guest-start.log
/home/osstest/pub/logs/79008/test-armhf-armhf-xl-xsm/15.ts-repeat-test.log
/home/osstest/pub/logs/79088/test-armhf-armhf-xl-credit2/15.ts-repeat-test.log
/home/osstest/pub/logs/79090/test-armhf-armhf-xl-credit2/11.ts-guest-start.log
/home/osstest/pub/logs/79090/test-armhf-armhf-xl-cubietruck/11.ts-guest-start.log
/home/osstest/pub/logs/79090/test-armhf-armhf-xl-xsm/11.ts-guest-start.log
/home/osstest/pub/logs/79120/test-armhf-armhf-xl-xsm/15.ts-repeat-test.log
From flights (machines in same order as tests above):
[linux-linus bisection] 78033: tested test-armhf-armhf-xl-cubietruck
cubietruck-braque
[linux-4.1 test] 78925: regressions - trouble: broken/fail/pass
cubietruck-metzinger
cubietruck-gleizes
cubietruck-gleizes
[linux-4.1 test] 79008: regressions - FAIL
cubietruck-metzinger
[linux-4.1 bisection] 79088: testing test-armhf-armhf-xl-credit2
cubietruck-metzinger
79090 is still in progress (nearly complete) but is on linux-4.1
cubietruck-metzinger
cubietruck-gleizes
cubietruck-gleizes
[linux-4.1 bisection] 79120: testing test-armhf-armhf-xl-xsm
cubietruck-gleizes
Neither of the bisects look likely to make any progress:
http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-4.1/test-armhf-armhf-xl-xsm.guest-start--debian.repeat.html
http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-4.1/test-armhf-armhf-xl-credit2.guest-start--debian.repeat.html
We only have logs back as far as 20 Jan, so it's possible there isn't
enough data here to decide anything, but the span does include some logs
from linux-3.18 which seems to not be showing this. OTOH linux-linus has
only hit it once, maybe this was fixed already upstream.
I suspect linux-4.1 has a cubietruck specific hardware issue (perhaps in
the network driver). I'm not sure how best to approach it though,
especially given that it doesn't seem to be reproducible by the bisector.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-4.1 test] 79008: regressions - FAIL
2016-01-27 11:18 ` Ian Campbell
@ 2016-01-27 12:05 ` Ian Campbell
2016-02-02 14:10 ` Ian Campbell
0 siblings, 1 reply; 5+ messages in thread
From: Ian Campbell @ 2016-01-27 12:05 UTC (permalink / raw)
To: osstest service owner, xen-devel, Ian Jackson
On Wed, 2016-01-27 at 11:18 +0000, Ian Campbell wrote:
> On Tue, 2016-01-26 at 13:11 +0000, osstest service owner wrote:
> > flight 79008 linux-4.1 real [real]
> > http://logs.test-lab.xenproject.org/osstest/logs/79008/
> >
> > Regressions :-(
> >
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> > test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR. vs. 66399
> > test-armhf-armhf-xl-xsm 15 guest-start/debian.repeat fail REGR. vs. 66399
>
> These were both:
>
> 2016-01-26 01:20:33 Z executing ssh ... root@172.16.147.101 echo guest debian.guest.osstest: ok
> Warning: Permanently added '172.16.147.101' (ECDSA) to the list of known hosts.
> key_verify failed for server_host_key
While poking through the test histories I also spotted:
http://logs.test-lab.xenproject.org/osstest/logs/78799/test-armhf-armhf-xl-cubietruck/info.html
http://logs.test-lab.xenproject.org/osstest/logs/78997/test-armhf-armhf-xl-cubietruck/info.html
Which are:
2016-01-26 01:03:34 Z executing ssh ... root@172.16.144.48 ps -wwef
Corrupted MAC on input.
Disconnecting: Packet corrupt
(this is dom0 rather than the "server_host_key" thing which was guest)
But it seems like it could be a different symptom of the same underlying
cause. Data-mining the logs a bit I find:
78033 test-armhf-armhf-xl-cubietruck 16.ts-logs-capture.log host cubietruck-braque
78799 test-armhf-armhf-libvirt-raw 8.ts-leak-check.log host~ cubietruck-braque
78799 test-armhf-armhf-libvirt-raw 9.ts-logs-capture.log host~ cubietruck-braque
78799 test-armhf-armhf-libvirt-xsm 8.ts-leak-check.log host~ cubietruck-braque
78799 test-armhf-armhf-libvirt-xsm 9.ts-logs-capture.log host~ cubietruck-braque
78799 test-armhf-armhf-xl 8.ts-leak-check.log host~ cubietruck-picasso
78799 test-armhf-armhf-xl 9.ts-logs-capture.log host~ cubietruck-picasso
78799 test-armhf-armhf-xl-credit2 8.ts-leak-check.log host~ cubietruck-picasso
78799 test-armhf-armhf-xl-credit2 9.ts-logs-capture.log host~ cubietruck-picasso
78799 test-armhf-armhf-xl-cubietruck 8.ts-leak-check.log host~ cubietruck-braque
78799 test-armhf-armhf-xl-cubietruck 9.ts-logs-capture.log host~ cubietruck-braque
78799 test-armhf-armhf-xl-multivcpu 8.ts-leak-check.log host~ cubietruck-gleizes
78799 test-armhf-armhf-xl-multivcpu 9.ts-logs-capture.log host~ cubietruck-gleizes
78799 test-armhf-armhf-xl-vhd 10.ts-logs-capture.log host~ cubietruck-metzinger
78799 test-armhf-armhf-xl-xsm 8.ts-leak-check.log host~ cubietruck-metzinger
78799 test-armhf-armhf-xl-xsm 9.ts-logs-capture.log host~ cubietruck-metzinger
78925 test-armhf-armhf-libvirt-raw 10.ts-logs-capture.log host~ cubietruck-picasso
78925 test-armhf-armhf-xl-credit2 12.ts-logs-capture.log host~ cubietruck-metzinger
78925 test-armhf-armhf-xl-cubietruck 16.ts-logs-capture.log host~ cubietruck-gleizes
78925 test-armhf-armhf-xl-multivcpu 19.ts-logs-capture.log host~ cubietruck-metzinger
78925 test-armhf-armhf-xl-xsm 16.ts-logs-capture.log host~ cubietruck-gleizes
78997 test-armhf-armhf-libvirt-raw 10.ts-logs-capture.log host~ cubietruck-braque
78997 test-armhf-armhf-libvirt-xsm 8.ts-leak-check.log host~ cubietruck-braque
78997 test-armhf-armhf-libvirt-xsm 9.ts-logs-capture.log host~ cubietruck-braque
78997 test-armhf-armhf-xl 8.ts-leak-check.log host~ cubietruck-gleizes
78997 test-armhf-armhf-xl 9.ts-logs-capture.log host~ cubietruck-gleizes
78997 test-armhf-armhf-xl-credit2 8.ts-leak-check.log host~ cubietruck-picasso
78997 test-armhf-armhf-xl-credit2 9.ts-logs-capture.log host~ cubietruck-picasso
78997 test-armhf-armhf-xl-cubietruck 8.ts-leak-check.log host~ cubietruck-braque
78997 test-armhf-armhf-xl-cubietruck 9.ts-logs-capture.log host~ cubietruck-braque
78997 test-armhf-armhf-xl-multivcpu 8.ts-leak-check.log host~ cubietruck-metzinger
78997 test-armhf-armhf-xl-multivcpu 9.ts-logs-capture.log host~ cubietruck-metzinger
78997 test-armhf-armhf-xl-vhd 10.ts-logs-capture.log host~ cubietruck-metzinger
78997 test-armhf-armhf-xl-xsm 8.ts-leak-check.log host~ cubietruck-gleizes
78997 test-armhf-armhf-xl-xsm 9.ts-logs-capture.log host~ cubietruck-gleizes
79008 test-armhf-armhf-libvirt-raw 10.ts-logs-capture.log host~ cubietruck-picasso
79008 test-armhf-armhf-xl-credit2 16.ts-logs-capture.log host~ cubietruck-metzinger
79008 test-armhf-armhf-xl-cubietruck 12.ts-logs-capture.log host~ cubietruck-gleizes
79008 test-armhf-armhf-xl-multivcpu 19.ts-logs-capture.log host~ cubietruck-picasso
79008 test-armhf-armhf-xl-xsm 16.ts-logs-capture.log host~ cubietruck-gleizes
79088 test-armhf-armhf-xl-credit2 16.ts-logs-capture.log host cubietruck-metzinger
79090 test-armhf-armhf-libvirt-raw 10.ts-logs-capture.log host~ cubietruck-picasso
79090 test-armhf-armhf-xl-credit2 12.ts-logs-capture.log host~ cubietruck-metzinger
79090 test-armhf-armhf-xl-cubietruck 12.ts-logs-capture.log host~ cubietruck-gleizes
79090 test-armhf-armhf-xl-vhd 10.ts-logs-capture.log host~ cubietruck-braque
79090 test-armhf-armhf-xl-xsm 12.ts-logs-capture.log host~ cubietruck-gleizes
79120 test-armhf-armhf-xl-xsm 16.ts-logs-capture.log host cubietruck-gleizes
and:
78033|linux-linus
78799|linux-next
78925|linux-4.1
78997|linux-next
79008|linux-4.1
79088|linux-4.1
79090|linux-4.1
79120|linux-4.1
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-4.1 test] 79008: regressions - FAIL
2016-01-27 12:05 ` Ian Campbell
@ 2016-02-02 14:10 ` Ian Campbell
2016-02-05 14:51 ` Ian Campbell
0 siblings, 1 reply; 5+ messages in thread
From: Ian Campbell @ 2016-02-02 14:10 UTC (permalink / raw)
To: osstest service owner, xen-devel, Ian Jackson
[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]
On Wed, 2016-01-27 at 12:05 +0000, Ian Campbell wrote:
> On Wed, 2016-01-27 at 11:18 +0000, Ian Campbell wrote:
> > On Tue, 2016-01-26 at 13:11 +0000, osstest service owner wrote:
> > > flight 79008 linux-4.1 real [real]
> > > http://logs.test-lab.xenproject.org/osstest/logs/79008/
> > >
> > > Regressions :-(
> > >
> > > Tests which did not succeed and are blocking,
> > > including tests which could not be run:
> > > test-armhf-armhf-xl-credit2 15 guest-start/debian.repeat fail REGR.
> > > vs. 66399
> > > test-armhf-armhf-xl-xsm 15 guest-start/debian.repeat fail REGR.
> > > vs. 66399
> >
> > These were both:
> >
> > 2016-01-26 01:20:33 Z executing ssh ... root@172.16.147.101 echo guest
> > debian.guest.osstest: ok
> > Warning: Permanently added '172.16.147.101' (ECDSA) to the list of
> > known hosts.
> > key_verify failed for server_host_key
So I've narrowed this down a bit, but not yet sufficiently to actually
diagnose.
The issue only occurs when the userspace is Debian Jessie. Debian Wheezy
does not, for some reason, expose this. It seems unlikely (although not
impossible) to be a real issue in Jessie vs Wheezy, more likely some
different behaviour in Jessie's sshd just exposes some issue somewhere
else.
When running Jessie userspace the issue only appeared somewhere between
Linux v3.18 and v3.19, I'm currently looking at bisecting that range in
case the commit which exposed the issue gives a hint (I fear it wont
though).
The attached ts-fetch-check-file exposes this pretty readily against either
dom0 or domU (slightly differing symptoms) on effected versions.
I can reproduce on the cubietruck on my desk as well as in the COLO.
I cannot reproduce on the same cubietruck on my desk when running native
4.1, it only happens when running under Xen.
I cannot reproduce on the arndale on my desk. I've not tried in the COLO
since the test results suggest there would be no point -- there is no sign
of this class of failure in the colo on arndale nor on any x86 box.
Ian.
[-- Attachment #2: 0001-ts-fetch-check-file-new-ts-to-fetch-a-file-and-check.patch --]
[-- Type: text/x-patch, Size: 4400 bytes --]
From 337f663c13e46f815ce1f13b070b492f8d248b0c Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Tue, 2 Feb 2016 10:54:42 +0000
Subject: [PATCH] ts-fetch-check-file: new ts to fetch a file and check for
corruption
Compares a checksum computed on the target with one computed after
cat'ting the file over ssh. Picks up on network corruption errors etc
which might be missed with smaller interactions.
Works for guests or hosts.
To support this add a variant of target_cmd_output which returns the
file descriptor instead of the actual data (which could be large),
allowing us to pipe it to the local sum.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
Osstest/TestSupport.pm | 10 ++++++--
ts-fetch-check-file | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 76 insertions(+), 2 deletions(-)
create mode 100755 ts-fetch-check-file
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 2141905..3c287b2 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -51,6 +51,7 @@ BEGIN {
target_cmd_root target_cmd target_cmd_build
target_cmd_output_root target_cmd_output
+ target_cmd_stdoutfd target_cmd_stdoutfd_root
target_cmd_inputfh_root sshuho
target_getfile target_getfile_root
target_putfile target_putfile_root
@@ -646,9 +647,11 @@ sub target_cmd ($$;$$) { tcmd(undef,undef,'osstest',@_); }
sub target_cmd_root ($$;$$) { tcmd(undef,undef,'root',@_); }
sub tcmdout {
+ my $wantfd = shift;
my $stdout= IO::File::new_tmpfile();
tcmd(undef,$stdout,@_);
$stdout->seek(0,0) or die "$stdout $!";
+ return $stdout if $wantfd;
my $r;
{ local ($/) = undef;
$r= <$stdout>; }
@@ -657,8 +660,11 @@ sub tcmdout {
return $r;
}
-sub target_cmd_output ($$;$) { tcmdout('osstest',@_); }
-sub target_cmd_output_root ($$;$) { tcmdout('root',@_); }
+sub target_cmd_output ($$;$) { tcmdout(0,'osstest',@_); }
+sub target_cmd_output_root ($$;$) { tcmdout(0,'root',@_); }
+
+sub target_cmd_stdoutfd ($$;$$) { tcmdout(1,'osstest',@_); }
+sub target_cmd_stdoutfd_root ($$;$$) { tcmdout(1,'root',@_); }
sub target_cmd_inputfh_root ($$$;$$) {
my ($tho,$stdinfh,$tcmd,@rest) = @_;
diff --git a/ts-fetch-check-file b/ts-fetch-check-file
new file mode 100755
index 0000000..bfceb6b
--- /dev/null
+++ b/ts-fetch-check-file
@@ -0,0 +1,68 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2016 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::TestSupport;
+
+use IO::Pipe;
+
+tsreadconfig();
+
+our ($whhost,$guest) = @ARGV;
+$whhost ||= 'host';
+
+our ($ho,$gho);
+our $fn = "/bin/bash"; # reasonable size, present in most guests and hosts
+
+$ho= selecthost($whhost);
+$gho= selectguest($guest,$ho) if $guest;
+
+sub fetch_and_check_file ($$) {
+ my ($t,$fn) = @_;
+
+ target_check_ip($t);
+
+ target_cmd_root($t, "ls -lH $fn");
+
+ my $expect = target_cmd_output_root($t,"sum $fn");
+ logm($expect);
+
+ my $stdout= target_cmd_stdoutfd_root($t,"cat $fn",5,[qw(-v)]);
+
+ my $pipe= IO::Pipe->new();
+ my $child= fork; die $! unless defined $child;
+ if (!$child) {
+ $pipe->writer();
+ open STDIN, "<&", $stdout or die "STDIN $!";
+ open STDOUT, ">&", $pipe or die "STDOUT $!";
+ exec("sum") or die "pipe writer $!";
+ }
+
+ $pipe->reader();
+
+ my $got = <$pipe>;
+ chomp($got);
+
+ logm("expected $expect");
+ logm("got $got");
+
+ die unless $expect eq $got;
+}
+
+fetch_and_check_file($gho ? $gho : $ho, $fn);
--
2.6.1
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [linux-4.1 test] 79008: regressions - FAIL
2016-02-02 14:10 ` Ian Campbell
@ 2016-02-05 14:51 ` Ian Campbell
0 siblings, 0 replies; 5+ messages in thread
From: Ian Campbell @ 2016-02-05 14:51 UTC (permalink / raw)
To: osstest service owner, xen-devel, Ian Jackson; +Cc: Stefano Stabellini
On Tue, 2016-02-02 at 14:10 +0000, Ian Campbell wrote:
>
> When running Jessie userspace the issue only appeared somewhere between
> Linux v3.18 and v3.19, I'm currently looking at bisecting that range in
> case the commit which exposed the issue gives a hint (I fear it wont
> though).
Bisecting the dom0 failure lead me to:
3567258d281b5b515d5165ed23851d9f84087e7d is the first bad commit
commit 3567258d281b5b515d5165ed23851d9f84087e7d
Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Date: Fri Nov 21 11:05:39 2014 +0000
xen/arm: use hypercall to flush caches in map_page
In xen_dma_map_page, if the page is a local page, call the native
map_page dma_ops. If the page is foreign, call __xen_dma_map_page that
issues any required cache maintenane operations via hypercall.
The reason for doing this is that the native dma_ops map_page could
allocate buffers than need to be freed. If the page is foreign we don't
call the native unmap_page dma_ops function, resulting in a memory leak.
Suggested-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
However AIUI this commit is supposed to be a NOP for all dom0 initiated
I/O, which is all which should be occurring in a test which only involves
ssh to dom0.
Something to do with cache flushes, dma and/or barriers does seem like a
plausible candidate for there error though.
Ian.
NB last few were combined with
git cherry-pick --no-commit 28603d13997e2ef47f18589cc9a44553aad49c86
else the NIC driver just crashes on boot.
git bisect start
# bad: [bfa76d49576599a4b9f9b7a71f23d73d6dcff735] Linux 3.19
git bisect bad bfa76d49576599a4b9f9b7a71f23d73d6dcff735
# good: [b2776bf7149bddd1f4161f14f79520f17fc1d71d] Linux 3.18
git bisect good b2776bf7149bddd1f4161f14f79520f17fc1d71d
# bad: [54850e73e86e3bc092680d1bdb84eb322f982ab1] zram: change parameter from vaild_io_request()
git bisect bad 54850e73e86e3bc092680d1bdb84eb322f982ab1
# good: [6b9e2cea428cf7af93a84bcb865e478d8bf1c165] Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost
git bisect good 6b9e2cea428cf7af93a84bcb865e478d8bf1c165
# good: [b5f185f33d0432cef6ff78765e033dfa8f4de068] Merge tag 'master-2014-12-08' of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next
git bisect good b5f185f33d0432cef6ff78765e033dfa8f4de068
# good: [bae41e45b7400496b9bf0c70c6004419d9987819] Merge tag 'sound-3.19-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect good bae41e45b7400496b9bf0c70c6004419d9987819
# good: [c0222ac086669a631814bbf857f8c8023452a4d7] Merge branch 'upstream' of git://git.linux-mips.org/pub/scm/ralf/upstream-linus
git bisect good c0222ac086669a631814bbf857f8c8023452a4d7
# bad: [a7cb7bb664543e4562ab0e9a072470d2d18c761f] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
git bisect bad a7cb7bb664543e4562ab0e9a072470d2d18c761f
# bad: [9bfccec24e31f4f83445cfe0c1b0a5ef97900628] Merge tag 'ext4_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4
git bisect bad 9bfccec24e31f4f83445cfe0c1b0a5ef97900628
# bad: [4e8790f77f051d4cc745a57b48a73052521e8dfc] Merge branch 'for-3.19' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/libata
git bisect bad 4e8790f77f051d4cc745a57b48a73052521e8dfc
# skip: [b1df4a56bf4a61113e8928f932d346bed6eef553] xen/pciback: Restore configuration space when detaching from a guest.
git bisect skip b1df4a56bf4a61113e8928f932d346bed6eef553
# bad: [9d050966e2eb37a643ac15904b6a8fda7fcfabe9] Merge tag 'stable/for-linus-3.19-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
git bisect bad 9d050966e2eb37a643ac15904b6a8fda7fcfabe9
# bad: [9490c6c67e2f41760de8ece4e4f56f75f84ceb9e] swiotlb-xen: call xen_dma_sync_single_for_device when appropriate
git bisect bad 9490c6c67e2f41760de8ece4e4f56f75f84ceb9e
# good: [a0f2dee0cd651efb5fac6a1d35b0a14460ebcdd4] xen: add a dma_addr_t dev_addr argument to xen_dma_map_page
git bisect good a0f2dee0cd651efb5fac6a1d35b0a14460ebcdd4
# bad: [a4dba130891271084344c12537731542ec77cb85] xen/arm/arm64: introduce xen_arch_need_swiotlb
git bisect bad a4dba130891271084344c12537731542ec77cb85
# bad: [3567258d281b5b515d5165ed23851d9f84087e7d] xen/arm: use hypercall to flush caches in map_page
git bisect bad 3567258d281b5b515d5165ed23851d9f84087e7d
# first bad commit: [3567258d281b5b515d5165ed23851d9f84087e7d] xen/arm: use hypercall to flush caches in map_page
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-02-05 14:51 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-01-26 13:11 [linux-4.1 test] 79008: regressions - FAIL osstest service owner
2016-01-27 11:18 ` Ian Campbell
2016-01-27 12:05 ` Ian Campbell
2016-02-02 14:10 ` Ian Campbell
2016-02-05 14:51 ` Ian Campbell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.