* [xen-4.5-testing baseline-only test] 67737: regressions - FAIL
@ 2016-09-21 14:09 Platform Team regression test user
2016-09-21 16:13 ` Problem with Xen 4.5 failing XTF tests on old AMD cpus ? Ian Jackson
0 siblings, 1 reply; 12+ messages in thread
From: Platform Team regression test user @ 2016-09-21 14:09 UTC (permalink / raw)
To: xen-devel, osstest-admin
This run is configured for baseline tests only.
flight 67737 xen-4.5-testing real [real]
http://osstest.xs.citrite.net/~osstest/testlogs/logs/67737/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-xtf-amd64-amd64-1 19 xtf/test-hvm32-invlpg~shadow fail REGR. vs. 67706
test-xtf-amd64-amd64-1 26 xtf/test-hvm32pae-invlpg~shadow fail REGR. vs. 67706
test-xtf-amd64-amd64-1 34 xtf/test-hvm64-invlpg~shadow fail REGR. vs. 67706
test-xtf-amd64-amd64-5 19 xtf/test-hvm32-invlpg~shadow fail REGR. vs. 67706
test-xtf-amd64-amd64-5 26 xtf/test-hvm32pae-invlpg~shadow fail REGR. vs. 67706
test-xtf-amd64-amd64-5 34 xtf/test-hvm64-invlpg~shadow fail REGR. vs. 67706
test-amd64-amd64-libvirt-pair 9 xen-boot/src_host fail REGR. vs. 67706
Regressions which are regarded as allowable (not blocking):
test-amd64-amd64-qemuu-nested-intel 13 xen-boot/l1 fail like 67706
test-amd64-amd64-xl-qemut-win7-amd64 16 guest-stop fail like 67706
test-amd64-amd64-amd64-pvgrub 10 guest-start fail like 67706
test-amd64-amd64-xl-rtds 6 xen-boot fail like 67706
test-amd64-amd64-xl-qemuu-win7-amd64 15 guest-localmigrate/x10 fail like 67706
test-amd64-amd64-xl-qemut-winxpsp3 9 windows-install fail like 67706
Tests which did not succeed, but are not blocking:
test-amd64-i386-rumprun-i386 1 build-check(1) blocked n/a
test-amd64-amd64-rumprun-amd64 1 build-check(1) blocked n/a
build-i386-rumprun 5 rumprun-build fail never pass
test-armhf-armhf-libvirt-raw 10 guest-start fail never pass
test-amd64-amd64-libvirt-vhd 11 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 12 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 13 saverestore-support-check fail never pass
test-armhf-armhf-libvirt-qcow2 10 guest-start fail never pass
test-xtf-amd64-amd64-2 19 xtf/test-hvm32-invlpg~shadow fail never pass
test-xtf-amd64-amd64-2 26 xtf/test-hvm32pae-invlpg~shadow fail never pass
test-xtf-amd64-amd64-2 34 xtf/test-hvm64-invlpg~shadow fail never pass
test-armhf-armhf-xl 12 migrate-support-check fail never pass
test-armhf-armhf-xl 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-midway 12 migrate-support-check fail never pass
test-armhf-armhf-xl-midway 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-multivcpu 12 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 13 saverestore-support-check fail never pass
test-armhf-armhf-xl-rtds 12 migrate-support-check fail never pass
test-armhf-armhf-xl-rtds 13 saverestore-support-check fail never pass
build-amd64-rumprun 5 rumprun-build fail never pass
test-armhf-armhf-libvirt 12 migrate-support-check fail never pass
test-armhf-armhf-libvirt 14 guest-saverestore fail never pass
test-armhf-armhf-xl-vhd 10 guest-start fail never pass
test-amd64-i386-xl-qemut-win7-amd64 16 guest-stop fail never pass
test-amd64-amd64-libvirt 12 migrate-support-check fail never pass
test-amd64-i386-libvirt 12 migrate-support-check fail never pass
test-amd64-amd64-xl-pvh-intel 11 guest-start fail never pass
test-amd64-i386-xl-qemuu-win7-amd64 16 guest-stop fail never pass
test-amd64-amd64-xl-pvh-amd 11 guest-start fail never pass
test-amd64-amd64-qemuu-nested-amd 16 debian-hvm-install/l1/l2 fail never pass
version targeted for testing:
xen e4ae4b03d35babc9624b7286f1ea4c6749bad84b
baseline version:
xen c18dfbb88670e9f2cabd93bbb32f661b71ffb73a
Last test of basis 67706 2016-09-13 20:15:55 Z 7 days
Testing same since 67737 2016-09-21 02:18:55 Z 0 days 1 attempts
------------------------------------------------------------
People who touched revisions under test:
Jan Beulich <jbeulich@suse.com>
jobs:
build-amd64-xtf pass
build-amd64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-prev pass
build-i386-prev pass
build-amd64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
build-amd64-rumprun fail
build-i386-rumprun fail
test-xtf-amd64-amd64-1 pass
test-xtf-amd64-amd64-2 pass
test-xtf-amd64-amd64-3 pass
test-xtf-amd64-amd64-4 pass
test-xtf-amd64-amd64-5 pass
test-amd64-amd64-xl pass
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-qemuu-nested-amd fail
test-amd64-amd64-xl-pvh-amd fail
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-rumprun-amd64 blocked
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-amd64-amd64-xl-credit2 pass
test-armhf-armhf-xl-credit2 pass
test-amd64-i386-freebsd10-i386 pass
test-amd64-i386-rumprun-i386 blocked
test-amd64-amd64-qemuu-nested-intel fail
test-amd64-amd64-xl-pvh-intel fail
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-armhf-armhf-libvirt fail
test-amd64-i386-libvirt pass
test-armhf-armhf-xl-midway pass
test-amd64-amd64-migrupgrade pass
test-amd64-i386-migrupgrade pass
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-libvirt-pair fail
test-amd64-i386-libvirt-pair pass
test-amd64-amd64-amd64-pvgrub fail
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-pygrub pass
test-armhf-armhf-libvirt-qcow2 fail
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-libvirt-raw fail
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-rtds fail
test-armhf-armhf-xl-rtds pass
test-amd64-i386-xl-qemut-winxpsp3-vcpus1 pass
test-amd64-i386-xl-qemuu-winxpsp3-vcpus1 pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-xl-vhd fail
test-amd64-amd64-xl-qemut-winxpsp3 fail
test-amd64-i386-xl-qemut-winxpsp3 pass
test-amd64-amd64-xl-qemuu-winxpsp3 pass
test-amd64-i386-xl-qemuu-winxpsp3 pass
------------------------------------------------------------
sg-report-flight on osstest.xs.citrite.net
logs: /home/osstest/logs
images: /home/osstest/images
Logs, config files, etc. are available at
http://osstest.xs.citrite.net/~osstest/testlogs/logs
Test harness code can be found at
http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
Push not applicable.
------------------------------------------------------------
commit e4ae4b03d35babc9624b7286f1ea4c6749bad84b
Author: Jan Beulich <jbeulich@suse.com>
Date: Tue Sep 20 07:59:24 2016 +0200
update Xen version to 4.5.5
One of the qemu tags did get created the wrong way, so we need to skip
4.5.4.
commit 22857ab3492c35aac27691ae7184dcc935c5fb2a
Author: Jan Beulich <jbeulich@suse.com>
Date: Mon Sep 19 17:50:59 2016 +0200
update Xen version to 4.5.4
(qemu changes not included)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread* Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-21 14:09 [xen-4.5-testing baseline-only test] 67737: regressions - FAIL Platform Team regression test user @ 2016-09-21 16:13 ` Ian Jackson 2016-09-21 16:15 ` Ian Jackson ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: Ian Jackson @ 2016-09-21 16:13 UTC (permalink / raw) To: Boris Ostrovsky, Suravee Suthikulpanit, Aravind Gopalakrishnan Cc: Andrew Cooper, xen-devel, Tim Deegan, Wei Liu, Jan Beulich Platform Team regression test user writes ("[xen-4.5-testing baseline-only test] 67737: regressions - FAIL"): > test-xtf-amd64-amd64-1 19 xtf/test-hvm32-invlpg~shadow fail REGR. vs. 67706 > test-xtf-amd64-amd64-1 26 xtf/test-hvm32pae-invlpg~shadow fail REGR. vs. 67706 Several of these, 32bit and 64bit HVM. This is in the Citrix Cambridge osstest instance. The Xen Project colo instance is unaffected (flight 101045 there passed with the same revisions of everything) This is with: xen e4ae4b03d35babc9624b7286f1ea4c6749bad84b xtf b5c5332de4268d33a6f8eadc1d17c7b9cf0e7dc3 linux b65f2f457c49b2cfd7967c34b7a0b04c25587f13 linux-firmware c530a75c1e6a472b0eb9558310b518f0dfcd8860 The log says this: 2016-09-21 06:07:36 Z ---------- substep 19 xtf/test-hvm32-invlpg~shadow running ---------- 2016-09-21 06:07:36 Z executing ssh ... root@10.80.228.78 /home/xtf/xtf-runner -m logfile test-hvm32-invlpg~shadow 1>&2; echo $? Using logfile '/var/log/xen/console/guest-test-hvm32-invlpg~shadow.log' Executing 'xl create -F tests/invlpg/test-hvm32-invlpg~shadow.cfg' --- Xen Test Framework --- Environment: HVM 32bit (No paging) Testing 'invlpg' in normally-faulting conditions Test: Mapped address Test: Unmapped address Test: NULL segment override Test: Past segment limit Fail: Unexpected #GP[0000] Test: Before expand-down segment limit Fail: Unexpected #GP[0000] Test result: FAILURE Combined test results: test-hvm32-invlpg~shadow FAILURE Sadly we haven't yet managed to make the logs from this instance public. Do you have any idea what might be causing this ? Is there a real problem with the Xen 4.5 branch ? The Citrix Cambridge instance has old hardware. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-21 16:13 ` Problem with Xen 4.5 failing XTF tests on old AMD cpus ? Ian Jackson @ 2016-09-21 16:15 ` Ian Jackson 2016-09-21 16:42 ` Boris Ostrovsky 2016-09-21 17:27 ` Andrew Cooper 2 siblings, 0 replies; 12+ messages in thread From: Ian Jackson @ 2016-09-21 16:15 UTC (permalink / raw) To: Boris Ostrovsky, Suravee Suthikulpanit, Aravind Gopalakrishnan, Jan Beulich, Tim Deegan, xen-devel, Andrew Cooper, Wei Liu Ian Jackson writes ("Problem with Xen 4.5 failing XTF tests on old AMD cpus ?"): > Sadly we haven't yet managed to make the logs from this instance > public. I have copied the logs from this one test job to here: http://xenbits.xen.org/people/iwj/2016/67737/test-xtf-amd64-amd64-1/info.html Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-21 16:13 ` Problem with Xen 4.5 failing XTF tests on old AMD cpus ? Ian Jackson 2016-09-21 16:15 ` Ian Jackson @ 2016-09-21 16:42 ` Boris Ostrovsky 2016-09-21 17:36 ` Wei Liu 2016-09-21 17:27 ` Andrew Cooper 2 siblings, 1 reply; 12+ messages in thread From: Boris Ostrovsky @ 2016-09-21 16:42 UTC (permalink / raw) To: Ian Jackson, Suravee Suthikulpanit, Aravind Gopalakrishnan Cc: Andrew Cooper, xen-devel, Tim Deegan, Wei Liu, Jan Beulich On 09/21/2016 12:13 PM, Ian Jackson wrote: > Platform Team regression test user writes ("[xen-4.5-testing baseline-only test] 67737: regressions - FAIL"): >> test-xtf-amd64-amd64-1 19 xtf/test-hvm32-invlpg~shadow fail REGR. vs. 67706 >> test-xtf-amd64-amd64-1 26 xtf/test-hvm32pae-invlpg~shadow fail REGR. vs. 67706 > Several of these, 32bit and 64bit HVM. This is in the Citrix > Cambridge osstest instance. The Xen Project colo instance is > unaffected (flight 101045 there passed with the same revisions of > everything) > > This is with: > > xen e4ae4b03d35babc9624b7286f1ea4c6749bad84b > xtf b5c5332de4268d33a6f8eadc1d17c7b9cf0e7dc3 > linux b65f2f457c49b2cfd7967c34b7a0b04c25587f13 > linux-firmware c530a75c1e6a472b0eb9558310b518f0dfcd8860 I can't get these commits neither for Xen nor for Linux. Are these from Citrix trees? > > The log says this: > > 2016-09-21 06:07:36 Z ---------- substep 19 xtf/test-hvm32-invlpg~shadow running ---------- > 2016-09-21 06:07:36 Z executing ssh ... root@10.80.228.78 /home/xtf/xtf-runner -m logfile test-hvm32-invlpg~shadow 1>&2; echo $? > > Using logfile '/var/log/xen/console/guest-test-hvm32-invlpg~shadow.log' > Executing 'xl create -F tests/invlpg/test-hvm32-invlpg~shadow.cfg' > --- Xen Test Framework --- > Environment: HVM 32bit (No paging) > Testing 'invlpg' in normally-faulting conditions > Test: Mapped address > Test: Unmapped address > Test: NULL segment override > Test: Past segment limit > Fail: Unexpected #GP[0000] > Test: Before expand-down segment limit > Fail: Unexpected #GP[0000] > Test result: FAILURE > > Combined test results: > test-hvm32-invlpg~shadow FAILURE > > Sadly we haven't yet managed to make the logs from this instance > public. I looked at the logs you posted but I can't find guest config file. Are they part of the logs? > > Do you have any idea what might be causing this ? Is there a real > problem with the Xen 4.5 branch ? The Citrix Cambridge instance has > old hardware. We run 4.5 every week or so here but we don't run without HAP (which, based on the name, I assume is what this guest is). I'll give it a try. -boris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-21 16:42 ` Boris Ostrovsky @ 2016-09-21 17:36 ` Wei Liu 2016-09-21 18:00 ` Boris Ostrovsky 0 siblings, 1 reply; 12+ messages in thread From: Wei Liu @ 2016-09-21 17:36 UTC (permalink / raw) To: Boris Ostrovsky Cc: xen-devel, Wei Liu, Suravee Suthikulpanit, Tim Deegan, Ian Jackson, Aravind Gopalakrishnan, Jan Beulich, Andrew Cooper On Wed, Sep 21, 2016 at 12:42:51PM -0400, Boris Ostrovsky wrote: > On 09/21/2016 12:13 PM, Ian Jackson wrote: > > Platform Team regression test user writes ("[xen-4.5-testing baseline-only test] 67737: regressions - FAIL"): > >> test-xtf-amd64-amd64-1 19 xtf/test-hvm32-invlpg~shadow fail REGR. vs. 67706 > >> test-xtf-amd64-amd64-1 26 xtf/test-hvm32pae-invlpg~shadow fail REGR. vs. 67706 > > Several of these, 32bit and 64bit HVM. This is in the Citrix > > Cambridge osstest instance. The Xen Project colo instance is > > unaffected (flight 101045 there passed with the same revisions of > > everything) > > > > This is with: > > > > xen e4ae4b03d35babc9624b7286f1ea4c6749bad84b > > xtf b5c5332de4268d33a6f8eadc1d17c7b9cf0e7dc3 > > linux b65f2f457c49b2cfd7967c34b7a0b04c25587f13 > > linux-firmware c530a75c1e6a472b0eb9558310b518f0dfcd8860 > > I can't get these commits neither for Xen nor for Linux. Are these from > Citrix trees? No. They are all upstream commits. > > > > > The log says this: > > > > 2016-09-21 06:07:36 Z ---------- substep 19 xtf/test-hvm32-invlpg~shadow running ---------- > > 2016-09-21 06:07:36 Z executing ssh ... root@10.80.228.78 /home/xtf/xtf-runner -m logfile test-hvm32-invlpg~shadow 1>&2; echo $? > > > > Using logfile '/var/log/xen/console/guest-test-hvm32-invlpg~shadow.log' > > Executing 'xl create -F tests/invlpg/test-hvm32-invlpg~shadow.cfg' > > --- Xen Test Framework --- > > Environment: HVM 32bit (No paging) > > Testing 'invlpg' in normally-faulting conditions > > Test: Mapped address > > Test: Unmapped address > > Test: NULL segment override > > Test: Past segment limit > > Fail: Unexpected #GP[0000] > > Test: Before expand-down segment limit > > Fail: Unexpected #GP[0000] > > Test result: FAILURE > > > > Combined test results: > > test-hvm32-invlpg~shadow FAILURE > > > > Sadly we haven't yet managed to make the logs from this instance > > public. > > I looked at the logs you posted but I can't find guest config file. Are > they part of the logs? The guest config file is not stored because it is part of xtf. So ... > > > > > Do you have any idea what might be causing this ? Is there a real > > problem with the Xen 4.5 branch ? The Citrix Cambridge instance has > > old hardware. > > We run 4.5 every week or so here but we don't run without HAP (which, > based on the name, I assume is what this guest is). > > I'll give it a try. > ... the best way to try is to build and run xtf. After building: ./xtf-runner --host --functional > -boris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-21 17:36 ` Wei Liu @ 2016-09-21 18:00 ` Boris Ostrovsky 2016-09-21 18:06 ` Wei Liu 0 siblings, 1 reply; 12+ messages in thread From: Boris Ostrovsky @ 2016-09-21 18:00 UTC (permalink / raw) To: Wei Liu Cc: xen-devel, Suravee Suthikulpanit, Tim Deegan, Ian Jackson, Aravind Gopalakrishnan, Jan Beulich, Andrew Cooper On 09/21/2016 01:36 PM, Wei Liu wrote: > On Wed, Sep 21, 2016 at 12:42:51PM -0400, Boris Ostrovsky wrote: >> On 09/21/2016 12:13 PM, Ian Jackson wrote: >>> Platform Team regression test user writes ("[xen-4.5-testing baseline-only test] 67737: regressions - FAIL"): >>>> test-xtf-amd64-amd64-1 19 xtf/test-hvm32-invlpg~shadow fail REGR. vs. 67706 >>>> test-xtf-amd64-amd64-1 26 xtf/test-hvm32pae-invlpg~shadow fail REGR. vs. 67706 >>> Several of these, 32bit and 64bit HVM. This is in the Citrix >>> Cambridge osstest instance. The Xen Project colo instance is >>> unaffected (flight 101045 there passed with the same revisions of >>> everything) >>> >>> This is with: >>> >>> xen e4ae4b03d35babc9624b7286f1ea4c6749bad84b >>> xtf b5c5332de4268d33a6f8eadc1d17c7b9cf0e7dc3 >>> linux b65f2f457c49b2cfd7967c34b7a0b04c25587f13 >>> linux-firmware c530a75c1e6a472b0eb9558310b518f0dfcd8860 >> I can't get these commits neither for Xen nor for Linux. Are these from >> Citrix trees? > No. They are all upstream commits. Apparently xen commit *just* made it to the tree, after I checked it. And I still don't see Linux one. I ran a quick test (not xtf, our internal one) with 32-bit shadow guest and didn't see anything. But then Andrew seems to have pointed out what the problem is. -boris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-21 18:00 ` Boris Ostrovsky @ 2016-09-21 18:06 ` Wei Liu 2016-09-21 18:20 ` Andrew Cooper 0 siblings, 1 reply; 12+ messages in thread From: Wei Liu @ 2016-09-21 18:06 UTC (permalink / raw) To: Boris Ostrovsky Cc: xen-devel, Wei Liu, Suravee Suthikulpanit, Tim Deegan, Ian Jackson, Aravind Gopalakrishnan, Jan Beulich, Andrew Cooper On Wed, Sep 21, 2016 at 02:00:50PM -0400, Boris Ostrovsky wrote: > On 09/21/2016 01:36 PM, Wei Liu wrote: > > On Wed, Sep 21, 2016 at 12:42:51PM -0400, Boris Ostrovsky wrote: > >> On 09/21/2016 12:13 PM, Ian Jackson wrote: > >>> Platform Team regression test user writes ("[xen-4.5-testing baseline-only test] 67737: regressions - FAIL"): > >>>> test-xtf-amd64-amd64-1 19 xtf/test-hvm32-invlpg~shadow fail REGR. vs. 67706 > >>>> test-xtf-amd64-amd64-1 26 xtf/test-hvm32pae-invlpg~shadow fail REGR. vs. 67706 > >>> Several of these, 32bit and 64bit HVM. This is in the Citrix > >>> Cambridge osstest instance. The Xen Project colo instance is > >>> unaffected (flight 101045 there passed with the same revisions of > >>> everything) > >>> > >>> This is with: > >>> > >>> xen e4ae4b03d35babc9624b7286f1ea4c6749bad84b > >>> xtf b5c5332de4268d33a6f8eadc1d17c7b9cf0e7dc3 > >>> linux b65f2f457c49b2cfd7967c34b7a0b04c25587f13 > >>> linux-firmware c530a75c1e6a472b0eb9558310b518f0dfcd8860 > >> I can't get these commits neither for Xen nor for Linux. Are these from > >> Citrix trees? > > No. They are all upstream commits. > > Apparently xen commit *just* made it to the tree, after I checked it. Yes, it passed in Mass COLO. We suspected it can be related to generations of AMD cpus (hence the "old" in email title). > And I still don't see Linux one. > It should be upstream commit, too. But I don't think it matters that much anyway. > I ran a quick test (not xtf, our internal one) with 32-bit shadow guest > and didn't see anything. But then Andrew seems to have pointed out what That's probably because your guest is a well-behaved guest. > the problem is. > > -boris > _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-21 18:06 ` Wei Liu @ 2016-09-21 18:20 ` Andrew Cooper 2016-09-22 9:29 ` Ian Jackson 0 siblings, 1 reply; 12+ messages in thread From: Andrew Cooper @ 2016-09-21 18:20 UTC (permalink / raw) To: Wei Liu, Boris Ostrovsky Cc: xen-devel, Suravee Suthikulpanit, Tim Deegan, Ian Jackson, Aravind Gopalakrishnan, Jan Beulich On 21/09/16 19:06, Wei Liu wrote: > On Wed, Sep 21, 2016 at 02:00:50PM -0400, Boris Ostrovsky wrote: >> On 09/21/2016 01:36 PM, Wei Liu wrote: >>> On Wed, Sep 21, 2016 at 12:42:51PM -0400, Boris Ostrovsky wrote: >>>> On 09/21/2016 12:13 PM, Ian Jackson wrote: >>>>> Platform Team regression test user writes ("[xen-4.5-testing baseline-only test] 67737: regressions - FAIL"): >>>>>> test-xtf-amd64-amd64-1 19 xtf/test-hvm32-invlpg~shadow fail REGR. vs. 67706 >>>>>> test-xtf-amd64-amd64-1 26 xtf/test-hvm32pae-invlpg~shadow fail REGR. vs. 67706 >>>>> Several of these, 32bit and 64bit HVM. This is in the Citrix >>>>> Cambridge osstest instance. The Xen Project colo instance is >>>>> unaffected (flight 101045 there passed with the same revisions of >>>>> everything) >>>>> >>>>> This is with: >>>>> >>>>> xen e4ae4b03d35babc9624b7286f1ea4c6749bad84b >>>>> xtf b5c5332de4268d33a6f8eadc1d17c7b9cf0e7dc3 >>>>> linux b65f2f457c49b2cfd7967c34b7a0b04c25587f13 >>>>> linux-firmware c530a75c1e6a472b0eb9558310b518f0dfcd8860 >>>> I can't get these commits neither for Xen nor for Linux. Are these from >>>> Citrix trees? >>> No. They are all upstream commits. >> Apparently xen commit *just* made it to the tree, after I checked it. > Yes, it passed in Mass COLO. We suspected it can be related to > generations of AMD cpus (hence the "old" in email title). It will be because of Gen1 SVM which doesn't have NRIP support. This case requires emulation of the invlpg instruction, rather than just using the information provided by the intercept. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-21 18:20 ` Andrew Cooper @ 2016-09-22 9:29 ` Ian Jackson 2016-09-26 16:56 ` Ian Jackson 0 siblings, 1 reply; 12+ messages in thread From: Ian Jackson @ 2016-09-22 9:29 UTC (permalink / raw) To: Andrew Cooper Cc: xen-devel, Wei Liu, Suravee Suthikulpanit, Tim Deegan, Aravind Gopalakrishnan, Jan Beulich, Boris Ostrovsky Andrew Cooper writes ("Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ?"): > It will be because of Gen1 SVM which doesn't have NRIP support. This > case requires emulation of the invlpg instruction, rather than just > using the information provided by the intercept. So it seems that the xtf test is not effective at detecting the Xen bug except on old hardware ? Is there some way it could be improved ? It's obviously not desirable that we should have tests which pass in the production colo and fail in the ancient Citrix Cambridge instance. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-22 9:29 ` Ian Jackson @ 2016-09-26 16:56 ` Ian Jackson 0 siblings, 0 replies; 12+ messages in thread From: Ian Jackson @ 2016-09-26 16:56 UTC (permalink / raw) To: Andrew Cooper, Wei Liu, Boris Ostrovsky, Suravee Suthikulpanit, Aravind Gopalakrishnan, Jan Beulich, Tim Deegan, xen-devel Ian Jackson writes ("Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ?"): > Andrew Cooper writes ("Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ?"): > > It will be because of Gen1 SVM which doesn't have NRIP support. This > > case requires emulation of the invlpg instruction, rather than just > > using the information provided by the intercept. > > So it seems that the xtf test is not effective at detecting the Xen > bug except on old hardware ? Is there some way it could be improved ? > > It's obviously not desirable that we should have tests which pass in > the production colo and fail in the ancient Citrix Cambridge instance. Andrew and I discussed this IRL. I thought it worth writing down what was said so that we can refer to it later. This test failure is due to genuine bug(s) in Xen 4.5, in that it doesn't have various fixes (see the rest of the thread). The bugs are only exposed on old hardware, which uses different codepaths in Xen. On new hardware Xen takes a different approach. This is why the test failure appears in the Citrix (Cambridge) osstest but not in the Xen Project (Massachusetts) instance. Xen decides which approach to take based on hardware features. There is not currently any way to tell Xen not to use these hardware features (at least, not in this case - the AMD SVM NextRIP feature) if they are available. Andrew has a long-term plan to add more of such a facility - but that is not going to be available any time soon. In this particular case, the old hardware uses the Xen instruction emulator where newer hardware uses hardware support. (Andrew tells me that without NextRIP support, Xen must use the instruction emulator when handling `invlpg` instructions on behalf of the guest, to calculate how many bytes to move the instruction pointer forward by. And it is the emulator which has the bug here.) So FEP could be used to cause the bug to manifest even on new hardware and indeed where FEP is available, XTF does then use FEP to run exactly the same set of tests. However, FEP is not available in Xen 4.5 and there are good reasons for not backporting it there. It would be possible to backport the bugfixes to Xen 4.5. However, the bugs address only very rare problems. Andrew thinks the bugs are, insofar they are bugs which might cause lossage, more likely to bbe roughly "crashes obscure or very oddly-behaved guests" than "crashes commonly used guests but only with very low probability. The latter kind of bug would be worth a backport; the former much less so (especially in a very old stable release, and especially when the fixes involve behavioural changes). The fixes would also provide an unquantified performance improvement on AMD hardware, due to avoiding extraneous TLB flushes, but Andrew says he doubts that's worth caring about. We discussed host stickiness, host-specific bug detection, and regression detection, in osstest. I reassured Andrew that I think the current osstest algorithms will deal with this situation tolerably well (if not perfectly). The conclusion is that there is nothing to be done, at least in the short term. There are good reasons for the bug to persist in 4.5 and good reasons for it being hard to detect on newer hardware. Ian. (Thanks to Andrew for the IRL explanation and for review of this email.) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-21 16:13 ` Problem with Xen 4.5 failing XTF tests on old AMD cpus ? Ian Jackson 2016-09-21 16:15 ` Ian Jackson 2016-09-21 16:42 ` Boris Ostrovsky @ 2016-09-21 17:27 ` Andrew Cooper 2016-09-22 9:49 ` Ian Jackson 2 siblings, 1 reply; 12+ messages in thread From: Andrew Cooper @ 2016-09-21 17:27 UTC (permalink / raw) To: Ian Jackson, Boris Ostrovsky, Suravee Suthikulpanit, Aravind Gopalakrishnan Cc: Wei Liu, xen-devel, Tim Deegan, Jan Beulich On 21/09/16 17:13, Ian Jackson wrote: > Platform Team regression test user writes ("[xen-4.5-testing baseline-only test] 67737: regressions - FAIL"): >> test-xtf-amd64-amd64-1 19 xtf/test-hvm32-invlpg~shadow fail REGR. vs. 67706 >> test-xtf-amd64-amd64-1 26 xtf/test-hvm32pae-invlpg~shadow fail REGR. vs. 67706 > Several of these, 32bit and 64bit HVM. This is in the Citrix > Cambridge osstest instance. The Xen Project colo instance is > unaffected (flight 101045 there passed with the same revisions of > everything) > > This is with: > > xen e4ae4b03d35babc9624b7286f1ea4c6749bad84b > xtf b5c5332de4268d33a6f8eadc1d17c7b9cf0e7dc3 > linux b65f2f457c49b2cfd7967c34b7a0b04c25587f13 > linux-firmware c530a75c1e6a472b0eb9558310b518f0dfcd8860 > > The log says this: > > 2016-09-21 06:07:36 Z ---------- substep 19 xtf/test-hvm32-invlpg~shadow running ---------- > 2016-09-21 06:07:36 Z executing ssh ... root@10.80.228.78 /home/xtf/xtf-runner -m logfile test-hvm32-invlpg~shadow 1>&2; echo $? > > Using logfile '/var/log/xen/console/guest-test-hvm32-invlpg~shadow.log' > Executing 'xl create -F tests/invlpg/test-hvm32-invlpg~shadow.cfg' > --- Xen Test Framework --- > Environment: HVM 32bit (No paging) > Testing 'invlpg' in normally-faulting conditions > Test: Mapped address > Test: Unmapped address > Test: NULL segment override > Test: Past segment limit > Fail: Unexpected #GP[0000] > Test: Before expand-down segment limit > Fail: Unexpected #GP[0000] > Test result: FAILURE > > Combined test results: > test-hvm32-invlpg~shadow FAILURE > > Sadly we haven't yet managed to make the logs from this instance > public. > > Do you have any idea what might be causing this ? Is there a real > problem with the Xen 4.5 branch ? The Citrix Cambridge instance has > old hardware. This is the 4.5 tree missing some fixes: * 31d961f - x86/hvm: Fix invalidation for emulated invlpg instructions (4 months ago) <Andrew Cooper> * eee511d - x86/svm: Don't unconditionally use a new ASID in svm_invlpg_intercept() (4 months ago) <Andrew Cooper> * a373db2 - x86/hvm: Correct the emulated interaction of invlpg with segments (4 months ago) <Andrew Cooper> * a94b35d - x86/hvm: Raise #SS faults for %ss-based segmentation violations (4 months ago) <Andrew Cooper> * 6093515 - x86/hvm: Always return the linear address from hvm_virtual_to_linear_addr() (4 months ago) <Andrew Cooper> In this case, XTF is complaining that `invlpg`, as used in a shadow guest, is not behaving in the architecturally specified way. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ? 2016-09-21 17:27 ` Andrew Cooper @ 2016-09-22 9:49 ` Ian Jackson 0 siblings, 0 replies; 12+ messages in thread From: Ian Jackson @ 2016-09-22 9:49 UTC (permalink / raw) To: Andrew Cooper Cc: xen-devel, Wei Liu, Suravee Suthikulpanit, Tim Deegan, Aravind Gopalakrishnan, Jan Beulich, Boris Ostrovsky Andrew Cooper writes ("Re: Problem with Xen 4.5 failing XTF tests on old AMD cpus ?"): > This is the 4.5 tree missing some fixes: > > * 31d961f - x86/hvm: Fix invalidation for emulated invlpg instructions > (4 months ago) <Andrew Cooper> > * eee511d - x86/svm: Don't unconditionally use a new ASID in > svm_invlpg_intercept() (4 months ago) <Andrew Cooper> > * a373db2 - x86/hvm: Correct the emulated interaction of invlpg with > segments (4 months ago) <Andrew Cooper> > * a94b35d - x86/hvm: Raise #SS faults for %ss-based segmentation > violations (4 months ago) <Andrew Cooper> > * 6093515 - x86/hvm: Always return the linear address from > hvm_virtual_to_linear_addr() (4 months ago) <Andrew Cooper> > > In this case, XTF is complaining that `invlpg`, as used in a shadow > guest, is not behaving in the architecturally specified way. I think it is probably not sensible to backport these kind of changes (and especially not to 4.5). Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2016-09-26 16:56 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-09-21 14:09 [xen-4.5-testing baseline-only test] 67737: regressions - FAIL Platform Team regression test user 2016-09-21 16:13 ` Problem with Xen 4.5 failing XTF tests on old AMD cpus ? Ian Jackson 2016-09-21 16:15 ` Ian Jackson 2016-09-21 16:42 ` Boris Ostrovsky 2016-09-21 17:36 ` Wei Liu 2016-09-21 18:00 ` Boris Ostrovsky 2016-09-21 18:06 ` Wei Liu 2016-09-21 18:20 ` Andrew Cooper 2016-09-22 9:29 ` Ian Jackson 2016-09-26 16:56 ` Ian Jackson 2016-09-21 17:27 ` Andrew Cooper 2016-09-22 9:49 ` Ian Jackson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).