* [xen-unstable test] 123379: regressions - FAIL
@ 2018-05-31 6:00 osstest service owner
2018-05-31 8:32 ` Juergen Gross
0 siblings, 1 reply; 22+ messages in thread
From: osstest service owner @ 2018-05-31 6:00 UTC (permalink / raw)
To: xen-devel, osstest-admin
flight 123379 xen-unstable real [real]
http://logs.test-lab.xenproject.org/osstest/logs/123379/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2 fail REGR. vs. 123323
test-armhf-armhf-xl-arndale 5 host-ping-check-native fail REGR. vs. 123323
Tests which did not succeed, but are not blocking:
test-amd64-amd64-xl-qemut-win7-amd64 17 guest-stop fail like 123323
test-amd64-amd64-xl-qemuu-win7-amd64 17 guest-stop fail like 123323
test-armhf-armhf-libvirt 14 saverestore-support-check fail like 123323
test-armhf-armhf-libvirt-xsm 14 saverestore-support-check fail like 123323
test-amd64-i386-xl-qemuu-win7-amd64 17 guest-stop fail like 123323
test-amd64-i386-xl-qemut-win7-amd64 17 guest-stop fail like 123323
test-amd64-i386-xl-qemuu-ws16-amd64 17 guest-stop fail like 123323
test-armhf-armhf-libvirt-raw 13 saverestore-support-check fail like 123323
test-amd64-amd64-xl-qemuu-ws16-amd64 17 guest-stop fail like 123323
test-amd64-amd64-xl-qemut-ws16-amd64 17 guest-stop fail like 123323
test-amd64-i386-xl-pvshim 12 guest-start fail never pass
test-amd64-amd64-libvirt 13 migrate-support-check fail never pass
test-amd64-i386-libvirt 13 migrate-support-check fail never pass
test-arm64-arm64-xl-credit2 13 migrate-support-check fail never pass
test-arm64-arm64-xl-credit2 14 saverestore-support-check fail never pass
test-arm64-arm64-libvirt-xsm 13 migrate-support-check fail never pass
test-arm64-arm64-libvirt-xsm 14 saverestore-support-check fail never pass
test-amd64-i386-libvirt-xsm 13 migrate-support-check fail never pass
test-arm64-arm64-xl-xsm 13 migrate-support-check fail never pass
test-arm64-arm64-xl-xsm 14 saverestore-support-check fail never pass
test-amd64-amd64-libvirt-xsm 13 migrate-support-check fail never pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 11 migrate-support-check fail never pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 11 migrate-support-check fail never pass
test-amd64-amd64-qemuu-nested-amd 17 debian-hvm-install/l1/l2 fail never pass
test-amd64-amd64-libvirt-vhd 12 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 13 migrate-support-check fail never pass
test-armhf-armhf-xl-credit2 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-xsm 13 migrate-support-check fail never pass
test-armhf-armhf-xl-xsm 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-multivcpu 13 migrate-support-check fail never pass
test-armhf-armhf-xl-multivcpu 14 saverestore-support-check fail never pass
test-armhf-armhf-xl 13 migrate-support-check fail never pass
test-armhf-armhf-xl 14 saverestore-support-check fail never pass
test-armhf-armhf-xl-rtds 13 migrate-support-check fail never pass
test-armhf-armhf-xl-rtds 14 saverestore-support-check fail never pass
test-armhf-armhf-libvirt 13 migrate-support-check fail never pass
test-armhf-armhf-libvirt-xsm 13 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 13 migrate-support-check fail never pass
test-armhf-armhf-xl-cubietruck 14 saverestore-support-check fail never pass
test-arm64-arm64-xl 13 migrate-support-check fail never pass
test-arm64-arm64-xl 14 saverestore-support-check fail never pass
test-armhf-armhf-libvirt-raw 12 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 12 migrate-support-check fail never pass
test-armhf-armhf-xl-vhd 13 saverestore-support-check fail never pass
test-amd64-i386-xl-qemut-ws16-amd64 17 guest-stop fail never pass
test-amd64-i386-xl-qemuu-win10-i386 10 windows-install fail never pass
test-amd64-amd64-xl-qemuu-win10-i386 10 windows-install fail never pass
test-amd64-amd64-xl-qemut-win10-i386 10 windows-install fail never pass
test-amd64-i386-xl-qemut-win10-i386 10 windows-install fail never pass
version targeted for testing:
xen 06f542f8f2e446c01bd0edab51e9450af7f6e05b
baseline version:
xen fc5805daef091240cd5fc06634a8bcdb2f3bb843
Last test of basis 123323 2018-05-28 23:34:10 Z 2 days
Testing same since 123379 2018-05-29 21:42:20 Z 1 days 1 attempts
------------------------------------------------------------
People who touched revisions under test:
Andrew Cooper <andrew.cooper3@citrix.com>
Ian Jackson <Ian.Jackson@eu.citrix.com>
Jan Beulich <jbeulich@suse.com>
Juergen Gross <jgross@suse.com>
Lars Kurth <lars.kurth@citrix.com>
Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Tim Deegan <tim@xen.org>
Wei Liu <wei.liu2@citrix.com>
jobs:
build-amd64-xsm pass
build-arm64-xsm pass
build-armhf-xsm pass
build-i386-xsm pass
build-amd64-xtf pass
build-amd64 pass
build-arm64 pass
build-armhf pass
build-i386 pass
build-amd64-libvirt pass
build-arm64-libvirt pass
build-armhf-libvirt pass
build-i386-libvirt pass
build-amd64-prev pass
build-i386-prev pass
build-amd64-pvops pass
build-arm64-pvops pass
build-armhf-pvops pass
build-i386-pvops pass
build-amd64-rumprun pass
build-i386-rumprun pass
test-xtf-amd64-amd64-1 pass
test-xtf-amd64-amd64-2 pass
test-xtf-amd64-amd64-3 pass
test-xtf-amd64-amd64-4 pass
test-xtf-amd64-amd64-5 pass
test-amd64-amd64-xl pass
test-arm64-arm64-xl pass
test-armhf-armhf-xl pass
test-amd64-i386-xl pass
test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm fail
test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm pass
test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-i386-xl-qemut-stubdom-debianhvm-amd64-xsm pass
test-amd64-amd64-libvirt-xsm pass
test-arm64-arm64-libvirt-xsm pass
test-armhf-armhf-libvirt-xsm pass
test-amd64-i386-libvirt-xsm pass
test-amd64-amd64-xl-xsm pass
test-arm64-arm64-xl-xsm pass
test-armhf-armhf-xl-xsm pass
test-amd64-i386-xl-xsm pass
test-amd64-amd64-qemuu-nested-amd fail
test-amd64-amd64-xl-pvhv2-amd pass
test-amd64-i386-qemut-rhel6hvm-amd pass
test-amd64-i386-qemuu-rhel6hvm-amd pass
test-amd64-amd64-xl-qemut-debianhvm-amd64 pass
test-amd64-i386-xl-qemut-debianhvm-amd64 pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-xl-qemuu-debianhvm-amd64 pass
test-amd64-i386-freebsd10-amd64 pass
test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
test-amd64-i386-xl-qemuu-ovmf-amd64 pass
test-amd64-amd64-rumprun-amd64 pass
test-amd64-amd64-xl-qemut-win7-amd64 fail
test-amd64-i386-xl-qemut-win7-amd64 fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-i386-xl-qemuu-win7-amd64 fail
test-amd64-amd64-xl-qemut-ws16-amd64 fail
test-amd64-i386-xl-qemut-ws16-amd64 fail
test-amd64-amd64-xl-qemuu-ws16-amd64 fail
test-amd64-i386-xl-qemuu-ws16-amd64 fail
test-armhf-armhf-xl-arndale fail
test-amd64-amd64-xl-credit2 pass
test-arm64-arm64-xl-credit2 pass
test-armhf-armhf-xl-credit2 pass
test-armhf-armhf-xl-cubietruck pass
test-amd64-amd64-examine pass
test-arm64-arm64-examine pass
test-armhf-armhf-examine pass
test-amd64-i386-examine pass
test-amd64-i386-freebsd10-i386 pass
test-amd64-i386-rumprun-i386 pass
test-amd64-amd64-xl-qemut-win10-i386 fail
test-amd64-i386-xl-qemut-win10-i386 fail
test-amd64-amd64-xl-qemuu-win10-i386 fail
test-amd64-i386-xl-qemuu-win10-i386 fail
test-amd64-amd64-qemuu-nested-intel pass
test-amd64-amd64-xl-pvhv2-intel pass
test-amd64-i386-qemut-rhel6hvm-intel pass
test-amd64-i386-qemuu-rhel6hvm-intel pass
test-amd64-amd64-libvirt pass
test-armhf-armhf-libvirt pass
test-amd64-i386-libvirt pass
test-amd64-amd64-livepatch pass
test-amd64-i386-livepatch pass
test-amd64-amd64-migrupgrade pass
test-amd64-i386-migrupgrade pass
test-amd64-amd64-xl-multivcpu pass
test-armhf-armhf-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-amd64-amd64-libvirt-pair pass
test-amd64-i386-libvirt-pair pass
test-amd64-amd64-amd64-pvgrub pass
test-amd64-amd64-i386-pvgrub pass
test-amd64-amd64-xl-pvshim pass
test-amd64-i386-xl-pvshim fail
test-amd64-amd64-pygrub pass
test-amd64-amd64-xl-qcow2 pass
test-armhf-armhf-libvirt-raw pass
test-amd64-i386-xl-raw pass
test-amd64-amd64-xl-rtds pass
test-armhf-armhf-xl-rtds pass
test-amd64-amd64-xl-qemuu-debianhvm-amd64-shadow pass
test-amd64-i386-xl-qemuu-debianhvm-amd64-shadow pass
test-amd64-amd64-xl-shadow pass
test-amd64-i386-xl-shadow pass
test-amd64-amd64-libvirt-vhd pass
test-armhf-armhf-xl-vhd pass
------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images
Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs
Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
Not pushing.
------------------------------------------------------------
commit 06f542f8f2e446c01bd0edab51e9450af7f6e05b
Author: Jan Beulich <jbeulich@suse.com>
Date: Tue May 29 12:39:24 2018 +0200
x86/CPUID: don't override tool stack decision to hide STIBP
Other than in the feature sets, where we indeed want to offer the
feature even if not enumerated on hardware, we shouldn't dictate the
feature being available if tool stack or host admin have decided to not
expose it (for whatever [questionable?] reason). That feature set side
override is sufficient to achieve the intended guest side safety
property (in offering - by default - STIBP independent of actual
availability in hardware).
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
commit d6239f64713df819278bf048446d3187c6ac4734
Author: Jan Beulich <jbeulich@suse.com>
Date: Tue May 29 12:38:52 2018 +0200
x86: correct default_xen_spec_ctrl calculation
Even with opt_msr_sc_{pv,hvm} both false we should set up the variable
as usual, to ensure proper one-time setup during boot and CPU bringup.
This then also brings the code in line with the comment immediately
ahead of the printk() being modified saying "irrespective of guests".
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
commit b7eb9d8bd61ecdc399e8fc41ea4bbff35cbe0755
Author: Jan Beulich <jbeulich@suse.com>
Date: Tue May 29 12:38:09 2018 +0200
x86: suppress sync when XPTI is disabled for a domain
Now that we have a per-domain flag we can and should control sync-ing in
a more fine grained manner: Only domains having XPTI enabled need the
sync to occur.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
commit 849cc9ac56eff8a8d575ed9f484aad72f383862c
Author: Jan Beulich <JBeulich@suse.com>
Date: Tue May 22 05:40:02 2018 -0600
libxc/x86/PV: don't hand through CPUID leaf 0x80000008 as is
Just like for HVM the feature set should be used for EBX output, while
EAX should be restricted to the low 16 bits and ECX/EDX should be zero.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
commit 2de2b10b2252761baa5dd0077df384dbfcca8212
Author: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Date: Tue May 22 21:47:45 2018 +0200
tools/kdd: alternative way of muting spurious gcc warning
Older gcc does not support #pragma GCC diagnostics, so use alternative
approach - change variable type to uint32_t (this code handle 32-bit
requests only anyway), which apparently also avoid gcc complaining about
this (otherwise correct) code.
Fixes 437e00fea04becc91c1b6bc1c0baa636b067a5cc "tools/kdd: mute spurious
gcc warning"
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
Acked-by: Tim Deegan <tim@xen.org>
commit 09afb9e78e1e90ce77d5107677a8464e8410802b
Author: Ian Jackson <ian.jackson@eu.citrix.com>
Date: Wed Dec 13 11:58:00 2017 +0000
docs/process/xen-release-management: Lesson to learn
The 4.10 release preparation was significantly more hairy than ideal.
(We seem to have a good overall outcome despite, rather than because
of, our approach.)
This is the second time (at least) that we have come close to failure
by committing to a release date before the exact code to be released
is known and has been made and tested.
Evidently our docs makes it insufficiently clear not to do that.
CC: Julien Grall <julien.grall@arm.com>
Acked-by: Juergen Gross <jgross@suse.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Lars Kurth <lars.kurth@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
commit 480b5ebcc98810aa8bb670a28900a62d02a48cbc
Author: Ian Jackson <ian.jackson@eu.citrix.com>
Date: Tue May 22 17:39:52 2018 +0100
docs/process: Add RUBRIC
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Juergen Gross <jgross@suse.com>
commit 4712c0a231f010253a5471531e335a5a13dcec76
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date: Thu May 24 15:06:16 2018 +0100
x86/traps: Dump the instruction stream even for double faults
This helps debug #DF's which occur in alternative patches
Reported-by: George Dunlap <george.dunlap@eu.citrix.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
commit 6b9562dac1746014ab376bd2cf8ba400acf34c6d
Author: Jan Beulich <jbeulich@suse.com>
Date: Mon May 28 11:20:26 2018 +0200
x86/XPTI: fix S3 resume (and CPU offlining in general)
We should index an L1 table with an L1 index.
Reported-by: Simon Gaiser <simon@invisiblethingslab.com>
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
(qemu changes not included)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-05-31 6:00 [xen-unstable test] 123379: regressions - FAIL osstest service owner
@ 2018-05-31 8:32 ` Juergen Gross
2018-05-31 9:14 ` Juergen Gross
0 siblings, 1 reply; 22+ messages in thread
From: Juergen Gross @ 2018-05-31 8:32 UTC (permalink / raw)
To: xen-devel
On 31/05/18 08:00, osstest service owner wrote:
> flight 123379 xen-unstable real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/123379/
>
> Regressions :-(
>
> Tests which did not succeed and are blocking,
> including tests which could not be run:
> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2 fail REGR. vs. 123323
AFAICS this seems to be the suspected Windows reboot again?
> test-armhf-armhf-xl-arndale 5 host-ping-check-native fail REGR. vs. 123323
Flaky hardware again?
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-05-31 8:32 ` Juergen Gross
@ 2018-05-31 9:14 ` Juergen Gross
2018-06-01 8:10 ` Jan Beulich
0 siblings, 1 reply; 22+ messages in thread
From: Juergen Gross @ 2018-05-31 9:14 UTC (permalink / raw)
To: xen-devel
On 31/05/18 10:32, Juergen Gross wrote:
> On 31/05/18 08:00, osstest service owner wrote:
>> flight 123379 xen-unstable real [real]
>> http://logs.test-lab.xenproject.org/osstest/logs/123379/
>>
>> Regressions :-(
>>
>> Tests which did not succeed and are blocking,
>> including tests which could not be run:
>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2 fail REGR. vs. 123323
>
> AFAICS this seems to be the suspected Windows reboot again?
Hmm, thinking more about it: xl save is done with the domU paused,
so the guest rebooting concurrently is rather improbable.
As this is an issue occurring sporadically not only during 4.11
development phase I don't think this should be a blocker.
Thoughts?
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-05-31 9:14 ` Juergen Gross
@ 2018-06-01 8:10 ` Jan Beulich
2018-06-01 9:08 ` Juergen Gross
0 siblings, 1 reply; 22+ messages in thread
From: Jan Beulich @ 2018-06-01 8:10 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel
>>> On 31.05.18 at 11:14, <jgross@suse.com> wrote:
> On 31/05/18 10:32, Juergen Gross wrote:
>> On 31/05/18 08:00, osstest service owner wrote:
>>> flight 123379 xen-unstable real [real]
>>> http://logs.test-lab.xenproject.org/osstest/logs/123379/
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking,
>>> including tests which could not be run:
>>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
> fail REGR. vs. 123323
>>
>> AFAICS this seems to be the suspected Windows reboot again?
>
> Hmm, thinking more about it: xl save is done with the domU paused,
> so the guest rebooting concurrently is rather improbable.
Not sure, considering e.g.
libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
When looking into the Windows reboot issue (note this we're not dealing
with Windows here), I had noticed that there was a problem with trying
to save the guest at the "wrong" time. Generally, as explained back then,
I think the tool stack should honor the guest trying to reboot when it is
already in the process of being migrated/saved, and migration/save
should not even be attempted when the guest has already signaled
reboot (iirc it's only the former that is an actual issue). Otherwise the
tool stack will internally try to drive the same guest into two distinct new
states at the same time. Giving reboot (or shutdown) higher priority than
migration/save seems natural to me: A rebooting guest can be moved to
the new host with no migration cost at all, and a shut down guest doesn't
need (live) moving in the first place.
> As this is an issue occurring sporadically not only during 4.11
> development phase I don't think this should be a blocker.
Yes and no: Yes, it's not a regression. But as long as we don't make this
a blocker, I don't think the issue will be addressed, considering for how
long it has been there already.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-01 8:10 ` Jan Beulich
@ 2018-06-01 9:08 ` Juergen Gross
2018-06-05 16:16 ` Ian Jackson
` (2 more replies)
0 siblings, 3 replies; 22+ messages in thread
From: Juergen Gross @ 2018-06-01 9:08 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel, Wei Liu, Ian Jackson
On 01/06/18 10:10, Jan Beulich wrote:
>>>> On 31.05.18 at 11:14, <jgross@suse.com> wrote:
>> On 31/05/18 10:32, Juergen Gross wrote:
>>> On 31/05/18 08:00, osstest service owner wrote:
>>>> flight 123379 xen-unstable real [real]
>>>> http://logs.test-lab.xenproject.org/osstest/logs/123379/
>>>>
>>>> Regressions :-(
>>>>
>>>> Tests which did not succeed and are blocking,
>>>> including tests which could not be run:
>>>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
>> fail REGR. vs. 123323
>>>
>>> AFAICS this seems to be the suspected Windows reboot again?
>>
>> Hmm, thinking more about it: xl save is done with the domU paused,
>> so the guest rebooting concurrently is rather improbable.
>
> Not sure, considering e.g.
>
> libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
That was at 2018-05-30 22:12:49.650+0000
Before that there was:
2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14
= Bad address): Internal error
But looking at the messages issued some seconds before that I see some
xenstore watch related messages in:
http://logs.test-lab.xenproject.org/osstest/logs/123379/test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm/huxelrebe1---var-log-libvirt-libxl-libxl-driver.log
which make me wonder whether the libxl watch handling is really
correct: e.g. libxl__ev_xswatch_register() first registers the watch
with xenstore and only then writes the data needed for processing the
watch in the related structure. Could it be that the real suspend watch
event was interpreted as a @releaseDomain event?
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-01 9:08 ` Juergen Gross
@ 2018-06-05 16:16 ` Ian Jackson
2018-06-06 7:39 ` Juergen Gross
2018-06-05 16:19 ` Ian Jackson
2018-06-08 14:25 ` Ad-hoc test instructions (was Re: [xen-unstable test] 123379: regressions - FAIL) Ian Jackson
2 siblings, 1 reply; 22+ messages in thread
From: Ian Jackson @ 2018-06-05 16:16 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel, Wei Liu, Jan Beulich
Juergen Gross writes ("Re: [Xen-devel] [xen-unstable test] 123379: regressions - FAIL"):
> Before that there was:
>
> 2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14
> = Bad address): Internal error
This seems to be the only message about the root cause.
> But looking at the messages issued some seconds before that I see some
> xenstore watch related messages in:
>
> http://logs.test-lab.xenproject.org/osstest/logs/123379/test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm/huxelrebe1---var-log-libvirt-libxl-libxl-driver.log
I think this is all a red herring.
What I see happening is:
2018-05-30 22:12:44.695+0000: libxl: libxl_event.c:636:libxl__ev_xswatch_register: watch w=0xb40005e8 wpath=/local/domain/3/control/shutdown token=2/b: register slotnum=2
libxl starts watching the domain's shutdown control node. I think
this is done from near libxl_dom_suspend.c:202.
2018-05-30 22:12:44.696+0000: libxl: libxl_event.c:573:watchfd_callback: watch w=0xb40005e8 wpath=/local/domain/3/control/shutdown token=2/b: event epath=/local/domain/3/control/shutdown
The watch we just set triggers. This happens with every xenstore
watch, after it is set up - ie, it does not mean that anything had
written the node.
2018-05-30 22:12:44.696+0000: libxl: libxl_event.c:673:libxl__ev_xswatch_deregister: watch w=0xb40005e8 wpath=/local/domain/3/control/shutdown token=2/b: deregister slotnum=2
libxl stops watching the domain's shutdown control node. This is
done, I think, by domain_suspend_common_pvcontrol_suspending
(libxl_dom_suspend.c:222).
We can conclude that
if (!rc && !domain_suspend_pvcontrol_acked(state))
was not taken. It seems unlikely that rc!=0, because the
node is read in xswait_xswatch_callback using libxl__xs_read_checked
which I think would log a message. So probably
/local/domain/3/control/shutdown was `suspend', meaning the domain had
indeed acked the suspend request.
2018-05-30 22:12:44.696+0000: libxl: libxl_event.c:636:libxl__ev_xswatch_register: watch w=0xb40005f8 wpath=@releaseDomain token=2/c: register slotnum=2
This is the watch registration in domain_suspend_common_wait_guest.
2018-05-30 22:12:44.696+0000: libxl: libxl_event.c:548:watchfd_callback: watch w=0xb40005f8 epath=/local/domain/3/control/shutdown token=2/b: counter != c
This is a watch event for the watch we set up at 2018-05-30
22:12:44.696+0000. You can tell because the token is the same. But
that watch was cancelled within libxl at 2018-05-30
22:12:44.696+0000. libxl's watch handling machinery knows this, and
discards the event. "counter != c", libxl_event.c:547.
It does indeed use the same slot in the libxl xswatch data structure,
but libxl can distinguish it by the counter and the event path. (In
any case xs watch handlers should tolerate spurious events and be
idempotent, although that does not matter here.)
I think this must be the watch event from the guest actually writing
its acknowledgement to the control node - we would indeed expect two
such events, one generated by the watch setup, and one from the
guest's write. The timing meant that here we processed the guest's
written value as a result of the first watch event. This is fine.
2018-05-30 22:12:44.696+0000: libxl: libxl_event.c:573:watchfd_callback: watch w=0xb40005f8 wpath=@releaseDomain token=2/c: event epath=@releaseDomain
This is the immediate-auto-firing of the @releaseDomain event set up
at 2018-05-30 22:12:44.696+0000. libxl's xswatch machinery looks this
up in slot 2 and finds that the counter and paths are right, so it
will dispatch it to suspend_common_wait_guest_watch which is a
frontend for suspend_common_wait_guest_check.
In the absence of log messages from that function we can conclude that
!(info.flags & XEN_DOMINF_shutdown)
ie the guest has not shut down yet.
2018-05-30 22:12:44.720+0000: libxl: libxl_event.c:573:watchfd_callback: watch w=0xb2a26708 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
This is a watch event which was set up much earlier at 2018-05-30
21:58:02.182+0000. The surrounding context there (references to
domain_death_xswatch_callback) makes it clear that this is pursuant to
libxl_evenable_domain_death - ie, libvirt asked libxl to monitor for
the death of the domain.
2018-05-30 22:12:44.724+0000: libxl: libxl_domain.c:816:domain_death_xswatch_callback: shutdown reporting
The output here is a bit perplexing. I don't understand how we can
have the message "shutdown reporting" without any previous message
"Exists shutdown_reported=%d" or "[evg=%p] nentries=%d rc=%d %ld..%ld"
both of which seem to precede the "shutdown reporting" message in
domain_death_xswatch_callback.
However, we can conclude that, at this point, libxl finds that
got->flags & XEN_DOMINF_shutdown
and it decides to inform libvirt that the domain has shut down,
by providing a DOMAIN_SHUTDOWN libxl event.
(This event is not passed to libvirt immediately yet because it lives
on either (a) a queue on this thread's stack, which will be processed
on return to libvirt, or (b) a queue associated with the CTX, whose
lock we hold. The callback to libvirt will be reported later.)
2018-05-30 22:12:44.724+0000: libxl: libxl_domain.c:771:domain_death_xswatch_callback: [evg=0] all reported
2018-05-30 22:12:44.724+0000: libxl: libxl_domain.c:833:domain_death_xswatch_callback: domain death search done
This is the end of the search ing domain_death_xswatch_callback for
domains which need to be reported. libvirt was listening only for one
domain.
2018-05-30 22:12:44.724+0000: libxl: libxl_event.c:573:watchfd_callback: watch w=0xb40005f8 wpath=@releaseDomain token=2/c: event epath=@releaseDomain
Another xs watch event for the same domain shutdown, because libxl had
set up two watches for it.
(These will probably have been written very quickly together into the
xs ring and/or socket, and handled within the loop in
libxl_event.c:watchfd_callback. So we have not yet released the CTX
lock or returned to libvirt: therefore, probably, the domain shutdown
event notification to libvirt is still queued with libxl. Indeed as
we will see, that occurs a bit later.)
This is another watch event from the registration in
domain_suspend_common_wait_guest and again it will call
suspend_common_wait_guest_check.
2018-05-30 22:12:44.724+0000: libxl: libxl_event.c:673:libxl__ev_xswatch_deregister: watch w=0xb40005f8 wpath=@releaseDomain token=2/c: deregister slotnum=2
This must logically be one of the libxl__ev_xswatch_deregister calls
in domain_suspend_common_guest_suspended / domain_suspend_common_done.
However, looking at suspend_common_wait_guest_check, reaching either
of those should have produced some kind of log message - either "guest
has suspended" (DEBUG) or an ERROR of some sort.
... oh I have just spotted this logfile ...
http://logs.test-lab.xenproject.org/osstest/logs/123379/test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm/huxelrebe1---var-log-libvirt-libxl-debianhvm.guest.osstest.log
... which contains the other half of the messages.
So we see there:
2018-05-30 22:12:44.724+0000: libxl: libxl_dom_suspend.c:350:suspend_common_wait_guest_check: Domain 3:guest has suspended
OK.
2018-05-30 22:12:44.731+0000: libxl: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0xb40005f8: deregister unregistered
This is the second call to libxl__ev_xswatch_deregister for the same
watch event. That tells us that the first call must have been in
domain_suspend_common_guest_suspended. So all is going well, and we
called domain_suspend_common_done, whose idempotent cleanup
deregisters the already-deregisterd watch.
2018-05-30 22:12:44.731+0000: libxl: libxl_event.c:1398:egc_run_callbacks: event 0xb40068f0 callback type=domain_shutdown
This is libxl calling libvirt to tell libvirt that the domain has shut
down. libvirt does not seem to respond.
2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
2018-05-30 22:12:49.483+0000: xc: Save failed (14 = Bad address): Internal error
2018-05-30 22:12:49.648+0000: libxl-save-helper: complete r=-1: Bad address
And this is the first thing that goes wrong. You can see similar
messages in the other logfile:
2018-05-30 22:12:49.650+0000: libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
All of these are reports of the same thing: xc_get_pfn_type_batch at
xc_sr_save.c:133 failed with EFAULT.
2018-05-30 22:12:49.650+0000: libxl: libxl.c:746:libxl__fd_flags_restore: fnctl F_SETFL of fd 33 to 0x8001
2018-05-30 22:12:49.650+0000: libxl: libxl_event.c:1869:libxl__ao_complete: ao 0xb4000478: complete, rc=-3
2018-05-30 22:12:49.650+0000: libxl: libxl_event.c:1838:libxl__ao__destroy: ao 0xb4000478: destroy
> which make me wonder whether the libxl watch handling is really
> correct: e.g. libxl__ev_xswatch_register() first registers the watch
> with xenstore and only then writes the data needed for processing the
> watch in the related structure. Could it be that the real suspend watch
> event was interpreted as a @releaseDomain event?
No. The code in libxl__ev_xswatch_register all runs with the libxl
CTX lock held so it cannot be interrupted in this way. As you see
above I have analysed the log and it is all operating correctly,
albeit rather noisily in the debug log.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-01 9:08 ` Juergen Gross
2018-06-05 16:16 ` Ian Jackson
@ 2018-06-05 16:19 ` Ian Jackson
2018-06-06 9:35 ` Jan Beulich
[not found] ` <5B17AAE102000078001C8972@suse.com>
2018-06-08 14:25 ` Ad-hoc test instructions (was Re: [xen-unstable test] 123379: regressions - FAIL) Ian Jackson
2 siblings, 2 replies; 22+ messages in thread
From: Ian Jackson @ 2018-06-05 16:19 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel, Wei Liu, Ian Jackson, Jan Beulich
>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
I thought I would reply again with the key point from my earlier mail
highlighted, and go a bit further. The first thing to go wrong in
this was:
2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
2018-05-30 22:12:49.483+0000: xc: Save failed (14 = Bad address): Internal error
2018-05-30 22:12:49.648+0000: libxl-save-helper: complete r=-1: Bad address
You can see similar messages in the other logfile:
2018-05-30 22:12:49.650+0000: libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
All of these are reports of the same thing: xc_get_pfn_type_batch at
xc_sr_save.c:133 failed with EFAULT. I'm afraid I don't know why.
There is no corresponding message in the host's serial log nor the
dom0 kernel log.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-05 16:16 ` Ian Jackson
@ 2018-06-06 7:39 ` Juergen Gross
0 siblings, 0 replies; 22+ messages in thread
From: Juergen Gross @ 2018-06-06 7:39 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel, Wei Liu, Jan Beulich
On 05/06/18 18:16, Ian Jackson wrote:
> 2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
This is worrying me.
The message is issued as a result of xc_get_pfn_type_batch() failing.
I see no other possibility for the failure with errno being 14 (EFAULT)
than the hypervisor failing a copy from/to guest for either struct
xen_domctl or the pfn array passed via struct xen_domctl (op
XEN_DOMCTL_getpageframeinfo3). Both should be accessible as they have
been correctly declared via DECLARE_HYPERCALL_BOUNCE() in xc_private.c.
Any ideas how that could have happened?
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-05 16:19 ` Ian Jackson
@ 2018-06-06 9:35 ` Jan Beulich
[not found] ` <5B17AAE102000078001C8972@suse.com>
1 sibling, 0 replies; 22+ messages in thread
From: Jan Beulich @ 2018-06-06 9:35 UTC (permalink / raw)
To: Ian Jackson, Juergen Gross; +Cc: Ian Jackson, Wei Liu, xen-devel
>>> On 05.06.18 at 18:19, <ian.jackson@citrix.com> wrote:
>> > test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
>
> I thought I would reply again with the key point from my earlier mail
> highlighted, and go a bit further. The first thing to go wrong in
> this was:
>
> 2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
> 2018-05-30 22:12:49.483+0000: xc: Save failed (14 = Bad address): Internal error
> 2018-05-30 22:12:49.648+0000: libxl-save-helper: complete r=-1: Bad address
>
> You can see similar messages in the other logfile:
>
> 2018-05-30 22:12:49.650+0000: libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
>
> All of these are reports of the same thing: xc_get_pfn_type_batch at
> xc_sr_save.c:133 failed with EFAULT. I'm afraid I don't know why.
>
> There is no corresponding message in the host's serial log nor the
> dom0 kernel log.
I vaguely recall from the time when I had looked at the similar Windows
migration issues that the guest is already in the process of being cleaned
up when these occur. Commit 2dbe9c3cd2 ("x86/mm: silence a pointless
warning") intentionally suppressed a log message here, and the
immediately following debugging code (933f966bcd x86/mm: add
temporary debugging code to get_page_from_gfn_p2m()) was reverted
a little over a month later. This wasn't as a follow-up to another patch
(fix), but following the discussion rooted at
https://lists.xenproject.org/archives/html/xen-devel/2017-06/msg00324.html
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
[not found] ` <5B17AAE102000078001C8972@suse.com>
@ 2018-06-06 9:40 ` Juergen Gross
2018-06-07 11:30 ` Juergen Gross
0 siblings, 1 reply; 22+ messages in thread
From: Juergen Gross @ 2018-06-06 9:40 UTC (permalink / raw)
To: Jan Beulich, Ian Jackson <ian.jackson@citrix.com>
Cc: Ian Jackson, Wei Liu, xen-devel
On 06/06/18 11:35, Jan Beulich wrote:
>>>> On 05.06.18 at 18:19, <ian.jackson@citrix.com> wrote:
>>>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
>>
>> I thought I would reply again with the key point from my earlier mail
>> highlighted, and go a bit further. The first thing to go wrong in
>> this was:
>>
>> 2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
>> 2018-05-30 22:12:49.483+0000: xc: Save failed (14 = Bad address): Internal error
>> 2018-05-30 22:12:49.648+0000: libxl-save-helper: complete r=-1: Bad address
>>
>> You can see similar messages in the other logfile:
>>
>> 2018-05-30 22:12:49.650+0000: libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
>>
>> All of these are reports of the same thing: xc_get_pfn_type_batch at
>> xc_sr_save.c:133 failed with EFAULT. I'm afraid I don't know why.
>>
>> There is no corresponding message in the host's serial log nor the
>> dom0 kernel log.
>
> I vaguely recall from the time when I had looked at the similar Windows
> migration issues that the guest is already in the process of being cleaned
> up when these occur. Commit 2dbe9c3cd2 ("x86/mm: silence a pointless
> warning") intentionally suppressed a log message here, and the
> immediately following debugging code (933f966bcd x86/mm: add
> temporary debugging code to get_page_from_gfn_p2m()) was reverted
> a little over a month later. This wasn't as a follow-up to another patch
> (fix), but following the discussion rooted at
> https://lists.xenproject.org/archives/html/xen-devel/2017-06/msg00324.html
That was -ESRCH, not -EFAULT.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-06 9:40 ` Juergen Gross
@ 2018-06-07 11:30 ` Juergen Gross
2018-06-08 10:12 ` Juergen Gross
0 siblings, 1 reply; 22+ messages in thread
From: Juergen Gross @ 2018-06-07 11:30 UTC (permalink / raw)
To: Jan Beulich, Ian Jackson <ian.jackson@citrix.com>
Cc: Ian Jackson, Wei Liu, xen-devel
On 06/06/18 11:40, Juergen Gross wrote:
> On 06/06/18 11:35, Jan Beulich wrote:
>>>>> On 05.06.18 at 18:19, <ian.jackson@citrix.com> wrote:
>>>>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
>>>
>>> I thought I would reply again with the key point from my earlier mail
>>> highlighted, and go a bit further. The first thing to go wrong in
>>> this was:
>>>
>>> 2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
>>> 2018-05-30 22:12:49.483+0000: xc: Save failed (14 = Bad address): Internal error
>>> 2018-05-30 22:12:49.648+0000: libxl-save-helper: complete r=-1: Bad address
>>>
>>> You can see similar messages in the other logfile:
>>>
>>> 2018-05-30 22:12:49.650+0000: libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
>>>
>>> All of these are reports of the same thing: xc_get_pfn_type_batch at
>>> xc_sr_save.c:133 failed with EFAULT. I'm afraid I don't know why.
>>>
>>> There is no corresponding message in the host's serial log nor the
>>> dom0 kernel log.
>>
>> I vaguely recall from the time when I had looked at the similar Windows
>> migration issues that the guest is already in the process of being cleaned
>> up when these occur. Commit 2dbe9c3cd2 ("x86/mm: silence a pointless
>> warning") intentionally suppressed a log message here, and the
>> immediately following debugging code (933f966bcd x86/mm: add
>> temporary debugging code to get_page_from_gfn_p2m()) was reverted
>> a little over a month later. This wasn't as a follow-up to another patch
>> (fix), but following the discussion rooted at
>> https://lists.xenproject.org/archives/html/xen-devel/2017-06/msg00324.html
>
> That was -ESRCH, not -EFAULT.
I've looked a little bit more into this.
As we are seeing EFAULT being returned by the hypervisor this either
means the tools are specifying an invalid address (quite unlikely)
or the buffers are not as MAP_LOCKED as we wish them to be.
Is there a way to see whether the host was experiencing some memory
shortage, so the buffers might have been swapped out?
man mmap tells me: "This implementation will try to populate (prefault)
the whole range but the mmap call doesn't fail with ENOMEM if this
fails. Therefore major faults might happen later on."
And: "One should use mmap(2) plus mlock(2) when major faults are not
acceptable after the initialization of the mapping."
With osdep_alloc_pages() in tools/libs/call/linux.c touching all the
hypercall buffer pages before doing the hypercall I'm not sure this
could be an issue.
Any thoughts on that?
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-07 11:30 ` Juergen Gross
@ 2018-06-08 10:12 ` Juergen Gross
2018-06-12 15:58 ` Juergen Gross
0 siblings, 1 reply; 22+ messages in thread
From: Juergen Gross @ 2018-06-08 10:12 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel, Wei Liu, Jan Beulich
On 07/06/18 13:30, Juergen Gross wrote:
> On 06/06/18 11:40, Juergen Gross wrote:
>> On 06/06/18 11:35, Jan Beulich wrote:
>>>>>> On 05.06.18 at 18:19, <ian.jackson@citrix.com> wrote:
>>>>>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
>>>>
>>>> I thought I would reply again with the key point from my earlier mail
>>>> highlighted, and go a bit further. The first thing to go wrong in
>>>> this was:
>>>>
>>>> 2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
>>>> 2018-05-30 22:12:49.483+0000: xc: Save failed (14 = Bad address): Internal error
>>>> 2018-05-30 22:12:49.648+0000: libxl-save-helper: complete r=-1: Bad address
>>>>
>>>> You can see similar messages in the other logfile:
>>>>
>>>> 2018-05-30 22:12:49.650+0000: libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
>>>>
>>>> All of these are reports of the same thing: xc_get_pfn_type_batch at
>>>> xc_sr_save.c:133 failed with EFAULT. I'm afraid I don't know why.
>>>>
>>>> There is no corresponding message in the host's serial log nor the
>>>> dom0 kernel log.
>>>
>>> I vaguely recall from the time when I had looked at the similar Windows
>>> migration issues that the guest is already in the process of being cleaned
>>> up when these occur. Commit 2dbe9c3cd2 ("x86/mm: silence a pointless
>>> warning") intentionally suppressed a log message here, and the
>>> immediately following debugging code (933f966bcd x86/mm: add
>>> temporary debugging code to get_page_from_gfn_p2m()) was reverted
>>> a little over a month later. This wasn't as a follow-up to another patch
>>> (fix), but following the discussion rooted at
>>> https://lists.xenproject.org/archives/html/xen-devel/2017-06/msg00324.html
>>
>> That was -ESRCH, not -EFAULT.
>
> I've looked a little bit more into this.
>
> As we are seeing EFAULT being returned by the hypervisor this either
> means the tools are specifying an invalid address (quite unlikely)
> or the buffers are not as MAP_LOCKED as we wish them to be.
>
> Is there a way to see whether the host was experiencing some memory
> shortage, so the buffers might have been swapped out?
>
> man mmap tells me: "This implementation will try to populate (prefault)
> the whole range but the mmap call doesn't fail with ENOMEM if this
> fails. Therefore major faults might happen later on."
>
> And: "One should use mmap(2) plus mlock(2) when major faults are not
> acceptable after the initialization of the mapping."
>
> With osdep_alloc_pages() in tools/libs/call/linux.c touching all the
> hypercall buffer pages before doing the hypercall I'm not sure this
> could be an issue.
>
> Any thoughts on that?
Ian, is there a chance to dedicate a machine to a specific test trying
to reproduce the problem? In case we manage to get this failure in a
reasonable time frame I guess the most promising approach would be to
use a test hypervisor producing more debug data. If you think this is
worth doing I can write a patch.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Ad-hoc test instructions (was Re: [xen-unstable test] 123379: regressions - FAIL)
2018-06-01 9:08 ` Juergen Gross
2018-06-05 16:16 ` Ian Jackson
2018-06-05 16:19 ` Ian Jackson
@ 2018-06-08 14:25 ` Ian Jackson
2018-06-08 15:42 ` Juergen Gross
2 siblings, 1 reply; 22+ messages in thread
From: Ian Jackson @ 2018-06-08 14:25 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel
Apropos of the irc conversation below, in particular my suggestion to
use mg-repro-setup. Probably, an appropriate rune is
./mg-repro-setup -f123855 -Ejgross@suse.com 123855 test-amd64-amd64-xl-qemuu-ovmf-amd64 guest-saverestore.2 host=alloc:'{equiv-albana}'
(You will have wanted to
git clone ~osstest/testing.git
cd testing
to make yourself a working tree. To run pieces of osstest, your
cwd should be the root of the osstest source tree.)
What mg-repro-setup will do is:
* Make a new flight consisting of one job
test-amd64-amd64-xl-qemuu-ovmf-amd64
whose recipe and parameters are copied from the corresponding job
in 123855 - and which will reuse builds from 123855 too.
* Try to allocate a host according to '{equiv-albana}', which is an
expression meaning "any host with the equiv-albana flag set", ie
one of the (two) albanas. This will have to wait for a slot,
but as a command line user you get a highish priority.
* Run the repro flight, including all the steps up to and including
the one with testid "guest-saverestore.2". This will wipe the
albana machine allocated above and install on it the same
versions of everything as used for the same job in 123855.
(You can also ask mg-repro-setup to wipe and reinstall an existing
host you have already allocated, or to reuse existing host and put
the relevant Xen and kernel on it but without wiping it. See the
usage message.)
* Email you with a report, comparing the results with 123855
After this, you will still have the relevant albana[01] machine
allocated and you may log into it etc. etc. If you want to connect to
its serial port,
ssh -vt serial2.test-lab sympathy -r albanaN
(from your own workstation)
You can power cycle it with
mg-hosts power albanaN reboot
Also you may use the flight constructed by ./mg-repro-setup for your
own ad-hoc-tests:
export OSSTEST_FLIGHT=<whatever number you were told by mg-repro-setup>
export OSSTEST_JOB=test-amd64-amd64-xl-qemuu-ovmf-amd64
./ts-guest-saverestore host=albanaN debian
When you are done, you must release the host manually
mg-allocate ^albananN
Please ask me on IRC if you have any questions. There are also docs
in the osstest tree. HTH.
Ian.
14:48 <juergen_gross> Okay. Already found some interesting samples, e.g. from
4.9 tests. All found up to now on different hosts
(silvana0, chardonnay0, huxelrebe1)
14:49 <juergen_gross> So the chances for some hardware specific bug are rather
slim, other than x86 :-)
14:55 <juergen_gross> Diziet: are most tests now running with dom0 4.14 kernel?
14:55 <Diziet> Basically all the x86 tests that aren't tests of some Linux
version.
14:56 <juergen_gross> Okay, so it is no strange coincidents all failures have
been with 4.14. :-) BTW: even 4.6 Xen has the same problem
14:58 <juergen_gross> Is it possible to run ts-guest-saverestore in a loop on a
machine to have an idea how long it takes to reproduce it?
14:58 <Diziet> Yes.
14:58 <juergen_gross> I'll write a hypervisor patch to get more diagnostics then
14:58 <Diziet> The easiest way is probably to use mg-repro-setup to find a
host, set everything up, repro the test up to that point, and
then run ts-repeat-test by hand in a shell
14:59 <Diziet> You could indeed also install a new hypervisor after
mg-repro-setup has done its thing
15:00 <juergen_gross> I guess the best would be if you could send me mail with
either the commands or a hint where to find the needed
info. I should do those tests myself to learn using them,
but I'd appreciate some help to start
15:00 <Diziet> Of course.
15:00 <juergen_gross> Thanks.
15:00 <Diziet> You mentioned having found some other occurrences.
15:00 <juergen_gross> Yes.
15:01 <Diziet> You might want to pick one that has a relatively simple guest
(and ideally not a Windows one)
15:01 <Diziet> Can you tell me the flight number and job name from that ?
15:01 <juergen_gross> Flight 123855 test-amd64-amd64-xl-qemuu-ovmf-amd64
15:02 <Diziet> t\z
15:02 <Diziet> ta I mean
...
15:09 <juergen_gross> Shows clearly the same problem
15:09 <Diziet> Indeed
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: Ad-hoc test instructions (was Re: [xen-unstable test] 123379: regressions - FAIL)
2018-06-08 14:25 ` Ad-hoc test instructions (was Re: [xen-unstable test] 123379: regressions - FAIL) Ian Jackson
@ 2018-06-08 15:42 ` Juergen Gross
0 siblings, 0 replies; 22+ messages in thread
From: Juergen Gross @ 2018-06-08 15:42 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel
On 08/06/18 16:25, Ian Jackson wrote:
> Apropos of the irc conversation below, in particular my suggestion to
> use mg-repro-setup. Probably, an appropriate rune is
>
> ./mg-repro-setup -f123855 -Ejgross@suse.com 123855 test-amd64-amd64-xl-qemuu-ovmf-amd64 guest-saverestore.2 host=alloc:'{equiv-albana}'
>
> (You will have wanted to
> git clone ~osstest/testing.git
> cd testing
> to make yourself a working tree. To run pieces of osstest, your
> cwd should be the root of the osstest source tree.)
./mg-repro-setup should do a "mkdir -p tmp":
./mg-repro-setup -f123855 -Ejgross@suse.com 123855
test-amd64-amd64-xl-qemuu-ovmf-amd64 guest-saverestore.2
host=alloc:'{equiv-albana}'
logging to tmp/mg-repro-setup.log
touch: cannot touch 'tmp/mg-repro-setup.log': No such file or directory
savelog: could not touch tmp/mg-repro-setup.log
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [xen-unstable test] 123379: regressions - FAIL
2018-06-08 10:12 ` Juergen Gross
@ 2018-06-12 15:58 ` Juergen Gross
2018-06-13 6:11 ` Jan Beulich
` (2 more replies)
0 siblings, 3 replies; 22+ messages in thread
From: Juergen Gross @ 2018-06-12 15:58 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel, Wei Liu, Jan Beulich
On 08/06/18 12:12, Juergen Gross wrote:
> On 07/06/18 13:30, Juergen Gross wrote:
>> On 06/06/18 11:40, Juergen Gross wrote:
>>> On 06/06/18 11:35, Jan Beulich wrote:
>>>>>>> On 05.06.18 at 18:19, <ian.jackson@citrix.com> wrote:
>>>>>>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
>>>>>
>>>>> I thought I would reply again with the key point from my earlier mail
>>>>> highlighted, and go a bit further. The first thing to go wrong in
>>>>> this was:
>>>>>
>>>>> 2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
>>>>> 2018-05-30 22:12:49.483+0000: xc: Save failed (14 = Bad address): Internal error
>>>>> 2018-05-30 22:12:49.648+0000: libxl-save-helper: complete r=-1: Bad address
>>>>>
>>>>> You can see similar messages in the other logfile:
>>>>>
>>>>> 2018-05-30 22:12:49.650+0000: libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
>>>>>
>>>>> All of these are reports of the same thing: xc_get_pfn_type_batch at
>>>>> xc_sr_save.c:133 failed with EFAULT. I'm afraid I don't know why.
>>>>>
>>>>> There is no corresponding message in the host's serial log nor the
>>>>> dom0 kernel log.
>>>>
>>>> I vaguely recall from the time when I had looked at the similar Windows
>>>> migration issues that the guest is already in the process of being cleaned
>>>> up when these occur. Commit 2dbe9c3cd2 ("x86/mm: silence a pointless
>>>> warning") intentionally suppressed a log message here, and the
>>>> immediately following debugging code (933f966bcd x86/mm: add
>>>> temporary debugging code to get_page_from_gfn_p2m()) was reverted
>>>> a little over a month later. This wasn't as a follow-up to another patch
>>>> (fix), but following the discussion rooted at
>>>> https://lists.xenproject.org/archives/html/xen-devel/2017-06/msg00324.html
>>>
>>> That was -ESRCH, not -EFAULT.
>>
>> I've looked a little bit more into this.
>>
>> As we are seeing EFAULT being returned by the hypervisor this either
>> means the tools are specifying an invalid address (quite unlikely)
>> or the buffers are not as MAP_LOCKED as we wish them to be.
>>
>> Is there a way to see whether the host was experiencing some memory
>> shortage, so the buffers might have been swapped out?
>>
>> man mmap tells me: "This implementation will try to populate (prefault)
>> the whole range but the mmap call doesn't fail with ENOMEM if this
>> fails. Therefore major faults might happen later on."
>>
>> And: "One should use mmap(2) plus mlock(2) when major faults are not
>> acceptable after the initialization of the mapping."
>>
>> With osdep_alloc_pages() in tools/libs/call/linux.c touching all the
>> hypercall buffer pages before doing the hypercall I'm not sure this
>> could be an issue.
>>
>> Any thoughts on that?
>
> Ian, is there a chance to dedicate a machine to a specific test trying
> to reproduce the problem? In case we manage to get this failure in a
> reasonable time frame I guess the most promising approach would be to
> use a test hypervisor producing more debug data. If you think this is
> worth doing I can write a patch.
Trying to reproduce the problem in a limited test environment finally
worked: doing a loop of "xl save -c" produced the problem after 198
iterations.
I have asked a SUSE engineer doing kernel memory management if he
could think of something. His idea is that maybe some kthread could be
the reason for our problem, e.g. trying page migration or compaction
(at least on the test machine I've looked at compaction of mlocked
pages is allowed: /proc/sys/vm/compact_unevictable_allowed is 1).
In order to be really sure nothing in the kernel can temporarily
switch hypercall buffer pages read-only or invalid for the hypervisor
we'll have to modify the privcmd driver interface: it will have to
gain knowledge which pages are handed over to the hypervisor as buffers
in order to be able to lock them accordingly via get_user_pages().
While this is a possible explanation of the fault we are seeing it might
be related to another reason. So I'm going to apply some modifications
to the hypervisor to get some more diagnostics in order to verify the
suspected kernel behavior is really the reason for the hypervisor to
return EFAULT.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-12 15:58 ` Juergen Gross
@ 2018-06-13 6:11 ` Jan Beulich
[not found] ` <5B20B5A602000078001CAACA@suse.com>
2018-06-13 8:52 ` Juergen Gross
2 siblings, 0 replies; 22+ messages in thread
From: Jan Beulich @ 2018-06-13 6:11 UTC (permalink / raw)
To: Juergen Gross; +Cc: Ian Jackson, Wei Liu, xen-devel
>>> On 12.06.18 at 17:58, <jgross@suse.com> wrote:
> Trying to reproduce the problem in a limited test environment finally
> worked: doing a loop of "xl save -c" produced the problem after 198
> iterations.
>
> I have asked a SUSE engineer doing kernel memory management if he
> could think of something. His idea is that maybe some kthread could be
> the reason for our problem, e.g. trying page migration or compaction
> (at least on the test machine I've looked at compaction of mlocked
> pages is allowed: /proc/sys/vm/compact_unevictable_allowed is 1).
Iirc the primary goal of compaction is to make contiguous memory
available for huge page allocations. PV not using huge pages, this is
of no interest here. The secondary consideration of physically
contiguous I/O buffer is an illusion only under PV, so perhaps not
much more of an interest (albeit I can see drivers wanting to allocate
physically contiguous buffers nevertheless now and then, but I'd
expect this to be mostly limited to driver initialization and device hot
add).
So it is perhaps at least worth considering whether to turn off
compaction/migration when running PV. But the problem would still
need addressing then mid-term, as PVH Dom0 would have the same
issue (and of course DomU, i.e. including HVM, can make hypercalls
too, and hence would be affected as well, just perhaps not as
visibly).
> In order to be really sure nothing in the kernel can temporarily
> switch hypercall buffer pages read-only or invalid for the hypervisor
> we'll have to modify the privcmd driver interface: it will have to
> gain knowledge which pages are handed over to the hypervisor as buffers
> in order to be able to lock them accordingly via get_user_pages().
So are you / is he saying that mlock() doesn't protect against such
playing with process memory? Teaching the privcmd driver of all
the indirections in hypercall request structures doesn't look very
attractive (or maintainable). Or are you thinking of the caller
providing sideband information describing the buffers involved,
perhaps along the lines of how dm_op was designed?
There's another option, but that has potentially severe drawbacks
too: Instead of returning -EFAULT on buffer access issues, we
could raise #PF on the very hypercall insn. Maybe something to
consider as an opt-in for PV/PVH, and as default for HVM.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
[not found] ` <5B20B5A602000078001CAACA@suse.com>
@ 2018-06-13 6:50 ` Juergen Gross
2018-06-13 7:21 ` Jan Beulich
[not found] ` <5B20C5E002000078001CAB80@suse.com>
0 siblings, 2 replies; 22+ messages in thread
From: Juergen Gross @ 2018-06-13 6:50 UTC (permalink / raw)
To: Jan Beulich; +Cc: Ian Jackson, Wei Liu, xen-devel
On 13/06/18 08:11, Jan Beulich wrote:
>>>> On 12.06.18 at 17:58, <jgross@suse.com> wrote:
>> Trying to reproduce the problem in a limited test environment finally
>> worked: doing a loop of "xl save -c" produced the problem after 198
>> iterations.
>>
>> I have asked a SUSE engineer doing kernel memory management if he
>> could think of something. His idea is that maybe some kthread could be
>> the reason for our problem, e.g. trying page migration or compaction
>> (at least on the test machine I've looked at compaction of mlocked
>> pages is allowed: /proc/sys/vm/compact_unevictable_allowed is 1).
>
> Iirc the primary goal of compaction is to make contiguous memory
> available for huge page allocations. PV not using huge pages, this is
> of no interest here. The secondary consideration of physically
> contiguous I/O buffer is an illusion only under PV, so perhaps not
> much more of an interest (albeit I can see drivers wanting to allocate
> physically contiguous buffers nevertheless now and then, but I'd
> expect this to be mostly limited to driver initialization and device hot
> add).
>
> So it is perhaps at least worth considering whether to turn off
> compaction/migration when running PV. But the problem would still
> need addressing then mid-term, as PVH Dom0 would have the same
> issue (and of course DomU, i.e. including HVM, can make hypercalls
> too, and hence would be affected as well, just perhaps not as
> visibly).
I think we should try to solve the problem by being aware of such
possibilities. Another potential source would be NUMA memory
migration (not now in pv, of course). And who knows what will come
in the next years.
>
>> In order to be really sure nothing in the kernel can temporarily
>> switch hypercall buffer pages read-only or invalid for the hypervisor
>> we'll have to modify the privcmd driver interface: it will have to
>> gain knowledge which pages are handed over to the hypervisor as buffers
>> in order to be able to lock them accordingly via get_user_pages().
>
> So are you / is he saying that mlock() doesn't protect against such
> playing with process memory?
Right. Due to proper locking in the kernel this is just a guarantee you
won't ever see a fault for such a page in user mode.
> Teaching the privcmd driver of all
> the indirections in hypercall request structures doesn't look very
> attractive (or maintainable). Or are you thinking of the caller
> providing sideband information describing the buffers involved,
> perhaps along the lines of how dm_op was designed?
I thought about that, yes. libxencall already has all the needed data
for that. Another possibility would be a dedicated ioctl for registering
a hypercall buffer (or some of them).
> There's another option, but that has potentially severe drawbacks
> too: Instead of returning -EFAULT on buffer access issues, we
> could raise #PF on the very hypercall insn. Maybe something to
> consider as an opt-in for PV/PVH, and as default for HVM.
Hmm, I'm not sure this will solve any problem. I'm not aware that it
is considered good practice to just access a user buffer from kernel
without using copyin()/copyout() when you haven't locked the page(s)
via get_user_pages(), even if the buffer was mlock()ed. Returning
-EFAULT is the right thing to do, I believe.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-13 6:50 ` Juergen Gross
@ 2018-06-13 7:21 ` Jan Beulich
[not found] ` <5B20C5E002000078001CAB80@suse.com>
1 sibling, 0 replies; 22+ messages in thread
From: Jan Beulich @ 2018-06-13 7:21 UTC (permalink / raw)
To: Juergen Gross; +Cc: Ian Jackson, Wei Liu, xen-devel
>>> On 13.06.18 at 08:50, <jgross@suse.com> wrote:
> On 13/06/18 08:11, Jan Beulich wrote:
>> Teaching the privcmd driver of all
>> the indirections in hypercall request structures doesn't look very
>> attractive (or maintainable). Or are you thinking of the caller
>> providing sideband information describing the buffers involved,
>> perhaps along the lines of how dm_op was designed?
>
> I thought about that, yes. libxencall already has all the needed data
> for that. Another possibility would be a dedicated ioctl for registering
> a hypercall buffer (or some of them).
I'm not sure that's an option: Is it legitimate (secure) to retain the
effects of get_user_pages() across system calls?
>> There's another option, but that has potentially severe drawbacks
>> too: Instead of returning -EFAULT on buffer access issues, we
>> could raise #PF on the very hypercall insn. Maybe something to
>> consider as an opt-in for PV/PVH, and as default for HVM.
>
> Hmm, I'm not sure this will solve any problem. I'm not aware that it
> is considered good practice to just access a user buffer from kernel
> without using copyin()/copyout() when you haven't locked the page(s)
> via get_user_pages(), even if the buffer was mlock()ed. Returning
> -EFAULT is the right thing to do, I believe.
But we're talking about the very copyin()/copyout(), just that here
it's being amortized by doing the operation just once (in the
hypervisor). A #PF would arise from syscall buffer copyin()/copyout(),
and the suggestion was to produce the same effect for the squashed
operation. Perhaps we wouldn't want #PF to come back from ordinary
(kernel invoked) hypercalls, but ones relayed by privcmd are different
in many ways anyway (see the stac()/clac() pair around the actual
call, for example).
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
[not found] ` <5B20C5E002000078001CAB80@suse.com>
@ 2018-06-13 7:57 ` Juergen Gross
0 siblings, 0 replies; 22+ messages in thread
From: Juergen Gross @ 2018-06-13 7:57 UTC (permalink / raw)
To: Jan Beulich; +Cc: Ian Jackson, Wei Liu, xen-devel
On 13/06/18 09:21, Jan Beulich wrote:
>>>> On 13.06.18 at 08:50, <jgross@suse.com> wrote:
>> On 13/06/18 08:11, Jan Beulich wrote:
>>> Teaching the privcmd driver of all
>>> the indirections in hypercall request structures doesn't look very
>>> attractive (or maintainable). Or are you thinking of the caller
>>> providing sideband information describing the buffers involved,
>>> perhaps along the lines of how dm_op was designed?
>>
>> I thought about that, yes. libxencall already has all the needed data
>> for that. Another possibility would be a dedicated ioctl for registering
>> a hypercall buffer (or some of them).
>
> I'm not sure that's an option: Is it legitimate (secure) to retain the
> effects of get_user_pages() across system calls?
I have to check that.
>>> There's another option, but that has potentially severe drawbacks
>>> too: Instead of returning -EFAULT on buffer access issues, we
>>> could raise #PF on the very hypercall insn. Maybe something to
>>> consider as an opt-in for PV/PVH, and as default for HVM.
>>
>> Hmm, I'm not sure this will solve any problem. I'm not aware that it
>> is considered good practice to just access a user buffer from kernel
>> without using copyin()/copyout() when you haven't locked the page(s)
>> via get_user_pages(), even if the buffer was mlock()ed. Returning
>> -EFAULT is the right thing to do, I believe.
>
> But we're talking about the very copyin()/copyout(), just that here
> it's being amortized by doing the operation just once (in the
> hypervisor). A #PF would arise from syscall buffer copyin()/copyout(),
> and the suggestion was to produce the same effect for the squashed
> operation. Perhaps we wouldn't want #PF to come back from ordinary
> (kernel invoked) hypercalls, but ones relayed by privcmd are different
> in many ways anyway (see the stac()/clac() pair around the actual
> call, for example).
Aah, okay. This is an option, but it would require some kind of
interface to tell the hypervisor it should raise the #PF instead of
returning -EFAULT, of course, as the kernel has to be prepared for
that.
I like that idea very much!
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-12 15:58 ` Juergen Gross
2018-06-13 6:11 ` Jan Beulich
[not found] ` <5B20B5A602000078001CAACA@suse.com>
@ 2018-06-13 8:52 ` Juergen Gross
2018-06-13 8:58 ` Andrew Cooper
2 siblings, 1 reply; 22+ messages in thread
From: Juergen Gross @ 2018-06-13 8:52 UTC (permalink / raw)
To: Ian Jackson; +Cc: xen-devel, Wei Liu, Jan Beulich
On 12/06/18 17:58, Juergen Gross wrote:
> On 08/06/18 12:12, Juergen Gross wrote:
>> On 07/06/18 13:30, Juergen Gross wrote:
>>> On 06/06/18 11:40, Juergen Gross wrote:
>>>> On 06/06/18 11:35, Jan Beulich wrote:
>>>>>>>> On 05.06.18 at 18:19, <ian.jackson@citrix.com> wrote:
>>>>>>>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
>>>>>>
>>>>>> I thought I would reply again with the key point from my earlier mail
>>>>>> highlighted, and go a bit further. The first thing to go wrong in
>>>>>> this was:
>>>>>>
>>>>>> 2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
>>>>>> 2018-05-30 22:12:49.483+0000: xc: Save failed (14 = Bad address): Internal error
>>>>>> 2018-05-30 22:12:49.648+0000: libxl-save-helper: complete r=-1: Bad address
>>>>>>
>>>>>> You can see similar messages in the other logfile:
>>>>>>
>>>>>> 2018-05-30 22:12:49.650+0000: libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
>>>>>>
>>>>>> All of these are reports of the same thing: xc_get_pfn_type_batch at
>>>>>> xc_sr_save.c:133 failed with EFAULT. I'm afraid I don't know why.
>>>>>>
>>>>>> There is no corresponding message in the host's serial log nor the
>>>>>> dom0 kernel log.
>>>>>
>>>>> I vaguely recall from the time when I had looked at the similar Windows
>>>>> migration issues that the guest is already in the process of being cleaned
>>>>> up when these occur. Commit 2dbe9c3cd2 ("x86/mm: silence a pointless
>>>>> warning") intentionally suppressed a log message here, and the
>>>>> immediately following debugging code (933f966bcd x86/mm: add
>>>>> temporary debugging code to get_page_from_gfn_p2m()) was reverted
>>>>> a little over a month later. This wasn't as a follow-up to another patch
>>>>> (fix), but following the discussion rooted at
>>>>> https://lists.xenproject.org/archives/html/xen-devel/2017-06/msg00324.html
>>>>
>>>> That was -ESRCH, not -EFAULT.
>>>
>>> I've looked a little bit more into this.
>>>
>>> As we are seeing EFAULT being returned by the hypervisor this either
>>> means the tools are specifying an invalid address (quite unlikely)
>>> or the buffers are not as MAP_LOCKED as we wish them to be.
>>>
>>> Is there a way to see whether the host was experiencing some memory
>>> shortage, so the buffers might have been swapped out?
>>>
>>> man mmap tells me: "This implementation will try to populate (prefault)
>>> the whole range but the mmap call doesn't fail with ENOMEM if this
>>> fails. Therefore major faults might happen later on."
>>>
>>> And: "One should use mmap(2) plus mlock(2) when major faults are not
>>> acceptable after the initialization of the mapping."
>>>
>>> With osdep_alloc_pages() in tools/libs/call/linux.c touching all the
>>> hypercall buffer pages before doing the hypercall I'm not sure this
>>> could be an issue.
>>>
>>> Any thoughts on that?
>>
>> Ian, is there a chance to dedicate a machine to a specific test trying
>> to reproduce the problem? In case we manage to get this failure in a
>> reasonable time frame I guess the most promising approach would be to
>> use a test hypervisor producing more debug data. If you think this is
>> worth doing I can write a patch.
>
> Trying to reproduce the problem in a limited test environment finally
> worked: doing a loop of "xl save -c" produced the problem after 198
> iterations.
>
> I have asked a SUSE engineer doing kernel memory management if he
> could think of something. His idea is that maybe some kthread could be
> the reason for our problem, e.g. trying page migration or compaction
> (at least on the test machine I've looked at compaction of mlocked
> pages is allowed: /proc/sys/vm/compact_unevictable_allowed is 1).
>
> In order to be really sure nothing in the kernel can temporarily
> switch hypercall buffer pages read-only or invalid for the hypervisor
> we'll have to modify the privcmd driver interface: it will have to
> gain knowledge which pages are handed over to the hypervisor as buffers
> in order to be able to lock them accordingly via get_user_pages().
>
> While this is a possible explanation of the fault we are seeing it might
> be related to another reason. So I'm going to apply some modifications
> to the hypervisor to get some more diagnostics in order to verify the
> suspected kernel behavior is really the reason for the hypervisor to
> return EFAULT.
I was lucky. Took only 39 iterations this time.
The debug data confirms the theory that the kernel is setting the PTE to
invalid or read only for a short amount of time:
(XEN) fixup for address 00007ffb9904fe44, error_code 0002:
(XEN) Pagetable walk from 00007ffb9904fe44:
(XEN) L4[0x0ff] = 0000000458da6067 0000000000019190
(XEN) L3[0x1ee] = 0000000457d26067 0000000000018210
(XEN) L2[0x0c8] = 0000000445ab3067 0000000000006083
(XEN) L1[0x04f] = 8000000458cdc107 000000000001925a
(XEN) Xen call trace:
(XEN) [<ffff82d0802abe31>] __copy_to_user_ll+0x27/0x30
(XEN) [<ffff82d080272edb>] arch_do_domctl+0x5a8/0x2648
(XEN) [<ffff82d080206d5d>] do_domctl+0x18fb/0x1c4e
(XEN) [<ffff82d08036d1ba>] pv_hypercall+0x1f4/0x43e
(XEN) [<ffff82d0803734a6>] lstar_enter+0x116/0x120
The page was writable again when the page walk data has been collected,
but A and D bits still are 0 (which should not be the case in case the
kernel didn't touch the PTE, as the hypervisor read from that page some
instructions before the failed write).
Starting with the Xen patches now...
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-13 8:52 ` Juergen Gross
@ 2018-06-13 8:58 ` Andrew Cooper
2018-06-13 9:02 ` Juergen Gross
0 siblings, 1 reply; 22+ messages in thread
From: Andrew Cooper @ 2018-06-13 8:58 UTC (permalink / raw)
To: Juergen Gross, Ian Jackson; +Cc: xen-devel, Wei Liu, Jan Beulich
On 13/06/18 09:52, Juergen Gross wrote:
> On 12/06/18 17:58, Juergen Gross wrote:
>> On 08/06/18 12:12, Juergen Gross wrote:
>>> On 07/06/18 13:30, Juergen Gross wrote:
>>>> On 06/06/18 11:40, Juergen Gross wrote:
>>>>> On 06/06/18 11:35, Jan Beulich wrote:
>>>>>>>>> On 05.06.18 at 18:19, <ian.jackson@citrix.com> wrote:
>>>>>>>>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
>>>>>>> I thought I would reply again with the key point from my earlier mail
>>>>>>> highlighted, and go a bit further. The first thing to go wrong in
>>>>>>> this was:
>>>>>>>
>>>>>>> 2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
>>>>>>> 2018-05-30 22:12:49.483+0000: xc: Save failed (14 = Bad address): Internal error
>>>>>>> 2018-05-30 22:12:49.648+0000: libxl-save-helper: complete r=-1: Bad address
>>>>>>>
>>>>>>> You can see similar messages in the other logfile:
>>>>>>>
>>>>>>> 2018-05-30 22:12:49.650+0000: libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
>>>>>>>
>>>>>>> All of these are reports of the same thing: xc_get_pfn_type_batch at
>>>>>>> xc_sr_save.c:133 failed with EFAULT. I'm afraid I don't know why.
>>>>>>>
>>>>>>> There is no corresponding message in the host's serial log nor the
>>>>>>> dom0 kernel log.
>>>>>> I vaguely recall from the time when I had looked at the similar Windows
>>>>>> migration issues that the guest is already in the process of being cleaned
>>>>>> up when these occur. Commit 2dbe9c3cd2 ("x86/mm: silence a pointless
>>>>>> warning") intentionally suppressed a log message here, and the
>>>>>> immediately following debugging code (933f966bcd x86/mm: add
>>>>>> temporary debugging code to get_page_from_gfn_p2m()) was reverted
>>>>>> a little over a month later. This wasn't as a follow-up to another patch
>>>>>> (fix), but following the discussion rooted at
>>>>>> https://lists.xenproject.org/archives/html/xen-devel/2017-06/msg00324.html
>>>>> That was -ESRCH, not -EFAULT.
>>>> I've looked a little bit more into this.
>>>>
>>>> As we are seeing EFAULT being returned by the hypervisor this either
>>>> means the tools are specifying an invalid address (quite unlikely)
>>>> or the buffers are not as MAP_LOCKED as we wish them to be.
>>>>
>>>> Is there a way to see whether the host was experiencing some memory
>>>> shortage, so the buffers might have been swapped out?
>>>>
>>>> man mmap tells me: "This implementation will try to populate (prefault)
>>>> the whole range but the mmap call doesn't fail with ENOMEM if this
>>>> fails. Therefore major faults might happen later on."
>>>>
>>>> And: "One should use mmap(2) plus mlock(2) when major faults are not
>>>> acceptable after the initialization of the mapping."
>>>>
>>>> With osdep_alloc_pages() in tools/libs/call/linux.c touching all the
>>>> hypercall buffer pages before doing the hypercall I'm not sure this
>>>> could be an issue.
>>>>
>>>> Any thoughts on that?
>>> Ian, is there a chance to dedicate a machine to a specific test trying
>>> to reproduce the problem? In case we manage to get this failure in a
>>> reasonable time frame I guess the most promising approach would be to
>>> use a test hypervisor producing more debug data. If you think this is
>>> worth doing I can write a patch.
>> Trying to reproduce the problem in a limited test environment finally
>> worked: doing a loop of "xl save -c" produced the problem after 198
>> iterations.
>>
>> I have asked a SUSE engineer doing kernel memory management if he
>> could think of something. His idea is that maybe some kthread could be
>> the reason for our problem, e.g. trying page migration or compaction
>> (at least on the test machine I've looked at compaction of mlocked
>> pages is allowed: /proc/sys/vm/compact_unevictable_allowed is 1).
>>
>> In order to be really sure nothing in the kernel can temporarily
>> switch hypercall buffer pages read-only or invalid for the hypervisor
>> we'll have to modify the privcmd driver interface: it will have to
>> gain knowledge which pages are handed over to the hypervisor as buffers
>> in order to be able to lock them accordingly via get_user_pages().
>>
>> While this is a possible explanation of the fault we are seeing it might
>> be related to another reason. So I'm going to apply some modifications
>> to the hypervisor to get some more diagnostics in order to verify the
>> suspected kernel behavior is really the reason for the hypervisor to
>> return EFAULT.
> I was lucky. Took only 39 iterations this time.
>
> The debug data confirms the theory that the kernel is setting the PTE to
> invalid or read only for a short amount of time:
>
> (XEN) fixup for address 00007ffb9904fe44, error_code 0002:
> (XEN) Pagetable walk from 00007ffb9904fe44:
> (XEN) L4[0x0ff] = 0000000458da6067 0000000000019190
> (XEN) L3[0x1ee] = 0000000457d26067 0000000000018210
> (XEN) L2[0x0c8] = 0000000445ab3067 0000000000006083
> (XEN) L1[0x04f] = 8000000458cdc107 000000000001925a
> (XEN) Xen call trace:
> (XEN) [<ffff82d0802abe31>] __copy_to_user_ll+0x27/0x30
> (XEN) [<ffff82d080272edb>] arch_do_domctl+0x5a8/0x2648
> (XEN) [<ffff82d080206d5d>] do_domctl+0x18fb/0x1c4e
> (XEN) [<ffff82d08036d1ba>] pv_hypercall+0x1f4/0x43e
> (XEN) [<ffff82d0803734a6>] lstar_enter+0x116/0x120
>
> The page was writable again when the page walk data has been collected,
> but A and D bits still are 0 (which should not be the case in case the
> kernel didn't touch the PTE, as the hypervisor read from that page some
> instructions before the failed write).
>
> Starting with the Xen patches now...
Given that walk, I'd expect the spurious pagefault logic to have kicked
in, and retried.
Presumably the spurious walk logic saw the non-present/read-only mappings?
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [xen-unstable test] 123379: regressions - FAIL
2018-06-13 8:58 ` Andrew Cooper
@ 2018-06-13 9:02 ` Juergen Gross
0 siblings, 0 replies; 22+ messages in thread
From: Juergen Gross @ 2018-06-13 9:02 UTC (permalink / raw)
To: Andrew Cooper, Ian Jackson; +Cc: xen-devel, Wei Liu, Jan Beulich
On 13/06/18 10:58, Andrew Cooper wrote:
> On 13/06/18 09:52, Juergen Gross wrote:
>> On 12/06/18 17:58, Juergen Gross wrote:
>>> On 08/06/18 12:12, Juergen Gross wrote:
>>>> On 07/06/18 13:30, Juergen Gross wrote:
>>>>> On 06/06/18 11:40, Juergen Gross wrote:
>>>>>> On 06/06/18 11:35, Jan Beulich wrote:
>>>>>>>>>> On 05.06.18 at 18:19, <ian.jackson@citrix.com> wrote:
>>>>>>>>>> test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 14 guest-saverestore.2
>>>>>>>> I thought I would reply again with the key point from my earlier mail
>>>>>>>> highlighted, and go a bit further. The first thing to go wrong in
>>>>>>>> this was:
>>>>>>>>
>>>>>>>> 2018-05-30 22:12:49.320+0000: xc: Failed to get types for pfn batch (14 = Bad address): Internal error
>>>>>>>> 2018-05-30 22:12:49.483+0000: xc: Save failed (14 = Bad address): Internal error
>>>>>>>> 2018-05-30 22:12:49.648+0000: libxl-save-helper: complete r=-1: Bad address
>>>>>>>>
>>>>>>>> You can see similar messages in the other logfile:
>>>>>>>>
>>>>>>>> 2018-05-30 22:12:49.650+0000: libxl: libxl_stream_write.c:350:libxl__xc_domain_save_done: Domain 3:saving domain: domain responded to suspend request: Bad address
>>>>>>>>
>>>>>>>> All of these are reports of the same thing: xc_get_pfn_type_batch at
>>>>>>>> xc_sr_save.c:133 failed with EFAULT. I'm afraid I don't know why.
>>>>>>>>
>>>>>>>> There is no corresponding message in the host's serial log nor the
>>>>>>>> dom0 kernel log.
>>>>>>> I vaguely recall from the time when I had looked at the similar Windows
>>>>>>> migration issues that the guest is already in the process of being cleaned
>>>>>>> up when these occur. Commit 2dbe9c3cd2 ("x86/mm: silence a pointless
>>>>>>> warning") intentionally suppressed a log message here, and the
>>>>>>> immediately following debugging code (933f966bcd x86/mm: add
>>>>>>> temporary debugging code to get_page_from_gfn_p2m()) was reverted
>>>>>>> a little over a month later. This wasn't as a follow-up to another patch
>>>>>>> (fix), but following the discussion rooted at
>>>>>>> https://lists.xenproject.org/archives/html/xen-devel/2017-06/msg00324.html
>>>>>> That was -ESRCH, not -EFAULT.
>>>>> I've looked a little bit more into this.
>>>>>
>>>>> As we are seeing EFAULT being returned by the hypervisor this either
>>>>> means the tools are specifying an invalid address (quite unlikely)
>>>>> or the buffers are not as MAP_LOCKED as we wish them to be.
>>>>>
>>>>> Is there a way to see whether the host was experiencing some memory
>>>>> shortage, so the buffers might have been swapped out?
>>>>>
>>>>> man mmap tells me: "This implementation will try to populate (prefault)
>>>>> the whole range but the mmap call doesn't fail with ENOMEM if this
>>>>> fails. Therefore major faults might happen later on."
>>>>>
>>>>> And: "One should use mmap(2) plus mlock(2) when major faults are not
>>>>> acceptable after the initialization of the mapping."
>>>>>
>>>>> With osdep_alloc_pages() in tools/libs/call/linux.c touching all the
>>>>> hypercall buffer pages before doing the hypercall I'm not sure this
>>>>> could be an issue.
>>>>>
>>>>> Any thoughts on that?
>>>> Ian, is there a chance to dedicate a machine to a specific test trying
>>>> to reproduce the problem? In case we manage to get this failure in a
>>>> reasonable time frame I guess the most promising approach would be to
>>>> use a test hypervisor producing more debug data. If you think this is
>>>> worth doing I can write a patch.
>>> Trying to reproduce the problem in a limited test environment finally
>>> worked: doing a loop of "xl save -c" produced the problem after 198
>>> iterations.
>>>
>>> I have asked a SUSE engineer doing kernel memory management if he
>>> could think of something. His idea is that maybe some kthread could be
>>> the reason for our problem, e.g. trying page migration or compaction
>>> (at least on the test machine I've looked at compaction of mlocked
>>> pages is allowed: /proc/sys/vm/compact_unevictable_allowed is 1).
>>>
>>> In order to be really sure nothing in the kernel can temporarily
>>> switch hypercall buffer pages read-only or invalid for the hypervisor
>>> we'll have to modify the privcmd driver interface: it will have to
>>> gain knowledge which pages are handed over to the hypervisor as buffers
>>> in order to be able to lock them accordingly via get_user_pages().
>>>
>>> While this is a possible explanation of the fault we are seeing it might
>>> be related to another reason. So I'm going to apply some modifications
>>> to the hypervisor to get some more diagnostics in order to verify the
>>> suspected kernel behavior is really the reason for the hypervisor to
>>> return EFAULT.
>> I was lucky. Took only 39 iterations this time.
>>
>> The debug data confirms the theory that the kernel is setting the PTE to
>> invalid or read only for a short amount of time:
>>
>> (XEN) fixup for address 00007ffb9904fe44, error_code 0002:
>> (XEN) Pagetable walk from 00007ffb9904fe44:
>> (XEN) L4[0x0ff] = 0000000458da6067 0000000000019190
>> (XEN) L3[0x1ee] = 0000000457d26067 0000000000018210
>> (XEN) L2[0x0c8] = 0000000445ab3067 0000000000006083
>> (XEN) L1[0x04f] = 8000000458cdc107 000000000001925a
>> (XEN) Xen call trace:
>> (XEN) [<ffff82d0802abe31>] __copy_to_user_ll+0x27/0x30
>> (XEN) [<ffff82d080272edb>] arch_do_domctl+0x5a8/0x2648
>> (XEN) [<ffff82d080206d5d>] do_domctl+0x18fb/0x1c4e
>> (XEN) [<ffff82d08036d1ba>] pv_hypercall+0x1f4/0x43e
>> (XEN) [<ffff82d0803734a6>] lstar_enter+0x116/0x120
>>
>> The page was writable again when the page walk data has been collected,
>> but A and D bits still are 0 (which should not be the case in case the
>> kernel didn't touch the PTE, as the hypervisor read from that page some
>> instructions before the failed write).
>>
>> Starting with the Xen patches now...
>
> Given that walk, I'd expect the spurious pagefault logic to have kicked
> in, and retried.
>
> Presumably the spurious walk logic saw the non-present/read-only mappings?
I guess so.
Otherwise my debug coding wouldn't have been called...
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2018-06-13 9:02 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-05-31 6:00 [xen-unstable test] 123379: regressions - FAIL osstest service owner
2018-05-31 8:32 ` Juergen Gross
2018-05-31 9:14 ` Juergen Gross
2018-06-01 8:10 ` Jan Beulich
2018-06-01 9:08 ` Juergen Gross
2018-06-05 16:16 ` Ian Jackson
2018-06-06 7:39 ` Juergen Gross
2018-06-05 16:19 ` Ian Jackson
2018-06-06 9:35 ` Jan Beulich
[not found] ` <5B17AAE102000078001C8972@suse.com>
2018-06-06 9:40 ` Juergen Gross
2018-06-07 11:30 ` Juergen Gross
2018-06-08 10:12 ` Juergen Gross
2018-06-12 15:58 ` Juergen Gross
2018-06-13 6:11 ` Jan Beulich
[not found] ` <5B20B5A602000078001CAACA@suse.com>
2018-06-13 6:50 ` Juergen Gross
2018-06-13 7:21 ` Jan Beulich
[not found] ` <5B20C5E002000078001CAB80@suse.com>
2018-06-13 7:57 ` Juergen Gross
2018-06-13 8:52 ` Juergen Gross
2018-06-13 8:58 ` Andrew Cooper
2018-06-13 9:02 ` Juergen Gross
2018-06-08 14:25 ` Ad-hoc test instructions (was Re: [xen-unstable test] 123379: regressions - FAIL) Ian Jackson
2018-06-08 15:42 ` Juergen Gross
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.