* dom0 PV looping on search_pre_exception_table()
@ 2020-12-08 17:57 Manuel Bouyer
2020-12-08 18:13 ` Andrew Cooper
0 siblings, 1 reply; 25+ messages in thread
From: Manuel Bouyer @ 2020-12-08 17:57 UTC (permalink / raw)
To: xen-devel
Hello,
for the first time I tried to boot a xen kernel from devel with
a NetBSD PV dom0. The kernel boots, but when the first userland prcess
is launched, it seems to enter a loop involving search_pre_exception_table()
(I see an endless stream from the dprintk() at arch/x86/extable.c:202)
With xen 4.13 I see it, but exactly once:
(XEN) extable.c:202: Pre-exception: ffff82d08038c304 -> ffff82d08038c8c8
with devel:
(XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8
(XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8
(XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8
(XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8
(XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8
[...]
the dom0 kernel is the same.
At first glance it looks like a fault in the guest is not handled at it should,
and the userland process keeps faulting on the same address.
Any idea what to look at ?
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: dom0 PV looping on search_pre_exception_table() 2020-12-08 17:57 dom0 PV looping on search_pre_exception_table() Manuel Bouyer @ 2020-12-08 18:13 ` Andrew Cooper 2020-12-09 8:39 ` Jan Beulich 2020-12-09 10:15 ` Manuel Bouyer 0 siblings, 2 replies; 25+ messages in thread From: Andrew Cooper @ 2020-12-08 18:13 UTC (permalink / raw) To: Manuel Bouyer, xen-devel On 08/12/2020 17:57, Manuel Bouyer wrote: > Hello, > for the first time I tried to boot a xen kernel from devel with > a NetBSD PV dom0. The kernel boots, but when the first userland prcess > is launched, it seems to enter a loop involving search_pre_exception_table() > (I see an endless stream from the dprintk() at arch/x86/extable.c:202) > > With xen 4.13 I see it, but exactly once: > (XEN) extable.c:202: Pre-exception: ffff82d08038c304 -> ffff82d08038c8c8 > > with devel: > (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > [...] > > the dom0 kernel is the same. > > At first glance it looks like a fault in the guest is not handled at it should, > and the userland process keeps faulting on the same address. > > Any idea what to look at ? That is a reoccurring fault on IRET back to guest context, and is probably caused by some unwise-in-hindsight cleanup which doesn't escalate the failure to the failsafe callback. This ought to give something useful to debug with: ~Andrew diff --git a/xen/arch/x86/extable.c b/xen/arch/x86/extable.c index 70972f1085..62a7bcbe38 100644 --- a/xen/arch/x86/extable.c +++ b/xen/arch/x86/extable.c @@ -191,6 +191,10 @@ static int __init stub_selftest(void) __initcall(stub_selftest); #endif +#include <xen/sched.h> +#include <xen/softirq.h> +const char *vec_name(unsigned int vec); + unsigned long search_pre_exception_table(struct cpu_user_regs *regs) { @@ -199,7 +203,13 @@ search_pre_exception_table(struct cpu_user_regs *regs) __start___pre_ex_table, __stop___pre_ex_table-1, addr); if ( fixup ) { - dprintk(XENLOG_INFO, "Pre-exception: %p -> %p\n", _p(addr), _p(fixup)); + printk(XENLOG_ERR "IRET fault: %s[%04x]\n", + vec_name(regs->entry_vector), regs->error_code); + + domain_crash(current->domain); + for ( ;; ) + do_softirq(); + perfc_incr(exception_fixed); } return fixup; diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index 0459cee9fb..1059f3ce66 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -687,7 +687,7 @@ const char *trapstr(unsigned int trapnr) return trapnr < ARRAY_SIZE(strings) ? strings[trapnr] : "???"; } -static const char *vec_name(unsigned int vec) +const char *vec_name(unsigned int vec) { static const char names[][4] = { #define P(x) [X86_EXC_ ## x] = "#" #x ^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-08 18:13 ` Andrew Cooper @ 2020-12-09 8:39 ` Jan Beulich 2020-12-09 9:49 ` Manuel Bouyer 2020-12-09 10:15 ` Manuel Bouyer 1 sibling, 1 reply; 25+ messages in thread From: Jan Beulich @ 2020-12-09 8:39 UTC (permalink / raw) To: Andrew Cooper, Manuel Bouyer; +Cc: xen-devel On 08.12.2020 19:13, Andrew Cooper wrote: > On 08/12/2020 17:57, Manuel Bouyer wrote: >> Hello, >> for the first time I tried to boot a xen kernel from devel with >> a NetBSD PV dom0. The kernel boots, but when the first userland prcess >> is launched, it seems to enter a loop involving search_pre_exception_table() >> (I see an endless stream from the dprintk() at arch/x86/extable.c:202) >> >> With xen 4.13 I see it, but exactly once: >> (XEN) extable.c:202: Pre-exception: ffff82d08038c304 -> ffff82d08038c8c8 >> >> with devel: >> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 >> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 >> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 >> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 >> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 >> [...] >> >> the dom0 kernel is the same. >> >> At first glance it looks like a fault in the guest is not handled at it should, >> and the userland process keeps faulting on the same address. >> >> Any idea what to look at ? > > That is a reoccurring fault on IRET back to guest context, and is > probably caused by some unwise-in-hindsight cleanup which doesn't > escalate the failure to the failsafe callback. But is this a 32-bit Dom0? 64-bit ones get well-known selectors installed for CS and SS by create_bounce_frame(), and we don't permit registration of non-canonical trap handler entry point addresses. I have to admit I also find curious the difference between 4.13 and master. Jan ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 8:39 ` Jan Beulich @ 2020-12-09 9:49 ` Manuel Bouyer 0 siblings, 0 replies; 25+ messages in thread From: Manuel Bouyer @ 2020-12-09 9:49 UTC (permalink / raw) To: Jan Beulich; +Cc: Andrew Cooper, xen-devel On Wed, Dec 09, 2020 at 09:39:49AM +0100, Jan Beulich wrote: > On 08.12.2020 19:13, Andrew Cooper wrote: > > On 08/12/2020 17:57, Manuel Bouyer wrote: > >> Hello, > >> for the first time I tried to boot a xen kernel from devel with > >> a NetBSD PV dom0. The kernel boots, but when the first userland prcess > >> is launched, it seems to enter a loop involving search_pre_exception_table() > >> (I see an endless stream from the dprintk() at arch/x86/extable.c:202) > >> > >> With xen 4.13 I see it, but exactly once: > >> (XEN) extable.c:202: Pre-exception: ffff82d08038c304 -> ffff82d08038c8c8 > >> > >> with devel: > >> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > >> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > >> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > >> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > >> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > >> [...] > >> > >> the dom0 kernel is the same. > >> > >> At first glance it looks like a fault in the guest is not handled at it should, > >> and the userland process keeps faulting on the same address. > >> > >> Any idea what to look at ? > > > > That is a reoccurring fault on IRET back to guest context, and is > > probably caused by some unwise-in-hindsight cleanup which doesn't > > escalate the failure to the failsafe callback. > > But is this a 32-bit Dom0? 64-bit ones get well-known selectors > installed for CS and SS by create_bounce_frame(), and we don't > permit registration of non-canonical trap handler entry point > addresses. No, it's a 64bits dom0. -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-08 18:13 ` Andrew Cooper 2020-12-09 8:39 ` Jan Beulich @ 2020-12-09 10:15 ` Manuel Bouyer 2020-12-09 13:28 ` Andrew Cooper 1 sibling, 1 reply; 25+ messages in thread From: Manuel Bouyer @ 2020-12-09 10:15 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Tue, Dec 08, 2020 at 06:13:46PM +0000, Andrew Cooper wrote: > On 08/12/2020 17:57, Manuel Bouyer wrote: > > Hello, > > for the first time I tried to boot a xen kernel from devel with > > a NetBSD PV dom0. The kernel boots, but when the first userland prcess > > is launched, it seems to enter a loop involving search_pre_exception_table() > > (I see an endless stream from the dprintk() at arch/x86/extable.c:202) > > > > With xen 4.13 I see it, but exactly once: > > (XEN) extable.c:202: Pre-exception: ffff82d08038c304 -> ffff82d08038c8c8 > > > > with devel: > > (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > > (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > > (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > > (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > > (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 > > [...] > > > > the dom0 kernel is the same. > > > > At first glance it looks like a fault in the guest is not handled at it should, > > and the userland process keeps faulting on the same address. > > > > Any idea what to look at ? > > That is a reoccurring fault on IRET back to guest context, and is > probably caused by some unwise-in-hindsight cleanup which doesn't > escalate the failure to the failsafe callback. > > This ought to give something useful to debug with: thanks, I got: (XEN) IRET fault: #PF[0000] (XEN) domain_crash called from extable.c:209 (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.15-unstable x86_64 debug=y Tainted: C ]---- (XEN) CPU: 0 (XEN) RIP: 0047:[<00007f7e184007d0>] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: ffff82d04038c309 rbx: 0000000000000000 rcx: 000000000000e008 (XEN) rdx: 0000000000010086 rsi: ffff83007fcb7f78 rdi: 000000000000e010 (XEN) rbp: 0000000000000000 rsp: 00007f7fff53e3e0 r8: 0000000e00000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 (XEN) cr3: 0000000079cdb000 cr2: 00007f7fff53e3e0 (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: ffffffff80cf2dc0 (XEN) ds: 0023 es: 0023 fs: 0000 gs: 0000 ss: 003f cs: 0047 (XEN) Guest stack trace from rsp=00007f7fff53e3e0: (XEN) 0000000000000001 00007f7fff53e8f8 0000000000000000 0000000000000000 (XEN) 0000000000000003 000000004b600040 0000000000000004 0000000000000038 (XEN) 0000000000000005 0000000000000008 0000000000000006 0000000000001000 (XEN) 0000000000000007 00007f7e18400000 0000000000000008 0000000000000000 (XEN) 0000000000000009 000000004b601cd0 00000000000007d0 0000000000000000 (XEN) 00000000000007d1 0000000000000000 00000000000007d2 0000000000000000 (XEN) 00000000000007d3 0000000000000000 000000000000000d 00007f7fff53f000 (XEN) 00000000000007de 00007f7fff53e4e0 0000000000000000 0000000000000000 (XEN) 6e692f6e6962732f 0000000000007469 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 10:15 ` Manuel Bouyer @ 2020-12-09 13:28 ` Andrew Cooper 2020-12-09 13:59 ` Manuel Bouyer 0 siblings, 1 reply; 25+ messages in thread From: Andrew Cooper @ 2020-12-09 13:28 UTC (permalink / raw) To: Manuel Bouyer; +Cc: xen-devel [-- Attachment #1: Type: text/plain, Size: 5677 bytes --] On 09/12/2020 10:15, Manuel Bouyer wrote: > On Tue, Dec 08, 2020 at 06:13:46PM +0000, Andrew Cooper wrote: >> On 08/12/2020 17:57, Manuel Bouyer wrote: >>> Hello, >>> for the first time I tried to boot a xen kernel from devel with >>> a NetBSD PV dom0. The kernel boots, but when the first userland prcess >>> is launched, it seems to enter a loop involving search_pre_exception_table() >>> (I see an endless stream from the dprintk() at arch/x86/extable.c:202) >>> >>> With xen 4.13 I see it, but exactly once: >>> (XEN) extable.c:202: Pre-exception: ffff82d08038c304 -> ffff82d08038c8c8 >>> >>> with devel: >>> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 >>> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 >>> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 >>> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 >>> (XEN) extable.c:202: Pre-exception: ffff82d040393309 -> ffff82d0403938c8 >>> [...] >>> >>> the dom0 kernel is the same. >>> >>> At first glance it looks like a fault in the guest is not handled at it should, >>> and the userland process keeps faulting on the same address. >>> >>> Any idea what to look at ? >> That is a reoccurring fault on IRET back to guest context, and is >> probably caused by some unwise-in-hindsight cleanup which doesn't >> escalate the failure to the failsafe callback. >> >> This ought to give something useful to debug with: > thanks, I got: > (XEN) IRET fault: #PF[0000] > (XEN) domain_crash called from extable.c:209 > (XEN) Domain 0 (vcpu#0) crashed on cpu#0: > (XEN) ----[ Xen-4.15-unstable x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 0 > (XEN) RIP: 0047:[<00007f7e184007d0>] > (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v0) > (XEN) rax: ffff82d04038c309 rbx: 0000000000000000 rcx: 000000000000e008 > (XEN) rdx: 0000000000010086 rsi: ffff83007fcb7f78 rdi: 000000000000e010 > (XEN) rbp: 0000000000000000 rsp: 00007f7fff53e3e0 r8: 0000000e00000000 > (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 > (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 > (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 > (XEN) cr3: 0000000079cdb000 cr2: 00007f7fff53e3e0 > (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: ffffffff80cf2dc0 > (XEN) ds: 0023 es: 0023 fs: 0000 gs: 0000 ss: 003f cs: 0047 > (XEN) Guest stack trace from rsp=00007f7fff53e3e0: > (XEN) 0000000000000001 00007f7fff53e8f8 0000000000000000 0000000000000000 > (XEN) 0000000000000003 000000004b600040 0000000000000004 0000000000000038 > (XEN) 0000000000000005 0000000000000008 0000000000000006 0000000000001000 > (XEN) 0000000000000007 00007f7e18400000 0000000000000008 0000000000000000 > (XEN) 0000000000000009 000000004b601cd0 00000000000007d0 0000000000000000 > (XEN) 00000000000007d1 0000000000000000 00000000000007d2 0000000000000000 > (XEN) 00000000000007d3 0000000000000000 000000000000000d 00007f7fff53f000 > (XEN) 00000000000007de 00007f7fff53e4e0 0000000000000000 0000000000000000 > (XEN) 6e692f6e6962732f 0000000000007469 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. Pagefaults on IRET come either from stack accesses for operands (not the case here as Xen is otherwise working fine), or from segement selector loads for %cs and %ss. In this example, %ss is in the LDT, which specifically does use pagefaults to promote the frame to PGT_segdesc. I suspect that what is happening is that handle_ldt_mapping_fault() is failing to promote the page (for some reason), and we're taking the "In hypervisor mode? Leave it to the #PF handler to fix up." path due to the confusion in context, and Xen's #PF handler is concluding "nothing else to do". The older behaviour of escalating to the failsafe callback would have broken this cycle by rewriting %ss and re-entering the kernel. Please try the attached debugging patch, which is an extension of what I gave you yesterday. First, it ought to print %cr2, which I expect will point to Xen's virtual mapping of the vcpu's LDT. The logic ought to loop a few times so we can inspect the hypervisor codepaths which are effectively livelocked in this state, and I've also instrumented check_descriptor() failures because I've got a gut feeling that is the root cause of the problem. ~Andrew [-- Attachment #2: 0001-extable-dbg.patch --] [-- Type: text/x-patch, Size: 2272 bytes --] From 841a6950fec5b43b370653e0c833a54fed64882e Mon Sep 17 00:00:00 2001 From: Andrew Cooper <andrew.cooper3@citrix.com> Date: Wed, 9 Dec 2020 12:50:38 +0000 Subject: extable-dbg diff --git a/xen/arch/x86/extable.c b/xen/arch/x86/extable.c index 70972f1085..88b05bef38 100644 --- a/xen/arch/x86/extable.c +++ b/xen/arch/x86/extable.c @@ -191,6 +191,10 @@ static int __init stub_selftest(void) __initcall(stub_selftest); #endif +#include <xen/sched.h> +#include <xen/softirq.h> +const char *vec_name(unsigned int vec); + unsigned long search_pre_exception_table(struct cpu_user_regs *regs) { @@ -199,7 +203,21 @@ search_pre_exception_table(struct cpu_user_regs *regs) __start___pre_ex_table, __stop___pre_ex_table-1, addr); if ( fixup ) { - dprintk(XENLOG_INFO, "Pre-exception: %p -> %p\n", _p(addr), _p(fixup)); + static int count; + + printk(XENLOG_ERR "IRET fault: %s[%04x]\n", + vec_name(regs->entry_vector), regs->error_code); + + if ( regs->entry_vector == X86_EXC_PF ) + printk(XENLOG_ERR "%%cr2 %016lx\n", read_cr2()); + + if ( count++ > 2 ) + { + domain_crash(current->domain); + for ( ;; ) + do_softirq(); + } + perfc_incr(exception_fixed); } return fixup; diff --git a/xen/arch/x86/pv/descriptor-tables.c b/xen/arch/x86/pv/descriptor-tables.c index 39c1a2311a..6bc58bba67 100644 --- a/xen/arch/x86/pv/descriptor-tables.c +++ b/xen/arch/x86/pv/descriptor-tables.c @@ -282,6 +282,10 @@ int validate_segdesc_page(struct page_info *page) unmap_domain_page(descs); + if ( i != 512 ) + printk_once("Check Descriptor failed: idx %u, a: %08x, b: %08x\n", + i, descs[i].a, descs[i].b); + return i == 512 ? 0 : -EINVAL; } diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index 0459cee9fb..1059f3ce66 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -687,7 +687,7 @@ const char *trapstr(unsigned int trapnr) return trapnr < ARRAY_SIZE(strings) ? strings[trapnr] : "???"; } -static const char *vec_name(unsigned int vec) +const char *vec_name(unsigned int vec) { static const char names[][4] = { #define P(x) [X86_EXC_ ## x] = "#" #x ^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 13:28 ` Andrew Cooper @ 2020-12-09 13:59 ` Manuel Bouyer 2020-12-09 14:41 ` Andrew Cooper 0 siblings, 1 reply; 25+ messages in thread From: Manuel Bouyer @ 2020-12-09 13:59 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Wed, Dec 09, 2020 at 01:28:54PM +0000, Andrew Cooper wrote: > > Pagefaults on IRET come either from stack accesses for operands (not the > case here as Xen is otherwise working fine), or from segement selector > loads for %cs and %ss. > > In this example, %ss is in the LDT, which specifically does use > pagefaults to promote the frame to PGT_segdesc. > > I suspect that what is happening is that handle_ldt_mapping_fault() is > failing to promote the page (for some reason), and we're taking the "In > hypervisor mode? Leave it to the #PF handler to fix up." path due to the > confusion in context, and Xen's #PF handler is concluding "nothing else > to do". > > The older behaviour of escalating to the failsafe callback would have > broken this cycle by rewriting %ss and re-entering the kernel. > > > Please try the attached debugging patch, which is an extension of what I > gave you yesterday. First, it ought to print %cr2, which I expect will > point to Xen's virtual mapping of the vcpu's LDT. The logic ought to > loop a few times so we can inspect the hypervisor codepaths which are > effectively livelocked in this state, and I've also instrumented > check_descriptor() failures because I've got a gut feeling that is the > root cause of the problem. here's the output: (XEN) IRET fault: #PF[0000] [23/1999] (XEN) %cr2 ffff820000010040 (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040 (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040 (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040 (XEN) domain_crash called from extable.c:216 (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.15-unstable x86_64 debug=y Tainted: C ]---- (XEN) CPU: 0 (XEN) RIP: 0047:[<00007f7ff60007d0>] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: ffff82d04038c309 rbx: 0000000000000000 rcx: 000000000000e008 (XEN) rdx: 0000000000010086 rsi: ffff83007fcb7f78 rdi: 000000000000e010 (XEN) rbp: 0000000000000000 rsp: 00007f7fff4876c0 r8: 0000000e00000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 (XEN) cr3: 0000000079cdb000 cr2: ffffa1000000a040 (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: ffffffff80cf2dc0 (XEN) ds: 0023 es: 0023 fs: 0000 gs: 0000 ss: 003f cs: 0047 (XEN) Guest stack trace from rsp=00007f7fff4876c0: (XEN) 0000000000000001 00007f7fff487bd8 0000000000000000 0000000000000000 (XEN) 0000000000000003 00000000aee00040 0000000000000004 0000000000000038 (XEN) 0000000000000005 0000000000000008 0000000000000006 0000000000001000 (XEN) 0000000000000007 00007f7ff6000000 0000000000000008 0000000000000000 (XEN) 0000000000000009 00000000aee01cd0 00000000000007d0 0000000000000000 (XEN) 00000000000007d1 0000000000000000 00000000000007d2 0000000000000000 (XEN) 00000000000007d3 0000000000000000 000000000000000d 00007f7fff488000 (XEN) 00000000000007de 00007f7fff4877c0 0000000000000000 0000000000000000 (XEN) 6e692f6e6962732f 0000000000007469 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 13:59 ` Manuel Bouyer @ 2020-12-09 14:41 ` Andrew Cooper 2020-12-09 15:44 ` Manuel Bouyer 0 siblings, 1 reply; 25+ messages in thread From: Andrew Cooper @ 2020-12-09 14:41 UTC (permalink / raw) To: Manuel Bouyer; +Cc: xen-devel On 09/12/2020 13:59, Manuel Bouyer wrote: > On Wed, Dec 09, 2020 at 01:28:54PM +0000, Andrew Cooper wrote: >> Pagefaults on IRET come either from stack accesses for operands (not the >> case here as Xen is otherwise working fine), or from segement selector >> loads for %cs and %ss. >> >> In this example, %ss is in the LDT, which specifically does use >> pagefaults to promote the frame to PGT_segdesc. >> >> I suspect that what is happening is that handle_ldt_mapping_fault() is >> failing to promote the page (for some reason), and we're taking the "In >> hypervisor mode? Leave it to the #PF handler to fix up." path due to the >> confusion in context, and Xen's #PF handler is concluding "nothing else >> to do". >> >> The older behaviour of escalating to the failsafe callback would have >> broken this cycle by rewriting %ss and re-entering the kernel. >> >> >> Please try the attached debugging patch, which is an extension of what I >> gave you yesterday. First, it ought to print %cr2, which I expect will >> point to Xen's virtual mapping of the vcpu's LDT. The logic ought to >> loop a few times so we can inspect the hypervisor codepaths which are >> effectively livelocked in this state, and I've also instrumented >> check_descriptor() failures because I've got a gut feeling that is the >> root cause of the problem. > here's the output: > (XEN) IRET fault: #PF[0000] [23/1999] > (XEN) %cr2 ffff820000010040 > (XEN) IRET fault: #PF[0000] > (XEN) %cr2 ffff820000010040 > (XEN) IRET fault: #PF[0000] > (XEN) %cr2 ffff820000010040 > (XEN) IRET fault: #PF[0000] > (XEN) %cr2 ffff820000010040 > (XEN) domain_crash called from extable.c:216 > (XEN) Domain 0 (vcpu#0) crashed on cpu#0: > (XEN) ----[ Xen-4.15-unstable x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 0 > (XEN) RIP: 0047:[<00007f7ff60007d0>] > (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v0) > (XEN) rax: ffff82d04038c309 rbx: 0000000000000000 rcx: 000000000000e008 > (XEN) rdx: 0000000000010086 rsi: ffff83007fcb7f78 rdi: 000000000000e010 > (XEN) rbp: 0000000000000000 rsp: 00007f7fff4876c0 r8: 0000000e00000000 > (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 > (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 > (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 > (XEN) cr3: 0000000079cdb000 cr2: ffffa1000000a040 > (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: ffffffff80cf2dc0 > (XEN) ds: 0023 es: 0023 fs: 0000 gs: 0000 ss: 003f cs: 0047 > (XEN) Guest stack trace from rsp=00007f7fff4876c0: > (XEN) 0000000000000001 00007f7fff487bd8 0000000000000000 0000000000000000 > (XEN) 0000000000000003 00000000aee00040 0000000000000004 0000000000000038 > (XEN) 0000000000000005 0000000000000008 0000000000000006 0000000000001000 > (XEN) 0000000000000007 00007f7ff6000000 0000000000000008 0000000000000000 > (XEN) 0000000000000009 00000000aee01cd0 00000000000007d0 0000000000000000 > (XEN) 00000000000007d1 0000000000000000 00000000000007d2 0000000000000000 > (XEN) 00000000000007d3 0000000000000000 000000000000000d 00007f7fff488000 > (XEN) 00000000000007de 00007f7fff4877c0 0000000000000000 0000000000000000 > (XEN) 6e692f6e6962732f 0000000000007469 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. Huh, so it is the LDT, but we're not getting as far as inspecting the target frame. I wonder if the LDT is set up correctly. How about this incremental delta? ~Andrew diff --git a/xen/arch/x86/extable.c b/xen/arch/x86/extable.c index 88b05bef38..be59a3e216 100644 --- a/xen/arch/x86/extable.c +++ b/xen/arch/x86/extable.c @@ -203,13 +203,16 @@ search_pre_exception_table(struct cpu_user_regs *regs) __start___pre_ex_table, __stop___pre_ex_table-1, addr); if ( fixup ) { + struct vcpu *curr = current; static int count; printk(XENLOG_ERR "IRET fault: %s[%04x]\n", vec_name(regs->entry_vector), regs->error_code); if ( regs->entry_vector == X86_EXC_PF ) - printk(XENLOG_ERR "%%cr2 %016lx\n", read_cr2()); + printk(XENLOG_ERR "%%cr2 %016lx, LDT base %016lx, limit %04x\n", + read_cr2(), curr->arch.pv.ldt_base, + (curr->arch.pv.ldt_ents << 3) | 7); if ( count++ > 2 ) { diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index 1059f3ce66..3ac07a84c3 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -1233,6 +1233,8 @@ static int handle_ldt_mapping_fault(unsigned int offset, } else { + printk(XENLOG_ERR "*** pv_map_ldt_shadow_page(%#x) failed\n", offset); + /* In hypervisor mode? Leave it to the #PF handler to fix up. */ if ( !guest_mode(regs) ) return 0; ^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 14:41 ` Andrew Cooper @ 2020-12-09 15:44 ` Manuel Bouyer 2020-12-09 16:00 ` Andrew Cooper 0 siblings, 1 reply; 25+ messages in thread From: Manuel Bouyer @ 2020-12-09 15:44 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Wed, Dec 09, 2020 at 02:41:23PM +0000, Andrew Cooper wrote: > > Huh, so it is the LDT, but we're not getting as far as inspecting the > target frame. > > I wonder if the LDT is set up correctly. I guess it is, otherwise it wouldn't boot with a Xen 4.13 kernel, isn't it ? > How about this incremental delta? Here's the output (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 (XEN) *** pv_map_ldt_shadow_page(0x40) failed (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 (XEN) *** pv_map_ldt_shadow_page(0x40) failed (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 (XEN) *** pv_map_ldt_shadow_page(0x40) failed (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 (XEN) domain_crash called from extable.c:219 (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.15-unstable x86_64 debug=y Tainted: C ]---- (XEN) CPU: 0 (XEN) RIP: 0047:[<00007f7ecaa007d0>] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: ffff82d04038c309 rbx: 0000000000000000 rcx: 000000000000e008 (XEN) rdx: 0000000000010086 rsi: ffff83007fcb7f78 rdi: 000000000000e010 (XEN) rbp: 0000000000000000 rsp: 00007f7fff32e3f0 r8: 0000000e00000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 (XEN) cr3: 0000000079cdb000 cr2: ffffc4800000a040 (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: ffffffff80cf2dc0 (XEN) ds: 0023 es: 0023 fs: 0000 gs: 0000 ss: 003f cs: 0047 (XEN) Guest stack trace from rsp=00007f7fff32e3f0: (XEN) 0000000000000001 00007f7fff32e908 0000000000000000 0000000000000000 (XEN) 0000000000000003 0000000173e00040 0000000000000004 0000000000000038 (XEN) 0000000000000005 0000000000000008 0000000000000006 0000000000001000 (XEN) 0000000000000007 00007f7ecaa00000 0000000000000008 0000000000000000 (XEN) 0000000000000009 0000000173e01cd0 00000000000007d0 0000000000000000 (XEN) 00000000000007d1 0000000000000000 00000000000007d2 0000000000000000 (XEN) 00000000000007d3 0000000000000000 000000000000000d 00007f7fff32f000 (XEN) 00000000000007de 00007f7fff32e4f0 0000000000000000 0000000000000000 (XEN) 6e692f6e6962732f 0000000000007469 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 15:44 ` Manuel Bouyer @ 2020-12-09 16:00 ` Andrew Cooper 2020-12-09 16:30 ` Manuel Bouyer 0 siblings, 1 reply; 25+ messages in thread From: Andrew Cooper @ 2020-12-09 16:00 UTC (permalink / raw) To: Manuel Bouyer; +Cc: xen-devel On 09/12/2020 15:44, Manuel Bouyer wrote: > On Wed, Dec 09, 2020 at 02:41:23PM +0000, Andrew Cooper wrote: >> Huh, so it is the LDT, but we're not getting as far as inspecting the >> target frame. >> >> I wonder if the LDT is set up correctly. > I guess it is, otherwise it wouldn't boot with a Xen 4.13 kernel, isn't it ? Well - you said you always saw it once on 4.13, which clearly shows that something was wonky, but it managed to unblock itself. >> How about this incremental delta? > Here's the output > (XEN) IRET fault: #PF[0000] > (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > (XEN) IRET fault: #PF[0000] > (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > (XEN) IRET fault: #PF[0000] Ok, so the promotion definitely fails, but we don't get as far as inspecting the content of the LDT frame. This probably means it failed to change the page type, which probably means there are still outstanding writeable references. I'm expecting the final printk to be the one which triggers. ~Andrew diff --git a/xen/arch/x86/pv/mm.c b/xen/arch/x86/pv/mm.c index 5d74d11cba..2823dc2894 100644 --- a/xen/arch/x86/pv/mm.c +++ b/xen/arch/x86/pv/mm.c @@ -87,14 +87,23 @@ bool pv_map_ldt_shadow_page(unsigned int offset) gl1e = guest_get_eff_kern_l1e(linear); if ( unlikely(!(l1e_get_flags(gl1e) & _PAGE_PRESENT)) ) + { + printk(XENLOG_ERR "*** LDT: gl1e %"PRIpte" not present\n", gl1e.l1); return false; + } page = get_page_from_gfn(currd, l1e_get_pfn(gl1e), NULL, P2M_ALLOC); if ( unlikely(!page) ) + { + printk(XENLOG_ERR "*** LDT: failed to get gfn %05lx reference\n", + l1e_get_pfn(gl1e)); return false; + } if ( unlikely(!get_page_type(page, PGT_seg_desc_page)) ) { + printk(XENLOG_ERR "*** LDT: bad type: caf %016lx, taf=%016lx\n", + page->count_info, page->u.inuse.type_info); put_page(page); return false; } ^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 16:00 ` Andrew Cooper @ 2020-12-09 16:30 ` Manuel Bouyer 2020-12-09 18:08 ` Andrew Cooper 0 siblings, 1 reply; 25+ messages in thread From: Manuel Bouyer @ 2020-12-09 16:30 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Wed, Dec 09, 2020 at 04:00:02PM +0000, Andrew Cooper wrote: > [...] > >> I wonder if the LDT is set up correctly. > > I guess it is, otherwise it wouldn't boot with a Xen 4.13 kernel, isn't it ? > > Well - you said you always saw it once on 4.13, which clearly shows that > something was wonky, but it managed to unblock itself. > > >> How about this incremental delta? > > Here's the output > > (XEN) IRET fault: #PF[0000] > > (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 > > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > > (XEN) IRET fault: #PF[0000] > > (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 > > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > > (XEN) IRET fault: #PF[0000] > > Ok, so the promotion definitely fails, but we don't get as far as > inspecting the content of the LDT frame. This probably means it failed > to change the page type, which probably means there are still > outstanding writeable references. > > I'm expecting the final printk to be the one which triggers. It's not. Here's the output: (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 (XEN) *** LDT: gl1e 0000000000000000 not present (XEN) *** pv_map_ldt_shadow_page(0x40) failed (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 (XEN) *** LDT: gl1e 0000000000000000 not present (XEN) *** pv_map_ldt_shadow_page(0x40) failed (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 (XEN) *** LDT: gl1e 0000000000000000 not present (XEN) *** pv_map_ldt_shadow_page(0x40) failed (XEN) IRET fault: #PF[0000] (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 (XEN) domain_crash called from extable.c:219 (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.15-unstable x86_64 debug=y Tainted: C ]---- (XEN) CPU: 0 (XEN) RIP: 0047:[<00007f7f5dc007d0>] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: ffff82d04038c309 rbx: 0000000000000000 rcx: 000000000000e008 (XEN) rdx: 0000000000010086 rsi: ffff83007fcb7f78 rdi: 000000000000e010 (XEN) rbp: 0000000000000000 rsp: 00007f7fffcfc8d0 r8: 0000000e00000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000002660 (XEN) cr3: 0000000079cdb000 cr2: ffffbd000000a040 (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: ffffffff80cf2dc0 (XEN) ds: 0023 es: 0023 fs: 0000 gs: 0000 ss: 003f cs: 0047 (XEN) Guest stack trace from rsp=00007f7fffcfc8d0: (XEN) 0000000000000001 00007f7fffcfcde8 0000000000000000 0000000000000000 (XEN) 0000000000000003 000000000e200040 0000000000000004 0000000000000038 (XEN) 0000000000000005 0000000000000008 0000000000000006 0000000000001000 (XEN) 0000000000000007 00007f7f5dc00000 0000000000000008 0000000000000000 (XEN) 0000000000000009 000000000e201cd0 00000000000007d0 0000000000000000 (XEN) 00000000000007d1 0000000000000000 00000000000007d2 0000000000000000 (XEN) 00000000000007d3 0000000000000000 000000000000000d 00007f7fffcfd000 (XEN) 00000000000007de 00007f7fffcfc9d0 0000000000000000 0000000000000000 (XEN) 6e692f6e6962732f 0000000000007469 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 16:30 ` Manuel Bouyer @ 2020-12-09 18:08 ` Andrew Cooper 2020-12-09 18:57 ` Manuel Bouyer 0 siblings, 1 reply; 25+ messages in thread From: Andrew Cooper @ 2020-12-09 18:08 UTC (permalink / raw) To: Manuel Bouyer; +Cc: xen-devel On 09/12/2020 16:30, Manuel Bouyer wrote: > On Wed, Dec 09, 2020 at 04:00:02PM +0000, Andrew Cooper wrote: >> [...] >>>> I wonder if the LDT is set up correctly. >>> I guess it is, otherwise it wouldn't boot with a Xen 4.13 kernel, isn't it ? >> Well - you said you always saw it once on 4.13, which clearly shows that >> something was wonky, but it managed to unblock itself. >> >>>> How about this incremental delta? >>> Here's the output >>> (XEN) IRET fault: #PF[0000] >>> (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 >>> (XEN) *** pv_map_ldt_shadow_page(0x40) failed >>> (XEN) IRET fault: #PF[0000] >>> (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 >>> (XEN) *** pv_map_ldt_shadow_page(0x40) failed >>> (XEN) IRET fault: #PF[0000] >> Ok, so the promotion definitely fails, but we don't get as far as >> inspecting the content of the LDT frame. This probably means it failed >> to change the page type, which probably means there are still >> outstanding writeable references. >> >> I'm expecting the final printk to be the one which triggers. > It's not. > Here's the output: > (XEN) IRET fault: #PF[0000] > (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 > (XEN) *** LDT: gl1e 0000000000000000 not present > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > (XEN) IRET fault: #PF[0000] > (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 > (XEN) *** LDT: gl1e 0000000000000000 not present > (XEN) *** pv_map_ldt_shadow_page(0x40) failed Ok. So the mapping registered for the LDT is not yet present. Xen should be raising #PF with the guest, and would be in every case other than the weird context on IRET, where we've confused bad guest state with bad hypervisor state. diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index 3ac07a84c3..35c24ed668 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -1235,10 +1235,6 @@ static int handle_ldt_mapping_fault(unsigned int offset, { printk(XENLOG_ERR "*** pv_map_ldt_shadow_page(%#x) failed\n", offset); - /* In hypervisor mode? Leave it to the #PF handler to fix up. */ - if ( !guest_mode(regs) ) - return 0; - /* Access would have become non-canonical? Pass #GP[sel] back. */ if ( unlikely(!is_canonical_address(curr->arch.pv.ldt_base + offset)) ) { This bodge ought to cause a #PF to be delivered suitably, but may make other corner cases not quite work correctly, so isn't a clean fix. ~Andrew ^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 18:08 ` Andrew Cooper @ 2020-12-09 18:57 ` Manuel Bouyer 2020-12-09 19:08 ` Andrew Cooper 0 siblings, 1 reply; 25+ messages in thread From: Manuel Bouyer @ 2020-12-09 18:57 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Wed, Dec 09, 2020 at 06:08:53PM +0000, Andrew Cooper wrote: > On 09/12/2020 16:30, Manuel Bouyer wrote: > > On Wed, Dec 09, 2020 at 04:00:02PM +0000, Andrew Cooper wrote: > >> [...] > >>>> I wonder if the LDT is set up correctly. > >>> I guess it is, otherwise it wouldn't boot with a Xen 4.13 kernel, isn't it ? > >> Well - you said you always saw it once on 4.13, which clearly shows that > >> something was wonky, but it managed to unblock itself. > >> > >>>> How about this incremental delta? > >>> Here's the output > >>> (XEN) IRET fault: #PF[0000] > >>> (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 > >>> (XEN) *** pv_map_ldt_shadow_page(0x40) failed > >>> (XEN) IRET fault: #PF[0000] > >>> (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 > >>> (XEN) *** pv_map_ldt_shadow_page(0x40) failed > >>> (XEN) IRET fault: #PF[0000] > >> Ok, so the promotion definitely fails, but we don't get as far as > >> inspecting the content of the LDT frame. This probably means it failed > >> to change the page type, which probably means there are still > >> outstanding writeable references. > >> > >> I'm expecting the final printk to be the one which triggers. > > It's not. > > Here's the output: > > (XEN) IRET fault: #PF[0000] > > (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 > > (XEN) *** LDT: gl1e 0000000000000000 not present > > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > > (XEN) IRET fault: #PF[0000] > > (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 > > (XEN) *** LDT: gl1e 0000000000000000 not present > > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > > Ok. So the mapping registered for the LDT is not yet present. Xen > should be raising #PF with the guest, and would be in every case other > than the weird context on IRET, where we've confused bad guest state > with bad hypervisor state. Unfortunably it doesn't fix the problem. I'm now getting a loop of (XEN) *** LDT: gl1e 0000000000000000 not present (XEN) *** pv_map_ldt_shadow_page(0x40) failed -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 18:57 ` Manuel Bouyer @ 2020-12-09 19:08 ` Andrew Cooper 2020-12-10 9:51 ` Manuel Bouyer 0 siblings, 1 reply; 25+ messages in thread From: Andrew Cooper @ 2020-12-09 19:08 UTC (permalink / raw) To: Manuel Bouyer; +Cc: xen-devel On 09/12/2020 18:57, Manuel Bouyer wrote: > On Wed, Dec 09, 2020 at 06:08:53PM +0000, Andrew Cooper wrote: >> On 09/12/2020 16:30, Manuel Bouyer wrote: >>> On Wed, Dec 09, 2020 at 04:00:02PM +0000, Andrew Cooper wrote: >>>> [...] >>>>>> I wonder if the LDT is set up correctly. >>>>> I guess it is, otherwise it wouldn't boot with a Xen 4.13 kernel, isn't it ? >>>> Well - you said you always saw it once on 4.13, which clearly shows that >>>> something was wonky, but it managed to unblock itself. >>>> >>>>>> How about this incremental delta? >>>>> Here's the output >>>>> (XEN) IRET fault: #PF[0000] >>>>> (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 >>>>> (XEN) *** pv_map_ldt_shadow_page(0x40) failed >>>>> (XEN) IRET fault: #PF[0000] >>>>> (XEN) %cr2 ffff820000010040, LDT base ffffc4800000a000, limit 0057 >>>>> (XEN) *** pv_map_ldt_shadow_page(0x40) failed >>>>> (XEN) IRET fault: #PF[0000] >>>> Ok, so the promotion definitely fails, but we don't get as far as >>>> inspecting the content of the LDT frame. This probably means it failed >>>> to change the page type, which probably means there are still >>>> outstanding writeable references. >>>> >>>> I'm expecting the final printk to be the one which triggers. >>> It's not. >>> Here's the output: >>> (XEN) IRET fault: #PF[0000] >>> (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 >>> (XEN) *** LDT: gl1e 0000000000000000 not present >>> (XEN) *** pv_map_ldt_shadow_page(0x40) failed >>> (XEN) IRET fault: #PF[0000] >>> (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 >>> (XEN) *** LDT: gl1e 0000000000000000 not present >>> (XEN) *** pv_map_ldt_shadow_page(0x40) failed >> Ok. So the mapping registered for the LDT is not yet present. Xen >> should be raising #PF with the guest, and would be in every case other >> than the weird context on IRET, where we've confused bad guest state >> with bad hypervisor state. > Unfortunably it doesn't fix the problem. I'm now getting a loop of > (XEN) *** LDT: gl1e 0000000000000000 not present > (XEN) *** pv_map_ldt_shadow_page(0x40) failed Oh of course - we don't follow the exit-to-guest path on the way out here. As a gross hack to check that we've at least diagnosed the issue appropriately, could you modify NetBSD to explicitly load the %ss selector into %es (or any other free segment) before first entering user context? If it a sequence of LDT demand-faulting issues, that should cause them to be fully resolved before Xen's IRET becomes the first actual LDT load. ~Andrew ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-09 19:08 ` Andrew Cooper @ 2020-12-10 9:51 ` Manuel Bouyer 2020-12-10 10:41 ` Jan Beulich ` (2 more replies) 0 siblings, 3 replies; 25+ messages in thread From: Manuel Bouyer @ 2020-12-10 9:51 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Wed, Dec 09, 2020 at 07:08:41PM +0000, Andrew Cooper wrote: > Oh of course - we don't follow the exit-to-guest path on the way out here. > > As a gross hack to check that we've at least diagnosed the issue > appropriately, could you modify NetBSD to explicitly load the %ss > selector into %es (or any other free segment) before first entering user > context? If I understood it properly, the user %ss is loaded by Xen from the trapframe when the guest swictes from kernel to user mode, isn't it ? So you mean setting %es to the same value in the trapframe ? Actually I used %fs because %es is set equal to %ds. Xen 4.13 boots fine with this change, but with 4.15 I get a loop of: (XEN) *** LDT: gl1e 0000000000000000 not present (XEN) *** pv_map_ldt_shadow_page(0x40) failed [ 12.3586540] Process (pid 1) got sig 11 which means that the dom0 gets the trap, and decides that the fault address is not mapped. Without the change the dom0 doesn't show the "Process (pid 1) got sig 11" I activated the NetBSD trap debug code, and this shows: [ 6.7165877] kern.module.path=/stand/amd64-xen/9.1/modules (XEN) *** LDT: gl1e 0000000000000000 not present (XEN) *** pv_map_ldt_shadow_page(0x40) failed [ 6.9462322] pid 1.1 (init): signal 11 code=1 (trap 0x6) @rip 0x7f7ef0c007d0 a ddr 0xffffbd800000a040 error=14 [ 7.0647896] trapframe 0xffffbd80381cff00 [ 7.1126288] rip 0x00007f7ef0c007d0 rsp 0x00007f7fff10aa30 rfl 0x00000000000 00202 [ 7.2041518] rdi 000000000000000000 rsi 000000000000000000 rdx 0000000000000 00000 [ 7.2956758] rcx 000000000000000000 r8 000000000000000000 r9 0000000000000 00000 [ 7.3872013] r10 000000000000000000 r11 000000000000000000 r12 0000000000000 00000 [ 7.4787216] r13 000000000000000000 r14 000000000000000000 r15 0000000000000 00000 [ 7.5702439] rbp 000000000000000000 rbx 0x00007f7fff10afe0 rax 0000000000000 00000 [ 7.6617663] cs 0x47 ds 0x23 es 0x23 fs 0000 gs 0000 ss 0x3f [ 7.7345663] fsbase 000000000000000000 gsbase 000000000000000000 so it looks like something resets %fs to 0 ... Anyway the fault address 0xffffbd800000a040 is in the hypervisor's range, isn't it ? -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-10 9:51 ` Manuel Bouyer @ 2020-12-10 10:41 ` Jan Beulich 2020-12-10 15:51 ` Andrew Cooper 2020-12-11 8:58 ` Jan Beulich 2 siblings, 0 replies; 25+ messages in thread From: Jan Beulich @ 2020-12-10 10:41 UTC (permalink / raw) To: Manuel Bouyer, Andrew Cooper; +Cc: xen-devel On 10.12.2020 10:51, Manuel Bouyer wrote: > On Wed, Dec 09, 2020 at 07:08:41PM +0000, Andrew Cooper wrote: >> Oh of course - we don't follow the exit-to-guest path on the way out here. >> >> As a gross hack to check that we've at least diagnosed the issue >> appropriately, could you modify NetBSD to explicitly load the %ss >> selector into %es (or any other free segment) before first entering user >> context? > > If I understood it properly, the user %ss is loaded by Xen from the > trapframe when the guest swictes from kernel to user mode, isn't it ? > So you mean setting %es to the same value in the trapframe ? > > Actually I used %fs because %es is set equal to %ds. > Xen 4.13 boots fine with this change, but with 4.15 I get a loop of: > > > (XEN) *** LDT: gl1e 0000000000000000 not present > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > [ 12.3586540] Process (pid 1) got sig 11 > > which means that the dom0 gets the trap, and decides that the fault address > is not mapped. Without the change the dom0 doesn't show the > "Process (pid 1) got sig 11" > > I activated the NetBSD trap debug code, and this shows: > [ 6.7165877] kern.module.path=/stand/amd64-xen/9.1/modules (XEN) *** LDT: gl1e 0000000000000000 not present > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > [ 6.9462322] pid 1.1 (init): signal 11 code=1 (trap 0x6) @rip 0x7f7ef0c007d0 a > ddr 0xffffbd800000a040 error=14 > [ 7.0647896] trapframe 0xffffbd80381cff00 > [ 7.1126288] rip 0x00007f7ef0c007d0 rsp 0x00007f7fff10aa30 rfl 0x00000000000 > 00202 > [ 7.2041518] rdi 000000000000000000 rsi 000000000000000000 rdx 0000000000000 > 00000 > [ 7.2956758] rcx 000000000000000000 r8 000000000000000000 r9 0000000000000 > 00000 > [ 7.3872013] r10 000000000000000000 r11 000000000000000000 r12 0000000000000 > 00000 > [ 7.4787216] r13 000000000000000000 r14 000000000000000000 r15 0000000000000 > 00000 > [ 7.5702439] rbp 000000000000000000 rbx 0x00007f7fff10afe0 rax 0000000000000 > 00000 > [ 7.6617663] cs 0x47 ds 0x23 es 0x23 fs 0000 gs 0000 ss 0x3f > [ 7.7345663] fsbase 000000000000000000 gsbase 000000000000000000 > > so it looks like something resets %fs to 0 ... > > Anyway the fault address 0xffffbd800000a040 is in the hypervisor's range, > isn't it ? No, the hypervisor range is 0xffff800000000000-0xffff880000000000. Jan ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-10 9:51 ` Manuel Bouyer 2020-12-10 10:41 ` Jan Beulich @ 2020-12-10 15:51 ` Andrew Cooper 2020-12-10 17:03 ` Manuel Bouyer 2020-12-11 8:58 ` Jan Beulich 2 siblings, 1 reply; 25+ messages in thread From: Andrew Cooper @ 2020-12-10 15:51 UTC (permalink / raw) To: Manuel Bouyer; +Cc: xen-devel On 10/12/2020 09:51, Manuel Bouyer wrote: > On Wed, Dec 09, 2020 at 07:08:41PM +0000, Andrew Cooper wrote: >> Oh of course - we don't follow the exit-to-guest path on the way out here. >> >> As a gross hack to check that we've at least diagnosed the issue >> appropriately, could you modify NetBSD to explicitly load the %ss >> selector into %es (or any other free segment) before first entering user >> context? > If I understood it properly, the user %ss is loaded by Xen from the > trapframe when the guest swictes from kernel to user mode, isn't it ? Yes. The kernel involves HYPERCALL_iret, and Xen copies/audits the provided trapframe, and uses it to actually enter userspace. > So you mean setting %es to the same value in the trapframe ? Yes - specifically I wanted to force the LDT reference to happen in a context where demand-faulting should work, so all the mappings get set up properly before we first encounter the LDT reference in Xen's IRET instruction. And to be clear, there is definitely a bug needing fixing here in Xen in terms of handling IRET faults caused by guest state. However, it looks like this isn't the root of the problem - merely some very weird collateral damage. > Actually I used %fs because %es is set equal to %ds. > Xen 4.13 boots fine with this change, but with 4.15 I get a loop of: > > > (XEN) *** LDT: gl1e 0000000000000000 not present > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > [ 12.3586540] Process (pid 1) got sig 11 > > which means that the dom0 gets the trap, and decides that the fault address > is not mapped. Without the change the dom0 doesn't show the > "Process (pid 1) got sig 11" > > I activated the NetBSD trap debug code, and this shows: > [ 6.7165877] kern.module.path=/stand/amd64-xen/9.1/modules > (XEN) *** LDT: gl1e 0000000000000000 not present > (XEN) *** pv_map_ldt_shadow_page(0x40) failed > [ 6.9462322] pid 1.1 (init): signal 11 code=1 (trap 0x6) @rip 0x7f7ef0c007d0 addr 0xffffbd800000a040 error=14 > [ 7.0647896] trapframe 0xffffbd80381cff00 > [ 7.1126288] rip 0x00007f7ef0c007d0 rsp 0x00007f7fff10aa30 rfl 0x0000000000000202 > [ 7.2041518] rdi 000000000000000000 rsi 000000000000000000 rdx 000000000000000000 > [ 7.2956758] rcx 000000000000000000 r8 000000000000000000 r9 000000000000000000 > [ 7.3872013] r10 000000000000000000 r11 000000000000000000 r12 000000000000000000 > [ 7.4787216] r13 000000000000000000 r14 000000000000000000 r15 000000000000000000 > [ 7.5702439] rbp 000000000000000000 rbx 0x00007f7fff10afe0 rax 000000000000000000 > [ 7.6617663] cs 0x47 ds 0x23 es 0x23 fs 0000 gs 0000 ss 0x3f > [ 7.7345663] fsbase 000000000000000000 gsbase 000000000000000000 > > so it looks like something resets %fs to 0 ... > > Anyway the fault address 0xffffbd800000a040 is in the hypervisor's range, > isn't it ? No. Its the kernel's LDT. From previous debugging: > (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 LDT handling in Xen is a bit complicated. To maintain host safety, we must map it into Xen's range, and we explicitly support a PV guest doing on-demand mapping of the LDT. (This pertains to the experimental Windows XP PV support which never made it beyond a prototype. Windows can page out the LDT.) Either way, we lazily map the LDT frames on first use. So %cr2 is the real hardware faulting address, and is in the Xen range. We spot that it is an LDT access, and try to lazily map the frame (at LDT base), but find that the kernel's virtual address mapping 0xffffbd000000a000 is not present (the gl1e printk). Therefore, we pass #PF to the guest kernel, adjusting vCR2 to what would have happened had Xen not mapped the real LDT elsewhere, which is expected to cause the guest kernel to do whatever demand mapping is necessary to pull the LDT back in. I suppose it is worth taking a step back and ascertaining how exactly NetBSD handles (or, should be handling) the LDT. Do you mind elaborating on how it is supposed to work? ~Andrew ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-10 15:51 ` Andrew Cooper @ 2020-12-10 17:03 ` Manuel Bouyer 2020-12-10 17:18 ` Andrew Cooper 0 siblings, 1 reply; 25+ messages in thread From: Manuel Bouyer @ 2020-12-10 17:03 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Thu, Dec 10, 2020 at 03:51:46PM +0000, Andrew Cooper wrote: > > [ 7.6617663] cs 0x47 ds 0x23 es 0x23 fs 0000 gs 0000 ss 0x3f > > [ 7.7345663] fsbase 000000000000000000 gsbase 000000000000000000 > > > > so it looks like something resets %fs to 0 ... > > > > Anyway the fault address 0xffffbd800000a040 is in the hypervisor's range, > > isn't it ? > > No. Its the kernel's LDT. From previous debugging: > > (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 > > LDT handling in Xen is a bit complicated. To maintain host safety, we > must map it into Xen's range, and we explicitly support a PV guest doing > on-demand mapping of the LDT. (This pertains to the experimental > Windows XP PV support which never made it beyond a prototype. Windows > can page out the LDT.) Either way, we lazily map the LDT frames on > first use. > > So %cr2 is the real hardware faulting address, and is in the Xen range. > We spot that it is an LDT access, and try to lazily map the frame (at > LDT base), but find that the kernel's virtual address mapping > 0xffffbd000000a000 is not present (the gl1e printk). > > Therefore, we pass #PF to the guest kernel, adjusting vCR2 to what would > have happened had Xen not mapped the real LDT elsewhere, which is > expected to cause the guest kernel to do whatever demand mapping is > necessary to pull the LDT back in. > > > I suppose it is worth taking a step back and ascertaining how exactly > NetBSD handles (or, should be handling) the LDT. > > Do you mind elaborating on how it is supposed to work? Note that I'm not familiar with this selector stuff; and I usually get it wrong the first time I go back to it. AFAIK, in the Xen PV case, a page is allocated an mapped in kernel space, and registered to Xen with MMUEXT_SET_LDT. From what I found, in the common case the LDT is the same for all processes. Does it make sense ? -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-10 17:03 ` Manuel Bouyer @ 2020-12-10 17:18 ` Andrew Cooper 2020-12-10 17:35 ` Manuel Bouyer 0 siblings, 1 reply; 25+ messages in thread From: Andrew Cooper @ 2020-12-10 17:18 UTC (permalink / raw) To: Manuel Bouyer; +Cc: xen-devel On 10/12/2020 17:03, Manuel Bouyer wrote: > On Thu, Dec 10, 2020 at 03:51:46PM +0000, Andrew Cooper wrote: >>> [ 7.6617663] cs 0x47 ds 0x23 es 0x23 fs 0000 gs 0000 ss 0x3f >>> [ 7.7345663] fsbase 000000000000000000 gsbase 000000000000000000 >>> >>> so it looks like something resets %fs to 0 ... >>> >>> Anyway the fault address 0xffffbd800000a040 is in the hypervisor's range, >>> isn't it ? >> No. Its the kernel's LDT. From previous debugging: >>> (XEN) %cr2 ffff820000010040, LDT base ffffbd000000a000, limit 0057 >> LDT handling in Xen is a bit complicated. To maintain host safety, we >> must map it into Xen's range, and we explicitly support a PV guest doing >> on-demand mapping of the LDT. (This pertains to the experimental >> Windows XP PV support which never made it beyond a prototype. Windows >> can page out the LDT.) Either way, we lazily map the LDT frames on >> first use. >> >> So %cr2 is the real hardware faulting address, and is in the Xen range. >> We spot that it is an LDT access, and try to lazily map the frame (at >> LDT base), but find that the kernel's virtual address mapping >> 0xffffbd000000a000 is not present (the gl1e printk). >> >> Therefore, we pass #PF to the guest kernel, adjusting vCR2 to what would >> have happened had Xen not mapped the real LDT elsewhere, which is >> expected to cause the guest kernel to do whatever demand mapping is >> necessary to pull the LDT back in. >> >> >> I suppose it is worth taking a step back and ascertaining how exactly >> NetBSD handles (or, should be handling) the LDT. >> >> Do you mind elaborating on how it is supposed to work? > Note that I'm not familiar with this selector stuff; and I usually get > it wrong the first time I go back to it. > > AFAIK, in the Xen PV case, a page is allocated an mapped in kernel > space, and registered to Xen with MMUEXT_SET_LDT. > From what I found, in the common case the LDT is the same for all processes. > Does it make sense ? The debugging earlier shows that MMUEXT_SET_LDT has indeed been called. Presumably 0xffffbd000000a000 is a plausible virtual address for NetBSD to position the LDT? However, Xen finds the mapping not-present when trying to demand-map it, hence why the #PF is forwarded to the kernel. The way we pull guest virtual addresses was altered by XSA-286 (released not too long ago despite its apparent age), but *should* have been no functional change. I wonder if we accidentally broke something there. What exactly are you running, Xen-wise, with the 4.13 version? Given that this is init failing, presumably the issue would repro with the net installer version too? ~Andrew ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-10 17:18 ` Andrew Cooper @ 2020-12-10 17:35 ` Manuel Bouyer 2020-12-10 21:01 ` Andrew Cooper 0 siblings, 1 reply; 25+ messages in thread From: Manuel Bouyer @ 2020-12-10 17:35 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Thu, Dec 10, 2020 at 05:18:39PM +0000, Andrew Cooper wrote: > The debugging earlier shows that MMUEXT_SET_LDT has indeed been called. > Presumably 0xffffbd000000a000 is a plausible virtual address for NetBSD > to position the LDT? Yes, it is. > > However, Xen finds the mapping not-present when trying to demand-map it, > hence why the #PF is forwarded to the kernel. > > The way we pull guest virtual addresses was altered by XSA-286 (released > not too long ago despite its apparent age), but *should* have been no > functional change. I wonder if we accidentally broke something there. > What exactly are you running, Xen-wise, with the 4.13 version? It is 4.13.2, with the patch for XSA351 > > Given that this is init failing, presumably the issue would repro with > the net installer version too? Hopefully yes, maybe even as a domU. But I don't have a linux dom0 to test. If you have a Xen setup you can test with http://ftp.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz note that this won't boot as a dom0 kernel. -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-10 17:35 ` Manuel Bouyer @ 2020-12-10 21:01 ` Andrew Cooper 2020-12-11 10:47 ` Manuel Bouyer 0 siblings, 1 reply; 25+ messages in thread From: Andrew Cooper @ 2020-12-10 21:01 UTC (permalink / raw) To: Manuel Bouyer; +Cc: xen-devel On 10/12/2020 17:35, Manuel Bouyer wrote: > On Thu, Dec 10, 2020 at 05:18:39PM +0000, Andrew Cooper wrote: >> However, Xen finds the mapping not-present when trying to demand-map it, >> hence why the #PF is forwarded to the kernel. >> >> The way we pull guest virtual addresses was altered by XSA-286 (released >> not too long ago despite its apparent age), but *should* have been no >> functional change. I wonder if we accidentally broke something there. >> What exactly are you running, Xen-wise, with the 4.13 version? > It is 4.13.2, with the patch for XSA351 Thanks, >> Given that this is init failing, presumably the issue would repro with >> the net installer version too? > Hopefully yes, maybe even as a domU. But I don't have a linux dom0 to test. > > If you have a Xen setup you can test with > http://ftp.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz > > note that this won't boot as a dom0 kernel. I've repro'd the problem. When I modify Xen to explicitly demand-map the LDT in the MMUEXT_SET_LDT hypercall, everything works fine. Specifically, this delta: diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c index 723cc1070f..71a791d877 100644 --- a/xen/arch/x86/mm.c +++ b/xen/arch/x86/mm.c @@ -3742,12 +3742,31 @@ long do_mmuext_op( else if ( (curr->arch.pv.ldt_ents != ents) || (curr->arch.pv.ldt_base != ptr) ) { + unsigned int err = 0, tmp; + if ( pv_destroy_ldt(curr) ) flush_tlb_local(); curr->arch.pv.ldt_base = ptr; curr->arch.pv.ldt_ents = ents; load_LDT(curr); + + printk("Probe new LDT\n"); + asm volatile ( + "mov %%es, %[tmp];\n\t" + "1: mov %[sel], %%es;\n\t" + "mov %[tmp], %%es;\n\t" + "2:\n\t" + ".section .fixup,\"ax\"\n" + "3: mov $1, %[err];\n\t" + "jmp 2b\n\t" + ".previous\n\t" + _ASM_EXTABLE(1b, 3b) + : [err] "+r" (err), + [tmp] "=&r" (tmp) + : [sel] "r" (0x3f) + : "memory"); + printk(" => err %u\n", err); } break; } Which stashes %es, explicitly loads init's %ss selector to trigger the #PF and Xen's lazy mapping, then restores %es. (XEN) d1v0 Dropping PAT write of 0007010600070106 (XEN) Probe new LDT (XEN) *** LDT Successful map, slot 0 (XEN) => err 0 (XEN) d1 L1TF-vulnerable L4e 0000000801e88000 - Shadowing And the domain is up and running: # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 2656 8 r----- 44.6 netbsd 1 256 1 -b---- 5.3 (Probably confused about the fact I gave it no disk...) Now, in this case, we find that the virtual address provided for the LDT is mapped, so we successfully copy the mapping into Xen's area, and init runs happily. So the mystery is why the LDT virtual address is not-present when Xen tries to lazily map the LDT at the normal point... Presumably you've got no Meltdown mitigations going on within the NetBSD kernel? (I suspect not, seeing as changing Xen changes the behaviour, but it is worth asking). ~Andrew ^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-10 21:01 ` Andrew Cooper @ 2020-12-11 10:47 ` Manuel Bouyer 0 siblings, 0 replies; 25+ messages in thread From: Manuel Bouyer @ 2020-12-11 10:47 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel On Thu, Dec 10, 2020 at 09:01:12PM +0000, Andrew Cooper wrote: > I've repro'd the problem. > > When I modify Xen to explicitly demand-map the LDT in the MMUEXT_SET_LDT > hypercall, everything works fine. > > Specifically, this delta: > > diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c > index 723cc1070f..71a791d877 100644 > --- a/xen/arch/x86/mm.c > +++ b/xen/arch/x86/mm.c > @@ -3742,12 +3742,31 @@ long do_mmuext_op( > else if ( (curr->arch.pv.ldt_ents != ents) || > (curr->arch.pv.ldt_base != ptr) ) > { > + unsigned int err = 0, tmp; > + > if ( pv_destroy_ldt(curr) ) > flush_tlb_local(); > > curr->arch.pv.ldt_base = ptr; > curr->arch.pv.ldt_ents = ents; > load_LDT(curr); > + > + printk("Probe new LDT\n"); > + asm volatile ( > + "mov %%es, %[tmp];\n\t" > + "1: mov %[sel], %%es;\n\t" > + "mov %[tmp], %%es;\n\t" > + "2:\n\t" > + ".section .fixup,\"ax\"\n" > + "3: mov $1, %[err];\n\t" > + "jmp 2b\n\t" > + ".previous\n\t" > + _ASM_EXTABLE(1b, 3b) > + : [err] "+r" (err), > + [tmp] "=&r" (tmp) > + : [sel] "r" (0x3f) > + : "memory"); > + printk(" => err %u\n", err); > } > break; > } > > Which stashes %es, explicitly loads init's %ss selector to trigger the > #PF and Xen's lazy mapping, then restores %es. Yes, this works for dom0 too, I have it running multiuser > [...] > > Presumably you've got no Meltdown mitigations going on within the NetBSD > kernel? (I suspect not, seeing as changing Xen changes the behaviour, > but it is worth asking). No, there's no Meltdown mitigations for PV in NetBSD. as I see it, for amd64 at last, the Xen kernel has to do it anyway, so it's not usefull to implement it in the guest's kernel. Did I miss something ? -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-10 9:51 ` Manuel Bouyer 2020-12-10 10:41 ` Jan Beulich 2020-12-10 15:51 ` Andrew Cooper @ 2020-12-11 8:58 ` Jan Beulich 2020-12-11 11:15 ` Manuel Bouyer 2 siblings, 1 reply; 25+ messages in thread From: Jan Beulich @ 2020-12-11 8:58 UTC (permalink / raw) To: Manuel Bouyer; +Cc: xen-devel, Andrew Cooper On 10.12.2020 10:51, Manuel Bouyer wrote: > On Wed, Dec 09, 2020 at 07:08:41PM +0000, Andrew Cooper wrote: >> Oh of course - we don't follow the exit-to-guest path on the way out here. >> >> As a gross hack to check that we've at least diagnosed the issue >> appropriately, could you modify NetBSD to explicitly load the %ss >> selector into %es (or any other free segment) before first entering user >> context? > > If I understood it properly, the user %ss is loaded by Xen from the > trapframe when the guest swictes from kernel to user mode, isn't it ? > So you mean setting %es to the same value in the trapframe ? > > Actually I used %fs because %es is set equal to %ds. > Xen 4.13 boots fine with this change, but with 4.15 I get a loop of: > > > (XEN) *** LDT: gl1e 0000000000000000 not present > (XEN) *** pv_map_ldt_shadow_page(0x40) failed Could you please revert 9ff970564764 ("x86/mm: drop guest_get_eff_l1e()")? I think there was a thinko there in that the change can't be split from the bigger one which was part of the originally planned set for XSA-286. We mustn't avoid the switching of page tables as long as guest_get_eff{,_kern}_l1e() makes use of the linear page tables. Jan ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-11 8:58 ` Jan Beulich @ 2020-12-11 11:15 ` Manuel Bouyer 2020-12-11 13:56 ` Andrew Cooper 0 siblings, 1 reply; 25+ messages in thread From: Manuel Bouyer @ 2020-12-11 11:15 UTC (permalink / raw) To: Jan Beulich; +Cc: xen-devel, Andrew Cooper On Fri, Dec 11, 2020 at 09:58:54AM +0100, Jan Beulich wrote: > Could you please revert 9ff970564764 ("x86/mm: drop guest_get_eff_l1e()")? > I think there was a thinko there in that the change can't be split from > the bigger one which was part of the originally planned set for XSA-286. > We mustn't avoid the switching of page tables as long as > guest_get_eff{,_kern}_l1e() makes use of the linear page tables. Yes, reverting this commit also makes the dom0 boot. -- Manuel Bouyer <bouyer@antioche.eu.org> NetBSD: 26 ans d'experience feront toujours la difference -- ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: dom0 PV looping on search_pre_exception_table() 2020-12-11 11:15 ` Manuel Bouyer @ 2020-12-11 13:56 ` Andrew Cooper 0 siblings, 0 replies; 25+ messages in thread From: Andrew Cooper @ 2020-12-11 13:56 UTC (permalink / raw) To: Manuel Bouyer, Jan Beulich; +Cc: xen-devel On 11/12/2020 11:15, Manuel Bouyer wrote: > On Fri, Dec 11, 2020 at 09:58:54AM +0100, Jan Beulich wrote: >> Could you please revert 9ff970564764 ("x86/mm: drop guest_get_eff_l1e()")? >> I think there was a thinko there in that the change can't be split from >> the bigger one which was part of the originally planned set for XSA-286. >> We mustn't avoid the switching of page tables as long as >> guest_get_eff{,_kern}_l1e() makes use of the linear page tables. > Yes, reverting this commit also makes the dom0 boot. > This was going to be my next area of investigation. Thanks for confirming. In hindsight, the bug is very obvious... ~Andrew ^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2020-12-11 13:57 UTC | newest] Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-12-08 17:57 dom0 PV looping on search_pre_exception_table() Manuel Bouyer 2020-12-08 18:13 ` Andrew Cooper 2020-12-09 8:39 ` Jan Beulich 2020-12-09 9:49 ` Manuel Bouyer 2020-12-09 10:15 ` Manuel Bouyer 2020-12-09 13:28 ` Andrew Cooper 2020-12-09 13:59 ` Manuel Bouyer 2020-12-09 14:41 ` Andrew Cooper 2020-12-09 15:44 ` Manuel Bouyer 2020-12-09 16:00 ` Andrew Cooper 2020-12-09 16:30 ` Manuel Bouyer 2020-12-09 18:08 ` Andrew Cooper 2020-12-09 18:57 ` Manuel Bouyer 2020-12-09 19:08 ` Andrew Cooper 2020-12-10 9:51 ` Manuel Bouyer 2020-12-10 10:41 ` Jan Beulich 2020-12-10 15:51 ` Andrew Cooper 2020-12-10 17:03 ` Manuel Bouyer 2020-12-10 17:18 ` Andrew Cooper 2020-12-10 17:35 ` Manuel Bouyer 2020-12-10 21:01 ` Andrew Cooper 2020-12-11 10:47 ` Manuel Bouyer 2020-12-11 8:58 ` Jan Beulich 2020-12-11 11:15 ` Manuel Bouyer 2020-12-11 13:56 ` Andrew Cooper
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.