* Xen 4.6.1 crash with altp2m enabled by default
@ 2016-07-29 7:33 Kevin.Mayer
2016-07-29 9:57 ` Dario Faggioli
2016-07-29 10:05 ` Andrew Cooper
0 siblings, 2 replies; 20+ messages in thread
From: Kevin.Mayer @ 2016-07-29 7:33 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 5893 bytes --]
Hi guys
We are using Xen 4.6.1 to manage our virtual machines on x86-64-servers.
We start dozens of VMs and destroy them again after 60 seconds, which works fine as it is, but the next step in our approach requires the use of the altp2m functionality.
Since libvirt does not pass the altp2m-enable flag to the hypervisor we enabled altp2m unconditionally by patching the hvm.c . Since all of our machines support the altp2m this seemed to be ok.
d->arch.hvm_domain.params[HVM_PARAM_HPET_ENABLED] = 1;
d->arch.hvm_domain.params[HVM_PARAM_TRIPLE_FAULT_REASON] = SHUTDOWN_reboot;
+ d->arch.hvm_domain.params[HVM_PARAM_ALTP2M] = 1;
+
vpic_init(d);
rc = vioapic_init(d);
Since applying this patch the hypervisor crashes after several hundred restarted VMs (without any altp2m-functionality used by us) with the following dmesg:
(XEN) ----[ Xen-4.6.1 x86_64 debug=n Not tainted ]----
(XEN) CPU: 7
(XEN) RIP: e008:[<ffff82d0801f5a55>] vmx_vmenter_helper+0x2b5/0x340
(XEN) RFLAGS: 0000000000010003 CONTEXT: hypervisor (d0v3)
(XEN) rax: 000000008005003b rbx: ffff8300e7038000 rcx: 0000000000000008
(XEN) rdx: 0000000000006c00 rsi: ffff83062eb5e000 rdi: ffff8300e7038000
(XEN) rbp: ffff830c17e3f000 rsp: ffff830617fc7d70 r8: 0000000000000000
(XEN) r9: ffff83014f8d7028 r10: 000002700f858000 r11: 00002201be6861f0
(XEN) r12: ffff83062eb5e000 r13: ffff8300e752f000 r14: ffff82d08030ea40
(XEN) r15: 0000000000000007 cr0: 000000008005003b cr4: 00000000000026e0
(XEN) cr3: 00000001bf4da000 cr2: 00000000dd840c00
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830617fc7d70:
(XEN) ffff8300e7038000 ffff82d080170c04 0000000000000000 0000000780109f6a
(XEN) ffff830617fc7f18 ffff83000000001e 0000000000000000 ffff8300e752f19c
(XEN) 0000000000000286 ffff8300e752f000 ffff8300e72fc000 0000000000000007
(XEN) ffff830c17e3f000 ffff830c14ee1000 ffff82d08030ea40 ffff82d080173d6a
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) ffff82d08030ea40 ffff8300e72fc000 000002700f481091 0000000000000001
(XEN) ffff82d080324560 ffff82d08030ea40 ffff8300e752f000 ffff82d080128004
(XEN) 0000000000000001 0000000001c9c380 ffff830c14ef60e8 0000000017fce600
(XEN) 0000000000000001 ffff82d0801bd18b ffff82d0801d9e88 ffff8300e752f000
(XEN) 0000000001c9c380 ffff82d08012e700 0000006e00000171 ffffffffffffffff
(XEN) ffff830617fc0000 ffff82d0802f8f80 00000000ffffffff ffff83062eb5e000
(XEN) ffff82d08030ea40 ffff82d08012b040 ffff8300e7038000 ffff830617fc0000
(XEN) ffff8300e7038000 00000000ffffffff ffff830c14ee1000 ffff82d080170970
(XEN) ffff8300e72fc000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000080550f50 00000000ffdffc70 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 000000002fcffe19
(XEN) 00000000ffdffc70 0000000000000000 00000000ffdffc50 00000000853b0918
(XEN) 000000fa00000000 00000000f0e48162 0000000000000000 0000000000000246
(XEN) 0000000080550f34 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000007 ffff8300e752f000
(XEN) Xen call trace:
(XEN) [<ffff82d0801f5a55>] vmx_vmenter_helper+0x2b5/0x340
(XEN) [<ffff82d080170c04>] __context_switch+0xb4/0x350
(XEN) [<ffff82d080173d6a>] context_switch+0xca/0xef0
(XEN) [<ffff82d080128004>] schedule+0x264/0x5f0
(XEN) [<ffff82d0801bd18b>] mwait_idle+0x25b/0x3a0
(XEN) [<ffff82d0801d9e88>] hvm_vcpu_has_pending_irq+0x58/0xc0
(XEN) [<ffff82d08012e700>] timer_softirq_action+0x80/0x250
(XEN) [<ffff82d08012b040>] __do_softirq+0x60/0x90
(XEN) [<ffff82d080170970>] idle_loop+0x20/0x50
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 7:
(XEN) FATAL TRAP: vector = 6 (invalid opcode)
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) Executing kexec image on cpu7
(XEN) Shot down all CPUs
The RIP points to ud2
0xffff82d0801f5a55: ud2
>From the RFLAGS we concluded that the vmwrite failed due to an invalid vmcs-pointer (CF = 1), but this is where we are stuck since we have no idea how the pointer could have gotten corrupted.
crash> vcpu
gives vmcs = 0xffffffff817cbc20 for vcpu_id = 7,
and vcpus gives
VCID PCID VCPU ST T DOMID DOMAIN
0 0 ffff8300e75f2000 RU I 32767 ffff830c14ee1000
1 1 ffff8300e72fe000 RU I 32767 ffff830c14ee1000
2 2 ffff8300e7527000 RU I 32767 ffff830c14ee1000
> 3 3 ffff8300e7526000 RU I 32767 ffff830c14ee1000
4 4 ffff8300e75f1000 RU I 32767 ffff830c14ee1000
> 5 5 ffff8300e75f0000 RU I 32767 ffff830c14ee1000
> 6 6 ffff8300e72fd000 RU I 32767 ffff830c14ee1000
7 7 ffff8300e72fc000 RU I 32767 ffff830c14ee1000
0 0 ffff8300e72fa000 BL 0 0 ffff830c17e3f000
1 6 ffff8300e72f9000 BL 0 0 ffff830c17e3f000
2 3 ffff8300e72f8000 BL 0 0 ffff830c17e3f000
> 3 7 ffff8300e752f000 RU 0 0 ffff830c17e3f000
4 5 ffff8300e752e000 RU 0 0 ffff830c17e3f000
> 5 2 ffff8300e752d000 RU 0 0 ffff830c17e3f000
> 6 1 ffff8300e752c000 BL 0 0 ffff830c17e3f000
>* 7 0 ffff8300e752b000 RU 0 0 ffff830c17e3f000
0 4 ffff8300e7042000 OF U 127 ffff830475bbe000
> 0 4 ffff8300e7040000 RU U 128 ffff83062a7bc000
0 1 ffff8300e7038000 RU U 129 ffff83062eb5e000
0 5 ffff8300e703e000 BL U 130 ffff830475bd1000
Do you have any ideas what could cause this crash or how to proceed?
Cheers
Kevin
____________
Virus checked by G Data MailSecurity
Version: AVA 25.7615 dated 29.07.2016
Virus news: www.antiviruslab.com
[-- Attachment #1.2: Type: text/html, Size: 14595 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabled by default
2016-07-29 7:33 Xen 4.6.1 crash with altp2m enabled by default Kevin.Mayer
@ 2016-07-29 9:57 ` Dario Faggioli
2016-07-29 10:05 ` Andrew Cooper
1 sibling, 0 replies; 20+ messages in thread
From: Dario Faggioli @ 2016-07-29 9:57 UTC (permalink / raw)
To: Kevin.Mayer, xen-devel; +Cc: Andrew Cooper, Tamas K Lengyel, George Dunlap
[-- Attachment #1.1: Type: text/plain, Size: 855 bytes --]
On Fri, 2016-07-29 at 07:33 +0000, Kevin.Mayer@gdata.de wrote:
> Hi guys
>
Hi,
I'm pretty much just Cc-ing maintainers/key people, to see if they have
ideas.
Only one thing. Since you are rebuilding Xen anyway, I think it could
be helpful to try a debug build, and post the dump it will produce.
> (XEN) ----[ Xen-4.6.1 x86_64 debug=n Not tainted ]----
>
I.e., this would need to become debug=y.
Since you said you're using 4.6.x, I think putting "debug=y" in a
.config file (and then rebuilding and reinstalling, of course) should
be enough.
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabled by default
2016-07-29 7:33 Xen 4.6.1 crash with altp2m enabled by default Kevin.Mayer
2016-07-29 9:57 ` Dario Faggioli
@ 2016-07-29 10:05 ` Andrew Cooper
2016-08-02 11:45 ` Kevin.Mayer
1 sibling, 1 reply; 20+ messages in thread
From: Andrew Cooper @ 2016-07-29 10:05 UTC (permalink / raw)
To: Kevin.Mayer, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 6703 bytes --]
On 29/07/16 08:33, Kevin.Mayer@gdata.de wrote:
>
> Hi guys
>
>
>
> We are using Xen 4.6.1 to manage our virtual machines on x86-64-servers.
>
> We start dozens of VMs and destroy them again after 60 seconds, which
> works fine as it is, but the next step in our approach requires the
> use of the altp2m functionality.
>
> Since libvirt does not pass the altp2m-enable flag to the hypervisor
> we enabled altp2m unconditionally by patching the hvm.c . Since all of
> our machines support the altp2m this seemed to be ok.
>
altp2m is emulated in software when hardware support isn't available, so
it should work on all hardware (albeit with rather higher overhead).
>
>
> d->arch.hvm_domain.params[HVM_PARAM_HPET_ENABLED] = 1;
>
> d->arch.hvm_domain.params[HVM_PARAM_TRIPLE_FAULT_REASON] =
> SHUTDOWN_reboot;
>
> + d->arch.hvm_domain.params[HVM_PARAM_ALTP2M] = 1;
>
> +
>
This looks to be ok, given your situation.
> vpic_init(d);
>
> rc = vioapic_init(d);
>
>
>
> Since applying this patch the hypervisor crashes after several hundred
> restarted VMs (without any altp2m-functionality used by us) with the
> following dmesg:
>
>
>
> (XEN) ----[ Xen-4.6.1 x86_64 debug=n Not tainted ]----
>
As a start, please always use a debug hypervisor for investigating
issues like this.
> (XEN) CPU: 7
>
> (XEN) RIP: e008:[<ffff82d0801f5a55>] vmx_vmenter_helper+0x2b5/0x340
>
> (XEN) RFLAGS: 0000000000010003 CONTEXT: hypervisor (d0v3)
>
> (XEN) rax: 000000008005003b rbx: ffff8300e7038000 rcx:
> 0000000000000008
>
> (XEN) rdx: 0000000000006c00 rsi: ffff83062eb5e000 rdi:
> ffff8300e7038000
>
> (XEN) rbp: ffff830c17e3f000 rsp: ffff830617fc7d70 r8:
> 0000000000000000
>
> (XEN) r9: ffff83014f8d7028 r10: 000002700f858000 r11:
> 00002201be6861f0
>
> (XEN) r12: ffff83062eb5e000 r13: ffff8300e752f000 r14:
> ffff82d08030ea40
>
> (XEN) r15: 0000000000000007 cr0: 000000008005003b cr4:
> 00000000000026e0
>
> (XEN) cr3: 00000001bf4da000 cr2: 00000000dd840c00
>
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
>
> (XEN) Xen stack trace from rsp=ffff830617fc7d70:
>
> (XEN) ffff8300e7038000 ffff82d080170c04 0000000000000000
> 0000000780109f6a
>
> (XEN) ffff830617fc7f18 ffff83000000001e 0000000000000000
> ffff8300e752f19c
>
> (XEN) 0000000000000286 ffff8300e752f000 ffff8300e72fc000
> 0000000000000007
>
> (XEN) ffff830c17e3f000 ffff830c14ee1000 ffff82d08030ea40
> ffff82d080173d6a
>
> (XEN) 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000
>
> (XEN) ffff82d08030ea40 ffff8300e72fc000 000002700f481091
> 0000000000000001
>
> (XEN) ffff82d080324560 ffff82d08030ea40 ffff8300e752f000
> ffff82d080128004
>
> (XEN) 0000000000000001 0000000001c9c380 ffff830c14ef60e8
> 0000000017fce600
>
> (XEN) 0000000000000001 ffff82d0801bd18b ffff82d0801d9e88
> ffff8300e752f000
>
> (XEN) 0000000001c9c380 ffff82d08012e700 0000006e00000171
> ffffffffffffffff
>
> (XEN) ffff830617fc0000 ffff82d0802f8f80 00000000ffffffff
> ffff83062eb5e000
>
> (XEN) ffff82d08030ea40 ffff82d08012b040 ffff8300e7038000
> ffff830617fc0000
>
> (XEN) ffff8300e7038000 00000000ffffffff ffff830c14ee1000
> ffff82d080170970
>
> (XEN) ffff8300e72fc000 0000000000000000 0000000000000000
> 0000000000000000
>
> (XEN) 0000000000000000 0000000080550f50 00000000ffdffc70
> 0000000000000000
>
> (XEN) 0000000000000000 0000000000000000 0000000000000000
> 000000002fcffe19
>
> (XEN) 00000000ffdffc70 0000000000000000 00000000ffdffc50
> 00000000853b0918
>
> (XEN) 000000fa00000000 00000000f0e48162 0000000000000000
> 0000000000000246
>
> (XEN) 0000000080550f34 0000000000000000 0000000000000000
> 0000000000000000
>
> (XEN) 0000000000000000 0000000000000000 0000000000000007
> ffff8300e752f000
>
> (XEN) Xen call trace:
>
> (XEN) [<ffff82d0801f5a55>] vmx_vmenter_helper+0x2b5/0x340
>
> (XEN) [<ffff82d080170c04>] __context_switch+0xb4/0x350
>
> (XEN) [<ffff82d080173d6a>] context_switch+0xca/0xef0
>
> (XEN) [<ffff82d080128004>] schedule+0x264/0x5f0
>
> (XEN) [<ffff82d0801bd18b>] mwait_idle+0x25b/0x3a0
>
> (XEN) [<ffff82d0801d9e88>] hvm_vcpu_has_pending_irq+0x58/0xc0
>
> (XEN) [<ffff82d08012e700>] timer_softirq_action+0x80/0x250
>
> (XEN) [<ffff82d08012b040>] __do_softirq+0x60/0x90
>
> (XEN) [<ffff82d080170970>] idle_loop+0x20/0x50
>
> (XEN)
>
> (XEN)
>
> (XEN) ****************************************
>
> (XEN) Panic on CPU 7:
>
> (XEN) FATAL TRAP: vector = 6 (invalid opcode)
>
> (XEN) ****************************************
>
> (XEN)
>
> (XEN) Reboot in five seconds...
>
> (XEN) Executing kexec image on cpu7
>
> (XEN) Shot down all CPUs
>
>
>
> The RIP points to ud2
>
> 0xffff82d0801f5a55: ud2
>
> From the RFLAGS we concluded that the vmwrite failed due to an invalid
> vmcs-pointer (CF = 1), but this is where we are stuck since we have no
> idea how the pointer could have gotten corrupted.
>
> crash> vcpu
>
> gives vmcs = 0xffffffff817cbc20 for vcpu_id = 7,
>
>
>
> and vcpus gives
>
>
>
> VCID PCID VCPU ST T DOMID DOMAIN
>
> 0 0 ffff8300e75f2000 RU I 32767 ffff830c14ee1000
>
> 1 1 ffff8300e72fe000 RU I 32767 ffff830c14ee1000
>
> 2 2 ffff8300e7527000 RU I 32767 ffff830c14ee1000
>
> > 3 3 ffff8300e7526000 RU I 32767 ffff830c14ee1000
>
> 4 4 ffff8300e75f1000 RU I 32767 ffff830c14ee1000
>
> > 5 5 ffff8300e75f0000 RU I 32767 ffff830c14ee1000
>
> > 6 6 ffff8300e72fd000 RU I 32767 ffff830c14ee1000
>
> 7 7 ffff8300e72fc000 RU I 32767 ffff830c14ee1000
>
> 0 0 ffff8300e72fa000 BL 0 0 ffff830c17e3f000
>
> 1 6 ffff8300e72f9000 BL 0 0 ffff830c17e3f000
>
> 2 3 ffff8300e72f8000 BL 0 0 ffff830c17e3f000
>
> > 3 7 ffff8300e752f000 RU 0 0 ffff830c17e3f000
>
> 4 5 ffff8300e752e000 RU 0 0 ffff830c17e3f000
>
> > 5 2 ffff8300e752d000 RU 0 0 ffff830c17e3f000
>
> > 6 1 ffff8300e752c000 BL 0 0 ffff830c17e3f000
>
> >* 7 0 ffff8300e752b000 RU 0 0 ffff830c17e3f000
>
> 0 4 ffff8300e7042000 OF U 127 ffff830475bbe000
>
> > 0 4 ffff8300e7040000 RU U 128 ffff83062a7bc000
>
> 0 1 ffff8300e7038000 RU U 129 ffff83062eb5e000
>
> 0 5 ffff8300e703e000 BL U 130 ffff830475bd1000
>
>
>
> Do you have any ideas what could cause this crash or how to proceed?
>
As a start, use a debug hypervisor. That will get you accurate
backtraces, and you might get lucky and hit an earlier assertion. Can
you identify which domain this vmcs should belong to, and whether it is
in the process of being destroyed?
~Andrew
[-- Attachment #1.2: Type: text/html, Size: 15662 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabled by default
2016-07-29 10:05 ` Andrew Cooper
@ 2016-08-02 11:45 ` Kevin.Mayer
2016-08-02 12:34 ` Jan Beulich
0 siblings, 1 reply; 20+ messages in thread
From: Kevin.Mayer @ 2016-08-02 11:45 UTC (permalink / raw)
To: andrew.cooper3; +Cc: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 17412 bytes --]
Thanks for your reply.
I installed the debug hypervisor and got a new crash dump now.
I must confess that I have little to no experience debugging crash dumps, but this seems to be a different kind of error, or at least the way the error is reached is different.
The pattern with “page number X invalid” and the “restore” repeats for all preceding domains visible in the dump.
[…]
(XEN) memory.c:269:d164v0 Domain 164 page number 54fc invalid
(XEN) memory.c:269:d164v0 Domain 164 page number 54fd invalid
(XEN) grant_table.c:1491:d164v0 Expanding dom (164) grant table from (4) to (32) frames.
(XEN) Dom164 callback via changed to GSI 28
(XEN) HVM165 restore: VM saved on one CPU (0x206c2) and restored on another (0x106a5).
(XEN) HVM165 restore: CPU 0
(XEN) HVM165 restore: PIC 0
(XEN) HVM165 restore: PIC 1
(XEN) HVM165 restore: IOAPIC 0
(XEN) HVM165 restore: LAPIC 0
(XEN) HVM165 restore: LAPIC_REGS 0
(XEN) HVM165 restore: PCI_IRQ 0
(XEN) HVM165 restore: ISA_IRQ 0
(XEN) HVM165 restore: PCI_LINK 0
(XEN) HVM165 restore: PIT 0
(XEN) HVM165 restore: RTC 0
(XEN) HVM165 restore: HPET 0
(XEN) HVM165 restore: PMTIMER 0
(XEN) HVM165 restore: MTRR 0
(XEN) HVM165 restore: VMCE_VCPU 0
(XEN) HVM165 restore: TSC_ADJUST 0
(XEN) memory.c:269:d165v0 Domain 165 page number 54de invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54df invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54e0 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54e1 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54e2 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54e3 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54e4 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54e5 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54e6 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54e7 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54e8 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54e9 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54ea invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54eb invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54ec invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54ed invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54ee invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54ef invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54f0 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54f1 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54f2 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54f3 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54f4 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54f5 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54f6 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54f7 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54f8 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54f9 invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54fa invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54fb invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54fc invalid
(XEN) memory.c:269:d165v0 Domain 165 page number 54fd invalid
(XEN) grant_table.c:1491:d165v0 Expanding dom (165) grant table from (4) to (32) frames.
(XEN) Dom165 callback via changed to GSI 28
(XEN) Debugging connection not set up.
(XEN) ----[ Xen-4.6.1 x86_64 debug=y Not tainted ]----
(XEN) CPU: 6
(XEN) RIP: e008:[<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a
(XEN) RFLAGS: 0000000000010003 CONTEXT: hypervisor
(XEN) rax: 000000008005003b rbx: ffff8300e72fc000 rcx: 0000000000000000
(XEN) rdx: 0000000000006c00 rsi: ffff830617fd7fc0 rdi: ffff8300e6fc0000
(XEN) rbp: ffff830617fd7c40 rsp: ffff830617fd7c30 r8: 0000000000000000
(XEN) r9: ffff830be8dc9310 r10: 0000000000000000 r11: 00003475e9cf85d0
(XEN) r12: 0000000000000006 r13: ffff830c14ee1000 r14: ffff8300e6fc0000
(XEN) r15: ffff830617fd0000 cr0: 000000008005003b cr4: 00000000000026e0
(XEN) cr3: 00000001bd665000 cr2: 0000000004510000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830617fd7c30:
(XEN) ffff830617fd7c40 ffff8300e72fc000 ffff830617fd7ca0 ffff82d080174f91
(XEN) ffff830617fd7f18 ffff830be8dc9000 0000000000000286 ffff830617fd7c90
(XEN) 0000000000000206 0000000000000246 0000000000000001 ffff830617e91250
(XEN) ffff8300e72fc000 ffff830be8dc9000 ffff830617fd7cc0 ffff82d080178c19
(XEN) 0000000000bdeeae ffff8300e72fc000 ffff830617fd7cd0 ffff82d080178c3e
(XEN) ffff830617fd7d20 ffff82d080179740 ffff8300e6fc2000 ffff830c17e38e80
(XEN) ffff830617e91250 ffff820080000000 0000000000000002 ffff830617e91250
(XEN) ffff830617e91240 ffff830be8dc9000 ffff830617fd7d70 ffff82d080196152
(XEN) ffff830617fd7d50 ffff82d0801f7c6b ffff8300e6fc2000 ffff830617e91250
(XEN) ffff8300e6fc2000 ffff830617e91250 ffff830617e91240 ffff830be8dc9000
(XEN) ffff830617fd7d80 ffff82d080244a62 ffff830617fd7db0 ffff82d0801d3fe2
(XEN) ffff8300e6fc2000 0000000000000000 ffff830617e91f28 ffff830617e91000
(XEN) ffff830617fd7dd0 ffff82d080175c2c ffff8300e6fc2000 ffff8300e6fc2000
(XEN) ffff830617fd7e00 ffff82d080105dd4 ffff830c17e38040 0000000000000000
(XEN) 0000000000000000 ffff830617fd0000 ffff830617fd7e30 ffff82d0801215fd
(XEN) ffff8300e6fc0000 ffff82d080329280 ffff82d080328f80 fffffffffffffffd
(XEN) ffff830617fd7e60 ffff82d08012caf8 0000000000000006 ffff830c17e3bc60
(XEN) 0000000000000002 ffff830c17e3bbe0 ffff830617fd7e70 ffff82d08012cb3b
(XEN) ffff830617fd7ef0 ffff82d0801c23a8 ffff8300e72fc000 ffffffffffffffff
(XEN) ffff82d0801f3200 ffff830617fd7f08 ffff82d080329280 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a
(XEN) [<ffff82d080174f91>] __context_switch+0xdb/0x3b5
(XEN) [<ffff82d080178c19>] __sync_local_execstate+0x5e/0x7a
(XEN) [<ffff82d080178c3e>] sync_local_execstate+0x9/0xb
(XEN) [<ffff82d080179740>] map_domain_page+0xa0/0x5d4
(XEN) [<ffff82d080196152>] destroy_perdomain_mapping+0x8f/0x1e8
(XEN) [<ffff82d080244a62>] free_compat_arg_xlat+0x26/0x28
(XEN) [<ffff82d0801d3fe2>] hvm_vcpu_destroy+0x73/0xb0
(XEN) [<ffff82d080175c2c>] vcpu_destroy+0x5d/0x72
(XEN) [<ffff82d080105dd4>] complete_domain_destroy+0x49/0x192
(XEN) [<ffff82d0801215fd>] rcu_process_callbacks+0x19a/0x1fb
(XEN) [<ffff82d08012caf8>] __do_softirq+0x82/0x8d
(XEN) [<ffff82d08012cb3b>] process_pending_softirqs+0x38/0x3a
(XEN) [<ffff82d0801c23a8>] mwait_idle+0x10c/0x315
(XEN) [<ffff82d080174825>] idle_loop+0x51/0x6b
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 6:
(XEN) FATAL TRAP: vector = 6 (invalid opcode)
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) Debugging connection not set up.
(XEN) Executing kexec image on cpu6
(XEN) Shot down all CPUs
bt gives a longer backtrace for cpu 6 with an additional call ( #11 [ffff830617fd7d38] vmx_vcpu_update_eptp at ffff82d0801f7c6b ) between #12 ( free_compat_arg_xlat ) and #10 (destroy_perdomain_mapping ).
This additional call should not have happened according to the source code of free_compat_arg_xlat, so this kind of baffles me.
PCPU: 6 VCPU: ffff8300e72fc000
#0 [ffff830617fd7a90] kexec_crash at ffff82d080115bb9
#1 [ffff830617fd7ab0] panic at ffff82d080144202
#2 [ffff830617fd7b20] do_invalid_op at ffff82d0801a2bba
#3 [ffff830617fd7b30] pmt_update_time at ffff82d0801e0c88
#4 [ffff830617fd7b80] handle_exception_saved at ffff82d08024e5d0
#5 [ffff830617fd7c08] vmx_vmenter_helper at ffff82d0801fd23a
#6 [ffff830617fd7c48] __context_switch at ffff82d080174f91
#7 [ffff830617fd7ca8] __sync_local_execstate at ffff82d080178c19
#8 [ffff830617fd7cc8] sync_local_execstate at ffff82d080178c3e
#9 [ffff830617fd7cd8] map_domain_page at ffff82d080179740
#10 [ffff830617fd7d28] destroy_perdomain_mapping at ffff82d080196152
#11 [ffff830617fd7d38] vmx_vcpu_update_eptp at ffff82d0801f7c6b
#12 [ffff830617fd7d78] free_compat_arg_xlat at ffff82d080244a62
#13 [ffff830617fd7d88] hvm_vcpu_destroy at ffff82d0801d3fe2
#14 [ffff830617fd7db8] vcpu_destroy at ffff82d080175c2c
#15 [ffff830617fd7dd8] complete_domain_destroy at ffff82d080105dd4
#16 [ffff830617fd7e08] rcu_process_callbacks at ffff82d0801215fd
#17 [ffff830617fd7e38] __do_softirq at ffff82d08012caf8
#18 [ffff830617fd7e68] process_pending_softirqs at ffff82d08012cb3b
#19 [ffff830617fd7e78] mwait_idle at ffff82d0801c23a8
#20 [ffff830617fd7e90] vmx_intr_assist at ffff82d0801f3200
#21 [ffff830617fd7ef8] idle_loop at ffff82d080174825
#22 [ffff830617fd7f00] do_softirq at ffff82d08012cb50
Instead vmx_vcpu_update_eptp should be called before free_compat_arg_xlat by hvm_vcpu_destroy->altp2m_vcpu_destroy->altp2m_vcpu_update_p2m->hvm_funcs.altp2m_vcpu_update_p2m
Set by .altp2m_vcpu_update_p2m = vmx_vcpu_update_eptp in vmx.c
The vmcs of cpu 6 is 0x0
struct vcpu {
vcpu_id = 6,
processor = 6,
vcpu_info = 0x0,
domain = 0xffff830c14ee1000,
[...]
vmx = {
vmcs = 0x0,
vcpus
VCID PCID VCPU ST T DOMID DOMAIN
> 0 0 ffff8300e7557000 RU I 32767 ffff830c14ee1000
> 1 1 ffff8300e75f2000 RU I 32767 ffff830c14ee1000
2 2 ffff8300e72fe000 RU I 32767 ffff830c14ee1000
> 3 3 ffff8300e75f1000 RU I 32767 ffff830c14ee1000
> 4 4 ffff8300e75f0000 RU I 32767 ffff830c14ee1000
> 5 5 ffff8300e72fd000 RU I 32767 ffff830c14ee1000
>* 6 6 ffff8300e72fc000 RU I 32767 ffff830c14ee1000
> 7 7 ffff8300e72fb000 RU I 32767 ffff830c14ee1000
> 0 2 ffff8300e72f9000 RU 0 0 ffff830c17e32000
1 3 ffff8300e72f8000 BL 0 0 ffff830c17e32000
2 5 ffff8300e755f000 BL 0 0 ffff830c17e32000
3 0 ffff8300e755e000 BL 0 0 ffff830c17e32000
4 6 ffff8300e755d000 BL 0 0 ffff830c17e32000
5 4 ffff8300e755c000 BL 0 0 ffff830c17e32000
6 7 ffff8300e755b000 BL 0 0 ffff830c17e32000
7 5 ffff8300e755a000 BL 0 0 ffff830c17e32000
0 1 ffff8300e6fc7000 BL U 162 ffff830bdee8f000
0 3 ffff8300e6fc9000 BL U 163 ffff830be20d3000
0 6 ffff8300e6fc0000 BL U 164 ffff830be8dc9000
0 0 ffff8300e6fc6000 BL U 165 ffff830bd0cc0000
So in contrast to the last dump the crashing CPU is running DOMID 32767 (the Dom-0) if I understand the output correctly.
Kevin
Von: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
Gesendet: Freitag, 29. Juli 2016 12:05
An: Mayer, Kevin <Kevin.Mayer@gdata.de>; xen-devel@lists.xen.org
Betreff: Re: [Xen-devel] Xen 4.6.1 crash with altp2m enabled by default
On 29/07/16 08:33, Kevin.Mayer@gdata.de<mailto:Kevin.Mayer@gdata.de> wrote:
Hi guys
We are using Xen 4.6.1 to manage our virtual machines on x86-64-servers.
We start dozens of VMs and destroy them again after 60 seconds, which works fine as it is, but the next step in our approach requires the use of the altp2m functionality.
Since libvirt does not pass the altp2m-enable flag to the hypervisor we enabled altp2m unconditionally by patching the hvm.c . Since all of our machines support the altp2m this seemed to be ok.
altp2m is emulated in software when hardware support isn't available, so it should work on all hardware (albeit with rather higher overhead).
d->arch.hvm_domain.params[HVM_PARAM_HPET_ENABLED] = 1;
d->arch.hvm_domain.params[HVM_PARAM_TRIPLE_FAULT_REASON] = SHUTDOWN_reboot;
+ d->arch.hvm_domain.params[HVM_PARAM_ALTP2M] = 1;
+
This looks to be ok, given your situation.
vpic_init(d);
rc = vioapic_init(d);
Since applying this patch the hypervisor crashes after several hundred restarted VMs (without any altp2m-functionality used by us) with the following dmesg:
(XEN) ----[ Xen-4.6.1 x86_64 debug=n Not tainted ]----
As a start, please always use a debug hypervisor for investigating issues like this.
(XEN) CPU: 7
(XEN) RIP: e008:[<ffff82d0801f5a55>] vmx_vmenter_helper+0x2b5/0x340
(XEN) RFLAGS: 0000000000010003 CONTEXT: hypervisor (d0v3)
(XEN) rax: 000000008005003b rbx: ffff8300e7038000 rcx: 0000000000000008
(XEN) rdx: 0000000000006c00 rsi: ffff83062eb5e000 rdi: ffff8300e7038000
(XEN) rbp: ffff830c17e3f000 rsp: ffff830617fc7d70 r8: 0000000000000000
(XEN) r9: ffff83014f8d7028 r10: 000002700f858000 r11: 00002201be6861f0
(XEN) r12: ffff83062eb5e000 r13: ffff8300e752f000 r14: ffff82d08030ea40
(XEN) r15: 0000000000000007 cr0: 000000008005003b cr4: 00000000000026e0
(XEN) cr3: 00000001bf4da000 cr2: 00000000dd840c00
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff830617fc7d70:
(XEN) ffff8300e7038000 ffff82d080170c04 0000000000000000 0000000780109f6a
(XEN) ffff830617fc7f18 ffff83000000001e 0000000000000000 ffff8300e752f19c
(XEN) 0000000000000286 ffff8300e752f000 ffff8300e72fc000 0000000000000007
(XEN) ffff830c17e3f000 ffff830c14ee1000 ffff82d08030ea40 ffff82d080173d6a
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) ffff82d08030ea40 ffff8300e72fc000 000002700f481091 0000000000000001
(XEN) ffff82d080324560 ffff82d08030ea40 ffff8300e752f000 ffff82d080128004
(XEN) 0000000000000001 0000000001c9c380 ffff830c14ef60e8 0000000017fce600
(XEN) 0000000000000001 ffff82d0801bd18b ffff82d0801d9e88 ffff8300e752f000
(XEN) 0000000001c9c380 ffff82d08012e700 0000006e00000171 ffffffffffffffff
(XEN) ffff830617fc0000 ffff82d0802f8f80 00000000ffffffff ffff83062eb5e000
(XEN) ffff82d08030ea40 ffff82d08012b040 ffff8300e7038000 ffff830617fc0000
(XEN) ffff8300e7038000 00000000ffffffff ffff830c14ee1000 ffff82d080170970
(XEN) ffff8300e72fc000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000080550f50 00000000ffdffc70 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 000000002fcffe19
(XEN) 00000000ffdffc70 0000000000000000 00000000ffdffc50 00000000853b0918
(XEN) 000000fa00000000 00000000f0e48162 0000000000000000 0000000000000246
(XEN) 0000000080550f34 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000007 ffff8300e752f000
(XEN) Xen call trace:
(XEN) [<ffff82d0801f5a55>] vmx_vmenter_helper+0x2b5/0x340
(XEN) [<ffff82d080170c04>] __context_switch+0xb4/0x350
(XEN) [<ffff82d080173d6a>] context_switch+0xca/0xef0
(XEN) [<ffff82d080128004>] schedule+0x264/0x5f0
(XEN) [<ffff82d0801bd18b>] mwait_idle+0x25b/0x3a0
(XEN) [<ffff82d0801d9e88>] hvm_vcpu_has_pending_irq+0x58/0xc0
(XEN) [<ffff82d08012e700>] timer_softirq_action+0x80/0x250
(XEN) [<ffff82d08012b040>] __do_softirq+0x60/0x90
(XEN) [<ffff82d080170970>] idle_loop+0x20/0x50
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 7:
(XEN) FATAL TRAP: vector = 6 (invalid opcode)
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) Executing kexec image on cpu7
(XEN) Shot down all CPUs
The RIP points to ud2
0xffff82d0801f5a55: ud2
From the RFLAGS we concluded that the vmwrite failed due to an invalid vmcs-pointer (CF = 1), but this is where we are stuck since we have no idea how the pointer could have gotten corrupted.
crash> vcpu
gives vmcs = 0xffffffff817cbc20 for vcpu_id = 7,
and vcpus gives
VCID PCID VCPU ST T DOMID DOMAIN
0 0 ffff8300e75f2000 RU I 32767 ffff830c14ee1000
1 1 ffff8300e72fe000 RU I 32767 ffff830c14ee1000
2 2 ffff8300e7527000 RU I 32767 ffff830c14ee1000
> 3 3 ffff8300e7526000 RU I 32767 ffff830c14ee1000
4 4 ffff8300e75f1000 RU I 32767 ffff830c14ee1000
> 5 5 ffff8300e75f0000 RU I 32767 ffff830c14ee1000
> 6 6 ffff8300e72fd000 RU I 32767 ffff830c14ee1000
7 7 ffff8300e72fc000 RU I 32767 ffff830c14ee1000
0 0 ffff8300e72fa000 BL 0 0 ffff830c17e3f000
1 6 ffff8300e72f9000 BL 0 0 ffff830c17e3f000
2 3 ffff8300e72f8000 BL 0 0 ffff830c17e3f000
> 3 7 ffff8300e752f000 RU 0 0 ffff830c17e3f000
4 5 ffff8300e752e000 RU 0 0 ffff830c17e3f000
> 5 2 ffff8300e752d000 RU 0 0 ffff830c17e3f000
> 6 1 ffff8300e752c000 BL 0 0 ffff830c17e3f000
>* 7 0 ffff8300e752b000 RU 0 0 ffff830c17e3f000
0 4 ffff8300e7042000 OF U 127 ffff830475bbe000
> 0 4 ffff8300e7040000 RU U 128 ffff83062a7bc000
0 1 ffff8300e7038000 RU U 129 ffff83062eb5e000
0 5 ffff8300e703e000 BL U 130 ffff830475bd1000
Do you have any ideas what could cause this crash or how to proceed?
As a start, use a debug hypervisor. That will get you accurate backtraces, and you might get lucky and hit an earlier assertion. Can you identify which domain this vmcs should belong to, and whether it is in the process of being destroyed?
~Andrew
____________
Virus checked by G Data MailSecurity
Version: AVA 25.7688 dated 02.08.2016
Virus news: www.antiviruslab.com
[-- Attachment #1.2: Type: text/html, Size: 49561 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabled by default
2016-08-02 11:45 ` Kevin.Mayer
@ 2016-08-02 12:34 ` Jan Beulich
2016-08-03 13:24 ` Kevin.Mayer
0 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2016-08-02 12:34 UTC (permalink / raw)
To: Kevin.Mayer; +Cc: andrew.cooper3, xen-devel
>>> On 02.08.16 at 13:45, <Kevin.Mayer@gdata.de> wrote:
> (XEN) ----[ Xen-4.6.1 x86_64 debug=y Not tainted ]----
> (XEN) CPU: 6
> (XEN) RIP: e008:[<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a
> (XEN) RFLAGS: 0000000000010003 CONTEXT: hypervisor
> (XEN) rax: 000000008005003b rbx: ffff8300e72fc000 rcx: 0000000000000000
> (XEN) rdx: 0000000000006c00 rsi: ffff830617fd7fc0 rdi: ffff8300e6fc0000
> (XEN) rbp: ffff830617fd7c40 rsp: ffff830617fd7c30 r8: 0000000000000000
> (XEN) r9: ffff830be8dc9310 r10: 0000000000000000 r11: 00003475e9cf85d0
> (XEN) r12: 0000000000000006 r13: ffff830c14ee1000 r14: ffff8300e6fc0000
> (XEN) r15: ffff830617fd0000 cr0: 000000008005003b cr4: 00000000000026e0
> (XEN) cr3: 00000001bd665000 cr2: 0000000004510000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
> (XEN) Xen stack trace from rsp=ffff830617fd7c30:
> (XEN) ffff830617fd7c40 ffff8300e72fc000 ffff830617fd7ca0 ffff82d080174f91
> (XEN) ffff830617fd7f18 ffff830be8dc9000 0000000000000286 ffff830617fd7c90
> (XEN) 0000000000000206 0000000000000246 0000000000000001 ffff830617e91250
> (XEN) ffff8300e72fc000 ffff830be8dc9000 ffff830617fd7cc0 ffff82d080178c19
> (XEN) 0000000000bdeeae ffff8300e72fc000 ffff830617fd7cd0 ffff82d080178c3e
> (XEN) ffff830617fd7d20 ffff82d080179740 ffff8300e6fc2000 ffff830c17e38e80
> (XEN) ffff830617e91250 ffff820080000000 0000000000000002 ffff830617e91250
> (XEN) ffff830617e91240 ffff830be8dc9000 ffff830617fd7d70 ffff82d080196152
> (XEN) ffff830617fd7d50 ffff82d0801f7c6b ffff8300e6fc2000 ffff830617e91250
> (XEN) ffff8300e6fc2000 ffff830617e91250 ffff830617e91240 ffff830be8dc9000
> (XEN) ffff830617fd7d80 ffff82d080244a62 ffff830617fd7db0 ffff82d0801d3fe2
> (XEN) ffff8300e6fc2000 0000000000000000 ffff830617e91f28 ffff830617e91000
> (XEN) ffff830617fd7dd0 ffff82d080175c2c ffff8300e6fc2000 ffff8300e6fc2000
> (XEN) ffff830617fd7e00 ffff82d080105dd4 ffff830c17e38040 0000000000000000
> (XEN) 0000000000000000 ffff830617fd0000 ffff830617fd7e30 ffff82d0801215fd
> (XEN) ffff8300e6fc0000 ffff82d080329280 ffff82d080328f80 fffffffffffffffd
> (XEN) ffff830617fd7e60 ffff82d08012caf8 0000000000000006 ffff830c17e3bc60
> (XEN) 0000000000000002 ffff830c17e3bbe0 ffff830617fd7e70 ffff82d08012cb3b
> (XEN) ffff830617fd7ef0 ffff82d0801c23a8 ffff8300e72fc000 ffffffffffffffff
> (XEN) ffff82d0801f3200 ffff830617fd7f08 ffff82d080329280 0000000000000000
> (XEN) Xen call trace:
> (XEN) [<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a
> (XEN) [<ffff82d080174f91>] __context_switch+0xdb/0x3b5
> (XEN) [<ffff82d080178c19>] __sync_local_execstate+0x5e/0x7a
> (XEN) [<ffff82d080178c3e>] sync_local_execstate+0x9/0xb
> (XEN) [<ffff82d080179740>] map_domain_page+0xa0/0x5d4
> (XEN) [<ffff82d080196152>] destroy_perdomain_mapping+0x8f/0x1e8
> (XEN) [<ffff82d080244a62>] free_compat_arg_xlat+0x26/0x28
> (XEN) [<ffff82d0801d3fe2>] hvm_vcpu_destroy+0x73/0xb0
> (XEN) [<ffff82d080175c2c>] vcpu_destroy+0x5d/0x72
> (XEN) [<ffff82d080105dd4>] complete_domain_destroy+0x49/0x192
> (XEN) [<ffff82d0801215fd>] rcu_process_callbacks+0x19a/0x1fb
> (XEN) [<ffff82d08012caf8>] __do_softirq+0x82/0x8d
> (XEN) [<ffff82d08012cb3b>] process_pending_softirqs+0x38/0x3a
> (XEN) [<ffff82d0801c23a8>] mwait_idle+0x10c/0x315
> (XEN) [<ffff82d080174825>] idle_loop+0x51/0x6b
On this deep a stack execution can't validly end up in
vmx_vmenter_helper: That's a function called only when the stack
is almost empty. Nor is the caller of it the context switch code.
Hence your problem starts quite a bit earlier - perhaps memory
corruption?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabled by default
2016-08-02 12:34 ` Jan Beulich
@ 2016-08-03 13:24 ` Kevin.Mayer
2016-08-03 13:54 ` Jan Beulich
0 siblings, 1 reply; 20+ messages in thread
From: Kevin.Mayer @ 2016-08-03 13:24 UTC (permalink / raw)
To: JBeulich; +Cc: andrew.cooper3, xen-devel
Hi guys
I got around to take a closer look at the crash dump today.
tl;dr:
You were right, vmx_vmenter_helper is not called at all in the call stack.
The real reason behind the [<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a should be a failed
__vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0); in static void vmx_fpu_leave(struct vcpu *v).
Long story in "Chapter1".
Concerning the stray vmx_vcpu_update_eptp:
This seems to be leftovers (either due to a corrupted stack or simply uninitialized local variables) of previous function calls originating in hvm_vcpu_destroy.
More precisely:
if ( hvm_altp2m_supported() )
altp2m_vcpu_destroy(v);
being called BEFORE free_compat_arg_xlat.
I assume some kind of error in the altp2m_vcpu_destroy-path to be responsible for the crash, but I have no idea where and how to start investigating.
Long story in "Chapter2".
Chapter1:
I started with a function in the callstack and followed the assembly code to deduce where the (XEN) [<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a comes from:
sync_local_execstate:
0xffff82d080178c36 <sync_local_execstate+1>: mov %rsp,%rbp
0xffff82d080178c39 <sync_local_execstate+4>: callq 0xffff82d080178bbb <__sync_local_execstate>
0xffff82d080178c3e <sync_local_execstate+9>: pop %rbp
__sync_local_execstate:
0xffff82d080178c09 <__sync_local_execstate+78>: cmp %rsi,0x7fe8(%rax)
0xffff82d080178c10 <__sync_local_execstate+85>: je 0xffff82d080178c14 <__sync_local_execstate+89>
0xffff82d080178c12 <__sync_local_execstate+87>: ud2
0xffff82d080178c13 <__sync_local_execstate+88>: or %eax,%ebp
0xffff82d080178c15 <__sync_local_execstate+90>: popfq
0xffff82d080178c16 <__sync_local_execstate+91>: retq $0xffff
0xffff82d080178c19 <__sync_local_execstate+94>: and $0x200,%ebx
Here crash / gdb seem to get confused with the je.
crash> x /3i __sync_local_execstate+89
0xffff82d080178c14 <__sync_local_execstate+89>: callq 0xffff82d080174eb6 <__context_switch>
0xffff82d080178c19 <__sync_local_execstate+94>: and $0x200,%ebx
0xffff82d080178c1f <__sync_local_execstate+100>: pushfq
It seems this code calls the __context_switch:
switch_required = (this_cpu(curr_vcpu) != current);
if ( switch_required )
{
ASSERT(current == idle_vcpu[smp_processor_id()]);
__context_switch();
}
Up to the __context_switch everything seems to be running as it should. Except for the stray
[ffff830617fd7d38] vmx_vcpu_update_eptp at ffff82d0801f7c6b
which can be found in the "crash> bt" output but not in the "dmesg".
__context_switch:
0xffff82d080174f7f <__context_switch+201>: mov %r14,%rdi
0xffff82d080174f82 <__context_switch+204>: callq 0xffff82d08017c474 <vcpu_save_fpu>
0xffff82d080174f87 <__context_switch+209>: mov %r14,%rdi
0xffff82d080174f8a <__context_switch+212>: callq *0x3a8(%r14)
Following r14 / rdi ( 0xffff8300e6fc0000 ) as given in the crash dump seemingly leads to a vtable with a function pointer at the offset 0x3a8:
0xffff82d0801fa06e
crash> x /i 0xffff82d0801fa06e
0xffff82d0801fa06e <vmx_ctxt_switch_from>: push %rbp
This call, which does not show up in the backtrace, is expected at this position when looking at the C-code:
static void __context_switch(void)
[...]
if ( !is_idle_domain(pd) )
{
memcpy(&p->arch.user_regs, stack_regs, CTXT_SWITCH_STACK_BYTES);
vcpu_save_fpu(p);
p->arch.ctxt_switch_from(p);
[...]
as it is set in:
static int vmx_vcpu_initialise(struct vcpu *v)
[...]
v->arch.ctxt_switch_from = vmx_ctxt_switch_from;
[...]
Finally at:
0xffff82d0801fa0c3 <vmx_ctxt_switch_from+85>: mov $0x6c00,%edx
0xffff82d0801fa0c8 <vmx_ctxt_switch_from+90>: vmwrite %rax,%rdx
0xffff82d0801fa0cb <vmx_ctxt_switch_from+93>: jbe 0xffff82d0801fd23a
The jump to [<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a (ud2 following vmx_vmenter_helper) is done.
vmx_ctxt_switch_from is rather short in C and the called static functions are inlined.
static void vmx_ctxt_switch_from(struct vcpu *v)
{
/*
* Return early if trying to do a context switch without VMX enabled,
* this can happen when the hypervisor shuts down with HVM guests
* still running.
*/
if ( unlikely(!this_cpu(vmxon)) )
return;
vmx_fpu_leave(v);
vmx_save_guest_msrs(v);
vmx_restore_host_msrs();
vmx_save_dr(v);
}
The unlikely path is not taken and the two ud2 (I assume the ud2 are the ASSERTs in vmx_fpu_leave?) are not reached either:
0xffff82d0801fa077 <vmx_ctxt_switch_from+9>: lea 0x15c692(%rip),%rax # 0xffff82d080356710 <per_cpu__vmxon>
0xffff82d0801fa07e <vmx_ctxt_switch_from+16>: mov %rsp,%rdx
0xffff82d0801fa081 <vmx_ctxt_switch_from+19>: and $0xffffffffffff8000,%rdx
0xffff82d0801fa088 <vmx_ctxt_switch_from+26>: mov 0x7ff0(%rdx),%rdx
0xffff82d0801fa08f <vmx_ctxt_switch_from+33>: cmpb $0x0,(%rdx,%rax,1)
0xffff82d0801fa093 <vmx_ctxt_switch_from+37>: je 0xffff82d0801fa1d9 <vmx_ctxt_switch_from+363>
0xffff82d0801fa099 <vmx_ctxt_switch_from+43>: cmpb $0x0,0x109(%rdi)
0xffff82d0801fa0a0 <vmx_ctxt_switch_from+50>: je 0xffff82d0801fa0a4 <vmx_ctxt_switch_from+54>
0xffff82d0801fa0a2 <vmx_ctxt_switch_from+52>: ud2
0xffff82d0801fa0a3 <vmx_ctxt_switch_from+53>: or (%rdi),%ecx
0xffff82d0801fa0a5 <vmx_ctxt_switch_from+55>: and %al,%al
0xffff82d0801fa0a7 <vmx_ctxt_switch_from+57>: test $0x8,%al
0xffff82d0801fa0a9 <vmx_ctxt_switch_from+59>: jne 0xffff82d0801fa0ad <vmx_ctxt_switch_from+63>
0xffff82d0801fa0ab <vmx_ctxt_switch_from+61>: ud2
0xffff82d0801fa0ac <vmx_ctxt_switch_from+62>: or -0x75(%rax),%ecx
0xffff82d0801fa0af <vmx_ctxt_switch_from+65>: xchg %eax,(%rax)
0xffff82d0801fa0b1 <vmx_ctxt_switch_from+67>: (bad)
0xffff82d0801fa0b2 <vmx_ctxt_switch_from+68>: add %al,(%rax)
0xffff82d0801fa0b4 <vmx_ctxt_switch_from+70>: test $0x8,%al
0xffff82d0801fa0b6 <vmx_ctxt_switch_from+72>: jne 0xffff82d0801fa0d1 <vmx_ctxt_switch_from+99>
0xffff82d0801fa0b8 <vmx_ctxt_switch_from+74>: or $0x8,%rax
0xffff82d0801fa0bc <vmx_ctxt_switch_from+78>: mov %rax,0x700(%rdi)
0xffff82d0801fa0c3 <vmx_ctxt_switch_from+85>: mov $0x6c00,%edx
0xffff82d0801fa0c8 <vmx_ctxt_switch_from+90>: vmwrite %rax,%rdx
0xffff82d0801fa0cb <vmx_ctxt_switch_from+93>: jbe 0xffff82d0801fd23a
The two test, jne should be the if ( !(v->arch.hvm_vmx.host_cr0 & X86_CR0_TS) ) and if ( !(v->arch.hvm_vcpu.guest_cr[0] & X86_CR0_TS) ) conditions.
mov $0x6c00,%edx
vmwrite %rax,%rdx
Should be the
__vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0);
So the real reason the hypervisor panics should be a failing __vmwrite in
static void vmx_fpu_leave(struct vcpu *v)
which simply jumps to a location behind vmx_vmenter_helper, thereby creating a slightly confusing stack trace.
Chapter2:
This does not seem to have anything to do with the altp2m so I looked at the stray vmx_vcpu_update_eptp which can be seen in the bt, but not in the xen dmesg.
#10 [ffff830617fd7d28] destroy_perdomain_mapping at ffff82d080196152
#11 [ffff830617fd7d38] vmx_vcpu_update_eptp at ffff82d0801f7c6b
#12 [ffff830617fd7d78] free_compat_arg_xlat at ffff82d080244a62
This function is not called by free_compat_arg_xlat:
void free_compat_arg_xlat(struct vcpu *v)
{
destroy_perdomain_mapping(v->domain, ARG_XLAT_START(v),
PFN_UP(COMPAT_ARG_XLAT_SIZE));
}
Instead it is called BEFORE free_compat_arg_xlat in hvm_vcpu_destroy->altp2m_vcpu_destroy->altp2m_vcpu_update_p2m->hvm_funcs.altp2m_vcpu_update_p2m (.altp2m_vcpu_update_p2m = vmx_vcpu_update_eptp,) :
void hvm_vcpu_destroy(struct vcpu *v)
{
hvm_all_ioreq_servers_remove_vcpu(v->domain, v);
if ( hvm_altp2m_supported() )
altp2m_vcpu_destroy(v);
nestedhvm_vcpu_destroy(v);
free_compat_arg_xlat(v);
[...]
Here the enabling of the altp2m has an effect, but I have no idea how it could lead to a failing __vmwrite.
Any ideas where in the altp2m-code the error could be, or how I could help in finding it?
Cheers
Kevin
> -----Ursprüngliche Nachricht-----
> Von: Jan Beulich [mailto:JBeulich@suse.com]
> Gesendet: Dienstag, 2. August 2016 14:34
> An: Mayer, Kevin <Kevin.Mayer@gdata.de>
> Cc: andrew.cooper3@citrix.com; xen-devel@lists.xen.org
> Betreff: Re: [Xen-devel] Xen 4.6.1 crash with altp2m enabled by default
>
> >>> On 02.08.16 at 13:45, <Kevin.Mayer@gdata.de> wrote:
> > (XEN) ----[ Xen-4.6.1 x86_64 debug=y Not tainted ]----
> > (XEN) CPU: 6
> > (XEN) RIP: e008:[<ffff82d0801fd23a>]
> vmx_vmenter_helper+0x27e/0x30a
> > (XEN) RFLAGS: 0000000000010003 CONTEXT: hypervisor
> > (XEN) rax: 000000008005003b rbx: ffff8300e72fc000 rcx:
> 0000000000000000
> > (XEN) rdx: 0000000000006c00 rsi: ffff830617fd7fc0 rdi: ffff8300e6fc0000
> > (XEN) rbp: ffff830617fd7c40 rsp: ffff830617fd7c30 r8: 0000000000000000
> > (XEN) r9: ffff830be8dc9310 r10: 0000000000000000 r11: 00003475e9cf85d0
> > (XEN) r12: 0000000000000006 r13: ffff830c14ee1000 r14: ffff8300e6fc0000
> > (XEN) r15: ffff830617fd0000 cr0: 000000008005003b cr4:
> 00000000000026e0
> > (XEN) cr3: 00000001bd665000 cr2: 0000000004510000
> > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
> > (XEN) Xen stack trace from rsp=ffff830617fd7c30:
> > (XEN) ffff830617fd7c40 ffff8300e72fc000 ffff830617fd7ca0
> ffff82d080174f91
> > (XEN) ffff830617fd7f18 ffff830be8dc9000 0000000000000286
> ffff830617fd7c90
> > (XEN) 0000000000000206 0000000000000246 0000000000000001
> ffff830617e91250
> > (XEN) ffff8300e72fc000 ffff830be8dc9000 ffff830617fd7cc0
> ffff82d080178c19
> > (XEN) 0000000000bdeeae ffff8300e72fc000 ffff830617fd7cd0
> ffff82d080178c3e
> > (XEN) ffff830617fd7d20 ffff82d080179740 ffff8300e6fc2000
> ffff830c17e38e80
> > (XEN) ffff830617e91250 ffff820080000000 0000000000000002
> ffff830617e91250
> > (XEN) ffff830617e91240 ffff830be8dc9000 ffff830617fd7d70
> ffff82d080196152
> > (XEN) ffff830617fd7d50 ffff82d0801f7c6b ffff8300e6fc2000
> ffff830617e91250
> > (XEN) ffff8300e6fc2000 ffff830617e91250 ffff830617e91240
> ffff830be8dc9000
> > (XEN) ffff830617fd7d80 ffff82d080244a62 ffff830617fd7db0
> ffff82d0801d3fe2
> > (XEN) ffff8300e6fc2000 0000000000000000 ffff830617e91f28
> ffff830617e91000
> > (XEN) ffff830617fd7dd0 ffff82d080175c2c ffff8300e6fc2000
> ffff8300e6fc2000
> > (XEN) ffff830617fd7e00 ffff82d080105dd4 ffff830c17e38040
> 0000000000000000
> > (XEN) 0000000000000000 ffff830617fd0000 ffff830617fd7e30
> ffff82d0801215fd
> > (XEN) ffff8300e6fc0000 ffff82d080329280 ffff82d080328f80 fffffffffffffffd
> > (XEN) ffff830617fd7e60 ffff82d08012caf8 0000000000000006
> ffff830c17e3bc60
> > (XEN) 0000000000000002 ffff830c17e3bbe0 ffff830617fd7e70
> ffff82d08012cb3b
> > (XEN) ffff830617fd7ef0 ffff82d0801c23a8 ffff8300e72fc000 ffffffffffffffff
> > (XEN) ffff82d0801f3200 ffff830617fd7f08 ffff82d080329280
> 0000000000000000
> > (XEN) Xen call trace:
> > (XEN) [<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a
> > (XEN) [<ffff82d080174f91>] __context_switch+0xdb/0x3b5
> > (XEN) [<ffff82d080178c19>] __sync_local_execstate+0x5e/0x7a
> > (XEN) [<ffff82d080178c3e>] sync_local_execstate+0x9/0xb
> > (XEN) [<ffff82d080179740>] map_domain_page+0xa0/0x5d4
> > (XEN) [<ffff82d080196152>] destroy_perdomain_mapping+0x8f/0x1e8
> > (XEN) [<ffff82d080244a62>] free_compat_arg_xlat+0x26/0x28
> > (XEN) [<ffff82d0801d3fe2>] hvm_vcpu_destroy+0x73/0xb0
> > (XEN) [<ffff82d080175c2c>] vcpu_destroy+0x5d/0x72
> > (XEN) [<ffff82d080105dd4>] complete_domain_destroy+0x49/0x192
> > (XEN) [<ffff82d0801215fd>] rcu_process_callbacks+0x19a/0x1fb
> > (XEN) [<ffff82d08012caf8>] __do_softirq+0x82/0x8d
> > (XEN) [<ffff82d08012cb3b>] process_pending_softirqs+0x38/0x3a
> > (XEN) [<ffff82d0801c23a8>] mwait_idle+0x10c/0x315
> > (XEN) [<ffff82d080174825>] idle_loop+0x51/0x6b
>
> On this deep a stack execution can't validly end up in
> vmx_vmenter_helper: That's a function called only when the stack is almost
> empty. Nor is the caller of it the context switch code.
> Hence your problem starts quite a bit earlier - perhaps memory corruption?
>
> Jan
____________
Virus checked by G Data MailSecurity
Version: AVA 25.7708 dated 03.08.2016
Virus news: www.antiviruslab.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabled by default
2016-08-03 13:24 ` Kevin.Mayer
@ 2016-08-03 13:54 ` Jan Beulich
2016-08-04 15:08 ` Kevin.Mayer
0 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2016-08-03 13:54 UTC (permalink / raw)
To: Kevin.Mayer; +Cc: andrew.cooper3, xen-devel
>>> On 03.08.16 at 15:24, <Kevin.Mayer@gdata.de> wrote:
> I got around to take a closer look at the crash dump today.
>
> tl;dr:
> You were right, vmx_vmenter_helper is not called at all in the call stack.
> The real reason behind the [<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a
> should be a failed
> __vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0); in static void
> vmx_fpu_leave(struct vcpu *v).
Ah - that's what you get for not using most recent code, and what
I get for not considering the effect of you being on 4.6.x. In any
event - the call stack is then fine, and you'll want to figure out
which bit(s) of the new CR0 value are in conflict with the rest of the
active VMCS.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabled by default
2016-08-03 13:54 ` Jan Beulich
@ 2016-08-04 15:08 ` Kevin.Mayer
2016-08-04 15:35 ` Jan Beulich
0 siblings, 1 reply; 20+ messages in thread
From: Kevin.Mayer @ 2016-08-04 15:08 UTC (permalink / raw)
To: JBeulich; +Cc: andrew.cooper3, xen-devel
According to the crash-dump ( output of vcpu ) the v->arch.hvm_vmx.host_cr0 is " 0 ".
This cannot be the correct result because of
if ( !(v->arch.hvm_vmx.host_cr0 & X86_CR0_TS) )
{
v->arch.hvm_vmx.host_cr0 |= X86_CR0_TS;
__vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0);
}
It should at least be 0x8.
Also the v->arch.hvm_vmx.vmcs is " 0 " which I assume leads to the crash.
Since I assumed that somehow the wrong VCPU is used I tried to find the correct one:
vcpus gives
VCID PCID VCPU ST T DOMID DOMAIN
> 0 0 ffff8300e7557000 RU I 32767 ffff830c14ee1000
> 1 1 ffff8300e75f2000 RU I 32767 ffff830c14ee1000
2 2 ffff8300e72fe000 RU I 32767 ffff830c14ee1000
> 3 3 ffff8300e75f1000 RU I 32767 ffff830c14ee1000
> 4 4 ffff8300e75f0000 RU I 32767 ffff830c14ee1000
> 5 5 ffff8300e72fd000 RU I 32767 ffff830c14ee1000
>* 6 6 ffff8300e72fc000 RU I 32767 ffff830c14ee1000
> 7 7 ffff8300e72fb000 RU I 32767 ffff830c14ee1000
> 0 2 ffff8300e72f9000 RU 0 0 ffff830c17e32000
1 3 ffff8300e72f8000 BL 0 0 ffff830c17e32000
2 5 ffff8300e755f000 BL 0 0 ffff830c17e32000
3 0 ffff8300e755e000 BL 0 0 ffff830c17e32000
4 6 ffff8300e755d000 BL 0 0 ffff830c17e32000
5 4 ffff8300e755c000 BL 0 0 ffff830c17e32000
6 7 ffff8300e755b000 BL 0 0 ffff830c17e32000
7 5 ffff8300e755a000 BL 0 0 ffff830c17e32000
0 1 ffff8300e6fc7000 BL U 162 ffff830bdee8f000
0 3 ffff8300e6fc9000 BL U 163 ffff830be20d3000
0 6 ffff8300e6fc0000 BL U 164 ffff830be8dc9000
0 0 ffff8300e6fc6000 BL U 165 ffff830bd0cc0000
Since I see the domain ffff830be8dc9000 all over the xen dmesg this should be the correct VCPU.
On this CPU the v->arch.hvm_vmx.host_cr0 is 2147811387 (0x 8005003B) which corresponds to the cr0 in the xen dmesg.
v->arch.hvm_vmx.vmcs is 0xffff830bd0da1000
crash> x /10x 0xffff830bd0da1000
0xffff830bd0da1000: 0x000000000000000e 0x0000000000000000
0xffff830bd0da1010: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1020: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1030: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1040: 0x0000000000000000 0x0000000000000000
So the vmcs revision id is 0xe.
rdmsr 0x480 (the IA32_VMX_BASIC MSR ) gives da04000000000e which confirms the revision ID.
Size should be 0x400 bytes.
crash> x /130x 0xffff830bd0da1000
0xffff830bd0da1000: 0x000000000000000e 0x0000000000000000
0xffff830bd0da1010: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1020: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1030: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1040: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1050: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1060: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1070: 0x0000000000000000 0x0000000bd0da3000
0xffff830bd0da1080: 0x0000000c17e36000 0x0000000000000000
0xffff830bd0da1090: 0x0000000000000000 0x0000000000000000
0xffff830bd0da10a0: 0x00000000e7512000 0x00000000e7513000
0xffff830bd0da10b0: 0x0000000bd0da0000 0x0000000000000000
0xffff830bd0da10c0: 0x0000000000000000 0x0000000000000000
0xffff830bd0da10d0: 0x0000000000000000 0x0000006fedea809b
0xffff830bd0da10e0: 0x00000001a379e000 0x0000000610f9101e
0xffff830bd0da10f0: 0x0000000000000000 0xffffffffffffffff
0xffff830bd0da1100: 0x0000000000000000 0x0007010600070106
0xffff830bd0da1110: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1120: 0x0000006bb6a075fa 0x000600420000003f
0xffff830bd0da1130: 0x0000000000000000 0x000fefff00000000
0xffff830bd0da1140: 0x0000000000000000 0x00000000000051ff
0xffff830bd0da1150: 0x0000000000000041 0x0000000000000000
0xffff830bd0da1160: 0x0000000000000000 0x0000000c00000000
0xffff830bd0da1170: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1180: 0x0000000000000001 0x0000000000000000
0xffff830bd0da1190: 0x0000000800000000 0x0000000000000000
0xffff830bd0da11a0: 0x0000000000000001 0x0000000000000096
0xffff830bd0da11b0: 0xffff82d0802bc208 0x00000000806f6dbc
0xffff830bd0da11c0: 0x0000000000000000 0x0000000000000400
0xffff830bd0da11d0: 0x0000000080550f34 0x00000000f0e48161
0xffff830bd0da11e0: 0x0000000000000246 0x0000000000000000
0xffff830bd0da11f0: 0x00000000f79c3000 0x00000000804de6f0
0xffff830bd0da1200: 0x0000000000000023 0x0000000000000000
0xffff830bd0da1210: 0x00c0f300ffffffff 0x0000000000000008
0xffff830bd0da1220: 0x0000000000000000 0x00c09b00ffffffff
0xffff830bd0da1230: 0x0000000000000010 0x0000000000000000
0xffff830bd0da1240: 0x00c09300ffffffff 0x0000000000000023
0xffff830bd0da1250: 0x0000000000000000 0x00c0f300ffffffff
0xffff830bd0da1260: 0x0000000000000030 0x00000000ffdff000
0xffff830bd0da1270: 0x00c0930000001fff 0x0000000000000000
0xffff830bd0da1280: 0x0000000000000000 0x01c00000ffffffff
0xffff830bd0da1290: 0x0000000000000000 0x0000000000000000
0xffff830bd0da12a0: 0x01c00000ffffffff 0x0000000000000028
0xffff830bd0da12b0: 0x0000000080042000 0x00008b00000020ab
0xffff830bd0da12c0: 0x000000008003f000 0x000000008003f400
0xffff830bd0da12d0: 0x000007ff000003ff 0x000000008001003b
0xffff830bd0da12e0: 0x0000000000039000 0x00000000000026d9
0xffff830bd0da12f0: 0x000000000000dc3c 0x0000000000000000
0xffff830bd0da1300: 0x0000e00800000000 0x0000000000000000
0xffff830bd0da1310: 0x0000000000000000 0x000000000000e040
0xffff830bd0da1320: 0x0000050100070406 0x0000000000000000
0xffff830bd0da1330: 0x0000000000000000 0x0000000080050033
0xffff830bd0da1340: 0x00000001bd665000 0x00000000000026e0
0xffff830bd0da1350: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1360: 0xffff830c17e38c80 0xffff830617fd3000
0xffff830bd0da1370: 0xffff830617fcf000 0xffff830617fd7fc0
0xffff830bd0da1380: 0xffff82d08024e150 0xffff830617fd7f90
0xffff830bd0da1390: 0xffff82d080201bb0 0x000000000000e008
0xffff830bd0da13a0: 0x0000006000000000 0x0000000000000000
0xffff830bd0da13b0: 0x0000000000000000 0x0000000000000000
0xffff830bd0da13c0: 0xffffffffffffffff 0xffffffffffffffff
0xffff830bd0da13d0: 0x000000008001003b 0x00000000000006d9
0xffff830bd0da13e0: 0x0000000000000000 0x0000000000000000
0xffff830bd0da13f0: 0x0000000000000000 0x0000000000000000
0xffff830bd0da1400: 0x0000000000000000 0x0000000000000000
I don't quite understand the Intel developer manual at this point. How do I have to read this data?
Since if ( !(v->arch.hvm_vmx.host_cr0 & X86_CR0_TS) ) must be true I assume the __vmwrite tries to | 0x8 into the host_cr0 leading to the 0x0000000080050033 for the current host_cr0 ( or better the 0x80050033 ).
Or at least this is what I think was intended to happen.
> -----Ursprüngliche Nachricht-----
> Von: Jan Beulich [mailto:JBeulich@suse.com]
> Gesendet: Mittwoch, 3. August 2016 15:54
> An: Mayer, Kevin <Kevin.Mayer@gdata.de>
> Cc: andrew.cooper3@citrix.com; xen-devel@lists.xen.org
> Betreff: Re: AW: [Xen-devel] Xen 4.6.1 crash with altp2m enabled by default
>
> >>> On 03.08.16 at 15:24, <Kevin.Mayer@gdata.de> wrote:
> > I got around to take a closer look at the crash dump today.
> >
> > tl;dr:
> > You were right, vmx_vmenter_helper is not called at all in the call stack.
> > The real reason behind the [<ffff82d0801fd23a>]
> > vmx_vmenter_helper+0x27e/0x30a should be a failed
> __vmwrite(HOST_CR0,
> > v->arch.hvm_vmx.host_cr0); in static void vmx_fpu_leave(struct vcpu
> > *v).
>
> Ah - that's what you get for not using most recent code, and what I get for
> not considering the effect of you being on 4.6.x. In any event - the call stack
> is then fine, and you'll want to figure out which bit(s) of the new CR0 value
> are in conflict with the rest of the active VMCS.
>
> Jan
____________
Virus checked by G Data MailSecurity
Version: AVA 25.7724 dated 04.08.2016
Virus news: www.antiviruslab.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabled by default
2016-08-04 15:08 ` Kevin.Mayer
@ 2016-08-04 15:35 ` Jan Beulich
2016-08-05 12:51 ` Xen 4.6.1 crash with altp2m enabled bydefault Kevin.Mayer
0 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2016-08-04 15:35 UTC (permalink / raw)
To: Kevin.Mayer; +Cc: andrew.cooper3, xen-devel
>>> On 04.08.16 at 17:08, <Kevin.Mayer@gdata.de> wrote:
> crash> x /130x 0xffff830bd0da1000
> 0xffff830bd0da1000: 0x000000000000000e 0x0000000000000000
> 0xffff830bd0da1010: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da1020: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da1030: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da1040: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da1050: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da1060: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da1070: 0x0000000000000000 0x0000000bd0da3000
> 0xffff830bd0da1080: 0x0000000c17e36000 0x0000000000000000
> 0xffff830bd0da1090: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da10a0: 0x00000000e7512000 0x00000000e7513000
> 0xffff830bd0da10b0: 0x0000000bd0da0000 0x0000000000000000
> 0xffff830bd0da10c0: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da10d0: 0x0000000000000000 0x0000006fedea809b
> 0xffff830bd0da10e0: 0x00000001a379e000 0x0000000610f9101e
> 0xffff830bd0da10f0: 0x0000000000000000 0xffffffffffffffff
> 0xffff830bd0da1100: 0x0000000000000000 0x0007010600070106
> 0xffff830bd0da1110: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da1120: 0x0000006bb6a075fa 0x000600420000003f
> 0xffff830bd0da1130: 0x0000000000000000 0x000fefff00000000
> 0xffff830bd0da1140: 0x0000000000000000 0x00000000000051ff
> 0xffff830bd0da1150: 0x0000000000000041 0x0000000000000000
> 0xffff830bd0da1160: 0x0000000000000000 0x0000000c00000000
> 0xffff830bd0da1170: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da1180: 0x0000000000000001 0x0000000000000000
> 0xffff830bd0da1190: 0x0000000800000000 0x0000000000000000
> 0xffff830bd0da11a0: 0x0000000000000001 0x0000000000000096
> 0xffff830bd0da11b0: 0xffff82d0802bc208 0x00000000806f6dbc
> 0xffff830bd0da11c0: 0x0000000000000000 0x0000000000000400
> 0xffff830bd0da11d0: 0x0000000080550f34 0x00000000f0e48161
> 0xffff830bd0da11e0: 0x0000000000000246 0x0000000000000000
> 0xffff830bd0da11f0: 0x00000000f79c3000 0x00000000804de6f0
> 0xffff830bd0da1200: 0x0000000000000023 0x0000000000000000
> 0xffff830bd0da1210: 0x00c0f300ffffffff 0x0000000000000008
> 0xffff830bd0da1220: 0x0000000000000000 0x00c09b00ffffffff
> 0xffff830bd0da1230: 0x0000000000000010 0x0000000000000000
> 0xffff830bd0da1240: 0x00c09300ffffffff 0x0000000000000023
> 0xffff830bd0da1250: 0x0000000000000000 0x00c0f300ffffffff
> 0xffff830bd0da1260: 0x0000000000000030 0x00000000ffdff000
> 0xffff830bd0da1270: 0x00c0930000001fff 0x0000000000000000
> 0xffff830bd0da1280: 0x0000000000000000 0x01c00000ffffffff
> 0xffff830bd0da1290: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da12a0: 0x01c00000ffffffff 0x0000000000000028
> 0xffff830bd0da12b0: 0x0000000080042000 0x00008b00000020ab
> 0xffff830bd0da12c0: 0x000000008003f000 0x000000008003f400
> 0xffff830bd0da12d0: 0x000007ff000003ff 0x000000008001003b
> 0xffff830bd0da12e0: 0x0000000000039000 0x00000000000026d9
> 0xffff830bd0da12f0: 0x000000000000dc3c 0x0000000000000000
> 0xffff830bd0da1300: 0x0000e00800000000 0x0000000000000000
> 0xffff830bd0da1310: 0x0000000000000000 0x000000000000e040
> 0xffff830bd0da1320: 0x0000050100070406 0x0000000000000000
> 0xffff830bd0da1330: 0x0000000000000000 0x0000000080050033
> 0xffff830bd0da1340: 0x00000001bd665000 0x00000000000026e0
> 0xffff830bd0da1350: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da1360: 0xffff830c17e38c80 0xffff830617fd3000
> 0xffff830bd0da1370: 0xffff830617fcf000 0xffff830617fd7fc0
> 0xffff830bd0da1380: 0xffff82d08024e150 0xffff830617fd7f90
> 0xffff830bd0da1390: 0xffff82d080201bb0 0x000000000000e008
> 0xffff830bd0da13a0: 0x0000006000000000 0x0000000000000000
> 0xffff830bd0da13b0: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da13c0: 0xffffffffffffffff 0xffffffffffffffff
> 0xffff830bd0da13d0: 0x000000008001003b 0x00000000000006d9
> 0xffff830bd0da13e0: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da13f0: 0x0000000000000000 0x0000000000000000
> 0xffff830bd0da1400: 0x0000000000000000 0x0000000000000000
>
> I don't quite understand the Intel developer manual at this point. How do I
> have to read this data?
I don't think this is formally specified anywhere (publicly). After all that's
why one has to use vmread/vmwrite.
> Since if ( !(v->arch.hvm_vmx.host_cr0 & X86_CR0_TS) ) must be true I assume the
> __vmwrite tries to | 0x8 into the host_cr0 leading to the 0x0000000080050033
> for the current host_cr0 ( or better the 0x80050033 ).
Well, together with the disassembly it should be possible without
consulting the crash dump to tell what value it was that was
attempted to be written (the disassembly tells you which register
and the state dumped to the log tells you the value). If it is (as
you indicated earlier up) indeed zero that gets written, then
you'd want to try to find out where that zero is coming from.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabled bydefault
2016-08-04 15:35 ` Jan Beulich
@ 2016-08-05 12:51 ` Kevin.Mayer
2016-08-05 14:48 ` Jan Beulich
0 siblings, 1 reply; 20+ messages in thread
From: Kevin.Mayer @ 2016-08-05 12:51 UTC (permalink / raw)
To: JBeulich; +Cc: andrew.cooper3, xen-devel
According to the xen dmesg
(XEN) RIP: e008:[<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a
(XEN) RFLAGS: 0000000000010003 CONTEXT: hypervisor
(XEN) rax: 000000008005003b rbx: ffff8300e72fc000 rcx: 0000000000000000
(XEN) rdx: 0000000000006c00 rsi: ffff830617fd7fc0 rdi: ffff8300e6fc0000
(XEN) rbp: ffff830617fd7c40 rsp: ffff830617fd7c30 r8: 0000000000000000
(XEN) r9: ffff830be8dc9310 r10: 0000000000000000 r11: 00003475e9cf85d0
(XEN) r12: 0000000000000006 r13: ffff830c14ee1000 r14: ffff8300e6fc0000
(XEN) r15: ffff830617fd0000 cr0: 000000008005003b cr4: 00000000000026e0
(XEN) cr3: 00000001bd665000 cr2: 0000000004510000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
0xffff82d0801fa0c3 <vmx_ctxt_switch_from+85>: mov $0x6c00,%edx
0xffff82d0801fa0c8 <vmx_ctxt_switch_from+90>: vmwrite %rax,%rdx
The vmwrite tries to write 0x000000008005003b to 0x6c00.
But the active VCPU has a 0-vmcs-pointer.
> -----Ursprüngliche Nachricht-----
> Von: Jan Beulich [mailto:JBeulich@suse.com]
> Gesendet: Donnerstag, 4. August 2016 17:36
> An: Mayer, Kevin <Kevin.Mayer@gdata.de>
> Cc: andrew.cooper3@citrix.com; xen-devel@lists.xen.org
> Betreff: Re: AW: AW: [Xen-devel] Xen 4.6.1 crash with altp2m enabled by
> default
>
> >>> On 04.08.16 at 17:08, <Kevin.Mayer@gdata.de> wrote:
> > crash> x /130x 0xffff830bd0da1000
> > 0xffff830bd0da1000: 0x000000000000000e 0x0000000000000000
> > 0xffff830bd0da1010: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da1020: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da1030: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da1040: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da1050: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da1060: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da1070: 0x0000000000000000 0x0000000bd0da3000
> > 0xffff830bd0da1080: 0x0000000c17e36000 0x0000000000000000
> > 0xffff830bd0da1090: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da10a0: 0x00000000e7512000 0x00000000e7513000
> > 0xffff830bd0da10b0: 0x0000000bd0da0000 0x0000000000000000
> > 0xffff830bd0da10c0: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da10d0: 0x0000000000000000 0x0000006fedea809b
> > 0xffff830bd0da10e0: 0x00000001a379e000 0x0000000610f9101e
> > 0xffff830bd0da10f0: 0x0000000000000000 0xffffffffffffffff
> > 0xffff830bd0da1100: 0x0000000000000000 0x0007010600070106
> > 0xffff830bd0da1110: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da1120: 0x0000006bb6a075fa 0x000600420000003f
> > 0xffff830bd0da1130: 0x0000000000000000 0x000fefff00000000
> > 0xffff830bd0da1140: 0x0000000000000000 0x00000000000051ff
> > 0xffff830bd0da1150: 0x0000000000000041 0x0000000000000000
> > 0xffff830bd0da1160: 0x0000000000000000 0x0000000c00000000
> > 0xffff830bd0da1170: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da1180: 0x0000000000000001 0x0000000000000000
> > 0xffff830bd0da1190: 0x0000000800000000 0x0000000000000000
> > 0xffff830bd0da11a0: 0x0000000000000001 0x0000000000000096
> > 0xffff830bd0da11b0: 0xffff82d0802bc208 0x00000000806f6dbc
> > 0xffff830bd0da11c0: 0x0000000000000000 0x0000000000000400
> > 0xffff830bd0da11d0: 0x0000000080550f34 0x00000000f0e48161
> > 0xffff830bd0da11e0: 0x0000000000000246 0x0000000000000000
> > 0xffff830bd0da11f0: 0x00000000f79c3000 0x00000000804de6f0
> > 0xffff830bd0da1200: 0x0000000000000023 0x0000000000000000
> > 0xffff830bd0da1210: 0x00c0f300ffffffff 0x0000000000000008
> > 0xffff830bd0da1220: 0x0000000000000000 0x00c09b00ffffffff
> > 0xffff830bd0da1230: 0x0000000000000010 0x0000000000000000
> > 0xffff830bd0da1240: 0x00c09300ffffffff 0x0000000000000023
> > 0xffff830bd0da1250: 0x0000000000000000 0x00c0f300ffffffff
> > 0xffff830bd0da1260: 0x0000000000000030 0x00000000ffdff000
> > 0xffff830bd0da1270: 0x00c0930000001fff 0x0000000000000000
> > 0xffff830bd0da1280: 0x0000000000000000 0x01c00000ffffffff
> > 0xffff830bd0da1290: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da12a0: 0x01c00000ffffffff 0x0000000000000028
> > 0xffff830bd0da12b0: 0x0000000080042000 0x00008b00000020ab
> > 0xffff830bd0da12c0: 0x000000008003f000 0x000000008003f400
> > 0xffff830bd0da12d0: 0x000007ff000003ff 0x000000008001003b
> > 0xffff830bd0da12e0: 0x0000000000039000 0x00000000000026d9
> > 0xffff830bd0da12f0: 0x000000000000dc3c 0x0000000000000000
> > 0xffff830bd0da1300: 0x0000e00800000000 0x0000000000000000
> > 0xffff830bd0da1310: 0x0000000000000000 0x000000000000e040
> > 0xffff830bd0da1320: 0x0000050100070406 0x0000000000000000
> > 0xffff830bd0da1330: 0x0000000000000000 0x0000000080050033
> > 0xffff830bd0da1340: 0x00000001bd665000 0x00000000000026e0
> > 0xffff830bd0da1350: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da1360: 0xffff830c17e38c80 0xffff830617fd3000
> > 0xffff830bd0da1370: 0xffff830617fcf000 0xffff830617fd7fc0
> > 0xffff830bd0da1380: 0xffff82d08024e150 0xffff830617fd7f90
> > 0xffff830bd0da1390: 0xffff82d080201bb0 0x000000000000e008
> > 0xffff830bd0da13a0: 0x0000006000000000 0x0000000000000000
> > 0xffff830bd0da13b0: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da13c0: 0xffffffffffffffff 0xffffffffffffffff
> > 0xffff830bd0da13d0: 0x000000008001003b 0x00000000000006d9
> > 0xffff830bd0da13e0: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da13f0: 0x0000000000000000 0x0000000000000000
> > 0xffff830bd0da1400: 0x0000000000000000 0x0000000000000000
> >
> > I don't quite understand the Intel developer manual at this point. How
> > do I have to read this data?
>
> I don't think this is formally specified anywhere (publicly). After all that's why
> one has to use vmread/vmwrite.
>
> > Since if ( !(v->arch.hvm_vmx.host_cr0 & X86_CR0_TS) ) must be true I
> > assume the __vmwrite tries to | 0x8 into the host_cr0 leading to the
> > 0x0000000080050033 for the current host_cr0 ( or better the 0x80050033 ).
>
> Well, together with the disassembly it should be possible without consulting
> the crash dump to tell what value it was that was attempted to be written
> (the disassembly tells you which register and the state dumped to the log
> tells you the value). If it is (as you indicated earlier up) indeed zero that gets
> written, then you'd want to try to find out where that zero is coming from.
>
> Jan
____________
Virus checked by G Data MailSecurity
Version: AVA 25.7740 dated 05.08.2016
Virus news: www.antiviruslab.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabled bydefault
2016-08-05 12:51 ` Xen 4.6.1 crash with altp2m enabled bydefault Kevin.Mayer
@ 2016-08-05 14:48 ` Jan Beulich
2016-08-08 9:48 ` Xen 4.6.1 crash with altp2m enabledbydefault Kevin.Mayer
0 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2016-08-05 14:48 UTC (permalink / raw)
To: Kevin.Mayer; +Cc: andrew.cooper3, xen-devel
>>> On 05.08.16 at 14:51, <Kevin.Mayer@gdata.de> wrote:
> According to the xen dmesg
>
> (XEN) RIP: e008:[<ffff82d0801fd23a>] vmx_vmenter_helper+0x27e/0x30a
> (XEN) RFLAGS: 0000000000010003 CONTEXT: hypervisor
> (XEN) rax: 000000008005003b rbx: ffff8300e72fc000 rcx: 0000000000000000
> (XEN) rdx: 0000000000006c00 rsi: ffff830617fd7fc0 rdi: ffff8300e6fc0000
> (XEN) rbp: ffff830617fd7c40 rsp: ffff830617fd7c30 r8: 0000000000000000
> (XEN) r9: ffff830be8dc9310 r10: 0000000000000000 r11: 00003475e9cf85d0
> (XEN) r12: 0000000000000006 r13: ffff830c14ee1000 r14: ffff8300e6fc0000
> (XEN) r15: ffff830617fd0000 cr0: 000000008005003b cr4: 00000000000026e0
> (XEN) cr3: 00000001bd665000 cr2: 0000000004510000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
>
> 0xffff82d0801fa0c3 <vmx_ctxt_switch_from+85>: mov $0x6c00,%edx
> 0xffff82d0801fa0c8 <vmx_ctxt_switch_from+90>: vmwrite %rax,%rdx
>
> The vmwrite tries to write 0x000000008005003b to 0x6c00.
> But the active VCPU has a 0-vmcs-pointer.
Which likely means altp2m manages to confuse some of VMX'es
VMCS management - vmx_vmenter_helper() being on the path
back to the guest, it should be impossible for the VMCS pointer to
be zero here. Can you perhaps identify the most recent vmread or
vmwrite which worked fine? There ought to be many on that path,
and the state corruption could then perhaps be narrowed to quite
small a range of code.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabledbydefault
2016-08-05 14:48 ` Jan Beulich
@ 2016-08-08 9:48 ` Kevin.Mayer
2016-08-08 10:29 ` Jan Beulich
0 siblings, 1 reply; 20+ messages in thread
From: Kevin.Mayer @ 2016-08-08 9:48 UTC (permalink / raw)
To: JBeulich; +Cc: andrew.cooper3, xen-devel
vmx_vmenter_helper is not part of the call stack. The address is simply the location of the ud2 to which the
__vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0);
In
static void vmx_fpu_leave(struct vcpu *v)
jumps.
There are two vmwrites in vmx_vcpu_update_eptp (called by altp2m_vcpu_destroy):
__vmwrite(EPT_POINTER, ept_get_eptp(ept));
__vmwrite(EPTP_INDEX, vcpu_altp2m(v).p2midx);
And four in vmx_vcpu_update_vmfunc_ve (also called by altp2m_vcpu_destroy)
__vmwrite(VM_FUNCTION_CONTROL, VMX_VMFUNC_EPTP_SWITCHING);
__vmwrite(EPTP_LIST_ADDR, virt_to_maddr(d->arch.altp2m_eptp));
__vmwrite(VIRT_EXCEPTION_INFO, mfn_x(mfn) << PAGE_SHIFT);
__vmwrite(SECONDARY_VM_EXEC_CONTROL, v->arch.hvm_vmx.secondary_exec_control);
After the altp2m-part hvm_vcpu_destroy also calls nestedhvm_vcpu_destroy(v), but this code path is executed unconditionally so I assume that the error lies somewhere in the altp2m_vcpu_destroy(v).
What exactly are the vmx_vmcs_enter / exit required for? I often see the vmx_vmcs_enter; __vmwrite; vmx_vmcs_exit combination. Need the __vmwrites be guarded by an enter / exit ( which Is not the case in the static void vmx_fpu_leave(struct vcpu *v) )?
Is it possible that the altp2m_vcpu_destroy->vmx_vcpu_update_eptp->vmx_vmcs_exit->vmx_clear_vmcs invalidates the vmcs for the current vcpu?
Cheers
Kevin
> -----Ursprüngliche Nachricht-----
> Von: Jan Beulich [mailto:JBeulich@suse.com]
> Gesendet: Freitag, 5. August 2016 16:49
> An: Mayer, Kevin <Kevin.Mayer@gdata.de>
> Cc: andrew.cooper3@citrix.com; xen-devel@lists.xen.org
> Betreff: Re: AW: AW: AW: [Xen-devel] Xen 4.6.1 crash with altp2m enabled
> bydefault
>
> >>> On 05.08.16 at 14:51, <Kevin.Mayer@gdata.de> wrote:
> > According to the xen dmesg
> >
> > (XEN) RIP: e008:[<ffff82d0801fd23a>]
> vmx_vmenter_helper+0x27e/0x30a
> > (XEN) RFLAGS: 0000000000010003 CONTEXT: hypervisor
> > (XEN) rax: 000000008005003b rbx: ffff8300e72fc000 rcx:
> 0000000000000000
> > (XEN) rdx: 0000000000006c00 rsi: ffff830617fd7fc0 rdi: ffff8300e6fc0000
> > (XEN) rbp: ffff830617fd7c40 rsp: ffff830617fd7c30 r8: 0000000000000000
> > (XEN) r9: ffff830be8dc9310 r10: 0000000000000000 r11: 00003475e9cf85d0
> > (XEN) r12: 0000000000000006 r13: ffff830c14ee1000 r14: ffff8300e6fc0000
> > (XEN) r15: ffff830617fd0000 cr0: 000000008005003b cr4:
> 00000000000026e0
> > (XEN) cr3: 00000001bd665000 cr2: 0000000004510000
> > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
> >
> > 0xffff82d0801fa0c3 <vmx_ctxt_switch_from+85>: mov $0x6c00,%edx
> > 0xffff82d0801fa0c8 <vmx_ctxt_switch_from+90>: vmwrite %rax,%rdx
> >
> > The vmwrite tries to write 0x000000008005003b to 0x6c00.
> > But the active VCPU has a 0-vmcs-pointer.
>
> Which likely means altp2m manages to confuse some of VMX'es VMCS
> management - vmx_vmenter_helper() being on the path back to the guest,
> it should be impossible for the VMCS pointer to be zero here. Can you
> perhaps identify the most recent vmread or vmwrite which worked fine?
> There ought to be many on that path, and the state corruption could then
> perhaps be narrowed to quite small a range of code.
>
> Jan
____________
Virus checked by G Data MailSecurity
Version: AVA 25.7794 dated 08.08.2016
Virus news: www.antiviruslab.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabledbydefault
2016-08-08 9:48 ` Xen 4.6.1 crash with altp2m enabledbydefault Kevin.Mayer
@ 2016-08-08 10:29 ` Jan Beulich
2016-08-19 10:01 ` Kevin.Mayer
0 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2016-08-08 10:29 UTC (permalink / raw)
To: Kevin.Mayer; +Cc: andrew.cooper3, xen-devel
>>> On 08.08.16 at 11:48, <Kevin.Mayer@gdata.de> wrote:
> vmx_vmenter_helper is not part of the call stack. The address is simply the
> location of the ud2 to which the
> __vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0);
> In
> static void vmx_fpu_leave(struct vcpu *v)
> jumps.
> There are two vmwrites in vmx_vcpu_update_eptp (called by
> altp2m_vcpu_destroy):
> __vmwrite(EPT_POINTER, ept_get_eptp(ept));
> __vmwrite(EPTP_INDEX, vcpu_altp2m(v).p2midx);
>
> And four in vmx_vcpu_update_vmfunc_ve (also called by altp2m_vcpu_destroy)
> __vmwrite(VM_FUNCTION_CONTROL, VMX_VMFUNC_EPTP_SWITCHING);
> __vmwrite(EPTP_LIST_ADDR, virt_to_maddr(d->arch.altp2m_eptp));
> __vmwrite(VIRT_EXCEPTION_INFO, mfn_x(mfn) << PAGE_SHIFT);
> __vmwrite(SECONDARY_VM_EXEC_CONTROL,
> v->arch.hvm_vmx.secondary_exec_control);
>
> After the altp2m-part hvm_vcpu_destroy also calls nestedhvm_vcpu_destroy(v),
> but this code path is executed unconditionally so I assume that the error
> lies somewhere in the altp2m_vcpu_destroy(v).
>
> What exactly are the vmx_vmcs_enter / exit required for? I often see the
> vmx_vmcs_enter; __vmwrite; vmx_vmcs_exit combination. Need the __vmwrites be
> guarded by an enter / exit ( which Is not the case in the static void
> vmx_fpu_leave(struct vcpu *v) )?
On code paths where the correct VMCS may not be the current one
it is necessary to frame vmread / vmwrite accordingly.
> Is it possible that the
> altp2m_vcpu_destroy->vmx_vcpu_update_eptp->vmx_vmcs_exit->vmx_clear_vmcs
> invalidates the vmcs for the current vcpu?
I certainly can't exclude this possibility.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabledbydefault
2016-08-08 10:29 ` Jan Beulich
@ 2016-08-19 10:01 ` Kevin.Mayer
2016-08-22 11:58 ` Andrew Cooper
0 siblings, 1 reply; 20+ messages in thread
From: Kevin.Mayer @ 2016-08-19 10:01 UTC (permalink / raw)
To: JBeulich; +Cc: andrew.cooper3, xen-devel
Hi
I took another look at Xen and a new crashdump.
The last successful __vmwrite should be in
static void vmx_vcpu_update_vmfunc_ve(struct vcpu *v)
[...]
__vmwrite(SECONDARY_VM_EXEC_CONTROL,
v->arch.hvm_vmx.secondary_exec_control);
[...]
After this the altp2m_vcpu_destroy wakes up the vcpu and is then finished.
In nestedhvm_vcpu_destroy (nvmx_vcpu_destroy) the vmcs can overwritten (but is not reached in our case as far as I can see):
if ( nvcpu->nv_n1vmcx )
v->arch.hvm_vmx.vmcs = nvcpu->nv_n1vmcx;
In conclusion:
When destroying a domain the altp2m_vcpu_destroy(v); path seems to mess up the vmcs which ( only ) sometimes leads to a failed __vmwrite in vmx_fpu_leave.
That is as far as I can get with my understanding of the Xen code.
Do you guys have any additional ideas what I could test / analyse?
> -----Ursprüngliche Nachricht-----
> Von: Jan Beulich [mailto:JBeulich@suse.com]
> Gesendet: Montag, 8. August 2016 12:29
> An: Mayer, Kevin <Kevin.Mayer@gdata.de>
> Cc: andrew.cooper3@citrix.com; xen-devel@lists.xen.org
> Betreff: Re: [Xen-devel] Xen 4.6.1 crash with altp2m enabledbydefault
>
> >>> On 08.08.16 at 11:48, <Kevin.Mayer@gdata.de> wrote:
> > vmx_vmenter_helper is not part of the call stack. The address is
> > simply the location of the ud2 to which the __vmwrite(HOST_CR0,
> > v->arch.hvm_vmx.host_cr0); In static void vmx_fpu_leave(struct vcpu
> > *v) jumps.
> > There are two vmwrites in vmx_vcpu_update_eptp (called by
> > altp2m_vcpu_destroy):
> > __vmwrite(EPT_POINTER, ept_get_eptp(ept)); __vmwrite(EPTP_INDEX,
> > vcpu_altp2m(v).p2midx);
> >
> > And four in vmx_vcpu_update_vmfunc_ve (also called by
> > altp2m_vcpu_destroy) __vmwrite(VM_FUNCTION_CONTROL,
> > VMX_VMFUNC_EPTP_SWITCHING); __vmwrite(EPTP_LIST_ADDR,
> > virt_to_maddr(d->arch.altp2m_eptp));
> > __vmwrite(VIRT_EXCEPTION_INFO, mfn_x(mfn) << PAGE_SHIFT);
> > __vmwrite(SECONDARY_VM_EXEC_CONTROL,
> > v->arch.hvm_vmx.secondary_exec_control);
> >
> > After the altp2m-part hvm_vcpu_destroy also calls
> > nestedhvm_vcpu_destroy(v), but this code path is executed
> > unconditionally so I assume that the error lies somewhere in the
> altp2m_vcpu_destroy(v).
> >
> > What exactly are the vmx_vmcs_enter / exit required for? I often see
> > the vmx_vmcs_enter; __vmwrite; vmx_vmcs_exit combination. Need the
> > __vmwrites be guarded by an enter / exit ( which Is not the case in
> > the static void vmx_fpu_leave(struct vcpu *v) )?
>
> On code paths where the correct VMCS may not be the current one it is
> necessary to frame vmread / vmwrite accordingly.
>
> > Is it possible that the
> > altp2m_vcpu_destroy->vmx_vcpu_update_eptp->vmx_vmcs_exit-
> >vmx_clear_vm
> > cs invalidates the vmcs for the current vcpu?
>
> I certainly can't exclude this possibility.
>
> Jan
____________
Virus checked by G Data MailSecurity
Version: AVA 25.7943 dated 19.08.2016
Virus news: www.antiviruslab.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabledbydefault
2016-08-19 10:01 ` Kevin.Mayer
@ 2016-08-22 11:58 ` Andrew Cooper
2016-08-22 12:22 ` Kevin.Mayer
` (2 more replies)
0 siblings, 3 replies; 20+ messages in thread
From: Andrew Cooper @ 2016-08-22 11:58 UTC (permalink / raw)
To: Kevin.Mayer, JBeulich; +Cc: xen-devel
On 19/08/16 11:01, Kevin.Mayer@gdata.de wrote:
> Hi
>
> I took another look at Xen and a new crashdump.
> The last successful __vmwrite should be in
> static void vmx_vcpu_update_vmfunc_ve(struct vcpu *v)
> [...]
> __vmwrite(SECONDARY_VM_EXEC_CONTROL,
> v->arch.hvm_vmx.secondary_exec_control);
> [...]
> After this the altp2m_vcpu_destroy wakes up the vcpu and is then finished.
>
> In nestedhvm_vcpu_destroy (nvmx_vcpu_destroy) the vmcs can overwritten (but is not reached in our case as far as I can see):
> if ( nvcpu->nv_n1vmcx )
> v->arch.hvm_vmx.vmcs = nvcpu->nv_n1vmcx;
>
> In conclusion:
> When destroying a domain the altp2m_vcpu_destroy(v); path seems to mess up the vmcs which ( only ) sometimes leads to a failed __vmwrite in vmx_fpu_leave.
> That is as far as I can get with my understanding of the Xen code.
>
> Do you guys have any additional ideas what I could test / analyse?
Do you have easy reproduction instructions you could share? Sadly, this
is looking like an issue which isn't viable to debug over email.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabledbydefault
2016-08-22 11:58 ` Andrew Cooper
@ 2016-08-22 12:22 ` Kevin.Mayer
2016-09-07 8:35 ` Kevin.Mayer
2016-09-21 14:18 ` Kevin.Mayer
2 siblings, 0 replies; 20+ messages in thread
From: Kevin.Mayer @ 2016-08-22 12:22 UTC (permalink / raw)
To: andrew.cooper3, JBeulich; +Cc: xen-devel
Hi
The reproduction should be pretty simple:
Apply the patch to enable altp2m unconditionally:
d->arch.hvm_domain.params[HVM_PARAM_HPET_ENABLED] = 1;
d->arch.hvm_domain.params[HVM_PARAM_TRIPLE_FAULT_REASON] = SHUTDOWN_reboot;
+ d->arch.hvm_domain.params[HVM_PARAM_ALTP2M] = 1;
+
vpic_init(d);
rc = vioapic_init(d);
For the guest we use one state file ( Windows 10 ) from which the guests are restored with libvirt.
Simply restore and destroy several guests (5-7 in our current setup) in fast succession (every guest has about 1-2minutes runtime).
The amount of guest-VMs seems to correlate with the time until the crash occurs, but other, random factors seem to be more important.
More VMs => the crash happens faster.
Is the following debug-setup possible?
L0: Xen / VMWare
L1: Xen with altp2m enabled
L2: Several guest-VMs being constantly restored / destroyed
Then periodically take snapshots until the hypervisor panics and try to debug from the latest snapshot on.
> -----Ursprüngliche Nachricht-----
> Von: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
> Gesendet: Montag, 22. August 2016 13:58
> An: Mayer, Kevin <Kevin.Mayer@gdata.de>; JBeulich@suse.com
> Cc: xen-devel@lists.xen.org
> Betreff: Re: AW: [Xen-devel] Xen 4.6.1 crash with altp2m enabledbydefault
>
> On 19/08/16 11:01, Kevin.Mayer@gdata.de wrote:
> > Hi
> >
> > I took another look at Xen and a new crashdump.
> > The last successful __vmwrite should be in static void
> > vmx_vcpu_update_vmfunc_ve(struct vcpu *v) [...]
> > __vmwrite(SECONDARY_VM_EXEC_CONTROL,
> > v->arch.hvm_vmx.secondary_exec_control);
> > [...]
> > After this the altp2m_vcpu_destroy wakes up the vcpu and is then
> finished.
> >
> > In nestedhvm_vcpu_destroy (nvmx_vcpu_destroy) the vmcs can
> overwritten (but is not reached in our case as far as I can see):
> > if ( nvcpu->nv_n1vmcx )
> > v->arch.hvm_vmx.vmcs = nvcpu->nv_n1vmcx;
> >
> > In conclusion:
> > When destroying a domain the altp2m_vcpu_destroy(v); path seems to
> mess up the vmcs which ( only ) sometimes leads to a failed __vmwrite in
> vmx_fpu_leave.
> > That is as far as I can get with my understanding of the Xen code.
> >
> > Do you guys have any additional ideas what I could test / analyse?
>
> Do you have easy reproduction instructions you could share? Sadly, this is
> looking like an issue which isn't viable to debug over email.
>
> ~Andrew
____________
Virus checked by G Data MailSecurity
Version: AVA 25.7981 dated 22.08.2016
Virus news: www.antiviruslab.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabledbydefault
2016-08-22 11:58 ` Andrew Cooper
2016-08-22 12:22 ` Kevin.Mayer
@ 2016-09-07 8:35 ` Kevin.Mayer
2016-09-21 14:18 ` Kevin.Mayer
2 siblings, 0 replies; 20+ messages in thread
From: Kevin.Mayer @ 2016-09-07 8:35 UTC (permalink / raw)
To: andrew.cooper3, JBeulich; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 2719 bytes --]
Hi
I took the time to write a small script which restores and destroys domains from provided state files.
Just apply the patch to a xen 4.6.1, provide some images + state files and start the script.
python VmStarter.py -FILE /path/to/domU-0.state -FILE /path/to/domU-1.state --loggingLevel DEBUG
You can provide an arbitrary amount of state files and the script will start an additional thread for each one.
Each thread restores one guest domain from the provided state file, waits for a random time between 20 and 30 seconds (sleepTime = random.randint(20,30) ) , destroys the domain and then starts the process again.
The guest domains and the corresponding state files need to have the same name since the script extracts the domain name from the state file name.
When starting about one guest domain for every physical core of the CPU the crash should occur in 5 to 10 minutes. Since the crashes are pretty random the hypervisor sometimes panics almost instantly and sometimes it takes a while, but it seems to correlate with the amount of started guest domains.
More domains => faster crash
Kevin
> -----Ursprüngliche Nachricht-----
> Von: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
> Gesendet: Montag, 22. August 2016 13:58
> An: Mayer, Kevin <Kevin.Mayer@gdata.de>; JBeulich@suse.com
> Cc: xen-devel@lists.xen.org
> Betreff: Re: AW: [Xen-devel] Xen 4.6.1 crash with altp2m enabledbydefault
>
> On 19/08/16 11:01, Kevin.Mayer@gdata.de wrote:
> > Hi
> >
> > I took another look at Xen and a new crashdump.
> > The last successful __vmwrite should be in static void
> > vmx_vcpu_update_vmfunc_ve(struct vcpu *v) [...]
> > __vmwrite(SECONDARY_VM_EXEC_CONTROL,
> > v->arch.hvm_vmx.secondary_exec_control);
> > [...]
> > After this the altp2m_vcpu_destroy wakes up the vcpu and is then
> finished.
> >
> > In nestedhvm_vcpu_destroy (nvmx_vcpu_destroy) the vmcs can
> overwritten (but is not reached in our case as far as I can see):
> > if ( nvcpu->nv_n1vmcx )
> > v->arch.hvm_vmx.vmcs = nvcpu->nv_n1vmcx;
> >
> > In conclusion:
> > When destroying a domain the altp2m_vcpu_destroy(v); path seems to
> mess up the vmcs which ( only ) sometimes leads to a failed __vmwrite in
> vmx_fpu_leave.
> > That is as far as I can get with my understanding of the Xen code.
> >
> > Do you guys have any additional ideas what I could test / analyse?
>
> Do you have easy reproduction instructions you could share? Sadly, this is
> looking like an issue which isn't viable to debug over email.
>
> ~Andrew
____________
Virus checked by G Data MailSecurity
Version: AVA 25.8183 dated 07.09.2016
Virus news: www.antiviruslab.com
[-- Attachment #2: xen-altp2menable.patch --]
[-- Type: application/octet-stream, Size: 402 bytes --]
--- a/xen/arch/x86/hvm/hvm.c 2015-10-05 16:33:39.000000000 +0200
+++ b/xen/arch/x86/hvm/hvm.c 2016-02-10 13:36:31.971062764 +0100
@@ -1597,6 +1597,8 @@
d->arch.hvm_domain.params[HVM_PARAM_HPET_ENABLED] = 1;
d->arch.hvm_domain.params[HVM_PARAM_TRIPLE_FAULT_REASON] = SHUTDOWN_reboot;
+ d->arch.hvm_domain.params[HVM_PARAM_ALTP2M] = 1;
+
vpic_init(d);
rc = vioapic_init(d);
[-- Attachment #3: VmStarter.py --]
[-- Type: application/octet-stream, Size: 4331 bytes --]
import argparse
import libvirt
import logging
from multiprocessing import Process
import os
import random
import threading
import time
def manageVM(endSignal, workerNumber, statefile, loggingLevel):
logging.basicConfig(level = loggingLevel)
logging.debug("Worker " + str(workerNumber) + ": begins work")
domainName = os.path.basename(statefile).split(".")[0]
domainID = 0
domain = None
connection = libvirt.open(None)
if connection == None:
logging.error('Failed to open connection to hypervisor')
exit(1)
logging.debug("Worker " + str(workerNumber) + ": connected to libvirt")
while not endSignal.wait(3):
sleepTime = random.randint(20,30)
logging.debug("Worker " + str(workerNumber) + ": starts VM from statefile " + str(statefile))
try:
id = connection.restore(statefile)
if id < 0:
logging.error('Unable to restore guest from ' + statefile)
exit(1)
except libvirt.libvirtError as err:
logging.error("Worker " + str(workerNumber) + ": Error when restoring domain " + domainName + " " + str(err))
logging.debug("Worker " + str(workerNumber) + ": Restored domain. Check if it succeeded.")
try:
domain = connection.lookupByName(domainName)
if domain == None:
logging.error('Unable to find guest with name ' + domainName)
exit(1)
domainID = domain.ID()
except libvirt.libvirtError as err:
logging.error("Worker " + str(workerNumber) + ": Error when looking up domain " + domainName + " " + str(err))
logging.debug("Worker " + str(workerNumber) + ": VM " + domainName + " started (id " + str (domainID) + "), waiting " + str(sleepTime))
time.sleep(sleepTime)
logging.debug("Worker " + str(workerNumber) + ": sleep time up. Destroying VM")
try:
if domain:
domain.destroy()
except libvirt.libvirtError as err:
logging.error("Worker " + str(workerNumber) + ": Error when managing domain " + domainName + " " + str(err))
logging.debug("Worker " + str(workerNumber) + ": VM destroyed")
logging.debug("Worker " + str(workerNumber) + ": ending")
def startTestRun(stateFiles, loggingLevel):
activeThreads = []
logging.debug("Starting worker threads")
endSignal = threading.Event()
for i in range (len(stateFiles)):
vmWorkerThread = threading.Thread(target = manageVM, args=(endSignal, i, stateFiles[i], loggingLevel))
activeThreads.append(vmWorkerThread)
vmWorkerThread.start()
time.sleep(5)
logging.debug("Waiting for program termination")
try:
while 1:
time.sleep(.1)
except KeyboardInterrupt:
logging.debug("Got Interrupt")
endSignal.set()
for thread in activeThreads:
thread.join()
logging.debug("All workers finished")
def inputSanityCheck(args):
numeric_level = getattr(logging, args.loggingLevel.upper(), None)
if not isinstance(numeric_level, int):
raise ValueError('Invalid log level: %s' % args.loggingLevel)
logging.basicConfig(level = args.loggingLevel)
if not args.stateFiles:
raise ValueError("No state files given. Aborting")
for file in args.stateFiles:
if not os.path.exists(file):
raise ValueError("State file " + file + " does not exist. Aborting")
def main():
parser = argparse.ArgumentParser(description='Restore and destroy VMs from statefile based on virsh.')
parser.add_argument('-FILE', action='append', dest='stateFiles',
default=[],
help='List of state files from which to restore the VMs')
parser.add_argument('--loggingLevel' ,
default = 'WARNING',
help='Set the logging level. Valid levels are: DEBUG, INFO, WARNING (default), ERROR, CRITICAL')
args = parser.parse_args()
print (args)
inputSanityCheck(args)
stateFiles = args.stateFiles
loggingLevel = args.loggingLevel
startTestRun(stateFiles, loggingLevel)
if __name__ == '__main__':
main()
[-- Attachment #4: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabledbydefault
2016-08-22 11:58 ` Andrew Cooper
2016-08-22 12:22 ` Kevin.Mayer
2016-09-07 8:35 ` Kevin.Mayer
@ 2016-09-21 14:18 ` Kevin.Mayer
2016-09-22 10:13 ` Jan Beulich
2 siblings, 1 reply; 20+ messages in thread
From: Kevin.Mayer @ 2016-09-21 14:18 UTC (permalink / raw)
To: andrew.cooper3, JBeulich; +Cc: xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 5863 bytes --]
Hi guys
I have found the problem (after hours and hours of gruesome
debugging with the almighty print) and it seems that this could potentially
have quite a bit of impact if altp2m is enabled for a guest domain (even if
the
functionality is never actively used), since destroying any vcpu of this
guest could lead to a hypervisor panic.
So a malicious user could simply destroy and restart his VM(s) in order to
DOS the VMs of other users by killing the hypervisor.
Granted, this is not very effective, but, depending on the environment, it
is extremely easy to implement.
The bug persists in Xen 4.7 and I do not that it was fixed in the current
master branch.
The following happens.
The call
void hvm_vcpu_destroy(struct vcpu *v)
{
hvm_all_ioreq_servers_remove_vcpu(v->domain, v);
if ( hvm_altp2m_supported() )
altp2m_vcpu_destroy(v);
at some time reaches vmx_vcpu_update_eptp which ends with a
vmx_vmcs_exit(v);.
There vmx_clear_vmcs(v); -> __vmx_clear_vmcs is called where the
current_vmcs is invalidated if the current vmcs in the CPU is the same as
virt_to_maddr (v->arch.hvm_vmx->vmcs):
__vmpclear(virt_to_maddr(arch_vmx->vmcs)); (
http://www.tptp.cc/mirrors/siyobik.info/instruction/VMCLEAR.html )
To check this assumption I implemented a basic __vmptrst (
http://www.tptp.cc/mirrors/siyobik.info/instruction/VMPTRST.html ) and added
the result to the debug output.
(XEN) vmcs.c:519:IDLEv4 __vmx_clear_vmcs: realVMCS BEFORE __vmpclear
82415a000
(XEN) vmcs.c:522:IDLEv4 __vmx_clear_vmcs: realVMCS AFTER __vmpclear
ffffffffffffffff
After that no vmcs_load / enter is executed so the vmcs in the CPU remains
invalidated.
For the next function in hvm_vcpu_destroy, the nestedhvm_vcpu_destroy(v) the
missing vmcs is no problem (at least in our use case), but the
free_compat_arg_xlat crashes.
The callstack is as follows:
hvm_vcpu_destroy
free_compat_arg_xlat
destroy_perdomain_mapping
map_domain_page
(probably inlined) mapcache_current_vcpu
sync_local_execstate
__sync_local_execstate
__context_switch
(with function pointer v->arch.ctxt_switch_from = vmx_ctxt_switch_from)
vmx_ctxt_switch_from
(probably inlined) vmx_fpu_leave
There a vmwrite is tried if either ( !(v->arch.hvm_vmx.host_cr0 &
X86_CR0_TS) ) or ( !(v->arch.hvm_vcpu.guest_cr[0] & X86_CR0_TS) ) is true.
The executed vmwrite then crashes.
As my knowledge of Xen is not that comprehensive, could you tell me when the
TS-bits are set / cleared and what they are used for?
static void vmx_fpu_leave(struct vcpu *v)
{
ASSERT(!v->fpu_dirtied);
ASSERT(read_cr0() & X86_CR0_TS);
if ( !(v->arch.hvm_vmx.host_cr0 & X86_CR0_TS) )
{
v->arch.hvm_vmx.host_cr0 |= X86_CR0_TS;
__vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0);
}
if ( !(v->arch.hvm_vcpu.guest_cr[0] & X86_CR0_TS) )
{
v->arch.hvm_vcpu.hw_cr[0] |= X86_CR0_TS;
__vmwrite(GUEST_CR0, v->arch.hvm_vcpu.hw_cr[0]);
v->arch.hvm_vmx.exception_bitmap |= (1u << TRAP_no_device);
vmx_update_exception_bitmap(v);
}
}
In the crash dump the additional debug output shows that at least one
__vmwrite will be tried and that the VMCS in the CPU is invalidated:
(XEN) vmx.c:698:IDLEv4 vmx_fpu_leave: vcpu ffff8300defae000 vmcs
ffff8301586c9000 host_cr0-case FALSE guest_cr[0]-case TRUE curr
ffff8300df2fb000 curr->arch.hvm_vmx.vmcs 0000000000000000 realVMCS
ffffffffffffffff
As a quick fix I patched the fpu_leave to only allow the __vmwrite when the
realVMCS is valid.
This seems to work fine, but requires a call to __vmptrst every time
vmx_fpu_leave is called. Also I do not know if an ignored TS has any
negative consequences when destroying a vcpu. I assume that this is not
case. In our tests nothing pointed to any problems.
I added the patch to enable altp2m unconditionally and a patch which evades
the panic in vmx_fpu_leave.
They are not pretty or anywhere near production ready, but I think you will
get the idea.
I tried to implement the __vmptrst with the #ifdef HAVE_GAS_VM parts (
analogue to the other functions in vmx.h ) but failed miserably since I lack
the required knowledge about the OPCODE definitions. :-D
Cheers
Kevin
> -----Ursprüngliche Nachricht-----
> Von: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
> Gesendet: Montag, 22. August 2016 13:58
> An: Mayer, Kevin <Kevin.Mayer@gdata.de>; JBeulich@suse.com
> Cc: xen-devel@lists.xen.org
> Betreff: Re: AW: [Xen-devel] Xen 4.6.1 crash with altp2m enabledbydefault
>
> On 19/08/16 11:01, Kevin.Mayer@gdata.de wrote:
> > Hi
> >
> > I took another look at Xen and a new crashdump.
> > The last successful __vmwrite should be in static void
> > vmx_vcpu_update_vmfunc_ve(struct vcpu *v) [...]
> > __vmwrite(SECONDARY_VM_EXEC_CONTROL,
> > v->arch.hvm_vmx.secondary_exec_control);
> > [...]
> > After this the altp2m_vcpu_destroy wakes up the vcpu and is then
> finished.
> >
> > In nestedhvm_vcpu_destroy (nvmx_vcpu_destroy) the vmcs can
> overwritten (but is not reached in our case as far as I can see):
> > if ( nvcpu->nv_n1vmcx )
> > v->arch.hvm_vmx.vmcs = nvcpu->nv_n1vmcx;
> >
> > In conclusion:
> > When destroying a domain the altp2m_vcpu_destroy(v); path seems to
> mess up the vmcs which ( only ) sometimes leads to a failed __vmwrite in
> vmx_fpu_leave.
> > That is as far as I can get with my understanding of the Xen code.
> >
> > Do you guys have any additional ideas what I could test / analyse?
>
> Do you have easy reproduction instructions you could share? Sadly, this
is
> looking like an issue which isn't viable to debug over email.
>
> ~Andrew
____________
Virus checked by G Data MailSecurity
Version: AVA 25.8368 dated 21.09.2016
Virus news: www.antiviruslab.com
[-- Attachment #1.1.2: xen-altp2menable.patch --]
[-- Type: application/octet-stream, Size: 415 bytes --]
--- a/xen/arch/x86/hvm/hvm.c 2015-10-05 16:33:39.000000000 +0200
+++ b/xen/arch/x86/hvm/hvm.c 2016-02-10 13:36:31.971062764 +0100
@@ -1597,6 +1597,8 @@
d->arch.hvm_domain.params[HVM_PARAM_HPET_ENABLED] = 1;
d->arch.hvm_domain.params[HVM_PARAM_TRIPLE_FAULT_REASON] = SHUTDOWN_reboot;
+ d->arch.hvm_domain.params[HVM_PARAM_ALTP2M] = 1;
+
vpic_init(d);
rc = vioapic_init(d);
[-- Attachment #1.1.3: xen-vmx_altp2m_bug.patch --]
[-- Type: application/octet-stream, Size: 2539 bytes --]
--- a/xen/arch/x86/hvm/vmx/vmx.c 2016-02-09 15:44:19.000000000 +0100
+++ b/xen/arch/x86/hvm/vmx/vmx.c 2016-09-21 15:46:20.000000000 +0200
@@ -683,27 +683,31 @@
static void vmx_fpu_leave(struct vcpu *v)
{
+ u64 realVMCS = 0xffffffffffffffff;
ASSERT(!v->fpu_dirtied);
ASSERT(read_cr0() & X86_CR0_TS);
-
- if ( !(v->arch.hvm_vmx.host_cr0 & X86_CR0_TS) )
+ __vmptrst ( &realVMCS );
+ if ( realVMCS != 0xffffffffffffffff )
{
- v->arch.hvm_vmx.host_cr0 |= X86_CR0_TS;
- __vmwrite(HOST_CR0, v->arch.hvm_vmx.host_cr0);
- }
+ if ( !( v->arch.hvm_vmx.host_cr0 & X86_CR0_TS ) )
+ {
+ v->arch.hvm_vmx.host_cr0 |= X86_CR0_TS;
+ __vmwrite ( HOST_CR0 , v->arch.hvm_vmx.host_cr0 );
+ }
- /*
- * If the guest does not have TS enabled then we must cause and handle an
- * exception on first use of the FPU. If the guest *does* have TS enabled
- * then this is not necessary: no FPU activity can occur until the guest
- * clears CR0.TS, and we will initialise the FPU when that happens.
- */
- if ( !(v->arch.hvm_vcpu.guest_cr[0] & X86_CR0_TS) )
- {
- v->arch.hvm_vcpu.hw_cr[0] |= X86_CR0_TS;
- __vmwrite(GUEST_CR0, v->arch.hvm_vcpu.hw_cr[0]);
- v->arch.hvm_vmx.exception_bitmap |= (1u << TRAP_no_device);
- vmx_update_exception_bitmap(v);
+ /*
+ * If the guest does not have TS enabled then we must cause and handle an
+ * exception on first use of the FPU. If the guest *does* have TS enabled
+ * then this is not necessary: no FPU activity can occur until the guest
+ * clears CR0.TS, and we will initialise the FPU when that happens.
+ */
+ if ( !( v->arch.hvm_vcpu.guest_cr[ 0 ] & X86_CR0_TS ) )
+ {
+ v->arch.hvm_vcpu.hw_cr[ 0 ] |= X86_CR0_TS;
+ __vmwrite ( GUEST_CR0 , v->arch.hvm_vcpu.hw_cr[ 0 ] );
+ v->arch.hvm_vmx.exception_bitmap |= ( 1u << TRAP_no_device );
+ vmx_update_exception_bitmap ( v );
+ }
}
}
--- a/xen/include/asm-x86/hvm/vmx/vmx.h 2016-02-09 15:44:19.000000000 +0100
+++ b/xen/include/asm-x86/hvm/vmx/vmx.h 2016-09-21 15:48:56.000000000 +0200
@@ -307,6 +307,11 @@
: "memory");
}
+static inline void __vmptrst ( u64 *addr )
+{
+ __asm __volatile ( "vmptrst %[addr]" ::[ addr ]"m" ( *addr ) : "memory" );
+}
+
static inline void __vmpclear(u64 addr)
{
asm volatile (
[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 4024 bytes --]
[-- Attachment #2: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2m enabledbydefault
2016-09-21 14:18 ` Kevin.Mayer
@ 2016-09-22 10:13 ` Jan Beulich
[not found] ` <5C9C3B9BEF1B354596EAE3D6800D876B2DCF7138@e3.gdata.de>
0 siblings, 1 reply; 20+ messages in thread
From: Jan Beulich @ 2016-09-22 10:13 UTC (permalink / raw)
To: andrew.cooper3, Kevin.Mayer; +Cc: xen-devel
>>> On 21.09.16 at 16:18, <Kevin.Mayer@gdata.de> wrote:
> I have found the problem (after hours and hours of gruesome
> debugging with the almighty print) and it seems that this could potentially
> have quite a bit of impact if altp2m is enabled for a guest domain (even if
> the
> functionality is never actively used), since destroying any vcpu of this
> guest could lead to a hypervisor panic.
> So a malicious user could simply destroy and restart his VM(s) in order to
> DOS the VMs of other users by killing the hypervisor.
> Granted, this is not very effective, but, depending on the environment, it
> is extremely easy to implement.
So this is not a security problem because altp2m isn't a supported
feature yet, albeit the features page doesn't explicitly state this one
way or the other. The correct way to report a suspected security
issue would, however, have been to contact security@xenproject.org
(see also https://www.xenproject.org/security-policy.html).
> The bug persists in Xen 4.7 and I do not that it was fixed in the current
> master branch.
>
> The following happens.
> The call
> void hvm_vcpu_destroy(struct vcpu *v)
> {
> hvm_all_ioreq_servers_remove_vcpu(v->domain, v);
> if ( hvm_altp2m_supported() )
> altp2m_vcpu_destroy(v);
>
> at some time reaches vmx_vcpu_update_eptp which ends with a
> vmx_vmcs_exit(v);.
I don't see how this can be a problem - it is properly paired with
a vmx_vmcs_enter().
> For the next function in hvm_vcpu_destroy, the nestedhvm_vcpu_destroy(v) the
> missing vmcs is no problem (at least in our use case), but the
> free_compat_arg_xlat crashes.
> The callstack is as follows:
> hvm_vcpu_destroy
> free_compat_arg_xlat
> destroy_perdomain_mapping
> map_domain_page
> (probably inlined) mapcache_current_vcpu
> sync_local_execstate
For you to get here, you must be running on the idle vCPU, yet
proof of this is not visible from the partial call stack you provide.
And anyway, things breaking here suggest something going wrong
earlier, or else - afaict - we'd run into this problem also without use
of altp2m (basically whenever map_domain_page() would get used
on the guest cleanup path, which - as you see from the call tree -
happens always). So I'm afraid the patch you've put together is
papering over a problem rather than fixing it, and the actual bug
remains non-understood.
Perhaps a relevant aspect is you saying "some time reaches
vmx_vcpu_update_eptp": Why only sometimes? Afaics
altp2m_vcpu_destroy() unconditionally calls
altp2m_vcpu_update_p2m(), which is just a wrapper around
vmx_vcpu_update_eptp().
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Xen 4.6.1 crash with altp2menabledbydefault
[not found] ` <5C9C3B9BEF1B354596EAE3D6800D876B2DCF7254@e3.gdata.de>
@ 2016-09-26 9:20 ` Jan Beulich
0 siblings, 0 replies; 20+ messages in thread
From: Jan Beulich @ 2016-09-26 9:20 UTC (permalink / raw)
To: Kevin.Mayer; +Cc: andrew.cooper3, xen-devel
>>> On 22.09.16 at 17:11, <Kevin.Mayer@gdata.de> wrote:
> Here is a call stack from dmesg.
> Keep in mind that the compiler omits some function names (most importantly
> the vmx_fpu_leave) and also that vmx_vmenter_helper is not actually called.
> The backtrace just thinks it is called because the ud2 which panics the
> hypervisor lies somewhere behind its epilogue.
>
> (XEN) Xen call trace:
> (XEN) [<ffff82d0801fd8ec>] vmx_vmenter_helper+0x280/0x30a
> (XEN) [<ffff82d080174f91>] __context_switch+0xdb/0x3b5
> (XEN) [<ffff82d080178c19>] __sync_local_execstate+0x5e/0x7a
> (XEN) [<ffff82d080178c3e>] sync_local_execstate+0x9/0xb
> (XEN) [<ffff82d080179740>] map_domain_page+0xa0/0x5d4
> (XEN) [<ffff82d080196152>] destroy_perdomain_mapping+0x8f/0x1e8
> (XEN) [<ffff82d080244a62>] free_compat_arg_xlat+0x26/0x28
> (XEN) [<ffff82d0801d4081>] hvm_vcpu_destroy+0x112/0x176
> (XEN) [<ffff82d080175c2c>] vcpu_destroy+0x5d/0x72
> (XEN) [<ffff82d080105dd4>] complete_domain_destroy+0x49/0x192
> (XEN) [<ffff82d0801215fd>] rcu_process_callbacks+0x19a/0x1fb
> (XEN) [<ffff82d08012caf8>] __do_softirq+0x82/0x8d
> (XEN) [<ffff82d08012cb3b>] process_pending_softirqs+0x38/0x3a
> (XEN) [<ffff82d0801c23a8>] mwait_idle+0x10c/0x315
> (XEN) [<ffff82d080174825>] idle_loop+0x51/0x6b
So one possible solution would be to simply avoid calling
altp2m_vcpu_update_p2m() and altp2m_vcpu_update_vmfunc_ve()
from altp2m_vcpu_destroy() for dying domains. However, it looks
as if this would still only paper over the underlying problem.
Yet I continue to have difficulty seeing how we can end up with the
call stack above, without some other earlier bug: I don't think
un-paused vCPU-s are supposed to make it into vcpu_destroy().
Yet at the moment a vCPU gets paused, sync_vcpu_execstate()
would have got called for it already. And while both
vcpu_check_shutdown() and domain_shutdown() call
vcpu_pause_nosync() (which hence wouldn't result in the needed
call to sync_vcpu_execstate()), domain_kill() calls domain_pause()
first thing, while it drops the domain reference almost last thing.
And only the dropping of the last domain reference can cause
execution to reach complete_domain_destroy().
Could you verify this is what is actually happening, i.e. you're not
suffering from a stray put_domain() somewhere? And just to
double check - you're not having any other code changes in your
tree beyond the default enabling of altp2m?
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2016-09-26 9:21 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-07-29 7:33 Xen 4.6.1 crash with altp2m enabled by default Kevin.Mayer
2016-07-29 9:57 ` Dario Faggioli
2016-07-29 10:05 ` Andrew Cooper
2016-08-02 11:45 ` Kevin.Mayer
2016-08-02 12:34 ` Jan Beulich
2016-08-03 13:24 ` Kevin.Mayer
2016-08-03 13:54 ` Jan Beulich
2016-08-04 15:08 ` Kevin.Mayer
2016-08-04 15:35 ` Jan Beulich
2016-08-05 12:51 ` Xen 4.6.1 crash with altp2m enabled bydefault Kevin.Mayer
2016-08-05 14:48 ` Jan Beulich
2016-08-08 9:48 ` Xen 4.6.1 crash with altp2m enabledbydefault Kevin.Mayer
2016-08-08 10:29 ` Jan Beulich
2016-08-19 10:01 ` Kevin.Mayer
2016-08-22 11:58 ` Andrew Cooper
2016-08-22 12:22 ` Kevin.Mayer
2016-09-07 8:35 ` Kevin.Mayer
2016-09-21 14:18 ` Kevin.Mayer
2016-09-22 10:13 ` Jan Beulich
[not found] ` <5C9C3B9BEF1B354596EAE3D6800D876B2DCF7138@e3.gdata.de>
[not found] ` <57E405970200007800111B26@prv-mh.provo.novell.com>
[not found] ` <5C9C3B9BEF1B354596EAE3D6800D876B2DCF7254@e3.gdata.de>
2016-09-26 9:20 ` Xen 4.6.1 crash with altp2menabledbydefault Jan Beulich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).