* Failed vm entry with heavy use of emulator @ 2016-01-05 11:49 Tamas K Lengyel 2016-01-05 11:56 ` Andrew Cooper 2016-01-05 12:35 ` Razvan Cojocaru 0 siblings, 2 replies; 15+ messages in thread From: Tamas K Lengyel @ 2016-01-05 11:49 UTC (permalink / raw) To: Xen-devel, Razvan Cojocaru [-- Attachment #1.1: Type: text/plain, Size: 3254 bytes --] Hi all, I've been stress-testing the built-in emulator using the vm_event response VM_EVENT_FLAG_EMULATE feature. In the test I've turned all pages non-readable by default and all trapped instructions to be emulated. My test code can be found at https://github.com/tklengyel/xen/compare/read_emul?expand=1. The following crash is reproducible and has been verified by Razvan as well. (XEN) p2m.c:1726:d1v0 calling mem_access_emulate_one, kind 0 (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0). (XEN) ************* VMCS Area ************** (XEN) *** Guest State *** (XEN) CR0: actual=0x000000008001003b, shadow=0x000000008001003b, gh_mask=ffffffffffffffff (XEN) CR4: actual=0x00000000000426f9, shadow=0x00000000000406f9, gh_mask=ffffffffffffffff (XEN) CR3 = 0x0000000000185000 (XEN) PDPTE0 = 0x0000000000186001 PDPTE1 = 0x0000000000187001 (XEN) PDPTE2 = 0x0000000000188001 PDPTE3 = 0x0000000000189001 (XEN) RSP = 0x000000008276dc28 (0x000000008276dc28) RIP = 0x00000000826bce1c (0x00000000826bce1c) (XEN) RFLAGS=0x00000002 (0x00000002) DR7 = 0x0000000000000400 (XEN) Sysenter RSP=000000008078b000 CS:RIP=0008:00000000826830c0 (XEN) sel attr limit base (XEN) CS: 0008 0c09b ffffffff 0000000000000000 (XEN) DS: 0023 0c0f3 ffffffff 0000000000000000 (XEN) SS: 0010 0c093 ffffffff 0000000000000000 (XEN) ES: 0023 0c0f3 ffffffff 0000000000000000 (XEN) FS: 0030 04093 00003748 0000000082770c00 (XEN) GS: 0000 1c000 ffffffff 0000000000000000 (XEN) GDTR: 000003ff 0000000080b95000 (XEN) LDTR: 0000 1c000 ffffffff 0000000000000000 (XEN) IDTR: 000007ff 0000000080b95400 (XEN) TR: 0028 0008b 000020ab 00000000801da000 (XEN) EFER = 0x0000000000000000 PAT = 0x0007010600070106 (XEN) PreemptionTimer = 0x00000000 SM Base = 0x00000000 (XEN) DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 (XEN) Interruptibility = 00000000 ActivityState = 00000000 (XEN) *** Host State *** (XEN) RIP = 0xffff82d0802075c0 (vmx_asm_vmexit_handler) RSP = 0xffff830430d97f90 (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 (XEN) FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff830430d9bc00 (XEN) GDTBase=ffff830430d8c000 IDTBase=ffff830430d98000 (XEN) CR0=000000008005003b CR3=00000004136d0000 CR4=00000000000426e0 (XEN) Sysenter RSP=ffff830430d97fc0 CS:RIP=e008:ffff82d08024db30 (XEN) EFER = 0x0000000000000000 PAT = 0x0000050100070406 (XEN) *** Control State *** (XEN) PinBased=0000003f CPUBased=b6a075fa SecondaryExec=000000eb (XEN) EntryControls=000051ff ExitControls=000fefff (XEN) ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000 (XEN) VMEntry: intr_info=800000d1 errcode=00000000 ilen=00000000 (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 (XEN) reason=80000021 qualification=0000000000000000 (XEN) IDTVectoring: info=800000d1 errcode=00000000 (XEN) TSC Offset = 0x0000004ed9c86354 (XEN) TPR Threshold = 0x00 PostedIntrVec = 0x00 (XEN) EPT pointer = 0x000000041124e01e EPTP index = 0x0000 (XEN) Virtual processor ID = 0x0011 VMfunc controls = 0000000000000000 (XEN) ************************************** (XEN) domain_crash called from vmx.c:2761 Any tips on how to further debug this issue? Thanks, Tamas [-- Attachment #1.2: Type: text/html, Size: 3603 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 11:49 Failed vm entry with heavy use of emulator Tamas K Lengyel @ 2016-01-05 11:56 ` Andrew Cooper 2016-01-05 12:05 ` Tamas K Lengyel 2016-01-05 13:39 ` Razvan Cojocaru 2016-01-05 12:35 ` Razvan Cojocaru 1 sibling, 2 replies; 15+ messages in thread From: Andrew Cooper @ 2016-01-05 11:56 UTC (permalink / raw) To: Tamas K Lengyel, Xen-devel, Razvan Cojocaru [-- Attachment #1.1: Type: text/plain, Size: 3533 bytes --] On 05/01/16 11:49, Tamas K Lengyel wrote: > Hi all, > I've been stress-testing the built-in emulator using the vm_event > response VM_EVENT_FLAG_EMULATE feature. In the test I've turned all > pages non-readable by default and all trapped instructions to be > emulated. My test code can be found at > https://github.com/tklengyel/xen/compare/read_emul?expand=1. > > The following crash is reproducible and has been verified by Razvan as > well. > > (XEN) p2m.c:1726:d1v0 calling mem_access_emulate_one, kind 0 > (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest > state (0). > (XEN) ************* VMCS Area ************** > (XEN) *** Guest State *** > (XEN) CR0: actual=0x000000008001003b, shadow=0x000000008001003b, > gh_mask=ffffffffffffffff > (XEN) CR4: actual=0x00000000000426f9, shadow=0x00000000000406f9, > gh_mask=ffffffffffffffff > (XEN) CR3 = 0x0000000000185000 > (XEN) PDPTE0 = 0x0000000000186001 PDPTE1 = 0x0000000000187001 > (XEN) PDPTE2 = 0x0000000000188001 PDPTE3 = 0x0000000000189001 > (XEN) RSP = 0x000000008276dc28 (0x000000008276dc28) RIP = > 0x00000000826bce1c (0x00000000826bce1c) > (XEN) RFLAGS=0x00000002 (0x00000002) DR7 = 0x0000000000000400 > (XEN) Sysenter RSP=000000008078b000 CS:RIP=0008:00000000826830c0 > (XEN) sel attr limit base > (XEN) CS: 0008 0c09b ffffffff 0000000000000000 > (XEN) DS: 0023 0c0f3 ffffffff 0000000000000000 > (XEN) SS: 0010 0c093 ffffffff 0000000000000000 > (XEN) ES: 0023 0c0f3 ffffffff 0000000000000000 > (XEN) FS: 0030 04093 00003748 0000000082770c00 > (XEN) GS: 0000 1c000 ffffffff 0000000000000000 > (XEN) GDTR: 000003ff 0000000080b95000 > (XEN) LDTR: 0000 1c000 ffffffff 0000000000000000 > (XEN) IDTR: 000007ff 0000000080b95400 > (XEN) TR: 0028 0008b 000020ab 00000000801da000 > (XEN) EFER = 0x0000000000000000 PAT = 0x0007010600070106 > (XEN) PreemptionTimer = 0x00000000 SM Base = 0x00000000 > (XEN) DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 > (XEN) Interruptibility = 00000000 ActivityState = 00000000 > (XEN) *** Host State *** > (XEN) RIP = 0xffff82d0802075c0 (vmx_asm_vmexit_handler) RSP = > 0xffff830430d97f90 > (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 > (XEN) FSBase=0000000000000000 GSBase=0000000000000000 > TRBase=ffff830430d9bc00 > (XEN) GDTBase=ffff830430d8c000 IDTBase=ffff830430d98000 > (XEN) CR0=000000008005003b CR3=00000004136d0000 CR4=00000000000426e0 > (XEN) Sysenter RSP=ffff830430d97fc0 CS:RIP=e008:ffff82d08024db30 > (XEN) EFER = 0x0000000000000000 PAT = 0x0000050100070406 > (XEN) *** Control State *** > (XEN) PinBased=0000003f CPUBased=b6a075fa SecondaryExec=000000eb > (XEN) EntryControls=000051ff ExitControls=000fefff > (XEN) ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000 > (XEN) VMEntry: intr_info=800000d1 errcode=00000000 ilen=00000000 > (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 > (XEN) reason=80000021 qualification=0000000000000000 > (XEN) IDTVectoring: info=800000d1 errcode=00000000 > (XEN) TSC Offset = 0x0000004ed9c86354 > (XEN) TPR Threshold = 0x00 PostedIntrVec = 0x00 > (XEN) EPT pointer = 0x000000041124e01e EPTP index = 0x0000 > (XEN) Virtual processor ID = 0x0011 VMfunc controls = 0000000000000000 > (XEN) ************************************** > (XEN) domain_crash called from vmx.c:2761 > > Any tips on how to further debug this issue? Do you have a log of the instructions emulated? Has the emulator by any chance just emulated setting CR4.PAE? ~Andrew [-- Attachment #1.2: Type: text/html, Size: 5022 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 11:56 ` Andrew Cooper @ 2016-01-05 12:05 ` Tamas K Lengyel 2016-01-05 13:45 ` Andrew Cooper 2016-01-06 14:21 ` Jan Beulich 2016-01-05 13:39 ` Razvan Cojocaru 1 sibling, 2 replies; 15+ messages in thread From: Tamas K Lengyel @ 2016-01-05 12:05 UTC (permalink / raw) To: Andrew Cooper; +Cc: Xen-devel, Razvan Cojocaru [-- Attachment #1.1: Type: text/plain, Size: 3927 bytes --] On Tue, Jan 5, 2016 at 12:56 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote: > On 05/01/16 11:49, Tamas K Lengyel wrote: > > Hi all, > I've been stress-testing the built-in emulator using the vm_event response > VM_EVENT_FLAG_EMULATE feature. In the test I've turned all pages > non-readable by default and all trapped instructions to be emulated. My > test code can be found at > <https://github.com/tklengyel/xen/compare/read_emul?expand=1> > https://github.com/tklengyel/xen/compare/read_emul?expand=1. > > The following crash is reproducible and has been verified by Razvan as > well. > > (XEN) p2m.c:1726:d1v0 calling mem_access_emulate_one, kind 0 > (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest > state (0). > (XEN) ************* VMCS Area ************** > (XEN) *** Guest State *** > (XEN) CR0: actual=0x000000008001003b, shadow=0x000000008001003b, > gh_mask=ffffffffffffffff > (XEN) CR4: actual=0x00000000000426f9, shadow=0x00000000000406f9, > gh_mask=ffffffffffffffff > (XEN) CR3 = 0x0000000000185000 > (XEN) PDPTE0 = 0x0000000000186001 PDPTE1 = 0x0000000000187001 > (XEN) PDPTE2 = 0x0000000000188001 PDPTE3 = 0x0000000000189001 > (XEN) RSP = 0x000000008276dc28 (0x000000008276dc28) RIP = > 0x00000000826bce1c (0x00000000826bce1c) > (XEN) RFLAGS=0x00000002 (0x00000002) DR7 = 0x0000000000000400 > (XEN) Sysenter RSP=000000008078b000 CS:RIP=0008:00000000826830c0 > (XEN) sel attr limit base > (XEN) CS: 0008 0c09b ffffffff 0000000000000000 > (XEN) DS: 0023 0c0f3 ffffffff 0000000000000000 > (XEN) SS: 0010 0c093 ffffffff 0000000000000000 > (XEN) ES: 0023 0c0f3 ffffffff 0000000000000000 > (XEN) FS: 0030 04093 00003748 0000000082770c00 > (XEN) GS: 0000 1c000 ffffffff 0000000000000000 > (XEN) GDTR: 000003ff 0000000080b95000 > (XEN) LDTR: 0000 1c000 ffffffff 0000000000000000 > (XEN) IDTR: 000007ff 0000000080b95400 > (XEN) TR: 0028 0008b 000020ab 00000000801da000 > (XEN) EFER = 0x0000000000000000 PAT = 0x0007010600070106 > (XEN) PreemptionTimer = 0x00000000 SM Base = 0x00000000 > (XEN) DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 > (XEN) Interruptibility = 00000000 ActivityState = 00000000 > (XEN) *** Host State *** > (XEN) RIP = 0xffff82d0802075c0 (vmx_asm_vmexit_handler) RSP = > 0xffff830430d97f90 > (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 > (XEN) FSBase=0000000000000000 GSBase=0000000000000000 > TRBase=ffff830430d9bc00 > (XEN) GDTBase=ffff830430d8c000 IDTBase=ffff830430d98000 > (XEN) CR0=000000008005003b CR3=00000004136d0000 CR4=00000000000426e0 > (XEN) Sysenter RSP=ffff830430d97fc0 CS:RIP=e008:ffff82d08024db30 > (XEN) EFER = 0x0000000000000000 PAT = 0x0000050100070406 > (XEN) *** Control State *** > (XEN) PinBased=0000003f CPUBased=b6a075fa SecondaryExec=000000eb > (XEN) EntryControls=000051ff ExitControls=000fefff > (XEN) ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000 > (XEN) VMEntry: intr_info=800000d1 errcode=00000000 ilen=00000000 > (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 > (XEN) reason=80000021 qualification=0000000000000000 > (XEN) IDTVectoring: info=800000d1 errcode=00000000 > (XEN) TSC Offset = 0x0000004ed9c86354 > (XEN) TPR Threshold = 0x00 PostedIntrVec = 0x00 > (XEN) EPT pointer = 0x000000041124e01e EPTP index = 0x0000 > (XEN) Virtual processor ID = 0x0011 VMfunc controls = 0000000000000000 > (XEN) ************************************** > (XEN) domain_crash called from vmx.c:2761 > > Any tips on how to further debug this issue? > > > Do you have a log of the instructions emulated? > I don't. Is there an easy way to get that beside manually sprinkling debug messages around in the emulator? > > Has the emulator by any chance just emulated setting CR4.PAE? > Possibly but I don't think so as the guest has already been fully booted so I would not expect it to touch that. Thanks, Tamas [-- Attachment #1.2: Type: text/html, Size: 5702 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 12:05 ` Tamas K Lengyel @ 2016-01-05 13:45 ` Andrew Cooper 2016-01-06 14:21 ` Jan Beulich 1 sibling, 0 replies; 15+ messages in thread From: Andrew Cooper @ 2016-01-05 13:45 UTC (permalink / raw) To: Tamas K Lengyel; +Cc: Xen-devel, Razvan Cojocaru [-- Attachment #1.1: Type: text/plain, Size: 4681 bytes --] On 05/01/16 12:05, Tamas K Lengyel wrote: > > > On Tue, Jan 5, 2016 at 12:56 PM, Andrew Cooper > <andrew.cooper3@citrix.com <mailto:andrew.cooper3@citrix.com>> wrote: > > On 05/01/16 11:49, Tamas K Lengyel wrote: >> Hi all, >> I've been stress-testing the built-in emulator using the vm_event >> response VM_EVENT_FLAG_EMULATE feature. In the test I've turned >> all pages non-readable by default and all trapped instructions to >> be emulated. My test code can be found at >> https://github.com/tklengyel/xen/compare/read_emul?expand=1. >> >> The following crash is reproducible and has been verified by >> Razvan as well. >> >> (XEN) p2m.c:1726:d1v0 calling mem_access_emulate_one, kind 0 >> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid >> guest state (0). >> (XEN) ************* VMCS Area ************** >> (XEN) *** Guest State *** >> (XEN) CR0: actual=0x000000008001003b, shadow=0x000000008001003b, >> gh_mask=ffffffffffffffff >> (XEN) CR4: actual=0x00000000000426f9, shadow=0x00000000000406f9, >> gh_mask=ffffffffffffffff >> (XEN) CR3 = 0x0000000000185000 >> (XEN) PDPTE0 = 0x0000000000186001 PDPTE1 = 0x0000000000187001 >> (XEN) PDPTE2 = 0x0000000000188001 PDPTE3 = 0x0000000000189001 >> (XEN) RSP = 0x000000008276dc28 (0x000000008276dc28) RIP = >> 0x00000000826bce1c (0x00000000826bce1c) >> (XEN) RFLAGS=0x00000002 (0x00000002) DR7 = 0x0000000000000400 >> (XEN) Sysenter RSP=000000008078b000 CS:RIP=0008:00000000826830c0 >> (XEN) sel attr limit base >> (XEN) CS: 0008 0c09b ffffffff 0000000000000000 >> (XEN) DS: 0023 0c0f3 ffffffff 0000000000000000 >> (XEN) SS: 0010 0c093 ffffffff 0000000000000000 >> (XEN) ES: 0023 0c0f3 ffffffff 0000000000000000 >> (XEN) FS: 0030 04093 00003748 0000000082770c00 >> (XEN) GS: 0000 1c000 ffffffff 0000000000000000 >> (XEN) GDTR: 000003ff 0000000080b95000 >> (XEN) LDTR: 0000 1c000 ffffffff 0000000000000000 >> (XEN) IDTR: 000007ff 0000000080b95400 >> (XEN) TR: 0028 0008b 000020ab 00000000801da000 >> (XEN) EFER = 0x0000000000000000 PAT = 0x0007010600070106 >> (XEN) PreemptionTimer = 0x00000000 SM Base = 0x00000000 >> (XEN) DebugCtl = 0x0000000000000000 DebugExceptions = >> 0x0000000000000000 >> (XEN) Interruptibility = 00000000 ActivityState = 00000000 >> (XEN) *** Host State *** >> (XEN) RIP = 0xffff82d0802075c0 (vmx_asm_vmexit_handler) RSP = >> 0xffff830430d97f90 >> (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 >> (XEN) FSBase=0000000000000000 GSBase=0000000000000000 >> TRBase=ffff830430d9bc00 >> (XEN) GDTBase=ffff830430d8c000 IDTBase=ffff830430d98000 >> (XEN) CR0=000000008005003b CR3=00000004136d0000 CR4=00000000000426e0 >> (XEN) Sysenter RSP=ffff830430d97fc0 CS:RIP=e008:ffff82d08024db30 >> (XEN) EFER = 0x0000000000000000 PAT = 0x0000050100070406 >> (XEN) *** Control State *** >> (XEN) PinBased=0000003f CPUBased=b6a075fa SecondaryExec=000000eb >> (XEN) EntryControls=000051ff ExitControls=000fefff >> (XEN) ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000 >> (XEN) VMEntry: intr_info=800000d1 errcode=00000000 ilen=00000000 >> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 >> (XEN) reason=80000021 qualification=0000000000000000 >> (XEN) IDTVectoring: info=800000d1 errcode=00000000 >> (XEN) TSC Offset = 0x0000004ed9c86354 >> (XEN) TPR Threshold = 0x00 PostedIntrVec = 0x00 >> (XEN) EPT pointer = 0x000000041124e01e EPTP index = 0x0000 >> (XEN) Virtual processor ID = 0x0011 VMfunc controls = >> 0000000000000000 >> (XEN) ************************************** >> (XEN) domain_crash called from vmx.c:2761 >> >> Any tips on how to further debug this issue? > > Do you have a log of the instructions emulated? > > > I don't. Is there an easy way to get that beside manually sprinkling > debug messages around in the emulator? Not trivially, sadly. > > > > Has the emulator by any chance just emulated setting CR4.PAE? > > > Possibly but I don't think so as the guest has already been fully > booted so I would not expect it to touch that. At a guess, I think the fault is an emulated 'mov %reg, %cr3' while in 32bit PAE mode. The PDPTE{0..3} values look wonky. I encountered a similar crash with the xen test framework in HAP mode with a bad %cr3 update. The VMM is expected to emulate updates to PDPTE{0..3} if writes to %cr3 are trapped. See vmx_update_guest_cr() and vmx_load_pdptrs(). ~Andrew [-- Attachment #1.2: Type: text/html, Size: 9194 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 12:05 ` Tamas K Lengyel 2016-01-05 13:45 ` Andrew Cooper @ 2016-01-06 14:21 ` Jan Beulich 1 sibling, 0 replies; 15+ messages in thread From: Jan Beulich @ 2016-01-06 14:21 UTC (permalink / raw) To: Tamas K Lengyel; +Cc: Andrew Cooper, Razvan Cojocaru, Xen-devel >>> On 05.01.16 at 13:05, <tamas.k.lengyel@gmail.com> wrote: > On Tue, Jan 5, 2016 at 12:56 PM, Andrew Cooper <andrew.cooper3@citrix.com> > wrote: >> Do you have a log of the instructions emulated? > > I don't. Is there an easy way to get that beside manually sprinkling debug > messages around in the emulator? Assuming you have the guest kernel binary (or binaries, in e.g. the Windows case), just disassemble the kernel binary. Considering later replies, maybe you even see crashes on different instructions, which may allow deriving a pattern. Jan ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 11:56 ` Andrew Cooper 2016-01-05 12:05 ` Tamas K Lengyel @ 2016-01-05 13:39 ` Razvan Cojocaru 2016-01-05 13:49 ` Andrew Cooper 1 sibling, 1 reply; 15+ messages in thread From: Razvan Cojocaru @ 2016-01-05 13:39 UTC (permalink / raw) To: Andrew Cooper, Tamas K Lengyel, Xen-devel On 01/05/2016 01:56 PM, Andrew Cooper wrote: > On 05/01/16 11:49, Tamas K Lengyel wrote: >> Hi all, >> I've been stress-testing the built-in emulator using the vm_event >> response VM_EVENT_FLAG_EMULATE feature. In the test I've turned all >> pages non-readable by default and all trapped instructions to be >> emulated. My test code can be found at >> <https://github.com/tklengyel/xen/compare/read_emul?expand=1>https://github.com/tklengyel/xen/compare/read_emul?expand=1. >> >> The following crash is reproducible and has been verified by Razvan as >> well. >> >> (XEN) p2m.c:1726:d1v0 calling mem_access_emulate_one, kind 0 >> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest >> state (0). >> (XEN) ************* VMCS Area ************** >> (XEN) *** Guest State *** >> (XEN) CR0: actual=0x000000008001003b, shadow=0x000000008001003b, >> gh_mask=ffffffffffffffff >> (XEN) CR4: actual=0x00000000000426f9, shadow=0x00000000000406f9, >> gh_mask=ffffffffffffffff >> (XEN) CR3 = 0x0000000000185000 >> (XEN) PDPTE0 = 0x0000000000186001 PDPTE1 = 0x0000000000187001 >> (XEN) PDPTE2 = 0x0000000000188001 PDPTE3 = 0x0000000000189001 >> (XEN) RSP = 0x000000008276dc28 (0x000000008276dc28) RIP = >> 0x00000000826bce1c (0x00000000826bce1c) >> (XEN) RFLAGS=0x00000002 (0x00000002) DR7 = 0x0000000000000400 >> (XEN) Sysenter RSP=000000008078b000 CS:RIP=0008:00000000826830c0 >> (XEN) sel attr limit base >> (XEN) CS: 0008 0c09b ffffffff 0000000000000000 >> (XEN) DS: 0023 0c0f3 ffffffff 0000000000000000 >> (XEN) SS: 0010 0c093 ffffffff 0000000000000000 >> (XEN) ES: 0023 0c0f3 ffffffff 0000000000000000 >> (XEN) FS: 0030 04093 00003748 0000000082770c00 >> (XEN) GS: 0000 1c000 ffffffff 0000000000000000 >> (XEN) GDTR: 000003ff 0000000080b95000 >> (XEN) LDTR: 0000 1c000 ffffffff 0000000000000000 >> (XEN) IDTR: 000007ff 0000000080b95400 >> (XEN) TR: 0028 0008b 000020ab 00000000801da000 >> (XEN) EFER = 0x0000000000000000 PAT = 0x0007010600070106 >> (XEN) PreemptionTimer = 0x00000000 SM Base = 0x00000000 >> (XEN) DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 >> (XEN) Interruptibility = 00000000 ActivityState = 00000000 >> (XEN) *** Host State *** >> (XEN) RIP = 0xffff82d0802075c0 (vmx_asm_vmexit_handler) RSP = >> 0xffff830430d97f90 >> (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 >> (XEN) FSBase=0000000000000000 GSBase=0000000000000000 >> TRBase=ffff830430d9bc00 >> (XEN) GDTBase=ffff830430d8c000 IDTBase=ffff830430d98000 >> (XEN) CR0=000000008005003b CR3=00000004136d0000 CR4=00000000000426e0 >> (XEN) Sysenter RSP=ffff830430d97fc0 CS:RIP=e008:ffff82d08024db30 >> (XEN) EFER = 0x0000000000000000 PAT = 0x0000050100070406 >> (XEN) *** Control State *** >> (XEN) PinBased=0000003f CPUBased=b6a075fa SecondaryExec=000000eb >> (XEN) EntryControls=000051ff ExitControls=000fefff >> (XEN) ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000 >> (XEN) VMEntry: intr_info=800000d1 errcode=00000000 ilen=00000000 >> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 >> (XEN) reason=80000021 qualification=0000000000000000 >> (XEN) IDTVectoring: info=800000d1 errcode=00000000 >> (XEN) TSC Offset = 0x0000004ed9c86354 >> (XEN) TPR Threshold = 0x00 PostedIntrVec = 0x00 >> (XEN) EPT pointer = 0x000000041124e01e EPTP index = 0x0000 >> (XEN) Virtual processor ID = 0x0011 VMfunc controls = 0000000000000000 >> (XEN) ************************************** >> (XEN) domain_crash called from vmx.c:2761 >> >> Any tips on how to further debug this issue? > > Do you have a log of the instructions emulated? Here's a quick log of the emulated instructions on my setup: http://pastebin.com/raw/XXQ0Lnzh Hope this helps. Thanks, Razvan ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 13:39 ` Razvan Cojocaru @ 2016-01-05 13:49 ` Andrew Cooper 2016-01-05 14:01 ` Razvan Cojocaru 0 siblings, 1 reply; 15+ messages in thread From: Andrew Cooper @ 2016-01-05 13:49 UTC (permalink / raw) To: xen-devel On 05/01/16 13:39, Razvan Cojocaru wrote: > On 01/05/2016 01:56 PM, Andrew Cooper wrote: >> On 05/01/16 11:49, Tamas K Lengyel wrote: >>> Hi all, >>> I've been stress-testing the built-in emulator using the vm_event >>> response VM_EVENT_FLAG_EMULATE feature. In the test I've turned all >>> pages non-readable by default and all trapped instructions to be >>> emulated. My test code can be found at >>> <https://github.com/tklengyel/xen/compare/read_emul?expand=1>https://github.com/tklengyel/xen/compare/read_emul?expand=1. >>> >>> The following crash is reproducible and has been verified by Razvan as >>> well. >>> >>> (XEN) p2m.c:1726:d1v0 calling mem_access_emulate_one, kind 0 >>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest >>> state (0). >>> (XEN) ************* VMCS Area ************** >>> (XEN) *** Guest State *** >>> (XEN) CR0: actual=0x000000008001003b, shadow=0x000000008001003b, >>> gh_mask=ffffffffffffffff >>> (XEN) CR4: actual=0x00000000000426f9, shadow=0x00000000000406f9, >>> gh_mask=ffffffffffffffff >>> (XEN) CR3 = 0x0000000000185000 >>> (XEN) PDPTE0 = 0x0000000000186001 PDPTE1 = 0x0000000000187001 >>> (XEN) PDPTE2 = 0x0000000000188001 PDPTE3 = 0x0000000000189001 >>> (XEN) RSP = 0x000000008276dc28 (0x000000008276dc28) RIP = >>> 0x00000000826bce1c (0x00000000826bce1c) >>> (XEN) RFLAGS=0x00000002 (0x00000002) DR7 = 0x0000000000000400 >>> (XEN) Sysenter RSP=000000008078b000 CS:RIP=0008:00000000826830c0 >>> (XEN) sel attr limit base >>> (XEN) CS: 0008 0c09b ffffffff 0000000000000000 >>> (XEN) DS: 0023 0c0f3 ffffffff 0000000000000000 >>> (XEN) SS: 0010 0c093 ffffffff 0000000000000000 >>> (XEN) ES: 0023 0c0f3 ffffffff 0000000000000000 >>> (XEN) FS: 0030 04093 00003748 0000000082770c00 >>> (XEN) GS: 0000 1c000 ffffffff 0000000000000000 >>> (XEN) GDTR: 000003ff 0000000080b95000 >>> (XEN) LDTR: 0000 1c000 ffffffff 0000000000000000 >>> (XEN) IDTR: 000007ff 0000000080b95400 >>> (XEN) TR: 0028 0008b 000020ab 00000000801da000 >>> (XEN) EFER = 0x0000000000000000 PAT = 0x0007010600070106 >>> (XEN) PreemptionTimer = 0x00000000 SM Base = 0x00000000 >>> (XEN) DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 >>> (XEN) Interruptibility = 00000000 ActivityState = 00000000 >>> (XEN) *** Host State *** >>> (XEN) RIP = 0xffff82d0802075c0 (vmx_asm_vmexit_handler) RSP = >>> 0xffff830430d97f90 >>> (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 >>> (XEN) FSBase=0000000000000000 GSBase=0000000000000000 >>> TRBase=ffff830430d9bc00 >>> (XEN) GDTBase=ffff830430d8c000 IDTBase=ffff830430d98000 >>> (XEN) CR0=000000008005003b CR3=00000004136d0000 CR4=00000000000426e0 >>> (XEN) Sysenter RSP=ffff830430d97fc0 CS:RIP=e008:ffff82d08024db30 >>> (XEN) EFER = 0x0000000000000000 PAT = 0x0000050100070406 >>> (XEN) *** Control State *** >>> (XEN) PinBased=0000003f CPUBased=b6a075fa SecondaryExec=000000eb >>> (XEN) EntryControls=000051ff ExitControls=000fefff >>> (XEN) ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000 >>> (XEN) VMEntry: intr_info=800000d1 errcode=00000000 ilen=00000000 >>> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 >>> (XEN) reason=80000021 qualification=0000000000000000 >>> (XEN) IDTVectoring: info=800000d1 errcode=00000000 >>> (XEN) TSC Offset = 0x0000004ed9c86354 >>> (XEN) TPR Threshold = 0x00 PostedIntrVec = 0x00 >>> (XEN) EPT pointer = 0x000000041124e01e EPTP index = 0x0000 >>> (XEN) Virtual processor ID = 0x0011 VMfunc controls = 0000000000000000 >>> (XEN) ************************************** >>> (XEN) domain_crash called from vmx.c:2761 >>> >>> Any tips on how to further debug this issue? >> Do you have a log of the instructions emulated? > Here's a quick log of the emulated instructions on my setup: > http://pastebin.com/raw/XXQ0Lnzh Hmm - according to that, the final instruction emulated was d1v0 32bit @ 0008:828925db -> fa which is the 'cli' instruction. I would start there - I doubt it is an instruction which is emulated often. ~Andrew ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 13:49 ` Andrew Cooper @ 2016-01-05 14:01 ` Razvan Cojocaru 2016-01-05 14:12 ` Andrew Cooper 2016-01-05 14:16 ` Tamas K Lengyel 0 siblings, 2 replies; 15+ messages in thread From: Razvan Cojocaru @ 2016-01-05 14:01 UTC (permalink / raw) To: Andrew Cooper, xen-devel, Tamas Lengyel On 01/05/2016 03:49 PM, Andrew Cooper wrote: > On 05/01/16 13:39, Razvan Cojocaru wrote: >> Here's a quick log of the emulated instructions on my setup: >> http://pastebin.com/raw/XXQ0Lnzh > > Hmm - according to that, the final instruction emulated was > > d1v0 32bit @ 0008:828925db -> fa > > which is the 'cli' instruction. > > I would start there - I doubt it is an instruction which is emulated often. My code (arch/x86/x86_emulate/x86_emulate.c) does have a case label for it: 3677 case 0xfa: /* cli */ 3678 generate_exception_if(!mode_iopl(), EXC_GP, 0); 3679 _regs.eflags &= ~EFLG_IF; 3680 break; Maybe the IOPL test fails there? Tamas, does your guest die after a CLI as well? Thanks, Razvan ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 14:01 ` Razvan Cojocaru @ 2016-01-05 14:12 ` Andrew Cooper 2016-01-05 14:16 ` Tamas K Lengyel 1 sibling, 0 replies; 15+ messages in thread From: Andrew Cooper @ 2016-01-05 14:12 UTC (permalink / raw) To: Razvan Cojocaru, xen-devel, Tamas Lengyel On 05/01/16 14:01, Razvan Cojocaru wrote: > On 01/05/2016 03:49 PM, Andrew Cooper wrote: >> On 05/01/16 13:39, Razvan Cojocaru wrote: >>> Here's a quick log of the emulated instructions on my setup: >>> http://pastebin.com/raw/XXQ0Lnzh >> Hmm - according to that, the final instruction emulated was >> >> d1v0 32bit @ 0008:828925db -> fa >> >> which is the 'cli' instruction. >> >> I would start there - I doubt it is an instruction which is emulated often. > My code (arch/x86/x86_emulate/x86_emulate.c) does have a case label for it: > > 3677 case 0xfa: /* cli */ > 3678 generate_exception_if(!mode_iopl(), EXC_GP, 0); > 3679 _regs.eflags &= ~EFLG_IF; > 3680 break; > > Maybe the IOPL test fails there? Tamas, does your guest die after a CLI > as well? The iopl test looks correct. It is quite possible that eflags.IF is expected to match a separate piece of control state in the vmcs, and that is the cause of the vmentry failure. ~Andrew ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 14:01 ` Razvan Cojocaru 2016-01-05 14:12 ` Andrew Cooper @ 2016-01-05 14:16 ` Tamas K Lengyel 2016-01-05 14:37 ` Razvan Cojocaru 1 sibling, 1 reply; 15+ messages in thread From: Tamas K Lengyel @ 2016-01-05 14:16 UTC (permalink / raw) To: Razvan Cojocaru; +Cc: Andrew Cooper, Xen-devel [-- Attachment #1.1: Type: text/plain, Size: 982 bytes --] On Tue, Jan 5, 2016 at 3:01 PM, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote: > On 01/05/2016 03:49 PM, Andrew Cooper wrote: > > On 05/01/16 13:39, Razvan Cojocaru wrote: > >> Here's a quick log of the emulated instructions on my setup: > >> http://pastebin.com/raw/XXQ0Lnzh > > > > Hmm - according to that, the final instruction emulated was > > > > d1v0 32bit @ 0008:828925db -> fa > > > > which is the 'cli' instruction. > > > > I would start there - I doubt it is an instruction which is emulated > often. > > My code (arch/x86/x86_emulate/x86_emulate.c) does have a case label for it: > > 3677 case 0xfa: /* cli */ > 3678 generate_exception_if(!mode_iopl(), EXC_GP, 0); > 3679 _regs.eflags &= ~EFLG_IF; > 3680 break; > > Maybe the IOPL test fails there? Tamas, does your guest die after a CLI > as well? > I've added a gdprintk(XENLOG_DEBUG, "emulate 0xfa cli\n"); into that switch case but it wasn't printed before the guest crashed. Tamas [-- Attachment #1.2: Type: text/html, Size: 1629 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 14:16 ` Tamas K Lengyel @ 2016-01-05 14:37 ` Razvan Cojocaru 2016-09-06 23:31 ` Tamas K Lengyel 0 siblings, 1 reply; 15+ messages in thread From: Razvan Cojocaru @ 2016-01-05 14:37 UTC (permalink / raw) To: Tamas K Lengyel; +Cc: Andrew Cooper, Xen-devel On 01/05/2016 04:16 PM, Tamas K Lengyel wrote: > > > On Tue, Jan 5, 2016 at 3:01 PM, Razvan Cojocaru > <rcojocaru@bitdefender.com <mailto:rcojocaru@bitdefender.com>> wrote: > > On 01/05/2016 03:49 PM, Andrew Cooper wrote: > > On 05/01/16 13:39, Razvan Cojocaru wrote: > >> Here's a quick log of the emulated instructions on my setup: > >> http://pastebin.com/raw/XXQ0Lnzh > > > > Hmm - according to that, the final instruction emulated was > > > > d1v0 32bit @ 0008:828925db -> fa > > > > which is the 'cli' instruction. > > > > I would start there - I doubt it is an instruction which is emulated often. > > My code (arch/x86/x86_emulate/x86_emulate.c) does have a case label > for it: > > 3677 case 0xfa: /* cli */ > 3678 generate_exception_if(!mode_iopl(), EXC_GP, 0); > 3679 _regs.eflags &= ~EFLG_IF; > 3680 break; > > Maybe the IOPL test fails there? Tamas, does your guest die after a CLI > as well? > > > I've added a gdprintk(XENLOG_DEBUG, "emulate 0xfa cli\n"); into that > switch case but it wasn't printed before the guest crashed. It's possible that your guest crashes after emulating a different instruction. I've added a line to xen/arch/x86/hvm/emulate.c, in hvm_mem_access_emulate_one(): 1790 switch ( kind ) 1791 { 1792 case EMUL_KIND_NOWRITE: 1793 rc = hvm_emulate_one_no_write(&ctx); 1794 break; 1795 case EMUL_KIND_SET_CONTEXT: 1796 ctx.set_context = 1; 1797 /* Intentional fall-through. */ 1798 default: 1799 rc = hvm_emulate_one(&ctx); 1800 hvm_dump_emulation_state(XENLOG_G_DEBUG, &ctx); 1801 } so I can then see which instruction was the last before the stack trace with xl dmesg (or looking at the log file, etc.) It's possible the problem is not specific to CLI, or maybe it's even something that happens prior to emulating the last instruction that leads to a corruption in the guest's state later on. Cheers, Razvan ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 14:37 ` Razvan Cojocaru @ 2016-09-06 23:31 ` Tamas K Lengyel 2016-09-07 5:59 ` Razvan Cojocaru 0 siblings, 1 reply; 15+ messages in thread From: Tamas K Lengyel @ 2016-09-06 23:31 UTC (permalink / raw) To: Razvan Cojocaru; +Cc: Tamas K Lengyel, Xen-devel, Andrew Cooper On Tue, Jan 5, 2016 at 7:37 AM, Razvan Cojocaru <rcojocaru@bitdefender.com> wrote: > On 01/05/2016 04:16 PM, Tamas K Lengyel wrote: >> >> >> On Tue, Jan 5, 2016 at 3:01 PM, Razvan Cojocaru >> <rcojocaru@bitdefender.com <mailto:rcojocaru@bitdefender.com>> wrote: >> >> On 01/05/2016 03:49 PM, Andrew Cooper wrote: >> > On 05/01/16 13:39, Razvan Cojocaru wrote: >> >> Here's a quick log of the emulated instructions on my setup: >> >> http://pastebin.com/raw/XXQ0Lnzh >> > >> > Hmm - according to that, the final instruction emulated was >> > >> > d1v0 32bit @ 0008:828925db -> fa >> > >> > which is the 'cli' instruction. >> > >> > I would start there - I doubt it is an instruction which is emulated often. >> >> My code (arch/x86/x86_emulate/x86_emulate.c) does have a case label >> for it: >> >> 3677 case 0xfa: /* cli */ >> 3678 generate_exception_if(!mode_iopl(), EXC_GP, 0); >> 3679 _regs.eflags &= ~EFLG_IF; >> 3680 break; >> >> Maybe the IOPL test fails there? Tamas, does your guest die after a CLI >> as well? >> >> >> I've added a gdprintk(XENLOG_DEBUG, "emulate 0xfa cli\n"); into that >> switch case but it wasn't printed before the guest crashed. > > It's possible that your guest crashes after emulating a different > instruction. I've added a line to xen/arch/x86/hvm/emulate.c, in > hvm_mem_access_emulate_one(): > > 1790 switch ( kind ) > 1791 { > 1792 case EMUL_KIND_NOWRITE: > 1793 rc = hvm_emulate_one_no_write(&ctx); > 1794 break; > 1795 case EMUL_KIND_SET_CONTEXT: > 1796 ctx.set_context = 1; > 1797 /* Intentional fall-through. */ > 1798 default: > 1799 rc = hvm_emulate_one(&ctx); > 1800 hvm_dump_emulation_state(XENLOG_G_DEBUG, &ctx); > 1801 } > > so I can then see which instruction was the last before the stack trace > with xl dmesg (or looking at the log file, etc.) > > It's possible the problem is not specific to CLI, or maybe it's even > something that happens prior to emulating the last instruction that > leads to a corruption in the guest's state later on. > Just a quick update on this issue, I also now see the following emulation error when I use xen-access with the emulation response for the exec case. The issue previously reported with the failed vm entry seem to happen only when emulating in response to the write-access. (XEN) Mem event emulation failed: d5v1 32bit @ 0008:826c602c -> 0f 21 c0 89 82 dc 02 00 00 0f 21 c8 89 82 e0 02 (XEN) Mem event emulation failed: d5v0 32bit @ 0008:82678caa -> cf 90 f7 45 70 00 00 02 00 75 09 f7 45 6c 01 00 (XEN) d5v1 Triple fault - invoking HVM shutdown action (XEN) *** Dumping Dom5 vcpu#1 state: *** (XEN) ----[ Xen-4.7.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: 0008:[<00000000827194f8>] (XEN) RFLAGS: 0000000000014292 CONTEXT: hvm guest (d5v1) (XEN) rax: 00000000badb10cc rbx: 00000000807c1534 rcx: 0000000000000000 (XEN) rdx: 0000000000000000 rsi: 000000008271957f rdi: 00000000badb10cc (XEN) rbp: 00000000807c10e4 rsp: 00000000807c0d30 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 000000008001003b cr4: 00000000000406f9 (XEN) cr3: 0000000000185000 cr2: 00000000807c0d2c (XEN) ds: 0023 es: 0023 fs: 0030 gs: 0000 ss: 0010 cs: 0008 Tamas _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-09-06 23:31 ` Tamas K Lengyel @ 2016-09-07 5:59 ` Razvan Cojocaru 2016-09-07 9:36 ` Jan Beulich 0 siblings, 1 reply; 15+ messages in thread From: Razvan Cojocaru @ 2016-09-07 5:59 UTC (permalink / raw) To: Tamas K Lengyel; +Cc: Tamas K Lengyel, Xen-devel, Andrew Cooper On 09/07/16 02:31, Tamas K Lengyel wrote: > On Tue, Jan 5, 2016 at 7:37 AM, Razvan Cojocaru > <rcojocaru@bitdefender.com> wrote: >> On 01/05/2016 04:16 PM, Tamas K Lengyel wrote: >>> >>> >>> On Tue, Jan 5, 2016 at 3:01 PM, Razvan Cojocaru >>> <rcojocaru@bitdefender.com <mailto:rcojocaru@bitdefender.com>> wrote: >>> >>> On 01/05/2016 03:49 PM, Andrew Cooper wrote: >>> > On 05/01/16 13:39, Razvan Cojocaru wrote: >>> >> Here's a quick log of the emulated instructions on my setup: >>> >> http://pastebin.com/raw/XXQ0Lnzh >>> > >>> > Hmm - according to that, the final instruction emulated was >>> > >>> > d1v0 32bit @ 0008:828925db -> fa >>> > >>> > which is the 'cli' instruction. >>> > >>> > I would start there - I doubt it is an instruction which is emulated often. >>> >>> My code (arch/x86/x86_emulate/x86_emulate.c) does have a case label >>> for it: >>> >>> 3677 case 0xfa: /* cli */ >>> 3678 generate_exception_if(!mode_iopl(), EXC_GP, 0); >>> 3679 _regs.eflags &= ~EFLG_IF; >>> 3680 break; >>> >>> Maybe the IOPL test fails there? Tamas, does your guest die after a CLI >>> as well? >>> >>> >>> I've added a gdprintk(XENLOG_DEBUG, "emulate 0xfa cli\n"); into that >>> switch case but it wasn't printed before the guest crashed. >> >> It's possible that your guest crashes after emulating a different >> instruction. I've added a line to xen/arch/x86/hvm/emulate.c, in >> hvm_mem_access_emulate_one(): >> >> 1790 switch ( kind ) >> 1791 { >> 1792 case EMUL_KIND_NOWRITE: >> 1793 rc = hvm_emulate_one_no_write(&ctx); >> 1794 break; >> 1795 case EMUL_KIND_SET_CONTEXT: >> 1796 ctx.set_context = 1; >> 1797 /* Intentional fall-through. */ >> 1798 default: >> 1799 rc = hvm_emulate_one(&ctx); >> 1800 hvm_dump_emulation_state(XENLOG_G_DEBUG, &ctx); >> 1801 } >> >> so I can then see which instruction was the last before the stack trace >> with xl dmesg (or looking at the log file, etc.) >> >> It's possible the problem is not specific to CLI, or maybe it's even >> something that happens prior to emulating the last instruction that >> leads to a corruption in the guest's state later on. >> > > Just a quick update on this issue, I also now see the following > emulation error when I use xen-access with the emulation response for > the exec case. The issue previously reported with the failed vm entry > seem to happen only when emulating in response to the write-access. > > (XEN) Mem event emulation failed: d5v1 32bit @ 0008:826c602c -> 0f 21 > c0 89 82 dc 02 00 00 0f 21 c8 89 82 e0 02 > (XEN) Mem event emulation failed: d5v0 32bit @ 0008:82678caa -> cf 90 > f7 45 70 00 00 02 00 75 09 f7 45 6c 01 00 > (XEN) d5v1 Triple fault - invoking HVM shutdown action > (XEN) *** Dumping Dom5 vcpu#1 state: *** > (XEN) ----[ Xen-4.7.0 x86_64 debug=n Not tainted ]---- > (XEN) CPU: 6 > (XEN) RIP: 0008:[<00000000827194f8>] > (XEN) RFLAGS: 0000000000014292 CONTEXT: hvm guest (d5v1) > (XEN) rax: 00000000badb10cc rbx: 00000000807c1534 rcx: 0000000000000000 > (XEN) rdx: 0000000000000000 rsi: 000000008271957f rdi: 00000000badb10cc > (XEN) rbp: 00000000807c10e4 rsp: 00000000807c0d30 r8: 0000000000000000 > (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 > (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 > (XEN) r15: 0000000000000000 cr0: 000000008001003b cr4: 00000000000406f9 > (XEN) cr3: 0000000000185000 cr2: 00000000807c0d2c > (XEN) ds: 0023 es: 0023 fs: 0030 gs: 0000 ss: 0010 cs: 0008 I see that there were two failed emulation attempts previously (most likely there's no support in the emulator for the listed instructions). Does the issue still occur even when there are no failed emulation attempts? You might try to check and see if the emulation has failed, and if so to try to run the pending instruction under the MTF and see if this still occurs. Or, just lift the page restrictions and let the instruction run if emulation fails. Or, of course, add support for emulating those instructions. Cheers, Razvan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-09-07 5:59 ` Razvan Cojocaru @ 2016-09-07 9:36 ` Jan Beulich 0 siblings, 0 replies; 15+ messages in thread From: Jan Beulich @ 2016-09-07 9:36 UTC (permalink / raw) To: Razvan Cojocaru, Tamas K Lengyel Cc: Andrew Cooper, Xen-devel, Tamas K Lengyel >>> On 07.09.16 at 07:59, <rcojocaru@bitdefender.com> wrote: > On 09/07/16 02:31, Tamas K Lengyel wrote: >> Just a quick update on this issue, I also now see the following >> emulation error when I use xen-access with the emulation response for >> the exec case. The issue previously reported with the failed vm entry >> seem to happen only when emulating in response to the write-access. >> >> (XEN) Mem event emulation failed: d5v1 32bit @ 0008:826c602c -> 0f 21 >> c0 89 82 dc 02 00 00 0f 21 c8 89 82 e0 02 >> (XEN) Mem event emulation failed: d5v0 32bit @ 0008:82678caa -> cf 90 >> f7 45 70 00 00 02 00 75 09 f7 45 6c 01 00 >> (XEN) d5v1 Triple fault - invoking HVM shutdown action >> (XEN) *** Dumping Dom5 vcpu#1 state: *** >> (XEN) ----[ Xen-4.7.0 x86_64 debug=n Not tainted ]---- >> (XEN) CPU: 6 >> (XEN) RIP: 0008:[<00000000827194f8>] >> (XEN) RFLAGS: 0000000000014292 CONTEXT: hvm guest (d5v1) >> (XEN) rax: 00000000badb10cc rbx: 00000000807c1534 rcx: 0000000000000000 >> (XEN) rdx: 0000000000000000 rsi: 000000008271957f rdi: 00000000badb10cc >> (XEN) rbp: 00000000807c10e4 rsp: 00000000807c0d30 r8: 0000000000000000 >> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 >> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 >> (XEN) r15: 0000000000000000 cr0: 000000008001003b cr4: 00000000000406f9 >> (XEN) cr3: 0000000000185000 cr2: 00000000807c0d2c >> (XEN) ds: 0023 es: 0023 fs: 0030 gs: 0000 ss: 0010 cs: 0008 > > I see that there were two failed emulation attempts previously (most > likely there's no support in the emulator for the listed instructions). The first one would have support in the emulator (it's a DR0 access), if the read_dr() hook was set (which it looks like isn't the case, but should be easy to fix). The second one (IRET) currently gets handled by the emulator only when the guest is in real mode. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Failed vm entry with heavy use of emulator 2016-01-05 11:49 Failed vm entry with heavy use of emulator Tamas K Lengyel 2016-01-05 11:56 ` Andrew Cooper @ 2016-01-05 12:35 ` Razvan Cojocaru 1 sibling, 0 replies; 15+ messages in thread From: Razvan Cojocaru @ 2016-01-05 12:35 UTC (permalink / raw) To: Tamas K Lengyel, Xen-devel On 01/05/2016 01:49 PM, Tamas K Lengyel wrote: > Hi all, > I've been stress-testing the built-in emulator using the vm_event > response VM_EVENT_FLAG_EMULATE feature. In the test I've turned all > pages non-readable by default and all trapped instructions to be > emulated. My test code can be found at > https://github.com/tklengyel/xen/compare/read_emul?expand=1. > > The following crash is reproducible and has been verified by Razvan as well. Indeed, but I should point out that it only occurs when all the pages are not readable. For all other cases (restricting write or execute, that have been heavily tested here) there seem to be no issues, regardless of how hard we're driving the emulator. Tamas found this strange (and I agree) since we've assumed that the emulator doesn't care about EPT restrictions. But our use cases so far have never blocked page reads. Cheers, Razvan ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2016-09-07 9:36 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-01-05 11:49 Failed vm entry with heavy use of emulator Tamas K Lengyel 2016-01-05 11:56 ` Andrew Cooper 2016-01-05 12:05 ` Tamas K Lengyel 2016-01-05 13:45 ` Andrew Cooper 2016-01-06 14:21 ` Jan Beulich 2016-01-05 13:39 ` Razvan Cojocaru 2016-01-05 13:49 ` Andrew Cooper 2016-01-05 14:01 ` Razvan Cojocaru 2016-01-05 14:12 ` Andrew Cooper 2016-01-05 14:16 ` Tamas K Lengyel 2016-01-05 14:37 ` Razvan Cojocaru 2016-09-06 23:31 ` Tamas K Lengyel 2016-09-07 5:59 ` Razvan Cojocaru 2016-09-07 9:36 ` Jan Beulich 2016-01-05 12:35 ` Razvan Cojocaru
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.