On Wed, Feb 03, 2016 at 10:07:27AM -0500, Konrad Rzeszutek Wilk wrote: > On Wed, Feb 03, 2016 at 02:34:47AM -0700, Jan Beulich wrote: > > >>> On 02.02.16 at 23:05, wrote: > > > This is getting more and more bizzare. > > > > > > I realized that this machine has VMCS shadowing so Xen does not trap on > > > any vmwrite or vmread. Unless I update the VMCS shadowing bitmap - which > > > I did for vmwrite and vmread to get a better view of this. It never > > > traps on VIRTUAL_APIC_PAGE_ADDR accesses. It does trap on: > > > VIRTUAL_PROCESSOR_ID, > > > VM_EXIT_MSR_LOAD_ADDR and GUEST_[ES,DS,FS,GS,TR]_SELECTORS. > > > > > > (It may also trap on IO_BITMAP_A,B but I didn't print that out). > > > > > > To confirm that the VMCS that will be given to the L2 guest is correct > > > I added some printking of some states that ought to be pretty OK such > > > as HOST_RIP or HOST_RSP - which are all 0! > > > > But did you also check what the field of interest starts out as? > > I will do that. Attached is the patch against staging (I had used 4.6 before as the only change between those two was the dynamic mapping/unmapping of the vmread/vmwrite bitmap). (d1) (d1) drive 0x000f6270: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=524288000 (d1) (d1) Space available for UMB: cb800-ed000, f5d30-f6270 (d1) Returned 258048 bytes of ZoneHigh (d1) e820 map has 7 items: (d1) 0: 0000000000000000 - 000000000009fc00 = 1 RAM (d1) 1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED (d1) 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED (d1) 3: 0000000000100000 - 00000000effff000 = 1 RAM (d1) 4: 00000000effff000 - 00000000f0000000 = 2 RESERVED (d1) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED (d1) 6: 0000000100000000 - 000000020f800000 = 1 RAM (d1) enter handle_19: (d1) NULL (d1) Booting from Hard Disk... (d1) Booting from 0000:7c00 (XEN) stdvga.c:178:d1v0 leaving stdvga mode (XEN) stdvga.c:173:d1v0 entering stdvga mode (XEN) nvmx_handle_vmwrite 1: IO_BITMAP_A(2000)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 0: IO_BITMAP_A(2000)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 1: IO_BITMAP_B(2002)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 2: IO_BITMAP_A(2000)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 1: VIRTUAL_APIC_PAGE_ADDR(2012)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 2: IO_BITMAP_B(2002)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 1: (2006)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 2: VIRTUAL_APIC_PAGE_ADDR(2012)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 1: VM_EXIT_MSR_LOAD_ADDR(2008)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 3: IO_BITMAP_A(2000)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 3: IO_BITMAP_B(2002)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 2: MSR_BITMAP(2004)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 1: MSR_BITMAP(2004)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 0: MSR_BITMAP(2004)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 3: (2006)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 3: VM_EXIT_MSR_LOAD_ADDR(2008)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 3: MSR_BITMAP(2004)[0=ffffffffffffffff] (XEN) nvmx_handle_vmwrite 1: VIRTUAL_PROCESSOR_ID(0)[0=9] (XEN) nvmx_handle_vmwrite 0: VIRTUAL_PROCESSOR_ID(0)[0=9] (XEN) nvmx_handle_vmwrite 1: MSR_BITMAP(2004)[ffffffffffffffff=1367ed000] (XEN) nvmx_handle_vmwrite 3: VIRTUAL_PROCESSOR_ID(0)[0=9] (XEN) nvmx_handle_vmwrite 0: MSR_BITMAP(2004)[ffffffffffffffff=1367ed000] (XEN) nvmx_handle_vmwrite 1: VM_EXIT_MSR_LOAD_ADDR(2008)[ffffffffffffffff=135639f40] (XEN) nvmx_handle_vmwrite 0: VM_EXIT_MSR_LOAD_ADDR(2008)[ffffffffffffffff=135666f40] (XEN) nvmx_handle_vmwrite 2: VIRTUAL_PROCESSOR_ID(0)[0=9] (XEN) nvmx_handle_vmwrite 3: MSR_BITMAP(2004)[ffffffffffffffff=1367ed000] (XEN) nvmx_handle_vmwrite 3: VM_EXIT_MSR_LOAD_ADDR(2008)[ffffffffffffffff=135693f40] (XEN) nvmx_handle_vmwrite 2: MSR_BITMAP(2004)[ffffffffffffffff=1367ed000] (XEN) nvmx_handle_vmwrite 2: VM_EXIT_MSR_LOAD_ADDR(2008)[ffffffffffffffff=135701f40] (XEN) nvmx_handle_vmwrite 3: VM_EXIT_MSR_LOAD_ADDR(2008)[135639f40=13763cf40] (XEN) nvmx_handle_vmwrite 1: VM_EXIT_MSR_LOAD_ADDR(2008)[135701f40=137a3cf40] (XEN) nvmx_handle_vmwrite 0: VM_EXIT_MSR_LOAD_ADDR(2008)[135693f40=13783cf40] (XEN) nvmx_handle_vmwrite 2: VM_EXIT_MSR_LOAD_ADDR(2008)[135666f40=137c3cf40] (XEN) nvmx_handle_vmwrite 3: (800)[0=0] (XEN) nvmx_handle_vmwrite 3: (804)[0=0] (XEN) nvmx_handle_vmwrite 3: (806)[0=0] (XEN) nvmx_handle_vmwrite 3: (80a)[0=0] (XEN) nvmx_handle_vmwrite 3: (80e)[0=0] (XEN) vvmx.c:2566:d1v3 Unknown nested vmexit reason 80000021. (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest state (4). (XEN) ************* VMCS Area ************** (XEN) *** Guest State *** (XEN) CR0: actual=0x0000000000000030, shadow=0x0000000000000000, gh_mask=ffffffffffffffff (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=ffffffffffffffff (XEN) CR3 = 0x0000000080c06000 (XEN) RSP = 0x0000000000000000 (0x0000000000000000) RIP = 0x0000000000000000 (0x0000000000000000) (XEN) RFLAGS=0x00000002 (0x00000002) DR7 = 0x0000000000000400 (XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000 (XEN) sel attr limit base (XEN) CS: 0000 00000 00000000 0000000000000000 (XEN) DS: 0000 00000 00000000 0000000000000000 (XEN) SS: 0000 00000 00000000 0000000000000000 (XEN) ES: 0000 00000 00000000 0000000000000000 (XEN) FS: 0000 00000 00000000 0000000000000000 (XEN) GS: 0000 00000 00000000 0000000000000000 (XEN) GDTR: 00000000 0000000000000000 (XEN) LDTR: 0000 00000 00000000 0000000000000000 (XEN) IDTR: 00000000 0000000000000000 (XEN) TR: 0000 00000 00000000 0000000000000000 (XEN) EFER = 0x0000000000000800 PAT = 0x0000000000000000 (XEN) PreemptionTimer = 0x00000000 SM Base = 0x00000000 (XEN) DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 (XEN) Interruptibility = 00000000 ActivityState = 00000000 (XEN) VIRTUAL_APIC_PAGE_ADDR = 0x0000000000000000 TPR threshold = 0x0000000000000000 (XEN) APIC_ACCESS_ADDR = 0x0000000000000000 (XEN) *** Host State *** (XEN) RIP = 0xffff82d0801f8f80 (vmx_asm_vmexit_handler) RSP = 0xffff834007897f90 (XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 (XEN) FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff83400789eb80 (XEN) GDTBase=ffff83400788f000 IDTBase=ffff83400789b000 (XEN) CR0=0000000080050033 CR3=00000040007a0000 CR4=00000000001526e0 (XEN) Sysenter RSP=ffff834007897fc0 CS:RIP=e008:ffff82d08023eb30 (XEN) EFER = 0x0000000000000000 PAT = 0x0000050100070406 (XEN) *** Control State *** (XEN) PinBased=0000003f CPUBased=b62065fa SecondaryExec=000054eb (XEN) EntryControls=000011fb ExitControls=001fefff (XEN) ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000 (XEN) VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000 (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000006 (XEN) reason=80000021 qualification=0000000000000004 (XEN) IDTVectoring: info=00000000 errcode=00000000 (XEN) TSC Offset = 0xffef355833aa1cd5 (XEN) TPR Threshold = 0x00 PostedIntrVec = 0x00 (XEN) EPT pointer = 0x00000040007d101e EPTP index = 0x0000 (XEN) PLE Gap=00000080 Window=00001000 (XEN) Virtual processor ID = 0x0050 VMfunc controls = 0000000000000000 (XEN) ************************************** (XEN) domain_crash called from vmx.c:2845 (XEN) Domain 1 (vcpu#3) crashed on cpu#54: (XEN) ----[ Xen-4.7-unstable x86_64 debug=y Tainted: C ]---- (XEN) CPU: 54 (XEN) RIP: 0000:[<0000000000000000>] (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (d1v3) (XEN) rax: 0000000000000000 rbx: 0000000000000000 rcx: 0000000000000000 (XEN) rdx: 00000000078bfbff rsi: 0000000000000000 rdi: 0000000000000000 (XEN) rbp: 0000000000000000 rsp: 0000000000000000 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000 (XEN) cr3: 0000000080c06000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: 0000 I am going to augment more of the tracing to get an idea of what is happening before this.