Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

["Zhang, Yang Z" <yang.z.zhang@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 3683 bytes --]

> -----Original Message-----
> From: Zhang, Yang Z
> Sent: Thursday, October 16, 2014 2:18 PM
> To: Jeroen Groenewegen van der Weyden; Tian, Kevin; Jan Beulich; Dong, Eddie;
> Nakajima, Jun
> Cc: xen-devel@lists.xen.org
> Subject: RE: [Xen-devel] Crash of guest with nested vmx with Unknown nested
> vmexit reason 80000021.
> 
> Jeroen Groenewegen van der Weyden wrote on 2014-10-10:
> > Hi all,
> >
> > Any input from my side necessary to keep this rolling?
> 
> Sorry for the later reply. Yes, this is a known issue to me but I didn't have time
> to cook a patch fix it. As Jan pointed out, the NMI handling logic is wrong in
> current nested logic. But it is not a trivial task to fix them. I will do it once I have
> the time or if you are interesting in it, a patch from you is welcome.

You can find the previous discussion here:
http://www.gossamer-threads.com/lists/xen/devel/322693?do=post_view_threaded

> 
> >
> > BR,
> > Jeroen.
> >
> > Tian, Kevin schreef op 7-10-2014 om 21:56:
> >> Add Yang as the nested vmx owner.
> >>
> >>> From: Jan Beulich [mailto:JBeulich@suse.com]
> >>> Sent: Tuesday, October 07, 2014 12:59 AM
> >>>
> >>>>>> On 30.09.14 at 17:55, <groen692@grosc.com> wrote:
> >>>> Recently I updated my openSuse box from 12.3 to 13.1. On this box I
> >>>> run xen with several guests. One of these guests is an appliance
> >>>> that has 4 kvm guests running.
> >>>> When I start this appliance with the nested vmx feature the
> >>>> appliance crashes either immediately or after a few minutes.
> >>>>
> >>>> This same guest was running without a problem on opensuse releases
> >>>> 11.4 until 12.3 [...] ==== outup xl demsg
> >>>> (XEN) vvmx.c:2459:d5 Unknown nested vmexit reason 80000021.
> >>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid
> >>>> guest state (0).
> >>>> (XEN) ************* VMCS Area ************** [...]
> >>> So the problem here is that
> >>>
> >>>> (XEN) Interruptibility=0008 ActivityState=0000
> >>> VMX_INTR_SHADOW_NMI is set while
> >>>
> >>>> (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb
> >>> PIN_BASED_VIRTUAL_NMIS is active and
> >>>
> >>>> (XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003
> >>>> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
> >>>> (XEN)         reason=80000021 qualification=00000000
> >>>> (XEN) IDTVectoring: info=80000202 errcode=00000000
> >>> an NMI is being injected. This case is explicitly mentioned in Vol
> >>> 3 section 31.7.1.2 (Resuming Guest Software after Handling an
> >>> Exception). Either there needs to be extra code in vvmx.c to clear
> >>> VMX_INTR_SHADOW_NMI (as the second sub-bullet point of the last
> >>> bullet point says), or the second half of vmx_idtv_reinject() needs
> >>> to be done without regard to nestedhvm_vcpu_in_guestmode(v) (and
> >>> maybe also without regard to EXIT_REASON_TASK_SWITCH).
> >>>
> >>> Speaking of SDM sections: There are quite a few references in the
> >>> code that name just section numbers (in the case here, several
> >>> references to sections 25.7.1.* exist). These numbers become stale
> >>> quite quickly (here they're now 31.7.1.*), so in order to help
> >>> digging through issues like the one here, can I please ask one of
> >>> you to go through and replace (or at least amend) these numbers with
> >>> the sections' titles (which I hope won't get altered that quickly)?
> >>>
> >>> Jan
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xen.org
> >> http://lists.xen.org/xen-devel
> >>
> 
> 
> Best regards,
> Yang
> 


[-- Attachment #2: Type: message/rfc822, Size: 6548 bytes --]

From: Steven Krikstone <skrikstone@gmail.com>
To: "Zhang, Yang Z" <yang.z.zhang@intel.com>
Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Subject: Re: [Xen-devel] [BUG] L2 Guest on hyper-v causes domain_crash
Date: Fri, 28 Mar 2014 22:20:47 +0000
Message-ID: <CAD6X+xVsySPm3So2qW0F-ew=dRcBDDTSu6U4p9imNNyFEwpOGQ@mail.gmail.com>


On Wed, Mar 26, 2014 at 3:22 AM, Zhang, Yang Z <yang.z.zhang@intel.com<mailto:yang.z.zhang@intel.com>> wrote:


   Steven Krikstone wrote on 2014-03-26:

   > Hello,
   >
   > Running a FreeBSD L2 guest in Hyper-V on Windows Server 2012 R2 as the
   > L1 causes the following guest domain crash.
   >
   > ### Setup: Intel server running Ubuntu 13.10 with Xen 4.4.1pre
   > (git:staging-4.4:babcef3 with 1 patch) The Ubuntu server is a stock
   > install with just the necessary packages installed to run Xen.
   > Windows Server 2012 R2 is a stock install with the Hyper-V role added.
   > The following patch from:
   > http://lists.xen.org/archives/html/xen-devel/2014-02/msg01071.html was
   > required or else Windows Server VM would hang at the boot splash screen.
   >
   > If I set "nestedhvm=0" Server 2012R2 runs without issue (minus a
   > working Hyper-v role).
   >
   > Let me know if you need further logs or traces provided.
   >
   > Thanks,
   >
   > -steve
   >
   > ### xl dmesg [..many L0 PIO lines snipped..] (XEN) L0 PIO 01f7 (XEN)
   > L0 PIO c220 (XEN) L0 PIO c222 (XEN) L0 PIO c222 (XEN) L0 PIO c220
   > (XEN) L0 PIO c222 (XEN) L0 PIO 01f7 (XEN) L0 PIO 01f7 (XEN) L0 PIO
   > c203 (XEN) L0 PIO c205 (XEN) L0 PIO c203 (XEN) L0 PIO c205 (XEN) L0
   > PIO c207 (XEN) L0 PIO c211 (XEN) L0 PIO c213 (XEN) L0 PIO c205 (XEN)
   > L0 PIO c207 (XEN)
   > vvmx.c:2477:d3 Unknown nested vmexit reason 80000021. (XEN) Failed vm
   > entry (exit reason 0x80000021) caused by invalid guest state (0).
   > (XEN)
   > ************* VMCS Area ************** (XEN) *** Guest State *** (XEN)
   > CR0: actual=0x0000000080010031, shadow=0x0000000080050031,
   > gh_mask=ffffffffffffffff (XEN) CR4: actual=0x00000000001526f8,
   > shadow=0x00000000001506f8, gh_mask=ffffffffffffffff (XEN) CR3:
   > actual=0x00000000001a7000, target_count=0 (XEN)
   > target0=0000000000000000, target1=0000000000000000 (XEN)
   > target2=0000000000000000, target3=0000000000000000 (XEN) RSP =
   > 0xffffd00020b3c890 (0xffffd00020b3c890)  RIP = 0xfffff80320f0c0ab
   > (0xfffff80320f0c0ab) (XEN) RFLAGS=0x0000000000000046
   > (0x0000000000000046)  DR7 = 0x0000000000000400 (XEN) Sysenter
   > RSP=0000000000000000 CS:RIP=0000:0000000000000000 (XEN) CS:
   > sel=0x0010, attr=0x0209b, limit=0x00000000, base=0x0000000000000000 (XEN) DS:
   > sel=0x002b, attr=0x0c0f3, limit=0xffffffff, base=0x0000000000000000
   > (XEN) SS: sel=0x0018, attr=0x0c093, limit=0xffffffff,
   > base=0x0000000000000000 (XEN) ES: sel=0x002b, attr=0x0c0f3,
   > limit=0xffffffff, base=0x0000000000000000 (XEN) FS: sel=0x0053,
   > attr=0x040f3, limit=0x0000fc00, base=0x00000000ff9ce000 (XEN) GS:
   > sel=0x002b, attr=0x0c0f3, limit=0xffffffff, base=0xffffd00020b12000
   > (XEN) GDTR:                           limit=0x0000007f,
   > base=0xffffd00020b1f040 (XEN) LDTR: sel=0x0000, attr=0x1c000,
   > limit=0xffffffff, base=0x0000000000000000 (XEN) IDTR:
   >        limit=0x00000fff, base=0xffffd00020b1f0c0 (XEN) TR: sel=0x0040,
   > attr=0x0008b, limit=0x00000067, base=0xffffd00020b18080 (XEN) Guest
   > PAT = 0x0000050100070406 (XEN) TSC Offset = fffffefc546de420 (XEN)
   > DebugCtl=0000000000000000 DebugExceptions=0000000000000000 (XEN)
   > Interruptibility=0008 ActivityState=0000


   Seems the virtual NMI handling for nested is not correct: blocking by NMI is set, but trying to inject a NMI to guest.

   Best regards,
   Yang





   I am more than willing to test any patches you may have.  I am also leaving this testbed setup in case you or others need further logs, debugs run, etc.


   Thanks,

   -steve


[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel