Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jeroen Groenewegen van der Weyden]

[-- Attachment #1.1: Type: text/plain, Size: 4318 bytes --]

Hi,

Recently I updated my openSuse box from 12.3 to 13.1. On this box I run 
xen with several guests. One of these guests is an appliance that has 4 
kvm guests running.
When I start this appliance with the nested vmx feature the appliance 
crashes either immediately or after a few minutes.

This same guest was running without a problem on opensuse releases 11.4 
until 12.3

( remark that about 3 year ago some patches were written by tim degan to make this appliance work )

dom0
opensuse 13.1 x86
kernel 3.11.10-21-xen
xen: 4.3..2_01-21.1

- domu (HVM)
- sles11sp2
- mem: 8 GB
- vcpu: 4
- kvm with 4 guest

any body any thought on this?

BR,
Jeroen


==== outup xl demsg
(XEN) vvmx.c:2459:d5 Unknown nested vmexit reason 80000021.
(XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest 
state (0).
(XEN) ************* VMCS Area **************
(XEN) *** Guest State ***
(XEN) CR0: actual=0x0000000080000033, shadow=0x0000000000000011, 
gh_mask=ffffffffffffffff
(XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, 
gh_mask=ffffffffffffffff
(XEN) CR3: actual=0x00000000feffc000, target_count=0
(XEN)      target0=0000000000000000, target1=0000000000000000
(XEN)      target2=0000000000000000, target3=0000000000000000
(XEN) RSP = 0x0000000000000398 (0x0000000000000398)  RIP = 
0x000000000000045f (0x000000000000045f)
(XEN) RFLAGS=0x0000000000000006 (0x0000000000000006)  DR7 = 
0x0000000000000400
(XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN) CS: sel=0x9dcc, attr=0x0009b, limit=0x00000ecb, 
base=0x0000000000689920
(XEN) DS: sel=0x9db4, attr=0x00093, limit=0x000038b9, 
base=0x00000000006778b0
(XEN) SS: sel=0x9e7c, attr=0x00093, limit=0x00000401, 
base=0x00000000006ba960
(XEN) ES: sel=0x3ec8, attr=0x00093, limit=0x00000099, 
base=0x000000000073c608
(XEN) FS: sel=0x0000, attr=0x1c000, limit=0xffffffff, 
base=0x0000000000000000
(XEN) GS: sel=0x0000, attr=0x1c000, limit=0xffffffff, 
base=0x0000000000000000
(XEN) GDTR:                           limit=0x0000ffff, 
base=0x0000000001000000
(XEN) LDTR: sel=0x0820, attr=0x00082, limit=0x0000ffff, 
base=0x0000000001010400
(XEN) IDTR:                           limit=0x000003ff, 
base=0x0000000001010000
(XEN) TR: sel=0x0690, attr=0x00083, limit=0x0000002b, 
base=0x0000000001213a30
(XEN) Guest PAT = 0x0000050100070406
(XEN) TSC Offset = fffd869460e8dc28
(XEN) DebugCtl=0000000000000000 DebugExceptions=0000000000000000
(XEN) Interruptibility=0008 ActivityState=0000
(XEN) *** Host State ***
(XEN) RSP = 0xffff83083ff27f90  RIP = 0xffff82c4c01d9330
(XEN) CS=e008 DS=0000 ES=0000 FS=0000 GS=0000 SS=0000 TR=e040
(XEN) FSBase=0000000000000000 GSBase=0000000000000000 
TRBase=ffff83083ff2dc80
(XEN) GDTBase=ffff8318035ee000 IDTBase=ffff8318035fa000
(XEN) CR0=0000000080050033 CR3=0000000eefcc0000 CR4=00000000000026f0
(XEN) Sysenter RSP=ffff83083ff27fc0 CS:RIP=e008:ffff82c4c0216c60
(XEN) Host PAT = 0x0000050100070406
(XEN) *** Control State ***
(XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb
(XEN) EntryControls=000011ff ExitControls=000fefff
(XEN) ExceptionBitmap=00040042
(XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003
(XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
(XEN)         reason=80000021 qualification=00000000
(XEN) IDTVectoring: info=80000202 errcode=00000000
(XEN) TPR Threshold = 0x00
(XEN) EPT pointer = 0x0000000ef022501e
(XEN) Virtual processor ID = 0xc0d0
(XEN) **************************************
(XEN) domain_crash called from vmx.c:2342
(XEN) Domain 5 (vcpu#3) crashed on cpu#20:
(XEN) ----[ Xen-4.3.2_01-21.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    20
(XEN) RIP:    9dcc:[<000000000000045f>]
(XEN) RFLAGS: 0000000000000006   CONTEXT: hvm guest
(XEN) rax: 0000000000000001   rbx: 00000000000026db   rcx: 000000009dd40003
(XEN) rdx: 00000000248816a0   rsi: 0000000000000000   rdi: 0000000026b010e4
(XEN) rbp: 00000000000003ae   rsp: 0000000000000398   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080000033   cr4: 0000000000002010
(XEN) cr3: 00000000feffc000   cr2: 0000000000000000
(XEN) ds: 9db4   es: 3ec8   fs: 0000   gs: 0000   ss: 9e7c   cs: 9dcc


[-- Attachment #1.2: Type: text/html, Size: 5620 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jan Beulich]

>>> On 30.09.14 at 17:55, <groen692@grosc.com> wrote:
> Recently I updated my openSuse box from 12.3 to 13.1. On this box I run 
> xen with several guests. One of these guests is an appliance that has 4 
> kvm guests running.
> When I start this appliance with the nested vmx feature the appliance 
> crashes either immediately or after a few minutes.

I see you also sent this to opensuse-virtual, which is where I think
it belongs at least initially. I'll take it up from there, but it may take
some time until I can get to it.

Jan

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jan Beulich]

>>> On 30.09.14 at 17:55, <groen692@grosc.com> wrote:
> Recently I updated my openSuse box from 12.3 to 13.1. On this box I run 
> xen with several guests. One of these guests is an appliance that has 4 
> kvm guests running.
> When I start this appliance with the nested vmx feature the appliance 
> crashes either immediately or after a few minutes.
> 
> This same guest was running without a problem on opensuse releases 11.4 
> until 12.3
>[...]
> ==== outup xl demsg
> (XEN) vvmx.c:2459:d5 Unknown nested vmexit reason 80000021.
> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest 
> state (0).
> (XEN) ************* VMCS Area **************
>[...]

So the problem here is that

> (XEN) Interruptibility=0008 ActivityState=0000

VMX_INTR_SHADOW_NMI is set while

> (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb

PIN_BASED_VIRTUAL_NMIS is active and

> (XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003
> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
> (XEN)         reason=80000021 qualification=00000000
> (XEN) IDTVectoring: info=80000202 errcode=00000000

an NMI is being injected. This case is explicitly mentioned in Vol 3
section 31.7.1.2 (Resuming Guest Software after Handling an
Exception). Either there needs to be extra code in vvmx.c to clear
VMX_INTR_SHADOW_NMI (as the second sub-bullet point of the last
bullet point says), or the second half of vmx_idtv_reinject() needs
to be done without regard to nestedhvm_vcpu_in_guestmode(v)
(and maybe also without regard to EXIT_REASON_TASK_SWITCH).

Speaking of SDM sections: There are quite a few references in the
code that name just section numbers (in the case here, several
references to sections 25.7.1.* exist). These numbers become stale
quite quickly (here they're now 31.7.1.*), so in order to help
digging through issues like the one here, can I please ask one of
you to go through and replace (or at least amend) these numbers
with the sections' titles (which I hope won't get altered that quickly)?

Jan

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jeroen Groenewegen van der Weyden]

Hi Jan,

I installed xen-4.5.29823-1.xen_unstable.1.x86_64
from the 
http://download.opensuse.org/repositories/home:/olh:/xen-unstable/openSUSE_Factory/ 


After about 45 minutes the guests crashed again.

mfg,
Jeroen


(XEN) vvmx.c:2476:d1v0 Unknown nested vmexit reason 80000021.
(XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest 
state (0).
(XEN) ************* VMCS Area **************
(XEN) *** Guest State ***
(XEN) CR0: actual=0x0000000000000033, shadow=0x0000000000000011, 
gh_mask=ffffffffffffffff
(XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, 
gh_mask=ffffffffffffffff
(XEN) CR3: actual=0x00000000feffc000, target_count=0
(XEN)      target0=0000000000000000, target1=0000000000000000
(XEN)      target2=0000000000000000, target3=0000000000000000
(XEN) RSP = 0x00000000000006be (0x00000000000006be)  RIP = 
0x000000000000017d (0x000000000000017d)
(XEN) RFLAGS=0x0000000000000002 (0x0000000000000002)  DR7 = 
0x0000000000000400
(XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN) CS: sel=0xbf70, attr=0x0009b, limit=0x0000075a, 
base=0x000000000194fec0
(XEN) DS: sel=0x3cec, attr=0x00093, limit=0x00000110, 
base=0x000000000bcbe276
(XEN) SS: sel=0x3cf4, attr=0x00093, limit=0x000007ff, 
base=0x000000000bcbe3aa
(XEN) ES: sel=0x3cfc, attr=0x00093, limit=0x00000099, 
base=0x000000000bcbebac
(XEN) FS: sel=0x0000, attr=0x1c000, limit=0xffffffff, 
base=0x0000000000000000
(XEN) GS: sel=0x0000, attr=0x1c000, limit=0xffffffff, 
base=0x0000000000000000
(XEN) GDTR:                           limit=0x0000ffff, 
base=0x0000000001000000
(XEN) LDTR: sel=0x0820, attr=0x00082, limit=0x0000ffff, 
base=0x0000000001010400
(XEN) IDTR:                           limit=0x000003ff, 
base=0x0000000001010000
(XEN) TR: sel=0x0690, attr=0x00083, limit=0x0000002b, 
base=0x0000000001213a30
(XEN) Guest PAT = 0x0000050100070406
(XEN) TSC Offset = fffffe8a52350fd4
(XEN) DebugCtl=0000000000000000 DebugExceptions=0000000000000000
(XEN) Interruptibility=0008 ActivityState=0000
(XEN) *** Host State ***
(XEN) RSP = 0xffff830836b5ff90  RIP = 0xffff82d0801e3ed0
(XEN) CS=e008 DS=0000 ES=0000 FS=0000 GS=0000 SS=0000 TR=e040
(XEN) FSBase=0000000000000000 GSBase=0000000000000000 
TRBase=ffff830836b64c80
(XEN) GDTBase=ffff830836b55000 IDTBase=ffff830836b61000
(XEN) CR0=0000000080050033 CR3=000000082ffdd000 CR4=00000000000026f0
(XEN) Sysenter RSP=ffff830836b5ffc0 CS:RIP=e008:ffff82d0802220a0
(XEN) Host PAT = 0x0000050100070406
(XEN) *** Control State ***
(XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb
(XEN) EntryControls=000011ff ExitControls=000fefff
(XEN) ExceptionBitmap=00040042
(XEN) VMEntry: intr_info=80000202 errcode=00000000 ilen=00000003
(XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
(XEN)         reason=80000021 qualification=00000000
(XEN) IDTVectoring: info=80000202 errcode=00000000
(XEN) TPR Threshold = 0x00
(XEN) EPT pointer = 0x000000082ffff01e
(XEN) Virtual processor ID = 0xe6a3
(XEN) **************************************
(XEN) domain_crash called from vmx.c:2483
(XEN) Domain 1 (vcpu#0) crashed on cpu#9:
(XEN) ----[ Xen-4.5.29823-1.xen_unstable.1  x86_64  debug=n  Not tainted 
]----
(XEN) CPU:    9
(XEN) RIP:    bf70:[<000000000000017d>]
(XEN) RFLAGS: 0000000000000002   CONTEXT: hvm guest
(XEN) rax: 0000000000040002   rbx: 0000000000042701   rcx: 00000000bf480003
(XEN) rdx: 00000000248816a0   rsi: 0000000000000000   rdi: 0000000026b00096
(XEN) rbp: 00000000000006d4   rsp: 00000000000006be   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000000000033   cr4: 0000000000002010
(XEN) cr3: 00000000feffc000   cr2: 0000000000000000
(XEN) ds: 3cec   es: 3cfc   fs: 0000   gs: 0000   ss: 3cf4   cs: bf70

Jan Beulich schreef op 7-10-2014 om 09:58:
>>>> On 30.09.14 at 17:55, <groen692@grosc.com> wrote:
>> Recently I updated my openSuse box from 12.3 to 13.1. On this box I run
>> xen with several guests. One of these guests is an appliance that has 4
>> kvm guests running.
>> When I start this appliance with the nested vmx feature the appliance
>> crashes either immediately or after a few minutes.
>>
>> This same guest was running without a problem on opensuse releases 11.4
>> until 12.3
>> [...]
>> ==== outup xl demsg
>> (XEN) vvmx.c:2459:d5 Unknown nested vmexit reason 80000021.
>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>> state (0).
>> (XEN) ************* VMCS Area **************
>> [...]
> So the problem here is that
>
>> (XEN) Interruptibility=0008 ActivityState=0000
> VMX_INTR_SHADOW_NMI is set while
>
>> (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb
> PIN_BASED_VIRTUAL_NMIS is active and
>
>> (XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003
>> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
>> (XEN)         reason=80000021 qualification=00000000
>> (XEN) IDTVectoring: info=80000202 errcode=00000000
> an NMI is being injected. This case is explicitly mentioned in Vol 3
> section 31.7.1.2 (Resuming Guest Software after Handling an
> Exception). Either there needs to be extra code in vvmx.c to clear
> VMX_INTR_SHADOW_NMI (as the second sub-bullet point of the last
> bullet point says), or the second half of vmx_idtv_reinject() needs
> to be done without regard to nestedhvm_vcpu_in_guestmode(v)
> (and maybe also without regard to EXIT_REASON_TASK_SWITCH).
>
> Speaking of SDM sections: There are quite a few references in the
> code that name just section numbers (in the case here, several
> references to sections 25.7.1.* exist). These numbers become stale
> quite quickly (here they're now 31.7.1.*), so in order to help
> digging through issues like the one here, can I please ask one of
> you to go through and replace (or at least amend) these numbers
> with the sections' titles (which I hope won't get altered that quickly)?
>
> Jan
>
>

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Tian, Kevin]

Add Yang as the nested vmx owner.

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Tuesday, October 07, 2014 12:59 AM
> 
> >>> On 30.09.14 at 17:55, <groen692@grosc.com> wrote:
> > Recently I updated my openSuse box from 12.3 to 13.1. On this box I run
> > xen with several guests. One of these guests is an appliance that has 4
> > kvm guests running.
> > When I start this appliance with the nested vmx feature the appliance
> > crashes either immediately or after a few minutes.
> >
> > This same guest was running without a problem on opensuse releases 11.4
> > until 12.3
> >[...]
> > ==== outup xl demsg
> > (XEN) vvmx.c:2459:d5 Unknown nested vmexit reason 80000021.
> > (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
> > state (0).
> > (XEN) ************* VMCS Area **************
> >[...]
> 
> So the problem here is that
> 
> > (XEN) Interruptibility=0008 ActivityState=0000
> 
> VMX_INTR_SHADOW_NMI is set while
> 
> > (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb
> 
> PIN_BASED_VIRTUAL_NMIS is active and
> 
> > (XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003
> > (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
> > (XEN)         reason=80000021 qualification=00000000
> > (XEN) IDTVectoring: info=80000202 errcode=00000000
> 
> an NMI is being injected. This case is explicitly mentioned in Vol 3
> section 31.7.1.2 (Resuming Guest Software after Handling an
> Exception). Either there needs to be extra code in vvmx.c to clear
> VMX_INTR_SHADOW_NMI (as the second sub-bullet point of the last
> bullet point says), or the second half of vmx_idtv_reinject() needs
> to be done without regard to nestedhvm_vcpu_in_guestmode(v)
> (and maybe also without regard to EXIT_REASON_TASK_SWITCH).
> 
> Speaking of SDM sections: There are quite a few references in the
> code that name just section numbers (in the case here, several
> references to sections 25.7.1.* exist). These numbers become stale
> quite quickly (here they're now 31.7.1.*), so in order to help
> digging through issues like the one here, can I please ask one of
> you to go through and replace (or at least amend) these numbers
> with the sections' titles (which I hope won't get altered that quickly)?
> 
> Jan

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jeroen Groenewegen van der Weyden]

Hi all,

Any input from my side necessary to keep this rolling?

BR,
Jeroen.

Tian, Kevin schreef op 7-10-2014 om 21:56:
> Add Yang as the nested vmx owner.
>
>> From: Jan Beulich [mailto:JBeulich@suse.com]
>> Sent: Tuesday, October 07, 2014 12:59 AM
>>
>>>>> On 30.09.14 at 17:55, <groen692@grosc.com> wrote:
>>> Recently I updated my openSuse box from 12.3 to 13.1. On this box I run
>>> xen with several guests. One of these guests is an appliance that has 4
>>> kvm guests running.
>>> When I start this appliance with the nested vmx feature the appliance
>>> crashes either immediately or after a few minutes.
>>>
>>> This same guest was running without a problem on opensuse releases 11.4
>>> until 12.3
>>> [...]
>>> ==== outup xl demsg
>>> (XEN) vvmx.c:2459:d5 Unknown nested vmexit reason 80000021.
>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest
>>> state (0).
>>> (XEN) ************* VMCS Area **************
>>> [...]
>> So the problem here is that
>>
>>> (XEN) Interruptibility=0008 ActivityState=0000
>> VMX_INTR_SHADOW_NMI is set while
>>
>>> (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb
>> PIN_BASED_VIRTUAL_NMIS is active and
>>
>>> (XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003
>>> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
>>> (XEN)         reason=80000021 qualification=00000000
>>> (XEN) IDTVectoring: info=80000202 errcode=00000000
>> an NMI is being injected. This case is explicitly mentioned in Vol 3
>> section 31.7.1.2 (Resuming Guest Software after Handling an
>> Exception). Either there needs to be extra code in vvmx.c to clear
>> VMX_INTR_SHADOW_NMI (as the second sub-bullet point of the last
>> bullet point says), or the second half of vmx_idtv_reinject() needs
>> to be done without regard to nestedhvm_vcpu_in_guestmode(v)
>> (and maybe also without regard to EXIT_REASON_TASK_SWITCH).
>>
>> Speaking of SDM sections: There are quite a few references in the
>> code that name just section numbers (in the case here, several
>> references to sections 25.7.1.* exist). These numbers become stale
>> quite quickly (here they're now 31.7.1.*), so in order to help
>> digging through issues like the one here, can I please ask one of
>> you to go through and replace (or at least amend) these numbers
>> with the sections' titles (which I hope won't get altered that quickly)?
>>
>> Jan
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Zhang, Yang Z]

Jeroen Groenewegen van der Weyden wrote on 2014-10-10:
> Hi all,
> 
> Any input from my side necessary to keep this rolling?

Sorry for the later reply. Yes, this is a known issue to me but I didn't have time to cook a patch fix it. As Jan pointed out, the NMI handling logic is wrong in current nested logic. But it is not a trivial task to fix them. I will do it once I have the time or if you are interesting in it, a patch from you is welcome.

> 
> BR,
> Jeroen.
> 
> Tian, Kevin schreef op 7-10-2014 om 21:56:
>> Add Yang as the nested vmx owner.
>> 
>>> From: Jan Beulich [mailto:JBeulich@suse.com]
>>> Sent: Tuesday, October 07, 2014 12:59 AM
>>> 
>>>>>> On 30.09.14 at 17:55, <groen692@grosc.com> wrote:
>>>> Recently I updated my openSuse box from 12.3 to 13.1. On this box
>>>> I run xen with several guests. One of these guests is an appliance
>>>> that has 4 kvm guests running.
>>>> When I start this appliance with the nested vmx feature the
>>>> appliance crashes either immediately or after a few minutes.
>>>> 
>>>> This same guest was running without a problem on opensuse releases
>>>> 11.4 until 12.3 [...] ==== outup xl demsg
>>>> (XEN) vvmx.c:2459:d5 Unknown nested vmexit reason 80000021.
>>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid
>>>> guest state (0).
>>>> (XEN) ************* VMCS Area ************** [...]
>>> So the problem here is that
>>> 
>>>> (XEN) Interruptibility=0008 ActivityState=0000
>>> VMX_INTR_SHADOW_NMI is set while
>>> 
>>>> (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb
>>> PIN_BASED_VIRTUAL_NMIS is active and
>>> 
>>>> (XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003
>>>> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
>>>> (XEN)         reason=80000021 qualification=00000000
>>>> (XEN) IDTVectoring: info=80000202 errcode=00000000
>>> an NMI is being injected. This case is explicitly mentioned in Vol
>>> 3 section 31.7.1.2 (Resuming Guest Software after Handling an
>>> Exception). Either there needs to be extra code in vvmx.c to clear
>>> VMX_INTR_SHADOW_NMI (as the second sub-bullet point of the last
>>> bullet point says), or the second half of vmx_idtv_reinject() needs
>>> to be done without regard to nestedhvm_vcpu_in_guestmode(v) (and
>>> maybe also without regard to EXIT_REASON_TASK_SWITCH).
>>> 
>>> Speaking of SDM sections: There are quite a few references in the
>>> code that name just section numbers (in the case here, several
>>> references to sections 25.7.1.* exist). These numbers become stale
>>> quite quickly (here they're now 31.7.1.*), so in order to help
>>> digging through issues like the one here, can I please ask one of
>>> you to go through and replace (or at least amend) these numbers
>>> with the sections' titles (which I hope won't get altered that quickly)?
>>> 
>>> Jan
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>>


Best regards,
Yang

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Zhang, Yang Z]

[-- Attachment #1: Type: text/plain, Size: 3683 bytes --]

> -----Original Message-----
> From: Zhang, Yang Z
> Sent: Thursday, October 16, 2014 2:18 PM
> To: Jeroen Groenewegen van der Weyden; Tian, Kevin; Jan Beulich; Dong, Eddie;
> Nakajima, Jun
> Cc: xen-devel@lists.xen.org
> Subject: RE: [Xen-devel] Crash of guest with nested vmx with Unknown nested
> vmexit reason 80000021.
> 
> Jeroen Groenewegen van der Weyden wrote on 2014-10-10:
> > Hi all,
> >
> > Any input from my side necessary to keep this rolling?
> 
> Sorry for the later reply. Yes, this is a known issue to me but I didn't have time
> to cook a patch fix it. As Jan pointed out, the NMI handling logic is wrong in
> current nested logic. But it is not a trivial task to fix them. I will do it once I have
> the time or if you are interesting in it, a patch from you is welcome.

You can find the previous discussion here:
http://www.gossamer-threads.com/lists/xen/devel/322693?do=post_view_threaded

> 
> >
> > BR,
> > Jeroen.
> >
> > Tian, Kevin schreef op 7-10-2014 om 21:56:
> >> Add Yang as the nested vmx owner.
> >>
> >>> From: Jan Beulich [mailto:JBeulich@suse.com]
> >>> Sent: Tuesday, October 07, 2014 12:59 AM
> >>>
> >>>>>> On 30.09.14 at 17:55, <groen692@grosc.com> wrote:
> >>>> Recently I updated my openSuse box from 12.3 to 13.1. On this box I
> >>>> run xen with several guests. One of these guests is an appliance
> >>>> that has 4 kvm guests running.
> >>>> When I start this appliance with the nested vmx feature the
> >>>> appliance crashes either immediately or after a few minutes.
> >>>>
> >>>> This same guest was running without a problem on opensuse releases
> >>>> 11.4 until 12.3 [...] ==== outup xl demsg
> >>>> (XEN) vvmx.c:2459:d5 Unknown nested vmexit reason 80000021.
> >>>> (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid
> >>>> guest state (0).
> >>>> (XEN) ************* VMCS Area ************** [...]
> >>> So the problem here is that
> >>>
> >>>> (XEN) Interruptibility=0008 ActivityState=0000
> >>> VMX_INTR_SHADOW_NMI is set while
> >>>
> >>>> (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb
> >>> PIN_BASED_VIRTUAL_NMIS is active and
> >>>
> >>>> (XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003
> >>>> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
> >>>> (XEN)         reason=80000021 qualification=00000000
> >>>> (XEN) IDTVectoring: info=80000202 errcode=00000000
> >>> an NMI is being injected. This case is explicitly mentioned in Vol
> >>> 3 section 31.7.1.2 (Resuming Guest Software after Handling an
> >>> Exception). Either there needs to be extra code in vvmx.c to clear
> >>> VMX_INTR_SHADOW_NMI (as the second sub-bullet point of the last
> >>> bullet point says), or the second half of vmx_idtv_reinject() needs
> >>> to be done without regard to nestedhvm_vcpu_in_guestmode(v) (and
> >>> maybe also without regard to EXIT_REASON_TASK_SWITCH).
> >>>
> >>> Speaking of SDM sections: There are quite a few references in the
> >>> code that name just section numbers (in the case here, several
> >>> references to sections 25.7.1.* exist). These numbers become stale
> >>> quite quickly (here they're now 31.7.1.*), so in order to help
> >>> digging through issues like the one here, can I please ask one of
> >>> you to go through and replace (or at least amend) these numbers with
> >>> the sections' titles (which I hope won't get altered that quickly)?
> >>>
> >>> Jan
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xen.org
> >> http://lists.xen.org/xen-devel
> >>
> 
> 
> Best regards,
> Yang
> 


[-- Attachment #2: Type: message/rfc822, Size: 6548 bytes --]

From: Steven Krikstone <skrikstone@gmail.com>
To: "Zhang, Yang Z" <yang.z.zhang@intel.com>
Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Subject: Re: [Xen-devel] [BUG] L2 Guest on hyper-v causes domain_crash
Date: Fri, 28 Mar 2014 22:20:47 +0000
Message-ID: <CAD6X+xVsySPm3So2qW0F-ew=dRcBDDTSu6U4p9imNNyFEwpOGQ@mail.gmail.com>


On Wed, Mar 26, 2014 at 3:22 AM, Zhang, Yang Z <yang.z.zhang@intel.com<mailto:yang.z.zhang@intel.com>> wrote:


   Steven Krikstone wrote on 2014-03-26:

   > Hello,
   >
   > Running a FreeBSD L2 guest in Hyper-V on Windows Server 2012 R2 as the
   > L1 causes the following guest domain crash.
   >
   > ### Setup: Intel server running Ubuntu 13.10 with Xen 4.4.1pre
   > (git:staging-4.4:babcef3 with 1 patch) The Ubuntu server is a stock
   > install with just the necessary packages installed to run Xen.
   > Windows Server 2012 R2 is a stock install with the Hyper-V role added.
   > The following patch from:
   > http://lists.xen.org/archives/html/xen-devel/2014-02/msg01071.html was
   > required or else Windows Server VM would hang at the boot splash screen.
   >
   > If I set "nestedhvm=0" Server 2012R2 runs without issue (minus a
   > working Hyper-v role).
   >
   > Let me know if you need further logs or traces provided.
   >
   > Thanks,
   >
   > -steve
   >
   > ### xl dmesg [..many L0 PIO lines snipped..] (XEN) L0 PIO 01f7 (XEN)
   > L0 PIO c220 (XEN) L0 PIO c222 (XEN) L0 PIO c222 (XEN) L0 PIO c220
   > (XEN) L0 PIO c222 (XEN) L0 PIO 01f7 (XEN) L0 PIO 01f7 (XEN) L0 PIO
   > c203 (XEN) L0 PIO c205 (XEN) L0 PIO c203 (XEN) L0 PIO c205 (XEN) L0
   > PIO c207 (XEN) L0 PIO c211 (XEN) L0 PIO c213 (XEN) L0 PIO c205 (XEN)
   > L0 PIO c207 (XEN)
   > vvmx.c:2477:d3 Unknown nested vmexit reason 80000021. (XEN) Failed vm
   > entry (exit reason 0x80000021) caused by invalid guest state (0).
   > (XEN)
   > ************* VMCS Area ************** (XEN) *** Guest State *** (XEN)
   > CR0: actual=0x0000000080010031, shadow=0x0000000080050031,
   > gh_mask=ffffffffffffffff (XEN) CR4: actual=0x00000000001526f8,
   > shadow=0x00000000001506f8, gh_mask=ffffffffffffffff (XEN) CR3:
   > actual=0x00000000001a7000, target_count=0 (XEN)
   > target0=0000000000000000, target1=0000000000000000 (XEN)
   > target2=0000000000000000, target3=0000000000000000 (XEN) RSP =
   > 0xffffd00020b3c890 (0xffffd00020b3c890)  RIP = 0xfffff80320f0c0ab
   > (0xfffff80320f0c0ab) (XEN) RFLAGS=0x0000000000000046
   > (0x0000000000000046)  DR7 = 0x0000000000000400 (XEN) Sysenter
   > RSP=0000000000000000 CS:RIP=0000:0000000000000000 (XEN) CS:
   > sel=0x0010, attr=0x0209b, limit=0x00000000, base=0x0000000000000000 (XEN) DS:
   > sel=0x002b, attr=0x0c0f3, limit=0xffffffff, base=0x0000000000000000
   > (XEN) SS: sel=0x0018, attr=0x0c093, limit=0xffffffff,
   > base=0x0000000000000000 (XEN) ES: sel=0x002b, attr=0x0c0f3,
   > limit=0xffffffff, base=0x0000000000000000 (XEN) FS: sel=0x0053,
   > attr=0x040f3, limit=0x0000fc00, base=0x00000000ff9ce000 (XEN) GS:
   > sel=0x002b, attr=0x0c0f3, limit=0xffffffff, base=0xffffd00020b12000
   > (XEN) GDTR:                           limit=0x0000007f,
   > base=0xffffd00020b1f040 (XEN) LDTR: sel=0x0000, attr=0x1c000,
   > limit=0xffffffff, base=0x0000000000000000 (XEN) IDTR:
   >        limit=0x00000fff, base=0xffffd00020b1f0c0 (XEN) TR: sel=0x0040,
   > attr=0x0008b, limit=0x00000067, base=0xffffd00020b18080 (XEN) Guest
   > PAT = 0x0000050100070406 (XEN) TSC Offset = fffffefc546de420 (XEN)
   > DebugCtl=0000000000000000 DebugExceptions=0000000000000000 (XEN)
   > Interruptibility=0008 ActivityState=0000


   Seems the virtual NMI handling for nested is not correct: blocking by NMI is set, but trying to inject a NMI to guest.

   Best regards,
   Yang





   I am more than willing to test any patches you may have.  I am also leaving this testbed setup in case you or others need further logs, debugs run, etc.


   Thanks,

   -steve


[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jan Beulich]

>>> On 16.10.14 at 08:18, <yang.z.zhang@intel.com> wrote:
> Sorry for the later reply. Yes, this is a known issue to me but I didn't 
> have time to cook a patch fix it. As Jan pointed out, the NMI handling logic 
> is wrong in current nested logic. But it is not a trivial task to fix them. I 
> will do it once I have the time or if you are interesting in it, a patch from 
> you is welcome.

If you were to at least comment on the two possible routes I
outlined (quoted below), I could see to get to this (considering
that the issue - as you point out subsequently - got brought up
more than half a year ago the first time, and is still unaddressed)
at least if the vmx_idtv_reinject() related route would be a
possible one.

Jan

>>>> From: Jan Beulich [mailto:JBeulich@suse.com]
>>>> So the problem here is that
>>>> 
>>>>> (XEN) Interruptibility=0008 ActivityState=0000
>>>> VMX_INTR_SHADOW_NMI is set while
>>>> 
>>>>> (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb
>>>> PIN_BASED_VIRTUAL_NMIS is active and
>>>> 
>>>>> (XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003
>>>>> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
>>>>> (XEN)         reason=80000021 qualification=00000000
>>>>> (XEN) IDTVectoring: info=80000202 errcode=00000000
>>>> an NMI is being injected. This case is explicitly mentioned in Vol
>>>> 3 section 31.7.1.2 (Resuming Guest Software after Handling an
>>>> Exception). Either there needs to be extra code in vvmx.c to clear
>>>> VMX_INTR_SHADOW_NMI (as the second sub-bullet point of the last
>>>> bullet point says), or the second half of vmx_idtv_reinject() needs
>>>> to be done without regard to nestedhvm_vcpu_in_guestmode(v) (and
>>>> maybe also without regard to EXIT_REASON_TASK_SWITCH).
>>>> 
>>>> Speaking of SDM sections: There are quite a few references in the
>>>> code that name just section numbers (in the case here, several
>>>> references to sections 25.7.1.* exist). These numbers become stale
>>>> quite quickly (here they're now 31.7.1.*), so in order to help
>>>> digging through issues like the one here, can I please ask one of
>>>> you to go through and replace (or at least amend) these numbers
>>>> with the sections' titles (which I hope won't get altered that quickly)?

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jeroen Groenewegen van der Weyden]

Hi,

Is there any test I can do on changed code?

mvg,
Jeroen.

Jan Beulich schreef op 16-10-2014 om 08:41:
>>>> On 16.10.14 at 08:18, <yang.z.zhang@intel.com> wrote:
>> Sorry for the later reply. Yes, this is a known issue to me but I didn't
>> have time to cook a patch fix it. As Jan pointed out, the NMI handling logic
>> is wrong in current nested logic. But it is not a trivial task to fix them. I
>> will do it once I have the time or if you are interesting in it, a patch from
>> you is welcome.
> If you were to at least comment on the two possible routes I
> outlined (quoted below), I could see to get to this (considering
> that the issue - as you point out subsequently - got brought up
> more than half a year ago the first time, and is still unaddressed)
> at least if the vmx_idtv_reinject() related route would be a
> possible one.
>
> Jan
>
>>>>> From: Jan Beulich [mailto:JBeulich@suse.com]
>>>>> So the problem here is that
>>>>>
>>>>>> (XEN) Interruptibility=0008 ActivityState=0000
>>>>> VMX_INTR_SHADOW_NMI is set while
>>>>>
>>>>>> (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb
>>>>> PIN_BASED_VIRTUAL_NMIS is active and
>>>>>
>>>>>> (XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003
>>>>>> (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003
>>>>>> (XEN)         reason=80000021 qualification=00000000
>>>>>> (XEN) IDTVectoring: info=80000202 errcode=00000000
>>>>> an NMI is being injected. This case is explicitly mentioned in Vol
>>>>> 3 section 31.7.1.2 (Resuming Guest Software after Handling an
>>>>> Exception). Either there needs to be extra code in vvmx.c to clear
>>>>> VMX_INTR_SHADOW_NMI (as the second sub-bullet point of the last
>>>>> bullet point says), or the second half of vmx_idtv_reinject() needs
>>>>> to be done without regard to nestedhvm_vcpu_in_guestmode(v) (and
>>>>> maybe also without regard to EXIT_REASON_TASK_SWITCH).
>>>>>
>>>>> Speaking of SDM sections: There are quite a few references in the
>>>>> code that name just section numbers (in the case here, several
>>>>> references to sections 25.7.1.* exist). These numbers become stale
>>>>> quite quickly (here they're now 31.7.1.*), so in order to help
>>>>> digging through issues like the one here, can I please ask one of
>>>>> you to go through and replace (or at least amend) these numbers
>>>>> with the sections' titles (which I hope won't get altered that quickly)?
>
>

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jan Beulich]

>>> On 29.10.14 at 15:17, <groen692@grosc.com> wrote:
> Is there any test I can do on changed code?

Strange question with there not being any changed code yet.

Jan

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[George Dunlap]

On Wed, Oct 29, 2014 at 4:42 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 29.10.14 at 15:17, <groen692@grosc.com> wrote:
>> Is there any test I can do on changed code?
>
> Strange question with there not being any changed code yet.

I think what Jan means is, sorry, we haven't had a chance to actually
work on this yet, but thanks for the ping. :-)

 -George

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jeroen Groenewegen van der Weyden]

Hi All,

Did anyone find the time yet? I'm still more then willing testing any 
patches.

BR,
Jeroen.

George Dunlap schreef op 3-11-2014 om 12:16:
> I think what Jan means is, sorry, we haven't had a chance to actually
> work on this yet, but thanks for the ping.:-)
>
>   -George

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jan Beulich]

>>> On 09.12.14 at 10:09, <groen692@grosc.com> wrote:
> Did anyone find the time yet? I'm still more then willing testing any 
> patches.

Just yesterday we were told by Intel that they still can't foresee when
they will find time.

Jan

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jeroen Groenewegen van der Weyden]

Hi Jan,

Anything planned concerning this?

BR,
Jeroen.

Jan Beulich schreef op 9-12-2014 om 10:17:
>>>> On 09.12.14 at 10:09, <groen692@grosc.com> wrote:
>> Did anyone find the time yet? I'm still more then willing testing any
>> patches.
> Just yesterday we were told by Intel that they still can't foresee when
> they will find time.
>
> Jan
>
>

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jan Beulich]

>>> On 26.02.15 at 19:56, <groen692@grosc.com> wrote:
> Anything planned concerning this?

Thanks for asking, but to be honest I don't understand why you're
(not the first time iirc) pinging me rather than the VMX maintainers
who had promised to do that work. All I can say is that I'm being
told it's still on their todo list, but no-one has found time yet. And
please remember that all of nested virt is still marked preview, i.e.
not fully supported - for reasons like the bug here.

Jan

> Jan Beulich schreef op 9-12-2014 om 10:17:
>>>>> On 09.12.14 at 10:09, <groen692@grosc.com> wrote:
>>> Did anyone find the time yet? I'm still more then willing testing any
>>> patches.
>> Just yesterday we were told by Intel that they still can't foresee when
>> they will find time.
>>
>> Jan
>>
>>

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Jeroen Groenewegen van der Weyden]

Hi Jan,

I am sorry, But  had the impression you were somehow the owner of these 
feature. My mistake.
I understand the feature is in preview, But are you feeling intense 
presure when I ask for some status every 2 months? :-))


BR,
Jeroen

Jan Beulich schreef op 27-2-2015 om 09:08:
>>>> On 26.02.15 at 19:56, <groen692@grosc.com> wrote:
>> Anything planned concerning this?
> Thanks for asking, but to be honest I don't understand why you're
> (not the first time iirc) pinging me rather than the VMX maintainers
> who had promised to do that work. All I can say is that I'm being
> told it's still on their todo list, but no-one has found time yet. And
> please remember that all of nested virt is still marked preview, i.e.
> not fully supported - for reasons like the bug here.
>
> Jan
>
>> Jan Beulich schreef op 9-12-2014 om 10:17:
>>>>>> On 09.12.14 at 10:09, <groen692@grosc.com> wrote:
>>>> Did anyone find the time yet? I'm still more then willing testing any
>>>> patches.
>>> Just yesterday we were told by Intel that they still can't foresee when
>>> they will find time.
>>>
>>> Jan
>>>
>>>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

Re: Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.

[Li, Liang Z]

Hi All,

       I found a way to preproduce the bug very easy by using the apic->send_IPI_all(NMI_VECTOR)
 in L2 in a kernel module to trigger MNI. And I have verified that the bug can be fixed as Jan's
 suggestion, ' the second half of vmx_idtv_reinject() needs to be done without regard to
 nestedhvm_vcpu_in_guestmode(v)', I will send a patch later.

Liang

> -----Original Message-----
> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> bounces@lists.xen.org] On Behalf Of Jeroen Groenewegen van der Weyden
> Sent: Wednesday, March 04, 2015 6:24 PM
> To: Jan Beulich
> Cc: Tian, Kevin; George Dunlap; Dong, Eddie; xen-devel@lists.xen.org;
> Nakajima, Jun; Zhang, Yang Z
> Subject: Re: [Xen-devel] Crash of guest with nested vmx with Unknown
> nested vmexit reason 80000021.
> 
> Hi Jan,
> 
> I am sorry, But  had the impression you were somehow the owner of these
> feature. My mistake.
> I understand the feature is in preview, But are you feeling intense presure
> when I ask for some status every 2 months? :-))
> 
> 
> BR,
> Jeroen
> 
> Jan Beulich schreef op 27-2-2015 om 09:08:
> >>>> On 26.02.15 at 19:56, <groen692@grosc.com> wrote:
> >> Anything planned concerning this?
> > Thanks for asking, but to be honest I don't understand why you're (not
> > the first time iirc) pinging me rather than the VMX maintainers who
> > had promised to do that work. All I can say is that I'm being told
> > it's still on their todo list, but no-one has found time yet. And
> > please remember that all of nested virt is still marked preview, i.e.
> > not fully supported - for reasons like the bug here.
> >
> > Jan
> >
> >> Jan Beulich schreef op 9-12-2014 om 10:17:
> >>>>>> On 09.12.14 at 10:09, <groen692@grosc.com> wrote:
> >>>> Did anyone find the time yet? I'm still more then willing testing
> >>>> any patches.
> >>> Just yesterday we were told by Intel that they still can't foresee
> >>> when they will find time.
> >>>
> >>> Jan
> >>>
> >>>
> >
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> >
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel