[BUG] Consistent LBR/TSX vmentry failure (0x80000022) calling domain_crash() in vmx.c:3324

[BUG] Consistent LBR/TSX vmentry failure (0x80000022) calling domain_crash() in vmx.c:3324

[Elliot Killick]

Hello,

Xen is crashing Windows 10 (64-bit) VMs consistently whenever IDA
Debugger
(https://www.hex-rays.com/products/ida/support/download_freeware/)
launches the Local Windows Debugger. The crash occurs when trying to
launch the debugger against any executable (e.g. calc.exe) right at the
time IDA says it is "Moving segment from <X address> to <Y address>".

Tested on Windows 7, 8 and Linux as well but the bug is only triggered
on Windows 10. Happens whether or not IDA is running with administrator
privileges. No drivers/VM tools installed. Windows has a bug check code
of zero, leaves no memory dump, nothing in the logs from QEMU in Dom0,
the domain just powers off immediately leaving a record of the incident
in the hypervisor.log. So, it does appear to be a Xen issue. Modern
Intel CPU.

Does anyone have some ideas on what may be causing this?

Thank you,
Elliot

hypervisor.log:

(XEN) d24v1 vmentry failure (reason 0x80000022): MSR loading (entry 1)
(XEN)   msr 000001dd val 1ffff80676f52be5 (mbz 0)
(XEN) ************* VMCS Area **************
(XEN) *** Guest State ***
(XEN) CR0: actual=0x0000000080050031, shadow=0x0000000080050031,
gh_mask=ffffffffffffffff
(XEN) CR4: actual=0x0000000000172678, shadow=0x0000000000170678,
gh_mask=ffffffffffffffff
(XEN) CR3 = 0x00000001b2725002
(XEN) RSP = 0xffff960c962d1af8 (0xffff960c962d1af8)  RIP =
0xfffff80676dc29d0 (0xfffff80676dc29d0)
(XEN) RFLAGS=0x00000002 (0x00000002)  DR7 = 0x0000000000000400
(XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN)        sel  attr  limit   base
(XEN)   CS: 0010 0209b 00000000 0000000000000000
(XEN)   DS: 002b 0c0f3 ffffffff 0000000000000000
(XEN)   SS: 0000 1c000 ffffffff 0000000000000000
(XEN)   ES: 002b 0c0f3 ffffffff 0000000000000000
(XEN)   FS: 0053 040f3 00007c00 0000000000000000
(XEN)   GS: 002b 0c0f3 ffffffff ffffb181c2d00000
(XEN) GDTR:            00000057 ffffb181c2d15fb0
(XEN) LDTR: 0000 1c000 ffffffff 0000000000000000
(XEN) IDTR:            00000fff ffffb181c2d13000
(XEN)   TR: 0040 0008b 00000067 ffffb181c2d14000
(XEN) EFER = 0x0000000000000400  PAT = 0x0007010600070106
(XEN) PreemptionTimer = 0x00000000  SM Base = 0x00000000
(XEN) DebugCtl = 0x0000000000000001  DebugExceptions = 0x0000000000000000
(XEN) Interruptibility = 00000000  ActivityState = 00000000
(XEN) *** Host State ***
(XEN) RIP = 0xffff82d0801f0840 (vmx_asm_vmexit_handler)  RSP =
0xffff8304204f7f70
(XEN) CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040
(XEN) FSBase=0000000000000000 GSBase=0000000000000000
TRBase=ffff83042bb02c80
(XEN) GDTBase=ffff83042baf2000 IDTBase=ffff8304204ee000
(XEN) CR0=0000000080050033 CR3=0000000417a40000 CR4=00000000001526e0
(XEN) Sysenter RSP=ffff8304204f7fa0 CS:RIP=e008:ffff82d0802144b0
(XEN) EFER = 0x0000000000000000  PAT = 0x0000050100070406
(XEN) *** Control State ***
(XEN) PinBased=0000003f CPUBased=b62065fa SecondaryExec=000054eb
(XEN) EntryControls=000053ff ExitControls=000fefff
(XEN) ExceptionBitmap=00060002 PFECmask=00000000 PFECmatch=00000000
(XEN) VMEntry: intr_info=0000002f errcode=00000000 ilen=00000000
(XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000002
(XEN)         reason=80000022 qualification=0000000000000002
(XEN) IDTVectoring: info=00000000 errcode=00000000
(XEN) TSC Offset = 0xffff797cd2ddfef4  TSC Multiplier = 0x0000000000000000
(XEN) TPR Threshold = 0x00  PostedIntrVec = 0x00
(XEN) EPT pointer = 0x000000041444701e  EPTP index = 0x0000
(XEN) PLE Gap=00000080 Window=00001000
(XEN) Virtual processor ID = 0xf71d VMfunc controls = 0000000000000000
(XEN) **************************************
(XEN) domain_crash called from vmx.c:3324
(XEN) Domain 24 (vcpu#1) crashed on cpu#1:
(XEN) ----[ Xen-4.8.5-15.fc25  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0010:[<fffff80676dc29d0>]
(XEN) RFLAGS: 0000000000000002   CONTEXT: hvm guest (d24v1)
(XEN) rax: 0000000000000001   rbx: 0000000000000000   rcx: 00000000000001d9
(XEN) rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
(XEN) rbp: ffff960c962d1b80   rsp: ffff960c962d1af8   r8:  0000000000000002
(XEN) r9:  ffffb181c2d00000   r10: ffffc48c879b6080   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050031   cr4: 0000000000170678
(XEN) cr3: 00000001b2725002   cr2: 00007ff89f231770
(XEN) fsb: 0000000000000000   gsb: ffffb181c2d00000   gss: 000000146673a000
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 0000   cs: 0010

Re: [BUG] Consistent LBR/TSX vmentry failure (0x80000022) calling domain_crash() in vmx.c:3324

[Andrew Cooper]

On 20/05/2020 11:33, Elliot Killick wrote:
> Hello,
>
> Xen is crashing Windows 10 (64-bit) VMs consistently whenever IDA
> Debugger
> (https://www.hex-rays.com/products/ida/support/download_freeware/)
> launches the Local Windows Debugger. The crash occurs when trying to
> launch the debugger against any executable (e.g. calc.exe) right at the
> time IDA says it is "Moving segment from <X address> to <Y address>".
>
> Tested on Windows 7, 8 and Linux as well but the bug is only triggered
> on Windows 10. Happens whether or not IDA is running with administrator
> privileges. No drivers/VM tools installed. Windows has a bug check code
> of zero, leaves no memory dump, nothing in the logs from QEMU in Dom0,
> the domain just powers off immediately leaving a record of the incident
> in the hypervisor.log. So, it does appear to be a Xen issue. Modern
> Intel CPU.
>
> Does anyone have some ideas on what may be causing this?

What exact CPU do you have?  This looks exactly like the
Haswell/Broadwell TSX errata.

~Andrew

Re: [BUG] Consistent LBR/TSX vmentry failure (0x80000022) calling domain_crash() in vmx.c:3324

[Elliot Killick]

On 2020-05-20 11:10, Andrew Cooper wrote:
> On 20/05/2020 11:33, Elliot Killick wrote:
>> Hello,
>>
>> Xen is crashing Windows 10 (64-bit) VMs consistently whenever IDA
>> Debugger
>> (https://www.hex-rays.com/products/ida/support/download_freeware/)
>> launches the Local Windows Debugger. The crash occurs when trying to
>> launch the debugger against any executable (e.g. calc.exe) right at the
>> time IDA says it is "Moving segment from <X address> to <Y address>".
>>
>> Tested on Windows 7, 8 and Linux as well but the bug is only triggered
>> on Windows 10. Happens whether or not IDA is running with administrator
>> privileges. No drivers/VM tools installed. Windows has a bug check code
>> of zero, leaves no memory dump, nothing in the logs from QEMU in Dom0,
>> the domain just powers off immediately leaving a record of the incident
>> in the hypervisor.log. So, it does appear to be a Xen issue. Modern
>> Intel CPU.
>>
>> Does anyone have some ideas on what may be causing this?
> 
> What exact CPU do you have?  This looks exactly like the
> Haswell/Broadwell TSX errata.
> 
> ~Andrew
> 

i5-4590

Re: [BUG] Consistent LBR/TSX vmentry failure (0x80000022) calling domain_crash() in vmx.c:3324

[Andrew Cooper]

On 20/05/2020 12:20, Elliot Killick wrote:
> On 2020-05-20 11:10, Andrew Cooper wrote:
>> On 20/05/2020 11:33, Elliot Killick wrote:
>>> Hello,
>>>
>>> Xen is crashing Windows 10 (64-bit) VMs consistently whenever IDA
>>> Debugger
>>> (https://www.hex-rays.com/products/ida/support/download_freeware/)
>>> launches the Local Windows Debugger. The crash occurs when trying to
>>> launch the debugger against any executable (e.g. calc.exe) right at the
>>> time IDA says it is "Moving segment from <X address> to <Y address>".
>>>
>>> Tested on Windows 7, 8 and Linux as well but the bug is only triggered
>>> on Windows 10. Happens whether or not IDA is running with administrator
>>> privileges. No drivers/VM tools installed. Windows has a bug check code
>>> of zero, leaves no memory dump, nothing in the logs from QEMU in Dom0,
>>> the domain just powers off immediately leaving a record of the incident
>>> in the hypervisor.log. So, it does appear to be a Xen issue. Modern
>>> Intel CPU.
>>>
>>> Does anyone have some ideas on what may be causing this?
>> What exact CPU do you have?  This looks exactly like the
>> Haswell/Broadwell TSX errata.
>>
>> ~Andrew
>>
> i5-4590

How about the output of `head /proc/cpuinfo` in dom0?

~Andrew

Re: [BUG] Consistent LBR/TSX vmentry failure (0x80000022) calling domain_crash() in vmx.c:3324

[Elliot Killick]

On 2020-05-20 11:27, Andrew Cooper wrote:
> On 20/05/2020 12:20, Elliot Killick wrote:
>> On 2020-05-20 11:10, Andrew Cooper wrote:
>>> On 20/05/2020 11:33, Elliot Killick wrote:
>>>> Hello,
>>>>
>>>> Xen is crashing Windows 10 (64-bit) VMs consistently whenever IDA
>>>> Debugger
>>>> (https://www.hex-rays.com/products/ida/support/download_freeware/)
>>>> launches the Local Windows Debugger. The crash occurs when trying to
>>>> launch the debugger against any executable (e.g. calc.exe) right at the
>>>> time IDA says it is "Moving segment from <X address> to <Y address>".
>>>>
>>>> Tested on Windows 7, 8 and Linux as well but the bug is only triggered
>>>> on Windows 10. Happens whether or not IDA is running with administrator
>>>> privileges. No drivers/VM tools installed. Windows has a bug check code
>>>> of zero, leaves no memory dump, nothing in the logs from QEMU in Dom0,
>>>> the domain just powers off immediately leaving a record of the incident
>>>> in the hypervisor.log. So, it does appear to be a Xen issue. Modern
>>>> Intel CPU.
>>>>
>>>> Does anyone have some ideas on what may be causing this?
>>> What exact CPU do you have?  This looks exactly like the
>>> Haswell/Broadwell TSX errata.
>>>
>>> ~Andrew
>>>
>> i5-4590
> 
> How about the output of `head /proc/cpuinfo` in dom0?
> 
> ~Andrew
> 

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 60
model name	: Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
stepping	: 3
microcode	: 0x27
cpu MHz		: 3299.926
cache size	: 6144 KB
physical id	: 0

Re: [BUG] Consistent LBR/TSX vmentry failure (0x80000022) calling domain_crash() in vmx.c:3324

[Andrew Cooper]

On 20/05/2020 12:46, Elliot Killick wrote:
> On 2020-05-20 11:27, Andrew Cooper wrote:
>> On 20/05/2020 12:20, Elliot Killick wrote:
>>> On 2020-05-20 11:10, Andrew Cooper wrote:
>>>> On 20/05/2020 11:33, Elliot Killick wrote:
>>>>> Hello,
>>>>>
>>>>> Xen is crashing Windows 10 (64-bit) VMs consistently whenever IDA
>>>>> Debugger
>>>>> (https://www.hex-rays.com/products/ida/support/download_freeware/)
>>>>> launches the Local Windows Debugger. The crash occurs when trying to
>>>>> launch the debugger against any executable (e.g. calc.exe) right at the
>>>>> time IDA says it is "Moving segment from <X address> to <Y address>".
>>>>>
>>>>> Tested on Windows 7, 8 and Linux as well but the bug is only triggered
>>>>> on Windows 10. Happens whether or not IDA is running with administrator
>>>>> privileges. No drivers/VM tools installed. Windows has a bug check code
>>>>> of zero, leaves no memory dump, nothing in the logs from QEMU in Dom0,
>>>>> the domain just powers off immediately leaving a record of the incident
>>>>> in the hypervisor.log. So, it does appear to be a Xen issue. Modern
>>>>> Intel CPU.
>>>>>
>>>>> Does anyone have some ideas on what may be causing this?
>>>> What exact CPU do you have?  This looks exactly like the
>>>> Haswell/Broadwell TSX errata.
>>>>
>>>> ~Andrew
>>>>
>>> i5-4590
>> How about the output of `head /proc/cpuinfo` in dom0?
>>
>> ~Andrew
>>
> processor	: 0
> vendor_id	: GenuineIntel
> cpu family	: 6
> model		: 60
> model name	: Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
> stepping	: 3
> microcode	: 0x27
> cpu MHz		: 3299.926
> cache size	: 6144 KB
> physical id	: 0

Ok, so the errata is one of HSM182/HSD172.

Xen has workaround for all of these.  However, I also see:

> (XEN) ----[ Xen-4.8.5-15.fc25  x86_64  debug=n   Not tainted ]----

which is an obsolete version of Xen these days.  It looks like these
issues were first fixed in Xen 4.9, but you should upgrade to something
rather newer.

~Andrew

Re: [BUG] Consistent LBR/TSX vmentry failure (0x80000022) calling domain_crash() in vmx.c:3324

[Elliot Killick]

On 2020-05-20 12:31, Andrew Cooper wrote:
> On 20/05/2020 12:46, Elliot Killick wrote:
>> On 2020-05-20 11:27, Andrew Cooper wrote:
>>> On 20/05/2020 12:20, Elliot Killick wrote:
>>>> On 2020-05-20 11:10, Andrew Cooper wrote:
>>>>> On 20/05/2020 11:33, Elliot Killick wrote:
>>>>>> Hello,
>>>>>>
>>>>>> Xen is crashing Windows 10 (64-bit) VMs consistently whenever IDA
>>>>>> Debugger
>>>>>> (https://www.hex-rays.com/products/ida/support/download_freeware/)
>>>>>> launches the Local Windows Debugger. The crash occurs when trying to
>>>>>> launch the debugger against any executable (e.g. calc.exe) right at the
>>>>>> time IDA says it is "Moving segment from <X address> to <Y address>".
>>>>>>
>>>>>> Tested on Windows 7, 8 and Linux as well but the bug is only triggered
>>>>>> on Windows 10. Happens whether or not IDA is running with administrator
>>>>>> privileges. No drivers/VM tools installed. Windows has a bug check code
>>>>>> of zero, leaves no memory dump, nothing in the logs from QEMU in Dom0,
>>>>>> the domain just powers off immediately leaving a record of the incident
>>>>>> in the hypervisor.log. So, it does appear to be a Xen issue. Modern
>>>>>> Intel CPU.
>>>>>>
>>>>>> Does anyone have some ideas on what may be causing this?
>>>>> What exact CPU do you have?  This looks exactly like the
>>>>> Haswell/Broadwell TSX errata.
>>>>>
>>>>> ~Andrew
>>>>>
>>>> i5-4590
>>> How about the output of `head /proc/cpuinfo` in dom0?
>>>
>>> ~Andrew
>>>
>> processor	: 0
>> vendor_id	: GenuineIntel
>> cpu family	: 6
>> model		: 60
>> model name	: Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
>> stepping	: 3
>> microcode	: 0x27
>> cpu MHz		: 3299.926
>> cache size	: 6144 KB
>> physical id	: 0
> 
> Ok, so the errata is one of HSM182/HSD172.
> 
> Xen has workaround for all of these.  However, I also see:
> 
>> (XEN) ----[ Xen-4.8.5-15.fc25  x86_64  debug=n   Not tainted ]----
> 
> which is an obsolete version of Xen these days.  It looks like these
> issues were first fixed in Xen 4.9, but you should upgrade to something
> rather newer.
> 
> ~Andrew
> 

Ah, so this is originally a CPU bug which Xen has had to patch over.

As for the Xen version, that's controlled by the "distribution" of Xen I
run which is Qubes. To remedy this I could run the testing stream of
Qubes which currently provides the latest version of Xen (4.13) but that
could bring its own set of problems.

Thank you for the help, Andrew!

Re: [BUG] Consistent LBR/TSX vmentry failure (0x80000022) calling domain_crash() in vmx.c:3324

[Andrew Cooper]

On 20/05/2020 15:59, Elliot Killick wrote:
> On 2020-05-20 12:31, Andrew Cooper wrote:
>> On 20/05/2020 12:46, Elliot Killick wrote:
>>> processor	: 0
>>> vendor_id	: GenuineIntel
>>> cpu family	: 6
>>> model		: 60
>>> model name	: Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
>>> stepping	: 3
>>> microcode	: 0x27
>>> cpu MHz		: 3299.926
>>> cache size	: 6144 KB
>>> physical id	: 0
>> Ok, so the errata is one of HSM182/HSD172.
>>
>> Xen has workaround for all of these.  However, I also see:
>>
>>> (XEN) ----[ Xen-4.8.5-15.fc25  x86_64  debug=n   Not tainted ]----
>> which is an obsolete version of Xen these days.  It looks like these
>> issues were first fixed in Xen 4.9, but you should upgrade to something
>> rather newer.
>>
>> ~Andrew
>>
> Ah, so this is originally a CPU bug which Xen has had to patch over.

Yes.  It was an unintended consequence of Intel being forced to disable
TSX in most of their Haswell/Broadwell CPUs.

> As for the Xen version, that's controlled by the "distribution" of Xen I
> run which is Qubes. To remedy this I could run the testing stream of
> Qubes which currently provides the latest version of Xen (4.13) but that
> could bring its own set of problems.

Ah, in which case Qubes will probably consider backporting the fixes. 
Open a bug with them, and I can probably point out a minimum set of
backports to make it work.

~Andrew

Re: [BUG] Consistent LBR/TSX vmentry failure (0x80000022) calling domain_crash() in vmx.c:3324

[Elliot Killick]

On 2020-05-20 16:53, Andrew Cooper wrote:
> On 20/05/2020 15:59, Elliot Killick wrote:
>> On 2020-05-20 12:31, Andrew Cooper wrote:
>>> On 20/05/2020 12:46, Elliot Killick wrote:
>>>> processor	: 0
>>>> vendor_id	: GenuineIntel
>>>> cpu family	: 6
>>>> model		: 60
>>>> model name	: Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
>>>> stepping	: 3
>>>> microcode	: 0x27
>>>> cpu MHz		: 3299.926
>>>> cache size	: 6144 KB
>>>> physical id	: 0
>>> Ok, so the errata is one of HSM182/HSD172.
>>>
>>> Xen has workaround for all of these.  However, I also see:
>>>
>>>> (XEN) ----[ Xen-4.8.5-15.fc25  x86_64  debug=n   Not tainted ]----
>>> which is an obsolete version of Xen these days.  It looks like these
>>> issues were first fixed in Xen 4.9, but you should upgrade to something
>>> rather newer.
>>>
>>> ~Andrew
>>>
>> Ah, so this is originally a CPU bug which Xen has had to patch over.
> 
> Yes.  It was an unintended consequence of Intel being forced to disable
> TSX in most of their Haswell/Broadwell CPUs.
> 
>> As for the Xen version, that's controlled by the "distribution" of Xen I
>> run which is Qubes. To remedy this I could run the testing stream of
>> Qubes which currently provides the latest version of Xen (4.13) but that
>> could bring its own set of problems.
> 
> Ah, in which case Qubes will probably consider backporting the fixes. 
> Open a bug with them, and I can probably point out a minimum set of
> backports to make it work.
> 
> ~Andrew
> 

Hi, Andrew

I've opened up a bug with Qubes here:
https://github.com/QubesOS/qubes-issues/issues/5848

Please highlight the required backports at your earliest convenience,
thank you.