* VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
@ 2021-02-01 15:26 James Dingwall
2021-02-03 14:55 ` Jan Beulich
0 siblings, 1 reply; 7+ messages in thread
From: James Dingwall @ 2021-02-01 15:26 UTC (permalink / raw)
To: xen-devel
Hi,
I am building the xen 4.11 branch at
310ab79875cb705cc2c7daddff412b5a4899f8c9 which includes commit
3b5de119f0399cbe745502cb6ebd5e6633cc139c "86/msr: fix handling of
MSR_IA32_PERF_{STATUS/CTL}". I think this should address this error
recorded in xen's dmesg:
(XEN) d11v0 VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
I have removed `viridian = [..]` from the xen config nut still get this
reliably when launching PassMark Performance Test and it is collecting
CPU information.
This is recorded in the domain qemu-dm log:
21244@1612191983.279616:xen_platform_log xen platform: XEN|BUGCHECK: ====>
21244@1612191983.279819:xen_platform_log xen platform: XEN|BUGCHECK: SYSTEM_SERVICE_EXCEPTION: 00000000C0000096 FFFFF800A43C72C5 FFFFD0014343D580 0000000000000000
21244@1612191983.279959:xen_platform_log xen platform: XEN|BUGCHECK: EXCEPTION (FFFFF800A43C72C5):
21244@1612191983.280075:xen_platform_log xen platform: XEN|BUGCHECK: - Code = C148320F
21244@1612191983.280205:xen_platform_log xen platform: XEN|BUGCHECK: - Flags = 0B4820E2
21244@1612191983.280346:xen_platform_log xen platform: XEN|BUGCHECK: - Address = 0000A824948D4800
21244@1612191983.280504:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[0] = 8B00000769850F07
21244@1612191983.280633:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[1] = 46B70F4024448906
21244@1612191983.280754:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[2] = 0F44442444896604
21244@1612191983.280876:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[3] = E983C88B410646B6
21244@1612191983.281012:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[4] = 0D7401E9831E7401
21244@1612191983.281172:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[5] = 54B70F217502F983
21244@1612191983.281304:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[6] = 54B70F15EBED4024
21244@1612191983.281426:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[7] = EBC0B70FED664024
21244@1612191983.281547:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[8] = 0FEC402454B70F09
21244@1612191983.281668:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[9] = 448B42244489C0B6
21244@1612191983.281809:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[10] = 2444B70F06894024
21244@1612191983.281932:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[11] = 4688440446896644
21244@1612191983.282052:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[12] = 0000073846C74906
21244@1612191983.282185:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[13] = F8830000070AE900
21244@1612191983.282340:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[14] = 8B000006F9850F07
21244@1612191983.282480:xen_platform_log xen platform: XEN|BUGCHECK: EXCEPTION (0000A824848948C2):
21244@1612191983.282617:xen_platform_log xen platform: XEN|BUGCHECK: CONTEXT (FFFFD0014343D580):
21244@1612191983.282717:xen_platform_log xen platform: XEN|BUGCHECK: - GS = 002B
21244@1612191983.282816:xen_platform_log xen platform: XEN|BUGCHECK: - FS = 0053
21244@1612191983.282914:xen_platform_log xen platform: XEN|BUGCHECK: - ES = 002B
21244@1612191983.283011:xen_platform_log xen platform: XEN|BUGCHECK: - DS = 002B
21244@1612191983.283127:xen_platform_log xen platform: XEN|BUGCHECK: - SS = 0018
21244@1612191983.283226:xen_platform_log xen platform: XEN|BUGCHECK: - CS = 0010
21244@1612191983.283332:xen_platform_log xen platform: XEN|BUGCHECK: - EFLAGS = 00000202
21244@1612191983.283444:xen_platform_log xen platform: XEN|BUGCHECK: - RDI = 00000000F64D5C20
21244@1612191983.283555:xen_platform_log xen platform: XEN|BUGCHECK: - RSI = 00000000F6367280
21244@1612191983.283666:xen_platform_log xen platform: XEN|BUGCHECK: - RBX = 000000008011E060
21244@1612191983.283810:xen_platform_log xen platform: XEN|BUGCHECK: - RDX = 00000000F64D5C20
21244@1612191983.283972:xen_platform_log xen platform: XEN|BUGCHECK: - RCX = 0000000000000199
21244@1612191983.284350:xen_platform_log xen platform: XEN|BUGCHECK: - RAX = 0000000000000004
21244@1612191983.284523:xen_platform_log xen platform: XEN|BUGCHECK: - RBP = 000000004343E891
21244@1612191983.284658:xen_platform_log xen platform: XEN|BUGCHECK: - RIP = 00000000A43C72C5
21244@1612191983.284842:xen_platform_log xen platform: XEN|BUGCHECK: - RSP = 000000004343DFA0
21244@1612191983.284959:xen_platform_log xen platform: XEN|BUGCHECK: - R8 = 0000000000000008
21244@1612191983.285073:xen_platform_log xen platform: XEN|BUGCHECK: - R9 = 000000000000000E
21244@1612191983.285188:xen_platform_log xen platform: XEN|BUGCHECK: - R10 = 0000000000000002
21244@1612191983.285304:xen_platform_log xen platform: XEN|BUGCHECK: - R11 = 000000004343E808
21244@1612191983.285420:xen_platform_log xen platform: XEN|BUGCHECK: - R12 = 0000000000000000
21244@1612191983.285564:xen_platform_log xen platform: XEN|BUGCHECK: - R13 = 00000000F7964E50
21244@1612191983.285680:xen_platform_log xen platform: XEN|BUGCHECK: - R14 = 00000000F64D5C20
21244@1612191983.285796:xen_platform_log xen platform: XEN|BUGCHECK: - R15 = 00000000F7964E50
21244@1612191983.285888:xen_platform_log xen platform: XEN|BUGCHECK: STACK:
21244@1612191983.286105:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343E810: (0000000000000000 000000004343E891 0000000000000002 00000000F75F08A0) ntoskrnl.exe + 0000000000485507
21244@1612191983.286340:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343E8E0: (00000000F75F0805 000000004343EB80 00000000F6A62CC0 00000000F75F08A0) ntoskrnl.exe + 0000000000486468
21244@1612191983.286547:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343EA20: (0000000000000000 0000000000000000 0000000000000000 0000000000000000) ntoskrnl.exe + 0000000000458CAE
21244@1612191983.286755:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343EA90: (0000000000000000 0000000000000000 000000007DBED000 000000007DA00028) ntoskrnl.exe + 00000000001501A3
21244@1612191983.286976:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE388: (00000000587D5673 0000000058F40000 0000000006002D2B 0000000000000000) 00007FFB5B3207CA
21244@1612191983.287171:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE390: (0000000058F40000 0000000006002D2B 0000000000000000 00000000160C86D8) 00007FFB587D5673
21244@1612191983.287390:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE398: (0000000006002D2B 0000000000000000 00000000160C86D8 0000000009ABE3E0) 00007FFB58F40000
21244@1612191983.287584:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE3A0: (0000000000000000 00000000160C86D8 0000000009ABE3E0 000000008011E060) 00007FFB06002D2B
21244@1612191983.287777:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE3A8: (00000000160C86D8 0000000009ABE3E0 000000008011E060 0000000009ABE4A0) 0000000000000000
21244@1612191983.287898:xen_platform_log xen platform: XEN|BUGCHECK: <====
The Windows guest is running winpv drivers 8.2.1.
I'm not quite sure what else to examine or change at this point so any
guidance would be welcome.
Thanks,
James
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
2021-02-01 15:26 VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0 James Dingwall
@ 2021-02-03 14:55 ` Jan Beulich
2021-02-03 15:04 ` Paul Durrant
2021-02-04 8:46 ` James Dingwall
0 siblings, 2 replies; 7+ messages in thread
From: Jan Beulich @ 2021-02-03 14:55 UTC (permalink / raw)
To: James Dingwall; +Cc: xen-devel
On 01.02.2021 16:26, James Dingwall wrote:
> I am building the xen 4.11 branch at
> 310ab79875cb705cc2c7daddff412b5a4899f8c9 which includes commit
> 3b5de119f0399cbe745502cb6ebd5e6633cc139c "86/msr: fix handling of
> MSR_IA32_PERF_{STATUS/CTL}". I think this should address this error
> recorded in xen's dmesg:
>
> (XEN) d11v0 VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
It seems to me that you imply some information here which might
better be spelled out. As it stands I do not see the immediate
connection between the cited commit and the crash. C0000096 is
STATUS_PRIVILEGED_INSTRUCTION, which to me ought to be impossible
for code running in ring 0. Of course I may simply not know enough
about modern Windows' internals to understand the connection.
> I have removed `viridian = [..]` from the xen config nut still get this
> reliably when launching PassMark Performance Test and it is collecting
> CPU information.
>
> This is recorded in the domain qemu-dm log:
>
> 21244@1612191983.279616:xen_platform_log xen platform: XEN|BUGCHECK: ====>
> 21244@1612191983.279819:xen_platform_log xen platform: XEN|BUGCHECK: SYSTEM_SERVICE_EXCEPTION: 00000000C0000096 FFFFF800A43C72C5 FFFFD0014343D580 0000000000000000
> 21244@1612191983.279959:xen_platform_log xen platform: XEN|BUGCHECK: EXCEPTION (FFFFF800A43C72C5):
> 21244@1612191983.280075:xen_platform_log xen platform: XEN|BUGCHECK: - Code = C148320F
> 21244@1612191983.280205:xen_platform_log xen platform: XEN|BUGCHECK: - Flags = 0B4820E2
> 21244@1612191983.280346:xen_platform_log xen platform: XEN|BUGCHECK: - Address = 0000A824948D4800
> 21244@1612191983.280504:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[0] = 8B00000769850F07
> 21244@1612191983.280633:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[1] = 46B70F4024448906
> 21244@1612191983.280754:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[2] = 0F44442444896604
> 21244@1612191983.280876:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[3] = E983C88B410646B6
> 21244@1612191983.281012:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[4] = 0D7401E9831E7401
> 21244@1612191983.281172:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[5] = 54B70F217502F983
> 21244@1612191983.281304:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[6] = 54B70F15EBED4024
> 21244@1612191983.281426:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[7] = EBC0B70FED664024
> 21244@1612191983.281547:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[8] = 0FEC402454B70F09
> 21244@1612191983.281668:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[9] = 448B42244489C0B6
> 21244@1612191983.281809:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[10] = 2444B70F06894024
> 21244@1612191983.281932:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[11] = 4688440446896644
> 21244@1612191983.282052:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[12] = 0000073846C74906
> 21244@1612191983.282185:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[13] = F8830000070AE900
> 21244@1612191983.282340:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[14] = 8B000006F9850F07
> 21244@1612191983.282480:xen_platform_log xen platform: XEN|BUGCHECK: EXCEPTION (0000A824848948C2):
> 21244@1612191983.282617:xen_platform_log xen platform: XEN|BUGCHECK: CONTEXT (FFFFD0014343D580):
> 21244@1612191983.282717:xen_platform_log xen platform: XEN|BUGCHECK: - GS = 002B
> 21244@1612191983.282816:xen_platform_log xen platform: XEN|BUGCHECK: - FS = 0053
> 21244@1612191983.282914:xen_platform_log xen platform: XEN|BUGCHECK: - ES = 002B
> 21244@1612191983.283011:xen_platform_log xen platform: XEN|BUGCHECK: - DS = 002B
> 21244@1612191983.283127:xen_platform_log xen platform: XEN|BUGCHECK: - SS = 0018
> 21244@1612191983.283226:xen_platform_log xen platform: XEN|BUGCHECK: - CS = 0010
> 21244@1612191983.283332:xen_platform_log xen platform: XEN|BUGCHECK: - EFLAGS = 00000202
> 21244@1612191983.283444:xen_platform_log xen platform: XEN|BUGCHECK: - RDI = 00000000F64D5C20
> 21244@1612191983.283555:xen_platform_log xen platform: XEN|BUGCHECK: - RSI = 00000000F6367280
> 21244@1612191983.283666:xen_platform_log xen platform: XEN|BUGCHECK: - RBX = 000000008011E060
> 21244@1612191983.283810:xen_platform_log xen platform: XEN|BUGCHECK: - RDX = 00000000F64D5C20
> 21244@1612191983.283972:xen_platform_log xen platform: XEN|BUGCHECK: - RCX = 0000000000000199
> 21244@1612191983.284350:xen_platform_log xen platform: XEN|BUGCHECK: - RAX = 0000000000000004
> 21244@1612191983.284523:xen_platform_log xen platform: XEN|BUGCHECK: - RBP = 000000004343E891
> 21244@1612191983.284658:xen_platform_log xen platform: XEN|BUGCHECK: - RIP = 00000000A43C72C5
> 21244@1612191983.284842:xen_platform_log xen platform: XEN|BUGCHECK: - RSP = 000000004343DFA0
> 21244@1612191983.284959:xen_platform_log xen platform: XEN|BUGCHECK: - R8 = 0000000000000008
> 21244@1612191983.285073:xen_platform_log xen platform: XEN|BUGCHECK: - R9 = 000000000000000E
> 21244@1612191983.285188:xen_platform_log xen platform: XEN|BUGCHECK: - R10 = 0000000000000002
> 21244@1612191983.285304:xen_platform_log xen platform: XEN|BUGCHECK: - R11 = 000000004343E808
> 21244@1612191983.285420:xen_platform_log xen platform: XEN|BUGCHECK: - R12 = 0000000000000000
> 21244@1612191983.285564:xen_platform_log xen platform: XEN|BUGCHECK: - R13 = 00000000F7964E50
> 21244@1612191983.285680:xen_platform_log xen platform: XEN|BUGCHECK: - R14 = 00000000F64D5C20
> 21244@1612191983.285796:xen_platform_log xen platform: XEN|BUGCHECK: - R15 = 00000000F7964E50
I'm also confused by this - the pointer given for CONTEXT suggests this
is a 64-bit kernel, yet none of the registers - including RIP and RSP -
have non-zero upper 32 bits. Or is qemu truncating these values?
> 21244@1612191983.285888:xen_platform_log xen platform: XEN|BUGCHECK: STACK:
> 21244@1612191983.286105:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343E810: (0000000000000000 000000004343E891 0000000000000002 00000000F75F08A0) ntoskrnl.exe + 0000000000485507
> 21244@1612191983.286340:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343E8E0: (00000000F75F0805 000000004343EB80 00000000F6A62CC0 00000000F75F08A0) ntoskrnl.exe + 0000000000486468
> 21244@1612191983.286547:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343EA20: (0000000000000000 0000000000000000 0000000000000000 0000000000000000) ntoskrnl.exe + 0000000000458CAE
> 21244@1612191983.286755:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343EA90: (0000000000000000 0000000000000000 000000007DBED000 000000007DA00028) ntoskrnl.exe + 00000000001501A3
> 21244@1612191983.286976:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE388: (00000000587D5673 0000000058F40000 0000000006002D2B 0000000000000000) 00007FFB5B3207CA
> 21244@1612191983.287171:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE390: (0000000058F40000 0000000006002D2B 0000000000000000 00000000160C86D8) 00007FFB587D5673
> 21244@1612191983.287390:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE398: (0000000006002D2B 0000000000000000 00000000160C86D8 0000000009ABE3E0) 00007FFB58F40000
> 21244@1612191983.287584:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE3A0: (0000000000000000 00000000160C86D8 0000000009ABE3E0 000000008011E060) 00007FFB06002D2B
> 21244@1612191983.287777:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE3A8: (00000000160C86D8 0000000009ABE3E0 000000008011E060 0000000009ABE4A0) 0000000000000000
> 21244@1612191983.287898:xen_platform_log xen platform: XEN|BUGCHECK: <====
>
> The Windows guest is running winpv drivers 8.2.1.
>
> I'm not quite sure what else to examine or change at this point so any
> guidance would be welcome.
The hypervisor log (at maximum log levels) accompanying this might
help some. And of course, if possible, trying on a newer Xen (ideally
current master).
Jan
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
2021-02-03 14:55 ` Jan Beulich
@ 2021-02-03 15:04 ` Paul Durrant
2021-02-03 15:42 ` Jan Beulich
2021-02-04 8:46 ` James Dingwall
1 sibling, 1 reply; 7+ messages in thread
From: Paul Durrant @ 2021-02-03 15:04 UTC (permalink / raw)
To: 'Jan Beulich', 'James Dingwall'; +Cc: xen-devel
> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Jan Beulich
> Sent: 03 February 2021 14:55
> To: James Dingwall <james-xen@dingwall.me.uk>
> Cc: xen-devel@lists.xenproject.org
> Subject: Re: VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
>
> On 01.02.2021 16:26, James Dingwall wrote:
> > I am building the xen 4.11 branch at
> > 310ab79875cb705cc2c7daddff412b5a4899f8c9 which includes commit
> > 3b5de119f0399cbe745502cb6ebd5e6633cc139c "86/msr: fix handling of
> > MSR_IA32_PERF_{STATUS/CTL}". I think this should address this error
> > recorded in xen's dmesg:
> >
> > (XEN) d11v0 VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
>
> It seems to me that you imply some information here which might
> better be spelled out. As it stands I do not see the immediate
> connection between the cited commit and the crash. C0000096 is
> STATUS_PRIVILEGED_INSTRUCTION, which to me ought to be impossible
> for code running in ring 0. Of course I may simply not know enough
> about modern Windows' internals to understand the connection.
>
> > I have removed `viridian = [..]` from the xen config nut still get this
> > reliably when launching PassMark Performance Test and it is collecting
> > CPU information.
> >
> > This is recorded in the domain qemu-dm log:
> >
> > 21244@1612191983.279616:xen_platform_log xen platform: XEN|BUGCHECK: ====>
> > 21244@1612191983.279819:xen_platform_log xen platform: XEN|BUGCHECK: SYSTEM_SERVICE_EXCEPTION:
> 00000000C0000096 FFFFF800A43C72C5 FFFFD0014343D580 0000000000000000
> > 21244@1612191983.279959:xen_platform_log xen platform: XEN|BUGCHECK: EXCEPTION (FFFFF800A43C72C5):
> > 21244@1612191983.280075:xen_platform_log xen platform: XEN|BUGCHECK: - Code = C148320F
> > 21244@1612191983.280205:xen_platform_log xen platform: XEN|BUGCHECK: - Flags = 0B4820E2
> > 21244@1612191983.280346:xen_platform_log xen platform: XEN|BUGCHECK: - Address = 0000A824948D4800
> > 21244@1612191983.280504:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[0] =
> 8B00000769850F07
> > 21244@1612191983.280633:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[1] =
> 46B70F4024448906
> > 21244@1612191983.280754:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[2] =
> 0F44442444896604
> > 21244@1612191983.280876:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[3] =
> E983C88B410646B6
> > 21244@1612191983.281012:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[4] =
> 0D7401E9831E7401
> > 21244@1612191983.281172:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[5] =
> 54B70F217502F983
> > 21244@1612191983.281304:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[6] =
> 54B70F15EBED4024
> > 21244@1612191983.281426:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[7] =
> EBC0B70FED664024
> > 21244@1612191983.281547:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[8] =
> 0FEC402454B70F09
> > 21244@1612191983.281668:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[9] =
> 448B42244489C0B6
> > 21244@1612191983.281809:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[10] =
> 2444B70F06894024
> > 21244@1612191983.281932:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[11] =
> 4688440446896644
> > 21244@1612191983.282052:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[12] =
> 0000073846C74906
> > 21244@1612191983.282185:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[13] =
> F8830000070AE900
> > 21244@1612191983.282340:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[14] =
> 8B000006F9850F07
> > 21244@1612191983.282480:xen_platform_log xen platform: XEN|BUGCHECK: EXCEPTION (0000A824848948C2):
> > 21244@1612191983.282617:xen_platform_log xen platform: XEN|BUGCHECK: CONTEXT (FFFFD0014343D580):
> > 21244@1612191983.282717:xen_platform_log xen platform: XEN|BUGCHECK: - GS = 002B
> > 21244@1612191983.282816:xen_platform_log xen platform: XEN|BUGCHECK: - FS = 0053
> > 21244@1612191983.282914:xen_platform_log xen platform: XEN|BUGCHECK: - ES = 002B
> > 21244@1612191983.283011:xen_platform_log xen platform: XEN|BUGCHECK: - DS = 002B
> > 21244@1612191983.283127:xen_platform_log xen platform: XEN|BUGCHECK: - SS = 0018
> > 21244@1612191983.283226:xen_platform_log xen platform: XEN|BUGCHECK: - CS = 0010
> > 21244@1612191983.283332:xen_platform_log xen platform: XEN|BUGCHECK: - EFLAGS = 00000202
> > 21244@1612191983.283444:xen_platform_log xen platform: XEN|BUGCHECK: - RDI = 00000000F64D5C20
> > 21244@1612191983.283555:xen_platform_log xen platform: XEN|BUGCHECK: - RSI = 00000000F6367280
> > 21244@1612191983.283666:xen_platform_log xen platform: XEN|BUGCHECK: - RBX = 000000008011E060
> > 21244@1612191983.283810:xen_platform_log xen platform: XEN|BUGCHECK: - RDX = 00000000F64D5C20
> > 21244@1612191983.283972:xen_platform_log xen platform: XEN|BUGCHECK: - RCX = 0000000000000199
> > 21244@1612191983.284350:xen_platform_log xen platform: XEN|BUGCHECK: - RAX = 0000000000000004
> > 21244@1612191983.284523:xen_platform_log xen platform: XEN|BUGCHECK: - RBP = 000000004343E891
> > 21244@1612191983.284658:xen_platform_log xen platform: XEN|BUGCHECK: - RIP = 00000000A43C72C5
> > 21244@1612191983.284842:xen_platform_log xen platform: XEN|BUGCHECK: - RSP = 000000004343DFA0
> > 21244@1612191983.284959:xen_platform_log xen platform: XEN|BUGCHECK: - R8 = 0000000000000008
> > 21244@1612191983.285073:xen_platform_log xen platform: XEN|BUGCHECK: - R9 = 000000000000000E
> > 21244@1612191983.285188:xen_platform_log xen platform: XEN|BUGCHECK: - R10 = 0000000000000002
> > 21244@1612191983.285304:xen_platform_log xen platform: XEN|BUGCHECK: - R11 = 000000004343E808
> > 21244@1612191983.285420:xen_platform_log xen platform: XEN|BUGCHECK: - R12 = 0000000000000000
> > 21244@1612191983.285564:xen_platform_log xen platform: XEN|BUGCHECK: - R13 = 00000000F7964E50
> > 21244@1612191983.285680:xen_platform_log xen platform: XEN|BUGCHECK: - R14 = 00000000F64D5C20
> > 21244@1612191983.285796:xen_platform_log xen platform: XEN|BUGCHECK: - R15 = 00000000F7964E50
>
> I'm also confused by this - the pointer given for CONTEXT suggests this
> is a 64-bit kernel, yet none of the registers - including RIP and RSP -
> have non-zero upper 32 bits. Or is qemu truncating these values?
The logging is coming from the PV drivers (in https://xenbits.xen.org/gitweb/?p=pvdrivers/win/xenbus.git;a=blob;f=src/xen/bug_check.c). The truncated values may just be due to a 32-bit user process I guess.
Paul
>
> > 21244@1612191983.285888:xen_platform_log xen platform: XEN|BUGCHECK: STACK:
> > 21244@1612191983.286105:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343E810:
> (0000000000000000 000000004343E891 0000000000000002 00000000F75F08A0) ntoskrnl.exe + 0000000000485507
> > 21244@1612191983.286340:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343E8E0:
> (00000000F75F0805 000000004343EB80 00000000F6A62CC0 00000000F75F08A0) ntoskrnl.exe + 0000000000486468
> > 21244@1612191983.286547:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343EA20:
> (0000000000000000 0000000000000000 0000000000000000 0000000000000000) ntoskrnl.exe + 0000000000458CAE
> > 21244@1612191983.286755:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343EA90:
> (0000000000000000 0000000000000000 000000007DBED000 000000007DA00028) ntoskrnl.exe + 00000000001501A3
> > 21244@1612191983.286976:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE388:
> (00000000587D5673 0000000058F40000 0000000006002D2B 0000000000000000) 00007FFB5B3207CA
> > 21244@1612191983.287171:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE390:
> (0000000058F40000 0000000006002D2B 0000000000000000 00000000160C86D8) 00007FFB587D5673
> > 21244@1612191983.287390:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE398:
> (0000000006002D2B 0000000000000000 00000000160C86D8 0000000009ABE3E0) 00007FFB58F40000
> > 21244@1612191983.287584:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE3A0:
> (0000000000000000 00000000160C86D8 0000000009ABE3E0 000000008011E060) 00007FFB06002D2B
> > 21244@1612191983.287777:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE3A8:
> (00000000160C86D8 0000000009ABE3E0 000000008011E060 0000000009ABE4A0) 0000000000000000
> > 21244@1612191983.287898:xen_platform_log xen platform: XEN|BUGCHECK: <====
> >
> > The Windows guest is running winpv drivers 8.2.1.
> >
> > I'm not quite sure what else to examine or change at this point so any
> > guidance would be welcome.
>
> The hypervisor log (at maximum log levels) accompanying this might
> help some. And of course, if possible, trying on a newer Xen (ideally
> current master).
>
> Jan
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
2021-02-03 15:04 ` Paul Durrant
@ 2021-02-03 15:42 ` Jan Beulich
2021-02-03 15:48 ` Paul Durrant
0 siblings, 1 reply; 7+ messages in thread
From: Jan Beulich @ 2021-02-03 15:42 UTC (permalink / raw)
To: paul; +Cc: xen-devel, 'James Dingwall'
On 03.02.2021 16:04, Paul Durrant wrote:
>> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Jan Beulich
>> Sent: 03 February 2021 14:55
>>
>> On 01.02.2021 16:26, James Dingwall wrote:
>>> 21244@1612191983.282480:xen_platform_log xen platform: XEN|BUGCHECK: EXCEPTION (0000A824848948C2):
>>> 21244@1612191983.282617:xen_platform_log xen platform: XEN|BUGCHECK: CONTEXT (FFFFD0014343D580):
>>> 21244@1612191983.282717:xen_platform_log xen platform: XEN|BUGCHECK: - GS = 002B
>>> 21244@1612191983.282816:xen_platform_log xen platform: XEN|BUGCHECK: - FS = 0053
>>> 21244@1612191983.282914:xen_platform_log xen platform: XEN|BUGCHECK: - ES = 002B
>>> 21244@1612191983.283011:xen_platform_log xen platform: XEN|BUGCHECK: - DS = 002B
>>> 21244@1612191983.283127:xen_platform_log xen platform: XEN|BUGCHECK: - SS = 0018
>>> 21244@1612191983.283226:xen_platform_log xen platform: XEN|BUGCHECK: - CS = 0010
>>> 21244@1612191983.283332:xen_platform_log xen platform: XEN|BUGCHECK: - EFLAGS = 00000202
>>> 21244@1612191983.283444:xen_platform_log xen platform: XEN|BUGCHECK: - RDI = 00000000F64D5C20
>>> 21244@1612191983.283555:xen_platform_log xen platform: XEN|BUGCHECK: - RSI = 00000000F6367280
>>> 21244@1612191983.283666:xen_platform_log xen platform: XEN|BUGCHECK: - RBX = 000000008011E060
>>> 21244@1612191983.283810:xen_platform_log xen platform: XEN|BUGCHECK: - RDX = 00000000F64D5C20
>>> 21244@1612191983.283972:xen_platform_log xen platform: XEN|BUGCHECK: - RCX = 0000000000000199
>>> 21244@1612191983.284350:xen_platform_log xen platform: XEN|BUGCHECK: - RAX = 0000000000000004
>>> 21244@1612191983.284523:xen_platform_log xen platform: XEN|BUGCHECK: - RBP = 000000004343E891
>>> 21244@1612191983.284658:xen_platform_log xen platform: XEN|BUGCHECK: - RIP = 00000000A43C72C5
>>> 21244@1612191983.284842:xen_platform_log xen platform: XEN|BUGCHECK: - RSP = 000000004343DFA0
>>> 21244@1612191983.284959:xen_platform_log xen platform: XEN|BUGCHECK: - R8 = 0000000000000008
>>> 21244@1612191983.285073:xen_platform_log xen platform: XEN|BUGCHECK: - R9 = 000000000000000E
>>> 21244@1612191983.285188:xen_platform_log xen platform: XEN|BUGCHECK: - R10 = 0000000000000002
>>> 21244@1612191983.285304:xen_platform_log xen platform: XEN|BUGCHECK: - R11 = 000000004343E808
>>> 21244@1612191983.285420:xen_platform_log xen platform: XEN|BUGCHECK: - R12 = 0000000000000000
>>> 21244@1612191983.285564:xen_platform_log xen platform: XEN|BUGCHECK: - R13 = 00000000F7964E50
>>> 21244@1612191983.285680:xen_platform_log xen platform: XEN|BUGCHECK: - R14 = 00000000F64D5C20
>>> 21244@1612191983.285796:xen_platform_log xen platform: XEN|BUGCHECK: - R15 = 00000000F7964E50
>>
>> I'm also confused by this - the pointer given for CONTEXT suggests this
>> is a 64-bit kernel, yet none of the registers - including RIP and RSP -
>> have non-zero upper 32 bits. Or is qemu truncating these values?
>
> The logging is coming from the PV drivers (in https://xenbits.xen.org/gitweb/?p=pvdrivers/win/xenbus.git;a=blob;f=src/xen/bug_check.c). The truncated values may just be due to a 32-bit user process I guess.
Since you pointed me at the code and truncation inside a string
not likely being due to some user process, I went and looked:
The driver uses %016X, instead of e.g. converting to (PVOID)
and using %p like code elsewhere in the file does (presumably
because there's no really convenient way to print 64-bit values
in Windows, short of using their custom "%016I64X" format
specifier, and the absence of a uniform specifier allowing to
format pointer-sized integers independent of architecture).
Jan
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
2021-02-03 15:42 ` Jan Beulich
@ 2021-02-03 15:48 ` Paul Durrant
0 siblings, 0 replies; 7+ messages in thread
From: Paul Durrant @ 2021-02-03 15:48 UTC (permalink / raw)
To: 'Jan Beulich'; +Cc: xen-devel, 'James Dingwall'
> -----Original Message-----
> From: Jan Beulich <jbeulich@suse.com>
> Sent: 03 February 2021 15:43
> To: paul@xen.org
> Cc: xen-devel@lists.xenproject.org; 'James Dingwall' <james-xen@dingwall.me.uk>
> Subject: Re: VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
>
> On 03.02.2021 16:04, Paul Durrant wrote:
> >> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Jan Beulich
> >> Sent: 03 February 2021 14:55
> >>
> >> On 01.02.2021 16:26, James Dingwall wrote:
> >>> 21244@1612191983.282480:xen_platform_log xen platform: XEN|BUGCHECK: EXCEPTION (0000A824848948C2):
> >>> 21244@1612191983.282617:xen_platform_log xen platform: XEN|BUGCHECK: CONTEXT (FFFFD0014343D580):
> >>> 21244@1612191983.282717:xen_platform_log xen platform: XEN|BUGCHECK: - GS = 002B
> >>> 21244@1612191983.282816:xen_platform_log xen platform: XEN|BUGCHECK: - FS = 0053
> >>> 21244@1612191983.282914:xen_platform_log xen platform: XEN|BUGCHECK: - ES = 002B
> >>> 21244@1612191983.283011:xen_platform_log xen platform: XEN|BUGCHECK: - DS = 002B
> >>> 21244@1612191983.283127:xen_platform_log xen platform: XEN|BUGCHECK: - SS = 0018
> >>> 21244@1612191983.283226:xen_platform_log xen platform: XEN|BUGCHECK: - CS = 0010
> >>> 21244@1612191983.283332:xen_platform_log xen platform: XEN|BUGCHECK: - EFLAGS = 00000202
> >>> 21244@1612191983.283444:xen_platform_log xen platform: XEN|BUGCHECK: - RDI = 00000000F64D5C20
> >>> 21244@1612191983.283555:xen_platform_log xen platform: XEN|BUGCHECK: - RSI = 00000000F6367280
> >>> 21244@1612191983.283666:xen_platform_log xen platform: XEN|BUGCHECK: - RBX = 000000008011E060
> >>> 21244@1612191983.283810:xen_platform_log xen platform: XEN|BUGCHECK: - RDX = 00000000F64D5C20
> >>> 21244@1612191983.283972:xen_platform_log xen platform: XEN|BUGCHECK: - RCX = 0000000000000199
> >>> 21244@1612191983.284350:xen_platform_log xen platform: XEN|BUGCHECK: - RAX = 0000000000000004
> >>> 21244@1612191983.284523:xen_platform_log xen platform: XEN|BUGCHECK: - RBP = 000000004343E891
> >>> 21244@1612191983.284658:xen_platform_log xen platform: XEN|BUGCHECK: - RIP = 00000000A43C72C5
> >>> 21244@1612191983.284842:xen_platform_log xen platform: XEN|BUGCHECK: - RSP = 000000004343DFA0
> >>> 21244@1612191983.284959:xen_platform_log xen platform: XEN|BUGCHECK: - R8 = 0000000000000008
> >>> 21244@1612191983.285073:xen_platform_log xen platform: XEN|BUGCHECK: - R9 = 000000000000000E
> >>> 21244@1612191983.285188:xen_platform_log xen platform: XEN|BUGCHECK: - R10 = 0000000000000002
> >>> 21244@1612191983.285304:xen_platform_log xen platform: XEN|BUGCHECK: - R11 = 000000004343E808
> >>> 21244@1612191983.285420:xen_platform_log xen platform: XEN|BUGCHECK: - R12 = 0000000000000000
> >>> 21244@1612191983.285564:xen_platform_log xen platform: XEN|BUGCHECK: - R13 = 00000000F7964E50
> >>> 21244@1612191983.285680:xen_platform_log xen platform: XEN|BUGCHECK: - R14 = 00000000F64D5C20
> >>> 21244@1612191983.285796:xen_platform_log xen platform: XEN|BUGCHECK: - R15 = 00000000F7964E50
> >>
> >> I'm also confused by this - the pointer given for CONTEXT suggests this
> >> is a 64-bit kernel, yet none of the registers - including RIP and RSP -
> >> have non-zero upper 32 bits. Or is qemu truncating these values?
> >
> > The logging is coming from the PV drivers (in
> https://xenbits.xen.org/gitweb/?p=pvdrivers/win/xenbus.git;a=blob;f=src/xen/bug_check.c). The
> truncated values may just be due to a 32-bit user process I guess.
>
> Since you pointed me at the code and truncation inside a string
> not likely being due to some user process, I went and looked:
> The driver uses %016X, instead of e.g. converting to (PVOID)
> and using %p like code elsewhere in the file does (presumably
> because there's no really convenient way to print 64-bit values
> in Windows, short of using their custom "%016I64X" format
> specifier, and the absence of a uniform specifier allowing to
> format pointer-sized integers independent of architecture).
Oh yes, good point... Other places in the code use the %p trick. It should be changed.
Paul
>
> Jan
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
2021-02-03 14:55 ` Jan Beulich
2021-02-03 15:04 ` Paul Durrant
@ 2021-02-04 8:46 ` James Dingwall
2021-02-04 8:52 ` Jan Beulich
1 sibling, 1 reply; 7+ messages in thread
From: James Dingwall @ 2021-02-04 8:46 UTC (permalink / raw)
To: Jan Beulich; +Cc: James Dingwall, xen-devel
Hi Jan,
Thank you for your reply.
On Wed, Feb 03, 2021 at 03:55:07PM +0100, Jan Beulich wrote:
> On 01.02.2021 16:26, James Dingwall wrote:
> > I am building the xen 4.11 branch at
> > 310ab79875cb705cc2c7daddff412b5a4899f8c9 which includes commit
> > 3b5de119f0399cbe745502cb6ebd5e6633cc139c "86/msr: fix handling of
> > MSR_IA32_PERF_{STATUS/CTL}". I think this should address this error
> > recorded in xen's dmesg:
> >
> > (XEN) d11v0 VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
>
> It seems to me that you imply some information here which might
> better be spelled out. As it stands I do not see the immediate
> connection between the cited commit and the crash. C0000096 is
> STATUS_PRIVILEGED_INSTRUCTION, which to me ought to be impossible
> for code running in ring 0. Of course I may simply not know enough
> about modern Windows' internals to understand the connection.
Searching for "VIRIDIAN CRASH: 3b" led me to this thread and then to the commit based on the commit log message.
https://patchwork.kernel.org/project/xen-devel/patch/20201007102032.98565-1-roger.pau@citrix.com/
I have naively assumed that the RCX register indicated MSR_IA32_PERF_CTL based on:
#define MSR_IA32_PERF_CTL 0x00000199
I've added this patch:
diff --git a/xen/arch/x86/msr.c b/xen/arch/x86/msr.c
index 99c848ff41..7a764907d5 100644
--- a/xen/arch/x86/msr.c
+++ b/xen/arch/x86/msr.c
@@ -232,12 +232,16 @@ int guest_rdmsr(const struct vcpu *v, uint32_t msr, uint64_t *val)
*/
case MSR_IA32_PERF_STATUS:
case MSR_IA32_PERF_CTL:
- if ( !(cp->x86_vendor & (X86_VENDOR_INTEL | X86_VENDOR_CENTAUR)) )
+ if ( !(cp->x86_vendor & (X86_VENDOR_INTEL | X86_VENDOR_CENTAUR)) ) {
+ printk(KERN_DEBUG "JKD: MSR %#x FAULT1: %#x & %#x\n", msr, cp->x86_vendor, (X86_VENDOR_INTEL | X86_VENDOR_CENTAUR));
+
goto gp_fault;
+ }
*val = 0;
if ( likely(!is_cpufreq_controller(d)) || rdmsr_safe(msr, *val) == 0 )
break;
+ printk(KERN_DEBUG "JKD: MSR FAULT2\n");
goto gp_fault;
/*
and now in the hypervisor log when the domain crashes:
(XEN) JKD: MSR 0x199 FAULT1: 0 & 0x2
(XEN) d11v0 VIRIDIAN CRASH: 3b c0000096 1146d2c5 6346d580 0
(XEN) avc: denied { reset } for domid=11 scontext=system_u:system_r:domU_t tcontext=system_u:system_r:domU_t_self tclass=event
I'm not sure what is expected in cp->x86_vendor but this is running on an Intel CPU so I would have thought 0x1 based on
#define X86_VENDOR_INTEL (1 << 0)
I have also booted with flask=disabled to to eliminate the reported avc denial as the cause.
>
> > I have removed `viridian = [..]` from the xen config nut still get this
> > reliably when launching PassMark Performance Test and it is collecting
> > CPU information.
> >
> > This is recorded in the domain qemu-dm log:
> >
> > 21244@1612191983.279616:xen_platform_log xen platform: XEN|BUGCHECK: ====>
> > 21244@1612191983.279819:xen_platform_log xen platform: XEN|BUGCHECK: SYSTEM_SERVICE_EXCEPTION: 00000000C0000096 FFFFF800A43C72C5 FFFFD0014343D580 0000000000000000
> > 21244@1612191983.279959:xen_platform_log xen platform: XEN|BUGCHECK: EXCEPTION (FFFFF800A43C72C5):
> > 21244@1612191983.280075:xen_platform_log xen platform: XEN|BUGCHECK: - Code = C148320F
> > 21244@1612191983.280205:xen_platform_log xen platform: XEN|BUGCHECK: - Flags = 0B4820E2
> > 21244@1612191983.280346:xen_platform_log xen platform: XEN|BUGCHECK: - Address = 0000A824948D4800
> > 21244@1612191983.280504:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[0] = 8B00000769850F07
> > 21244@1612191983.280633:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[1] = 46B70F4024448906
> > 21244@1612191983.280754:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[2] = 0F44442444896604
> > 21244@1612191983.280876:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[3] = E983C88B410646B6
> > 21244@1612191983.281012:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[4] = 0D7401E9831E7401
> > 21244@1612191983.281172:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[5] = 54B70F217502F983
> > 21244@1612191983.281304:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[6] = 54B70F15EBED4024
> > 21244@1612191983.281426:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[7] = EBC0B70FED664024
> > 21244@1612191983.281547:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[8] = 0FEC402454B70F09
> > 21244@1612191983.281668:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[9] = 448B42244489C0B6
> > 21244@1612191983.281809:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[10] = 2444B70F06894024
> > 21244@1612191983.281932:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[11] = 4688440446896644
> > 21244@1612191983.282052:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[12] = 0000073846C74906
> > 21244@1612191983.282185:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[13] = F8830000070AE900
> > 21244@1612191983.282340:xen_platform_log xen platform: XEN|BUGCHECK: - Parameter[14] = 8B000006F9850F07
> > 21244@1612191983.282480:xen_platform_log xen platform: XEN|BUGCHECK: EXCEPTION (0000A824848948C2):
> > 21244@1612191983.282617:xen_platform_log xen platform: XEN|BUGCHECK: CONTEXT (FFFFD0014343D580):
> > 21244@1612191983.282717:xen_platform_log xen platform: XEN|BUGCHECK: - GS = 002B
> > 21244@1612191983.282816:xen_platform_log xen platform: XEN|BUGCHECK: - FS = 0053
> > 21244@1612191983.282914:xen_platform_log xen platform: XEN|BUGCHECK: - ES = 002B
> > 21244@1612191983.283011:xen_platform_log xen platform: XEN|BUGCHECK: - DS = 002B
> > 21244@1612191983.283127:xen_platform_log xen platform: XEN|BUGCHECK: - SS = 0018
> > 21244@1612191983.283226:xen_platform_log xen platform: XEN|BUGCHECK: - CS = 0010
> > 21244@1612191983.283332:xen_platform_log xen platform: XEN|BUGCHECK: - EFLAGS = 00000202
> > 21244@1612191983.283444:xen_platform_log xen platform: XEN|BUGCHECK: - RDI = 00000000F64D5C20
> > 21244@1612191983.283555:xen_platform_log xen platform: XEN|BUGCHECK: - RSI = 00000000F6367280
> > 21244@1612191983.283666:xen_platform_log xen platform: XEN|BUGCHECK: - RBX = 000000008011E060
> > 21244@1612191983.283810:xen_platform_log xen platform: XEN|BUGCHECK: - RDX = 00000000F64D5C20
> > 21244@1612191983.283972:xen_platform_log xen platform: XEN|BUGCHECK: - RCX = 0000000000000199
> > 21244@1612191983.284350:xen_platform_log xen platform: XEN|BUGCHECK: - RAX = 0000000000000004
> > 21244@1612191983.284523:xen_platform_log xen platform: XEN|BUGCHECK: - RBP = 000000004343E891
> > 21244@1612191983.284658:xen_platform_log xen platform: XEN|BUGCHECK: - RIP = 00000000A43C72C5
> > 21244@1612191983.284842:xen_platform_log xen platform: XEN|BUGCHECK: - RSP = 000000004343DFA0
> > 21244@1612191983.284959:xen_platform_log xen platform: XEN|BUGCHECK: - R8 = 0000000000000008
> > 21244@1612191983.285073:xen_platform_log xen platform: XEN|BUGCHECK: - R9 = 000000000000000E
> > 21244@1612191983.285188:xen_platform_log xen platform: XEN|BUGCHECK: - R10 = 0000000000000002
> > 21244@1612191983.285304:xen_platform_log xen platform: XEN|BUGCHECK: - R11 = 000000004343E808
> > 21244@1612191983.285420:xen_platform_log xen platform: XEN|BUGCHECK: - R12 = 0000000000000000
> > 21244@1612191983.285564:xen_platform_log xen platform: XEN|BUGCHECK: - R13 = 00000000F7964E50
> > 21244@1612191983.285680:xen_platform_log xen platform: XEN|BUGCHECK: - R14 = 00000000F64D5C20
> > 21244@1612191983.285796:xen_platform_log xen platform: XEN|BUGCHECK: - R15 = 00000000F7964E50
>
> I'm also confused by this - the pointer given for CONTEXT suggests this
> is a 64-bit kernel, yet none of the registers - including RIP and RSP -
> have non-zero upper 32 bits. Or is qemu truncating these values?
>
> > 21244@1612191983.285888:xen_platform_log xen platform: XEN|BUGCHECK: STACK:
> > 21244@1612191983.286105:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343E810: (0000000000000000 000000004343E891 0000000000000002 00000000F75F08A0) ntoskrnl.exe + 0000000000485507
> > 21244@1612191983.286340:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343E8E0: (00000000F75F0805 000000004343EB80 00000000F6A62CC0 00000000F75F08A0) ntoskrnl.exe + 0000000000486468
> > 21244@1612191983.286547:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343EA20: (0000000000000000 0000000000000000 0000000000000000 0000000000000000) ntoskrnl.exe + 0000000000458CAE
> > 21244@1612191983.286755:xen_platform_log xen platform: XEN|BUGCHECK: 000000004343EA90: (0000000000000000 0000000000000000 000000007DBED000 000000007DA00028) ntoskrnl.exe + 00000000001501A3
> > 21244@1612191983.286976:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE388: (00000000587D5673 0000000058F40000 0000000006002D2B 0000000000000000) 00007FFB5B3207CA
> > 21244@1612191983.287171:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE390: (0000000058F40000 0000000006002D2B 0000000000000000 00000000160C86D8) 00007FFB587D5673
> > 21244@1612191983.287390:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE398: (0000000006002D2B 0000000000000000 00000000160C86D8 0000000009ABE3E0) 00007FFB58F40000
> > 21244@1612191983.287584:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE3A0: (0000000000000000 00000000160C86D8 0000000009ABE3E0 000000008011E060) 00007FFB06002D2B
> > 21244@1612191983.287777:xen_platform_log xen platform: XEN|BUGCHECK: 0000000009ABE3A8: (00000000160C86D8 0000000009ABE3E0 000000008011E060 0000000009ABE4A0) 0000000000000000
> > 21244@1612191983.287898:xen_platform_log xen platform: XEN|BUGCHECK: <====
> >
> > The Windows guest is running winpv drivers 8.2.1.
> >
> > I'm not quite sure what else to examine or change at this point so any
> > guidance would be welcome.
>
> The hypervisor log (at maximum log levels) accompanying this might
> help some. And of course, if possible, trying on a newer Xen (ideally
> current master).
We have a separate upgrade to 4.14.1 in progress and I will test on that too.
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
2021-02-04 8:46 ` James Dingwall
@ 2021-02-04 8:52 ` Jan Beulich
0 siblings, 0 replies; 7+ messages in thread
From: Jan Beulich @ 2021-02-04 8:52 UTC (permalink / raw)
To: James Dingwall; +Cc: James Dingwall, xen-devel
On 04.02.2021 09:46, James Dingwall wrote:
> On Wed, Feb 03, 2021 at 03:55:07PM +0100, Jan Beulich wrote:
>> On 01.02.2021 16:26, James Dingwall wrote:
>>> I am building the xen 4.11 branch at
>>> 310ab79875cb705cc2c7daddff412b5a4899f8c9 which includes commit
>>> 3b5de119f0399cbe745502cb6ebd5e6633cc139c "86/msr: fix handling of
>>> MSR_IA32_PERF_{STATUS/CTL}". I think this should address this error
>>> recorded in xen's dmesg:
>>>
>>> (XEN) d11v0 VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0
>>
>> It seems to me that you imply some information here which might
>> better be spelled out. As it stands I do not see the immediate
>> connection between the cited commit and the crash. C0000096 is
>> STATUS_PRIVILEGED_INSTRUCTION, which to me ought to be impossible
>> for code running in ring 0. Of course I may simply not know enough
>> about modern Windows' internals to understand the connection.
>
> Searching for "VIRIDIAN CRASH: 3b" led me to this thread and then to the commit based on the commit log message.
>
> https://patchwork.kernel.org/project/xen-devel/patch/20201007102032.98565-1-roger.pau@citrix.com/
>
> I have naively assumed that the RCX register indicated MSR_IA32_PERF_CTL based on:
>
> #define MSR_IA32_PERF_CTL 0x00000199
>
> I've added this patch:
>
> diff --git a/xen/arch/x86/msr.c b/xen/arch/x86/msr.c
> index 99c848ff41..7a764907d5 100644
> --- a/xen/arch/x86/msr.c
> +++ b/xen/arch/x86/msr.c
> @@ -232,12 +232,16 @@ int guest_rdmsr(const struct vcpu *v, uint32_t msr, uint64_t *val)
> */
> case MSR_IA32_PERF_STATUS:
> case MSR_IA32_PERF_CTL:
> - if ( !(cp->x86_vendor & (X86_VENDOR_INTEL | X86_VENDOR_CENTAUR)) )
> + if ( !(cp->x86_vendor & (X86_VENDOR_INTEL | X86_VENDOR_CENTAUR)) ) {
> + printk(KERN_DEBUG "JKD: MSR %#x FAULT1: %#x & %#x\n", msr, cp->x86_vendor, (X86_VENDOR_INTEL | X86_VENDOR_CENTAUR));
> +
> goto gp_fault;
> + }
>
> *val = 0;
> if ( likely(!is_cpufreq_controller(d)) || rdmsr_safe(msr, *val) == 0 )
> break;
> + printk(KERN_DEBUG "JKD: MSR FAULT2\n");
> goto gp_fault;
>
> /*
>
> and now in the hypervisor log when the domain crashes:
>
> (XEN) JKD: MSR 0x199 FAULT1: 0 & 0x2
> (XEN) d11v0 VIRIDIAN CRASH: 3b c0000096 1146d2c5 6346d580 0
> (XEN) avc: denied { reset } for domid=11 scontext=system_u:system_r:domU_t tcontext=system_u:system_r:domU_t_self tclass=event
>
> I'm not sure what is expected in cp->x86_vendor but this is running on an Intel CPU so I would have thought 0x1 based on
>
> #define X86_VENDOR_INTEL (1 << 0)
This is the problem - a bad backport. Therefore ...
>> The hypervisor log (at maximum log levels) accompanying this might
>> help some. And of course, if possible, trying on a newer Xen (ideally
>> current master).
>
> We have a separate upgrade to 4.14.1 in progress and I will test on that too.
I'm sure you'll find this work there. I'll make a patch for the affected
older tree(s).
Jan
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-02-04 8:52 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-01 15:26 VIRIDIAN CRASH: 3b c0000096 75b12c5 9e7f1580 0 James Dingwall
2021-02-03 14:55 ` Jan Beulich
2021-02-03 15:04 ` Paul Durrant
2021-02-03 15:42 ` Jan Beulich
2021-02-03 15:48 ` Paul Durrant
2021-02-04 8:46 ` James Dingwall
2021-02-04 8:52 ` Jan Beulich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).