AMD crash on restore. bug in MTRR restore. 4.0.2-rc2-pre

AMD crash on restore. bug in MTRR restore. 4.0.2-rc2-pre

[James Harper]

The mtrr save code on my AMD system reads the first MTRR as
0x00000000f0000000,0x000000fff8000800. When it tries to restore in
mtrr_var_range_msr_set, 0x000000fff8000800 is rejected as invalid and
not restored.

Commenting out the check against the mask, as well as Tim's patch to fix
the segment registers makes the output of xen-hvmctx match before the
save and after the restore.

Windows still crashes on restore though so I don't know if this has
anything to do with it.

James

RE: AMD crash on restore. bug in MTRR restore. 4.0.2-rc2-pre

[James Harper]

Additionally, WinDbg tells me that MSR's 175, and 176 are wiped out by
save/restore:

immediately before suspend
kd> rdmsr 174
msr[174] = 00000000`00000008
kd> rdmsr 175
msr[175] = 00000000`8039e000
kd> rdmsr 176
msr[176] = 00000000`816a4950

immediately after restore
kd> rdmsr 174
msr[174] = 00000000`00000000
kd> rdmsr 175
msr[175] = 00000000`00000000
kd> rdmsr 176
msr[176] = 00000000`00000000

although you're never too sure with WinDbg...

James

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of James Harper
> Sent: Wednesday, 26 January 2011 17:19
> To: xen-devel@lists.xensource.com
> Cc: Tim Deegan
> Subject: [Xen-devel] AMD crash on restore. bug in MTRR restore.
4.0.2-rc2-pre
> 
> The mtrr save code on my AMD system reads the first MTRR as
> 0x00000000f0000000,0x000000fff8000800. When it tries to restore in
> mtrr_var_range_msr_set, 0x000000fff8000800 is rejected as invalid and
> not restored.
> 
> Commenting out the check against the mask, as well as Tim's patch to
fix
> the segment registers makes the output of xen-hvmctx match before the
> save and after the restore.
> 
> Windows still crashes on restore though so I don't know if this has
> anything to do with it.
> 
> James
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: AMD crash on restore. bug in MTRR restore.4.0.2-rc2-pre

[James Harper]

By saving the sysenter msrs in my PV drivers before the suspend
hypercall and restoring them after, save/restore now works fine.

Does nobody else see a save/restore failure on AMD as a blocker to
4.0.2? It it something particular to my exact AMD CPU and OS (Windows
2008 x32)?

James

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of James Harper
> Sent: Thursday, 27 January 2011 13:32
> To: xen-devel@lists.xensource.com
> Cc: Tim Deegan
> Subject: RE: [Xen-devel] AMD crash on restore. bug in MTRR
restore.4.0.2-rc2-
> pre
> 
> Additionally, WinDbg tells me that MSR's 175, and 176 are wiped out by
> save/restore:
> 
> immediately before suspend
> kd> rdmsr 174
> msr[174] = 00000000`00000008
> kd> rdmsr 175
> msr[175] = 00000000`8039e000
> kd> rdmsr 176
> msr[176] = 00000000`816a4950
> 
> immediately after restore
> kd> rdmsr 174
> msr[174] = 00000000`00000000
> kd> rdmsr 175
> msr[175] = 00000000`00000000
> kd> rdmsr 176
> msr[176] = 00000000`00000000
> 
> although you're never too sure with WinDbg...
> 
> James
> 
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> > bounces@lists.xensource.com] On Behalf Of James Harper
> > Sent: Wednesday, 26 January 2011 17:19
> > To: xen-devel@lists.xensource.com
> > Cc: Tim Deegan
> > Subject: [Xen-devel] AMD crash on restore. bug in MTRR restore.
> 4.0.2-rc2-pre
> >
> > The mtrr save code on my AMD system reads the first MTRR as
> > 0x00000000f0000000,0x000000fff8000800. When it tries to restore in
> > mtrr_var_range_msr_set, 0x000000fff8000800 is rejected as invalid
and
> > not restored.
> >
> > Commenting out the check against the mask, as well as Tim's patch to
> fix
> > the segment registers makes the output of xen-hvmctx match before
the
> > save and after the restore.
> >
> > Windows still crashes on restore though so I don't know if this has
> > anything to do with it.
> >
> > James
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: AMD crash on restore. bug in MTRR restore.4.0.2-rc2-pre

[Jan Beulich]

>>> On 27.01.11 at 04:49, "James Harper" <james.harper@bendigoit.com.au> wrote:
> By saving the sysenter msrs in my PV drivers before the suspend
> hypercall and restoring them after, save/restore now works fine.
> 
> Does nobody else see a save/restore failure on AMD as a blocker to
> 4.0.2? It it something particular to my exact AMD CPU and OS (Windows
> 2008 x32)?

The 32-bit-ness of your Windows may matter: Looking at
xen/arch/hvm/svm/svm.c, I cannot see how
v->arch.hvm_svm.guest_sysenter_* would ever get set
in this case (given that the intercept for the respective MSRs
gets disabled in svm_update_guest_efer()), yet
svm_vmcb_save() reads this field unconditionally, whereas I
would think v->arch.hvm_svm.vmcb->sysenter_* should be
read/written in that case.

That code seems to go back to c/s 19856, fixing a boot
problem introduced in c/s 19648, which is the one that
removed the reads of the VMCB fields. Christoph?

Jan

>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of James Harper
>> Sent: Thursday, 27 January 2011 13:32
>> To: xen-devel@lists.xensource.com 
>> Cc: Tim Deegan
>> Subject: RE: [Xen-devel] AMD crash on restore. bug in MTRR
> restore.4.0.2-rc2-
>> pre
>> 
>> Additionally, WinDbg tells me that MSR's 175, and 176 are wiped out by
>> save/restore:
>> 
>> immediately before suspend
>> kd> rdmsr 174
>> msr[174] = 00000000`00000008
>> kd> rdmsr 175
>> msr[175] = 00000000`8039e000
>> kd> rdmsr 176
>> msr[176] = 00000000`816a4950
>> 
>> immediately after restore
>> kd> rdmsr 174
>> msr[174] = 00000000`00000000
>> kd> rdmsr 175
>> msr[175] = 00000000`00000000
>> kd> rdmsr 176
>> msr[176] = 00000000`00000000
>> 
>> although you're never too sure with WinDbg...
>> 
>> James

RE: AMD crash on restore. bug in MTRR restore.4.0.2-rc2-pre

[Wei Huang]

[-- Attachment #1: Type: text/plain, Size: 3031 bytes --]

Hi James,

Could you try the attached patch? The issue is because we don't
intercept SYSENTER_xx MSRs under 32bit mode (guest mode). As a result,
hvm_svm.guest_sysenter_xx contain invalid values. So save/restore will
eventually fail by using these values. This patch solves the problem.

If it works, please let me know and I will submit a formal one.

Best,
-Wei

On Wed, 2011-01-26 at 21:49 -0600, James Harper wrote:
> By saving the sysenter msrs in my PV drivers before the suspend
> hypercall and restoring them after, save/restore now works fine.
> 
> Does nobody else see a save/restore failure on AMD as a blocker to
> 4.0.2? It it something particular to my exact AMD CPU and OS (Windows
> 2008 x32)?
> 
> James
> 
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> > bounces@lists.xensource.com] On Behalf Of James Harper
> > Sent: Thursday, 27 January 2011 13:32
> > To: xen-devel@lists.xensource.com
> > Cc: Tim Deegan
> > Subject: RE: [Xen-devel] AMD crash on restore. bug in MTRR
> restore.4.0.2-rc2-
> > pre
> > 
> > Additionally, WinDbg tells me that MSR's 175, and 176 are wiped out by
> > save/restore:
> > 
> > immediately before suspend
> > kd> rdmsr 174
> > msr[174] = 00000000`00000008
> > kd> rdmsr 175
> > msr[175] = 00000000`8039e000
> > kd> rdmsr 176
> > msr[176] = 00000000`816a4950
> > 
> > immediately after restore
> > kd> rdmsr 174
> > msr[174] = 00000000`00000000
> > kd> rdmsr 175
> > msr[175] = 00000000`00000000
> > kd> rdmsr 176
> > msr[176] = 00000000`00000000
> > 
> > although you're never too sure with WinDbg...
> > 
> > James
> > 
> > > -----Original Message-----
> > > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> > > bounces@lists.xensource.com] On Behalf Of James Harper
> > > Sent: Wednesday, 26 January 2011 17:19
> > > To: xen-devel@lists.xensource.com
> > > Cc: Tim Deegan
> > > Subject: [Xen-devel] AMD crash on restore. bug in MTRR restore.
> > 4.0.2-rc2-pre
> > >
> > > The mtrr save code on my AMD system reads the first MTRR as
> > > 0x00000000f0000000,0x000000fff8000800. When it tries to restore in
> > > mtrr_var_range_msr_set, 0x000000fff8000800 is rejected as invalid
> and
> > > not restored.
> > >
> > > Commenting out the check against the mask, as well as Tim's patch to
> > fix
> > > the segment registers makes the output of xen-hvmctx match before
> the
> > > save and after the restore.
> > >
> > > Windows still crashes on restore though so I don't know if this has
> > > anything to do with it.
> > >
> > > James
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 


[-- Attachment #2: amd_fix_sysenter_msr.txt --]
[-- Type: text/plain, Size: 1605 bytes --]

diff -r e94cd141c7af xen/arch/x86/hvm/svm/svm.c
--- a/xen/arch/x86/hvm/svm/svm.c	Wed Jan 26 11:58:02 2011 -0600
+++ b/xen/arch/x86/hvm/svm/svm.c	Sat Jan 29 00:46:58 2011 -0600
@@ -158,15 +158,26 @@
 static int svm_vmcb_save(struct vcpu *v, struct hvm_hw_cpu *c)
 {
     struct vmcb_struct *vmcb = v->arch.hvm_svm.vmcb;
+    uint64_t cs, esp, eip;
 
     c->cr0 = v->arch.hvm_vcpu.guest_cr[0];
     c->cr2 = v->arch.hvm_vcpu.guest_cr[2];
     c->cr3 = v->arch.hvm_vcpu.guest_cr[3];
     c->cr4 = v->arch.hvm_vcpu.guest_cr[4];
 
-    c->sysenter_cs = v->arch.hvm_svm.guest_sysenter_cs;
-    c->sysenter_esp = v->arch.hvm_svm.guest_sysenter_esp;
-    c->sysenter_eip = v->arch.hvm_svm.guest_sysenter_eip;
+    cs = vmcb->sysenter_cs;
+    esp = vmcb->sysenter_esp;
+    eip = vmcb->sysenter_eip;
+
+    printk("cs=0x%llx, esp=0x%llx, eip=0x%llx\n", (unsigned long long)cs,
+           (unsigned long long)esp, (unsigned long long)eip);
+
+    //c->sysenter_cs = v->arch.hvm_svm.guest_sysenter_cs;
+    //c->sysenter_esp = v->arch.hvm_svm.guest_sysenter_esp;
+    //c->sysenter_eip = v->arch.hvm_svm.guest_sysenter_eip;
+    c->sysenter_cs = cs;
+    c->sysenter_esp = esp;
+    c->sysenter_eip = eip;
 
     c->pending_event = 0;
     c->error_code = 0;
@@ -228,6 +239,10 @@
     v->arch.hvm_svm.guest_sysenter_esp = c->sysenter_esp;
     v->arch.hvm_svm.guest_sysenter_eip = c->sysenter_eip;
 
+    vmcb->sysenter_cs = c->sysenter_cs;
+    vmcb->sysenter_esp = c->sysenter_esp;
+    vmcb->sysenter_eip = c->sysenter_eip;
+
     if ( paging_mode_hap(v->domain) )
     {
         vmcb_set_np_enable(vmcb, 1);

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel