All of lore.kernel.org
 help / color / mirror / Atom feed
* fxsave, fnsave, ltr hang for guest OS.
@ 2010-11-02 18:13 alarson
  2010-11-02 20:16 ` Keir Fraser
  0 siblings, 1 reply; 13+ messages in thread
From: alarson @ 2010-11-02 18:13 UTC (permalink / raw)
  To: xen-devel

Someone suggested that the following query would have been more properly 
addressed to xen-devel than xen-users.

The short version is that as soon as I turn on paging (CR0|=PG|PE), I 
loose the ability to do an fxsave, ltr, and several other things in my HVM 
client.

It would appear that Xen is making an assumption about the paging 
environment that my OS is not satisfying.

----- Forwarded by Aaron Larson/ddci on 11/02/2010 12:55 PM -----

Aaron Larson/ddci wrote on 10/28/2010 03:57:59 PM:

> I am attempting to get a 32-bit protected mode OS running under Xen as
> an HVM guest, but it is hanging on the execution of fxsave (and
> fnsave) and ltr (load task register) instructions.  The OS is known to
> run on a variety of X86 machines from 386 through Core2 DOU, and also
> VMware workstation and userland qemu.  I saw some discussion of Xen
> fxsave issue, most notably
> http://lists.xensource.com/archives/html/xen-devel/2008-07/msg00874.html
> but my I'm not trying to step, just execute so my interpretation is my
> problem is different (although I certainly could be wrong).  I didn't
> see any issues with fnsave or ltr.
> 
> I'm using Xen under OpenSuse 11.1 (Xen 3.3.1), although I've also
> tried OpenSuse 11.3 (Xen 4.0.0) with identical results.
> 
> Any assistance would be greatly appreciated.
> 
> The code snippet for the fxsave case:
> 
>     diagMsg << "\nSaving FP context @" << (DWORD) 
> (&initialFPcontext) << "...";
>     // Ensure buffer is writeable.
>     ((BYTE*)(&initialFPcontext))[0]=0;
>     ((BYTE*)(&initialFPcontext))[511]=0;
>     // Initialze the FPU.
>     diagMsg << "fninit...";
>     asm volatile ("fninit");
> 
>     if (simdContextSavingSupported())
>     {
>       diagMsg << "fxSave...";
>       asm volatile ("fxsave (%0)"::"r"(&initialFPcontext));
>       diagMsg << "done\n";
>     ...
> 
> The results are:
> 
>    Saving FP context @C082D470...fninit...fxSave...
> 
> An objdump -d of the binary yields:
> 
>    c0811ffb:    0f ae 00                fxsave (%eax)
> 
> "xm list" shows the VM continuing to get CPU cycles:
> 
>   # xm list
>   Name                    ID   Mem VCPUs      State   Time(s)
>   Domain-0                 0  3855     2     r-----    367.4
>   myos                    59    16     1     r-----     17.7
> 
> GDB attached to the running VM shows that the processor is executing
> the fxsave instruction and eax is as expected:
> 
>    (gdb) p/x $eip
>    $1 = 0xc0811ffb
>    (gdb) p/x $eax
>    $2 = 0xc082d470
> 
> The /var/log/xen/qemu-dm-myos.log:
>   domid: 59
>   qemu: the number of cpus is 1
>   Watching /local/domain/0/device-model/59/logdirty/next-active
>   Watching /local/domain/0/device-model/59/command
>   xs_read(): vncpasswd get error. /vm/9ec94cd1-20c5-07da-a1d4-
> fdd500b8aef6/vncpasswd.
>   qemu_map_cache_init nr_buckets = 4000 size 196608
>   /usr/src/packages/BUILD/xen-3.3.1-testing/tools/ioemu-
> dir/hw/xen_blktap.c:714: Init blktap pipes
>   shared page at pfn ffe
>   buffered io page at pfn ffc
>   Time offset set 0
>   Register xen platform.
>   Done register platform.
>   medium change watch on `hdc' (index: 0): /home/alarson/pci-scanner.iso
>   I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
> 
> The fxsave instruction lists the following faults, and I've checked
> them all.
> 
>   Protected Mode faults:
>     #GP(0) illegal memory operand effective address in segment
>     register. 
>     #SS(0) illegal address in SS segment
>     #PF(fault-code)
> 
>      The instruction does not have a segment prefix, and the
>      preceding code ensures the buffer is writable.
> 
>     #NM if CR0[ts] or CR0[em]=1 (bits 3 or 2).
> 
>      The CR0 is C0000033
> 
>     #UD If CPUID.01H:EDX.FXSR[bit 24] = 0.
> 
>       CPUID.01H is EAX=0001067A EBX=00020800 ECX=80082201 EDX=0789FBBF
> 
>       EDX[24] is 1
> 
>     #UD If the LOCK prefix is used
> 
>       Instruction has no prefixes.
> 
>     #AC  for alignment
> 
>       FP context address is 0xC082D470 (16 byte aligned)
> 
> cat /proc/cpuinfo
> processor       : 0
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 23
> model name      : Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz
> stepping        : 10
> cpu MHz         : 2992.582
> cache size      : 6144 KB
> fdiv_bug        : no
> hlt_bug         : no
> f00f_bug        : no
> coma_bug        : no
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 13
> wp              : yes
> flags           : fpu de tsc msr pae cx8 apic mtrr cmov pat clflush 
> acpi mmx fxsr sse sse2 ss ht nx constant_tsc pni est ssse3 sse4_1
> bogomips        : 5987.06
> clflush size    : 64
> power management:
> 
> processor       : 1
> [same as for processor 0]
> 
> An xm list -l yields the following for the domain:
> (domain
>     (domid 58)
>     (bootloader )
>     (on_crash preserve)
>     (uuid ca61601b-4728-d39c-2a4c-4fe0ced6954e)
>     (bootloader_args )
>     (vcpus 1)
>     (name myos)
>     (cpus (()))
>     (on_reboot restart)
>     (on_poweroff destroy)
>     (maxmem 16)
>     (memory 16)
>     (shadow_memory 2)
>     (features )
>     (on_xend_start ignore)
>     (on_xend_stop ignore)
>     (start_time 1288284595.39)
>     (cpu_time 319.894000271)
>     (online_vcpus 1)
>     (image
>         (hvm
>             (kernel )
>             (args 'VNC_VIEWER=127.0.0.2:5514 ')
>             (timer_mode 0)
>             (vnc 1)
>             (nographic 0)
>             (hpet 0)
>             (rtc_timeoffset 0)
>             (loader /usr/lib/xen/boot/hvmloader)
>             (pci ())
>             (guest_os_type default)
>             (apic 1)
>             (sdl 0)
>             (extid 0)
>             (hap 0)
>             (monitor 0)
>             (device_model /usr/lib/xen/bin/qemu-dm)
>             (boot d)
>             (display localhost:10.0)
>             (pae 0)
>             (stdvga 0)
>             (usb 0)
>             (xauthority /root/.Xauthority)
>             (isa 0)
>             (acpi 0)
>             (vncunused 1)
>             (localtime 0)
>             (notes (SUSPEND_CANCEL 1))
>         )
>     )
>     (status 1)
>     (state --p---)
>     (store_mfn 4093)
>     (device
>         (console
>             (protocol vt100)
>             (location 3)
>             (uuid 9b2b7ffe-8520-8a0b-6a0a-8864bcc3c8c4)
>         )
>     )
>     (device
>         (vbd
>             (uuid 65beb1fb-b726-a262-8837-6a99d24a45e3)
>             (bootable 1)
>             (dev hdc:cdrom)
>             (uname file:/home/alarson/pci-scanner.iso)
>             (mode r)
>             (backend 0)
>             (bootable 1)
>             (VDI )
>         )
>     )
>     (device
>         (vfb
>             (vncunused 1)
>             (type vnc)
>             (uuid 7cb2e9d4-8376-8f6b-7851-6bf7c4058914)
>             (location 127.0.0.1:5900)
>         )
>     )
> )


This message is intended only for the use of the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the sender of this E-Mail by return E-Mail or by telephone. 

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-02 18:13 fxsave, fnsave, ltr hang for guest OS alarson
@ 2010-11-02 20:16 ` Keir Fraser
  2010-11-03  1:10   ` alarson
  0 siblings, 1 reply; 13+ messages in thread
From: Keir Fraser @ 2010-11-02 20:16 UTC (permalink / raw)
  To: alarson, xen-devel

Since the guest is allegedly continually running, you might try the 'd'
debug key (hit d on Xen serial console) and get execution backtrace for
every physical CPU, and see what the hypervisor is spending its time doing
for your guest on that instruction. Or add tracing to Xen to see where time
is going. Eg. Could be endlessly trying to emulate fxsave in
arch/x86/x86_emulate/x86_emulate.c. Or could be taking endless faults via
arch/x86/hvm/vmx/vmx.c:vmx_fpu_dirty_intercept().

In short, you need to do some more spadework.

 -- Keir

On 02/11/2010 18:13, "alarson@ddci.com" <alarson@ddci.com> wrote:

> Someone suggested that the following query would have been more properly
> addressed to xen-devel than xen-users.
> 
> The short version is that as soon as I turn on paging (CR0|=PG|PE), I
> loose the ability to do an fxsave, ltr, and several other things in my HVM
> client.
> 
> It would appear that Xen is making an assumption about the paging
> environment that my OS is not satisfying.
> 
> ----- Forwarded by Aaron Larson/ddci on 11/02/2010 12:55 PM -----
> 
> Aaron Larson/ddci wrote on 10/28/2010 03:57:59 PM:
> 
>> I am attempting to get a 32-bit protected mode OS running under Xen as
>> an HVM guest, but it is hanging on the execution of fxsave (and
>> fnsave) and ltr (load task register) instructions.  The OS is known to
>> run on a variety of X86 machines from 386 through Core2 DOU, and also
>> VMware workstation and userland qemu.  I saw some discussion of Xen
>> fxsave issue, most notably
>> http://lists.xensource.com/archives/html/xen-devel/2008-07/msg00874.html
>> but my I'm not trying to step, just execute so my interpretation is my
>> problem is different (although I certainly could be wrong).  I didn't
>> see any issues with fnsave or ltr.
>> 
>> I'm using Xen under OpenSuse 11.1 (Xen 3.3.1), although I've also
>> tried OpenSuse 11.3 (Xen 4.0.0) with identical results.
>> 
>> Any assistance would be greatly appreciated.
>> 
>> The code snippet for the fxsave case:
>> 
>>     diagMsg << "\nSaving FP context @" << (DWORD)
>> (&initialFPcontext) << "...";
>>     // Ensure buffer is writeable.
>>     ((BYTE*)(&initialFPcontext))[0]=0;
>>     ((BYTE*)(&initialFPcontext))[511]=0;
>>     // Initialze the FPU.
>>     diagMsg << "fninit...";
>>     asm volatile ("fninit");
>> 
>>     if (simdContextSavingSupported())
>>     {
>>       diagMsg << "fxSave...";
>>       asm volatile ("fxsave (%0)"::"r"(&initialFPcontext));
>>       diagMsg << "done\n";
>>     ...
>> 
>> The results are:
>> 
>>    Saving FP context @C082D470...fninit...fxSave...
>> 
>> An objdump -d of the binary yields:
>> 
>>    c0811ffb:    0f ae 00                fxsave (%eax)
>> 
>> "xm list" shows the VM continuing to get CPU cycles:
>> 
>>   # xm list
>>   Name                    ID   Mem VCPUs      State   Time(s)
>>   Domain-0                 0  3855     2     r-----    367.4
>>   myos                    59    16     1     r-----     17.7
>> 
>> GDB attached to the running VM shows that the processor is executing
>> the fxsave instruction and eax is as expected:
>> 
>>    (gdb) p/x $eip
>>    $1 = 0xc0811ffb
>>    (gdb) p/x $eax
>>    $2 = 0xc082d470
>> 
>> The /var/log/xen/qemu-dm-myos.log:
>>   domid: 59
>>   qemu: the number of cpus is 1
>>   Watching /local/domain/0/device-model/59/logdirty/next-active
>>   Watching /local/domain/0/device-model/59/command
>>   xs_read(): vncpasswd get error. /vm/9ec94cd1-20c5-07da-a1d4-
>> fdd500b8aef6/vncpasswd.
>>   qemu_map_cache_init nr_buckets = 4000 size 196608
>>   /usr/src/packages/BUILD/xen-3.3.1-testing/tools/ioemu-
>> dir/hw/xen_blktap.c:714: Init blktap pipes
>>   shared page at pfn ffe
>>   buffered io page at pfn ffc
>>   Time offset set 0
>>   Register xen platform.
>>   Done register platform.
>>   medium change watch on `hdc' (index: 0): /home/alarson/pci-scanner.iso
>>   I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
>> 
>> The fxsave instruction lists the following faults, and I've checked
>> them all.
>> 
>>   Protected Mode faults:
>>     #GP(0) illegal memory operand effective address in segment
>>     register. 
>>     #SS(0) illegal address in SS segment
>>     #PF(fault-code)
>> 
>>      The instruction does not have a segment prefix, and the
>>      preceding code ensures the buffer is writable.
>> 
>>     #NM if CR0[ts] or CR0[em]=1 (bits 3 or 2).
>> 
>>      The CR0 is C0000033
>> 
>>     #UD If CPUID.01H:EDX.FXSR[bit 24] = 0.
>> 
>>       CPUID.01H is EAX=0001067A EBX=00020800 ECX=80082201 EDX=0789FBBF
>> 
>>       EDX[24] is 1
>> 
>>     #UD If the LOCK prefix is used
>> 
>>       Instruction has no prefixes.
>> 
>>     #AC  for alignment
>> 
>>       FP context address is 0xC082D470 (16 byte aligned)
>> 
>> cat /proc/cpuinfo
>> processor       : 0
>> vendor_id       : GenuineIntel
>> cpu family      : 6
>> model           : 23
>> model name      : Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz
>> stepping        : 10
>> cpu MHz         : 2992.582
>> cache size      : 6144 KB
>> fdiv_bug        : no
>> hlt_bug         : no
>> f00f_bug        : no
>> coma_bug        : no
>> fpu             : yes
>> fpu_exception   : yes
>> cpuid level     : 13
>> wp              : yes
>> flags           : fpu de tsc msr pae cx8 apic mtrr cmov pat clflush
>> acpi mmx fxsr sse sse2 ss ht nx constant_tsc pni est ssse3 sse4_1
>> bogomips        : 5987.06
>> clflush size    : 64
>> power management:
>> 
>> processor       : 1
>> [same as for processor 0]
>> 
>> An xm list -l yields the following for the domain:
>> (domain
>>     (domid 58)
>>     (bootloader )
>>     (on_crash preserve)
>>     (uuid ca61601b-4728-d39c-2a4c-4fe0ced6954e)
>>     (bootloader_args )
>>     (vcpus 1)
>>     (name myos)
>>     (cpus (()))
>>     (on_reboot restart)
>>     (on_poweroff destroy)
>>     (maxmem 16)
>>     (memory 16)
>>     (shadow_memory 2)
>>     (features )
>>     (on_xend_start ignore)
>>     (on_xend_stop ignore)
>>     (start_time 1288284595.39)
>>     (cpu_time 319.894000271)
>>     (online_vcpus 1)
>>     (image
>>         (hvm
>>             (kernel )
>>             (args 'VNC_VIEWER=127.0.0.2:5514 ')
>>             (timer_mode 0)
>>             (vnc 1)
>>             (nographic 0)
>>             (hpet 0)
>>             (rtc_timeoffset 0)
>>             (loader /usr/lib/xen/boot/hvmloader)
>>             (pci ())
>>             (guest_os_type default)
>>             (apic 1)
>>             (sdl 0)
>>             (extid 0)
>>             (hap 0)
>>             (monitor 0)
>>             (device_model /usr/lib/xen/bin/qemu-dm)
>>             (boot d)
>>             (display localhost:10.0)
>>             (pae 0)
>>             (stdvga 0)
>>             (usb 0)
>>             (xauthority /root/.Xauthority)
>>             (isa 0)
>>             (acpi 0)
>>             (vncunused 1)
>>             (localtime 0)
>>             (notes (SUSPEND_CANCEL 1))
>>         )
>>     )
>>     (status 1)
>>     (state --p---)
>>     (store_mfn 4093)
>>     (device
>>         (console
>>             (protocol vt100)
>>             (location 3)
>>             (uuid 9b2b7ffe-8520-8a0b-6a0a-8864bcc3c8c4)
>>         )
>>     )
>>     (device
>>         (vbd
>>             (uuid 65beb1fb-b726-a262-8837-6a99d24a45e3)
>>             (bootable 1)
>>             (dev hdc:cdrom)
>>             (uname file:/home/alarson/pci-scanner.iso)
>>             (mode r)
>>             (backend 0)
>>             (bootable 1)
>>             (VDI )
>>         )
>>     )
>>     (device
>>         (vfb
>>             (vncunused 1)
>>             (type vnc)
>>             (uuid 7cb2e9d4-8376-8f6b-7851-6bf7c4058914)
>>             (location 127.0.0.1:5900)
>>         )
>>     )
>> )
> 
> 
> This message is intended only for the use of the individual or entity to which
> it is addressed. If the reader of this message is not the intended recipient,
> or the employee or agent responsible for delivering the message to the
> intended recipient, you are hereby notified that any dissemination,
> distribution or copying of this message is strictly prohibited. If you have
> received this communication in error, please notify us immediately by replying
> to the sender of this E-Mail by return E-Mail or by telephone.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-02 20:16 ` Keir Fraser
@ 2010-11-03  1:10   ` alarson
  2010-11-03  6:48     ` Keir Fraser
  0 siblings, 1 reply; 13+ messages in thread
From: alarson @ 2010-11-03  1:10 UTC (permalink / raw)
  To: xen-devel

Keir Fraser <keir.xen@gmail.com> wrote on 11/02/2010 03:16:38 PM:

> Since the guest is allegedly continually running, you might try the 'd'
> debug key (hit d on Xen serial console) and get execution backtrace for
> every physical CPU, and see what the hypervisor is spending its time 
doing
> for your guest on that instruction. Or add tracing to Xen to see where 
time
> is going. Eg. Could be endlessly trying to emulate fxsave in
> arch/x86/x86_emulate/x86_emulate.c. Or could be taking endless faults 
via
> arch/x86/hvm/vmx/vmx.c:vmx_fpu_dirty_intercept().
> 
> In short, you need to do some more spadework.

Thanks Keir, I was headed in that general direction, but I was hoping to 
get lucky.

I setup the serial cable and did a number of backtraces.  I'm hoping that 
the backtrace for the host is correlated with the client backtrace, but 
I'm suspicious.  Some of the backtraces I got list my client (e.g., on 
CPU1) but don't list a CPU1 host state.  In all cases my clint is doing 
what I expect (attempting to execute an fnsave instruction).

I slurped the relevant host stacktraces from the console debug log, and I 
don't see a pattern (again hoping to get lucky).  I'll put away my private 
OS shovel, and see if I can start fashioning a Xen shovel.  If you see 
something obvious below that looks suspicious, let me know, otherwise I'll 
see what my Xen shovel digs up.  Thanks again for your help.

Host CPU traces from the CPU's associated with my OS client.

(XEN) Xen call trace:
(XEN)    [<ffff82c4801ee984>] __mfn_valid+0x44/0x50
(XEN)    [<ffff82c4801d513f>] sh_page_fault__guest_2+0x71f/0x1880
(XEN)    [<ffff82c4801480ea>] apic_timer_interrupt+0x2a/0x30
(XEN)    [<ffff82c4801a7bbb>] hvm_maybe_deassert_evtchn_irq+0x1b/0x60
(XEN)    [<ffff82c4801bc973>] vmx_vmexit_handler+0xe33/0x1bb0
(XEN)    [<ffff82c480119b2d>] _csched_cpu_pick+0xfd/0x350
(XEN)    [<ffff82c4801afbd9>] vpic_update_int_output+0x29/0x80
(XEN)    [<ffff82c4801a7b93>] hvm_isa_irq_deassert+0x93/0xa0
(XEN)    [<ffff82c4801b0a79>] pt_update_irq+0x159/0x1f0
(XEN)    [<ffff82c4801518bd>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c4801a7af5>] hvm_isa_irq_assert+0x95/0xa0
(XEN)    [<ffff82c4801ae2a0>] vlapic_accept_pic_intr+0x20/0x30
(XEN)    [<ffff82c4801a8184>] hvm_vcpu_has_pending_irq+0x24/0x60
(XEN)    [<ffff82c4801b5cd8>] vmx_intr_assist+0x1d8/0x1f0
(XEN)    [<ffff82c4801b5923>] vmx_asm_do_vmentry+0x0/0xdd

(XEN) Xen call trace:
(XEN)    [<ffff82c4801a7a98>] hvm_isa_irq_assert+0x38/0xa0
(XEN)    [<ffff82c4801b5b46>] vmx_intr_assist+0x46/0x1f0
(XEN)    [<ffff82c4801b5928>] vmx_asm_do_vmentry+0x5/0xdd

(XEN) Xen call trace:
(XEN)    [<ffff82c4801bbc05>] vmx_vmexit_handler+0xc5/0x1bb0
(XEN)    [<ffff82c480119b2d>] _csched_cpu_pick+0xfd/0x350
(XEN)    [<ffff82c4801afbd9>] vpic_update_int_output+0x29/0x80
(XEN)    [<ffff82c4801a7b93>] hvm_isa_irq_deassert+0x93/0xa0
(XEN)    [<ffff82c4801b0a79>] pt_update_irq+0x159/0x1f0
(XEN)    [<ffff82c4801518bd>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c4801a7af5>] hvm_isa_irq_assert+0x95/0xa0
(XEN)    [<ffff82c4801ae2a0>] vlapic_accept_pic_intr+0x20/0x30
(XEN)    [<ffff82c4801a8184>] hvm_vcpu_has_pending_irq+0x24/0x60
(XEN)    [<ffff82c4801b5cd8>] vmx_intr_assist+0x1d8/0x1f0
(XEN)    [<ffff82c4801b5923>] vmx_asm_do_vmentry+0x0/0xdd

(XEN) Xen call trace:
(XEN)    [<ffff82c4801bbc05>] vmx_vmexit_handler+0xc5/0x1bb0
(XEN)    [<ffff82c480119b2d>] _csched_cpu_pick+0xfd/0x350
(XEN)    [<ffff82c4801afbd9>] vpic_update_int_output+0x29/0x80
(XEN)    [<ffff82c4801a7b93>] hvm_isa_irq_deassert+0x93/0xa0
(XEN)    [<ffff82c4801b0a79>] pt_update_irq+0x159/0x1f0
(XEN)    [<ffff82c4801518bd>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c4801a7af5>] hvm_isa_irq_assert+0x95/0xa0
(XEN)    [<ffff82c4801ae2a0>] vlapic_accept_pic_intr+0x20/0x30
(XEN)    [<ffff82c4801a8184>] hvm_vcpu_has_pending_irq+0x24/0x60
(XEN)    [<ffff82c4801b5cd8>] vmx_intr_assist+0x1d8/0x1f0
(XEN)    [<ffff82c4801b5923>] vmx_asm_do_vmentry+0x0/0xdd

(XEN) Xen call trace:
(XEN)    [<ffff82c48017bd95>] x86_emulate+0x2e5/0x100b0
(XEN)    [<ffff82c480136500>] serial_putc+0x100/0x130
(XEN)    [<ffff82c480136166>] __serial_putc+0x86/0x190
(XEN)    [<ffff82c4801480ea>] apic_timer_interrupt+0x2a/0x30
(XEN)    [<ffff82c480149743>] smp_apic_timer_interrupt+0x53/0xa0
(XEN)    [<ffff82c480149743>] smp_apic_timer_interrupt+0x53/0xa0
(XEN)    [<ffff82c4801d2daa>] sh_gva_to_gfn__guest_2+0x10a/0x170
(XEN)    [<ffff82c4801a051b>] __hvm_copy+0x33b/0x3e0
(XEN)    [<ffff82c4801d5c19>] sh_page_fault__guest_2+0x11f9/0x1880
(XEN)    [<ffff82c480149743>] smp_apic_timer_interrupt+0x53/0xa0
(XEN)    [<ffff82c4801480ea>] apic_timer_interrupt+0x2a/0x30
(XEN)    [<ffff82c4801bc973>] vmx_vmexit_handler+0xe33/0x1bb0
(XEN)    [<ffff82c480119b2d>] _csched_cpu_pick+0xfd/0x350
(XEN)    [<ffff82c4801b0893>] pt_restore_timer+0x23/0xb0
(XEN)    [<ffff82c4801afbd9>] vpic_update_int_output+0x29/0x80
(XEN)    [<ffff82c4801a7b93>] hvm_isa_irq_deassert+0x93/0xa0
(XEN)    [<ffff82c4801b0a79>] pt_update_irq+0x159/0x1f0
(XEN)    [<ffff82c4801518bd>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c4801a7af5>] hvm_isa_irq_assert+0x95/0xa0
(XEN)    [<ffff82c4801ae2a0>] vlapic_accept_pic_intr+0x20/0x30
(XEN)    [<ffff82c4801a8184>] hvm_vcpu_has_pending_irq+0x24/0x60
(XEN)    [<ffff82c4801b5cd8>] vmx_intr_assist+0x1d8/0x1f0
(XEN)    [<ffff82c4801b5923>] vmx_asm_do_vmentry+0x0/0xdd

Full log:

My client can be identified by "cr3: 0000000000a3b000"

(XEN) 'd' pressed -> dumping registers
(XEN) 
(XEN) *** Dumping CPU1 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff82c4801ee984>] __mfn_valid+0x44/0x50
(XEN) RFLAGS: 0000000000000286   CONTEXT: hypervisor
(XEN) rax: 0000000000000001   rbx: ffff8301273e0000   rcx: 
0000000000000000
(XEN) rdx: 0000000000000000   rsi: ffff818000001818   rdi: 
000000000011fe04
(XEN) rbp: ffff8300cfa4a000   rsp: ffff830128f6fb20   r8: 00000000001223ed
(XEN) r9:  0000000000000000   r10: 0000000000303b90   r11: 
00000000001223e8
(XEN) r12: ffff830128f6ff28   r13: 0000000000000006   r14: 
0000000000000303
(XEN) r15: ffff830128f6ff28   cr0: 0000000080050033   cr4: 
00000000000426f0
(XEN) cr3: 0000000127230000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff830128f6fb20:
(XEN)    ffff82c4801d513f 00000000c082243e 0000000000000000 
0000000000000001
(XEN)    0000000000000000 0000000100000003 ffff8300cfa4bab8 
00000000001223ed
(XEN)    000000000011ef03 0000000000303b90 ffff8301273e0e30 
0000000000000303
(XEN)    ffff82c480266de0 ffff830128f6fca8 0000000000000060 
ffff830128f6ff28
(XEN)    ffff830128f6ff28 ffff818000001818 ffff830128f6fce4 
0000000000000000
(XEN)    ffffffffffffffff 0000000000000000 000000008046fe60 
0000000000000c0c
(XEN)    ffff830128f6fcc8 ffff830128f6ff28 0000002000000020 
0000000100000000
(XEN)    8dc0821d676836dd b5e850ffffff5885 f68910c483ffff37 
1fe58955f689feeb
(XEN)    00000000c081745c 0c9b000800000003 00000000ffffffff 
0c93001000000000
(XEN)    00000000ffffffff ffff82c400000000 ffff830128f6ff28 
ffff8300cfa4a000
(XEN)    ffff82c4801480ea 000000000000000e 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffff8300cfa4a000 ffff830128f6ff28 
0000000000000000
(XEN)    0000000000000000 0000000000303b90 0030306500a04c67 
000000000011fe3b
(XEN)    000000000011fe04 00000000c082243e 0000000000303b90 
000000011ef03025
(XEN)    0000000100000000 ffff82c4801a7bbb ffff830128f6ff28 
ffff8300cfa4a000
(XEN)    0000000000303b90 0000000000000000 0000000000000000 
000000000000000e
(XEN)    ffff82c4801bc973 0000000000000000 0000000000000001 
ffff830128f6fdf8
(XEN)    ffff830128f6fd98 ffff830128f6fdd8 ffff82c480119b2d 
ffff82c400000001
(XEN)    ffff82c480266ee0 ffff82c480266f00 ffff82c480266f00 
ffff82c480266080
(XEN)    00ff82c4801480ea ffff82c480266ee0 ffff82c480266080 
0000000000000001
(XEN) Xen call trace:
(XEN)    [<ffff82c4801ee984>] __mfn_valid+0x44/0x50
(XEN)    [<ffff82c4801d513f>] sh_page_fault__guest_2+0x71f/0x1880
(XEN)    [<ffff82c4801480ea>] apic_timer_interrupt+0x2a/0x30
(XEN)    [<ffff82c4801a7bbb>] hvm_maybe_deassert_evtchn_irq+0x1b/0x60
(XEN)    [<ffff82c4801bc973>] vmx_vmexit_handler+0xe33/0x1bb0
(XEN)    [<ffff82c480119b2d>] _csched_cpu_pick+0xfd/0x350
(XEN)    [<ffff82c4801afbd9>] vpic_update_int_output+0x29/0x80
(XEN)    [<ffff82c4801a7b93>] hvm_isa_irq_deassert+0x93/0xa0
(XEN)    [<ffff82c4801b0a79>] pt_update_irq+0x159/0x1f0
(XEN)    [<ffff82c4801518bd>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c4801a7af5>] hvm_isa_irq_assert+0x95/0xa0
(XEN)    [<ffff82c4801ae2a0>] vlapic_accept_pic_intr+0x20/0x30
(XEN)    [<ffff82c4801a8184>] hvm_vcpu_has_pending_irq+0x24/0x60
(XEN)    [<ffff82c4801b5cd8>] vmx_intr_assist+0x1d8/0x1f0
(XEN)    [<ffff82c4801b5923>] vmx_asm_do_vmentry+0x0/0xdd
(XEN) 
(XEN) *** Dumping CPU1 guest state (d1:v0): ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0008:[<00000000c081745c>]
(XEN) RFLAGS: 0000000000010046   CONTEXT: hvm guest
(XEN) rax: 0000000000303d9c   rbx: 00000000c0821d53   rcx: 
00000000000b8b98
(XEN) rdx: 00000000c082243e   rsi: 0000000000303b90   rdi: 
0000000000303d9c
(XEN) rbp: 0000000000303e44   rsp: 0000000000303904   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080000011   cr4: 
0000000000000200
(XEN) cr3: 0000000000a3b000   cr2: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008
(XEN) 
(XEN) *** Dumping CPU0 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48011faa2>] _spin_lock+0x12/0x20
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff8301273e0000   rcx: 
0000000000000000
(XEN) rdx: 0000000000000000   rsi: 0000000000000000   rdi: 
ffff8301273e0e30
(XEN) rbp: 0000000000000000   rsp: ffff82c48046fde0   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: ffff82c480204330   r11: 
0000000000000282
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 
00000000000026f0
(XEN) cr3: 000000002c9dd000   cr2: ffff8800c903b218
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48046fde0:
(XEN)    ffff82c4801d2142 0000000000000002 ffff82c48014fdbc 
0000000000000000
(XEN)    ffff82c4802660a8 ffff82c480121598 ffff8301273e0e30 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
000000000000140e
(XEN)    ffffffffffffffea 0000000000000006 00007ffffa3015f0 
ffff830128f40000
(XEN)    ffffffff802e2770 ffff8800ee7d47c0 ffff82c4801a4a4c 
0000000001c9c380
(XEN)    000002515ca4324d ffff830128f60330 ffff8300cfdfa000 
0000000000000001
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000206 ffffffff800073d0 0000000000000100 
00007f71258806b7
(XEN)    ffff8301273e0000 0000000000000206 ffff8300cfdfa000 
ffffffff80003440
(XEN)    0000000000305000 0000000000000000 ffffffff802e2770 
ffff8800ee7d47c0
(XEN)    ffff82c4801ee169 ffff8800ee7d47c0 ffffffff802e2770 
0000000000000000
(XEN)    0000000000305000 ffffffff80003440 ffff8800ce3c7e38 
0000000000000282
(XEN)    0000000000b00b60 0000000000000000 0000000000000001 
0000000000000022
(XEN)    ffffffff8000344a 0000000000000001 00007ffffa3015f0 
0000000000000006
(XEN)    0000010000000000 ffffffff8000344a 000000000000e033 
0000000000000282
(XEN)    ffff8800ce3c7db0 000000000000e02b 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
ffff8300cfdfa000
(XEN) Xen call trace:
(XEN)    [<ffff82c48011faa2>] _spin_lock+0x12/0x20
(XEN)    [<ffff82c4801d2142>] shadow_track_dirty_vram+0xd2/0x600
(XEN)    [<ffff82c48014fdbc>] context_switch+0x18c/0xec0
(XEN)    [<ffff82c480121598>] add_entry+0x38/0xa0
(XEN)    [<ffff82c4801a4a4c>] do_hvm_op+0x8dc/0x11b0
(XEN)    [<ffff82c4801ee169>] syscall_enter+0xa9/0xae
(XEN) 
(XEN) *** Dumping CPU0 guest state (d0:v1): ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<ffffffff8000344a>]
(XEN) RFLAGS: 0000000000000282   EM: 0   CONTEXT: pv guest
(XEN) rax: 0000000000000022   rbx: ffff8800ce3c7e38   rcx: 
ffffffff8000344a
(XEN) rdx: 0000000000000001   rsi: 00007ffffa3015f0   rdi: 
0000000000000006
(XEN) rbp: ffffffff80003440   rsp: ffff8800ce3c7db0   r8: 0000000000000001
(XEN) r9:  0000000000000000   r10: 0000000000b00b60   r11: 
0000000000000282
(XEN) r12: 0000000000305000   r13: 0000000000000000   r14: 
ffffffff802e2770
(XEN) r15: ffff8800ee7d47c0   cr0: 000000008005003b   cr4: 
00000000000026f0
(XEN) cr3: 000000002c9dd000   cr2: 00007f9c5e443000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffff8800ce3c7db0:
(XEN)    0000000000000246 0000000000000000 ffffffff802e288d 
ffff8800ce3e8ca0
(XEN)    ffff8800cc3ebf00 ffff880002024c80 ffff8800cc3ebf00 
ffff8800ce3e8c60
(XEN)    ffffffff800f3eb3 ffff8800ce3c7ed8 25602560ce3c7ed8 
0000000000000022
(XEN)    0000000000000006 00007ffffa3015f0 0000000000000001 
0000000000b00b60
(XEN)    0000000000000001 ffff8800ce3c7e38 ffff8800ce3c7e38 
ffff8800cc3ebf00
(XEN)    ffff8800010db140 ffff8800cc3386c0 0000000000305000 
0000000000000000
(XEN)    ffffffff802e2770 00007ffffa3015c0 ffffffff8016d134 
ffff8800ce3c7f30
(XEN)    ffff8800010db1b8 00007ffffa301000 ffff8800cc3386c0 
0000000000305000
(XEN)    00007ffffa3015c0 00007ffffa3015c0 0000000000000011 
0000000000000001
(XEN)    ffffffff80123ac9 ffff8800cc3386c0 0000000000000005 
0000000000305000
(XEN)    ffffffff80124060 0000000000000001 ffff8800ce3c7f30 
00007ffffa302000
(XEN)    0000000000000011 ffff8800cc3386c0 0000000000000005 
0000000000305000
(XEN)    ffffffff80124460 00007ffffa301000 0000000000001000 
00007ffffa301000
(XEN)    00000001e0744840 0000000000ae4448 0000000000000005 
0000000000000000
(XEN)    00007ffffa3015f0 0000000000ae4448 ffffffff80007438 
0000000000000246
(XEN)    00007ffffa301740 0000000000000000 0000000000000000 
0000000000000010
(XEN)    ffffffff800073d0 00007ffffa3015c0 0000000000305000 
0000000000000005
(XEN)    0000000000000010 00007f712587be87 000000000000e033 
0000000000000246
(XEN)    00007ffffa3015b8 000000000000e02b
(XEN) 
(XEN) 'd' pressed -> dumping registers
(XEN) 
(XEN) *** Dumping CPU1 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff82c4801a7a98>] hvm_isa_irq_assert+0x38/0xa0
(XEN) RFLAGS: 0000000000000202   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff8301273e0000   rcx: 
0000000000000001
(XEN) rdx: 0000000000000001   rsi: 0000000000000000   rdi: 
ffff8301273e09a8
(XEN) rbp: 0000000000000000   rsp: ffff830128f6fe98   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000002   r11: 
0000000000000000
(XEN) r12: ffff8301273e09a8   r13: 0000000000000002   r14: 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 
00000000000426f0
(XEN) cr3: 0000000127230000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff830128f6fe98:
(XEN)    0000000000303e44 ffff8300cfa4a000 0000000000303e44 
0000000000000000
(XEN)    0000000000004016 ffff82c4801b5b46 ffffffffffffffff 
ffff830128f6ff00
(XEN)    ffff82c480483280 0000000000000000 0000000000000000 
ffff8300cfa4a000
(XEN)    0000000000303e44 ffff8300cfa4a000 0000000000303e44 
0000000000000000
(XEN)    0000000000000000 ffff82c4801b5928 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000303e44 
00000000c0821d53
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000303d9c 00000000000b8b98 00000000c082243e 
0000000000303b90
(XEN)    0000000000303d9c 000000f100000003 00000000c081745c 
0000000000000000
(XEN)    0000000000010046 0000000000303904 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000001
(XEN)    ffff8300cfa4a000
(XEN) Xen call trace:
(XEN)    [<ffff82c4801a7a98>] hvm_isa_irq_assert+0x38/0xa0
(XEN)    [<ffff82c4801b5b46>] vmx_intr_assist+0x46/0x1f0
(XEN)    [<ffff82c4801b5928>] vmx_asm_do_vmentry+0x5/0xdd
(XEN) 
(XEN) *** Dumping CPU1 guest state (d1:v0): ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0008:[<00000000c081745c>]
(XEN) RFLAGS: 0000000000010046   CONTEXT: hvm guest
(XEN) rax: 0000000000303d9c   rbx: 00000000c0821d53   rcx: 
00000000000b8b98
(XEN) rdx: 00000000c082243e   rsi: 0000000000303b90   rdi: 
0000000000303d9c
(XEN) rbp: 0000000000303e44   rsp: 0000000000303904   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080000011   cr4: 
0000000000000200
(XEN) cr3: 0000000000a3b000   cr2: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008
(XEN) 
(XEN) *** Dumping CPU0 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: 0000000000000003   rbx: ffff830128f26c20   rcx: 
ffff82c480483280
(XEN) rdx: 0000000000000808   rsi: 0000000000000000   rdi: 
ffff830128f26c80
(XEN) rbp: ffff830128f26c80   rsp: ffff82c48046fe78   r8: ffff830128f60350
(XEN) r9:  000002534ef40b7f   r10: 0000000000000000   r11: 
ffff82c4804ad5c0
(XEN) r12: 00000000002c1327   r13: ffff82c48046fe18   r14: 
ffff8300cfdfc000
(XEN) r15: 0000000000000002   cr0: 000000008005003b   cr4: 
00000000000026f0
(XEN) cr3: 000000002c9dd000   cr2: ffff8800c903b218
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48046fe78:
(XEN)    ffff82c48018f1e2 ffff82c4802660a8 ffff8300cfdfc000 
ffffffffffffffff
(XEN)    ffff830128f60330 ffff82c48012181a ffff8300cfdfc000 
0000000000000000
(XEN)    0000000000000000 ffff82c480483280 00001aa200002221 
ffff82c48046ff28
(XEN)    ffff82c48046ff28 ffff82c480266c00 ffff82c48046ff28 
ffff82c48046fe18
(XEN)    ffff8300cfdfc000 0000000000000002 ffff82c48014d73e 
ffff8300cfdfc000
(XEN)    ffff8300cfc66000 00000000ffffffff 0000000000000000 
0000000000000000
(XEN)    ffffffffffffffff ffffffff806f5fd8 ffffffff80751100 
0000000000000000
(XEN)    0000000000000246 0000000000000001 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffffffff800033aa 0000000000000000 
0000000000000000
(XEN)    0000000000000001 0000010000000000 ffffffff800033aa 
000000000000e033
(XEN)    0000000000000246 ffffffff806f5f58 000000000000e02b 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    ffff8300cfdfc000
(XEN) Xen call trace:
(XEN)    [<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN)    [<ffff82c48018f1e2>] acpi_processor_idle+0x2d2/0x6d0
(XEN)    [<ffff82c48012181a>] timer_softirq_action+0x21a/0x360
(XEN)    [<ffff82c48014d73e>] idle_loop+0x2e/0x90
(XEN) 
(XEN) No guest context (CPU0 is idle).
(XEN) 
(XEN) 'd' pressed -> dumping registers
(XEN) *** Dumping CPU1 guest state (d1:v0): ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0008:[<00000000c081745c>]
(XEN) RFLAGS: 0000000000010046   CONTEXT: hvm guest
(XEN) rax: 0000000000303d9c   rbx: 00000000c0821d53   rcx: 
00000000000b8b98
(XEN) rdx: 00000000c082243e   rsi: 0000000000303b90   rdi: 
0000000000303d9c
(XEN) rbp: 0000000000303e44   rsp: 0000000000303904   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080000011   cr4: 
0000000000000200
(XEN) cr3: 0000000000a3b000   cr2: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008
(XEN) 
(XEN) *** Dumping CPU0 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: 0000000000000003   rbx: ffff830128f26c20   rcx: 
ffff82c480483280
(XEN) rdx: 0000000000000808   rsi: 0000000000000000   rdi: 
ffff830128f26c80
(XEN) rbp: ffff830128f26c80   rsp: ffff82c48046fe78   r8: ffff830128f60348
(XEN) r9:  00000254bcdb0417   r10: 0000000000000000   r11: 
ffff8300cfdfa060
(XEN) r12: 00000000009fcd27   r13: ffff82c48046fe18   r14: 
ffff8300cfdfc000
(XEN) r15: 0000000000000002   cr0: 000000008005003b   cr4: 
00000000000026f0
(XEN) cr3: 000000002c9dd000   cr2: ffff8800c903b218
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48046fe78:
(XEN)    ffff82c48018f1e2 ffff82c4802660a8 ffff8300cfdfc000 
ffffffffffffffff
(XEN)    ffff830128f60330 ffff82c48012181a ffff8300cfdfc000 
0000000000000000
(XEN)    0000000000000000 ffff82c480483280 0000042a00001ecf 
ffff82c48046ff28
(XEN)    ffff82c48046ff28 ffff82c480266c00 ffff82c48046ff28 
ffff82c48046fe18
(XEN)    ffff8300cfdfc000 0000000000000002 ffff82c48014d73e 
ffff8300cfdfc000
(XEN)    ffff8300cfc66000 00000000ffffffff 0000000000000000 
0000000000000000
(XEN)    ffffffffffffffff ffffffff806f5fd8 ffffffff80751100 
0000000000000000
(XEN)    0000000000000246 0000000000000001 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffffffff800033aa 0000000000000000 
0000000000000000
(XEN)    0000000000000001 0000010000000000 ffffffff800033aa 
000000000000e033
(XEN)    0000000000000246 ffffffff806f5f58 000000000000e02b 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    ffff8300cfdfc000
(XEN) Xen call trace:
(XEN)    [<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN)    [<ffff82c48018f1e2>] acpi_processor_idle+0x2d2/0x6d0
(XEN)    [<ffff82c48012181a>] timer_softirq_action+0x21a/0x360
(XEN)    [<ffff82c48014d73e>] idle_loop+0x2e/0x90
(XEN) 
(XEN) No guest context (CPU0 is idle).
(XEN) 
(XEN) 'd' pressed -> dumping registers
(XEN) *** Dumping CPU1 guest state (d1:v0): ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0008:[<00000000c081745c>]
(XEN) RFLAGS: 0000000000010046   CONTEXT: hvm guest
(XEN) rax: 0000000000303d9c   rbx: 00000000c0821d53   rcx: 
00000000000b8b98
(XEN) rdx: 00000000c082243e   rsi: 0000000000303b90   rdi: 
0000000000303d9c
(XEN) rbp: 0000000000303e44   rsp: 0000000000303904   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080000011   cr4: 
0000000000000200
(XEN) cr3: 0000000000a3b000   cr2: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008
(XEN) 
(XEN) *** Dumping CPU0 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: 0000000000000003   rbx: ffff830128f26c20   rcx: 
ffff82c480483280
(XEN) rdx: 0000000000000808   rsi: 0000000000000000   rdi: 
ffff830128f26c80
(XEN) rbp: ffff830128f26c80   rsp: ffff82c48046fe78   r8: ffff830128f60348
(XEN) r9:  00000256997a35e6   r10: 0000000000000000   r11: 
ffff82c4804b20e0
(XEN) r12: 000000000053e293   r13: ffff82c48046fe18   r14: 
ffff8300cfdfc000
(XEN) r15: 0000000000000002   cr0: 000000008005003b   cr4: 
00000000000026f0
(XEN) cr3: 000000002c9dd000   cr2: ffff8800c9093168
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48046fe78:
(XEN)    ffff82c48018f1e2 ffff82c4802660a8 ffff8300cfdfc000 
ffffffffffffffff
(XEN)    ffff830128f60330 ffff82c48012181a ffff8300cfdfc000 
0000000000000000
(XEN)    0000000000000000 ffff82c480483280 00001b8000002547 
ffff82c48046ff28
(XEN)    ffff82c48046ff28 ffff82c480266c00 ffff82c48046ff28 
ffff82c48046fe18
(XEN)    ffff8300cfdfc000 0000000000000002 ffff82c48014d73e 
ffff8300cfdfc000
(XEN)    ffff8300cfc66000 00000000ffffffff 0000000000000000 
0000000000000000
(XEN)    ffffffffffffffff ffffffff806f5fd8 ffffffff80751100 
0000000000000000
(XEN)    0000000000000246 0000000000000001 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffffffff800033aa 0000000000000000 
0000000000000000
(XEN)    0000000000000001 0000010000000000 ffffffff800033aa 
000000000000e033
(XEN)    0000000000000246 ffffffff806f5f58 000000000000e02b 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    ffff8300cfdfc000
(XEN) Xen call trace:
(XEN)    [<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN)    [<ffff82c48018f1e2>] acpi_processor_idle+0x2d2/0x6d0
(XEN)    [<ffff82c48012181a>] timer_softirq_action+0x21a/0x360
(XEN)    [<ffff82c48014d73e>] idle_loop+0x2e/0x90
(XEN) 
(XEN) No guest context (CPU0 is idle).
(XEN) 
(XEN) 'd' pressed -> dumping registers
(XEN) 
(XEN) *** Dumping CPU1 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff82c4801bbc05>] vmx_vmexit_handler+0xc5/0x1bb0
(XEN) RFLAGS: 0000000000000287   CONTEXT: hypervisor
(XEN) rax: 0000000000004404   rbx: ffff830128f6ff28   rcx: 
0000000080000b0e
(XEN) rdx: 00000000c082243e   rsi: 0000000000303b90   rdi: 
ffff830128f6ff28
(XEN) rbp: ffff8300cfa4a000   rsp: ffff830128f6fd28   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 000000000000000e   cr0: 0000000080050033   cr4: 
00000000000426f0
(XEN) cr3: 0000000127230000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff830128f6fd28:
(XEN)    0000000000000004 0000000000000001 ffff830128f6fdf8 
ffff830128f6fd98
(XEN)    ffff830128f6fdd8 ffff82c480119b2d ffff830100000001 
ffff82c480266ee0
(XEN)    ffff82c480266f00 ffff82c480266f00 ffff82c480266080 
00ff82c4801480ea
(XEN)    ffff82c480266ee0 ffff82c480266080 0000000000000001 
0000000080000b0e
(XEN)    0000000000000000 0000000000000000 0000000000000002 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000003 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffff82c4801afbd9 ffffffffffff8000 
ffff8301273e0000
(XEN)    ffff8301273e09a8 ffff82c4801a7b93 ffff83012bc78748 
ffff8300cfa4a000
(XEN)    0000000000000000 0000000000000001 ffff8300cfa4a000 
ffff82c4801b0a79
(XEN)    0000000000000002 ffff82c4801518bd 0000000000000000 
ffff8301273e0000
(XEN)    0000000000000000 ffff82c4801a7af5 ffff82c4801ae2a0 
ffff82c4801a8184
(XEN)    ffff8300cfa4a000 ffff8300cfa4a000 0000000000000001 
ffff82c4801b5cd8
(XEN)    ffffffffffffffff ffff830128f6ff00 ffff82c480483280 
0000000000000000
(XEN)    0000000000000000 ffff8300cfa4a000 0000000000303e44 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
ffff82c4801b5923
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000303e44 00000000c0821d53 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000303d9c 
00000000000b8b98
(XEN)    00000000c082243e 0000000000303b90 0000000000303d9c 
000000f100000003
(XEN) Xen call trace:
(XEN)    [<ffff82c4801bbc05>] vmx_vmexit_handler+0xc5/0x1bb0
(XEN)    [<ffff82c480119b2d>] _csched_cpu_pick+0xfd/0x350
(XEN)    [<ffff82c4801afbd9>] vpic_update_int_output+0x29/0x80
(XEN)    [<ffff82c4801a7b93>] hvm_isa_irq_deassert+0x93/0xa0
(XEN)    [<ffff82c4801b0a79>] pt_update_irq+0x159/0x1f0
(XEN)    [<ffff82c4801518bd>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c4801a7af5>] hvm_isa_irq_assert+0x95/0xa0
(XEN)    [<ffff82c4801ae2a0>] vlapic_accept_pic_intr+0x20/0x30
(XEN)    [<ffff82c4801a8184>] hvm_vcpu_has_pending_irq+0x24/0x60
(XEN)    [<ffff82c4801b5cd8>] vmx_intr_assist+0x1d8/0x1f0
(XEN)    [<ffff82c4801b5923>] vmx_asm_do_vmentry+0x0/0xdd
(XEN) 
(XEN) *** Dumping CPU1 guest state (d1:v0): ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0008:[<00000000c081745c>]
(XEN) RFLAGS: 0000000000010046   CONTEXT: hvm guest
(XEN) rax: 0000000000303d9c   rbx: 00000000c0821d53   rcx: 
00000000000b8b98
(XEN) rdx: 00000000c082243e   rsi: 0000000000303b90   rdi: 
0000000000303d9c
(XEN) rbp: 0000000000303e44   rsp: 0000000000303904   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080000011   cr4: 
0000000000000200
(XEN) cr3: 0000000000a3b000   cr2: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008
(XEN) 
(XEN) *** Dumping CPU0 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: 0000000000000003   rbx: ffff830128f26c20   rcx: 
ffff82c480483280
(XEN) rdx: 0000000000000808   rsi: 0000000000000000   rdi: 
ffff830128f26c80
(XEN) rbp: ffff830128f26c80   rsp: ffff82c48046fe78   r8: ffff830128f60348
(XEN) r9:  00000257e2bb7c5e   r10: 0000000000000000   r11: 
ffff8300cfdfa060
(XEN) r12: 000000000081459b   r13: ffff82c48046fe18   r14: 
ffff8300cfdfc000
(XEN) r15: 0000000000000002   cr0: 000000008005003b   cr4: 
00000000000026f0
(XEN) cr3: 000000002c9dd000   cr2: ffff8800c903b218
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48046fe78:
(XEN)    ffff82c48018f1e2 ffff82c4802660a8 ffff8300cfdfc000 
ffffffffffffffff
(XEN)    ffff830128f60330 ffff82c48012181a ffff8300cfdfc000 
0000000000000000
(XEN)    0000000000000000 ffff82c480483280 0000239f000029ff 
ffff82c48046ff28
(XEN)    ffff82c48046ff28 ffff82c480266c00 ffff82c48046ff28 
ffff82c48046fe18
(XEN)    ffff8300cfdfc000 0000000000000002 ffff82c48014d73e 
ffff8300cfdfc000
(XEN)    ffff8300cfc66000 00000000ffffffff 0000000000000000 
0000000000000000
(XEN)    ffffffffffffffff ffffffff806f5fd8 ffffffff80751100 
0000000000000000
(XEN)    0000000000000246 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffffffff800033aa 0000000000000000 
0000000000000000
(XEN)    0000000000000001 0000010000000000 ffffffff800033aa 
000000000000e033
(XEN)    0000000000000246 ffffffff806f5f58 000000000000e02b 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    ffff8300cfdfc000
(XEN) Xen call trace:
(XEN)    [<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN)    [<ffff82c48018f1e2>] acpi_processor_idle+0x2d2/0x6d0
(XEN)    [<ffff82c48012181a>] timer_softirq_action+0x21a/0x360
(XEN)    [<ffff82c48014d73e>] idle_loop+0x2e/0x90
(XEN) 
(XEN) No guest context (CPU0 is idle).
(XEN) 
(XEN) 'd' pressed -> dumping registers
(XEN) 
(XEN) *** Dumping CPU1 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff82c4801bbc05>] vmx_vmexit_handler+0xc5/0x1bb0
(XEN) RFLAGS: 0000000000000287   CONTEXT: hypervisor
(XEN) rax: 0000000000004404   rbx: ffff830128f6ff28   rcx: 
0000000080000b0e
(XEN) rdx: 00000000c082243e   rsi: 0000000000303b90   rdi: 
ffff830128f6ff28
(XEN) rbp: ffff8300cfa4a000   rsp: ffff830128f6fd28   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 000000000000000e   cr0: 0000000080050033   cr4: 
00000000000426f0
(XEN) cr3: 0000000127230000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff830128f6fd28:
(XEN)    0000000000000004 0000000000000001 ffff830128f6fdf8 
ffff830128f6fd98
(XEN)    ffff830128f6fdd8 ffff82c480119b2d ffff830100000001 
ffff82c480266ee0
(XEN)    ffff82c480266f00 ffff82c480266f00 ffff82c480266080 
0000000000000000
(XEN)    ffff82c480266ee0 ffff82c480266080 0000000000000001 
0000000080000b0e
(XEN)    0000000000000000 0000000000000000 0000000000000002 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000003 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffff82c4801afbd9 ffffffffffff8000 
ffff8301273e0000
(XEN)    ffff8301273e09a8 ffff82c4801a7b93 ffff83012bc78748 
ffff8300cfa4a000
(XEN)    0000000000000000 0000000000000001 ffff8300cfa4a000 
ffff82c4801b0a79
(XEN)    0000000000000002 ffff82c4801518bd 0000000000000000 
ffff8301273e0000
(XEN)    0000000000000000 ffff82c4801a7af5 ffff82c4801ae2a0 
ffff82c4801a8184
(XEN)    ffff8300cfa4a000 ffff8300cfa4a000 0000000000000001 
ffff82c4801b5cd8
(XEN)    ffffffffffffffff ffff830128f6ff00 ffff82c480483280 
0000000000000000
(XEN)    0000000000000000 ffff8300cfa4a000 0000000000303e44 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
ffff82c4801b5923
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000303e44 00000000c0821d53 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000303d9c 
00000000000b8b98
(XEN)    00000000c082243e 0000000000303b90 0000000000303d9c 
000000f100000003
(XEN) Xen call trace:
(XEN)    [<ffff82c4801bbc05>] vmx_vmexit_handler+0xc5/0x1bb0
(XEN)    [<ffff82c480119b2d>] _csched_cpu_pick+0xfd/0x350
(XEN)    [<ffff82c4801afbd9>] vpic_update_int_output+0x29/0x80
(XEN)    [<ffff82c4801a7b93>] hvm_isa_irq_deassert+0x93/0xa0
(XEN)    [<ffff82c4801b0a79>] pt_update_irq+0x159/0x1f0
(XEN)    [<ffff82c4801518bd>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c4801a7af5>] hvm_isa_irq_assert+0x95/0xa0
(XEN)    [<ffff82c4801ae2a0>] vlapic_accept_pic_intr+0x20/0x30
(XEN)    [<ffff82c4801a8184>] hvm_vcpu_has_pending_irq+0x24/0x60
(XEN)    [<ffff82c4801b5cd8>] vmx_intr_assist+0x1d8/0x1f0
(XEN)    [<ffff82c4801b5923>] vmx_asm_do_vmentry+0x0/0xdd
(XEN) 
(XEN) *** Dumping CPU1 guest state (d1:v0): ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0008:[<00000000c081745c>]
(XEN) RFLAGS: 0000000000010046   CONTEXT: hvm guest
(XEN) rax: 0000000000303d9c   rbx: 00000000c0821d53   rcx: 
00000000000b8b98
(XEN) rdx: 00000000c082243e   rsi: 0000000000303b90   rdi: 
0000000000303d9c
(XEN) rbp: 0000000000303e44   rsp: 0000000000303904   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080000011   cr4: 
0000000000000200
(XEN) cr3: 0000000000a3b000   cr2: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008
(XEN) 
(XEN) *** Dumping CPU0 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: 0000000000000003   rbx: ffff830128f26c20   rcx: 
ffff82c480483280
(XEN) rdx: 0000000000000808   rsi: 0000000000000000   rdi: 
ffff830128f26c80
(XEN) rbp: ffff830128f26c80   rsp: ffff82c48046fe78   r8: ffff830128f60348
(XEN) r9:  0000025bb52c7d0d   r10: 0000000000000000   r11: 
ffff82c4804b20e0
(XEN) r12: 000000000002c315   r13: ffff82c48046fe18   r14: 
ffff8300cfdfc000
(XEN) r15: 0000000000000002   cr0: 000000008005003b   cr4: 
00000000000026f0
(XEN) cr3: 000000002c9dd000   cr2: ffff8800c903b218
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48046fe78:
(XEN)    ffff82c48018f1e2 ffff82c4802660a8 ffff8300cfdfc000 
ffffffffffffffff
(XEN)    ffff830128f60330 ffff82c48012181a ffff8300cfdfc000 
0000000000000000
(XEN)    0000000000000000 ffff82c480483280 00001d1b0000269b 
ffff82c48046ff28
(XEN)    ffff82c48046ff28 ffff82c480266c00 ffff82c48046ff28 
ffff82c48046fe18
(XEN)    ffff8300cfdfc000 0000000000000002 ffff82c48014d73e 
ffff8300cfdfc000
(XEN)    ffff8300cfc66000 00000000ffffffff 0000000000000000 
0000000000000000
(XEN)    ffffffffffffffff ffffffff806f5fd8 ffffffff80751100 
0000000000000000
(XEN)    0000000000000246 0000000000000001 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffffffff800033aa 0000000000000000 
0000000000000000
(XEN)    0000000000000001 0000010000000000 ffffffff800033aa 
000000000000e033
(XEN)    0000000000000246 ffffffff806f5f58 000000000000e02b 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    ffff8300cfdfc000
(XEN) Xen call trace:
(XEN)    [<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN)    [<ffff82c48018f1e2>] acpi_processor_idle+0x2d2/0x6d0
(XEN)    [<ffff82c48012181a>] timer_softirq_action+0x21a/0x360
(XEN)    [<ffff82c48014d73e>] idle_loop+0x2e/0x90
(XEN) 
(XEN) No guest context (CPU0 is idle).
(XEN) 
(XEN) 'd' pressed -> dumping registers
(XEN) *** Dumping CPU1 guest state (d1:v0): ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0008:[<00000000c081745c>]
(XEN) RFLAGS: 0000000000010046   CONTEXT: hvm guest
(XEN) rax: 0000000000303d9c   rbx: 00000000c0821d53   rcx: 
00000000000b8b98
(XEN) rdx: 00000000c082243e   rsi: 0000000000303b90   rdi: 
0000000000303d9c
(XEN) rbp: 0000000000303e44   rsp: 0000000000303904   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080000011   cr4: 
0000000000000200
(XEN) cr3: 0000000000a3b000   cr2: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008
(XEN) 
(XEN) *** Dumping CPU0 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: 0000000000000003   rbx: ffff830128f26c20   rcx: 
ffff82c480483280
(XEN) rdx: 0000000000000808   rsi: 0000000000000000   rdi: 
ffff830128f26c80
(XEN) rbp: ffff830128f26c80   rsp: ffff82c48046fe78   r8: ffff830128f60348
(XEN) r9:  0000025dd29780f3   r10: 0000000000000000   r11: 
ffff8300cfdfa060
(XEN) r12: 0000000000f1ee84   r13: ffff82c48046fe18   r14: 
ffff8300cfdfc000
(XEN) r15: 0000000000000002   cr0: 000000008005003b   cr4: 
00000000000026f0
(XEN) cr3: 000000002c9dd000   cr2: ffff8800c903b218
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48046fe78:
(XEN)    ffff82c48018f1e2 ffff830128f60330 ffff82c48012181a 
ffffffffffffffff
(XEN)    ffff830128f60330 ffff82c48012181a ffff82c480483280 
0000000000000000
(XEN)    0000000000000000 ffff82c480483280 00000669000006f0 
0000000000000002
(XEN)    ffff82c48046ff28 ffff82c480266c00 ffff82c48046ff28 
ffff82c48046fe18
(XEN)    ffff8300cfdfc000 0000000000000002 ffff82c48014d73e 
ffff8300cfdfc000
(XEN)    ffff8300cfc66000 00000000ffffffff 0000000000000000 
0000000000000000
(XEN)    ffffffffffffffff ffffffff806f5fd8 ffffffff80751100 
0000000000000000
(XEN)    0000000000000246 0000000000000001 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffffffff800033aa 0000000000000000 
0000000000000000
(XEN)    0000000000000001 0000010000000000 ffffffff800033aa 
000000000000e033
(XEN)    0000000000000246 ffffffff806f5f58 000000000000e02b 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    ffff8300cfdfc000
(XEN) Xen call trace:
(XEN)    [<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN)    [<ffff82c48018f1e2>] acpi_processor_idle+0x2d2/0x6d0
(XEN)    [<ffff82c48012181a>] timer_softirq_action+0x21a/0x360
(XEN)    [<ffff82c48012181a>] timer_softirq_action+0x21a/0x360
(XEN)    [<ffff82c48014d73e>] idle_loop+0x2e/0x90
(XEN) 
(XEN) No guest context (CPU0 is idle).
(XEN) 
(XEN) 'd' pressed -> dumping registers
(XEN) 
(XEN) *** Dumping CPU1 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff82c48017bd95>] x86_emulate+0x2e5/0x100b0
(XEN) RFLAGS: 0000000000000282   CONTEXT: hypervisor
(XEN) rax: 00000000000000dd   rbx: 0000000000000000   rcx: 
0000000000000000
(XEN) rdx: 00000000000000c8   rsi: ffff830128f6fc01   rdi: 
ffff830128f6fad1
(XEN) rbp: ffff830128f6fbe8   rsp: ffff830128f6f788   r8: ffff830128f6fbe8
(XEN) r9:  0000000000000000   r10: 0000000000000001   r11: 
00000000001223e8
(XEN) r12: 00000000000000dd   r13: 0000000000000004   r14: 
ffff82c480230d80
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 
00000000000426f0
(XEN) cr3: 0000000127230000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff830128f6f788:
(XEN)    ffff82c48022aae0 0000000000000000 ffff82c400000000 
ffff82c480136500
(XEN)    00000008802667c0 0000000000000086 ffff82c400000004 
ffff82c4ffffffff
(XEN)    000000c800000096 ffff830128f6f848 ffff830128f6f958 
0000000400000002
(XEN)    ffff830128f6fad0 000000008048bb7f 000000c800000000 
0000000000303b90
(XEN)    00ff82c48022aae0 0000000100000000 ffff82c48022ab36 
00000000801365c0
(XEN)    3830613362303106 ffff82c480136166 ffff82c48022aa06 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000303e44 00000000c0821d53 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000303d9c 
00000000000b8b98
(XEN)    00000000c082243e 0000000000303b90 0000000000303d9c 
000000f100000003
(XEN)    00000000c081745d 0000000000000000 0000000000010046 
0000000000303904
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffff8300cfa4a000 00000000000c0817 
ffff82c4801480ea
(XEN)    0000000000000018 00000000c081745c ffff830128f6faa4 
ffff8300cfa4bab8
(XEN)    00000000000c0817 ffff8300cfa4a000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffff82c480149743 ffff830100000003 
ffff814000000028
(XEN)    0000000000000001 ffff82c480149743 ffff830128f6fae4 
ffff830128f6fca8
(XEN)    0000000000000001 ffff82c4801d2daa 0000000000000a1e 
000000011fe3b000
(XEN)    0000000000000a04 ffff830128f6ff28 0000000000000001 
0000000000a1e000
(XEN) Xen call trace:
(XEN)    [<ffff82c48017bd95>] x86_emulate+0x2e5/0x100b0
(XEN)    [<ffff82c480136500>] serial_putc+0x100/0x130
(XEN)    [<ffff82c480136166>] __serial_putc+0x86/0x190
(XEN)    [<ffff82c4801480ea>] apic_timer_interrupt+0x2a/0x30
(XEN)    [<ffff82c480149743>] smp_apic_timer_interrupt+0x53/0xa0
(XEN)    [<ffff82c480149743>] smp_apic_timer_interrupt+0x53/0xa0
(XEN)    [<ffff82c4801d2daa>] sh_gva_to_gfn__guest_2+0x10a/0x170
(XEN)    [<ffff82c4801a051b>] __hvm_copy+0x33b/0x3e0
(XEN)    [<ffff82c4801d5c19>] sh_page_fault__guest_2+0x11f9/0x1880
(XEN)    [<ffff82c480149743>] smp_apic_timer_interrupt+0x53/0xa0
(XEN)    [<ffff82c4801480ea>] apic_timer_interrupt+0x2a/0x30
(XEN)    [<ffff82c4801bc973>] vmx_vmexit_handler+0xe33/0x1bb0
(XEN)    [<ffff82c480119b2d>] _csched_cpu_pick+0xfd/0x350
(XEN)    [<ffff82c4801b0893>] pt_restore_timer+0x23/0xb0
(XEN)    [<ffff82c4801afbd9>] vpic_update_int_output+0x29/0x80
(XEN)    [<ffff82c4801a7b93>] hvm_isa_irq_deassert+0x93/0xa0
(XEN)    [<ffff82c4801b0a79>] pt_update_irq+0x159/0x1f0
(XEN)    [<ffff82c4801518bd>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c4801a7af5>] hvm_isa_irq_assert+0x95/0xa0
(XEN)    [<ffff82c4801ae2a0>] vlapic_accept_pic_intr+0x20/0x30
(XEN)    [<ffff82c4801a8184>] hvm_vcpu_has_pending_irq+0x24/0x60
(XEN)    [<ffff82c4801b5cd8>] vmx_intr_assist+0x1d8/0x1f0
(XEN)    [<ffff82c4801b5923>] vmx_asm_do_vmentry+0x0/0xdd
(XEN) 
(XEN) *** Dumping CPU1 guest state (d1:v0): ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0008:[<00000000c081745c>]
(XEN) RFLAGS: 0000000000010046   CONTEXT: hvm guest
(XEN) rax: 0000000000303d9c   rbx: 00000000c0821d53   rcx: 
00000000000b8b98
(XEN) rdx: 00000000c082243e   rsi: 0000000000303b90   rdi: 
0000000000303d9c
(XEN) rbp: 0000000000303e44   rsp: 0000000000303904   r8: 0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080000011   cr4: 
0000000000000200
(XEN) cr3: 0000000000a3b000   cr2: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008
(XEN) 
(XEN) *** Dumping CPU0 host state: ***
(XEN) ----[ Xen-4.0.0_21091_06-0.1.1  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: 0000000000000003   rbx: ffff830128f26c20   rcx: 
ffff82c480483280
(XEN) rdx: 0000000000000808   rsi: 0000000000000000   rdi: 
ffff830128f26c80
(XEN) rbp: ffff830128f26c80   rsp: ffff82c48046fe78   r8: ffff830128f60348
(XEN) r9:  0000026435d7983e   r10: 0000000000000000   r11: 
ffff8300cfdfa060
(XEN) r12: 0000000000cccdfb   r13: ffff82c48046fe18   r14: 
ffff8300cfdfc000
(XEN) r15: 0000000000000002   cr0: 000000008005003b   cr4: 
00000000000026f0
(XEN) cr3: 000000002c9dd000   cr2: ffff8800c903b218
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48046fe78:
(XEN)    ffff82c48018f1e2 ffff82c4802660a8 ffff8300cfdfc000 
ffffffffffffffff
(XEN)    ffff830128f60330 ffff82c48012181a ffff8300cfdfc000 
0000000000000000
(XEN)    0000000000000000 ffff82c480483280 000025660000288a 
ffff82c48046ff28
(XEN)    ffff82c48046ff28 ffff82c480266c00 ffff82c48046ff28 
ffff82c48046fe18
(XEN)    ffff8300cfdfc000 0000000000000002 ffff82c48014d73e 
ffff8300cfdfc000
(XEN)    ffff8300cfc66000 00000000ffffffff 0000000000000000 
0000000000000000
(XEN)    ffffffffffffffff ffffffff806f5fd8 ffffffff80751100 
0000000000000000
(XEN)    0000000000000246 0000000000000001 0000000000000000 
0000000000000000
(XEN)    0000000000000000 ffffffff800033aa 0000000000000000 
0000000000000000
(XEN)    0000000000000001 0000010000000000 ffffffff800033aa 
000000000000e033
(XEN)    0000000000000246 ffffffff806f5f58 000000000000e02b 
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN)    ffff8300cfdfc000
(XEN) Xen call trace:
(XEN)    [<ffff82c48018ec2a>] acpi_idle_do_entry+0x2a/0x60
(XEN)    [<ffff82c48018f1e2>] acpi_processor_idle+0x2d2/0x6d0
(XEN)    [<ffff82c48012181a>] timer_softirq_action+0x21a/0x360
(XEN)    [<ffff82c48014d73e>] idle_loop+0x2e/0x90
(XEN) 
(XEN) No guest context (CPU0 is idle).
(XEN) 


This message is intended only for the use of the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the sender of this E-Mail by return E-Mail or by telephone. 

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-03  1:10   ` alarson
@ 2010-11-03  6:48     ` Keir Fraser
  2010-11-04 16:32       ` alarson
  0 siblings, 1 reply; 13+ messages in thread
From: Keir Fraser @ 2010-11-03  6:48 UTC (permalink / raw)
  To: alarson, xen-devel

On 03/11/2010 01:10, "alarson@ddci.com" <alarson@ddci.com> wrote:

> Thanks Keir, I was headed in that general direction, but I was hoping to
> get lucky.
> 
> I setup the serial cable and did a number of backtraces.  I'm hoping that
> the backtrace for the host is correlated with the client backtrace, but
> I'm suspicious.  Some of the backtraces I got list my client (e.g., on
> CPU1) but don't list a CPU1 host state.  In all cases my clint is doing
> what I expect (attempting to execute an fnsave instruction).
> 
> I slurped the relevant host stacktraces from the console debug log, and I
> don't see a pattern (again hoping to get lucky).  I'll put away my private
> OS shovel, and see if I can start fashioning a Xen shovel.  If you see
> something obvious below that looks suspicious, let me know, otherwise I'll
> see what my Xen shovel digs up.  Thanks again for your help.
> 
> Host CPU traces from the CPU's associated with my OS client.

Do a 'debug=y make' build of Xen and your backtraces will be precise as wthe
build will include frame pointers. Looking at these backtraces, they mostly
seem to be in vmx_intr_assist() doing interrupt delivery work (e.g.,
hvm_isa_irq_assert()). Perhaps you are stuck in some kind of interrupt
delivery loop?

 -- Keir

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-03  6:48     ` Keir Fraser
@ 2010-11-04 16:32       ` alarson
  2010-11-04 16:50         ` Keir Fraser
  0 siblings, 1 reply; 13+ messages in thread
From: alarson @ 2010-11-04 16:32 UTC (permalink / raw)
  To: xen-devel

History: fnsave in my 32-bit protected mode OS 'hangs' when paging is
turned on.

Keir Fraser <keir.xen@gmail.com> wrote on 11/03/2010 01:48:53 AM:

> Do a 'debug=y make' build of Xen and your backtraces will be precise as 
wthe
> build will include frame pointers. Looking at these backtraces, they 
mostly
> seem to be in vmx_intr_assist() doing interrupt delivery work (e.g.,
> hvm_isa_irq_assert()). Perhaps you are stuck in some kind of interrupt
> delivery loop?

It turns out that OpenSuse provides a debug xen kernel so I was able
to use that yesterday, but I wasn't making much progress until it
occurred to me that by pressing 'd', I'm sampling the stack of xen,
but I don't know for sure which client the xen stack represents.  When
I restricted the two clients (dom0 and my os) to exclusively their own
CPU, then, if you exclude the first trace below, a pattern seems to
emerge, and it would seem that I should start with sh_page_fault().

I think I can see a path forward, but I figured I'd post hoping to get
lucky again...

(XEN) Xen call trace:
(XEN)    [<ffff82c480120ecf>] _spin_is_locked+0x0/0x20
(XEN)    [<ffff82c4801b1df4>] vpic_get_highest_priority_irq+0x52/0xc7
(XEN)    [<ffff82c4801b1eaa>] vpic_update_int_output+0x41/0x9d
(XEN)    [<ffff82c4801b27b9>] vpic_irq_negative_edge+0xab/0xc0
(XEN)    [<ffff82c4801a9bea>] hvm_isa_irq_deassert+0x8a/0xa7
(XEN)    [<ffff82c4801b2ee0>] pt_update_irq+0x1b9/0x1f2
(XEN)    [<ffff82c4801b8296>] vmx_intr_assist+0x6f/0x229

(XEN) Xen call trace:
(XEN)    [<ffff82c4801bb952>] vmx_get_segment_register+0x182/0x3a5
(XEN)    [<ffff82c4801ced63>] hvm_get_seg_reg+0x43/0x4d
(XEN)    [<ffff82c4801cf11f>] shadow_init_emulation+0xa9/0x144
(XEN)    [<ffff82c4801dfaac>] sh_page_fault__guest_2+0x18c1/0x1d45
(XEN)    [<ffff82c4801bed2c>] vmx_vmexit_handler+0x7f5/0x1ab8

(XEN) Xen call trace:
(XEN)    [<ffff82c4801d29bd>] sh_remove_write_access+0xcce/0xce0
(XEN)    [<ffff82c4801dfef8>] sh_page_fault__guest_2+0x1d0d/0x1d45
(XEN)    [<ffff82c4801bed2c>] vmx_vmexit_handler+0x7f5/0x1ab8

(XEN) Xen call trace:
(XEN)    [<ffff82c4801faacd>] __mfn_valid+0x3e/0x58
(XEN)    [<ffff82c4801d80b2>] l1e_propagate_from_guest+0xd2/0x644
(XEN)    [<ffff82c4801df0d1>] sh_page_fault__guest_2+0xee6/0x1d45
(XEN)    [<ffff82c4801bed2c>] vmx_vmexit_handler+0x7f5/0x1ab8

(XEN) Xen call trace:
(XEN)    [<ffff82c4801faad1>] __mfn_valid+0x42/0x58
(XEN)    [<ffff82c4801df0f7>] sh_page_fault__guest_2+0xf0c/0x1d45
(XEN)    [<ffff82c4801bed2c>] vmx_vmexit_handler+0x7f5/0x1ab8

(XEN) Xen call trace:
(XEN)    [<ffff82c4801faa90>] __mfn_valid+0x1/0x58
(XEN)    [<ffff82c4801de6e8>] sh_page_fault__guest_2+0x4fd/0x1d45
(XEN)    [<ffff82c4801bed2c>] vmx_vmexit_handler+0x7f5/0x1ab8

(XEN) Xen call trace:
(XEN)    [<ffff82c48017f5e0>] x86_emulate+0x2c/0xefc8
(XEN)    [<ffff82c4801dfab7>] sh_page_fault__guest_2+0x18cc/0x1d45
(XEN)    [<ffff82c4801bed2c>] vmx_vmexit_handler+0x7f5/0x1ab8

(XEN) Xen call trace:
(XEN)    [<ffff82c4801caae8>] set_ad_bits+0x0/0x34
(XEN)    [<ffff82c4801de6e8>] sh_page_fault__guest_2+0x4fd/0x1d45
(XEN)    [<ffff82c4801bed2c>] vmx_vmexit_handler+0x7f5/0x1ab8

(XEN) Xen call trace:
(XEN)    [<ffff82c4801be932>] vmx_vmexit_handler+0x3fb/0x1ab8

(XEN) Xen call trace:
(XEN)    [<ffff82c4801d769a>] sh_gva_to_gfn__guest_2+0x8/0x168
(XEN)    [<ffff82c4801a20ef>] __hvm_copy+0xe7/0x3ef
(XEN)    [<ffff82c4801a4fd9>] hvm_fetch_from_guest_virt_nofault+0x46/0x48
(XEN)    [<ffff82c4801cf196>] shadow_init_emulation+0x120/0x144
(XEN)    [<ffff82c4801dfaac>] sh_page_fault__guest_2+0x18c1/0x1d45
(XEN)    [<ffff82c4801bed2c>] vmx_vmexit_handler+0x7f5/0x1ab8

(XEN) Xen call trace:
(XEN)    [<ffff82c4801be932>] vmx_vmexit_handler+0x3fb/0x1ab8

(XEN) Xen call trace:
(XEN)    [<ffff82c4801ced20>] hvm_get_seg_reg+0x0/0x4d
(XEN)    [<ffff82c4801dfaac>] sh_page_fault__guest_2+0x18c1/0x1d45
(XEN)    [<ffff82c4801bed2c>] vmx_vmexit_handler+0x7f5/0x1ab8


This message is intended only for the use of the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the sender of this E-Mail by return E-Mail or by telephone. 

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-04 16:32       ` alarson
@ 2010-11-04 16:50         ` Keir Fraser
  2010-11-12 21:27           ` alarson
  0 siblings, 1 reply; 13+ messages in thread
From: Keir Fraser @ 2010-11-04 16:50 UTC (permalink / raw)
  To: alarson, xen-devel

On 04/11/2010 16:32, "alarson@ddci.com" <alarson@ddci.com> wrote:

> It turns out that OpenSuse provides a debug xen kernel so I was able
> to use that yesterday, but I wasn't making much progress until it
> occurred to me that by pressing 'd', I'm sampling the stack of xen,
> but I don't know for sure which client the xen stack represents.  When
> I restricted the two clients (dom0 and my os) to exclusively their own
> CPU, then, if you exclude the first trace below, a pattern seems to
> emerge, and it would seem that I should start with sh_page_fault().

Hm, well maybe. sh_page_fault() is the entry point to one of the most
complex pieces of the hypervisor, so you're likely to find it more of a
sticky tar pit than a source of salvation. It may be involved, but you might
want to wade in armed with some more debug info first so that your search
has some greater focus.

I would dispense with sampling the 'd' key -- clearly a bunch of stuff is
happening -- and instead go for printk() tracing in the vmexit handler
vmx_vmexit_handler(). Perhaps enable this tracing only when the saved guest
EIP happens to be the address that you know your 'hanging' instruction
resides at, so that the logging console only gets noisy when the bad
situation occurs. From that function you can log useful things like: what is
the vmexit for? Faulting linear address (if a page fault)? Etc etc. See if
there is a (looping) pattern to the vmexit reasons and see thereby if you
can work out the overall livelock loop you are experiencing (this assumes
your problem is a livelock loop of some kind, which seems quite possible
from what you've seen so far).

 -- Keir

> I think I can see a path forward, but I figured I'd post hoping to get
> lucky again...

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-04 16:50         ` Keir Fraser
@ 2010-11-12 21:27           ` alarson
  2010-11-12 22:34             ` Keir Fraser
  0 siblings, 1 reply; 13+ messages in thread
From: alarson @ 2010-11-12 21:27 UTC (permalink / raw)
  To: xen-devel

History: fnsave in my 32-bit protected mode OS 'hangs' when paging is
turned on.

Keir Fraser <keir.xen@gmail.com> wrote on 11/04/2010 11:50:52 AM:

> On 04/11/2010 16:32, "alarson@ddci.com" <alarson@ddci.com> wrote:
> 
AL> ... When I restricted the two clients (dom0 and my os) to
AL> exclusively their own CPU, then, if you exclude the first trace
AL> below, a pattern seems to emerge, and it would seem that I should
AL> start with sh_page_fault().

KF> Hm, well maybe. sh_page_fault() is the entry point to one of the
KF> most complex pieces of the hypervisor, so you're likely to find it
KF> more of a sticky tar pit than a source of salvation. It may be
KF> involved, but you might want to wade in armed with some more debug
KF> info first so that your search has some greater focus.

KF> I would dispense with sampling the 'd' key -- clearly a bunch of
KF> stuff is happening -- and instead go for printk() tracing in the
KF> vmexit handler vmx_vmexit_handler(). ...

You advice was most helpful once again.  However, I'm now even more
confused than I was before.  I downloaded the OpenSuse source RPM for
xen (xen-4.0.0_21091_06-0.1.1.src.rpm) and added printk debugging
statements as suggested.  The following are the outputs I eventually
found most useful.  All expressions denote existing source code
variables:

(XEN) sh_page_fault va=303b90, regs->error_code=3
(XEN) x86_emulate: b=dd, modrm=31, modrm_reg=6
(XEN) sh_page_fault called x86_emulate va=303b90,result=1
(XEN) vmx_vmexit_handler reason=0
(XEN) vmx_vmexit_handler intr_info=80000b0e, vector=e

The key line is:
(XEN) x86_emulate: b=dd, modrm=31, modrm_reg=6

The instruction I'm executing is:

c0817476: dd 31 fnsave (%ecx)

Opcode 0xDD and the modrm_reg (0x6) matches.  The source for
x86_emulate() opcode 0xdd has no case for modrm_reg=0x6, so
x86_emulate() returns X86EMUL_UNHANDLEABLE (i.e., 1).

What I don't understand is why fnsave works before I turn on paging
and doesn't afterwords.

Am I just looking at an unimplemented feature, or is there something
more nefarious going on?

I haven't looked at why load task register (ltr) and 
accesses to the APIC behave similarly.  Does Xen assume
fnsave, ltr, etc. happen with paging turned off?


This message is intended only for the use of the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the sender of this E-Mail by return E-Mail or by telephone. 

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-12 21:27           ` alarson
@ 2010-11-12 22:34             ` Keir Fraser
  2010-11-15 22:22               ` alarson
  0 siblings, 1 reply; 13+ messages in thread
From: Keir Fraser @ 2010-11-12 22:34 UTC (permalink / raw)
  To: alarson, xen-devel

On 12/11/2010 21:27, "alarson@ddci.com" <alarson@ddci.com> wrote:

> You advice was most helpful once again.  However, I'm now even more
> confused than I was before.  I downloaded the OpenSuse source RPM for
> xen (xen-4.0.0_21091_06-0.1.1.src.rpm) and added printk debugging
> statements as suggested.  The following are the outputs I eventually
> found most useful.  All expressions denote existing source code
> variables:
> 
> (XEN) sh_page_fault va=303b90, regs->error_code=3
> (XEN) x86_emulate: b=dd, modrm=31, modrm_reg=6

So you took a vmexit on a page fault while trying to execute fnsave. The
shadow fault handler then tries to emulate the faulting instruction (fnsave)
presumably because it thinks the instruction is attempting to write to a
page that is read-only in the shadow page tables (generally that would be
because Xen thinks the instruction is trying to write to a page-table page).

> (XEN) sh_page_fault called x86_emulate va=303b90,result=1

Emulation failed, because the emulator does not support fnsave. The shadow
fault handler will take this as a hint that the instrction is not writing to
a page table page, and hence flushes some shadow page table state which is
possibly out of date. This makes the previously read-lonely page writable,
and we expect that on return to guest context the CPU will execute the
previously-faulting instruction without problem.

Since you end up in a loop not progressing past the fnsave instruction, it
seems quite likely that you have a bug and are writing to a pagetable page.
In fact a pagetable page that maps something that is needed to execute the
fnsave instruction. You need that page to both be writable (so that fnsave
can write its data) and read-only (because it is a pagetable page that maps
something that is used by the fnsave instruction) and so I'm guessing you
end up in an endless loop with that page flipping between being read-only
and read-write in the shadow page table.

Hope that makes sense. :-)

> I haven't looked at why load task register (ltr) and
> accesses to the APIC behave similarly.  Does Xen assume
> fnsave, ltr, etc. happen with paging turned off?

In general those instructions would not even vmexit. The CPU would just
execute them directly. The reason you are having problems when you turn
paging on is probably because of some guest bug where data structures
written yo by FNSAVE, LTR, etc are overlapping with your page tables.

 -- Keir

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-12 22:34             ` Keir Fraser
@ 2010-11-15 22:22               ` alarson
  2010-11-16 17:09                 ` Tim Deegan
  2010-11-19 17:32                 ` alarson
  0 siblings, 2 replies; 13+ messages in thread
From: alarson @ 2010-11-15 22:22 UTC (permalink / raw)
  To: xen-devel

Keir Fraser <keir.xen@gmail.com> wrote on 11/12/2010 04:34:05 PM:

KF> On 12/11/2010 21:27, "alarson@ddci.com" <alarson@ddci.com> wrote:

...
AL> (XEN) sh_page_fault va=303b90, regs->error_code=3
AL> (XEN) x86_emulate: b=dd, modrm=31, modrm_reg=6

...

KF> Since you end up in a loop not progressing past the fnsave
KF> instruction, it seems quite likely that you have a bug and are
KF> writing to a pagetable page.  In fact a pagetable page that maps
KF> something that is needed to execute the fnsave instruction. You
KF> need that page to both be writable (so that fnsave can write its
KF> data) and read-only (because it is a pagetable page that maps
KF> something that is used by the fnsave instruction) and so I'm
KF> guessing you end up in an endless loop with that page flipping
KF> between being read-only and read-write in the shadow page table.

KF> Hope that makes sense. :-)

I understand what you are saying, but I'm confident that's not what's
happening.  Just to be sure, I modified one of our analysis programs
to list all the page tables and the PDT and confirmed that the page at
virtual address 303b90 doesn't map to any of them.  In fact I verified
that none of the present pages maps to any PT or the PDT.

If not, it seems like I need to find out who is calling sh_page_fault,
and that leads me to sh_paging_mode which seems even more of a sticky
tar pit than sh_page_fault.  I think I should be looking at
emulate_gva_to_mfn() and propagate_page_fault(), but I haven't spent
enough time with the code to be sure.  If you have other ideas, I'd
love to hear them.


This message is intended only for the use of the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the sender of this E-Mail by return E-Mail or by telephone. 

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-15 22:22               ` alarson
@ 2010-11-16 17:09                 ` Tim Deegan
  2010-11-19 17:32                 ` alarson
  1 sibling, 0 replies; 13+ messages in thread
From: Tim Deegan @ 2010-11-16 17:09 UTC (permalink / raw)
  To: alarson; +Cc: xen-devel

Hi, 

At 22:22 +0000 on 15 Nov (1289859751), alarson@ddci.com wrote:
> I understand what you are saying, but I'm confident that's not what's
> happening.  Just to be sure, I modified one of our analysis programs
> to list all the page tables and the PDT and confirmed that the page at
> virtual address 303b90 doesn't map to any of them.  In fact I verified
> that none of the present pages maps to any PT or the PDT.
> 
> If not, it seems like I need to find out who is calling sh_page_fault,
> and that leads me to sh_paging_mode which seems even more of a sticky
> tar pit than sh_page_fault.  I think I should be looking at
> emulate_gva_to_mfn() and propagate_page_fault(), but I haven't spent
> enough time with the code to be sure.  If you have other ideas, I'd
> love to hear them.

I think the most useful thing, since you know the virtual address, is to
instrument the sh_page_fault() function to see which path it takes, and
in particular why it returns what it does.  Adding printks guarded by
"if ( va == 0x303b90 )" should be OK.  It would also be interesting to
instrument guest_walk_pages() for just that address, say by adding a
print_gw() call at the end.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-15 22:22               ` alarson
  2010-11-16 17:09                 ` Tim Deegan
@ 2010-11-19 17:32                 ` alarson
  2010-11-19 18:01                   ` Keir Fraser
  1 sibling, 1 reply; 13+ messages in thread
From: alarson @ 2010-11-19 17:32 UTC (permalink / raw)
  To: xen-devel

> Keir Fraser <keir.xen@gmail.com> wrote on 11/12/2010 04:34:05 PM:
> KF> On 12/11/2010 21:27, "alarson@ddci.com" <alarson@ddci.com> wrote:
> ...
AL> (XEN) sh_page_fault va=303b90, regs->error_code=3
AL> (XEN) x86_emulate: b=dd, modrm=31, modrm_reg=6
> ...
KF> Since you end up in a loop not progressing past the fnsave
KF> instruction, it seems quite likely that you have a bug and are
KF> writing to a pagetable page.  In fact a pagetable page that maps
KF> something that is needed to execute the fnsave instruction. You
KF> need that page to both be writable (so that fnsave can write its
KF> data) and read-only (because it is a pagetable page that maps
KF> something that is used by the fnsave instruction) and so I'm
KF> guessing you end up in an endless loop with that page flipping
KF> between being read-only and read-write in the shadow page table.

KF> Hope that makes sense. :-)

AL> I understand what you are saying, but I'm confident that's not what's
AL> happening.  Just to be sure, I modified one of our analysis programs
AL> to list all the page tables and the PDT and confirmed that the page at
AL> virtual address 303b90 doesn't map to any of them.  In fact I verified
AL> that none of the present pages maps to any PT or the PDT.

Just to report final resolution.  The problem turned out to be that
the destination of the fnsave was to a page specifying read only
access and the OS was depending on CR0.WP=0 to permit write access in
kernel mode.  Xen doesn't allow this configuration unless using HAP
(Hardware Assisted Paging).

Many thanks to Keir for his invaluable assistance with this.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-19 17:32                 ` alarson
@ 2010-11-19 18:01                   ` Keir Fraser
  2010-11-22  9:51                     ` Tim Deegan
  0 siblings, 1 reply; 13+ messages in thread
From: Keir Fraser @ 2010-11-19 18:01 UTC (permalink / raw)
  To: alarson, xen-devel

On 19/11/2010 17:32, "alarson@ddci.com" <alarson@ddci.com> wrote:

> Just to report final resolution.  The problem turned out to be that
> the destination of the fnsave was to a page specifying read only
> access and the OS was depending on CR0.WP=0 to permit write access in
> kernel mode.  Xen doesn't allow this configuration unless using HAP
> (Hardware Assisted Paging).

To be precise, we have a go at emulating the instruction, but our emulator
is incomplete. Running a whole OS in CR0.WP=0 mode is pretty much doomed if
running on Xen without HAP. I think there is some supported OS that does do
some small amount of stuff with CR0.WP=0, and we are complete enough to work
in that case.

 -- Keir

> Many thanks to Keir for his invaluable assistance with this.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: fxsave, fnsave, ltr hang for guest OS.
  2010-11-19 18:01                   ` Keir Fraser
@ 2010-11-22  9:51                     ` Tim Deegan
  0 siblings, 0 replies; 13+ messages in thread
From: Tim Deegan @ 2010-11-22  9:51 UTC (permalink / raw)
  To: Keir Fraser; +Cc: xen-devel, alarson

At 18:01 +0000 on 19 Nov (1290189698), Keir Fraser wrote:
> On 19/11/2010 17:32, "alarson@ddci.com" <alarson@ddci.com> wrote:
> 
> > Just to report final resolution.  The problem turned out to be that
> > the destination of the fnsave was to a page specifying read only
> > access and the OS was depending on CR0.WP=0 to permit write access in
> > kernel mode.  Xen doesn't allow this configuration unless using HAP
> > (Hardware Assisted Paging).
> 
> To be precise, we have a go at emulating the instruction, but our emulator
> is incomplete. Running a whole OS in CR0.WP=0 mode is pretty much doomed if
> running on Xen without HAP. I think there is some supported OS that does do
> some small amount of stuff with CR0.WP=0, and we are complete enough to work
> in that case.

To be precise, the shadow code/emulator can run various flavours of
Windows when anti-virus software briefly turns off CR0.WP in order to
overwrite protected kernel structures (IDT &c).  That only involves
straightforward memory writes in an OS that otherwise obeys its own 
pagetable restrictions.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2010-11-22  9:51 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-11-02 18:13 fxsave, fnsave, ltr hang for guest OS alarson
2010-11-02 20:16 ` Keir Fraser
2010-11-03  1:10   ` alarson
2010-11-03  6:48     ` Keir Fraser
2010-11-04 16:32       ` alarson
2010-11-04 16:50         ` Keir Fraser
2010-11-12 21:27           ` alarson
2010-11-12 22:34             ` Keir Fraser
2010-11-15 22:22               ` alarson
2010-11-16 17:09                 ` Tim Deegan
2010-11-19 17:32                 ` alarson
2010-11-19 18:01                   ` Keir Fraser
2010-11-22  9:51                     ` Tim Deegan

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.