All of lore.kernel.org
 help / color / mirror / Atom feed
* Floating point instructions (e.g. cvtsi2sd) incorrect result
@ 2016-08-04 20:40 Alexandru Duţu
  2016-08-08  2:26 ` Yang Zhang
  0 siblings, 1 reply; 4+ messages in thread
From: Alexandru Duţu @ 2016-08-04 20:40 UTC (permalink / raw)
  To: kvm

Hi everyone,

I am using KVM to run a virtual machine (x86) without having a guest
OS. So the machine state is set for long mode before calling KVM_RUN
and I am trapping out of KVM on every syscall  to emulate the syscall
in the program that uses KVM. Also I am trapping out of KVM on
pagefaults to deal with stack growth.

This has been working great for integer workloads, however I have
encoutered some workloads which are using floating point instructions,
for example cvtsi2sd to cast an integer to a double, and their result
is incorrect.

Recently I have discovered that KVM does FPU context save on-demand,
meaning when the virtual machine actually uses the FPU [1]. I am
wondering if this machanism has some baked in assumptions about
running a virtual machine with a guest OS and if a guest OS is not
present the FPU state might be altered? Also I have not seen any #NM
exceptions, these usually trap out of KVM.

The FPU state that I am using [2] seems valid and it does not change
when trapping out of KVM for syscalls and pagefaults. In addition, I
have tried running with multiple linux versions 2.6.29, 3.13 and 4.0.4
and all of them give incorrect floating point instructions result,
however for different instructions.

Thank you,
-- 
Alex

[1] http://www.linux-kvm.org/images/e/ea/2010-forum-mtosatti_walkthrough_entry_exit.pdf
[2] FPU related machine state before calling KVM_RUN:
 Efer efer = 0;
 efer.sce = 1; // Enable system call extensions.
 efer.lme = 1; // Enable long mode.
 efer.lma = 1; // Activate long mode.
 efer.nxe = 0; // Enable nx support.
 efer.svme = 1; // Enable svm support for now.
 efer.ffxsr = 0; // Turn on fast fxsave and fxrstor.

 //Set up the registers that describe the operating mode.
 CR0 cr0 = 0;
 cr0.pg = 1; // Turn on paging.
 cr0.cd = 0; // Don't disable caching.
 cr0.nw = 0; // This is bit is defined to be ignored.
 cr0.am = 1; // No alignment checking
 cr0.wp = 1; // Supervisor mode can write read only pages
 cr0.ne = 1;
 cr0.et = 1; // This should always be 1
 cr0.ts = 0; // We don't do task switching, so causing fp exceptions
             // would be pointless.
 cr0.em = 0; // Allow x87 instructions to execute natively.
 cr0.mp = 1; // This doesn't really matter, but the manual suggests
             // setting it to one.
 cr0.pe = 1; // We're definitely in protected mode.

 CR4 cr4 = 0;
 //Turn on pae.
 cr4.osxsave = 1; // Enable XSAVE and Proc Extended States
 cr4.osxmmexcpt = 1; // Operating System Unmasked Exception
 cr4.osfxsr = 1; // Operating System FXSave/FSRSTOR Support
 cr4.pce = 0; // Performance-Monitoring Counter Enable
 cr4.pge = 0; // Page-Global Enable
 cr4.mce = 0; // Machine Check Enable
 cr4.pae = 1; // Physical-Address Extension
 cr4.pse = 0; // Page Size Extensions
 cr4.de = 0; // Debugging Extensions
 cr4.tsd = 0; // Time Stamp Disable
 cr4.pvi = 0; // Protected-Mode Virtual Interrupts
 cr4.vme = 0; // Virtual-8086 Mode Extensions

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Floating point instructions (e.g. cvtsi2sd) incorrect result
  2016-08-04 20:40 Floating point instructions (e.g. cvtsi2sd) incorrect result Alexandru Duţu
@ 2016-08-08  2:26 ` Yang Zhang
  2016-08-10 16:38   ` Alexandru Duţu
  0 siblings, 1 reply; 4+ messages in thread
From: Yang Zhang @ 2016-08-08  2:26 UTC (permalink / raw)
  To: Alexandru Duţu, kvm

On 2016/8/5 4:40, Alexandru Duţu wrote:
> Hi everyone,
>
> I am using KVM to run a virtual machine (x86) without having a guest
> OS. So the machine state is set for long mode before calling KVM_RUN
> and I am trapping out of KVM on every syscall  to emulate the syscall
> in the program that uses KVM. Also I am trapping out of KVM on
> pagefaults to deal with stack growth.
>
> This has been working great for integer workloads, however I have
> encoutered some workloads which are using floating point instructions,
> for example cvtsi2sd to cast an integer to a double, and their result
> is incorrect.
>
> Recently I have discovered that KVM does FPU context save on-demand,
> meaning when the virtual machine actually uses the FPU [1]. I am

Latest KVM will use eagerfpu if host is using eagerfpu.

> wondering if this machanism has some baked in assumptions about
> running a virtual machine with a guest OS and if a guest OS is not
> present the FPU state might be altered? Also I have not seen any #NM
> exceptions, these usually trap out of KVM.

#NM exceptions will trap into KVM only when current hardware FPU state 
not belongs to the running VCPU, i.e. first access to FPU.

>
> The FPU state that I am using [2] seems valid and it does not change
> when trapping out of KVM for syscalls and pagefaults. In addition, I

Have you check the state before vm entry?

> have tried running with multiple linux versions 2.6.29, 3.13 and 4.0.4
> and all of them give incorrect floating point instructions result,
> however for different instructions.
>
> Thank you,
>


-- 
Yang
Alibaba Cloud Computing

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Floating point instructions (e.g. cvtsi2sd) incorrect result
  2016-08-08  2:26 ` Yang Zhang
@ 2016-08-10 16:38   ` Alexandru Duţu
       [not found]     ` <a9ae5ca6-d9d7-fda5-1418-cb4cfbfcfeaf@gmail.com>
  0 siblings, 1 reply; 4+ messages in thread
From: Alexandru Duţu @ 2016-08-10 16:38 UTC (permalink / raw)
  To: Yang Zhang; +Cc: kvm

Hi Yang,

Yes, I have. In particular the FPU state is the following:

FPU registers (XSave):
    fcw: 0x0
    fsw: 0x0 (top: 0, conditions: , exceptions:  )
    ftwx: 0xff
    last_opcode: 0x0
    last_ip: 0x0
    last_dp: 0x0
    mxcsr_mask: 0x0
    mxcsr: 0x1f80
    FP Stack:
        ST0/0: 0x00000000000000000000 (0)
        ST1/1: 0x00000000000000000000 (0)
        ST2/2: 0x00000000000000000000 (0)
        ST3/3: 0x00000000000000000000 (0)
        ST4/4: 0x00000000000000000000 (0)
        ST5/5: 0x00000000000000000000 (0)
        ST6/6: 0x00000000000000000000 (0)
        ST7/7: 0x00000000000000000000 (0)
    XMM registers:
        0: 0x00000000000000400000000000000000
        1: 0x00000000000000000000000000000000
        2: 0x00000000000000000000000000000000
        3: 0x00000000000000000000000000000000
        4: 0x00000000000000000000000000000000
        5: 0x00000000000000000000000000000000
        6: 0x00000000000000000000000000000000
        7: 0x00000000000000000000000000000000
        8: 0x00000000000000000000000000000000
        9: 0x00000000000000000000000000000000
        10: 0x00000000000000000000000000000000
        11: 0x00000000000000000000000000000000
        12: 0x00000000000000000000000000000000
        13: 0x00000000000000000000000000000000
        14: 0x00000000000000000000000000000000
        15: 0x00000000000000000000000000000000

Do you have particular registers you are wondering about?

Thank you,
Alex

On Sun, Aug 7, 2016 at 7:26 PM, Yang Zhang <yang.zhang.wz@gmail.com> wrote:
> On 2016/8/5 4:40, Alexandru Duţu wrote:
>>
>> Hi everyone,
>>
>> I am using KVM to run a virtual machine (x86) without having a guest
>> OS. So the machine state is set for long mode before calling KVM_RUN
>> and I am trapping out of KVM on every syscall  to emulate the syscall
>> in the program that uses KVM. Also I am trapping out of KVM on
>> pagefaults to deal with stack growth.
>>
>> This has been working great for integer workloads, however I have
>> encoutered some workloads which are using floating point instructions,
>> for example cvtsi2sd to cast an integer to a double, and their result
>> is incorrect.
>>
>> Recently I have discovered that KVM does FPU context save on-demand,
>> meaning when the virtual machine actually uses the FPU [1]. I am
>
>
> Latest KVM will use eagerfpu if host is using eagerfpu.
>
>> wondering if this machanism has some baked in assumptions about
>> running a virtual machine with a guest OS and if a guest OS is not
>> present the FPU state might be altered? Also I have not seen any #NM
>> exceptions, these usually trap out of KVM.
>
>
> #NM exceptions will trap into KVM only when current hardware FPU state not
> belongs to the running VCPU, i.e. first access to FPU.
>
>>
>> The FPU state that I am using [2] seems valid and it does not change
>> when trapping out of KVM for syscalls and pagefaults. In addition, I
>
>
> Have you check the state before vm entry?
>
>> have tried running with multiple linux versions 2.6.29, 3.13 and 4.0.4
>> and all of them give incorrect floating point instructions result,
>> however for different instructions.
>>
>> Thank you,
>>
>
>
> --
> Yang
> Alibaba Cloud Computing



-- 
Alex

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Floating point instructions (e.g. cvtsi2sd) incorrect result
       [not found]     ` <a9ae5ca6-d9d7-fda5-1418-cb4cfbfcfeaf@gmail.com>
@ 2016-08-12 18:52       ` Alexandru Duţu
  0 siblings, 0 replies; 4+ messages in thread
From: Alexandru Duţu @ 2016-08-12 18:52 UTC (permalink / raw)
  To: Yang Zhang; +Cc: kvm

Leaving the data registers (i.e. XMM) aside -- the FP stack remains
unchanged -- the mxcsr is changing in the following way:

59143906000: system.cpu0: T0 : syscall clock_getres called w/arguments
4, 140737488348384, 140737488350520, 140737488347808, 140737341873792,
140737341867960
info:   mxcsr: 0x1f80
        (enter VM)
info:   mxcsr: 0x1f80
        (exit VM)
info:   mxcsr: 0x1f80
        (enter VM)
info:   mxcsr: 0x1f84
        (exit VM)
info:   mxcsr: 0x1f84
        (enter VM)
info:   mxcsr: 0x1f84
        (exit VM)
59282397000: system.cpu0: T0 : syscall clock_gettime called
w/arguments 4, 140737488348304, 140737488350520, 140737488347728,
140737341873792, 14073734186796
info:   mxcsr: 0x1f84
        (enter VM)
info:   mxcsr: 0x1f84
        (exit VM)
59324467500: system.cpu0: T0 : syscall clock_gettime called
w/arguments 4, 140737488348304, 0, 140737488347728, 140737341873792,
140737341867960
info:   mxcsr: 0x1f84
        (enter VM)
info:   mxcsr: 0x1fa4
        (exit VM)
59366650500: system.cpu0: T0 : syscall clock_gettime called
w/arguments 4, 140737488348304, 0, 140737488347728, 140737341873792,
140737341867960
info:   mxcsr: 0x1fa4
        (enter VM)
info:   mxcsr: 0x1fa5
        (exit VM)

Each of these mxcsr prints represent the content of mxcsr on entering
and exiting the VM, the first one being before entering. It seems that
bogus floting point results appear after the mxcsr is switched to
0x1f84, from 0x1f80 -- which is the default machine state used from
the begining of the execution. We can notice that all changes occur
when executing inside the VM. I will detail what seems to be happening
with the floating point unit based on these changes in the MXCSR:
0x1f80 -> 0x1f84 - overflow exception
0x1f84 -> 0x1fa4 - precision exception
0x1fa4 -> 0x1fa5 - invalid operand exception

I will be curious to know more about the interaction of KVM's eager
FPU context saving strategy and these changes in the FPU state. Please
let me know if you need more details.

Thank you,
Alex

On Thu, Aug 11, 2016 at 6:18 PM, Yang Zhang <yang.zhang.wz@gmail.com> wrote:
> On 2016/8/11 0:38, Alexandru Duţu wrote:
>>
>> Hi Yang,
>>
>> Yes, I have. In particular the FPU state is the following:
>
>
> I mean is the state same with the one after vmexit?
>
>
>>
>> FPU registers (XSave):
>>     fcw: 0x0
>>     fsw: 0x0 (top: 0, conditions: , exceptions:  )
>>     ftwx: 0xff
>>     last_opcode: 0x0
>>     last_ip: 0x0
>>     last_dp: 0x0
>>     mxcsr_mask: 0x0
>>     mxcsr: 0x1f80
>>     FP Stack:
>>         ST0/0: 0x00000000000000000000 (0)
>>         ST1/1: 0x00000000000000000000 (0)
>>         ST2/2: 0x00000000000000000000 (0)
>>         ST3/3: 0x00000000000000000000 (0)
>>         ST4/4: 0x00000000000000000000 (0)
>>         ST5/5: 0x00000000000000000000 (0)
>>         ST6/6: 0x00000000000000000000 (0)
>>         ST7/7: 0x00000000000000000000 (0)
>>     XMM registers:
>>         0: 0x00000000000000400000000000000000
>>         1: 0x00000000000000000000000000000000
>>         2: 0x00000000000000000000000000000000
>>         3: 0x00000000000000000000000000000000
>>         4: 0x00000000000000000000000000000000
>>         5: 0x00000000000000000000000000000000
>>         6: 0x00000000000000000000000000000000
>>         7: 0x00000000000000000000000000000000
>>         8: 0x00000000000000000000000000000000
>>         9: 0x00000000000000000000000000000000
>>         10: 0x00000000000000000000000000000000
>>         11: 0x00000000000000000000000000000000
>>         12: 0x00000000000000000000000000000000
>>         13: 0x00000000000000000000000000000000
>>         14: 0x00000000000000000000000000000000
>>         15: 0x00000000000000000000000000000000
>>
>> Do you have particular registers you are wondering about?
>>
>> Thank you,
>> Alex
>>
>> On Sun, Aug 7, 2016 at 7:26 PM, Yang Zhang <yang.zhang.wz@gmail.com>
>> wrote:
>>>
>>> On 2016/8/5 4:40, Alexandru Duţu wrote:
>>>>
>>>>
>>>> Hi everyone,
>>>>
>>>> I am using KVM to run a virtual machine (x86) without having a guest
>>>> OS. So the machine state is set for long mode before calling KVM_RUN
>>>> and I am trapping out of KVM on every syscall  to emulate the syscall
>>>> in the program that uses KVM. Also I am trapping out of KVM on
>>>> pagefaults to deal with stack growth.
>>>>
>>>> This has been working great for integer workloads, however I have
>>>> encoutered some workloads which are using floating point instructions,
>>>> for example cvtsi2sd to cast an integer to a double, and their result
>>>> is incorrect.
>>>>
>>>> Recently I have discovered that KVM does FPU context save on-demand,
>>>> meaning when the virtual machine actually uses the FPU [1]. I am
>>>
>>>
>>>
>>> Latest KVM will use eagerfpu if host is using eagerfpu.
>>>
>>>> wondering if this machanism has some baked in assumptions about
>>>> running a virtual machine with a guest OS and if a guest OS is not
>>>> present the FPU state might be altered? Also I have not seen any #NM
>>>> exceptions, these usually trap out of KVM.
>>>
>>>
>>>
>>> #NM exceptions will trap into KVM only when current hardware FPU state
>>> not
>>> belongs to the running VCPU, i.e. first access to FPU.
>>>
>>>>
>>>> The FPU state that I am using [2] seems valid and it does not change
>>>> when trapping out of KVM for syscalls and pagefaults. In addition, I
>>>
>>>
>>>
>>> Have you check the state before vm entry?
>>>
>>>> have tried running with multiple linux versions 2.6.29, 3.13 and 4.0.4
>>>> and all of them give incorrect floating point instructions result,
>>>> however for different instructions.
>>>>
>>>> Thank you,
>>>>
>>>
>>>
>>> --
>>> Yang
>>> Alibaba Cloud Computing
>>
>>
>>
>>
>
>
> --
> Yang
> Alibaba Cloud Computing



-- 
Alex

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2016-08-12 18:52 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-08-04 20:40 Floating point instructions (e.g. cvtsi2sd) incorrect result Alexandru Duţu
2016-08-08  2:26 ` Yang Zhang
2016-08-10 16:38   ` Alexandru Duţu
     [not found]     ` <a9ae5ca6-d9d7-fda5-1418-cb4cfbfcfeaf@gmail.com>
2016-08-12 18:52       ` Alexandru Duţu

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.