KVM PUSH ES size bug

KVM PUSH ES size bug

[Pedro Fonseca]

Hi,

During tests that we conducted on KVM, we noticed that executing a "PUSH 
%ES" instruction under KVM produces different results on both memory and 
the SP register depending on whether EPT support is enabled. With EPT 
the SP is reduced by 4 bytes (and the written value is 0-padded) but 
without EPT support it is only reduced by 2 bytes. The difference can be 
observed when the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).

The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and 
SP=0xFFE. Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The 
testing system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700 
CPU @ 3.60GHz.

The test case (https://pastebin.com/ZejdtGEk) produces the output 
bellow. Note that 0x10 is written to 0xFFA on EPT=1 but it's written to 
0xFFC on EPT=0.
> $ insmod kvm-intel.ko
> $ sudo ./reproduce-push_es
> Executing KVM_RUN
> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>   0000: 06 f4 00 00 00 00 00 00
>   0008: 00 00 00 00 00 00 00 00
>   0ff8: 00 00 10 00 00 00 00 00
>   1000: 00 00 00 00 00 00 00 00

> $ insmod kvm-intel.ko ept=0
> $ sudo ./reproduce-push_es
> Executing KVM_RUN
> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>   0000: 06 f4 00 00 00 00 00 00
>   0008: 00 00 00 00 00 00 00 00
>   0ff8: 00 00 00 00 10 00 00 00
>   1000: 00 00 00 00 00 00 00 00


Thanks,
Pedro

Re: KVM PUSH ES size bug

[Wanpeng Li]

2017-10-24 19:10 GMT+08:00 Pedro Fonseca <pfonseca@cs.washington.edu>:
> Hi,
>
> During tests that we conducted on KVM, we noticed that executing a "PUSH
> %ES" instruction under KVM produces different results on both memory and the
> SP register depending on whether EPT support is enabled. With EPT the SP is
> reduced by 4 bytes (and the written value is 0-padded) but without EPT
> support it is only reduced by 2 bytes. The difference can be observed when
> the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).
>
> The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and SP=0xFFE.
> Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The testing
> system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz.
>
> The test case (https://pastebin.com/ZejdtGEk) produces the output bellow.
> Note that 0x10 is written to 0xFFA on EPT=1 but it's written to 0xFFC on
> EPT=0.
>>
>> $ insmod kvm-intel.ko
>> $ sudo ./reproduce-push_es
>> Executing KVM_RUN
>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>   0000: 06 f4 00 00 00 00 00 00
>>   0008: 00 00 00 00 00 00 00 00
>>   0ff8: 00 00 10 00 00 00 00 00
>>   1000: 00 00 00 00 00 00 00 00
>
>
>> $ insmod kvm-intel.ko ept=0
>> $ sudo ./reproduce-push_es
>> Executing KVM_RUN
>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>   0000: 06 f4 00 00 00 00 00 00
>>   0008: 00 00 00 00 00 00 00 00
>>   0ff8: 00 00 00 00 10 00 00 00
>>   1000: 00 00 00 00 00 00 00 00
>

Easy to reproduce, I will have a look.

Regards,
Wanpeng Li

Re: KVM PUSH ES size bug

[Wanpeng Li]

Cc Radim, Nadav,
2017-10-24 19:10 GMT+08:00 Pedro Fonseca <pfonseca@cs.washington.edu>:
> Hi,
>
> During tests that we conducted on KVM, we noticed that executing a "PUSH
> %ES" instruction under KVM produces different results on both memory and the
> SP register depending on whether EPT support is enabled. With EPT the SP is
> reduced by 4 bytes (and the written value is 0-padded) but without EPT
> support it is only reduced by 2 bytes. The difference can be observed when
> the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).
>
> The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and SP=0xFFE.
> Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The testing
> system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz.
>
> The test case (https://pastebin.com/ZejdtGEk) produces the output bellow.
> Note that 0x10 is written to 0xFFA on EPT=1 but it's written to 0xFFC on
> EPT=0.
>>
>> $ insmod kvm-intel.ko
>> $ sudo ./reproduce-push_es
>> Executing KVM_RUN
>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>   0000: 06 f4 00 00 00 00 00 00
>>   0008: 00 00 00 00 00 00 00 00
>>   0ff8: 00 00 10 00 00 00 00 00
>>   1000: 00 00 00 00 00 00 00 00
>
>
>> $ insmod kvm-intel.ko ept=0
>> $ sudo ./reproduce-push_es
>> Executing KVM_RUN
>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>   0000: 06 f4 00 00 00 00 00 00
>>   0008: 00 00 00 00 00 00 00 00
>>   0ff8: 00 00 00 00 10 00 00 00
>>   1000: 00 00 00 00 00 00 00 00

The cause of your two reports are the same. I think it has associated
with EPT+unrestricted_guest and vm8086 instead of EPT itself. vm8086
emulates a real mode environment, so it will not respect CS.D=1 which
you give since there is no segment descriptors support. However, big
real mode is different, they still load the segment descriptors which
hand over from protect mode before the mode switch. Your testcase just
start a real mode guest in all its life time w/o switch to protect
mode or vice versa. And KVM(EPT=Y, unrestricted_guest=Y) can't
distinguish between a real mode guest w/ segment descriptors given by
userspace and big real mode which occurs when protect mode switch to
real mode.

Regards,
Wanpeng Li

Re: KVM PUSH ES size bug

[Nadav Amit]

[-- Attachment #1: Type: text/plain, Size: 2712 bytes --]

Wanpeng Li <kernellwp@gmail.com> wrote:

> Cc Radim, Nadav,
> 2017-10-24 19:10 GMT+08:00 Pedro Fonseca <pfonseca@cs.washington.edu>:
>> Hi,
>> 
>> During tests that we conducted on KVM, we noticed that executing a "PUSH
>> %ES" instruction under KVM produces different results on both memory and the
>> SP register depending on whether EPT support is enabled. With EPT the SP is
>> reduced by 4 bytes (and the written value is 0-padded) but without EPT
>> support it is only reduced by 2 bytes. The difference can be observed when
>> the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).
>> 
>> The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and SP=0xFFE.
>> Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The testing
>> system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz.
>> 
>> The test case (https://pastebin.com/ZejdtGEk) produces the output bellow.
>> Note that 0x10 is written to 0xFFA on EPT=1 but it's written to 0xFFC on
>> EPT=0.
>>> $ insmod kvm-intel.ko
>>> $ sudo ./reproduce-push_es
>>> Executing KVM_RUN
>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>  0000: 06 f4 00 00 00 00 00 00
>>>  0008: 00 00 00 00 00 00 00 00
>>>  0ff8: 00 00 10 00 00 00 00 00
>>>  1000: 00 00 00 00 00 00 00 00
>> 
>> 
>>> $ insmod kvm-intel.ko ept=0
>>> $ sudo ./reproduce-push_es
>>> Executing KVM_RUN
>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>  0000: 06 f4 00 00 00 00 00 00
>>>  0008: 00 00 00 00 00 00 00 00
>>>  0ff8: 00 00 00 00 10 00 00 00
>>>  1000: 00 00 00 00 00 00 00 00
> 
> The cause of your two reports are the same. I think it has associated
> with EPT+unrestricted_guest and vm8086 instead of EPT itself. vm8086
> emulates a real mode environment, so it will not respect CS.D=1 which
> you give since there is no segment descriptors support. However, big
> real mode is different, they still load the segment descriptors which
> hand over from protect mode before the mode switch. Your testcase just
> start a real mode guest in all its life time w/o switch to protect
> mode or vice versa. And KVM(EPT=Y, unrestricted_guest=Y) can't
> distinguish between a real mode guest w/ segment descriptors given by
> userspace and big real mode which occurs when protect mode switch to
> real mode.
> 

Interesting. I can guess that the Intel tests that I was running back at the
time had a setup code (prior to the random code) in protected-mode, which
would explain why I missed this problem.

Perhaps the problem comes from wrong setting of the guest segment selector
“unusable” bit. I see there are quite few hacks in the code in regard to
this bit.

Regards,
Nadav

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: KVM PUSH ES size bug

[Wanpeng Li]

2017-10-26 0:20 GMT+08:00 Nadav Amit <nadav.amit@gmail.com>:
> Wanpeng Li <kernellwp@gmail.com> wrote:
>
>> Cc Radim, Nadav,
>> 2017-10-24 19:10 GMT+08:00 Pedro Fonseca <pfonseca@cs.washington.edu>:
>>> Hi,
>>>
>>> During tests that we conducted on KVM, we noticed that executing a "PUSH
>>> %ES" instruction under KVM produces different results on both memory and the
>>> SP register depending on whether EPT support is enabled. With EPT the SP is
>>> reduced by 4 bytes (and the written value is 0-padded) but without EPT
>>> support it is only reduced by 2 bytes. The difference can be observed when
>>> the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).
>>>
>>> The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and SP=0xFFE.
>>> Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The testing
>>> system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz.
>>>
>>> The test case (https://pastebin.com/ZejdtGEk) produces the output bellow.
>>> Note that 0x10 is written to 0xFFA on EPT=1 but it's written to 0xFFC on
>>> EPT=0.
>>>> $ insmod kvm-intel.ko
>>>> $ sudo ./reproduce-push_es
>>>> Executing KVM_RUN
>>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>>  0000: 06 f4 00 00 00 00 00 00
>>>>  0008: 00 00 00 00 00 00 00 00
>>>>  0ff8: 00 00 10 00 00 00 00 00
>>>>  1000: 00 00 00 00 00 00 00 00
>>>
>>>
>>>> $ insmod kvm-intel.ko ept=0
>>>> $ sudo ./reproduce-push_es
>>>> Executing KVM_RUN
>>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>>  0000: 06 f4 00 00 00 00 00 00
>>>>  0008: 00 00 00 00 00 00 00 00
>>>>  0ff8: 00 00 00 00 10 00 00 00
>>>>  1000: 00 00 00 00 00 00 00 00
>>
>> The cause of your two reports are the same. I think it has associated
>> with EPT+unrestricted_guest and vm8086 instead of EPT itself. vm8086
>> emulates a real mode environment, so it will not respect CS.D=1 which
>> you give since there is no segment descriptors support. However, big
>> real mode is different, they still load the segment descriptors which
>> hand over from protect mode before the mode switch. Your testcase just
>> start a real mode guest in all its life time w/o switch to protect
>> mode or vice versa. And KVM(EPT=Y, unrestricted_guest=Y) can't
>> distinguish between a real mode guest w/ segment descriptors given by
>> userspace and big real mode which occurs when protect mode switch to
>> real mode.
>>
>
> Interesting. I can guess that the Intel tests that I was running back at the
> time had a setup code (prior to the random code) in protected-mode, which
> would explain why I missed this problem.
>
> Perhaps the problem comes from wrong setting of the guest segment selector
> “unusable” bit. I see there are quite few hacks in the code in regard to
> this bit.

I change the "present" bit of CS/DS/SS/ES to 0 in the testcase,
however, the guest vmentry fails. In addition, is there any idea how
to fix it in kvm? I can be the volunteer to implement the idea. :)

Regards,
Wanpeng Li

Re: KVM PUSH ES size bug

[Nadav Amit]

Wanpeng Li <kernellwp@gmail.com> wrote:

> 2017-10-26 0:20 GMT+08:00 Nadav Amit <nadav.amit@gmail.com>:
>> Wanpeng Li <kernellwp@gmail.com> wrote:
>> 
>>> Cc Radim, Nadav,
>>> 2017-10-24 19:10 GMT+08:00 Pedro Fonseca <pfonseca@cs.washington.edu>:
>>>> Hi,
>>>> 
>>>> During tests that we conducted on KVM, we noticed that executing a "PUSH
>>>> %ES" instruction under KVM produces different results on both memory and the
>>>> SP register depending on whether EPT support is enabled. With EPT the SP is
>>>> reduced by 4 bytes (and the written value is 0-padded) but without EPT
>>>> support it is only reduced by 2 bytes. The difference can be observed when
>>>> the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).
>>>> 
>>>> The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and SP=0xFFE.
>>>> Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The testing
>>>> system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz.
>>>> 
>>>> The test case (https://pastebin.com/ZejdtGEk) produces the output bellow.
>>>> Note that 0x10 is written to 0xFFA on EPT=1 but it's written to 0xFFC on
>>>> EPT=0.
>>>>> $ insmod kvm-intel.ko
>>>>> $ sudo ./reproduce-push_es
>>>>> Executing KVM_RUN
>>>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>>> 0000: 06 f4 00 00 00 00 00 00
>>>>> 0008: 00 00 00 00 00 00 00 00
>>>>> 0ff8: 00 00 10 00 00 00 00 00
>>>>> 1000: 00 00 00 00 00 00 00 00
>>>> 
>>>> 
>>>>> $ insmod kvm-intel.ko ept=0
>>>>> $ sudo ./reproduce-push_es
>>>>> Executing KVM_RUN
>>>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>>> 0000: 06 f4 00 00 00 00 00 00
>>>>> 0008: 00 00 00 00 00 00 00 00
>>>>> 0ff8: 00 00 00 00 10 00 00 00
>>>>> 1000: 00 00 00 00 00 00 00 00
>>> 
>>> The cause of your two reports are the same. I think it has associated
>>> with EPT+unrestricted_guest and vm8086 instead of EPT itself. vm8086
>>> emulates a real mode environment, so it will not respect CS.D=1 which
>>> you give since there is no segment descriptors support. However, big
>>> real mode is different, they still load the segment descriptors which
>>> hand over from protect mode before the mode switch. Your testcase just
>>> start a real mode guest in all its life time w/o switch to protect
>>> mode or vice versa. And KVM(EPT=Y, unrestricted_guest=Y) can't
>>> distinguish between a real mode guest w/ segment descriptors given by
>>> userspace and big real mode which occurs when protect mode switch to
>>> real mode.
>> 
>> Interesting. I can guess that the Intel tests that I was running back at the
>> time had a setup code (prior to the random code) in protected-mode, which
>> would explain why I missed this problem.
>> 
>> Perhaps the problem comes from wrong setting of the guest segment selector
>> “unusable” bit. I see there are quite few hacks in the code in regard to
>> this bit.
> 
> I change the "present" bit of CS/DS/SS/ES to 0 in the testcase,
> however, the guest vmentry fails. In addition, is there any idea how
> to fix it in kvm? I can be the volunteer to implement the idea. :)

Stupid me. I didn’t read the setup well enough. So I understand there is
actually emulation when EPT=0, and this emulation is wrong.

I don’t see where the operand size (op_bytes) for “Stack” operations in
x86_decode_insn() is updated in respect to cs.d, and there is also no
appropriate logic in em_push_sreg().

I am sorry, but don’t have the time to further look into it right now.

Regards,
Nadav

Re: KVM PUSH ES size bug

[Wanpeng Li]

2017-10-26 14:52 GMT+08:00 Nadav Amit <nadav.amit@gmail.com>:
> Wanpeng Li <kernellwp@gmail.com> wrote:
>
>> 2017-10-26 0:20 GMT+08:00 Nadav Amit <nadav.amit@gmail.com>:
>>> Wanpeng Li <kernellwp@gmail.com> wrote:
>>>
>>>> Cc Radim, Nadav,
>>>> 2017-10-24 19:10 GMT+08:00 Pedro Fonseca <pfonseca@cs.washington.edu>:
>>>>> Hi,
>>>>>
>>>>> During tests that we conducted on KVM, we noticed that executing a "PUSH
>>>>> %ES" instruction under KVM produces different results on both memory and the
>>>>> SP register depending on whether EPT support is enabled. With EPT the SP is
>>>>> reduced by 4 bytes (and the written value is 0-padded) but without EPT
>>>>> support it is only reduced by 2 bytes. The difference can be observed when
>>>>> the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).
>>>>>
>>>>> The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and SP=0xFFE.
>>>>> Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The testing
>>>>> system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz.
>>>>>
>>>>> The test case (https://pastebin.com/ZejdtGEk) produces the output bellow.
>>>>> Note that 0x10 is written to 0xFFA on EPT=1 but it's written to 0xFFC on
>>>>> EPT=0.
>>>>>> $ insmod kvm-intel.ko
>>>>>> $ sudo ./reproduce-push_es
>>>>>> Executing KVM_RUN
>>>>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>>>> 0000: 06 f4 00 00 00 00 00 00
>>>>>> 0008: 00 00 00 00 00 00 00 00
>>>>>> 0ff8: 00 00 10 00 00 00 00 00
>>>>>> 1000: 00 00 00 00 00 00 00 00
>>>>>
>>>>>
>>>>>> $ insmod kvm-intel.ko ept=0
>>>>>> $ sudo ./reproduce-push_es
>>>>>> Executing KVM_RUN
>>>>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>>>> 0000: 06 f4 00 00 00 00 00 00
>>>>>> 0008: 00 00 00 00 00 00 00 00
>>>>>> 0ff8: 00 00 00 00 10 00 00 00
>>>>>> 1000: 00 00 00 00 00 00 00 00
>>>>
>>>> The cause of your two reports are the same. I think it has associated
>>>> with EPT+unrestricted_guest and vm8086 instead of EPT itself. vm8086
>>>> emulates a real mode environment, so it will not respect CS.D=1 which
>>>> you give since there is no segment descriptors support. However, big
>>>> real mode is different, they still load the segment descriptors which
>>>> hand over from protect mode before the mode switch. Your testcase just
>>>> start a real mode guest in all its life time w/o switch to protect
>>>> mode or vice versa. And KVM(EPT=Y, unrestricted_guest=Y) can't
>>>> distinguish between a real mode guest w/ segment descriptors given by
>>>> userspace and big real mode which occurs when protect mode switch to
>>>> real mode.
>>>
>>> Interesting. I can guess that the Intel tests that I was running back at the
>>> time had a setup code (prior to the random code) in protected-mode, which
>>> would explain why I missed this problem.
>>>
>>> Perhaps the problem comes from wrong setting of the guest segment selector
>>> “unusable” bit. I see there are quite few hacks in the code in regard to
>>> this bit.
>>
>> I change the "present" bit of CS/DS/SS/ES to 0 in the testcase,
>> however, the guest vmentry fails. In addition, is there any idea how
>> to fix it in kvm? I can be the volunteer to implement the idea. :)
>
> Stupid me. I didn’t read the setup well enough. So I understand there is
> actually emulation when EPT=0, and this emulation is wrong.
>
> I don’t see where the operand size (op_bytes) for “Stack” operations in
> x86_decode_insn() is updated in respect to cs.d, and there is also no
> appropriate logic in em_push_sreg().

Do you mean vm8086 should still respect cs.d even if there is no
segment descriptors in real mode?

Regards,
Wanpeng Li

Re: KVM PUSH ES size bug

[Paolo Bonzini]

On 26/10/2017 13:48, Wanpeng Li wrote:
> 2017-10-26 14:52 GMT+08:00 Nadav Amit <nadav.amit@gmail.com>:
>> Wanpeng Li <kernellwp@gmail.com> wrote:
>>
>>> 2017-10-26 0:20 GMT+08:00 Nadav Amit <nadav.amit@gmail.com>:
>>>> Wanpeng Li <kernellwp@gmail.com> wrote:
>>>>
>>>>> Cc Radim, Nadav,
>>>>> 2017-10-24 19:10 GMT+08:00 Pedro Fonseca <pfonseca@cs.washington.edu>:
>>>>>> Hi,
>>>>>>
>>>>>> During tests that we conducted on KVM, we noticed that executing a "PUSH
>>>>>> %ES" instruction under KVM produces different results on both memory and the
>>>>>> SP register depending on whether EPT support is enabled. With EPT the SP is
>>>>>> reduced by 4 bytes (and the written value is 0-padded) but without EPT
>>>>>> support it is only reduced by 2 bytes. The difference can be observed when
>>>>>> the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).
>>>>>>
>>>>>> The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and SP=0xFFE.
>>>>>> Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The testing
>>>>>> system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz.
>>>>>>
>>>>>> The test case (https://pastebin.com/ZejdtGEk) produces the output bellow.
>>>>>> Note that 0x10 is written to 0xFFA on EPT=1 but it's written to 0xFFC on
>>>>>> EPT=0.
>>>>>>> $ insmod kvm-intel.ko
>>>>>>> $ sudo ./reproduce-push_es
>>>>>>> Executing KVM_RUN
>>>>>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>>>>> 0000: 06 f4 00 00 00 00 00 00
>>>>>>> 0008: 00 00 00 00 00 00 00 00
>>>>>>> 0ff8: 00 00 10 00 00 00 00 00
>>>>>>> 1000: 00 00 00 00 00 00 00 00
>>>>>>
>>>>>>
>>>>>>> $ insmod kvm-intel.ko ept=0
>>>>>>> $ sudo ./reproduce-push_es
>>>>>>> Executing KVM_RUN
>>>>>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>>>>> 0000: 06 f4 00 00 00 00 00 00
>>>>>>> 0008: 00 00 00 00 00 00 00 00
>>>>>>> 0ff8: 00 00 00 00 10 00 00 00
>>>>>>> 1000: 00 00 00 00 00 00 00 00
>>>>>
>>>>> The cause of your two reports are the same. I think it has associated
>>>>> with EPT+unrestricted_guest and vm8086 instead of EPT itself. vm8086
>>>>> emulates a real mode environment, so it will not respect CS.D=1 which
>>>>> you give since there is no segment descriptors support. However, big
>>>>> real mode is different, they still load the segment descriptors which
>>>>> hand over from protect mode before the mode switch. Your testcase just
>>>>> start a real mode guest in all its life time w/o switch to protect
>>>>> mode or vice versa. And KVM(EPT=Y, unrestricted_guest=Y) can't
>>>>> distinguish between a real mode guest w/ segment descriptors given by
>>>>> userspace and big real mode which occurs when protect mode switch to
>>>>> real mode.
>>>>
>>>> Interesting. I can guess that the Intel tests that I was running back at the
>>>> time had a setup code (prior to the random code) in protected-mode, which
>>>> would explain why I missed this problem.
>>>>
>>>> Perhaps the problem comes from wrong setting of the guest segment selector
>>>> “unusable” bit. I see there are quite few hacks in the code in regard to
>>>> this bit.
>>>
>>> I change the "present" bit of CS/DS/SS/ES to 0 in the testcase,
>>> however, the guest vmentry fails. In addition, is there any idea how
>>> to fix it in kvm? I can be the volunteer to implement the idea. :)
>>
>> Stupid me. I didn’t read the setup well enough. So I understand there is
>> actually emulation when EPT=0, and this emulation is wrong.
>>
>> I don’t see where the operand size (op_bytes) for “Stack” operations in
>> x86_decode_insn() is updated in respect to cs.d, and there is also no
>> appropriate logic in em_push_sreg().
> 
> Do you mean vm8086 should still respect cs.d even if there is no
> segment descriptors in real mode?

Yes, the segment descriptors exist everywhere---in real mode and, to a
lesser extent, in vm8086 mode, they are hidden behind the descriptor
cache, but they are there.

Paolo

Re: KVM PUSH ES size bug

[Wanpeng Li]

2017-10-26 23:05 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
> On 26/10/2017 13:48, Wanpeng Li wrote:
>> 2017-10-26 14:52 GMT+08:00 Nadav Amit <nadav.amit@gmail.com>:
>>> Wanpeng Li <kernellwp@gmail.com> wrote:
>>>
>>>> 2017-10-26 0:20 GMT+08:00 Nadav Amit <nadav.amit@gmail.com>:
>>>>> Wanpeng Li <kernellwp@gmail.com> wrote:
>>>>>
>>>>>> Cc Radim, Nadav,
>>>>>> 2017-10-24 19:10 GMT+08:00 Pedro Fonseca <pfonseca@cs.washington.edu>:
>>>>>>> Hi,
>>>>>>>
>>>>>>> During tests that we conducted on KVM, we noticed that executing a "PUSH
>>>>>>> %ES" instruction under KVM produces different results on both memory and the
>>>>>>> SP register depending on whether EPT support is enabled. With EPT the SP is
>>>>>>> reduced by 4 bytes (and the written value is 0-padded) but without EPT
>>>>>>> support it is only reduced by 2 bytes. The difference can be observed when
>>>>>>> the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).
>>>>>>>
>>>>>>> The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and SP=0xFFE.
>>>>>>> Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The testing
>>>>>>> system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz.
>>>>>>>
>>>>>>> The test case (https://pastebin.com/ZejdtGEk) produces the output bellow.
>>>>>>> Note that 0x10 is written to 0xFFA on EPT=1 but it's written to 0xFFC on
>>>>>>> EPT=0.
>>>>>>>> $ insmod kvm-intel.ko
>>>>>>>> $ sudo ./reproduce-push_es
>>>>>>>> Executing KVM_RUN
>>>>>>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>>>>>> 0000: 06 f4 00 00 00 00 00 00
>>>>>>>> 0008: 00 00 00 00 00 00 00 00
>>>>>>>> 0ff8: 00 00 10 00 00 00 00 00
>>>>>>>> 1000: 00 00 00 00 00 00 00 00
>>>>>>>
>>>>>>>
>>>>>>>> $ insmod kvm-intel.ko ept=0
>>>>>>>> $ sudo ./reproduce-push_es
>>>>>>>> Executing KVM_RUN
>>>>>>>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>>>>>>> 0000: 06 f4 00 00 00 00 00 00
>>>>>>>> 0008: 00 00 00 00 00 00 00 00
>>>>>>>> 0ff8: 00 00 00 00 10 00 00 00
>>>>>>>> 1000: 00 00 00 00 00 00 00 00
>>>>>>
>>>>>> The cause of your two reports are the same. I think it has associated
>>>>>> with EPT+unrestricted_guest and vm8086 instead of EPT itself. vm8086
>>>>>> emulates a real mode environment, so it will not respect CS.D=1 which
>>>>>> you give since there is no segment descriptors support. However, big
>>>>>> real mode is different, they still load the segment descriptors which
>>>>>> hand over from protect mode before the mode switch. Your testcase just
>>>>>> start a real mode guest in all its life time w/o switch to protect
>>>>>> mode or vice versa. And KVM(EPT=Y, unrestricted_guest=Y) can't
>>>>>> distinguish between a real mode guest w/ segment descriptors given by
>>>>>> userspace and big real mode which occurs when protect mode switch to
>>>>>> real mode.
>>>>>
>>>>> Interesting. I can guess that the Intel tests that I was running back at the
>>>>> time had a setup code (prior to the random code) in protected-mode, which
>>>>> would explain why I missed this problem.
>>>>>
>>>>> Perhaps the problem comes from wrong setting of the guest segment selector
>>>>> “unusable” bit. I see there are quite few hacks in the code in regard to
>>>>> this bit.
>>>>
>>>> I change the "present" bit of CS/DS/SS/ES to 0 in the testcase,
>>>> however, the guest vmentry fails. In addition, is there any idea how
>>>> to fix it in kvm? I can be the volunteer to implement the idea. :)
>>>
>>> Stupid me. I didn’t read the setup well enough. So I understand there is
>>> actually emulation when EPT=0, and this emulation is wrong.
>>>
>>> I don’t see where the operand size (op_bytes) for “Stack” operations in
>>> x86_decode_insn() is updated in respect to cs.d, and there is also no
>>> appropriate logic in em_push_sreg().
>>
>> Do you mean vm8086 should still respect cs.d even if there is no
>> segment descriptors in real mode?
>
> Yes, the segment descriptors exist everywhere---in real mode and, to a
> lesser extent, in vm8086 mode, they are hidden behind the descriptor
> cache, but they are there.

Ok, I just sent out a patch to fix it.

Regards,
Wanpeng Li

Re: KVM PUSH ES size bug

[Paolo Bonzini]

On 24/10/2017 13:10, Pedro Fonseca wrote:
> Hi,
> 
> During tests that we conducted on KVM, we noticed that executing a "PUSH
> %ES" instruction under KVM produces different results on both memory and
> the SP register depending on whether EPT support is enabled. With EPT
> the SP is reduced by 4 bytes (and the written value is 0-padded) but
> without EPT support it is only reduced by 2 bytes. The difference can be
> observed when the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).

Just one thing: your setup code is setting up selectors and DPL for a
protected mode guest, but you are not initializing CR0.PE=1.

Paolo


> The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and
> SP=0xFFE. Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The
> testing system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700
> CPU @ 3.60GHz.
> 
> The test case (https://pastebin.com/ZejdtGEk) produces the output
> bellow. Note that 0x10 is written to 0xFFA on EPT=1 but it's written to
> 0xFFC on EPT=0.
>> $ insmod kvm-intel.ko
>> $ sudo ./reproduce-push_es
>> Executing KVM_RUN
>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>   0000: 06 f4 00 00 00 00 00 00
>>   0008: 00 00 00 00 00 00 00 00
>>   0ff8: 00 00 10 00 00 00 00 00
>>   1000: 00 00 00 00 00 00 00 00
> 
>> $ insmod kvm-intel.ko ept=0
>> $ sudo ./reproduce-push_es
>> Executing KVM_RUN
>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>   0000: 06 f4 00 00 00 00 00 00
>>   0008: 00 00 00 00 00 00 00 00
>>   0ff8: 00 00 00 00 10 00 00 00
>>   1000: 00 00 00 00 00 00 00 00
> 
> 
> Thanks,
> Pedro
> 

Re: KVM PUSH ES size bug

[Nadav Amit]

Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 24/10/2017 13:10, Pedro Fonseca wrote:
>> Hi,
>> 
>> During tests that we conducted on KVM, we noticed that executing a "PUSH
>> %ES" instruction under KVM produces different results on both memory and
>> the SP register depending on whether EPT support is enabled. With EPT
>> the SP is reduced by 4 bytes (and the written value is 0-padded) but
>> without EPT support it is only reduced by 2 bytes. The difference can be
>> observed when the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).
> 
> Just one thing: your setup code is setting up selectors and DPL for a
> protected mode guest, but you are not initializing CR0.PE=1.

IIUC, the problem would not have occurred if CR0.PE=1. If CR0.PE=1,
init_emulate_ctxt() would set the mode to X86EMUL_MODE_PROT32, the
operand-size would have been determined to be 4, and the emulation
would have been correct.

Nadav

Re: KVM PUSH ES size bug

[Wanpeng Li]

Hi Pedro,
2017-10-24 19:10 GMT+08:00 Pedro Fonseca <pfonseca@cs.washington.edu>:

Could you try v2 https://www.spinics.net/lists/kvm/msg157668.html and
give your Tested-by to v2 if it pass your testing?

Regards,
Wanpeng Li

> Hi,
>
> During tests that we conducted on KVM, we noticed that executing a "PUSH
> %ES" instruction under KVM produces different results on both memory and the
> SP register depending on whether EPT support is enabled. With EPT the SP is
> reduced by 4 bytes (and the written value is 0-padded) but without EPT
> support it is only reduced by 2 bytes. The difference can be observed when
> the CS.DB field is 1 (32-bit) but not when it's 0 (16-bit).
>
> The test case initializes the VM with EIP=0, CS.DB=1, ES=0x10, and SP=0xFFE.
> Memory is initialized with 0x06 (PUSH %ES) and 0xF4 (HLT). The testing
> system was running Linux 4.12.5 and Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz.
>
> The test case (https://pastebin.com/ZejdtGEk) produces the output bellow.
> Note that 0x10 is written to 0xFFA on EPT=1 but it's written to 0xFFC on
> EPT=0.
>>
>> $ insmod kvm-intel.ko
>> $ sudo ./reproduce-push_es
>> Executing KVM_RUN
>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>   0000: 06 f4 00 00 00 00 00 00
>>   0008: 00 00 00 00 00 00 00 00
>>   0ff8: 00 00 10 00 00 00 00 00
>>   1000: 00 00 00 00 00 00 00 00
>
>
>> $ insmod kvm-intel.ko ept=0
>> $ sudo ./reproduce-push_es
>> Executing KVM_RUN
>> KVM_RUN exited (exit_reason: 5, KVM_EXIT_HLT)
>>   0000: 06 f4 00 00 00 00 00 00
>>   0008: 00 00 00 00 00 00 00 00
>>   0ff8: 00 00 00 00 10 00 00 00
>>   1000: 00 00 00 00 00 00 00 00
>
>
>
> Thanks,
> Pedro