KVM exit on UD interception

KVM exit on UD interception

[Alexandru Duţu]

Dear all,

It seems that currently, on UD interception KVM does not exit
completely. Virtualized execution finishes, KVM executes
ud_intercept() after which it enters virtualized execution again.

I am working on accelerating with virtualized execution a simulator
that emulates system calls. Essentially doing virtualized execution
without a OS kernel. In order to make this work, I had to modify my
the KVM kernel module such that ud_intercept() return 0 and not 1
which break KVM __vcpu_run loop. This is necessary as I need to trap
syscall instructions, exit virtualized execution with UD exception,
emulate the system call in the simulator and after the system call is
done enter back in virtualized mode and start execution with the help
of KVM.

So by modifying ud_intercept() to return 0, I got all this to work. Is
it possible to achieve the same effect (exit on undefined opcode)
without modifying ud_intercept()?

It seems that re-entering virtualized execution on UD interception
gives the user the flexibility of running binaries with newer
instructions on older hardware, if kvm is able to emulate the newer
instructions. I do not fully understand the details of this scenario,
is there such a scenario or is it likely that ud_interception() will
change?

Thank you in advance!

Best regards,
Alex

Re: KVM exit on UD interception

[Nakajima, Jun]

On Mon, May 5, 2014 at 8:56 AM, Alexandru Duţu <alex.dutu@gmail.com> wrote:
> Dear all,
>
> It seems that currently, on UD interception KVM does not exit
> completely. Virtualized execution finishes, KVM executes
> ud_intercept() after which it enters virtualized execution again.

Maybe you might want to take a look at the VMX side (to port it to
SVM). The MOVBE emulation, for example, should be helpful.

>
> I am working on accelerating with virtualized execution a simulator
> that emulates system calls. Essentially doing virtualized execution
> without a OS kernel. In order to make this work, I had to modify my
> the KVM kernel module such that ud_intercept() return 0 and not 1
> which break KVM __vcpu_run loop. This is necessary as I need to trap
> syscall instructions, exit virtualized execution with UD exception,
> emulate the system call in the simulator and after the system call is
> done enter back in virtualized mode and start execution with the help
> of KVM.
>
> So by modifying ud_intercept() to return 0, I got all this to work. Is
> it possible to achieve the same effect (exit on undefined opcode)
> without modifying ud_intercept()?
>
> It seems that re-entering virtualized execution on UD interception
> gives the user the flexibility of running binaries with newer
> instructions on older hardware, if kvm is able to emulate the newer
> instructions. I do not fully understand the details of this scenario,
> is there such a scenario or is it likely that ud_interception() will
> change?
>
> Thank you in advance!
>
> Best regards,
> Alex
> --

-- 
Jun
Intel Open Source Technology Center

Re: KVM exit on UD interception

[Alexandru Duţu]

Thank you Jun! I see that in case of VMX does not emulated the
instruction that produced a UD exception, it just queues the exception
and returns 1. After that KVM will still try to enter virtualized
execution and so forth, the execution probably finishing with a DF and
shut down. It does not seem that KVM, in case of VMX, will exit
immediately on UD.

I am not sure what you meant with MOVBE emulation.

Thanks,
Alex

On Mon, May 5, 2014 at 12:34 PM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
> On Mon, May 5, 2014 at 8:56 AM, Alexandru Duţu <alex.dutu@gmail.com> wrote:
>> Dear all,
>>
>> It seems that currently, on UD interception KVM does not exit
>> completely. Virtualized execution finishes, KVM executes
>> ud_intercept() after which it enters virtualized execution again.
>
> Maybe you might want to take a look at the VMX side (to port it to
> SVM). The MOVBE emulation, for example, should be helpful.
>
>>
>> I am working on accelerating with virtualized execution a simulator
>> that emulates system calls. Essentially doing virtualized execution
>> without a OS kernel. In order to make this work, I had to modify my
>> the KVM kernel module such that ud_intercept() return 0 and not 1
>> which break KVM __vcpu_run loop. This is necessary as I need to trap
>> syscall instructions, exit virtualized execution with UD exception,
>> emulate the system call in the simulator and after the system call is
>> done enter back in virtualized mode and start execution with the help
>> of KVM.
>>
>> So by modifying ud_intercept() to return 0, I got all this to work. Is
>> it possible to achieve the same effect (exit on undefined opcode)
>> without modifying ud_intercept()?
>>
>> It seems that re-entering virtualized execution on UD interception
>> gives the user the flexibility of running binaries with newer
>> instructions on older hardware, if kvm is able to emulate the newer
>> instructions. I do not fully understand the details of this scenario,
>> is there such a scenario or is it likely that ud_interception() will
>> change?
>>
>> Thank you in advance!
>>
>> Best regards,
>> Alex
>> --
>
> --
> Jun
> Intel Open Source Technology Center



-- 
Alex

Re: KVM exit on UD interception

[Nakajima, Jun]

On Mon, May 5, 2014 at 11:48 AM, Alexandru Duţu <alex.dutu@gmail.com> wrote:
> Thank you Jun! I see that in case of VMX does not emulated the
> instruction that produced a UD exception, it just queues the exception
> and returns 1. After that KVM will still try to enter virtualized
> execution and so forth, the execution probably finishing with a DF and
> shut down. It does not seem that KVM, in case of VMX, will exit
> immediately on UD.
>
> I am not sure what you meant with MOVBE emulation.

I meant:

commit 84cffe499b9418d6c3b4de2ad9599cc2ec50c607
Author: Borislav Petkov <bp@suse.de>
Date:   Tue Oct 29 12:54:56 2013 +0100

    kvm: Emulate MOVBE

    This basically came from the need to be able to boot 32-bit Atom SMP
    guests on an AMD host, i.e. a host which doesn't support MOVBE. As a
    matter of fact, qemu has since recently received MOVBE support but we
    cannot share that with kvm emulation and thus we have to do this in the
    host. We're waay faster in kvm anyway. :-)

    So, we piggyback on the #UD path and emulate the MOVBE functionality.
    With it, an 8-core SMP guest boots in under 6 seconds.

    Also, requesting MOVBE emulation needs to happen explicitly to work,
    i.e. qemu -cpu n270,+movbe...

    Just FYI, a fairly straight-forward boot of a MOVBE-enabled 3.9-rc6+
    kernel in kvm executes MOVBE ~60K times.

    Signed-off-by: Andre Przywara <andre@andrep.de>
    Signed-off-by: Borislav Petkov <bp@suse.de>
    Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>


-- 
Jun
Intel Open Source Technology Center

Re: KVM exit on UD interception

[Alexandru Duţu]

Thank you Jun!

Now I understand that there is a strong need to support this scenario
where the host might run into trouble executing binaries with
instructions unknown to it.

I am still wondering if there is a way to actually exit KVM on UD from
a syscall instruction without modifying the KVM kernel module?

Best regards,
Alex


On Mon, May 5, 2014 at 7:07 PM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
> On Mon, May 5, 2014 at 11:48 AM, Alexandru Duţu <alex.dutu@gmail.com> wrote:
>> Thank you Jun! I see that in case of VMX does not emulated the
>> instruction that produced a UD exception, it just queues the exception
>> and returns 1. After that KVM will still try to enter virtualized
>> execution and so forth, the execution probably finishing with a DF and
>> shut down. It does not seem that KVM, in case of VMX, will exit
>> immediately on UD.
>>
>> I am not sure what you meant with MOVBE emulation.
>
> I meant:
>
> commit 84cffe499b9418d6c3b4de2ad9599cc2ec50c607
> Author: Borislav Petkov <bp@suse.de>
> Date:   Tue Oct 29 12:54:56 2013 +0100
>
>     kvm: Emulate MOVBE
>
>     This basically came from the need to be able to boot 32-bit Atom SMP
>     guests on an AMD host, i.e. a host which doesn't support MOVBE. As a
>     matter of fact, qemu has since recently received MOVBE support but we
>     cannot share that with kvm emulation and thus we have to do this in the
>     host. We're waay faster in kvm anyway. :-)
>
>     So, we piggyback on the #UD path and emulate the MOVBE functionality.
>     With it, an 8-core SMP guest boots in under 6 seconds.
>
>     Also, requesting MOVBE emulation needs to happen explicitly to work,
>     i.e. qemu -cpu n270,+movbe...
>
>     Just FYI, a fairly straight-forward boot of a MOVBE-enabled 3.9-rc6+
>     kernel in kvm executes MOVBE ~60K times.
>
>     Signed-off-by: Andre Przywara <andre@andrep.de>
>     Signed-off-by: Borislav Petkov <bp@suse.de>
>     Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>
>
> --
> Jun
> Intel Open Source Technology Center



-- 
Alex

Re: KVM exit on UD interception

[Paolo Bonzini]

Il 05/05/2014 17:56, Alexandru Duţu ha scritto:
>
> It seems that re-entering virtualized execution on UD interception
> gives the user the flexibility of running binaries with newer
> instructions on older hardware, if kvm is able to emulate the newer
> instructions. I do not fully understand the details of this scenario,
> is there such a scenario or is it likely that ud_interception() will
> change?

Yes, you are correct.  This emulation capability is used in some cases, 
including: MOVBE, SYSCALL/SYSENTER on 32-bit guests, VMCALL and VMMCALL.

Paolo

Re: KVM exit on UD interception

[Alexandru Duţu]

Thank you Paolo!

What is puzzling thought is the fact that even if there is an
instruction that can't be emulated by KVM, re-entering virtualized
mode is still attempted. Re-entering will produce a double fault, in
which case KVM will finally exit leaving the user with a somewhat
diminished perspective of the exit reason (DF exception). The hardware
virtualization support (SVM and VMX for x86) is designed such that
exceptions are intercepted and double faults or triple faults do not
hinder the actual reason of exit.

So would it make sense to change ud_interception such that it produces
a KVM exit if the instruction is not emulated by KVM and it re-enters
virtualized execution if the instruction is emulated?

Best,
Alex

On Tue, May 6, 2014 at 11:56 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 05/05/2014 17:56, Alexandru Duţu ha scritto:
>
>>
>> It seems that re-entering virtualized execution on UD interception
>> gives the user the flexibility of running binaries with newer
>> instructions on older hardware, if kvm is able to emulate the newer
>> instructions. I do not fully understand the details of this scenario,
>> is there such a scenario or is it likely that ud_interception() will
>> change?
>
>
> Yes, you are correct.  This emulation capability is used in some cases,
> including: MOVBE, SYSCALL/SYSENTER on 32-bit guests, VMCALL and VMMCALL.
>
> Paolo



-- 
Alex

Re: KVM exit on UD interception

[Paolo Bonzini]

Il 06/05/2014 22:11, Alexandru Duţu ha scritto:
> What is puzzling thought is the fact that even if there is an
> instruction that can't be emulated by KVM, re-entering virtualized
> mode is still attempted. Re-entering will produce a double fault,

That would be a bug.  It should inject the #UD into the guest.

Paolo

Re: KVM exit on UD interception

[Alexandru Duţu]

Thanks Paolo! That makes sense, KVM has to inject the #UD in order for
the guest to deliver the exception in the guest user space.

On Wed, May 7, 2014 at 1:55 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 06/05/2014 22:11, Alexandru Duţu ha scritto:
>
>> What is puzzling thought is the fact that even if there is an
>> instruction that can't be emulated by KVM, re-entering virtualized
>> mode is still attempted. Re-entering will produce a double fault,
>
>
> That would be a bug.  It should inject the #UD into the guest.
>
> Paolo



-- 
Alex