[RFC] Hypervisor, x86 emulation deprivileged

[RFC] Hypervisor, x86 emulation deprivileged

[Anthony PERARD]

Hi,

I've taken over the work from Ben to have a deprivileged mode in the
hypervisor, but I'm unsure about which direction to take.

First, after understanding what have been done, and fixing a few things,
I did some benchmark to compare a simple "device" running in ring0 to
the same one running in ring3 and also in QEMU. This "device" would call
'rdtsc' on 'outl' and return the value in 'inl' (I actually do not use
the value). The measurement is done from a kernel module in the guest
(simply rdtsc;inl;rdtsc multiple time). This is the result I've found:

    ring3 ~3.5x slower than ring0
    qemu   ~22x slower than ring0
          ~6.5x slower than ring3

So that would be the worst-case scenario, where an emulator barely do
anything.


There have been different methods proposed to do the depriv mode, in
<55A8D477.2060909@citrix.com>, one of which was to implement a per-vcpu
stack which could be more elegant.

So, would you suggest that I start working on a per-vcpu stack? Or
should I continue with the current direction?


Some of the code I have, including the code for the benchmark, can be
found here:
git://xenbits.xen.org/people/aperard/xen-unstable.git
branch: hvm-x86-deprivileged-mode-wip

Regards,

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC] Hypervisor, x86 emulation deprivileged

[George Dunlap]

On Tue, Jul 5, 2016 at 12:22 PM, Anthony PERARD
<anthony.perard@citrix.com> wrote:
> Hi,
>
> I've taken over the work from Ben to have a deprivileged mode in the
> hypervisor, but I'm unsure about which direction to take.
>
> First, after understanding what have been done, and fixing a few things,
> I did some benchmark to compare a simple "device" running in ring0 to
> the same one running in ring3 and also in QEMU. This "device" would call
> 'rdtsc' on 'outl' and return the value in 'inl' (I actually do not use
> the value). The measurement is done from a kernel module in the guest
> (simply rdtsc;inl;rdtsc multiple time). This is the result I've found:
>
>     ring3 ~3.5x slower than ring0
>     qemu   ~22x slower than ring0
>           ~6.5x slower than ring3
>
> So that would be the worst-case scenario, where an emulator barely do
> anything.
>
>
> There have been different methods proposed to do the depriv mode, in
> <55A8D477.2060909@citrix.com>, one of which was to implement a per-vcpu
> stack which could be more elegant.
>
> So, would you suggest that I start working on a per-vcpu stack? Or
> should I continue with the current direction?
>
>
> Some of the code I have, including the code for the benchmark, can be
> found here:
> git://xenbits.xen.org/people/aperard/xen-unstable.git
> branch: hvm-x86-deprivileged-mode-wip

I think we need to take a step back for a minute.

Correct me if I'm wrong here: this work doesn't move the *x86
emulation* code into ring3, but instead moves the *device emulation*
code into ring3, is that right?

So the use case would be a device which we think is too complicated to
emulate in ring0, but is too slow to emulate in qemu.

So the question is:

1. Are there any devices currently in ring0 that we would like to be
able to move into ring3?
2. Are there any devices currently in qemu that we would like to move
into ring3?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC] Hypervisor, x86 emulation deprivileged

[Anthony PERARD]

On Tue, Jul 05, 2016 at 01:58:16PM +0100, George Dunlap wrote:
> On Tue, Jul 5, 2016 at 12:22 PM, Anthony PERARD
> <anthony.perard@citrix.com> wrote:
> > Hi,
> >
> > I've taken over the work from Ben to have a deprivileged mode in the
> > hypervisor, but I'm unsure about which direction to take.
> >
> > First, after understanding what have been done, and fixing a few things,
> > I did some benchmark to compare a simple "device" running in ring0 to
> > the same one running in ring3 and also in QEMU. This "device" would call
> > 'rdtsc' on 'outl' and return the value in 'inl' (I actually do not use
> > the value). The measurement is done from a kernel module in the guest
> > (simply rdtsc;inl;rdtsc multiple time). This is the result I've found:
> >
> >     ring3 ~3.5x slower than ring0
> >     qemu   ~22x slower than ring0
> >           ~6.5x slower than ring3
> >
> > So that would be the worst-case scenario, where an emulator barely do
> > anything.
> >
> >
> > There have been different methods proposed to do the depriv mode, in
> > <55A8D477.2060909@citrix.com>, one of which was to implement a per-vcpu
> > stack which could be more elegant.
> >
> > So, would you suggest that I start working on a per-vcpu stack? Or
> > should I continue with the current direction?
> >
> >
> > Some of the code I have, including the code for the benchmark, can be
> > found here:
> > git://xenbits.xen.org/people/aperard/xen-unstable.git
> > branch: hvm-x86-deprivileged-mode-wip
> 
> I think we need to take a step back for a minute.
> 
> Correct me if I'm wrong here: this work doesn't move the *x86
> emulation* code into ring3, but instead moves the *device emulation*
> code into ring3, is that right?

Actually, I think it would be for both. If the impact on performance is
small enough. I haven't try to move the x86 emulation into ring3.

> So the use case would be a device which we think is too complicated to
> emulate in ring0, but is too slow to emulate in qemu.
> 
> So the question is:
> 
> 1. Are there any devices currently in ring0 that we would like to be
> able to move into ring3?
> 2. Are there any devices currently in qemu that we would like to move
> into ring3?

Maybe having the pci root move from qemu to xen would help some use
cases. It has been suggested here:
https://lists.xen.org/archives/html/xen-devel/2016-06/msg00553.html
<alpine.DEB.2.10.1606031724280.2590@sstabellini-ThinkPad-X260>


-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC] Hypervisor, x86 emulation deprivileged

[Jan Beulich]

>>> On 05.07.16 at 13:22, <anthony.perard@citrix.com> wrote:
> Hi,
> 
> I've taken over the work from Ben to have a deprivileged mode in the
> hypervisor, but I'm unsure about which direction to take.
> 
> First, after understanding what have been done, and fixing a few things,
> I did some benchmark to compare a simple "device" running in ring0 to
> the same one running in ring3 and also in QEMU. This "device" would call
> 'rdtsc' on 'outl' and return the value in 'inl' (I actually do not use
> the value). The measurement is done from a kernel module in the guest
> (simply rdtsc;inl;rdtsc multiple time). This is the result I've found:
> 
>     ring3 ~3.5x slower than ring0
>     qemu   ~22x slower than ring0
>           ~6.5x slower than ring3
> 
> So that would be the worst-case scenario, where an emulator barely do
> anything.
> 
> 
> There have been different methods proposed to do the depriv mode, in
> <55A8D477.2060909@citrix.com>, one of which was to implement a per-vcpu
> stack which could be more elegant.

Sadly my mail frontend doesn't let me search for message IDs (and
this old a mail would have been purged anyway meanwhile), so I
think (also considering how much time has passed) it would be better
if you actually summarized where things stopped back then.

> So, would you suggest that I start working on a per-vcpu stack? Or
> should I continue with the current direction?

Was there any reason why using per-vCPU stacks would be assumed
to meaningfully improve above numbers? I'm not sure pursuing this
idea is really useful if more than a marginal performance degradation
results.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC] Hypervisor, x86 emulation deprivileged

[Anthony PERARD]

On Tue, Jul 05, 2016 at 07:02:39AM -0600, Jan Beulich wrote:
> >>> On 05.07.16 at 13:22, <anthony.perard@citrix.com> wrote:
> > Hi,
> > 
> > I've taken over the work from Ben to have a deprivileged mode in the
> > hypervisor, but I'm unsure about which direction to take.
> > 
> > First, after understanding what have been done, and fixing a few things,
> > I did some benchmark to compare a simple "device" running in ring0 to
> > the same one running in ring3 and also in QEMU. This "device" would call
> > 'rdtsc' on 'outl' and return the value in 'inl' (I actually do not use
> > the value). The measurement is done from a kernel module in the guest
> > (simply rdtsc;inl;rdtsc multiple time). This is the result I've found:
> > 
> >     ring3 ~3.5x slower than ring0
> >     qemu   ~22x slower than ring0
> >           ~6.5x slower than ring3
> > 
> > So that would be the worst-case scenario, where an emulator barely do
> > anything.
> > 
> > 
> > There have been different methods proposed to do the depriv mode, in
> > <55A8D477.2060909@citrix.com>, one of which was to implement a per-vcpu
> > stack which could be more elegant.
> 
> Sadly my mail frontend doesn't let me search for message IDs (and
> this old a mail would have been purged anyway meanwhile), so I
> think (also considering how much time has passed) it would be better
> if you actually summarized where things stopped back then.

https://lists.xen.org/archives/html/xen-devel/2015-07/msg03507.html

It has been said that a per-vcpu stack would be too much work for a
short term project.

> > So, would you suggest that I start working on a per-vcpu stack? Or
> > should I continue with the current direction?
> 
> Was there any reason why using per-vCPU stacks would be assumed
> to meaningfully improve above numbers?

Probably not. I guess the context switch alone takes most time, and it
does not matter where the stack is and if there is a copy of it.

> I'm not sure pursuing this
> idea is really useful if more than a marginal performance degradation
> results.

Maybe the instruction emulator would be big enough that the impact of a
context swith would not matter as much? I don't know much about it so I
can not make a guess of how much code is running.

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC] Hypervisor, x86 emulation deprivileged

[Andrew Cooper]

On 05/07/16 12:22, Anthony PERARD wrote:
> Hi,
>
> I've taken over the work from Ben to have a deprivileged mode in the
> hypervisor, but I'm unsure about which direction to take.

You should begin with an evaluation of available options, identifying
which issues are mitigated, those which are not, and which which new
risks are introduced.

E.g.

1) De-privileging the x86 instruction emulator into hypervisor ring3
Issues mitigated:
* Out of bounds pointer accesses.
Issues not mitigated:
* On-stack state corruption.
Risks:
* Introduces what is basically 3rd type of vcpu, including all the
subtly that comes user processes.


Performance is strictly a secondary consideration;  Make something which
works first, then make it fast.  Never the opposite way around.

Another exercise which might be useful is to look at the recent XSAs and
identified which of them could have been mitigated by one of the
suggestions.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC] Hypervisor, x86 emulation deprivileged

[Paul Durrant]

> -----Original Message-----
> From: Xen-devel [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of
> Andrew Cooper
> Sent: 05 July 2016 18:26
> To: Anthony Perard; xen-devel@lists.xen.org; Jan Beulich
> Subject: Re: [Xen-devel] [RFC] Hypervisor, x86 emulation deprivileged
> 
> On 05/07/16 12:22, Anthony PERARD wrote:
> > Hi,
> >
> > I've taken over the work from Ben to have a deprivileged mode in the
> > hypervisor, but I'm unsure about which direction to take.
> 
> You should begin with an evaluation of available options, identifying
> which issues are mitigated, those which are not, and which which new
> risks are introduced.
> 

Personally I can't see why we want to invent a new way of executing code in the hypervisor. What's wrong with modelling each emulator as a PV unikernel, running with all the usual domain and vcpu constructs? Doing anything else seems like:

a) a lot of work for little gain
b) re-inventing the wheel

  Paul

> E.g.
> 
> 1) De-privileging the x86 instruction emulator into hypervisor ring3
> Issues mitigated:
> * Out of bounds pointer accesses.
> Issues not mitigated:
> * On-stack state corruption.
> Risks:
> * Introduces what is basically 3rd type of vcpu, including all the
> subtly that comes user processes.
> 
> 
> Performance is strictly a secondary consideration;  Make something which
> works first, then make it fast.  Never the opposite way around.
> 
> Another exercise which might be useful is to look at the recent XSAs and
> identified which of them could have been mitigated by one of the
> suggestions.
> 
> ~Andrew
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [RFC] Hypervisor, x86 emulation deprivileged

[Tim Deegan]

At 07:59 +0000 on 06 Jul (1467791951), Paul Durrant wrote:
> > -----Original Message-----
> > From: Xen-devel [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of
> > Andrew Cooper
> > Sent: 05 July 2016 18:26
> > To: Anthony Perard; xen-devel@lists.xen.org; Jan Beulich
> > Subject: Re: [Xen-devel] [RFC] Hypervisor, x86 emulation deprivileged
> > 
> > On 05/07/16 12:22, Anthony PERARD wrote:
> > > Hi,
> > >
> > > I've taken over the work from Ben to have a deprivileged mode in the
> > > hypervisor, but I'm unsure about which direction to take.
> > 
> > You should begin with an evaluation of available options, identifying
> > which issues are mitigated, those which are not, and which which new
> > risks are introduced.
> > 
> 
> Personally I can't see why we want to invent a new way of executing
> code in the hypervisor. What's wrong with modelling each emulator as a
> PV unikernel, running with all the usual domain and vcpu constructs?

It would be useful to compare a unikernel/minios implementation of
this toy service to the other three.  I'd expect it to be slower
because of the extra context switch, but maybe directed yields could
help with that?

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel