xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
* [RFC] Hypervisor, x86 emulation deprivileged
@ 2016-07-05 11:22 Anthony PERARD
  2016-07-05 12:58 ` George Dunlap
                   ` (2 more replies)
  0 siblings, 3 replies; 8+ messages in thread
From: Anthony PERARD @ 2016-07-05 11:22 UTC (permalink / raw)
  To: xen-devel, Jan Beulich, Andrew Cooper

Hi,

I've taken over the work from Ben to have a deprivileged mode in the
hypervisor, but I'm unsure about which direction to take.

First, after understanding what have been done, and fixing a few things,
I did some benchmark to compare a simple "device" running in ring0 to
the same one running in ring3 and also in QEMU. This "device" would call
'rdtsc' on 'outl' and return the value in 'inl' (I actually do not use
the value). The measurement is done from a kernel module in the guest
(simply rdtsc;inl;rdtsc multiple time). This is the result I've found:

    ring3 ~3.5x slower than ring0
    qemu   ~22x slower than ring0
          ~6.5x slower than ring3

So that would be the worst-case scenario, where an emulator barely do
anything.


There have been different methods proposed to do the depriv mode, in
<55A8D477.2060909@citrix.com>, one of which was to implement a per-vcpu
stack which could be more elegant.

So, would you suggest that I start working on a per-vcpu stack? Or
should I continue with the current direction?


Some of the code I have, including the code for the benchmark, can be
found here:
git://xenbits.xen.org/people/aperard/xen-unstable.git
branch: hvm-x86-deprivileged-mode-wip

Regards,

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [RFC] Hypervisor, x86 emulation deprivileged
  2016-07-05 11:22 [RFC] Hypervisor, x86 emulation deprivileged Anthony PERARD
@ 2016-07-05 12:58 ` George Dunlap
  2016-07-05 17:20   ` Anthony PERARD
  2016-07-05 13:02 ` Jan Beulich
  2016-07-05 17:25 ` Andrew Cooper
  2 siblings, 1 reply; 8+ messages in thread
From: George Dunlap @ 2016-07-05 12:58 UTC (permalink / raw)
  To: Anthony PERARD; +Cc: Andrew Cooper, Paul Durrant, Jan Beulich, xen-devel

On Tue, Jul 5, 2016 at 12:22 PM, Anthony PERARD
<anthony.perard@citrix.com> wrote:
> Hi,
>
> I've taken over the work from Ben to have a deprivileged mode in the
> hypervisor, but I'm unsure about which direction to take.
>
> First, after understanding what have been done, and fixing a few things,
> I did some benchmark to compare a simple "device" running in ring0 to
> the same one running in ring3 and also in QEMU. This "device" would call
> 'rdtsc' on 'outl' and return the value in 'inl' (I actually do not use
> the value). The measurement is done from a kernel module in the guest
> (simply rdtsc;inl;rdtsc multiple time). This is the result I've found:
>
>     ring3 ~3.5x slower than ring0
>     qemu   ~22x slower than ring0
>           ~6.5x slower than ring3
>
> So that would be the worst-case scenario, where an emulator barely do
> anything.
>
>
> There have been different methods proposed to do the depriv mode, in
> <55A8D477.2060909@citrix.com>, one of which was to implement a per-vcpu
> stack which could be more elegant.
>
> So, would you suggest that I start working on a per-vcpu stack? Or
> should I continue with the current direction?
>
>
> Some of the code I have, including the code for the benchmark, can be
> found here:
> git://xenbits.xen.org/people/aperard/xen-unstable.git
> branch: hvm-x86-deprivileged-mode-wip

I think we need to take a step back for a minute.

Correct me if I'm wrong here: this work doesn't move the *x86
emulation* code into ring3, but instead moves the *device emulation*
code into ring3, is that right?

So the use case would be a device which we think is too complicated to
emulate in ring0, but is too slow to emulate in qemu.

So the question is:

1. Are there any devices currently in ring0 that we would like to be
able to move into ring3?
2. Are there any devices currently in qemu that we would like to move
into ring3?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [RFC] Hypervisor, x86 emulation deprivileged
  2016-07-05 11:22 [RFC] Hypervisor, x86 emulation deprivileged Anthony PERARD
  2016-07-05 12:58 ` George Dunlap
@ 2016-07-05 13:02 ` Jan Beulich
  2016-07-05 17:01   ` Anthony PERARD
  2016-07-05 17:25 ` Andrew Cooper
  2 siblings, 1 reply; 8+ messages in thread
From: Jan Beulich @ 2016-07-05 13:02 UTC (permalink / raw)
  To: Anthony PERARD; +Cc: Andrew Cooper, xen-devel

>>> On 05.07.16 at 13:22, <anthony.perard@citrix.com> wrote:
> Hi,
> 
> I've taken over the work from Ben to have a deprivileged mode in the
> hypervisor, but I'm unsure about which direction to take.
> 
> First, after understanding what have been done, and fixing a few things,
> I did some benchmark to compare a simple "device" running in ring0 to
> the same one running in ring3 and also in QEMU. This "device" would call
> 'rdtsc' on 'outl' and return the value in 'inl' (I actually do not use
> the value). The measurement is done from a kernel module in the guest
> (simply rdtsc;inl;rdtsc multiple time). This is the result I've found:
> 
>     ring3 ~3.5x slower than ring0
>     qemu   ~22x slower than ring0
>           ~6.5x slower than ring3
> 
> So that would be the worst-case scenario, where an emulator barely do
> anything.
> 
> 
> There have been different methods proposed to do the depriv mode, in
> <55A8D477.2060909@citrix.com>, one of which was to implement a per-vcpu
> stack which could be more elegant.

Sadly my mail frontend doesn't let me search for message IDs (and
this old a mail would have been purged anyway meanwhile), so I
think (also considering how much time has passed) it would be better
if you actually summarized where things stopped back then.

> So, would you suggest that I start working on a per-vcpu stack? Or
> should I continue with the current direction?

Was there any reason why using per-vCPU stacks would be assumed
to meaningfully improve above numbers? I'm not sure pursuing this
idea is really useful if more than a marginal performance degradation
results.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [RFC] Hypervisor, x86 emulation deprivileged
  2016-07-05 13:02 ` Jan Beulich
@ 2016-07-05 17:01   ` Anthony PERARD
  0 siblings, 0 replies; 8+ messages in thread
From: Anthony PERARD @ 2016-07-05 17:01 UTC (permalink / raw)
  To: Jan Beulich; +Cc: Andrew Cooper, xen-devel

On Tue, Jul 05, 2016 at 07:02:39AM -0600, Jan Beulich wrote:
> >>> On 05.07.16 at 13:22, <anthony.perard@citrix.com> wrote:
> > Hi,
> > 
> > I've taken over the work from Ben to have a deprivileged mode in the
> > hypervisor, but I'm unsure about which direction to take.
> > 
> > First, after understanding what have been done, and fixing a few things,
> > I did some benchmark to compare a simple "device" running in ring0 to
> > the same one running in ring3 and also in QEMU. This "device" would call
> > 'rdtsc' on 'outl' and return the value in 'inl' (I actually do not use
> > the value). The measurement is done from a kernel module in the guest
> > (simply rdtsc;inl;rdtsc multiple time). This is the result I've found:
> > 
> >     ring3 ~3.5x slower than ring0
> >     qemu   ~22x slower than ring0
> >           ~6.5x slower than ring3
> > 
> > So that would be the worst-case scenario, where an emulator barely do
> > anything.
> > 
> > 
> > There have been different methods proposed to do the depriv mode, in
> > <55A8D477.2060909@citrix.com>, one of which was to implement a per-vcpu
> > stack which could be more elegant.
> 
> Sadly my mail frontend doesn't let me search for message IDs (and
> this old a mail would have been purged anyway meanwhile), so I
> think (also considering how much time has passed) it would be better
> if you actually summarized where things stopped back then.

https://lists.xen.org/archives/html/xen-devel/2015-07/msg03507.html

It has been said that a per-vcpu stack would be too much work for a
short term project.

> > So, would you suggest that I start working on a per-vcpu stack? Or
> > should I continue with the current direction?
> 
> Was there any reason why using per-vCPU stacks would be assumed
> to meaningfully improve above numbers?

Probably not. I guess the context switch alone takes most time, and it
does not matter where the stack is and if there is a copy of it.

> I'm not sure pursuing this
> idea is really useful if more than a marginal performance degradation
> results.

Maybe the instruction emulator would be big enough that the impact of a
context swith would not matter as much? I don't know much about it so I
can not make a guess of how much code is running.

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [RFC] Hypervisor, x86 emulation deprivileged
  2016-07-05 12:58 ` George Dunlap
@ 2016-07-05 17:20   ` Anthony PERARD
  0 siblings, 0 replies; 8+ messages in thread
From: Anthony PERARD @ 2016-07-05 17:20 UTC (permalink / raw)
  To: George Dunlap; +Cc: Andrew Cooper, Paul Durrant, Jan Beulich, xen-devel

On Tue, Jul 05, 2016 at 01:58:16PM +0100, George Dunlap wrote:
> On Tue, Jul 5, 2016 at 12:22 PM, Anthony PERARD
> <anthony.perard@citrix.com> wrote:
> > Hi,
> >
> > I've taken over the work from Ben to have a deprivileged mode in the
> > hypervisor, but I'm unsure about which direction to take.
> >
> > First, after understanding what have been done, and fixing a few things,
> > I did some benchmark to compare a simple "device" running in ring0 to
> > the same one running in ring3 and also in QEMU. This "device" would call
> > 'rdtsc' on 'outl' and return the value in 'inl' (I actually do not use
> > the value). The measurement is done from a kernel module in the guest
> > (simply rdtsc;inl;rdtsc multiple time). This is the result I've found:
> >
> >     ring3 ~3.5x slower than ring0
> >     qemu   ~22x slower than ring0
> >           ~6.5x slower than ring3
> >
> > So that would be the worst-case scenario, where an emulator barely do
> > anything.
> >
> >
> > There have been different methods proposed to do the depriv mode, in
> > <55A8D477.2060909@citrix.com>, one of which was to implement a per-vcpu
> > stack which could be more elegant.
> >
> > So, would you suggest that I start working on a per-vcpu stack? Or
> > should I continue with the current direction?
> >
> >
> > Some of the code I have, including the code for the benchmark, can be
> > found here:
> > git://xenbits.xen.org/people/aperard/xen-unstable.git
> > branch: hvm-x86-deprivileged-mode-wip
> 
> I think we need to take a step back for a minute.
> 
> Correct me if I'm wrong here: this work doesn't move the *x86
> emulation* code into ring3, but instead moves the *device emulation*
> code into ring3, is that right?

Actually, I think it would be for both. If the impact on performance is
small enough. I haven't try to move the x86 emulation into ring3.

> So the use case would be a device which we think is too complicated to
> emulate in ring0, but is too slow to emulate in qemu.
> 
> So the question is:
> 
> 1. Are there any devices currently in ring0 that we would like to be
> able to move into ring3?
> 2. Are there any devices currently in qemu that we would like to move
> into ring3?

Maybe having the pci root move from qemu to xen would help some use
cases. It has been suggested here:
https://lists.xen.org/archives/html/xen-devel/2016-06/msg00553.html
<alpine.DEB.2.10.1606031724280.2590@sstabellini-ThinkPad-X260>


-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [RFC] Hypervisor, x86 emulation deprivileged
  2016-07-05 11:22 [RFC] Hypervisor, x86 emulation deprivileged Anthony PERARD
  2016-07-05 12:58 ` George Dunlap
  2016-07-05 13:02 ` Jan Beulich
@ 2016-07-05 17:25 ` Andrew Cooper
  2016-07-06  7:59   ` Paul Durrant
  2 siblings, 1 reply; 8+ messages in thread
From: Andrew Cooper @ 2016-07-05 17:25 UTC (permalink / raw)
  To: Anthony PERARD, xen-devel, Jan Beulich

On 05/07/16 12:22, Anthony PERARD wrote:
> Hi,
>
> I've taken over the work from Ben to have a deprivileged mode in the
> hypervisor, but I'm unsure about which direction to take.

You should begin with an evaluation of available options, identifying
which issues are mitigated, those which are not, and which which new
risks are introduced.

E.g.

1) De-privileging the x86 instruction emulator into hypervisor ring3
Issues mitigated:
* Out of bounds pointer accesses.
Issues not mitigated:
* On-stack state corruption.
Risks:
* Introduces what is basically 3rd type of vcpu, including all the
subtly that comes user processes.


Performance is strictly a secondary consideration;  Make something which
works first, then make it fast.  Never the opposite way around.

Another exercise which might be useful is to look at the recent XSAs and
identified which of them could have been mitigated by one of the
suggestions.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [RFC] Hypervisor, x86 emulation deprivileged
  2016-07-05 17:25 ` Andrew Cooper
@ 2016-07-06  7:59   ` Paul Durrant
  2016-07-08  9:07     ` Tim Deegan
  0 siblings, 1 reply; 8+ messages in thread
From: Paul Durrant @ 2016-07-06  7:59 UTC (permalink / raw)
  To: Andrew Cooper, Anthony Perard, xen-devel, Jan Beulich

> -----Original Message-----
> From: Xen-devel [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of
> Andrew Cooper
> Sent: 05 July 2016 18:26
> To: Anthony Perard; xen-devel@lists.xen.org; Jan Beulich
> Subject: Re: [Xen-devel] [RFC] Hypervisor, x86 emulation deprivileged
> 
> On 05/07/16 12:22, Anthony PERARD wrote:
> > Hi,
> >
> > I've taken over the work from Ben to have a deprivileged mode in the
> > hypervisor, but I'm unsure about which direction to take.
> 
> You should begin with an evaluation of available options, identifying
> which issues are mitigated, those which are not, and which which new
> risks are introduced.
> 

Personally I can't see why we want to invent a new way of executing code in the hypervisor. What's wrong with modelling each emulator as a PV unikernel, running with all the usual domain and vcpu constructs? Doing anything else seems like:

a) a lot of work for little gain
b) re-inventing the wheel

  Paul

> E.g.
> 
> 1) De-privileging the x86 instruction emulator into hypervisor ring3
> Issues mitigated:
> * Out of bounds pointer accesses.
> Issues not mitigated:
> * On-stack state corruption.
> Risks:
> * Introduces what is basically 3rd type of vcpu, including all the
> subtly that comes user processes.
> 
> 
> Performance is strictly a secondary consideration;  Make something which
> works first, then make it fast.  Never the opposite way around.
> 
> Another exercise which might be useful is to look at the recent XSAs and
> identified which of them could have been mitigated by one of the
> suggestions.
> 
> ~Andrew
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [RFC] Hypervisor, x86 emulation deprivileged
  2016-07-06  7:59   ` Paul Durrant
@ 2016-07-08  9:07     ` Tim Deegan
  0 siblings, 0 replies; 8+ messages in thread
From: Tim Deegan @ 2016-07-08  9:07 UTC (permalink / raw)
  To: Paul Durrant; +Cc: Anthony Perard, Andrew Cooper, Jan Beulich, xen-devel

At 07:59 +0000 on 06 Jul (1467791951), Paul Durrant wrote:
> > -----Original Message-----
> > From: Xen-devel [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of
> > Andrew Cooper
> > Sent: 05 July 2016 18:26
> > To: Anthony Perard; xen-devel@lists.xen.org; Jan Beulich
> > Subject: Re: [Xen-devel] [RFC] Hypervisor, x86 emulation deprivileged
> > 
> > On 05/07/16 12:22, Anthony PERARD wrote:
> > > Hi,
> > >
> > > I've taken over the work from Ben to have a deprivileged mode in the
> > > hypervisor, but I'm unsure about which direction to take.
> > 
> > You should begin with an evaluation of available options, identifying
> > which issues are mitigated, those which are not, and which which new
> > risks are introduced.
> > 
> 
> Personally I can't see why we want to invent a new way of executing
> code in the hypervisor. What's wrong with modelling each emulator as a
> PV unikernel, running with all the usual domain and vcpu constructs?

It would be useful to compare a unikernel/minios implementation of
this toy service to the other three.  I'd expect it to be slower
because of the extra context switch, but maybe directed yields could
help with that?

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2016-07-08  9:07 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-07-05 11:22 [RFC] Hypervisor, x86 emulation deprivileged Anthony PERARD
2016-07-05 12:58 ` George Dunlap
2016-07-05 17:20   ` Anthony PERARD
2016-07-05 13:02 ` Jan Beulich
2016-07-05 17:01   ` Anthony PERARD
2016-07-05 17:25 ` Andrew Cooper
2016-07-06  7:59   ` Paul Durrant
2016-07-08  9:07     ` Tim Deegan

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).