Future x86 emulator direction

Future x86 emulator direction

[Andrew Cooper]

Hello,

I bring this query up now as I realise it will influence how I proceed
with the MSR and CPUID faulting improvements.

During the most recent Cambridge Hackathon (April 2016), there was a
suggestion made (sorry - I don't recall from whom) to feed the the
SVM/VMX intercept information into a slightly more general emulate
framework, rather than to try to implement common functionality in 3
separate locations.

I think this is a very good idea, but it does involve a quite
substantial change to the way things currently work; It would involve
making an emulation of an entire CPU, rather than just instructions.

For example, hvm_task_switch() currently works from the the intercept
hooks alone, but there is no support in the instruction emulator for
`lcall $TSS, $0`.  Inverting the current layout and having both the
intercepts and instruction emulator calling emulate->task_switch() would
fix this without needing to double up the task switch code.  Other
examples like this are VMMCALL/VMCALL, which should invoke the hypercall
path even if emulated.

On a different stance, we currently have multiple bits of code
implementing accessing/caching/updating of segment registers for hvm
guests.  With the introduction of the ->validate() hook, we should be
able to share all of this logic between the shadow and general emulation
paths, as it isn't use-dependent.

Overall, I think would cause a net reduction of logic living in
{svm,vmx}_vmexit_handler(), and a net simplification of other
emulator-related logic in the hypervisor.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Jan Beulich]

>>> On 13.12.16 at 13:00, <andrew.cooper3@citrix.com> wrote:
> During the most recent Cambridge Hackathon (April 2016), there was a
> suggestion made (sorry - I don't recall from whom) to feed the the
> SVM/VMX intercept information into a slightly more general emulate
> framework, rather than to try to implement common functionality in 3
> separate locations.

I don't recall such a suggestion, so can you perhaps share a few
more of the details? Is this to basically invoke the insn emulator
here (once suitable extended)? If so, I think we may want to
approach this the other direction - first make the emulator
complete enough to be usable here, and only then funnel those
code paths into it. And to be honest, on the road towards
completion of the emulator I think the SVM/VMX insns are pretty
close to the end of the priority list.

> On a different stance, we currently have multiple bits of code
> implementing accessing/caching/updating of segment registers for hvm
> guests.  With the introduction of the ->validate() hook, we should be
> able to share all of this logic between the shadow and general emulation
> paths, as it isn't use-dependent.

I fully agree that any elimination of code duplication is a good
goal.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Andrew Cooper]

On 13/12/16 12:55, Jan Beulich wrote:
>>>> On 13.12.16 at 13:00, <andrew.cooper3@citrix.com> wrote:
>> During the most recent Cambridge Hackathon (April 2016), there was a
>> suggestion made (sorry - I don't recall from whom) to feed the the
>> SVM/VMX intercept information into a slightly more general emulate
>> framework, rather than to try to implement common functionality in 3
>> separate locations.
> I don't recall such a suggestion, so can you perhaps share a few
> more of the details?

It was an offhand comment over coffee.

> Is this to basically invoke the insn emulator
> here (once suitable extended)? If so, I think we may want to
> approach this the other direction - first make the emulator
> complete enough to be usable here, and only then funnel those
> code paths into it.

Perhaps I didn't describe it very well.

Think of it a bit more like introducing a new action emulator (name
definitely subject to improvement), which implements things such as
wrmsr, cpuid, pagewalk, task_switch, etc.

The vmexit helpers, given decode assistance from hardware, can directly
call action->task_switch().  If insufficient information is available
(e.g. LMSW on AMD), the helpers invoke the instruction emulator to work
out what to do, and the instruction emulator would invoke the action
emulator as part of its execute phase.

Wherever possible, the action emulator should be guest-neutral, and
ideally the single point of implementation of non-architectural actions
such as "the vm_event subsystem is interested in this."

> And to be honest, on the road towards
> completion of the emulator I think the SVM/VMX insns are pretty
> close to the end of the priority list.

I'd expect them to show up frequently during introspection, although
maybe I am wrong.  Razvan: Any thoughts?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Razvan Cojocaru]

Hello, and first of all thanks for the discussion!

> Think of it a bit more like introducing a new action emulator (name
> definitely subject to improvement), which implements things such as
> wrmsr, cpuid, pagewalk, task_switch, etc.
> 
> The vmexit helpers, given decode assistance from hardware, can directly
> call action->task_switch().  If insufficient information is available
> (e.g. LMSW on AMD), the helpers invoke the instruction emulator to work
> out what to do, and the instruction emulator would invoke the action
> emulator as part of its execute phase.
> 
> Wherever possible, the action emulator should be guest-neutral, and
> ideally the single point of implementation of non-architectural actions
> such as "the vm_event subsystem is interested in this."
> 
>> And to be honest, on the road towards
>> completion of the emulator I think the SVM/VMX insns are pretty
>> close to the end of the priority list.
> 
> I'd expect them to show up frequently during introspection, although
> maybe I am wrong.  Razvan: Any thoughts?

I definitely think this is a good idea as far as introspection goes -
having a single contact surface with the underlying logic would be a
significant improvement.

As for SVM/VMX instructions, we're interested in anything that is able
to trigger an EPT fault (and hence a mem_access event) - we've had our
share of adventures with VMX-specific instructions, so they're not low
priority for us.


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Andrew Cooper]

On 13/12/16 15:58, Razvan Cojocaru wrote:
> Hello, and first of all thanks for the discussion!
>
>> Think of it a bit more like introducing a new action emulator (name
>> definitely subject to improvement), which implements things such as
>> wrmsr, cpuid, pagewalk, task_switch, etc.
>>
>> The vmexit helpers, given decode assistance from hardware, can directly
>> call action->task_switch().  If insufficient information is available
>> (e.g. LMSW on AMD), the helpers invoke the instruction emulator to work
>> out what to do, and the instruction emulator would invoke the action
>> emulator as part of its execute phase.
>>
>> Wherever possible, the action emulator should be guest-neutral, and
>> ideally the single point of implementation of non-architectural actions
>> such as "the vm_event subsystem is interested in this."
>>
>>> And to be honest, on the road towards
>>> completion of the emulator I think the SVM/VMX insns are pretty
>>> close to the end of the priority list.
>> I'd expect them to show up frequently during introspection, although
>> maybe I am wrong.  Razvan: Any thoughts?
> I definitely think this is a good idea as far as introspection goes -
> having a single contact surface with the underlying logic would be a
> significant improvement.
>
> As for SVM/VMX instructions, we're interested in anything that is able
> to trigger an EPT fault (and hence a mem_access event) - we've had our
> share of adventures with VMX-specific instructions, so they're not low
> priority for us.

In reality, this is any instruction if you set EPT NX on a page, I presume?

Do you have stats on which instructions you most frequently have to
singlestep because of lack of emulator support, or is the spread
essentially random?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Razvan Cojocaru]

On 12/13/2016 07:10 PM, Andrew Cooper wrote:
> On 13/12/16 15:58, Razvan Cojocaru wrote:
>> Hello, and first of all thanks for the discussion!
>>
>>> Think of it a bit more like introducing a new action emulator (name
>>> definitely subject to improvement), which implements things such as
>>> wrmsr, cpuid, pagewalk, task_switch, etc.
>>>
>>> The vmexit helpers, given decode assistance from hardware, can directly
>>> call action->task_switch().  If insufficient information is available
>>> (e.g. LMSW on AMD), the helpers invoke the instruction emulator to work
>>> out what to do, and the instruction emulator would invoke the action
>>> emulator as part of its execute phase.
>>>
>>> Wherever possible, the action emulator should be guest-neutral, and
>>> ideally the single point of implementation of non-architectural actions
>>> such as "the vm_event subsystem is interested in this."
>>>
>>>> And to be honest, on the road towards
>>>> completion of the emulator I think the SVM/VMX insns are pretty
>>>> close to the end of the priority list.
>>> I'd expect them to show up frequently during introspection, although
>>> maybe I am wrong.  Razvan: Any thoughts?
>> I definitely think this is a good idea as far as introspection goes -
>> having a single contact surface with the underlying logic would be a
>> significant improvement.
>>
>> As for SVM/VMX instructions, we're interested in anything that is able
>> to trigger an EPT fault (and hence a mem_access event) - we've had our
>> share of adventures with VMX-specific instructions, so they're not low
>> priority for us.
> 
> In reality, this is any instruction if you set EPT NX on a page, I presume?
> 
> Do you have stats on which instructions you most frequently have to
> singlestep because of lack of emulator support, or is the spread
> essentially random?

We do set some pages NX, so those too, but there are also a lot of
events coming from instructions that simply try to write to a page
marked RX - so we'll have an EPT fault even for an instruction running
from a legitimate page, but which has a destination address in a
read-only page.

We've unfortunately not kept track of them beyond the discussions and
patches that occured on xen-devel, but it's been important enough to
warrant writing patches that basically set the MTF and "single-step"
intstructions that fail emulation.


Thanks,
Razvan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Andrew Cooper]

On 13/12/2016 20:51, Razvan Cojocaru wrote:
> On 12/13/2016 07:10 PM, Andrew Cooper wrote:
>> On 13/12/16 15:58, Razvan Cojocaru wrote:
>>> Hello, and first of all thanks for the discussion!
>>>
>>>> Think of it a bit more like introducing a new action emulator (name
>>>> definitely subject to improvement), which implements things such as
>>>> wrmsr, cpuid, pagewalk, task_switch, etc.
>>>>
>>>> The vmexit helpers, given decode assistance from hardware, can directly
>>>> call action->task_switch().  If insufficient information is available
>>>> (e.g. LMSW on AMD), the helpers invoke the instruction emulator to work
>>>> out what to do, and the instruction emulator would invoke the action
>>>> emulator as part of its execute phase.
>>>>
>>>> Wherever possible, the action emulator should be guest-neutral, and
>>>> ideally the single point of implementation of non-architectural actions
>>>> such as "the vm_event subsystem is interested in this."
>>>>
>>>>> And to be honest, on the road towards
>>>>> completion of the emulator I think the SVM/VMX insns are pretty
>>>>> close to the end of the priority list.
>>>> I'd expect them to show up frequently during introspection, although
>>>> maybe I am wrong.  Razvan: Any thoughts?
>>> I definitely think this is a good idea as far as introspection goes -
>>> having a single contact surface with the underlying logic would be a
>>> significant improvement.
>>>
>>> As for SVM/VMX instructions, we're interested in anything that is able
>>> to trigger an EPT fault (and hence a mem_access event) - we've had our
>>> share of adventures with VMX-specific instructions, so they're not low
>>> priority for us.
>> In reality, this is any instruction if you set EPT NX on a page, I presume?
>>
>> Do you have stats on which instructions you most frequently have to
>> singlestep because of lack of emulator support, or is the spread
>> essentially random?
> We do set some pages NX, so those too, but there are also a lot of
> events coming from instructions that simply try to write to a page
> marked RX - so we'll have an EPT fault even for an instruction running
> from a legitimate page, but which has a destination address in a
> read-only page.
>
> We've unfortunately not kept track of them beyond the discussions and
> patches that occured on xen-devel, but it's been important enough to
> warrant writing patches that basically set the MTF and "single-step"
> intstructions that fail emulation.

The eventual plan is to have a fully complete emulator (but I'd be lying
if I suggested that that was just around the corner).

Given the previous use of the instruction emulator, I'd be surprised if
there were many instructions with memory operands which we were still
missing, although I suppose our support of non-mov SSE instructions is
about 0.

It would certainly be interesting to see a list of "most frequent
instruction unsupported by the emulator" if you were to happen upon
one.  While there is no shortage of identifiable holes in the emulator,
there is also no priority order to fix them in.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Razvan Cojocaru]

On 12/13/2016 11:03 PM, Andrew Cooper wrote:
> On 13/12/2016 20:51, Razvan Cojocaru wrote:
>> On 12/13/2016 07:10 PM, Andrew Cooper wrote:
>>> On 13/12/16 15:58, Razvan Cojocaru wrote:
>>>> Hello, and first of all thanks for the discussion!
>>>>
>>>>> Think of it a bit more like introducing a new action emulator (name
>>>>> definitely subject to improvement), which implements things such as
>>>>> wrmsr, cpuid, pagewalk, task_switch, etc.
>>>>>
>>>>> The vmexit helpers, given decode assistance from hardware, can directly
>>>>> call action->task_switch().  If insufficient information is available
>>>>> (e.g. LMSW on AMD), the helpers invoke the instruction emulator to work
>>>>> out what to do, and the instruction emulator would invoke the action
>>>>> emulator as part of its execute phase.
>>>>>
>>>>> Wherever possible, the action emulator should be guest-neutral, and
>>>>> ideally the single point of implementation of non-architectural actions
>>>>> such as "the vm_event subsystem is interested in this."
>>>>>
>>>>>> And to be honest, on the road towards
>>>>>> completion of the emulator I think the SVM/VMX insns are pretty
>>>>>> close to the end of the priority list.
>>>>> I'd expect them to show up frequently during introspection, although
>>>>> maybe I am wrong.  Razvan: Any thoughts?
>>>> I definitely think this is a good idea as far as introspection goes -
>>>> having a single contact surface with the underlying logic would be a
>>>> significant improvement.
>>>>
>>>> As for SVM/VMX instructions, we're interested in anything that is able
>>>> to trigger an EPT fault (and hence a mem_access event) - we've had our
>>>> share of adventures with VMX-specific instructions, so they're not low
>>>> priority for us.
>>> In reality, this is any instruction if you set EPT NX on a page, I presume?
>>>
>>> Do you have stats on which instructions you most frequently have to
>>> singlestep because of lack of emulator support, or is the spread
>>> essentially random?
>> We do set some pages NX, so those too, but there are also a lot of
>> events coming from instructions that simply try to write to a page
>> marked RX - so we'll have an EPT fault even for an instruction running
>> from a legitimate page, but which has a destination address in a
>> read-only page.
>>
>> We've unfortunately not kept track of them beyond the discussions and
>> patches that occured on xen-devel, but it's been important enough to
>> warrant writing patches that basically set the MTF and "single-step"
>> intstructions that fail emulation.
> 
> The eventual plan is to have a fully complete emulator (but I'd be lying
> if I suggested that that was just around the corner).
> 
> Given the previous use of the instruction emulator, I'd be surprised if
> there were many instructions with memory operands which we were still
> missing, although I suppose our support of non-mov SSE instructions is
> about 0.

We've had some issues specifically with SSE issues in the past, although
I unfortunately cannot recall exactly which ones now (this happened back
in 2013 IIRC).

> It would certainly be interesting to see a list of "most frequent
> instruction unsupported by the emulator" if you were to happen upon
> one.  While there is no shortage of identifiable holes in the emulator,
> there is also no priority order to fix them in.

I'll try to run some tests tomorrow and see if anything pops up with
basic guest usage. It's unfortunately hard to predict what instruction
will fail emulation, since we're also protecting user applications, so
any instruction that, say, Chrome or Firefox uses and can trigger an EPT
fault is fair game for emulation. Combining user application with
"regular" guest behaviour yields such a number of potentially failed
emulation attempts that we thought it would be important to have a way
to quickly single step un-emulatable instructions.

On a somewhat related note, it's important to also figure out how best
to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
in the past. Maybe that's also worth taking into consideration at this
early stage.


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Andrew Cooper]

On 13/12/2016 21:55, Razvan Cojocaru wrote:
> On 12/13/2016 11:03 PM, Andrew Cooper wrote:
>> On 13/12/2016 20:51, Razvan Cojocaru wrote:
>>> On 12/13/2016 07:10 PM, Andrew Cooper wrote:
>>>> On 13/12/16 15:58, Razvan Cojocaru wrote:
>>>>> Hello, and first of all thanks for the discussion!
>>>>>
>>>>>> Think of it a bit more like introducing a new action emulator (name
>>>>>> definitely subject to improvement), which implements things such as
>>>>>> wrmsr, cpuid, pagewalk, task_switch, etc.
>>>>>>
>>>>>> The vmexit helpers, given decode assistance from hardware, can directly
>>>>>> call action->task_switch().  If insufficient information is available
>>>>>> (e.g. LMSW on AMD), the helpers invoke the instruction emulator to work
>>>>>> out what to do, and the instruction emulator would invoke the action
>>>>>> emulator as part of its execute phase.
>>>>>>
>>>>>> Wherever possible, the action emulator should be guest-neutral, and
>>>>>> ideally the single point of implementation of non-architectural actions
>>>>>> such as "the vm_event subsystem is interested in this."
>>>>>>
>>>>>>> And to be honest, on the road towards
>>>>>>> completion of the emulator I think the SVM/VMX insns are pretty
>>>>>>> close to the end of the priority list.
>>>>>> I'd expect them to show up frequently during introspection, although
>>>>>> maybe I am wrong.  Razvan: Any thoughts?
>>>>> I definitely think this is a good idea as far as introspection goes -
>>>>> having a single contact surface with the underlying logic would be a
>>>>> significant improvement.
>>>>>
>>>>> As for SVM/VMX instructions, we're interested in anything that is able
>>>>> to trigger an EPT fault (and hence a mem_access event) - we've had our
>>>>> share of adventures with VMX-specific instructions, so they're not low
>>>>> priority for us.
>>>> In reality, this is any instruction if you set EPT NX on a page, I presume?
>>>>
>>>> Do you have stats on which instructions you most frequently have to
>>>> singlestep because of lack of emulator support, or is the spread
>>>> essentially random?
>>> We do set some pages NX, so those too, but there are also a lot of
>>> events coming from instructions that simply try to write to a page
>>> marked RX - so we'll have an EPT fault even for an instruction running
>>> from a legitimate page, but which has a destination address in a
>>> read-only page.
>>>
>>> We've unfortunately not kept track of them beyond the discussions and
>>> patches that occured on xen-devel, but it's been important enough to
>>> warrant writing patches that basically set the MTF and "single-step"
>>> intstructions that fail emulation.
>> The eventual plan is to have a fully complete emulator (but I'd be lying
>> if I suggested that that was just around the corner).
>>
>> Given the previous use of the instruction emulator, I'd be surprised if
>> there were many instructions with memory operands which we were still
>> missing, although I suppose our support of non-mov SSE instructions is
>> about 0.
> We've had some issues specifically with SSE issues in the past, although
> I unfortunately cannot recall exactly which ones now (this happened back
> in 2013 IIRC).
>
>> It would certainly be interesting to see a list of "most frequent
>> instruction unsupported by the emulator" if you were to happen upon
>> one.  While there is no shortage of identifiable holes in the emulator,
>> there is also no priority order to fix them in.
> I'll try to run some tests tomorrow and see if anything pops up with
> basic guest usage. It's unfortunately hard to predict what instruction
> will fail emulation, since we're also protecting user applications, so
> any instruction that, say, Chrome or Firefox uses and can trigger an EPT
> fault is fair game for emulation. Combining user application with
> "regular" guest behaviour yields such a number of potentially failed
> emulation attempts that we thought it would be important to have a way
> to quickly single step un-emulatable instructions.
>
> On a somewhat related note, it's important to also figure out how best
> to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
> in the past. Maybe that's also worth taking into consideration at this
> early stage.

Funny you should ask that.

The only possible way to do this safely is to have the emulator map the
target frame(s) and execute a locked stub instruction with a memory
operand pointing at the mapping.  We have no other way of interacting
with the cache coherency fabric.

I think the shadow cmpxchg() hook is nearly there, although the
implementation does need to up into the body of the emulator along with
a map()/unmap() pair of primitives.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Tian, Kevin]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Tuesday, December 13, 2016 8:56 PM
> 
> >>> On 13.12.16 at 13:00, <andrew.cooper3@citrix.com> wrote:
> > During the most recent Cambridge Hackathon (April 2016), there was a
> > suggestion made (sorry - I don't recall from whom) to feed the the
> > SVM/VMX intercept information into a slightly more general emulate
> > framework, rather than to try to implement common functionality in 3
> > separate locations.
> 
> I don't recall such a suggestion, so can you perhaps share a few
> more of the details? Is this to basically invoke the insn emulator
> here (once suitable extended)? If so, I think we may want to
> approach this the other direction - first make the emulator
> complete enough to be usable here, and only then funnel those
> code paths into it. And to be honest, on the road towards
> completion of the emulator I think the SVM/VMX insns are pretty
> close to the end of the priority list.
> 
> > On a different stance, we currently have multiple bits of code
> > implementing accessing/caching/updating of segment registers for hvm
> > guests.  With the introduction of the ->validate() hook, we should be
> > able to share all of this logic between the shadow and general emulation
> > paths, as it isn't use-dependent.
> 
> I fully agree that any elimination of code duplication is a good
> goal.
> 

yes, it's a good thing.

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Jan Beulich]

>>> On 13.12.16 at 23:02, <andrew.cooper3@citrix.com> wrote:
> On 13/12/2016 21:55, Razvan Cojocaru wrote:
>> On a somewhat related note, it's important to also figure out how best
>> to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
>> in the past. Maybe that's also worth taking into consideration at this
>> early stage.
> 
> Funny you should ask that.
> 
> The only possible way to do this safely is to have the emulator map the
> target frame(s) and execute a locked stub instruction with a memory
> operand pointing at the mapping.  We have no other way of interacting
> with the cache coherency fabric.

Well, that approach is necessary only if one path (vCPU) can write
to a page, while another one needs emulation. If pages are globally
write-protected, an approach following the model from Razvan's
earlier patch (which I have no idea what has become of) would
seem to suffice.

> I think the shadow cmpxchg() hook is nearly there, although the
> implementation does need to up into the body of the emulator along with
> a map()/unmap() pair of primitives.

To be honest, I don't think we should have map/unmap primitives.
Instead the cmpxchg() hook needs to be told enough to properly
do its job.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Razvan Cojocaru]

On 12/14/2016 09:14 AM, Jan Beulich wrote:
>>>> On 13.12.16 at 23:02, <andrew.cooper3@citrix.com> wrote:
>> On 13/12/2016 21:55, Razvan Cojocaru wrote:
>>> On a somewhat related note, it's important to also figure out how best
>>> to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
>>> in the past. Maybe that's also worth taking into consideration at this
>>> early stage.
>>
>> Funny you should ask that.
>>
>> The only possible way to do this safely is to have the emulator map the
>> target frame(s) and execute a locked stub instruction with a memory
>> operand pointing at the mapping.  We have no other way of interacting
>> with the cache coherency fabric.
> 
> Well, that approach is necessary only if one path (vCPU) can write
> to a page, while another one needs emulation. If pages are globally
> write-protected, an approach following the model from Razvan's
> earlier patch (which I have no idea what has become of) would
> seem to suffice.

As previously stated, you've raised performance concerns which seemed to
require a different direction, namely the one Andrew is now suggesting,
which indeed, aside from being somewhat faster is also safer for all
cases (including the one you've mentioned, where one path can write
normally and the other does so via emulation).

The old patch itself is still alive in the XenServer patch queue, albeit
quite unlikely to be trivial to apply to the current Xen 4.9-unstable
code in its current form:

https://github.com/xenserver/xen-4.7.pg/blob/master/master/xen-x86-emulate-syncrhonise-LOCKed-instruction-emulation.patch

Again, if you decide that this patch is preferable, I can try to rework
it for the current version of Xen.


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Razvan Cojocaru]

On 12/13/2016 07:10 PM, Andrew Cooper wrote:
> On 13/12/16 15:58, Razvan Cojocaru wrote:
>> Hello, and first of all thanks for the discussion!
>>
>>> Think of it a bit more like introducing a new action emulator (name
>>> definitely subject to improvement), which implements things such as
>>> wrmsr, cpuid, pagewalk, task_switch, etc.
>>>
>>> The vmexit helpers, given decode assistance from hardware, can directly
>>> call action->task_switch().  If insufficient information is available
>>> (e.g. LMSW on AMD), the helpers invoke the instruction emulator to work
>>> out what to do, and the instruction emulator would invoke the action
>>> emulator as part of its execute phase.
>>>
>>> Wherever possible, the action emulator should be guest-neutral, and
>>> ideally the single point of implementation of non-architectural actions
>>> such as "the vm_event subsystem is interested in this."
>>>
>>>> And to be honest, on the road towards
>>>> completion of the emulator I think the SVM/VMX insns are pretty
>>>> close to the end of the priority list.
>>> I'd expect them to show up frequently during introspection, although
>>> maybe I am wrong.  Razvan: Any thoughts?
>> I definitely think this is a good idea as far as introspection goes -
>> having a single contact surface with the underlying logic would be a
>> significant improvement.
>>
>> As for SVM/VMX instructions, we're interested in anything that is able
>> to trigger an EPT fault (and hence a mem_access event) - we've had our
>> share of adventures with VMX-specific instructions, so they're not low
>> priority for us.
> 
> In reality, this is any instruction if you set EPT NX on a page, I presume?
> 
> Do you have stats on which instructions you most frequently have to
> singlestep because of lack of emulator support, or is the spread
> essentially random?

Here's what I've gathered just now with nothing more than just starting
up, firing up Internet Explorer and Firefox, and shutting down a few
Windows 7 (32 and 64 bit) guests:

64-bit guest 1:

(XEN) Mem event emulation failed: d1v0 32bit @ 0023:7489f1f8 -> 66 0f 12
44 24 04 ba 00 00 00 00 66 0f 28 e8 66

64-bit guest 2:

(XEN) Mem event emulation failed: d2v1 32bit @ 0023:734cf031 -> 66 0f ef
c0 51 53 8b c1 83 e0 0f 85 c0 75 7f 8b

32-bit guest:

(XEN) Mem event emulation failed: d3v0 32bit @ 001b:7359f1f8 -> 66 0f 12
44 24 04 ba 00 00 00 00 66 0f 28 e8 66
(XEN) Mem event emulation failed: d3v0 32bit @ 001b:7359e7d5 -> 66 0f 12
0d 90 f1 59 73 f2 0f 59 c8 66 0f 12 15
(XEN) Mem event emulation failed: d3v0 32bit @ 0008:821d385f -> 0f 6e 06
0f 72 d0 18 0f ef 05 08 f6 32 82 0f 61

This is with 4.9-unstable master, and only took about 5 minutes per guest.


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Jan Beulich]

>>> On 14.12.16 at 09:53, <rcojocaru@bitdefender.com> wrote:
> On 12/13/2016 07:10 PM, Andrew Cooper wrote:
>> Do you have stats on which instructions you most frequently have to
>> singlestep because of lack of emulator support, or is the spread
>> essentially random?
> 
> Here's what I've gathered just now with nothing more than just starting
> up, firing up Internet Explorer and Firefox, and shutting down a few
> Windows 7 (32 and 64 bit) guests:
> 
> 64-bit guest 1:
> 
> (XEN) Mem event emulation failed: d1v0 32bit @ 0023:7489f1f8 -> 66 0f 12
> 44 24 04 ba 00 00 00 00 66 0f 28 e8 66
> 
> 64-bit guest 2:
> 
> (XEN) Mem event emulation failed: d2v1 32bit @ 0023:734cf031 -> 66 0f ef
> c0 51 53 8b c1 83 e0 0f 85 c0 75 7f 8b
> 
> 32-bit guest:
> 
> (XEN) Mem event emulation failed: d3v0 32bit @ 001b:7359f1f8 -> 66 0f 12
> 44 24 04 ba 00 00 00 00 66 0f 28 e8 66
> (XEN) Mem event emulation failed: d3v0 32bit @ 001b:7359e7d5 -> 66 0f 12
> 0d 90 f1 59 73 f2 0f 59 c8 66 0f 12 15
> (XEN) Mem event emulation failed: d3v0 32bit @ 0008:821d385f -> 0f 6e 06
> 0f 72 d0 18 0f ef 05 08 f6 32 82 0f 61
> 
> This is with 4.9-unstable master, and only took about 5 minutes per guest.

Thanks. These appear to all fall into the SSEn category, which I hope
to find time to take care of as a few big groups, once some more of
my pending patch queue has gone in.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Andrew Cooper]

On 14/12/16 08:53, Razvan Cojocaru wrote:
> On 12/13/2016 07:10 PM, Andrew Cooper wrote:
>> On 13/12/16 15:58, Razvan Cojocaru wrote:
>>> Hello, and first of all thanks for the discussion!
>>>
>>>> Think of it a bit more like introducing a new action emulator (name
>>>> definitely subject to improvement), which implements things such as
>>>> wrmsr, cpuid, pagewalk, task_switch, etc.
>>>>
>>>> The vmexit helpers, given decode assistance from hardware, can directly
>>>> call action->task_switch().  If insufficient information is available
>>>> (e.g. LMSW on AMD), the helpers invoke the instruction emulator to work
>>>> out what to do, and the instruction emulator would invoke the action
>>>> emulator as part of its execute phase.
>>>>
>>>> Wherever possible, the action emulator should be guest-neutral, and
>>>> ideally the single point of implementation of non-architectural actions
>>>> such as "the vm_event subsystem is interested in this."
>>>>
>>>>> And to be honest, on the road towards
>>>>> completion of the emulator I think the SVM/VMX insns are pretty
>>>>> close to the end of the priority list.
>>>> I'd expect them to show up frequently during introspection, although
>>>> maybe I am wrong.  Razvan: Any thoughts?
>>> I definitely think this is a good idea as far as introspection goes -
>>> having a single contact surface with the underlying logic would be a
>>> significant improvement.
>>>
>>> As for SVM/VMX instructions, we're interested in anything that is able
>>> to trigger an EPT fault (and hence a mem_access event) - we've had our
>>> share of adventures with VMX-specific instructions, so they're not low
>>> priority for us.
>> In reality, this is any instruction if you set EPT NX on a page, I presume?
>>
>> Do you have stats on which instructions you most frequently have to
>> singlestep because of lack of emulator support, or is the spread
>> essentially random?
> Here's what I've gathered just now with nothing more than just starting
> up, firing up Internet Explorer and Firefox, and shutting down a few
> Windows 7 (32 and 64 bit) guests:
>
> 64-bit guest 1:
>
> (XEN) Mem event emulation failed: d1v0 32bit @ 0023:7489f1f8 -> 66 0f 12
> 44 24 04 ba 00 00 00 00 66 0f 28 e8 66

movlpd 0x4(%rsp),%xmm0

>
> 64-bit guest 2:
>
> (XEN) Mem event emulation failed: d2v1 32bit @ 0023:734cf031 -> 66 0f ef
> c0 51 53 8b c1 83 e0 0f 85 c0 75 7f 8b

pxor   %xmm0,%xmm0

>
> 32-bit guest:
>
> (XEN) Mem event emulation failed: d3v0 32bit @ 001b:7359f1f8 -> 66 0f 12
> 44 24 04 ba 00 00 00 00 66 0f 28 e8 66

Another movlpd 0x4(%esp),%xmm0

> (XEN) Mem event emulation failed: d3v0 32bit @ 001b:7359e7d5 -> 66 0f 12
> 0d 90 f1 59 73 f2 0f 59 c8 66 0f 12 15

movlpd 0x7359f190,%xmm1

The movlpd's should be easy to implement.  They aren't meaningfully
different from their integer counterparts in terms of needs for the
emulator.

> (XEN) Mem event emulation failed: d3v0 32bit @ 0008:821d385f -> 0f 6e 06
> 0f 72 d0 18 0f ef 05 08 f6 32 82 0f 61

This is just a straight movd (%esi),%mm0

I could have sworn we already had support for this...

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Jan Beulich]

>>> On 14.12.16 at 11:43, <andrew.cooper3@citrix.com> wrote:
> The movlpd's should be easy to implement.  They aren't meaningfully
> different from their integer counterparts in terms of needs for the
> emulator.

Well, the thing here is the increasing complexity of determining
the right size to do the actual memory access with. My plan with
all of SSEn/AVXn is to primarily group instructions by their memory
access patterns, rather than by base functionality (as is the case
now) - we're using a stub anyway, and what exactly an insn does
is of little interest (as long as it doesn't do anything unusual).
That'll then also hopefully allow to simplify the relatively
convoluted way we're currently dealing with the few insns we do
support.

>> (XEN) Mem event emulation failed: d3v0 32bit @ 0008:821d385f -> 0f 6e 06
>> 0f 72 d0 18 0f ef 05 08 f6 32 82 0f 61
> 
> This is just a straight movd (%esi),%mm0
> 
> I could have sworn we already had support for this...

See the set of three patches sent earlier today: So far we support
only their store forms, yet this is a load.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Andrew Cooper]

On 14/12/16 11:13, Jan Beulich wrote:
>>>> On 14.12.16 at 11:43, <andrew.cooper3@citrix.com> wrote:
>> The movlpd's should be easy to implement.  They aren't meaningfully
>> different from their integer counterparts in terms of needs for the
>> emulator.
> Well, the thing here is the increasing complexity of determining
> the right size to do the actual memory access with. My plan with
> all of SSEn/AVXn is to primarily group instructions by their memory
> access patterns, rather than by base functionality (as is the case
> now) - we're using a stub anyway, and what exactly an insn does
> is of little interest (as long as it doesn't do anything unusual).
> That'll then also hopefully allow to simplify the relatively
> convoluted way we're currently dealing with the few insns we do
> support.

Ok.  Sounds like a good plan.

>
>>> (XEN) Mem event emulation failed: d3v0 32bit @ 0008:821d385f -> 0f 6e 06
>>> 0f 72 d0 18 0f ef 05 08 f6 32 82 0f 61
>> This is just a straight movd (%esi),%mm0
>>
>> I could have sworn we already had support for this...
> See the set of three patches sent earlier today: So far we support
> only their store forms, yet this is a load.

Very true.  I will take a look at your other series.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Andrew Cooper]

On 14/12/16 07:37, Razvan Cojocaru wrote:
> On 12/14/2016 09:14 AM, Jan Beulich wrote:
>>>>> On 13.12.16 at 23:02, <andrew.cooper3@citrix.com> wrote:
>>> On 13/12/2016 21:55, Razvan Cojocaru wrote:
>>>> On a somewhat related note, it's important to also figure out how best
>>>> to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
>>>> in the past. Maybe that's also worth taking into consideration at this
>>>> early stage.
>>> Funny you should ask that.
>>>
>>> The only possible way to do this safely is to have the emulator map the
>>> target frame(s) and execute a locked stub instruction with a memory
>>> operand pointing at the mapping.  We have no other way of interacting
>>> with the cache coherency fabric.
>> Well, that approach is necessary only if one path (vCPU) can write
>> to a page, while another one needs emulation. If pages are globally
>> write-protected, an approach following the model from Razvan's
>> earlier patch (which I have no idea what has become of) would
>> seem to suffice.

How do you suggest evaluating whether the page is read-only, without
doing 99% of the work necessary to map it anyway?

> As previously stated, you've raised performance concerns which seemed to
> require a different direction, namely the one Andrew is now suggesting,
> which indeed, aside from being somewhat faster is also safer for all
> cases (including the one you've mentioned, where one path can write
> normally and the other does so via emulation).
>
> The old patch itself is still alive in the XenServer patch queue, albeit
> quite unlikely to be trivial to apply to the current Xen 4.9-unstable
> code in its current form:
>
> https://github.com/xenserver/xen-4.7.pg/blob/master/master/xen-x86-emulate-syncrhonise-LOCKed-instruction-emulation.patch
>
> Again, if you decide that this patch is preferable, I can try to rework
> it for the current version of Xen.

FWIW, I always have a version of the XenServer patch queue rebased on to
staging, which is how I submit proposed upstream development work for
testing in the XenServer automation system.  (This particular patch has
been a constant source of textural merge conflicts with the recent
emulator development work)

>> I think the shadow cmpxchg() hook is nearly there, although the
>> implementation does need to up into the body of the emulator along with
>> a map()/unmap() pair of primitives.
>
> To be honest, I don't think we should have map/unmap primitives.
> Instead the cmpxchg() hook needs to be told enough to properly
> do its job.

The current implementation of ->cmpxchg() is certainly sub-par.

However, you can't sensibly implement something like `lock btc ...` in
terms of cmpxchg(), so fixing the behaviour of cmpxchg() isn't
sufficient to fix the atomicity problems.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Jan Beulich]

>>> On 15.12.16 at 13:32, <andrew.cooper3@citrix.com> wrote:
> On 14/12/16 07:37, Razvan Cojocaru wrote:
>> On 12/14/2016 09:14 AM, Jan Beulich wrote:
>>>>>> On 13.12.16 at 23:02, <andrew.cooper3@citrix.com> wrote:
>>>> On 13/12/2016 21:55, Razvan Cojocaru wrote:
>>>>> On a somewhat related note, it's important to also figure out how best
>>>>> to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
>>>>> in the past. Maybe that's also worth taking into consideration at this
>>>>> early stage.
>>>> Funny you should ask that.
>>>>
>>>> The only possible way to do this safely is to have the emulator map the
>>>> target frame(s) and execute a locked stub instruction with a memory
>>>> operand pointing at the mapping.  We have no other way of interacting
>>>> with the cache coherency fabric.
>>> Well, that approach is necessary only if one path (vCPU) can write
>>> to a page, while another one needs emulation. If pages are globally
>>> write-protected, an approach following the model from Razvan's
>>> earlier patch (which I have no idea what has become of) would
>>> seem to suffice.
> 
> How do you suggest evaluating whether the page is read-only, without
> doing 99% of the work necessary to map it anyway?

That wasn't my point. I merely wanted to express that for a limited
set of use cases not using the stub approach would suffice.
Meanwhile I've in fact realized that there's another relevant aspect
requiring us to use an actual CMPXCHG here: Other than a CMP-and-
MOV sequence, CMPXCHG always writes its memory operand, which
for MMIO may make a difference.

>>> I think the shadow cmpxchg() hook is nearly there, although the
>>> implementation does need to up into the body of the emulator along with
>>> a map()/unmap() pair of primitives.
>>
>> To be honest, I don't think we should have map/unmap primitives.
>> Instead the cmpxchg() hook needs to be told enough to properly
>> do its job.
> 
> The current implementation of ->cmpxchg() is certainly sub-par.
> 
> However, you can't sensibly implement something like `lock btc ...` in
> terms of cmpxchg(), so fixing the behaviour of cmpxchg() isn't
> sufficient to fix the atomicity problems.

Well, I've realized that too. The set of LOCK-able insn luckily is
rather small, so I think we instead may want a ->locked_rmw()
op, which all guest uses of LOCK should go to (with an enum
specifying the actual operation).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Razvan Cojocaru]

On 12/14/2016 09:37 AM, Razvan Cojocaru wrote:
> On 12/14/2016 09:14 AM, Jan Beulich wrote:
>>>>> On 13.12.16 at 23:02, <andrew.cooper3@citrix.com> wrote:
>>> On 13/12/2016 21:55, Razvan Cojocaru wrote:
>>>> On a somewhat related note, it's important to also figure out how best
>>>> to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
>>>> in the past. Maybe that's also worth taking into consideration at this
>>>> early stage.
>>>
>>> Funny you should ask that.
>>>
>>> The only possible way to do this safely is to have the emulator map the
>>> target frame(s) and execute a locked stub instruction with a memory
>>> operand pointing at the mapping.  We have no other way of interacting
>>> with the cache coherency fabric.
>>
>> Well, that approach is necessary only if one path (vCPU) can write
>> to a page, while another one needs emulation. If pages are globally
>> write-protected, an approach following the model from Razvan's
>> earlier patch (which I have no idea what has become of) would
>> seem to suffice.
> 
> As previously stated, you've raised performance concerns which seemed to
> require a different direction, namely the one Andrew is now suggesting,
> which indeed, aside from being somewhat faster is also safer for all
> cases (including the one you've mentioned, where one path can write
> normally and the other does so via emulation).
> 
> The old patch itself is still alive in the XenServer patch queue, albeit
> quite unlikely to be trivial to apply to the current Xen 4.9-unstable
> code in its current form:
> 
> https://github.com/xenserver/xen-4.7.pg/blob/master/master/xen-x86-emulate-syncrhonise-LOCKed-instruction-emulation.patch
> 
> Again, if you decide that this patch is preferable, I can try to rework
> it for the current version of Xen.

Sorry to revive this old thread, but I'm still not sure what the
upstream solution for this very real problem should be. Should I bring
back the old patch that synchronizes LOCKed CMPXCHGs (perhaps with
Andrew's kind help, as he's stated that they keep an up-to-date patch
that works against staging)? Or are you considering implementing a stub
as part of the work being done on the emulator?


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Jan Beulich]

>>> On 14.03.17 at 22:07, <rcojocaru@bitdefender.com> wrote:
> On 12/14/2016 09:37 AM, Razvan Cojocaru wrote:
>> On 12/14/2016 09:14 AM, Jan Beulich wrote:
>>>>>> On 13.12.16 at 23:02, <andrew.cooper3@citrix.com> wrote:
>>>> On 13/12/2016 21:55, Razvan Cojocaru wrote:
>>>>> On a somewhat related note, it's important to also figure out how best
>>>>> to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
>>>>> in the past. Maybe that's also worth taking into consideration at this
>>>>> early stage.
>>>>
>>>> Funny you should ask that.
>>>>
>>>> The only possible way to do this safely is to have the emulator map the
>>>> target frame(s) and execute a locked stub instruction with a memory
>>>> operand pointing at the mapping.  We have no other way of interacting
>>>> with the cache coherency fabric.
>>>
>>> Well, that approach is necessary only if one path (vCPU) can write
>>> to a page, while another one needs emulation. If pages are globally
>>> write-protected, an approach following the model from Razvan's
>>> earlier patch (which I have no idea what has become of) would
>>> seem to suffice.
>> 
>> As previously stated, you've raised performance concerns which seemed to
>> require a different direction, namely the one Andrew is now suggesting,
>> which indeed, aside from being somewhat faster is also safer for all
>> cases (including the one you've mentioned, where one path can write
>> normally and the other does so via emulation).
>> 
>> The old patch itself is still alive in the XenServer patch queue, albeit
>> quite unlikely to be trivial to apply to the current Xen 4.9-unstable
>> code in its current form:
>> 
>> 
> https://github.com/xenserver/xen-4.7.pg/blob/master/master/xen-x86-emulate-sy 
> ncrhonise-LOCKed-instruction-emulation.patch
>> 
>> Again, if you decide that this patch is preferable, I can try to rework
>> it for the current version of Xen.
> 
> Sorry to revive this old thread, but I'm still not sure what the
> upstream solution for this very real problem should be. Should I bring
> back the old patch that synchronizes LOCKed CMPXCHGs (perhaps with
> Andrew's kind help, as he's stated that they keep an up-to-date patch
> that works against staging)? Or are you considering implementing a stub
> as part of the work being done on the emulator?

Both are options imo. The stub approach likely would be the long term
better solution, but carries with it quite a bit of emulator rework, since
we'd have to completely change the way memory writes get carried
out: As we'd need to act on the actual (guest) memory location, we'd
have to do a page walk (or possibly two for an access crossing a page
boundary) before running the stub, presumably completely replacing
the ->write() hook. Compared with this making the ->cmpxchg() hook
work as originally intended seems to be the more straightforward
solution.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Andrew Cooper]

On 15/03/17 07:49, Jan Beulich wrote:
>>>> On 14.03.17 at 22:07, <rcojocaru@bitdefender.com> wrote:
>> On 12/14/2016 09:37 AM, Razvan Cojocaru wrote:
>>> On 12/14/2016 09:14 AM, Jan Beulich wrote:
>>>>>>> On 13.12.16 at 23:02, <andrew.cooper3@citrix.com> wrote:
>>>>> On 13/12/2016 21:55, Razvan Cojocaru wrote:
>>>>>> On a somewhat related note, it's important to also figure out how best
>>>>>> to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
>>>>>> in the past. Maybe that's also worth taking into consideration at this
>>>>>> early stage.
>>>>> Funny you should ask that.
>>>>>
>>>>> The only possible way to do this safely is to have the emulator map the
>>>>> target frame(s) and execute a locked stub instruction with a memory
>>>>> operand pointing at the mapping.  We have no other way of interacting
>>>>> with the cache coherency fabric.
>>>> Well, that approach is necessary only if one path (vCPU) can write
>>>> to a page, while another one needs emulation. If pages are globally
>>>> write-protected, an approach following the model from Razvan's
>>>> earlier patch (which I have no idea what has become of) would
>>>> seem to suffice.
>>> As previously stated, you've raised performance concerns which seemed to
>>> require a different direction, namely the one Andrew is now suggesting,
>>> which indeed, aside from being somewhat faster is also safer for all
>>> cases (including the one you've mentioned, where one path can write
>>> normally and the other does so via emulation).
>>>
>>> The old patch itself is still alive in the XenServer patch queue, albeit
>>> quite unlikely to be trivial to apply to the current Xen 4.9-unstable
>>> code in its current form:
>>>
>>>
>> https://github.com/xenserver/xen-4.7.pg/blob/master/master/xen-x86-emulate-sy 
>> ncrhonise-LOCKed-instruction-emulation.patch
>>> Again, if you decide that this patch is preferable, I can try to rework
>>> it for the current version of Xen.
>> Sorry to revive this old thread, but I'm still not sure what the
>> upstream solution for this very real problem should be. Should I bring
>> back the old patch that synchronizes LOCKed CMPXCHGs (perhaps with
>> Andrew's kind help, as he's stated that they keep an up-to-date patch
>> that works against staging)? Or are you considering implementing a stub
>> as part of the work being done on the emulator?
> Both are options imo. The stub approach likely would be the long term
> better solution, but carries with it quite a bit of emulator rework, since
> we'd have to completely change the way memory writes get carried
> out: As we'd need to act on the actual (guest) memory location, we'd
> have to do a page walk (or possibly two for an access crossing a page
> boundary) before running the stub, presumably completely replacing
> the ->write() hook. Compared with this making the ->cmpxchg() hook
> work as originally intended seems to be the more straightforward
> solution.

We already need to change how reads and writes happen.  As it currently
stands, accesses which cross a page boundary are not handled correctly,
and will complete a partial read/write on the first page before finding
that the 2nd page takes a pagefault.  (The root of the problem is that
hvm_copy() has dual use; originally as a memcpy(), and later to
implement an individual instructions access.)

The HVM side of the code needs to be altered to work in the same way
that sh_x86_emulate_{write,cmpxchg}() currently uses
sh_emulate_map_dest(), except that the read side needs including as
well.  This important for handling MMIO where reads may have side effects.

Once that is complete, the cmpxchg hook at least should have proper
atomic properties.

The next question is how to go about making all other LOCKed
instructions have atomic properties.  One suggestion was to try and
implement all LOCKed instructions in terms of cmpxchg, but I suspect
that will come with an unreasonably high overhead for introspection when
all vcpus are hitting the same spinlock.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Jan Beulich]

>>> On 15.03.17 at 13:08, <andrew.cooper3@citrix.com> wrote:
> On 15/03/17 07:49, Jan Beulich wrote:
>>>>> On 14.03.17 at 22:07, <rcojocaru@bitdefender.com> wrote:
>>> On 12/14/2016 09:37 AM, Razvan Cojocaru wrote:
>>>> On 12/14/2016 09:14 AM, Jan Beulich wrote:
>>>>>>>> On 13.12.16 at 23:02, <andrew.cooper3@citrix.com> wrote:
>>>>>> On 13/12/2016 21:55, Razvan Cojocaru wrote:
>>>>>>> On a somewhat related note, it's important to also figure out how best
>>>>>>> to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
>>>>>>> in the past. Maybe that's also worth taking into consideration at this
>>>>>>> early stage.
>>>>>> Funny you should ask that.
>>>>>>
>>>>>> The only possible way to do this safely is to have the emulator map the
>>>>>> target frame(s) and execute a locked stub instruction with a memory
>>>>>> operand pointing at the mapping.  We have no other way of interacting
>>>>>> with the cache coherency fabric.
>>>>> Well, that approach is necessary only if one path (vCPU) can write
>>>>> to a page, while another one needs emulation. If pages are globally
>>>>> write-protected, an approach following the model from Razvan's
>>>>> earlier patch (which I have no idea what has become of) would
>>>>> seem to suffice.
>>>> As previously stated, you've raised performance concerns which seemed to
>>>> require a different direction, namely the one Andrew is now suggesting,
>>>> which indeed, aside from being somewhat faster is also safer for all
>>>> cases (including the one you've mentioned, where one path can write
>>>> normally and the other does so via emulation).
>>>>
>>>> The old patch itself is still alive in the XenServer patch queue, albeit
>>>> quite unlikely to be trivial to apply to the current Xen 4.9-unstable
>>>> code in its current form:
>>>>
>>>>
>>> https://github.com/xenserver/xen-4.7.pg/blob/master/master/xen-x86-emulate-sy 
>>> ncrhonise-LOCKed-instruction-emulation.patch
>>>> Again, if you decide that this patch is preferable, I can try to rework
>>>> it for the current version of Xen.
>>> Sorry to revive this old thread, but I'm still not sure what the
>>> upstream solution for this very real problem should be. Should I bring
>>> back the old patch that synchronizes LOCKed CMPXCHGs (perhaps with
>>> Andrew's kind help, as he's stated that they keep an up-to-date patch
>>> that works against staging)? Or are you considering implementing a stub
>>> as part of the work being done on the emulator?
>> Both are options imo. The stub approach likely would be the long term
>> better solution, but carries with it quite a bit of emulator rework, since
>> we'd have to completely change the way memory writes get carried
>> out: As we'd need to act on the actual (guest) memory location, we'd
>> have to do a page walk (or possibly two for an access crossing a page
>> boundary) before running the stub, presumably completely replacing
>> the ->write() hook. Compared with this making the ->cmpxchg() hook
>> work as originally intended seems to be the more straightforward
>> solution.
> 
> We already need to change how reads and writes happen.  As it currently
> stands, accesses which cross a page boundary are not handled correctly,
> and will complete a partial read/write on the first page before finding
> that the 2nd page takes a pagefault.  (The root of the problem is that
> hvm_copy() has dual use; originally as a memcpy(), and later to
> implement an individual instructions access.)

Well, even for the memcpy() purpose of the function it would be
better if no partial writes happened (at least when size is
meaningfully smaller than a page).

> The HVM side of the code needs to be altered to work in the same way
> that sh_x86_emulate_{write,cmpxchg}() currently uses
> sh_emulate_map_dest(), except that the read side needs including as
> well.  This important for handling MMIO where reads may have side effects.

Except that commonly MMIO crossing page boundaries is considered
undefined. Are you convinced CPUs never do partial writes when an
access spans pages?

> Once that is complete, the cmpxchg hook at least should have proper
> atomic properties.
> 
> The next question is how to go about making all other LOCKed
> instructions have atomic properties.  One suggestion was to try and
> implement all LOCKed instructions in terms of cmpxchg, but I suspect
> that will come with an unreasonably high overhead for introspection when
> all vcpus are hitting the same spinlock.

Hmm, that's a valid concern.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Razvan Cojocaru]

On 03/15/2017 02:42 PM, Jan Beulich wrote:
>>>> On 15.03.17 at 13:08, <andrew.cooper3@citrix.com> wrote:
>> On 15/03/17 07:49, Jan Beulich wrote:
>>>>>> On 14.03.17 at 22:07, <rcojocaru@bitdefender.com> wrote:
>>>> On 12/14/2016 09:37 AM, Razvan Cojocaru wrote:
>>>>> On 12/14/2016 09:14 AM, Jan Beulich wrote:
>>>>>>>>> On 13.12.16 at 23:02, <andrew.cooper3@citrix.com> wrote:
>>>>>>> On 13/12/2016 21:55, Razvan Cojocaru wrote:
>>>>>>>> On a somewhat related note, it's important to also figure out how best
>>>>>>>> to avoid emulation races such as the LOCK CMPXCHG issue we've discussed
>>>>>>>> in the past. Maybe that's also worth taking into consideration at this
>>>>>>>> early stage.
>>>>>>> Funny you should ask that.
>>>>>>>
>>>>>>> The only possible way to do this safely is to have the emulator map the
>>>>>>> target frame(s) and execute a locked stub instruction with a memory
>>>>>>> operand pointing at the mapping.  We have no other way of interacting
>>>>>>> with the cache coherency fabric.
>>>>>> Well, that approach is necessary only if one path (vCPU) can write
>>>>>> to a page, while another one needs emulation. If pages are globally
>>>>>> write-protected, an approach following the model from Razvan's
>>>>>> earlier patch (which I have no idea what has become of) would
>>>>>> seem to suffice.
>>>>> As previously stated, you've raised performance concerns which seemed to
>>>>> require a different direction, namely the one Andrew is now suggesting,
>>>>> which indeed, aside from being somewhat faster is also safer for all
>>>>> cases (including the one you've mentioned, where one path can write
>>>>> normally and the other does so via emulation).
>>>>>
>>>>> The old patch itself is still alive in the XenServer patch queue, albeit
>>>>> quite unlikely to be trivial to apply to the current Xen 4.9-unstable
>>>>> code in its current form:
>>>>>
>>>>>
>>>> https://github.com/xenserver/xen-4.7.pg/blob/master/master/xen-x86-emulate-sy 
>>>> ncrhonise-LOCKed-instruction-emulation.patch
>>>>> Again, if you decide that this patch is preferable, I can try to rework
>>>>> it for the current version of Xen.
>>>> Sorry to revive this old thread, but I'm still not sure what the
>>>> upstream solution for this very real problem should be. Should I bring
>>>> back the old patch that synchronizes LOCKed CMPXCHGs (perhaps with
>>>> Andrew's kind help, as he's stated that they keep an up-to-date patch
>>>> that works against staging)? Or are you considering implementing a stub
>>>> as part of the work being done on the emulator?
>>> Both are options imo. The stub approach likely would be the long term
>>> better solution, but carries with it quite a bit of emulator rework, since
>>> we'd have to completely change the way memory writes get carried
>>> out: As we'd need to act on the actual (guest) memory location, we'd
>>> have to do a page walk (or possibly two for an access crossing a page
>>> boundary) before running the stub, presumably completely replacing
>>> the ->write() hook. Compared with this making the ->cmpxchg() hook
>>> work as originally intended seems to be the more straightforward
>>> solution.
>>
>> We already need to change how reads and writes happen.  As it currently
>> stands, accesses which cross a page boundary are not handled correctly,
>> and will complete a partial read/write on the first page before finding
>> that the 2nd page takes a pagefault.  (The root of the problem is that
>> hvm_copy() has dual use; originally as a memcpy(), and later to
>> implement an individual instructions access.)
> 
> Well, even for the memcpy() purpose of the function it would be
> better if no partial writes happened (at least when size is
> meaningfully smaller than a page).
> 
>> The HVM side of the code needs to be altered to work in the same way
>> that sh_x86_emulate_{write,cmpxchg}() currently uses
>> sh_emulate_map_dest(), except that the read side needs including as
>> well.  This important for handling MMIO where reads may have side effects.
> 
> Except that commonly MMIO crossing page boundaries is considered
> undefined. Are you convinced CPUs never do partial writes when an
> access spans pages?
> 
>> Once that is complete, the cmpxchg hook at least should have proper
>> atomic properties.
>>
>> The next question is how to go about making all other LOCKed
>> instructions have atomic properties.  One suggestion was to try and
>> implement all LOCKed instructions in terms of cmpxchg, but I suspect
>> that will come with an unreasonably high overhead for introspection when
>> all vcpus are hitting the same spinlock.
> 
> Hmm, that's a valid concern.

If you'd like I could try to keep count of all the emulated LOCK
instructions that happen during a typical introspection session -
however I suspect there won't be that many, since the application will
not mark all of the guest's pages and hence not require emulation for
all LOCKed instructions. I remember that when I originally found the
issue there was a handful of them on guest start-up and then just a few
other here and there.

With that said, should I submit a new version of the original LOCK patch
to have in the meantime (until the fix suggested by Andrew is
implemented, and presumably to be reverted once it lands), or is it not
worth xen-devel's extra time?


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Future x86 emulator direction

[Jan Beulich]

>>> On 15.03.17 at 14:08, <rcojocaru@bitdefender.com> wrote:
> With that said, should I submit a new version of the original LOCK patch
> to have in the meantime (until the fix suggested by Andrew is
> implemented, and presumably to be reverted once it lands), or is it not
> worth xen-devel's extra time?

I think it would be worthwhile, but there's no promise that it'll be
taken.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel