[PATCH 0/4] Add support for XMM fast hypercalls

[PATCH 0/4] Add support for XMM fast hypercalls

[Siddharth Chandrasekaran]

Hyper-V supports the use of XMM registers to perform fast hypercalls.
This allows guests to take advantage of the improved performance of the
fast hypercall interface even though a hypercall may require more than
(the current maximum of) two general purpose registers.

The XMM fast hypercall interface uses an additional six XMM registers
(XMM0 to XMM5) to allow the caller to pass an input parameter block of
up to 112 bytes. Hyper-V can also return data back to the guest in the
remaining XMM registers that are not used by the current hypercall.

Although the Hyper-v TLFS mentions that a guest cannot use this feature
unless the hypervisor advertises support for it, some hypercalls which
we plan on upstreaming in future uses them anyway. This patchset adds
necessary infrastructure for handling input/output via XMM registers and
patches kvm_hv_flush_tlb() to use xmm input arguments.

~ Sid.

Siddharth Chandrasekaran (4):
  KVM: x86: Move FPU register accessors into fpu.h
  KVM: hyper-v: Collect hypercall params into struct
  KVM: x86: kvm_hv_flush_tlb use inputs from XMM registers
  KVM: hyper-v: Advertise support for fast XMM hypercalls

 arch/x86/include/asm/hyperv-tlfs.h |   4 +-
 arch/x86/kvm/emulate.c             | 138 +++--------------
 arch/x86/kvm/fpu.h                 | 140 +++++++++++++++++
 arch/x86/kvm/hyperv.c              | 241 +++++++++++++++++++----------
 4 files changed, 322 insertions(+), 201 deletions(-)
 create mode 100644 arch/x86/kvm/fpu.h

-- 
2.17.1




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Wei Liu]

On Wed, Apr 07, 2021 at 11:29:26PM +0200, Siddharth Chandrasekaran wrote:
> Hyper-V supports the use of XMM registers to perform fast hypercalls.
> This allows guests to take advantage of the improved performance of the
> fast hypercall interface even though a hypercall may require more than
> (the current maximum of) two general purpose registers.
> 
> The XMM fast hypercall interface uses an additional six XMM registers
> (XMM0 to XMM5) to allow the caller to pass an input parameter block of
> up to 112 bytes. Hyper-V can also return data back to the guest in the
> remaining XMM registers that are not used by the current hypercall.
> 
> Although the Hyper-v TLFS mentions that a guest cannot use this feature
> unless the hypervisor advertises support for it, some hypercalls which
> we plan on upstreaming in future uses them anyway. 

No, please don't do this. Check the feature bit(s) before you issue
hypercalls which rely on the extended interface.

Wei.

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Paolo Bonzini]

On 08/04/21 17:28, Wei Liu wrote:
>> Although the Hyper-v TLFS mentions that a guest cannot use this feature
>> unless the hypervisor advertises support for it, some hypercalls which
>> we plan on upstreaming in future uses them anyway.
>
> No, please don't do this. Check the feature bit(s) before you issue
> hypercalls which rely on the extended interface.

Perhaps Siddharth should clarify this, but I read it as Hyper-V being 
buggy and using XMM arguments unconditionally.

Paolo

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Wei Liu]

On Thu, Apr 08, 2021 at 05:30:26PM +0200, Paolo Bonzini wrote:
> On 08/04/21 17:28, Wei Liu wrote:
> > > Although the Hyper-v TLFS mentions that a guest cannot use this feature
> > > unless the hypervisor advertises support for it, some hypercalls which
> > > we plan on upstreaming in future uses them anyway.
> > 
> > No, please don't do this. Check the feature bit(s) before you issue
> > hypercalls which rely on the extended interface.
> 
> Perhaps Siddharth should clarify this, but I read it as Hyper-V being buggy
> and using XMM arguments unconditionally.
> 

There is no code in upstream Linux that uses the XMM fast hypercall
interface at the moment.

If there is such code, it has bugs in it and should be fixed. :-)

Wei.

> Paolo
> 

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Siddharth Chandrasekaran]

On Thu, Apr 08, 2021 at 05:30:26PM +0200, Paolo Bonzini wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> On 08/04/21 17:28, Wei Liu wrote:
> > > Although the Hyper-v TLFS mentions that a guest cannot use this feature
> > > unless the hypervisor advertises support for it, some hypercalls which
> > > we plan on upstreaming in future uses them anyway.
> >
> > No, please don't do this. Check the feature bit(s) before you issue
> > hypercalls which rely on the extended interface.
>
> Perhaps Siddharth should clarify this, but I read it as Hyper-V being
> buggy and using XMM arguments unconditionally.

The guest is at fault here as it expects Hyper-V to consume arguments
from XMM registers for certain hypercalls (that we are working) even if
we didn't expose the feature via CPUID bits.

~ Sid.



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Paolo Bonzini]

On 08/04/21 17:40, Siddharth Chandrasekaran wrote:
>>>> Although the Hyper-v TLFS mentions that a guest cannot use this feature
>>>> unless the hypervisor advertises support for it, some hypercalls which
>>>> we plan on upstreaming in future uses them anyway.
>>> No, please don't do this. Check the feature bit(s) before you issue
>>> hypercalls which rely on the extended interface.
>> Perhaps Siddharth should clarify this, but I read it as Hyper-V being
>> buggy and using XMM arguments unconditionally.
> The guest is at fault here as it expects Hyper-V to consume arguments
> from XMM registers for certain hypercalls (that we are working) even if
> we didn't expose the feature via CPUID bits.

What guest is that?

Paolo

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Siddharth Chandrasekaran]

On Thu, Apr 08, 2021 at 05:48:19PM +0200, Paolo Bonzini wrote:
> On 08/04/21 17:40, Siddharth Chandrasekaran wrote:
> > > > > Although the Hyper-v TLFS mentions that a guest cannot use this feature
> > > > > unless the hypervisor advertises support for it, some hypercalls which
> > > > > we plan on upstreaming in future uses them anyway.
> > > > No, please don't do this. Check the feature bit(s) before you issue
> > > > hypercalls which rely on the extended interface.
> > > Perhaps Siddharth should clarify this, but I read it as Hyper-V being
> > > buggy and using XMM arguments unconditionally.
> > The guest is at fault here as it expects Hyper-V to consume arguments
> > from XMM registers for certain hypercalls (that we are working) even if
> > we didn't expose the feature via CPUID bits.
>
> What guest is that?

It is a Windows Server 2016.



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Siddharth Chandrasekaran]

On Thu, Apr 08, 2021 at 03:38:13PM +0000, Wei Liu wrote:
> On Thu, Apr 08, 2021 at 05:30:26PM +0200, Paolo Bonzini wrote:
> > On 08/04/21 17:28, Wei Liu wrote:
> > > > Although the Hyper-v TLFS mentions that a guest cannot use this feature
> > > > unless the hypervisor advertises support for it, some hypercalls which
> > > > we plan on upstreaming in future uses them anyway.
> > >
> > > No, please don't do this. Check the feature bit(s) before you issue
> > > hypercalls which rely on the extended interface.
> >
> > Perhaps Siddharth should clarify this, but I read it as Hyper-V being buggy
> > and using XMM arguments unconditionally.
> >
> 
> There is no code in upstream Linux that uses the XMM fast hypercall
> interface at the moment.
> 
> If there is such code, it has bugs in it and should be fixed. :-)

None of the existing hypercalls are buggy. These are some hypercalls we
are working on (and planning on upstreaming in the near future). 

~ Sid.



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Wei Liu]

On Thu, Apr 08, 2021 at 05:54:43PM +0200, Siddharth Chandrasekaran wrote:
> On Thu, Apr 08, 2021 at 05:48:19PM +0200, Paolo Bonzini wrote:
> > On 08/04/21 17:40, Siddharth Chandrasekaran wrote:
> > > > > > Although the Hyper-v TLFS mentions that a guest cannot use this feature
> > > > > > unless the hypervisor advertises support for it, some hypercalls which
> > > > > > we plan on upstreaming in future uses them anyway.
> > > > > No, please don't do this. Check the feature bit(s) before you issue
> > > > > hypercalls which rely on the extended interface.
> > > > Perhaps Siddharth should clarify this, but I read it as Hyper-V being
> > > > buggy and using XMM arguments unconditionally.
> > > The guest is at fault here as it expects Hyper-V to consume arguments
> > > from XMM registers for certain hypercalls (that we are working) even if
> > > we didn't expose the feature via CPUID bits.
> >
> > What guest is that?
> 
> It is a Windows Server 2016.

Can you be more specific? Are you implementing some hypercalls from
TLFS? If so, which ones?

Wei.

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Siddharth Chandrasekaran]

On Thu, Apr 08, 2021 at 04:30:18PM +0000, Wei Liu wrote:
> On Thu, Apr 08, 2021 at 05:54:43PM +0200, Siddharth Chandrasekaran wrote:
> > On Thu, Apr 08, 2021 at 05:48:19PM +0200, Paolo Bonzini wrote:
> > > On 08/04/21 17:40, Siddharth Chandrasekaran wrote:
> > > > > > > Although the Hyper-v TLFS mentions that a guest cannot use this feature
> > > > > > > unless the hypervisor advertises support for it, some hypercalls which
> > > > > > > we plan on upstreaming in future uses them anyway.
> > > > > > No, please don't do this. Check the feature bit(s) before you issue
> > > > > > hypercalls which rely on the extended interface.
> > > > > Perhaps Siddharth should clarify this, but I read it as Hyper-V being
> > > > > buggy and using XMM arguments unconditionally.
> > > > The guest is at fault here as it expects Hyper-V to consume arguments
> > > > from XMM registers for certain hypercalls (that we are working) even if
> > > > we didn't expose the feature via CPUID bits.
> > >
> > > What guest is that?
> >
> > It is a Windows Server 2016.
> 
> Can you be more specific? Are you implementing some hypercalls from
> TLFS? If so, which ones?

Yes all of them are from TLFS. We are implementing VSM and there are a
bunch of hypercalls that we have implemented to manage VTL switches,
memory protection and virtual interrupts.

The following 3 hypercalls that use the XMM fast hypercalls are relevant
to this patch set:

HvCallModifyVtlProtectionMask
HvGetVpRegisters 
HvSetVpRegisters 

~ Sid.



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Vitaly Kuznetsov]

Siddharth Chandrasekaran <sidcha@amazon.de> writes:

> On Thu, Apr 08, 2021 at 04:30:18PM +0000, Wei Liu wrote:
>> On Thu, Apr 08, 2021 at 05:54:43PM +0200, Siddharth Chandrasekaran wrote:
>> > On Thu, Apr 08, 2021 at 05:48:19PM +0200, Paolo Bonzini wrote:
>> > > On 08/04/21 17:40, Siddharth Chandrasekaran wrote:
>> > > > > > > Although the Hyper-v TLFS mentions that a guest cannot use this feature
>> > > > > > > unless the hypervisor advertises support for it, some hypercalls which
>> > > > > > > we plan on upstreaming in future uses them anyway.
>> > > > > > No, please don't do this. Check the feature bit(s) before you issue
>> > > > > > hypercalls which rely on the extended interface.
>> > > > > Perhaps Siddharth should clarify this, but I read it as Hyper-V being
>> > > > > buggy and using XMM arguments unconditionally.
>> > > > The guest is at fault here as it expects Hyper-V to consume arguments
>> > > > from XMM registers for certain hypercalls (that we are working) even if
>> > > > we didn't expose the feature via CPUID bits.
>> > >
>> > > What guest is that?
>> >
>> > It is a Windows Server 2016.
>> 
>> Can you be more specific? Are you implementing some hypercalls from
>> TLFS? If so, which ones?
>
> Yes all of them are from TLFS. We are implementing VSM and there are a
> bunch of hypercalls that we have implemented to manage VTL switches,
> memory protection and virtual interrupts.

Wow, sounds awesome! Do you plan to upstream this work?

> The following 3 hypercalls that use the XMM fast hypercalls are relevant
> to this patch set:
>
> HvCallModifyVtlProtectionMask
> HvGetVpRegisters 
> HvSetVpRegisters 

It seems AccessVSM and AccessVpRegisters privilges have implicit
dependency on XMM input/output. This will need to be enforced in KVM
userspace.

-- 
Vitaly

Re: [PATCH 0/4] Add support for XMM fast hypercalls

[Siddharth Chandrasekaran]

On Fri, Apr 09, 2021 at 09:42:41AM +0200, Vitaly Kuznetsov wrote:
> Siddharth Chandrasekaran <sidcha@amazon.de> writes:
> > On Thu, Apr 08, 2021 at 04:30:18PM +0000, Wei Liu wrote:
> >> On Thu, Apr 08, 2021 at 05:54:43PM +0200, Siddharth Chandrasekaran wrote:
> >> > On Thu, Apr 08, 2021 at 05:48:19PM +0200, Paolo Bonzini wrote:
> >> > > On 08/04/21 17:40, Siddharth Chandrasekaran wrote:
> >> > > > > > > Although the Hyper-v TLFS mentions that a guest cannot use this feature
> >> > > > > > > unless the hypervisor advertises support for it, some hypercalls which
> >> > > > > > > we plan on upstreaming in future uses them anyway.
> >> > > > > > No, please don't do this. Check the feature bit(s) before you issue
> >> > > > > > hypercalls which rely on the extended interface.
> >> > > > > Perhaps Siddharth should clarify this, but I read it as Hyper-V being
> >> > > > > buggy and using XMM arguments unconditionally.
> >> > > > The guest is at fault here as it expects Hyper-V to consume arguments
> >> > > > from XMM registers for certain hypercalls (that we are working) even if
> >> > > > we didn't expose the feature via CPUID bits.
> >> > >
> >> > > What guest is that?
> >> >
> >> > It is a Windows Server 2016.
> >>
> >> Can you be more specific? Are you implementing some hypercalls from
> >> TLFS? If so, which ones?
> >
> > Yes all of them are from TLFS. We are implementing VSM and there are a
> > bunch of hypercalls that we have implemented to manage VTL switches,
> > memory protection and virtual interrupts.
> 
> Wow, sounds awesome! Do you plan to upstream this work?

Yes, that is the plan :-)

> > The following 3 hypercalls that use the XMM fast hypercalls are relevant
> > to this patch set:
> >
> > HvCallModifyVtlProtectionMask
> > HvGetVpRegisters
> > HvSetVpRegisters
> 
> It seems AccessVSM and AccessVpRegisters privilges have implicit
> dependency on XMM input/output. This will need to be enforced in KVM
> userspace.

Noted.

~ Sid.



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879