[PATCH v3 0/5] KVM: nVMX: nested VPID emulation

[PATCH v3 0/5] KVM: nVMX: nested VPID emulation

[Wanpeng Li]

v2 -> v3:
 * separate nested_vmx_vpid_caps and move checks to patch 3/5,
   only rejoin them when reading the MSR.

v1 -> v2:
 * set bit 32 of the VMX_EPT_VPID_CAP MSR
 * check against the supported types in the implementation of 
   the INVVPID instruction
 * the memory operand must always be read even if it isn't needed 
   (e.g., for type==global), similar to INVEPT
 * for single-context invalidation to check that VPID != 0, though in 
   practice that doesn't matter because we don't want to support
   single-context invalidation
 * don't set msr's ept related bits if !enable_ept 


VPID is used to tag address space and avoid a TLB flush. Currently L0 use
the same VPID to run L1 and all its guests. KVM flushes VPID when switching
between L1 and L2.

This patch advertises VPID to the L1 hypervisor, then address space of L1
and L2 can be separately treated and avoid TLB flush when swithing between
L1 and L2. For each nested vmentry, if vpid12 is changed, reuse shadow vpid
w/ an invvpid.

Performance:

run lmbench on L2 w/ 3.5 kernel.

Context switching - times in microseconds - smaller is better
-------------------------------------------------------------------------
Host                 OS  2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
                         ctxsw  ctxsw  ctxsw ctxsw  ctxsw   ctxsw   ctxsw
--------- ------------- ------ ------ ------ ------ ------ ------- -------
kernel    Linux 3.5.0-1 1.2200 1.3700 1.4500 4.7800 2.3300 5.60000 2.88000  nested VPID
kernel    Linux 3.5.0-1 1.2600 1.4300 1.5600   12.7   12.9 3.49000 7.46000  vanilla

Wanpeng Li (5):
  KVM: VMX: adjust interface to allocate/free_vpid
  KVM: VMX: introduce __vmx_flush_tlb to handle specific vpid
  KVM: nVMX: emulate the INVVPID instruction
  KVM: nVMX: nested VPID emulation
  KVM: nVMX: expose VPID capability to L1

 arch/x86/include/asm/vmx.h |   1 +
 arch/x86/kvm/vmx.c         | 151 ++++++++++++++++++++++++++++++++++++---------
 2 files changed, 123 insertions(+), 29 deletions(-)

-- 
2.1.0

Re: [PATCH v3 0/5] KVM: nVMX: nested VPID emulation

[Wanpeng Li]

Ping, :-)
On 10/14/15 12:12 AM, Wanpeng Li wrote:
> v2 -> v3:
>   * separate nested_vmx_vpid_caps and move checks to patch 3/5,
>     only rejoin them when reading the MSR.
>
> v1 -> v2:
>   * set bit 32 of the VMX_EPT_VPID_CAP MSR
>   * check against the supported types in the implementation of
>     the INVVPID instruction
>   * the memory operand must always be read even if it isn't needed
>     (e.g., for type==global), similar to INVEPT
>   * for single-context invalidation to check that VPID != 0, though in
>     practice that doesn't matter because we don't want to support
>     single-context invalidation
>   * don't set msr's ept related bits if !enable_ept
>
>
> VPID is used to tag address space and avoid a TLB flush. Currently L0 use
> the same VPID to run L1 and all its guests. KVM flushes VPID when switching
> between L1 and L2.
>
> This patch advertises VPID to the L1 hypervisor, then address space of L1
> and L2 can be separately treated and avoid TLB flush when swithing between
> L1 and L2. For each nested vmentry, if vpid12 is changed, reuse shadow vpid
> w/ an invvpid.
>
> Performance:
>
> run lmbench on L2 w/ 3.5 kernel.
>
> Context switching - times in microseconds - smaller is better
> -------------------------------------------------------------------------
> Host                 OS  2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
>                           ctxsw  ctxsw  ctxsw ctxsw  ctxsw   ctxsw   ctxsw
> --------- ------------- ------ ------ ------ ------ ------ ------- -------
> kernel    Linux 3.5.0-1 1.2200 1.3700 1.4500 4.7800 2.3300 5.60000 2.88000  nested VPID
> kernel    Linux 3.5.0-1 1.2600 1.4300 1.5600   12.7   12.9 3.49000 7.46000  vanilla
>
> Wanpeng Li (5):
>    KVM: VMX: adjust interface to allocate/free_vpid
>    KVM: VMX: introduce __vmx_flush_tlb to handle specific vpid
>    KVM: nVMX: emulate the INVVPID instruction
>    KVM: nVMX: nested VPID emulation
>    KVM: nVMX: expose VPID capability to L1
>
>   arch/x86/include/asm/vmx.h |   1 +
>   arch/x86/kvm/vmx.c         | 151 ++++++++++++++++++++++++++++++++++++---------
>   2 files changed, 123 insertions(+), 29 deletions(-)
>

Re: [PATCH v3 0/5] KVM: nVMX: nested VPID emulation

[Paolo Bonzini]

On 16/10/2015 15:13, Wanpeng Li wrote:
> Ping, :-)
> On 10/14/15 12:12 AM, Wanpeng Li wrote:
>> v2 -> v3:
>>   * separate nested_vmx_vpid_caps and move checks to patch 3/5,
>>     only rejoin them when reading the MSR.
>>
>> v1 -> v2:
>>   * set bit 32 of the VMX_EPT_VPID_CAP MSR
>>   * check against the supported types in the implementation of
>>     the INVVPID instruction
>>   * the memory operand must always be read even if it isn't needed
>>     (e.g., for type==global), similar to INVEPT
>>   * for single-context invalidation to check that VPID != 0, though in
>>     practice that doesn't matter because we don't want to support
>>     single-context invalidation
>>   * don't set msr's ept related bits if !enable_ept
>>
>>
>> VPID is used to tag address space and avoid a TLB flush. Currently L0 use
>> the same VPID to run L1 and all its guests. KVM flushes VPID when
>> switching
>> between L1 and L2.
>>
>> This patch advertises VPID to the L1 hypervisor, then address space of L1
>> and L2 can be separately treated and avoid TLB flush when swithing
>> between
>> L1 and L2. For each nested vmentry, if vpid12 is changed, reuse shadow
>> vpid
>> w/ an invvpid.
>>
>> Performance:
>>
>> run lmbench on L2 w/ 3.5 kernel.
>>
>> Context switching - times in microseconds - smaller is better
>> -------------------------------------------------------------------------
>> Host                 OS  2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K
>> 16p/64K
>>                           ctxsw  ctxsw  ctxsw ctxsw  ctxsw   ctxsw  
>> ctxsw
>> --------- ------------- ------ ------ ------ ------ ------ -------
>> -------
>> kernel    Linux 3.5.0-1 1.2200 1.3700 1.4500 4.7800 2.3300 5.60000
>> 2.88000  nested VPID
>> kernel    Linux 3.5.0-1 1.2600 1.4300 1.5600   12.7   12.9 3.49000
>> 7.46000  vanilla
>>
>> Wanpeng Li (5):
>>    KVM: VMX: adjust interface to allocate/free_vpid
>>    KVM: VMX: introduce __vmx_flush_tlb to handle specific vpid
>>    KVM: nVMX: emulate the INVVPID instruction
>>    KVM: nVMX: nested VPID emulation
>>    KVM: nVMX: expose VPID capability to L1
>>
>>   arch/x86/include/asm/vmx.h |   1 +
>>   arch/x86/kvm/vmx.c         | 151
>> ++++++++++++++++++++++++++++++++++++---------
>>   2 files changed, 123 insertions(+), 29 deletions(-)
>>
> 

Under test, will be committed to kvm/next soon.

Paolo