[PATCH v2] KVM: x86: Allow guests to see MSR_IA32_TSX_CTRL even if tsx=off

[PATCH v2] KVM: x86: Allow guests to see MSR_IA32_TSX_CTRL even if tsx=off

[Paolo Bonzini]

Userspace that does not know about KVM_GET_MSR_FEATURE_INDEX_LIST
will generally use the default value for MSR_IA32_ARCH_CAPABILITIES.
When this happens and the host has tsx=on, it is possible to end up with
virtual machines that have HLE and RTM disabled, but TSX_CTRL available.

If the fleet is then switched to tsx=off, kvm_get_arch_capabilities()
will clear the ARCH_CAP_TSX_CTRL_MSR bit and it will not be possible to
use the tsx=off hosts as migration destinations, even though the guests
do not have TSX enabled.

To allow this migration, allow guests to write to their TSX_CTRL MSR,
while keeping the host MSR unchanged for the entire life of the guests.
This ensures that TSX remains disabled and also saves MSR reads and
writes, and it's okay to do because with tsx=off we know that guests will
not have the HLE and RTM features in their CPUID.  (If userspace sets
bogus CPUID data, we do not expect HLE and RTM to work in guests anyway).

Cc: stable@vger.kernel.org
Fixes: cbbaa2727aa3 ("KVM: x86: fix presentation of TSX feature in ARCH_CAPABILITIES")
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/vmx/vmx.c | 17 +++++++++++++----
 arch/x86/kvm/x86.c     |  2 +-
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index cc60b1fc3ee7..eb69fef57485 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -6860,11 +6860,20 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu)
 		switch (index) {
 		case MSR_IA32_TSX_CTRL:
 			/*
-			 * No need to pass TSX_CTRL_CPUID_CLEAR through, so
-			 * let's avoid changing CPUID bits under the host
-			 * kernel's feet.
+			 * TSX_CTRL_CPUID_CLEAR is handled in the CPUID
+			 * interception.  Keep the host value unchanged to avoid
+			 * changing CPUID bits under the host kernel's feet.
+			 *
+			 * hle=0, rtm=0, tsx_ctrl=1 can be found with some
+			 * combinations of new kernel and old userspace.  If
+			 * those guests run on a tsx=off host, do allow guests
+			 * to use TSX_CTRL, but do not change the value on the
+			 * host so that TSX remains always disabled.
 			 */
-			vmx->guest_uret_msrs[j].mask = ~(u64)TSX_CTRL_CPUID_CLEAR;
+			if (boot_cpu_has(X86_FEATURE_RTM))
+				vmx->guest_uret_msrs[j].mask = ~(u64)TSX_CTRL_CPUID_CLEAR;
+			else
+				vmx->guest_uret_msrs[j].mask = 0;
 			break;
 		default:
 			vmx->guest_uret_msrs[j].mask = -1ull;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 76bce832cade..15733013b266 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1401,7 +1401,7 @@ static u64 kvm_get_arch_capabilities(void)
 	 *	  This lets the guest use VERW to clear CPU buffers.
 	 */
 	if (!boot_cpu_has(X86_FEATURE_RTM))
-		data &= ~(ARCH_CAP_TAA_NO | ARCH_CAP_TSX_CTRL_MSR);
+		data &= ~ARCH_CAP_TAA_NO;
 	else if (!boot_cpu_has_bug(X86_BUG_TAA))
 		data |= ARCH_CAP_TAA_NO;
 
-- 
2.26.2

Re: [PATCH v2] KVM: x86: Allow guests to see MSR_IA32_TSX_CTRL even if tsx=off

[Sean Christopherson]

On Fri, Jan 29, 2021, Paolo Bonzini wrote:
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 76bce832cade..15733013b266 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -1401,7 +1401,7 @@ static u64 kvm_get_arch_capabilities(void)
>  	 *	  This lets the guest use VERW to clear CPU buffers.


This comment be updated to call out the new TSX_CTRL behavior.

	/*
	 * On TAA affected systems:
	 *      - nothing to do if TSX is disabled on the host.
	 *      - we emulate TSX_CTRL if present on the host.
	 *	  This lets the guest use VERW to clear CPU buffers.
	 */

>  	 */
>  	if (!boot_cpu_has(X86_FEATURE_RTM))
> -		data &= ~(ARCH_CAP_TAA_NO | ARCH_CAP_TSX_CTRL_MSR);
> +		data &= ~ARCH_CAP_TAA_NO;

Hmm, simply clearing TSX_CTRL will only preserve the host value.  Since
ARCH_CAPABILITIES is unconditionally emulated by KVM, wouldn't it make sense to
unconditionally expose TSX_CTRL as well, as opposed to exposing it only if it's
supported in the host?  I.e. allow migrating a TSX-disabled guest to a host
without TSX.  Or am I misunderstanding how TSX_CTRL is checked/used?

>  	else if (!boot_cpu_has_bug(X86_BUG_TAA))
>  		data |= ARCH_CAP_TAA_NO;
>  
> -- 
> 2.26.2
> 

Re: [PATCH v2] KVM: x86: Allow guests to see MSR_IA32_TSX_CTRL even if tsx=off

[Paolo Bonzini]

On 29/01/21 17:58, Sean Christopherson wrote:
> On Fri, Jan 29, 2021, Paolo Bonzini wrote:
>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>> index 76bce832cade..15733013b266 100644
>> --- a/arch/x86/kvm/x86.c
>> +++ b/arch/x86/kvm/x86.c
>> @@ -1401,7 +1401,7 @@ static u64 kvm_get_arch_capabilities(void)
>>   	 *	  This lets the guest use VERW to clear CPU buffers.
> 
> 
> This comment be updated to call out the new TSX_CTRL behavior.
> 
> 	/*
> 	 * On TAA affected systems:
> 	 *      - nothing to do if TSX is disabled on the host.
> 	 *      - we emulate TSX_CTRL if present on the host.
> 	 *	  This lets the guest use VERW to clear CPU buffers.
> 	 */

Ok.

>>   	 */
>>   	if (!boot_cpu_has(X86_FEATURE_RTM))
>> -		data &= ~(ARCH_CAP_TAA_NO | ARCH_CAP_TSX_CTRL_MSR);
>> +		data &= ~ARCH_CAP_TAA_NO;
> 
> Hmm, simply clearing TSX_CTRL will only preserve the host value.  Since
> ARCH_CAPABILITIES is unconditionally emulated by KVM, wouldn't it make sense to
> unconditionally expose TSX_CTRL as well, as opposed to exposing it only if it's
> supported in the host?  I.e. allow migrating a TSX-disabled guest to a host
> without TSX.  Or am I misunderstanding how TSX_CTRL is checked/used?

I'm a bit wary of having a combination (MDS_NO=0, TSX_CTRL=1) that does 
not exist on bare metal.  There are other cases where such combinations 
can happen, especially with the Spectre and SSBD mitigations (for 
example due to AMD CPUID bits for Intel processors), but at least those 
are just redundancies in the CPUID bits and it's more likely that the 
guest does something sensible with them.

Paolo

>>   	else if (!boot_cpu_has_bug(X86_BUG_TAA))
>>   		data |= ARCH_CAP_TAA_NO;
>>   
>> -- 
>> 2.26.2
>>
> 

Re: [PATCH v2] KVM: x86: Allow guests to see MSR_IA32_TSX_CTRL even if tsx=off

[Paolo Bonzini]

On 01/02/21 09:46, Paolo Bonzini wrote:
>>
>> This comment be updated to call out the new TSX_CTRL behavior.
>>
>>     /*
>>      * On TAA affected systems:
>>      *      - nothing to do if TSX is disabled on the host.
>>      *      - we emulate TSX_CTRL if present on the host.
>>      *      This lets the guest use VERW to clear CPU buffers.
>>      */
> 
> Ok.

Hmm, but the comment is even more accurate now than before, isn't it? 
It said nothing about hiding TSX_CTRL, so now it matches the code below.

Paolo

Re: [PATCH v2] KVM: x86: Allow guests to see MSR_IA32_TSX_CTRL even if tsx=off

[Sean Christopherson]

On Mon, Feb 01, 2021, Paolo Bonzini wrote:
> On 01/02/21 09:46, Paolo Bonzini wrote:
> > > 
> > > This comment be updated to call out the new TSX_CTRL behavior.
> > > 
> > >     /*
> > >      * On TAA affected systems:
> > >      *      - nothing to do if TSX is disabled on the host.
> > >      *      - we emulate TSX_CTRL if present on the host.
> > >      *      This lets the guest use VERW to clear CPU buffers.
> > >      */
> > 
> > Ok.
> 
> Hmm, but the comment is even more accurate now than before, isn't it? It
> said nothing about hiding TSX_CTRL, so now it matches the code below.

Ha, that is technically true.  But it says "nothing to do..." and then clears a
flag.  The other interpretation of "nothing to do... at runtime" is also wrong
as KVM emulates the MSR as a nop.

I guess I just find the whole comment more confusing than the code itself.

Re: [PATCH v2] KVM: x86: Allow guests to see MSR_IA32_TSX_CTRL even if tsx=off

[Sean Christopherson]

On Mon, Feb 01, 2021, Paolo Bonzini wrote:
> On 29/01/21 17:58, Sean Christopherson wrote:
> > On Fri, Jan 29, 2021, Paolo Bonzini wrote:
> > >   	 */
> > >   	if (!boot_cpu_has(X86_FEATURE_RTM))
> > > -		data &= ~(ARCH_CAP_TAA_NO | ARCH_CAP_TSX_CTRL_MSR);
> > > +		data &= ~ARCH_CAP_TAA_NO;
> > 
> > Hmm, simply clearing TSX_CTRL will only preserve the host value.  Since
> > ARCH_CAPABILITIES is unconditionally emulated by KVM, wouldn't it make sense to
> > unconditionally expose TSX_CTRL as well, as opposed to exposing it only if it's
> > supported in the host?  I.e. allow migrating a TSX-disabled guest to a host
> > without TSX.  Or am I misunderstanding how TSX_CTRL is checked/used?
> 
> I'm a bit wary of having a combination (MDS_NO=0, TSX_CTRL=1) that does not
> exist on bare metal.  There are other cases where such combinations can
> happen, especially with the Spectre and SSBD mitigations (for example due to
> AMD CPUID bits for Intel processors), but at least those are just
> redundancies in the CPUID bits and it's more likely that the guest does
> something sensible with them.

Gotcha.  The vulnerability combos and all the double and triple negatives make
my head spin.

Thanks!

Re: [PATCH v2] KVM: x86: Allow guests to see MSR_IA32_TSX_CTRL even if tsx=off

[Paolo Bonzini]

On 01/02/21 17:38, Sean Christopherson wrote:
>>>>      /*
>>>>       * On TAA affected systems:
>>>>       *      - nothing to do if TSX is disabled on the host.
>>>>       *      - we emulate TSX_CTRL if present on the host.
>>>>       *      This lets the guest use VERW to clear CPU buffers.
>>>>       */
> 
> it says "nothing to do..." and then clears a
> flag.  The other interpretation of "nothing to do... at runtime" is also wrong
> as KVM emulates the MSR as a nop.
> 
> I guess I just find the whole comment more confusing than the code itself.

What about:


         if (!boot_cpu_has(X86_FEATURE_RTM)) {
                 /*
                  * If RTM=0 because the kernel has disabled TSX, the 
host might
                  * have TAA_NO or TSX_CTRL.  Clear TAA_NO (the guest 
sees RTM=0
                  * and therefore knows that there cannot be TAA) but keep
                  * TSX_CTRL: some buggy userspaces leave it set on 
tsx=on hosts,
                  * and we want to allow migrating those guests to 
tsx=off hosts.
                  */
                 data &= ~ARCH_CAP_TAA_NO;
         } else if (!boot_cpu_has_bug(X86_BUG_TAA)) {
                 data |= ARCH_CAP_TAA_NO;
         } else {
                 /*
                  * Nothing to do here; we emulate TSX_CTRL if present 
on the
                  * host so the guest can choose between disabling TSX or
                  * using VERW to clear CPU buffers.
                  */
         }

Paolo

Re: [PATCH v2] KVM: x86: Allow guests to see MSR_IA32_TSX_CTRL even if tsx=off

[Sean Christopherson]

On Mon, Feb 01, 2021, Paolo Bonzini wrote:
> On 01/02/21 17:38, Sean Christopherson wrote:
> > > > >      /*
> > > > >       * On TAA affected systems:
> > > > >       *      - nothing to do if TSX is disabled on the host.
> > > > >       *      - we emulate TSX_CTRL if present on the host.
> > > > >       *      This lets the guest use VERW to clear CPU buffers.
> > > > >       */
> > 
> > it says "nothing to do..." and then clears a
> > flag.  The other interpretation of "nothing to do... at runtime" is also wrong
> > as KVM emulates the MSR as a nop.
> > 
> > I guess I just find the whole comment more confusing than the code itself.
> 
> What about:
> 
> 
>         if (!boot_cpu_has(X86_FEATURE_RTM)) {
>                 /*
>                  * If RTM=0 because the kernel has disabled TSX, the host might
>                  * have TAA_NO or TSX_CTRL.  Clear TAA_NO (the guest sees RTM=0
>                  * and therefore knows that there cannot be TAA) but keep
>                  * TSX_CTRL: some buggy userspaces leave it set on tsx=on hosts,
>                  * and we want to allow migrating those guests to tsx=off hosts.
>                  */
>                 data &= ~ARCH_CAP_TAA_NO;
>         } else if (!boot_cpu_has_bug(X86_BUG_TAA)) {
>                 data |= ARCH_CAP_TAA_NO;
>         } else {
>                 /*
>                  * Nothing to do here; we emulate TSX_CTRL if present on the
>                  * host so the guest can choose between disabling TSX or
>                  * using VERW to clear CPU buffers.
>                  */
>         }

Awesome!  Thanks much!