[PATCH] KVM: nVMX: Filter out all unsupported controls when eVMCS was activated

[PATCH] KVM: nVMX: Filter out all unsupported controls when eVMCS was activated

[Vitaly Kuznetsov]

Windows Server 2022 with Hyper-V role enabled failed to boot on KVM when
enlightened VMCS is advertised. Debugging revealed there are two exposed
secondary controls it is not happy with: SECONDARY_EXEC_ENABLE_VMFUNC and
SECONDARY_EXEC_SHADOW_VMCS. These controls are known to be unsupported,
as there are no corresponding fields in eVMCSv1 (see the comment above
EVMCS1_UNSUPPORTED_2NDEXEC definition).

Previously, commit 31de3d2500e4 ("x86/kvm/hyper-v: move VMX controls
sanitization out of nested_enable_evmcs()") introduced the required
filtering mechanism for VMX MSRs but for some reason put only known
to be problematic (and not full EVMCS1_UNSUPPORTED_* lists) controls
there.

Note, Windows Server 2022 seems to have gained some sanity check for VMX
MSRs: it doesn't even try to launch a guest when there's something it
doesn't like, nested_evmcs_check_controls() mechanism can't catch the
problem.

Let's be bold this time and instead of playing whack-a-mole just filter out
all unsupported controls from VMX MSRs.

Fixes: 31de3d2500e4 ("x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()")
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 arch/x86/kvm/vmx/evmcs.c | 12 +++++++++---
 arch/x86/kvm/vmx/vmx.c   |  9 +++++----
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/vmx/evmcs.c b/arch/x86/kvm/vmx/evmcs.c
index 0dab1b7b529f..ba6f99f584ac 100644
--- a/arch/x86/kvm/vmx/evmcs.c
+++ b/arch/x86/kvm/vmx/evmcs.c
@@ -353,14 +353,20 @@ void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
 	switch (msr_index) {
 	case MSR_IA32_VMX_EXIT_CTLS:
 	case MSR_IA32_VMX_TRUE_EXIT_CTLS:
-		ctl_high &= ~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL;
+		ctl_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
 		break;
 	case MSR_IA32_VMX_ENTRY_CTLS:
 	case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
-		ctl_high &= ~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
+		ctl_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
 		break;
 	case MSR_IA32_VMX_PROCBASED_CTLS2:
-		ctl_high &= ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
+		ctl_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
+		break;
+	case MSR_IA32_VMX_PINBASED_CTLS:
+		ctl_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
+		break;
+	case MSR_IA32_VMX_VMFUNC:
+		ctl_low &= ~EVMCS1_UNSUPPORTED_VMFUNC;
 		break;
 	}
 
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index fada1055f325..d7c5257eb5c0 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -1837,10 +1837,11 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 				    &msr_info->data))
 			return 1;
 		/*
-		 * Enlightened VMCS v1 doesn't have certain fields, but buggy
-		 * Hyper-V versions are still trying to use corresponding
-		 * features when they are exposed. Filter out the essential
-		 * minimum.
+		 * Enlightened VMCS v1 doesn't have certain VMCS fields but
+		 * instead of just ignoring the features, different Hyper-V
+		 * versions are either trying to use them and fail or do some
+		 * sanity checking and refuse to boot. Filter all unsupported
+		 * features out.
 		 */
 		if (!msr_info->host_initiated &&
 		    vmx->nested.enlightened_vmcs_enabled)
-- 
2.31.1

Re: [PATCH] KVM: nVMX: Filter out all unsupported controls when eVMCS was activated

[Sean Christopherson]

On Tue, Sep 07, 2021, Vitaly Kuznetsov wrote:
> Let's be bold this time and instead of playing whack-a-mole just filter out
> all unsupported controls from VMX MSRs.

Out of curiosity, why didn't we do this from the get-go?

Re: [PATCH] KVM: nVMX: Filter out all unsupported controls when eVMCS was activated

[Vitaly Kuznetsov]

Sean Christopherson <seanjc@google.com> writes:

> On Tue, Sep 07, 2021, Vitaly Kuznetsov wrote:
>> Let's be bold this time and instead of playing whack-a-mole just filter out
>> all unsupported controls from VMX MSRs.
>
> Out of curiosity, why didn't we do this from the get-go?

We actually did, the initial implementation (57b119da3594f) was
filtering out everything but then things changed in "only clear controls
which are known to cause issues" (31de3d2500e4). I forgot everything
already but was able to google this suggestion from Paolo:

https://www.lkml.org/lkml/2020/1/22/1108

so finally we've settled on a shortened list. Now as new Windows version
is out, we have new problems to solve)

-- 
Vitaly

Re: [PATCH] KVM: nVMX: Filter out all unsupported controls when eVMCS was activated

[Paolo Bonzini]

On 09/09/21 09:03, Vitaly Kuznetsov wrote:
> Sean Christopherson <seanjc@google.com> writes:
> 
>> On Tue, Sep 07, 2021, Vitaly Kuznetsov wrote:
>>> Let's be bold this time and instead of playing whack-a-mole just filter out
>>> all unsupported controls from VMX MSRs.
>>
>> Out of curiosity, why didn't we do this from the get-go?
> 
> We actually did, the initial implementation (57b119da3594f) was
> filtering out everything but then things changed in "only clear controls
> which are known to cause issues" (31de3d2500e4). I forgot everything
> already but was able to google this suggestion from Paolo:
> 
> https://www.lkml.org/lkml/2020/1/22/1108

The doubt was whether userspaces could be enabling eVMCS blindly, and 
thus would lose features for Linux guests.

Paolo

Re: [PATCH] KVM: nVMX: Filter out all unsupported controls when eVMCS was activated

[Paolo Bonzini]

On 07/09/21 18:35, Vitaly Kuznetsov wrote:
> Windows Server 2022 with Hyper-V role enabled failed to boot on KVM when
> enlightened VMCS is advertised. Debugging revealed there are two exposed
> secondary controls it is not happy with: SECONDARY_EXEC_ENABLE_VMFUNC and
> SECONDARY_EXEC_SHADOW_VMCS. These controls are known to be unsupported,
> as there are no corresponding fields in eVMCSv1 (see the comment above
> EVMCS1_UNSUPPORTED_2NDEXEC definition).
> 
> Previously, commit 31de3d2500e4 ("x86/kvm/hyper-v: move VMX controls
> sanitization out of nested_enable_evmcs()") introduced the required
> filtering mechanism for VMX MSRs but for some reason put only known
> to be problematic (and not full EVMCS1_UNSUPPORTED_* lists) controls
> there.
> 
> Note, Windows Server 2022 seems to have gained some sanity check for VMX
> MSRs: it doesn't even try to launch a guest when there's something it
> doesn't like, nested_evmcs_check_controls() mechanism can't catch the
> problem.
> 
> Let's be bold this time and instead of playing whack-a-mole just filter out
> all unsupported controls from VMX MSRs.
> 
> Fixes: 31de3d2500e4 ("x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()")
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>   arch/x86/kvm/vmx/evmcs.c | 12 +++++++++---
>   arch/x86/kvm/vmx/vmx.c   |  9 +++++----
>   2 files changed, 14 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/x86/kvm/vmx/evmcs.c b/arch/x86/kvm/vmx/evmcs.c
> index 0dab1b7b529f..ba6f99f584ac 100644
> --- a/arch/x86/kvm/vmx/evmcs.c
> +++ b/arch/x86/kvm/vmx/evmcs.c
> @@ -353,14 +353,20 @@ void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata)
>   	switch (msr_index) {
>   	case MSR_IA32_VMX_EXIT_CTLS:
>   	case MSR_IA32_VMX_TRUE_EXIT_CTLS:
> -		ctl_high &= ~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL;
> +		ctl_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
>   		break;
>   	case MSR_IA32_VMX_ENTRY_CTLS:
>   	case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
> -		ctl_high &= ~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
> +		ctl_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
>   		break;
>   	case MSR_IA32_VMX_PROCBASED_CTLS2:
> -		ctl_high &= ~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
> +		ctl_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC;
> +		break;
> +	case MSR_IA32_VMX_PINBASED_CTLS:
> +		ctl_high &= ~EVMCS1_UNSUPPORTED_PINCTRL;
> +		break;
> +	case MSR_IA32_VMX_VMFUNC:
> +		ctl_low &= ~EVMCS1_UNSUPPORTED_VMFUNC;
>   		break;
>   	}
>   
> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> index fada1055f325..d7c5257eb5c0 100644
> --- a/arch/x86/kvm/vmx/vmx.c
> +++ b/arch/x86/kvm/vmx/vmx.c
> @@ -1837,10 +1837,11 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
>   				    &msr_info->data))
>   			return 1;
>   		/*
> -		 * Enlightened VMCS v1 doesn't have certain fields, but buggy
> -		 * Hyper-V versions are still trying to use corresponding
> -		 * features when they are exposed. Filter out the essential
> -		 * minimum.
> +		 * Enlightened VMCS v1 doesn't have certain VMCS fields but
> +		 * instead of just ignoring the features, different Hyper-V
> +		 * versions are either trying to use them and fail or do some
> +		 * sanity checking and refuse to boot. Filter all unsupported
> +		 * features out.
>   		 */
>   		if (!msr_info->host_initiated &&
>   		    vmx->nested.enlightened_vmcs_enabled)
> 

Queued, thanks.

Paolo