[PATCH 0/2] KVM: VMX: Sanitize VM-Entry/VM-Exit pairs during setup

[PATCH 0/2] KVM: VMX: Sanitize VM-Entry/VM-Exit pairs during setup

[Sean Christopherson]

Sanitize the VM-Entry/VM-Exit load+load and load+clear pairs when kvm_intel
is loaded instead of checking both controls at runtime.  Not sanitizing
means KVM ends up setting non-dynamic bits in the VMCS.

Add an opt-in knob to force kvm_intel to bail if an inconsistent pair is
detected instead of using a degraded and/or potentially broken setup.

Arguably patch 01 is stable material, but my mental coin flip came up
negative and I didn't Cc: stable.

And for patch 02, I'd definitely be favor of making it opt-out instead of
opt-in, but there's a non-zero chance that someone out there is running
KVM in a misconfigured VM...

Sean Christopherson (2):
  KVM: VMX: Sanitize VM-Entry/VM-Exit control pairs at kvm_intel load
    time
  KVM: VMX: Add knob to allow rejecting kvm_intel on inconsistent VMCS
    config

 arch/x86/kvm/vmx/capabilities.h | 13 +++-----
 arch/x86/kvm/vmx/vmx.c          | 55 +++++++++++++++++++++++++++++++--
 2 files changed, 56 insertions(+), 12 deletions(-)


base-commit: 90bde5bea810d766e7046bf5884f2ccf76dd78e9
-- 
2.36.1.124.g0e6072fb45-goog

[PATCH 1/2] KVM: VMX: Sanitize VM-Entry/VM-Exit control pairs at kvm_intel load time

[Sean Christopherson]

Sanitize the VM-Entry/VM-Exit control pairs (load+load or load+clear)
during setup instead of checking both controls in a pair at runtime.  If
only one control is supported, KVM will report the associated feature as
not available, but will leave the supported control bit set in the VMCS
config, which could lead to corruption of host state.  E.g. if only the
VM-Entry control is supported and the feature is not dynamically toggled,
KVM will set the control in all VMCSes and load zeros without restoring
host state.

Note, while this is technically a bug fix, practically speaking no sane
CPU or VMM would support only one control.  KVM's behavior of checking
both controls is mostly pedantry.

Cc: Chenyi Qiang <chenyi.qiang@intel.com>
Cc: Lei Wang <lei4.wang@intel.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/vmx/capabilities.h | 13 ++++--------
 arch/x86/kvm/vmx/vmx.c          | 35 +++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kvm/vmx/capabilities.h b/arch/x86/kvm/vmx/capabilities.h
index dc2cb8a16e76..464bf39e4835 100644
--- a/arch/x86/kvm/vmx/capabilities.h
+++ b/arch/x86/kvm/vmx/capabilities.h
@@ -97,20 +97,17 @@ static inline bool cpu_has_vmx_posted_intr(void)
 
 static inline bool cpu_has_load_ia32_efer(void)
 {
-	return (vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_EFER) &&
-	       (vmcs_config.vmexit_ctrl & VM_EXIT_LOAD_IA32_EFER);
+	return vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_EFER;
 }
 
 static inline bool cpu_has_load_perf_global_ctrl(void)
 {
-	return (vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL) &&
-	       (vmcs_config.vmexit_ctrl & VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL);
+	return vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
 }
 
 static inline bool cpu_has_vmx_mpx(void)
 {
-	return (vmcs_config.vmexit_ctrl & VM_EXIT_CLEAR_BNDCFGS) &&
-		(vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_BNDCFGS);
+	return vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_BNDCFGS;
 }
 
 static inline bool cpu_has_vmx_tpr_shadow(void)
@@ -377,7 +374,6 @@ static inline bool cpu_has_vmx_intel_pt(void)
 	rdmsrl(MSR_IA32_VMX_MISC, vmx_msr);
 	return (vmx_msr & MSR_IA32_VMX_MISC_INTEL_PT) &&
 		(vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_PT_USE_GPA) &&
-		(vmcs_config.vmexit_ctrl & VM_EXIT_CLEAR_IA32_RTIT_CTL) &&
 		(vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_RTIT_CTL);
 }
 
@@ -406,8 +402,7 @@ static inline bool vmx_pebs_supported(void)
 
 static inline bool cpu_has_vmx_arch_lbr(void)
 {
-	return (vmcs_config.vmexit_ctrl & VM_EXIT_CLEAR_IA32_LBR_CTL) &&
-		(vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_LBR_CTL);
+	return vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_LBR_CTL;
 }
 
 static inline u64 vmx_get_perf_capabilities(void)
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 6927f6e8ec31..2ea256de9aba 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -2462,6 +2462,9 @@ static __init u64 adjust_vmx_controls64(u64 ctl_opt, u32 msr)
 	return  ctl_opt & allowed;
 }
 
+#define VMCS_ENTRY_EXIT_PAIR(name, entry_action, exit_action) \
+	{ VM_ENTRY_##entry_action##_##name, VM_EXIT_##exit_action##_##name }
+
 static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
 				    struct vmx_capability *vmx_cap)
 {
@@ -2473,6 +2476,24 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
 	u64 _cpu_based_3rd_exec_control = 0;
 	u32 _vmexit_control = 0;
 	u32 _vmentry_control = 0;
+	int i;
+
+	/*
+	 * LOAD/SAVE_DEBUG_CONTROLS are absent because both are mandatory.
+	 * SAVE_IA32_PAT and SAVE_IA32_EFER are absent because KVM always
+	 * intercepts writes to PAT and EFER, i.e. never enables those controls.
+	 */
+	struct {
+		u32 entry_control;
+		u32 exit_control;
+	} vmcs_entry_exit_pairs[] = {
+		VMCS_ENTRY_EXIT_PAIR(IA32_PERF_GLOBAL_CTRL, LOAD, LOAD),
+		VMCS_ENTRY_EXIT_PAIR(IA32_PAT, LOAD, LOAD),
+		VMCS_ENTRY_EXIT_PAIR(IA32_EFER, LOAD, LOAD),
+		VMCS_ENTRY_EXIT_PAIR(BNDCFGS, LOAD, CLEAR),
+		VMCS_ENTRY_EXIT_PAIR(IA32_RTIT_CTL, LOAD, CLEAR),
+		VMCS_ENTRY_EXIT_PAIR(IA32_LBR_CTL, LOAD, CLEAR),
+	};
 
 	memset(vmcs_conf, 0, sizeof(*vmcs_conf));
 	min = CPU_BASED_HLT_EXITING |
@@ -2614,6 +2635,20 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
 				&_vmentry_control) < 0)
 		return -EIO;
 
+	for (i = 0; i < ARRAY_SIZE(vmcs_entry_exit_pairs); i++) {
+		u32 n_ctrl = vmcs_entry_exit_pairs[i].entry_control;
+		u32 x_ctrl = vmcs_entry_exit_pairs[i].exit_control;
+
+		if (!(_vmentry_control & n_ctrl) == !(_vmexit_control & x_ctrl))
+			continue;
+
+		pr_warn_once("Inconsistent VM-Entry/VM-Exit pair, entry = %x, exit = %x\n",
+			     _vmentry_control & n_ctrl, _vmexit_control & x_ctrl);
+
+		_vmentry_control &= ~n_ctrl;
+		_vmexit_control &= ~x_ctrl;
+	}
+
 	/*
 	 * Some cpus support VM_{ENTRY,EXIT}_IA32_PERF_GLOBAL_CTRL but they
 	 * can't be used due to an errata where VM Exit may incorrectly clear
-- 
2.36.1.124.g0e6072fb45-goog

[PATCH 2/2] KVM: VMX: Add knob to allow rejecting kvm_intel on inconsistent VMCS config

[Sean Christopherson]

Add an off-by-default module param, reject_inconsistent_vmcs_config, to
allow rejecting the load of kvm_intel if an inconsistent VMCS config is
detected.  Continuing on with an inconsistent, degraded config is
undesirable when the CPU is expected to support a given set of features,
e.g. can result in a misconfigured VM if userspace doesn't cross-check
KVM_GET_SUPPORTED_CPUID, and/or can result in poor performance due to
lack of fast MSR switching.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/vmx/vmx.c | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 2ea256de9aba..11413a8cc57f 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -119,6 +119,9 @@ module_param(nested, bool, S_IRUGO);
 bool __read_mostly enable_pml = 1;
 module_param_named(pml, enable_pml, bool, S_IRUGO);
 
+static bool __read_mostly reject_inconsistent_vmcs_config;
+module_param(reject_inconsistent_vmcs_config, bool, 0444);
+
 static bool __read_mostly dump_invalid_vmcs = 0;
 module_param(dump_invalid_vmcs, bool, 0644);
 
@@ -2577,15 +2580,23 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
 					     CPU_BASED_CR3_STORE_EXITING |
 					     CPU_BASED_INVLPG_EXITING);
 	} else if (vmx_cap->ept) {
-		vmx_cap->ept = 0;
 		pr_warn_once("EPT CAP should not exist if not support "
 				"1-setting enable EPT VM-execution control\n");
+
+		if (reject_inconsistent_vmcs_config)
+			return -EIO;
+
+		vmx_cap->ept = 0;
 	}
 	if (!(_cpu_based_2nd_exec_control & SECONDARY_EXEC_ENABLE_VPID) &&
-		vmx_cap->vpid) {
-		vmx_cap->vpid = 0;
+	    vmx_cap->vpid) {
 		pr_warn_once("VPID CAP should not exist if not support "
 				"1-setting enable VPID VM-execution control\n");
+
+		if (reject_inconsistent_vmcs_config)
+			return -EIO;
+
+		vmx_cap->vpid = 0;
 	}
 
 	if (_cpu_based_exec_control & CPU_BASED_ACTIVATE_TERTIARY_CONTROLS) {
@@ -2645,6 +2656,9 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
 		pr_warn_once("Inconsistent VM-Entry/VM-Exit pair, entry = %x, exit = %x\n",
 			     _vmentry_control & n_ctrl, _vmexit_control & x_ctrl);
 
+		if (reject_inconsistent_vmcs_config)
+			return -EIO;
+
 		_vmentry_control &= ~n_ctrl;
 		_vmexit_control &= ~x_ctrl;
 	}
-- 
2.36.1.124.g0e6072fb45-goog

Re: [PATCH 2/2] KVM: VMX: Add knob to allow rejecting kvm_intel on inconsistent VMCS config

[Jim Mattson]

On Wed, May 25, 2022 at 2:04 PM Sean Christopherson <seanjc@google.com> wrote:
>
> Add an off-by-default module param, reject_inconsistent_vmcs_config, to
> allow rejecting the load of kvm_intel if an inconsistent VMCS config is
> detected.  Continuing on with an inconsistent, degraded config is
> undesirable when the CPU is expected to support a given set of features,
> e.g. can result in a misconfigured VM if userspace doesn't cross-check
> KVM_GET_SUPPORTED_CPUID, and/or can result in poor performance due to
> lack of fast MSR switching.
>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
There are several inconsistent VMCS configs that are not rejected here
(e.g. "enable XSAVES/XRSTORS" on a CPU that doesn't support XSAVES).
Do you plan to include more checks in the future, or should this be,
"reject_some_inconsistent_vmcs_configs"? :-)

Re: [PATCH 1/2] KVM: VMX: Sanitize VM-Entry/VM-Exit control pairs at kvm_intel load time

[Yuan Yao]

On Wed, May 25, 2022 at 09:04:46PM +0000, Sean Christopherson wrote:
> Sanitize the VM-Entry/VM-Exit control pairs (load+load or load+clear)
> during setup instead of checking both controls in a pair at runtime.  If
> only one control is supported, KVM will report the associated feature as
> not available, but will leave the supported control bit set in the VMCS
> config, which could lead to corruption of host state.  E.g. if only the
> VM-Entry control is supported and the feature is not dynamically toggled,
> KVM will set the control in all VMCSes and load zeros without restoring
> host state.
>
> Note, while this is technically a bug fix, practically speaking no sane
> CPU or VMM would support only one control.  KVM's behavior of checking
> both controls is mostly pedantry.
>
> Cc: Chenyi Qiang <chenyi.qiang@intel.com>
> Cc: Lei Wang <lei4.wang@intel.com>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  arch/x86/kvm/vmx/capabilities.h | 13 ++++--------
>  arch/x86/kvm/vmx/vmx.c          | 35 +++++++++++++++++++++++++++++++++
>  2 files changed, 39 insertions(+), 9 deletions(-)
>
> diff --git a/arch/x86/kvm/vmx/capabilities.h b/arch/x86/kvm/vmx/capabilities.h
> index dc2cb8a16e76..464bf39e4835 100644
> --- a/arch/x86/kvm/vmx/capabilities.h
> +++ b/arch/x86/kvm/vmx/capabilities.h
> @@ -97,20 +97,17 @@ static inline bool cpu_has_vmx_posted_intr(void)
>
>  static inline bool cpu_has_load_ia32_efer(void)
>  {
> -	return (vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_EFER) &&
> -	       (vmcs_config.vmexit_ctrl & VM_EXIT_LOAD_IA32_EFER);
> +	return vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_EFER;
>  }
>
>  static inline bool cpu_has_load_perf_global_ctrl(void)
>  {
> -	return (vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL) &&
> -	       (vmcs_config.vmexit_ctrl & VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL);
> +	return vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL;
>  }
>
>  static inline bool cpu_has_vmx_mpx(void)
>  {
> -	return (vmcs_config.vmexit_ctrl & VM_EXIT_CLEAR_BNDCFGS) &&
> -		(vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_BNDCFGS);
> +	return vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_BNDCFGS;
>  }
>
>  static inline bool cpu_has_vmx_tpr_shadow(void)
> @@ -377,7 +374,6 @@ static inline bool cpu_has_vmx_intel_pt(void)
>  	rdmsrl(MSR_IA32_VMX_MISC, vmx_msr);
>  	return (vmx_msr & MSR_IA32_VMX_MISC_INTEL_PT) &&
>  		(vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_PT_USE_GPA) &&
> -		(vmcs_config.vmexit_ctrl & VM_EXIT_CLEAR_IA32_RTIT_CTL) &&
>  		(vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_RTIT_CTL);
>  }
>
> @@ -406,8 +402,7 @@ static inline bool vmx_pebs_supported(void)
>
>  static inline bool cpu_has_vmx_arch_lbr(void)
>  {
> -	return (vmcs_config.vmexit_ctrl & VM_EXIT_CLEAR_IA32_LBR_CTL) &&
> -		(vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_LBR_CTL);
> +	return vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_LBR_CTL;
>  }
>
>  static inline u64 vmx_get_perf_capabilities(void)
> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> index 6927f6e8ec31..2ea256de9aba 100644
> --- a/arch/x86/kvm/vmx/vmx.c
> +++ b/arch/x86/kvm/vmx/vmx.c
> @@ -2462,6 +2462,9 @@ static __init u64 adjust_vmx_controls64(u64 ctl_opt, u32 msr)
>  	return  ctl_opt & allowed;
>  }
>
> +#define VMCS_ENTRY_EXIT_PAIR(name, entry_action, exit_action) \
> +	{ VM_ENTRY_##entry_action##_##name, VM_EXIT_##exit_action##_##name }
> +
>  static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
>  				    struct vmx_capability *vmx_cap)
>  {
> @@ -2473,6 +2476,24 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
>  	u64 _cpu_based_3rd_exec_control = 0;
>  	u32 _vmexit_control = 0;
>  	u32 _vmentry_control = 0;
> +	int i;
> +
> +	/*
> +	 * LOAD/SAVE_DEBUG_CONTROLS are absent because both are mandatory.
> +	 * SAVE_IA32_PAT and SAVE_IA32_EFER are absent because KVM always
> +	 * intercepts writes to PAT and EFER, i.e. never enables those controls.
> +	 */
> +	struct {
> +		u32 entry_control;
> +		u32 exit_control;
> +	} vmcs_entry_exit_pairs[] = {
> +		VMCS_ENTRY_EXIT_PAIR(IA32_PERF_GLOBAL_CTRL, LOAD, LOAD),
> +		VMCS_ENTRY_EXIT_PAIR(IA32_PAT, LOAD, LOAD),
> +		VMCS_ENTRY_EXIT_PAIR(IA32_EFER, LOAD, LOAD),
> +		VMCS_ENTRY_EXIT_PAIR(BNDCFGS, LOAD, CLEAR),
> +		VMCS_ENTRY_EXIT_PAIR(IA32_RTIT_CTL, LOAD, CLEAR),
> +		VMCS_ENTRY_EXIT_PAIR(IA32_LBR_CTL, LOAD, CLEAR),
> +	};
>
>  	memset(vmcs_conf, 0, sizeof(*vmcs_conf));
>  	min = CPU_BASED_HLT_EXITING |
> @@ -2614,6 +2635,20 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
>  				&_vmentry_control) < 0)
>  		return -EIO;
>
> +	for (i = 0; i < ARRAY_SIZE(vmcs_entry_exit_pairs); i++) {
> +		u32 n_ctrl = vmcs_entry_exit_pairs[i].entry_control;
> +		u32 x_ctrl = vmcs_entry_exit_pairs[i].exit_control;
> +
> +		if (!(_vmentry_control & n_ctrl) == !(_vmexit_control & x_ctrl))
> +			continue;
> +
> +		pr_warn_once("Inconsistent VM-Entry/VM-Exit pair, entry = %x, exit = %x\n",
> +			     _vmentry_control & n_ctrl, _vmexit_control & x_ctrl);

How about "n_ctrl, x_ctrl);" ? In 0/1 or 1/0 case this
outputs all information of real inconsistent bits but not 0.

> +
> +		_vmentry_control &= ~n_ctrl;
> +		_vmexit_control &= ~x_ctrl;
> +	}
> +
>  	/*
>  	 * Some cpus support VM_{ENTRY,EXIT}_IA32_PERF_GLOBAL_CTRL but they
>  	 * can't be used due to an errata where VM Exit may incorrectly clear
> --
> 2.36.1.124.g0e6072fb45-goog
>

Re: [PATCH 1/2] KVM: VMX: Sanitize VM-Entry/VM-Exit control pairs at kvm_intel load time

[Sean Christopherson]

On Thu, May 26, 2022, Yuan Yao wrote:
> On Wed, May 25, 2022 at 09:04:46PM +0000, Sean Christopherson wrote:
> > @@ -2614,6 +2635,20 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
> >  				&_vmentry_control) < 0)
> >  		return -EIO;
> >
> > +	for (i = 0; i < ARRAY_SIZE(vmcs_entry_exit_pairs); i++) {
> > +		u32 n_ctrl = vmcs_entry_exit_pairs[i].entry_control;
> > +		u32 x_ctrl = vmcs_entry_exit_pairs[i].exit_control;
> > +
> > +		if (!(_vmentry_control & n_ctrl) == !(_vmexit_control & x_ctrl))
> > +			continue;
> > +
> > +		pr_warn_once("Inconsistent VM-Entry/VM-Exit pair, entry = %x, exit = %x\n",
> > +			     _vmentry_control & n_ctrl, _vmexit_control & x_ctrl);
> 
> How about "n_ctrl, x_ctrl);" ? In 0/1 or 1/0 case this
> outputs all information of real inconsistent bits but not 0.

I thought about adding the stringified control name to the output (yay macros),
but opted for the simplest approach because this should be a very, very rare
event.  All the necessary info is there, it just takes a bit of leg work to get
from a single control bit to the related control name and finally to its pair.

I'm not totally against printing more info, but if we're going to bother doing so,
my vote is to print names instead of numbers.

Re: [PATCH 2/2] KVM: VMX: Add knob to allow rejecting kvm_intel on inconsistent VMCS config

[Sean Christopherson]

On Wed, May 25, 2022, Jim Mattson wrote:
> On Wed, May 25, 2022 at 2:04 PM Sean Christopherson <seanjc@google.com> wrote:
> >
> > Add an off-by-default module param, reject_inconsistent_vmcs_config, to
> > allow rejecting the load of kvm_intel if an inconsistent VMCS config is
> > detected.  Continuing on with an inconsistent, degraded config is
> > undesirable when the CPU is expected to support a given set of features,
> > e.g. can result in a misconfigured VM if userspace doesn't cross-check
> > KVM_GET_SUPPORTED_CPUID, and/or can result in poor performance due to
> > lack of fast MSR switching.
> >
> > Signed-off-by: Sean Christopherson <seanjc@google.com>
> > ---
> There are several inconsistent VMCS configs that are not rejected here
> (e.g. "enable XSAVES/XRSTORS" on a CPU that doesn't support XSAVES).
> Do you plan to include more checks in the future, or should this be,
> "reject_some_inconsistent_vmcs_configs"? :-)

I have no plan, it was purely a reaction to continuing on with a known bad entry/exit
pair handling being awful.  I hesitated to even apply it to the EPT/VPID stuff, but
again it seemed silly to detect an inconsistency and do nothing about it.

I'm not opposed to adding more checks, though there is definitely a point of
diminishing returns.  I'm just picking the really low hanging fruit :-)

Re: [PATCH 2/2] KVM: VMX: Add knob to allow rejecting kvm_intel on inconsistent VMCS config

[Jim Mattson]

On Wed, May 25, 2022 at 5:45 PM Sean Christopherson <seanjc@google.com> wrote:
>
> On Wed, May 25, 2022, Jim Mattson wrote:
> > On Wed, May 25, 2022 at 2:04 PM Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > Add an off-by-default module param, reject_inconsistent_vmcs_config, to
> > > allow rejecting the load of kvm_intel if an inconsistent VMCS config is
> > > detected.  Continuing on with an inconsistent, degraded config is
> > > undesirable when the CPU is expected to support a given set of features,
> > > e.g. can result in a misconfigured VM if userspace doesn't cross-check
> > > KVM_GET_SUPPORTED_CPUID, and/or can result in poor performance due to
> > > lack of fast MSR switching.
> > >
> > > Signed-off-by: Sean Christopherson <seanjc@google.com>
> > > ---
> > There are several inconsistent VMCS configs that are not rejected here
> > (e.g. "enable XSAVES/XRSTORS" on a CPU that doesn't support XSAVES).
> > Do you plan to include more checks in the future, or should this be,
> > "reject_some_inconsistent_vmcs_configs"? :-)
>
> I have no plan, it was purely a reaction to continuing on with a known bad entry/exit
> pair handling being awful.  I hesitated to even apply it to the EPT/VPID stuff, but
> again it seemed silly to detect an inconsistency and do nothing about it.
>
> I'm not opposed to adding more checks, though there is definitely a point of
> diminishing returns.  I'm just picking the really low hanging fruit :-)

The usual KVM approach to a misconfigured guest is to let userspace
shoot itself in the foot, as long as it doesn't put the host at risk.
This change seems to run counter to that.

Re: [PATCH 1/2] KVM: VMX: Sanitize VM-Entry/VM-Exit control pairs at kvm_intel load time

[Yuan Yao]

On Thu, May 26, 2022 at 12:42:31AM +0000, Sean Christopherson wrote:
> On Thu, May 26, 2022, Yuan Yao wrote:
> > On Wed, May 25, 2022 at 09:04:46PM +0000, Sean Christopherson wrote:
> > > @@ -2614,6 +2635,20 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
> > >  				&_vmentry_control) < 0)
> > >  		return -EIO;
> > >
> > > +	for (i = 0; i < ARRAY_SIZE(vmcs_entry_exit_pairs); i++) {
> > > +		u32 n_ctrl = vmcs_entry_exit_pairs[i].entry_control;
> > > +		u32 x_ctrl = vmcs_entry_exit_pairs[i].exit_control;
> > > +
> > > +		if (!(_vmentry_control & n_ctrl) == !(_vmexit_control & x_ctrl))
> > > +			continue;
> > > +
> > > +		pr_warn_once("Inconsistent VM-Entry/VM-Exit pair, entry = %x, exit = %x\n",
> > > +			     _vmentry_control & n_ctrl, _vmexit_control & x_ctrl);
> >
> > How about "n_ctrl, x_ctrl);" ? In 0/1 or 1/0 case this
> > outputs all information of real inconsistent bits but not 0.
>
> I thought about adding the stringified control name to the output (yay macros),
> but opted for the simplest approach because this should be a very, very rare
> event.  All the necessary info is there, it just takes a bit of leg work to get
> from a single control bit to the related control name and finally to its pair.
>
> I'm not totally against printing more info, but if we're going to bother doing so,
> my vote is to print names instead of numbers.

Agree for simplest approach because this should be rare event, thanks.

Re: [PATCH 2/2] KVM: VMX: Add knob to allow rejecting kvm_intel on inconsistent VMCS config

[Paolo Bonzini]

On 5/25/22 23:04, Sean Christopherson wrote:
> Add an off-by-default module param, reject_inconsistent_vmcs_config, to
> allow rejecting the load of kvm_intel if an inconsistent VMCS config is
> detected.  Continuing on with an inconsistent, degraded config is
> undesirable when the CPU is expected to support a given set of features,
> e.g. can result in a misconfigured VM if userspace doesn't cross-check
> KVM_GET_SUPPORTED_CPUID, and/or can result in poor performance due to
> lack of fast MSR switching.
> 
> Signed-off-by: Sean Christopherson<seanjc@google.com>
> ---
>   arch/x86/kvm/vmx/vmx.c | 20 +++++++++++++++++---
>   1 file changed, 17 insertions(+), 3 deletions(-)

Yeah let's do this by default.

Paolo

Re: [PATCH 1/2] KVM: VMX: Sanitize VM-Entry/VM-Exit control pairs at kvm_intel load time

[Paolo Bonzini]

On 5/25/22 23:04, Sean Christopherson wrote:
>   
> +#define VMCS_ENTRY_EXIT_PAIR(name, entry_action, exit_action) \
> +	{ VM_ENTRY_##entry_action##_##name, VM_EXIT_##exit_action##_##name }
> +
>   static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
>   				    struct vmx_capability *vmx_cap)
>   {
> @@ -2473,6 +2476,24 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
>   	u64 _cpu_based_3rd_exec_control = 0;
>   	u32 _vmexit_control = 0;
>   	u32 _vmentry_control = 0;
> +	int i;
> +
> +	/*
> +	 * LOAD/SAVE_DEBUG_CONTROLS are absent because both are mandatory.
> +	 * SAVE_IA32_PAT and SAVE_IA32_EFER are absent because KVM always
> +	 * intercepts writes to PAT and EFER, i.e. never enables those controls.
> +	 */
> +	struct {
> +		u32 entry_control;
> +		u32 exit_control;
> +	} vmcs_entry_exit_pairs[] = {
> +		VMCS_ENTRY_EXIT_PAIR(IA32_PERF_GLOBAL_CTRL, LOAD, LOAD),
> +		VMCS_ENTRY_EXIT_PAIR(IA32_PAT, LOAD, LOAD),
> +		VMCS_ENTRY_EXIT_PAIR(IA32_EFER, LOAD, LOAD),
> +		VMCS_ENTRY_EXIT_PAIR(BNDCFGS, LOAD, CLEAR),
> +		VMCS_ENTRY_EXIT_PAIR(IA32_RTIT_CTL, LOAD, CLEAR),
> +		VMCS_ENTRY_EXIT_PAIR(IA32_LBR_CTL, LOAD, CLEAR),

No macros please, it's just as clear to expand them especially since the 
#define is far from the struct definition.

Paolo

Re: [PATCH 1/2] KVM: VMX: Sanitize VM-Entry/VM-Exit control pairs at kvm_intel load time

[Sean Christopherson]

On Thu, May 26, 2022, Paolo Bonzini wrote:
> On 5/25/22 23:04, Sean Christopherson wrote:
> > +#define VMCS_ENTRY_EXIT_PAIR(name, entry_action, exit_action) \
> > +	{ VM_ENTRY_##entry_action##_##name, VM_EXIT_##exit_action##_##name }
> > +
> >   static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
> >   				    struct vmx_capability *vmx_cap)
> >   {
> > @@ -2473,6 +2476,24 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
> >   	u64 _cpu_based_3rd_exec_control = 0;
> >   	u32 _vmexit_control = 0;
> >   	u32 _vmentry_control = 0;
> > +	int i;
> > +
> > +	/*
> > +	 * LOAD/SAVE_DEBUG_CONTROLS are absent because both are mandatory.
> > +	 * SAVE_IA32_PAT and SAVE_IA32_EFER are absent because KVM always
> > +	 * intercepts writes to PAT and EFER, i.e. never enables those controls.
> > +	 */
> > +	struct {
> > +		u32 entry_control;
> > +		u32 exit_control;
> > +	} vmcs_entry_exit_pairs[] = {
> > +		VMCS_ENTRY_EXIT_PAIR(IA32_PERF_GLOBAL_CTRL, LOAD, LOAD),
> > +		VMCS_ENTRY_EXIT_PAIR(IA32_PAT, LOAD, LOAD),
> > +		VMCS_ENTRY_EXIT_PAIR(IA32_EFER, LOAD, LOAD),
> > +		VMCS_ENTRY_EXIT_PAIR(BNDCFGS, LOAD, CLEAR),
> > +		VMCS_ENTRY_EXIT_PAIR(IA32_RTIT_CTL, LOAD, CLEAR),
> > +		VMCS_ENTRY_EXIT_PAIR(IA32_LBR_CTL, LOAD, CLEAR),
> 
> No macros please, it's just as clear to expand them especially since the
> #define is far from the struct definition.

It's not for clarity, it's to prevent plopping an EXIT control into the ENTRY
slot and vice versa.  I have a hell of a time trying to visually differentiate
those, and a buggy pair isn't guaranteed to be detected at runtime, e.g. if both
are swapped, all bets are off, and if one is duplicated, odds the warn may or may
not show up unless hardware actually supports at least one of the controls, if not
both.

With this, swapping LOAD and LOAD is obviously a nop, and swapping LOAD and CLEAR
will generate a compiler error.

FWIW, I did originally have the array declared as static __initdata immediately
after the #define.  I moved away from that because __initdata doesn't play nice
with const, but then of course I forgot to add back the "const".  /facepalm

Re: [PATCH 1/2] KVM: VMX: Sanitize VM-Entry/VM-Exit control pairs at kvm_intel load time

[Paolo Bonzini]

On 5/26/22 23:35, Sean Christopherson wrote:
> It's not for clarity, it's to prevent plopping an EXIT control into the ENTRY
> slot and vice versa.  I have a hell of a time trying to visually differentiate
> those, and a buggy pair isn't guaranteed to be detected at runtime, e.g. if both
> are swapped, all bets are off, and if one is duplicated, odds the warn may or may
> not show up unless hardware actually supports at least one of the controls, if not
> both.

Make the lines longer than 80 characters and align each element so that 
you have a line of VM_ENTRY_ and VM_EXIT_.  (It would not work if they 
were the same length, but they aren't).

Paolo