[RFC PATCH 0/9] Intel SPEC CTRL virtualization support

[RFC PATCH 0/9] Intel SPEC CTRL virtualization support

[Zhang Chen]

This series introduces "virtual IA32_SPEC_CTRL" support. Here are introduction
and use cases of this new feature.

### Virtual IA32_SPEC_CTRL

Virtual IA32_SPEC_CTRL [1] is a new VMX feature on Intel CPUs.
This feature allows VMM to fix some bits of IA32_SPEC_CTRL MSR even when
the MSR is pass-thru'd to a guest.


### Use cases of virtual IA32_SPEC_CTRL

Software mitigations like Retpoline and software BHB-clearing sequence
depend on CPU microarchitectures. And guest cannot know exactly
the underlying microarchitecture. When a guest is migrated between
processors of different microarchitectures, software mitigations which
work perfectly on previous microachitecture may be not effective on the
new one. To fix the problem, some hardware mitigations should be used in
conjunction with software mitigations. Using virtual IA32_SPEC_CTRL,
VMM can enforce hardware mitigations transparently to guests and avoid
those hardware mitigations being accidentally disabled when guest
changes IA32_SPEC_CTRL MSR.


### Intention of this series

This series adds the capability of enforcing hardware mitigations for
guests transparently and efficiently (i.e., without intecepting
IA32_SPEC_CTRL MSR accesses) to kvm. The capability can be used to solve
the VM migration issue in a pool consisting of processors of different
microarchitectures.

Specifically, below are two target scenarios of this series:

Scenario 1: If retpoline is used by a VM to mitigate IMBTI in CPL0, VMM
            can set RRSBA_DIS_S on parts enumerates RRSBA. Note that
            the VM is presented with a microarchitecture doesn't enumerate
            RRSBA.

Scenario 2: If a VM uses software BHB-clearing sequence on transitions
            into CPL0 to mitigate BHI, VMM can use "virtual
            IA32_SPEC_CTRL" to set BHI_DIS_S on newer
            hardware which doesn't enumerate BHI_NO.

Intel defines some virtual MSRs for guests to notify VMM of software
mitigations in use. Such information allows VMM to enable hardware
mitigations only when necessary (i.e., VMs care about the vulnerability
and are using software mitigiation) to minimize the performance impact
to other VMs. The support of the virtual MSRs also added in this series.

### Organization of this series

This series is developed based on v6.1-rc7 (commit ef4d3ea40565a781c25847e9cb96c1bd9f462bc6)

1. Patch 1 introduce Intel SPEC_CTRL BHI related definition.
2. Patch 2/3 advertises {RRSBA/BHI}_CTRL for retpoline/BHB-clearing
sequence mitigations for KVM.
3. Patch 4 introduce Intel virtual IA32_SPEC_CTRL
4. Patch 5-7 virtualize a set of virtual MSRs for guests to communicate
   software mitigations in use.
5. Patch 8-9 enable "virtual IA32_SPEC_CTRL" VMX feature with virtual
   MSRs support in KVM.

### Documentation

Refer to below link for more information:
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html



Pawan Gupta (1):
  x86/bugs: Use Virtual MSRs to request hardware mitigations

Zhang Chen (8):
  x86/speculation: Introduce Intel SPEC_CTRL BHI related definition
  KVM: x86: Add a kvm-only leaf for RRSBA_CTRL
  KVM: x86: Add a kvm-only leaf for BHI_CTRL
  x86/kvm/vmx: Virtualize Intel IA32_SPEC_CTRL
  kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest
    MSR_IA32_ARCH_CAPABILITIES
  kvm/x86: Add MSR_VIRTUAL_MITIGATION_ENUM/CTRL emulation
  x86/kvm/vmx: Initialize SPEC_CTRL MASK for RRSBA
  x86/kvm/vmx: Initialize SPEC_CTRL MASK for BHI

 arch/x86/include/asm/msr-index.h       |  29 +++++++
 arch/x86/include/asm/vmx.h             |   5 ++
 arch/x86/include/asm/vmxfeatures.h     |   2 +
 arch/x86/kernel/cpu/bugs.c             |  24 ++++++
 arch/x86/kvm/cpuid.c                   |   4 +
 arch/x86/kvm/reverse_cpuid.h           |   9 ++
 arch/x86/kvm/vmx/capabilities.h        |   5 ++
 arch/x86/kvm/vmx/vmx.c                 | 109 ++++++++++++++++++++++++-
 arch/x86/kvm/vmx/vmx.h                 |  27 +++++-
 arch/x86/kvm/x86.c                     |  23 +++++-
 tools/arch/x86/include/asm/msr-index.h |   6 ++
 11 files changed, 238 insertions(+), 5 deletions(-)

-- 
2.25.1

[RFC PATCH 1/9] x86/speculation: Introduce Intel SPEC_CTRL BHI related definition

[Zhang Chen]

Define BHI_NO bit and new control of BHI hardware mitigation in
IA32_SPEC_CTRL. These definitions are used by following KVM patches
to determine whether to enforce BHI hardware mitigiations for
guests transparently.

BHI_NO means the processor isn't vulnernable to BHI attacks. BHI_DIS_S
is a new indirect predictor control. Once enabled, BHI_DIS_S prevents
predicted targets of indirect branches executed in CPL0/1/2 from being
selected based on branch history from branches executed in CPL3.
While set in the VMX root, it also prevents predicted targets executed
in CPL0 from being selected based on branch history from branches
executed in a VMX non-root.

Branch History Injection (BHI) describes a specific form of intra-mode
BTI, where an attacker may manipulate branch history before transitioning
from user to supervisor mode (or from VMX non-root/guest to root mode)
in an effort to cause an indirect branch predictor to select a specific
predictor entry for an indirect branch, and a disclosure gadget at the
predicted target will transiently execute. This may be possible since
the relevant branch history may contain branches taken in previous
security contexts, and in particular, in other predictor modes.

Refer to below link for more information:
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 arch/x86/include/asm/msr-index.h       | 6 ++++++
 tools/arch/x86/include/asm/msr-index.h | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 4a2af82553e4..1143ac9400c3 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -53,6 +53,8 @@
 #define SPEC_CTRL_SSBD			BIT(SPEC_CTRL_SSBD_SHIFT)	/* Speculative Store Bypass Disable */
 #define SPEC_CTRL_RRSBA_DIS_S_SHIFT	6	   /* Disable RRSBA behavior */
 #define SPEC_CTRL_RRSBA_DIS_S		BIT(SPEC_CTRL_RRSBA_DIS_S_SHIFT)
+#define SPEC_CTRL_BHI_DIS_S_SHIFT	10	   /* Enable BHI_DIS_S behavior */
+#define SPEC_CTRL_BHI_DIS_S		BIT(SPEC_CTRL_BHI_DIS_S_SHIFT)
 
 #define MSR_IA32_PRED_CMD		0x00000049 /* Prediction Command */
 #define PRED_CMD_IBPB			BIT(0)	   /* Indirect Branch Prediction Barrier */
@@ -150,6 +152,10 @@
 						 * are restricted to targets in
 						 * kernel.
 						 */
+#define ARCH_CAP_BHI_NO			BIT(20)	/*
+						 * Not susceptible to Branch History
+						 * Injection.
+						 */
 #define ARCH_CAP_PBRSB_NO		BIT(24)	/*
 						 * Not susceptible to Post-Barrier
 						 * Return Stack Buffer Predictions.
diff --git a/tools/arch/x86/include/asm/msr-index.h b/tools/arch/x86/include/asm/msr-index.h
index f17ade084720..aed18b76dee0 100644
--- a/tools/arch/x86/include/asm/msr-index.h
+++ b/tools/arch/x86/include/asm/msr-index.h
@@ -53,6 +53,8 @@
 #define SPEC_CTRL_SSBD			BIT(SPEC_CTRL_SSBD_SHIFT)	/* Speculative Store Bypass Disable */
 #define SPEC_CTRL_RRSBA_DIS_S_SHIFT	6	   /* Disable RRSBA behavior */
 #define SPEC_CTRL_RRSBA_DIS_S		BIT(SPEC_CTRL_RRSBA_DIS_S_SHIFT)
+#define SPEC_CTRL_BHI_DIS_S_SHIFT	10         /* Enable BHI_DIS_S behavior */
+#define SPEC_CTRL_BHI_DIS_S		BIT(SPEC_CTRL_BHI_DIS_S_SHIFT)
 
 #define MSR_IA32_PRED_CMD		0x00000049 /* Prediction Command */
 #define PRED_CMD_IBPB			BIT(0)	   /* Indirect Branch Prediction Barrier */
@@ -150,6 +152,10 @@
 						 * are restricted to targets in
 						 * kernel.
 						 */
+#define ARCH_CAP_BHI_NO			BIT(20) /*
+						 * Not susceptible to Branch History
+						 * Injection.
+						 */
 #define ARCH_CAP_PBRSB_NO		BIT(24)	/*
 						 * Not susceptible to Post-Barrier
 						 * Return Stack Buffer Predictions.
-- 
2.25.1

[RFC PATCH 2/9] KVM: x86: Add a kvm-only leaf for RRSBA_CTRL

[Zhang Chen]

KVM needs to check if guests can see RRSBA_CTRL. If a guest is using
retpoline and cannot see RRSBA_CTRL and the host enumerates RRSBA,
KVM is responsible for setting RRSBA_DIS_S for the guest.

This allows VM migration from parts doesn't enumerates RRSBA to those
that enumerate RRSBA.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 arch/x86/kvm/cpuid.c         | 4 ++++
 arch/x86/kvm/reverse_cpuid.h | 7 +++++++
 2 files changed, 11 insertions(+)

diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index 62bc7a01cecc..8d45bc0b4b7c 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -668,6 +668,10 @@ void kvm_set_cpu_caps(void)
 		SF(SGX1) | SF(SGX2)
 	);
 
+	kvm_cpu_cap_init_scattered(CPUID_7_2_EDX,
+		SF(RRSBA_CTRL)
+	);
+
 	kvm_cpu_cap_mask(CPUID_8000_0001_ECX,
 		F(LAHF_LM) | F(CMP_LEGACY) | 0 /*SVM*/ | 0 /* ExtApicSpace */ |
 		F(CR8_LEGACY) | F(ABM) | F(SSE4A) | F(MISALIGNSSE) |
diff --git a/arch/x86/kvm/reverse_cpuid.h b/arch/x86/kvm/reverse_cpuid.h
index a19d473d0184..4c38ed61c505 100644
--- a/arch/x86/kvm/reverse_cpuid.h
+++ b/arch/x86/kvm/reverse_cpuid.h
@@ -13,6 +13,7 @@
  */
 enum kvm_only_cpuid_leafs {
 	CPUID_12_EAX	 = NCAPINTS,
+	CPUID_7_2_EDX,
 	NR_KVM_CPU_CAPS,
 
 	NKVMCAPINTS = NR_KVM_CPU_CAPS - NCAPINTS,
@@ -24,6 +25,9 @@ enum kvm_only_cpuid_leafs {
 #define KVM_X86_FEATURE_SGX1		KVM_X86_FEATURE(CPUID_12_EAX, 0)
 #define KVM_X86_FEATURE_SGX2		KVM_X86_FEATURE(CPUID_12_EAX, 1)
 
+/* Intel-defined sub-features, CPUID level 0x00000007:2 (EDX)*/
+#define KVM_X86_FEATURE_RRSBA_CTRL	KVM_X86_FEATURE(CPUID_7_2_EDX, 2)
+
 struct cpuid_reg {
 	u32 function;
 	u32 index;
@@ -46,6 +50,7 @@ static const struct cpuid_reg reverse_cpuid[] = {
 	[CPUID_8000_0007_EBX] = {0x80000007, 0, CPUID_EBX},
 	[CPUID_7_EDX]         = {         7, 0, CPUID_EDX},
 	[CPUID_7_1_EAX]       = {         7, 1, CPUID_EAX},
+	[CPUID_7_2_EDX]       = {         7, 2, CPUID_EDX},
 	[CPUID_12_EAX]        = {0x00000012, 0, CPUID_EAX},
 	[CPUID_8000_001F_EAX] = {0x8000001f, 0, CPUID_EAX},
 };
@@ -78,6 +83,8 @@ static __always_inline u32 __feature_translate(int x86_feature)
 		return KVM_X86_FEATURE_SGX1;
 	else if (x86_feature == X86_FEATURE_SGX2)
 		return KVM_X86_FEATURE_SGX2;
+	else if (x86_feature == X86_FEATURE_RRSBA_CTRL)
+		return KVM_X86_FEATURE_RRSBA_CTRL;
 
 	return x86_feature;
 }
-- 
2.25.1

[RFC PATCH 3/9] KVM: x86: Add a kvm-only leaf for BHI_CTRL

[Zhang Chen]

KVM needs to check if guests can see BHI_CTRL. If a guest is using
BHB-clearing sequence and cannot see BHI_CTRL and the host enumerates BHI,
KVM is responsible for setting BHI_DIS_S for the guest.

This allows VM migration from parts doesn't enumerate BHI to those
that enumerate BHI.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 arch/x86/kvm/cpuid.c         | 2 +-
 arch/x86/kvm/reverse_cpuid.h | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index 8d45bc0b4b7c..91af27cc57e5 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -669,7 +669,7 @@ void kvm_set_cpu_caps(void)
 	);
 
 	kvm_cpu_cap_init_scattered(CPUID_7_2_EDX,
-		SF(RRSBA_CTRL)
+		SF(RRSBA_CTRL) | F(BHI_CTRL)
 	);
 
 	kvm_cpu_cap_mask(CPUID_8000_0001_ECX,
diff --git a/arch/x86/kvm/reverse_cpuid.h b/arch/x86/kvm/reverse_cpuid.h
index 4c38ed61c505..cf4e209ce2f6 100644
--- a/arch/x86/kvm/reverse_cpuid.h
+++ b/arch/x86/kvm/reverse_cpuid.h
@@ -27,6 +27,8 @@ enum kvm_only_cpuid_leafs {
 
 /* Intel-defined sub-features, CPUID level 0x00000007:2 (EDX)*/
 #define KVM_X86_FEATURE_RRSBA_CTRL	KVM_X86_FEATURE(CPUID_7_2_EDX, 2)
+/* X86_FEATURE_BHI_CTRL only used by KVM */
+#define X86_FEATURE_BHI_CTRL		KVM_X86_FEATURE(CPUID_7_2_EDX, 4)
 
 struct cpuid_reg {
 	u32 function;
-- 
2.25.1

[RFC PATCH 4/9] x86/kvm/vmx: Virtualize Intel IA32_SPEC_CTRL

[Zhang Chen]

Currently KVM disables interception of IA32_SPEC_CTRL after a non-0 is
written to IA32_SPEC_CTRL by guest. Then, guest is allowed to write any
value to hardware.

"virtualize IA32_SPEC_CTRL" is a new tertiary vm-exec control.
It provides VMM the capability to restrict the value of IA32_SPEC_CTRL in
hardware even if the MSR isn't intercepted (e.g., prevent VMs from changing
some bits in IA32_SPEC_CTRL) in an efficient way.

Two new fields are added to VMCS,
IA32_SPEC_CTRL_MASK: Setting a bit in this field prevents guest software
from modifying the corresponding bit in the IA32_SPEC_CTRL MSR
IA32_SPEC_CTRL_SHADOW: This field contains the value that guest software
expects to be in the IA32_SPEC_CTRL MSR

In VMX non-root mode, when IA32_SPEC_CTRL isn't intercepted by VMM,
guest's accesses to IA32_SPEC_CTRL is virtualized by the processor
according to the two new fields:
RDMSR IA32_SPEC_CTRL returns the shadow value.
WRMSR IA32_SPEC_CTRL writes EDX:EAX to the shadow field and calculate
a new value according to the guest value (EDX:EAX), current value of
IA32_SPEC_CTRL MSR and the IA32_SPEC_CTRL_MASK field (specifically,
(cur_val & mask) | (guest_val & ~mask)) and write it to IA32_SPEC_CTRL MSR.

Enable "virtual IA32_SPEC_CTRL" if it is supported. With
"virtual IA32_SPEC_CTRL" enabled, IA32_SPEC_CTRL MSR value seen from
guest p.o.v is different from the value in hardware when guest is running.
We refer to the two values as below:

1. effective value of IA32_SPEC_CTRL. This value is the one programmed in
   hardware when the vCPU is running.
2. shadow value of IA32_SPEC_CTRL. This value is returned when rdmsr is
   used inside a guest to read IA32_SPEC_CTRL. This value doesn't affect
   CPU's enablement of indirect predictor controls.

In KVM, vmx->spec_ctrl always stores the effective value of IA32_SPEC_CTRL
when guest is running (even when "virtual IA32_SPEC_CTRL" is disabled.
In this case, the shadow value is equal to the effective one).
When "virtual IA32_SPEC_CTRL" is enabled, the shadow value of
IA32_SPEC_CTRL is stored in IA32_SPEC_CTRL_SHADOW field in VMCS.

IA32_SPEC_CTRL_MASK is always 0 for now. It means all supported bits
in hardware is allowed to be toggled by guest's wrmsr. The mask will
be changed by following patches.

Note "virtual IA32_SPEC_CTRL" is now used by VMM to enforce some bits
of IA32_SPEC_CTRL to 1 (i.e., enabled some HW mitigations transparently
for guests). In theory, VMM can disable some HW mitigations behind guests.
But to keep this series simple, we leave that for future work.

Co-developed-by: Chao Gao <chao.gao@intel.com>
Signed-off-by: Chao Gao <chao.gao@intel.com>
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 arch/x86/include/asm/vmx.h         |  5 +++++
 arch/x86/include/asm/vmxfeatures.h |  2 ++
 arch/x86/kvm/vmx/capabilities.h    |  5 +++++
 arch/x86/kvm/vmx/vmx.c             | 30 ++++++++++++++++++++++++++++--
 arch/x86/kvm/vmx/vmx.h             | 10 +++++++++-
 5 files changed, 49 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h
index 498dc600bd5c..c2efdad491c1 100644
--- a/arch/x86/include/asm/vmx.h
+++ b/arch/x86/include/asm/vmx.h
@@ -81,6 +81,7 @@
  * Definitions of Tertiary Processor-Based VM-Execution Controls.
  */
 #define TERTIARY_EXEC_IPI_VIRT			VMCS_CONTROL_BIT(IPI_VIRT)
+#define TERTIARY_EXEC_VIRT_SPEC_CTRL		VMCS_CONTROL_BIT(VIRT_SPEC_CTRL)
 
 #define PIN_BASED_EXT_INTR_MASK                 VMCS_CONTROL_BIT(INTR_EXITING)
 #define PIN_BASED_NMI_EXITING                   VMCS_CONTROL_BIT(NMI_EXITING)
@@ -233,6 +234,10 @@ enum vmcs_field {
 	TERTIARY_VM_EXEC_CONTROL_HIGH	= 0x00002035,
 	PID_POINTER_TABLE		= 0x00002042,
 	PID_POINTER_TABLE_HIGH		= 0x00002043,
+	IA32_SPEC_CTRL_MASK             = 0x0000204A,
+	IA32_SPEC_CTRL_MASK_HIGH        = 0x0000204B,
+	IA32_SPEC_CTRL_SHADOW           = 0x0000204C,
+	IA32_SPEC_CTRL_SHADOW_HIGH      = 0x0000204D,
 	GUEST_PHYSICAL_ADDRESS          = 0x00002400,
 	GUEST_PHYSICAL_ADDRESS_HIGH     = 0x00002401,
 	VMCS_LINK_POINTER               = 0x00002800,
diff --git a/arch/x86/include/asm/vmxfeatures.h b/arch/x86/include/asm/vmxfeatures.h
index c6a7eed03914..d3b7237d9c42 100644
--- a/arch/x86/include/asm/vmxfeatures.h
+++ b/arch/x86/include/asm/vmxfeatures.h
@@ -89,4 +89,6 @@
 
 /* Tertiary Processor-Based VM-Execution Controls, word 3 */
 #define VMX_FEATURE_IPI_VIRT		( 3*32+  4) /* Enable IPI virtualization */
+#define VMX_FEATURE_VIRT_SPEC_CTRL	( 3*32+  7) /* Virtualize IA32_SPEC_CTRL */
+
 #endif /* _ASM_X86_VMXFEATURES_H */
diff --git a/arch/x86/kvm/vmx/capabilities.h b/arch/x86/kvm/vmx/capabilities.h
index 07254314f3dd..a9a0adcd403b 100644
--- a/arch/x86/kvm/vmx/capabilities.h
+++ b/arch/x86/kvm/vmx/capabilities.h
@@ -138,6 +138,11 @@ static inline bool cpu_has_tertiary_exec_ctrls(void)
 		CPU_BASED_ACTIVATE_TERTIARY_CONTROLS;
 }
 
+static inline bool cpu_has_virt_spec_ctrl(void)
+{
+	return vmcs_config.cpu_based_3rd_exec_ctrl & TERTIARY_EXEC_VIRT_SPEC_CTRL;
+}
+
 static inline bool cpu_has_vmx_virtualize_apic_accesses(void)
 {
 	return vmcs_config.cpu_based_2nd_exec_ctrl &
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 63247c57c72c..407061b369b4 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -1898,7 +1898,10 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		    !guest_has_spec_ctrl_msr(vcpu))
 			return 1;
 
-		msr_info->data = to_vmx(vcpu)->spec_ctrl;
+		if (cpu_has_virt_spec_ctrl())
+			msr_info->data = vmcs_read64(IA32_SPEC_CTRL_SHADOW);
+		else
+			msr_info->data = to_vmx(vcpu)->spec_ctrl;
 		break;
 	case MSR_IA32_SYSENTER_CS:
 		msr_info->data = vmcs_read32(GUEST_SYSENTER_CS);
@@ -2160,10 +2163,22 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		    !guest_has_spec_ctrl_msr(vcpu))
 			return 1;
 
-		if (kvm_spec_ctrl_test_value(data))
+		if (kvm_spec_ctrl_test_value(data | vmx->spec_ctrl_mask))
 			return 1;
 
 		vmx->spec_ctrl = data;
+
+		if (cpu_has_virt_spec_ctrl()) {
+			vmcs_write64(IA32_SPEC_CTRL_SHADOW, data);
+			/*
+			 * Some bits are allowed to be toggled by guest.
+			 * Update the effective value of IA32_SPEC_CTRL
+			 * MSR according to the value written by guest
+			 * but keep bits in the mask set.
+			 */
+			vmx->spec_ctrl = data | vmx->spec_ctrl_mask;
+		}
+
 		if (!data)
 			break;
 
@@ -4673,6 +4688,11 @@ static void init_vmcs(struct vcpu_vmx *vmx)
 	if (cpu_has_vmx_xsaves())
 		vmcs_write64(XSS_EXIT_BITMAP, VMX_XSS_EXIT_BITMAP);
 
+	if (cpu_has_virt_spec_ctrl()) {
+		vmcs_write64(IA32_SPEC_CTRL_SHADOW, 0);
+		vmcs_write64(IA32_SPEC_CTRL_MASK, vmx->spec_ctrl_mask);
+	}
+
 	if (enable_pml) {
 		vmcs_write64(PML_ADDRESS, page_to_phys(vmx->pml_pg));
 		vmcs_write16(GUEST_PML_INDEX, PML_ENTITY_NUM - 1);
@@ -4738,6 +4758,12 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
 		__vmx_vcpu_reset(vcpu);
 
 	vmx->rmode.vm86_active = 0;
+
+	if (cpu_has_virt_spec_ctrl()) {
+		vmx->spec_ctrl_mask = 0;
+		vmcs_write64(IA32_SPEC_CTRL_MASK, vmx->spec_ctrl_mask);
+		vmcs_write64(IA32_SPEC_CTRL_SHADOW, 0);
+	}
 	vmx->spec_ctrl = 0;
 
 	vmx->msr_ia32_umwait_control = 0;
diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index a3da84f4ea45..c5a41ae14237 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -290,7 +290,14 @@ struct vcpu_vmx {
 	u64		      msr_guest_kernel_gs_base;
 #endif
 
+	/* The value of hardware IA32_SPEC_CTRL MSR when guest is running */
 	u64		      spec_ctrl;
+	/*
+	 * The bits KVM doesn't allow guest to toggle.
+	 * A bit set in the mask should always be set in guest
+	 * IA32_SPEC_CTRL_MSR.
+	 */
+	u64		      spec_ctrl_mask;
 	u32		      msr_ia32_umwait_control;
 
 	/*
@@ -589,7 +596,8 @@ static inline u8 vmx_get_rvi(void)
 
 #define KVM_REQUIRED_VMX_TERTIARY_VM_EXEC_CONTROL 0
 #define KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL			\
-	(TERTIARY_EXEC_IPI_VIRT)
+	(TERTIARY_EXEC_IPI_VIRT |					\
+	TERTIARY_EXEC_VIRT_SPEC_CTRL)
 
 #define BUILD_CONTROLS_SHADOW(lname, uname, bits)						\
 static inline void lname##_controls_set(struct vcpu_vmx *vmx, u##bits val)			\
-- 
2.25.1

[RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

[Zhang Chen]

From: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>

Guests that have different family/model than the host may not be aware
of hardware mitigations(such as RRSBA_DIS_S) available on host. This is
particularly true when guests migrate. To solve this problem Intel
processors have added a virtual MSR interface through which guests can
report their mitigation status and request VMM to deploy relevant
hardware mitigations.

Use this virtualized MSR interface to request relevant hardware controls
for retpoline mitigation.

Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
---
 arch/x86/include/asm/msr-index.h | 23 +++++++++++++++++++++++
 arch/x86/kernel/cpu/bugs.c       | 24 ++++++++++++++++++++++++
 2 files changed, 47 insertions(+)

diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 1143ac9400c3..1166b472377c 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -165,6 +165,7 @@
 						 * IA32_XAPIC_DISABLE_STATUS MSR
 						 * supported
 						 */
+#define ARCH_CAP_VIRTUAL_ENUM		BIT(63)	/* MSR_VIRTUAL_ENUMERATION supported */
 
 #define MSR_IA32_FLUSH_CMD		0x0000010b
 #define L1D_FLUSH			BIT(0)	/*
@@ -1062,6 +1063,28 @@
 #define MSR_IA32_VMX_MISC_INTEL_PT                 (1ULL << 14)
 #define MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS (1ULL << 29)
 #define MSR_IA32_VMX_MISC_PREEMPTION_TIMER_SCALE   0x1F
+
+/* Intel virtual MSRs */
+#define MSR_VIRTUAL_ENUMERATION			0x50000000
+#define VIRT_ENUM_MITIGATION_CTRL_SUPPORT	BIT(0)	/*
+							 * Mitigation ctrl via virtual
+							 * MSRs supported
+							 */
+
+#define MSR_VIRTUAL_MITIGATION_ENUM		0x50000001
+#define MITI_ENUM_BHB_CLEAR_SEQ_S_SUPPORT	BIT(0)	/* VMM supports BHI_DIS_S */
+#define MITI_ENUM_RETPOLINE_S_SUPPORT		BIT(1)	/* VMM supports RRSBA_DIS_S */
+
+#define MSR_VIRTUAL_MITIGATION_CTRL		0x50000002
+#define MITI_CTRL_BHB_CLEAR_SEQ_S_USED		BIT(0)	/*
+							 * Request VMM to deploy
+							 * BHI_DIS_S mitigation
+							 */
+#define MITI_CTRL_RETPOLINE_S_USED		BIT(1)	/*
+							 * Request VMM to deploy
+							 * RRSBA_DIS_S mitigation
+							 */
+
 /* AMD-V MSRs */
 
 #define MSR_VM_CR                       0xc0010114
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 3e3230cccaa7..a9e869f568ee 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1379,6 +1379,28 @@ static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_
 	dump_stack();
 }
 
+/* Speculation control using virtualized MSRs */
+static void __init spec_ctrl_setup_virtualized_msr(void)
+{
+	u64 msr_virt_enum, msr_mitigation_enum, msr_mitigation_ctrl;
+
+	if (!(x86_read_arch_cap_msr() & ARCH_CAP_VIRTUAL_ENUM))
+		return;
+
+	rdmsrl(MSR_VIRTUAL_ENUMERATION, msr_virt_enum);
+	if (!(msr_virt_enum & VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+		return;
+
+	rdmsrl(MSR_VIRTUAL_MITIGATION_ENUM, msr_mitigation_enum);
+	/* When retpoline is being used, request relevant hardware controls */
+	if (boot_cpu_has(X86_FEATURE_RETPOLINE) &&
+	    msr_mitigation_enum & MITI_ENUM_RETPOLINE_S_SUPPORT) {
+		rdmsrl(MSR_VIRTUAL_MITIGATION_CTRL, msr_mitigation_ctrl);
+		msr_mitigation_ctrl |= MITI_CTRL_RETPOLINE_S_USED;
+		wrmsrl(MSR_VIRTUAL_MITIGATION_CTRL, msr_mitigation_ctrl);
+	}
+}
+
 static void __init spectre_v2_select_mitigation(void)
 {
 	enum spectre_v2_mitigation_cmd cmd = spectre_v2_parse_cmdline();
@@ -1485,6 +1507,8 @@ static void __init spectre_v2_select_mitigation(void)
 	    mode == SPECTRE_V2_RETPOLINE)
 		spec_ctrl_disable_kernel_rrsba();
 
+	spec_ctrl_setup_virtualized_msr();
+
 	spectre_v2_enabled = mode;
 	pr_info("%s\n", spectre_v2_strings[mode]);
 
-- 
2.25.1

[RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

[Zhang Chen]

Add the 63 bit in MSR_IA32_ARCH_CAPABILITIES for enable the virtual MSRs.
Virtual MSRs can allow guests to notify VMM whether or not
they are using specific software mitigation, allowing a VMM
to enable there hardware control only where necessary.
As Intel spec defination, expose virtual MSR for guest.
Make guest have ability to check virtual MSR 0x50000000.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 arch/x86/kvm/vmx/vmx.c | 15 +++++++++++++++
 arch/x86/kvm/vmx/vmx.h |  1 +
 arch/x86/kvm/x86.c     | 16 +++++++++++++++-
 3 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 407061b369b4..6ed6b743be0e 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -2001,6 +2001,12 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_IA32_DEBUGCTLMSR:
 		msr_info->data = vmcs_read64(GUEST_IA32_DEBUGCTL);
 		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		if (!msr_info->host_initiated &&
+		    !(vcpu->arch.arch_capabilities & ARCH_CAP_VIRTUAL_ENUM))
+			return 1;
+		msr_info->data = vmx->msr_virtual_enumeration;
+		break;
 	default:
 	find_uret_msr:
 		msr = vmx_find_uret_msr(vmx, msr_info->index);
@@ -2375,6 +2381,15 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		}
 		ret = kvm_set_msr_common(vcpu, msr_info);
 		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		if (msr_info->host_initiated &&
+		    !(vcpu->arch.arch_capabilities & ARCH_CAP_VIRTUAL_ENUM))
+			return 1;
+		if (data & ~VIRT_ENUM_MITIGATION_CTRL_SUPPORT)
+			return 1;
+		vmx->msr_virtual_enumeration = data &
+						VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
+		break;
 
 	default:
 	find_uret_msr:
diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index c5a41ae14237..fc873cf45f70 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -298,6 +298,7 @@ struct vcpu_vmx {
 	 * IA32_SPEC_CTRL_MSR.
 	 */
 	u64		      spec_ctrl_mask;
+	u64		      msr_virtual_enumeration;
 	u32		      msr_ia32_umwait_control;
 
 	/*
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 2835bd796639..6be0a3f1281f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1532,6 +1532,8 @@ static const u32 emulated_msrs_all[] = {
 	MSR_IA32_VMX_EPT_VPID_CAP,
 	MSR_IA32_VMX_VMFUNC,
 
+	MSR_VIRTUAL_ENUMERATION,
+
 	MSR_K7_HWCR,
 	MSR_KVM_POLL_CONTROL,
 };
@@ -1567,6 +1569,7 @@ static const u32 msr_based_features_all[] = {
 	MSR_IA32_UCODE_REV,
 	MSR_IA32_ARCH_CAPABILITIES,
 	MSR_IA32_PERF_CAPABILITIES,
+	MSR_VIRTUAL_ENUMERATION,
 };
 
 static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all)];
@@ -1588,7 +1591,8 @@ static unsigned int num_msr_based_features;
 	 ARCH_CAP_SKIP_VMENTRY_L1DFLUSH | ARCH_CAP_SSB_NO | ARCH_CAP_MDS_NO | \
 	 ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \
 	 ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \
-	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO)
+	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | \
+	 ARCH_CAP_VIRTUAL_ENUM)
 
 static u64 kvm_get_arch_capabilities(void)
 {
@@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
 	 */
 	data |= ARCH_CAP_PSCHANGE_MC_NO;
 
+	/*
+	 * Virtual MSRs can allow guests to notify VMM whether or not
+	 * they are using specific software mitigation, allowing a VMM
+	 * to enable there hardware control only where necessary.
+	 */
+	data |= ARCH_CAP_VIRTUAL_ENUM;
+
 	/*
 	 * If we're doing cache flushes (either "always" or "cond")
 	 * we will do one whenever the guest does a vmlaunch/vmresume.
@@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct kvm_msr_entry *msr)
 	case MSR_IA32_UCODE_REV:
 		rdmsrl_safe(msr->index, &msr->data);
 		break;
+	case MSR_VIRTUAL_ENUMERATION:
+		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
+		break;
 	default:
 		return static_call(kvm_x86_get_msr_feature)(msr);
 	}
-- 
2.25.1

[RFC PATCH 7/9] kvm/x86: Add MSR_VIRTUAL_MITIGATION_ENUM/CTRL emulation

[Zhang Chen]

Introduce Intel virtual MSR_VIRTUAL_MITIGATION_ENUM(0x50000001)
and MSR_VIRTUAL_MITIGATION_CTRL(0x50000002).
The MSR_VIRTUAL_MITIGATION_ENUM to tell guest about supported
mitigations and enable the MSR_VIRTUAL_MITIGATION_CTRL virtual
MSRs for guest, VMM will help to setup virtual spec ctrl mask
for SPEC_CTRL_RRSBA_DIS_S, SPEC_CTRL_BHI_DIS_S as guest's needs.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 arch/x86/kvm/vmx/vmx.h | 16 ++++++++++++++++
 arch/x86/kvm/x86.c     |  7 +++++++
 2 files changed, 23 insertions(+)

diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index fc873cf45f70..6abda05cc426 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -29,6 +29,10 @@
 #endif
 
 #define MAX_NR_LOADSTORE_MSRS	8
+#define MITI_ENUM_SUPPORTED	(MITI_ENUM_BHB_CLEAR_SEQ_S_SUPPORT |          \
+				MITI_ENUM_RETPOLINE_S_SUPPORT)
+#define MITI_CTRL_USED		(MITI_CTRL_BHB_CLEAR_SEQ_S_USED |                 \
+				MITI_CTRL_RETPOLINE_S_USED)
 
 struct vmx_msrs {
 	unsigned int		nr;
@@ -301,6 +305,18 @@ struct vcpu_vmx {
 	u64		      msr_virtual_enumeration;
 	u32		      msr_ia32_umwait_control;
 
+	/*
+	 * Guest read only, Only available if MITIGATION_CTRL_SUPPORT
+	 * is enumerated.
+	 */
+	u64		      msr_virtual_mitigation_enum;
+
+	/*
+	 * Read/Write, Only available if MITIGATION_CTRL_SUPPORT
+	 * is enumerated.
+	 */
+	u64		      msr_virtual_mitigation_ctrl;
+
 	/*
 	 * loaded_vmcs points to the VMCS currently used in this vcpu. For a
 	 * non-nested (L1) guest, it always points to vmcs01. For a nested
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 6be0a3f1281f..f6c314def6a8 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1533,6 +1533,8 @@ static const u32 emulated_msrs_all[] = {
 	MSR_IA32_VMX_VMFUNC,
 
 	MSR_VIRTUAL_ENUMERATION,
+	MSR_VIRTUAL_MITIGATION_ENUM,
+	MSR_VIRTUAL_MITIGATION_CTRL,
 
 	MSR_K7_HWCR,
 	MSR_KVM_POLL_CONTROL,
@@ -1570,6 +1572,7 @@ static const u32 msr_based_features_all[] = {
 	MSR_IA32_ARCH_CAPABILITIES,
 	MSR_IA32_PERF_CAPABILITIES,
 	MSR_VIRTUAL_ENUMERATION,
+	MSR_VIRTUAL_MITIGATION_ENUM,
 };
 
 static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all)];
@@ -1671,6 +1674,10 @@ static int kvm_get_msr_feature(struct kvm_msr_entry *msr)
 	case MSR_VIRTUAL_ENUMERATION:
 		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
 		break;
+	case MSR_VIRTUAL_MITIGATION_ENUM:
+		msr->data = MITI_ENUM_BHB_CLEAR_SEQ_S_SUPPORT |
+			    MITI_ENUM_RETPOLINE_S_SUPPORT;
+		break;
 	default:
 		return static_call(kvm_x86_get_msr_feature)(msr);
 	}
-- 
2.25.1

[RFC PATCH 8/9] x86/kvm/vmx: Initialize SPEC_CTRL MASK for RRSBA

[Zhang Chen]

VMMs can address mitigations issues in migration pool by applying
the needed controls whenever the guest is operating on a newer
processor. If a guest is using retpoline to mitigate intra-mode BTI
in CPL0, the VMM can set RRSBA_DIS_S when the guest runs on hardware
which enumerates RRSBA.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 arch/x86/kvm/vmx/vmx.c | 57 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 56 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 6ed6b743be0e..fb0f3b1639b9 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -2007,6 +2007,20 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 			return 1;
 		msr_info->data = vmx->msr_virtual_enumeration;
 		break;
+	case MSR_VIRTUAL_MITIGATION_ENUM:
+		if (!msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		msr_info->data = vmx->msr_virtual_mitigation_enum;
+		break;
+	case MSR_VIRTUAL_MITIGATION_CTRL:
+		if (!msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		msr_info->data = vmx->msr_virtual_mitigation_ctrl;
+		break;
 	default:
 	find_uret_msr:
 		msr = vmx_find_uret_msr(vmx, msr_info->index);
@@ -2056,7 +2070,7 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	struct vmx_uret_msr *msr;
 	int ret = 0;
 	u32 msr_index = msr_info->index;
-	u64 data = msr_info->data;
+	u64 data = msr_info->data, arch_msr;
 	u32 index;
 
 	switch (msr_index) {
@@ -2390,6 +2404,46 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		vmx->msr_virtual_enumeration = data &
 						VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
 		break;
+	case MSR_VIRTUAL_MITIGATION_ENUM:
+		if (msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		if (data & ~MITI_ENUM_SUPPORTED)
+			return 1;
+		vmx->msr_virtual_mitigation_enum = data;
+		break;
+	case MSR_VIRTUAL_MITIGATION_CTRL:
+		if (!msr_info->host_initiated &&
+		    !(vmx->msr_virtual_enumeration &
+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
+			return 1;
+		if (data & ~MITI_CTRL_USED)
+			return 1;
+
+		if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES))
+			rdmsrl(MSR_IA32_ARCH_CAPABILITIES, arch_msr);
+
+		if (data & MITI_CTRL_RETPOLINE_S_USED &&
+		    boot_cpu_has(X86_FEATURE_RRSBA_CTRL) &&
+		    arch_msr & ARCH_CAP_RRSBA)
+			vmx->spec_ctrl_mask |= SPEC_CTRL_RRSBA_DIS_S;
+		else
+			vmx->spec_ctrl_mask &= ~SPEC_CTRL_RRSBA_DIS_S;
+
+		if (cpu_has_virt_spec_ctrl()) {
+			vmcs_write64(IA32_SPEC_CTRL_MASK, vmx->spec_ctrl_mask);
+		} else if (vmx->spec_ctrl_mask) {
+			pr_err_once("Virtual spec ctrl is missing. Cannot keep "
+					"bits in %llx always set\n",
+					vmx->spec_ctrl_mask);
+			vmx->spec_ctrl_mask = 0;
+		}
+
+		vmx->spec_ctrl = vmx->spec_ctrl | vmx->spec_ctrl_mask;
+
+		vmx->msr_virtual_mitigation_ctrl = data;
+		break;
 
 	default:
 	find_uret_msr:
@@ -4774,6 +4828,7 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
 
 	vmx->rmode.vm86_active = 0;
 
+	vmx->msr_virtual_mitigation_ctrl = 0;
 	if (cpu_has_virt_spec_ctrl()) {
 		vmx->spec_ctrl_mask = 0;
 		vmcs_write64(IA32_SPEC_CTRL_MASK, vmx->spec_ctrl_mask);
-- 
2.25.1

[RFC PATCH 9/9] x86/kvm/vmx: Initialize SPEC_CTRL MASK for BHI

[Zhang Chen]

VMMs can address mitigations issues in migration pool by applying
the needed controls whenever the guest is operating on a newer
processor. If a guest is using the BHB-clearing sequence
on transitions into CPL0 to mitigate BHI, the VMM can use the
“virtual IA32_SPEC_CTRL” VM-execution control to set BHI_DIS_S
on newer hardware which does not enumerate BHI_NO.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 arch/x86/kvm/vmx/vmx.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index fb0f3b1639b9..980d1ace9718 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -2431,6 +2431,13 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		else
 			vmx->spec_ctrl_mask &= ~SPEC_CTRL_RRSBA_DIS_S;
 
+		if (data & MITI_CTRL_BHB_CLEAR_SEQ_S_USED &&
+		    kvm_cpu_cap_has(X86_FEATURE_BHI_CTRL) &&
+		    !(arch_msr & ARCH_CAP_BHI_NO))
+			vmx->spec_ctrl_mask |= SPEC_CTRL_BHI_DIS_S;
+		else
+			vmx->spec_ctrl_mask &= ~SPEC_CTRL_BHI_DIS_S;
+
 		if (cpu_has_virt_spec_ctrl()) {
 			vmcs_write64(IA32_SPEC_CTRL_MASK, vmx->spec_ctrl_mask);
 		} else if (vmx->spec_ctrl_mask) {
-- 
2.25.1

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

[Pawan Gupta]

On Sun, Dec 11, 2022 at 12:00:42AM +0800, Zhang Chen wrote:
> From: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> 
> Guests that have different family/model than the host may not be aware
> of hardware mitigations(such as RRSBA_DIS_S) available on host. This is
> particularly true when guests migrate. To solve this problem Intel
> processors have added a virtual MSR interface through which guests can
> report their mitigation status and request VMM to deploy relevant
> hardware mitigations.
> 
> Use this virtualized MSR interface to request relevant hardware controls
> for retpoline mitigation.
> 
> Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> ---
>  arch/x86/include/asm/msr-index.h | 23 +++++++++++++++++++++++
>  arch/x86/kernel/cpu/bugs.c       | 24 ++++++++++++++++++++++++
>  2 files changed, 47 insertions(+)
> 
> diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
> index 1143ac9400c3..1166b472377c 100644
> --- a/arch/x86/include/asm/msr-index.h
> +++ b/arch/x86/include/asm/msr-index.h
> @@ -165,6 +165,7 @@
>  						 * IA32_XAPIC_DISABLE_STATUS MSR
>  						 * supported
>  						 */
> +#define ARCH_CAP_VIRTUAL_ENUM		BIT(63)	/* MSR_VIRTUAL_ENUMERATION supported */
>  
>  #define MSR_IA32_FLUSH_CMD		0x0000010b
>  #define L1D_FLUSH			BIT(0)	/*
> @@ -1062,6 +1063,28 @@
>  #define MSR_IA32_VMX_MISC_INTEL_PT                 (1ULL << 14)
>  #define MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS (1ULL << 29)
>  #define MSR_IA32_VMX_MISC_PREEMPTION_TIMER_SCALE   0x1F
> +
> +/* Intel virtual MSRs */
> +#define MSR_VIRTUAL_ENUMERATION			0x50000000
> +#define VIRT_ENUM_MITIGATION_CTRL_SUPPORT	BIT(0)	/*
> +							 * Mitigation ctrl via virtual
> +							 * MSRs supported
> +							 */
> +
> +#define MSR_VIRTUAL_MITIGATION_ENUM		0x50000001
> +#define MITI_ENUM_BHB_CLEAR_SEQ_S_SUPPORT	BIT(0)	/* VMM supports BHI_DIS_S */
> +#define MITI_ENUM_RETPOLINE_S_SUPPORT		BIT(1)	/* VMM supports RRSBA_DIS_S */
> +
> +#define MSR_VIRTUAL_MITIGATION_CTRL		0x50000002
> +#define MITI_CTRL_BHB_CLEAR_SEQ_S_USED		BIT(0)	/*
> +							 * Request VMM to deploy
> +							 * BHI_DIS_S mitigation
> +							 */
> +#define MITI_CTRL_RETPOLINE_S_USED		BIT(1)	/*
> +							 * Request VMM to deploy
> +							 * RRSBA_DIS_S mitigation
> +							 */
> +
>  /* AMD-V MSRs */
>  
>  #define MSR_VM_CR                       0xc0010114
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 3e3230cccaa7..a9e869f568ee 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -1379,6 +1379,28 @@ static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_
>  	dump_stack();
>  }
>  
> +/* Speculation control using virtualized MSRs */
> +static void __init spec_ctrl_setup_virtualized_msr(void)
> +{
> +	u64 msr_virt_enum, msr_mitigation_enum, msr_mitigation_ctrl;
> +
> +	if (!(x86_read_arch_cap_msr() & ARCH_CAP_VIRTUAL_ENUM))
> +		return;
> +
> +	rdmsrl(MSR_VIRTUAL_ENUMERATION, msr_virt_enum);
> +	if (!(msr_virt_enum & VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
> +		return;
> +
> +	rdmsrl(MSR_VIRTUAL_MITIGATION_ENUM, msr_mitigation_enum);
> +	/* When retpoline is being used, request relevant hardware controls */
> +	if (boot_cpu_has(X86_FEATURE_RETPOLINE) &&
> +	    msr_mitigation_enum & MITI_ENUM_RETPOLINE_S_SUPPORT) {
> +		rdmsrl(MSR_VIRTUAL_MITIGATION_CTRL, msr_mitigation_ctrl);
> +		msr_mitigation_ctrl |= MITI_CTRL_RETPOLINE_S_USED;
> +		wrmsrl(MSR_VIRTUAL_MITIGATION_CTRL, msr_mitigation_ctrl);
> +	}
> +}
> +
>  static void __init spectre_v2_select_mitigation(void)
>  {
>  	enum spectre_v2_mitigation_cmd cmd = spectre_v2_parse_cmdline();
> @@ -1485,6 +1507,8 @@ static void __init spectre_v2_select_mitigation(void)
>  	    mode == SPECTRE_V2_RETPOLINE)
>  		spec_ctrl_disable_kernel_rrsba();
>  
> +	spec_ctrl_setup_virtualized_msr();

I think this also needs to be called during secondary CPU
initialization.

RE: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

[Zhang, Chen]

> -----Original Message-----
> From: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> Sent: Tuesday, December 13, 2022 4:24 AM
> To: Zhang, Chen <chen.zhang@intel.com>
> Cc: x86@kernel.org; linux-kernel@vger.kernel.org; kvm@vger.kernel.org; Gao,
> Chao <chao.gao@intel.com>; Paolo Bonzini <pbonzini@redhat.com>;
> Christopherson,, Sean <seanjc@google.com>; H. Peter Anvin
> <hpa@zytor.com>; Dave Hansen <dave.hansen@linux.intel.com>; Borislav
> Petkov <bp@alien8.de>; Ingo Molnar <mingo@redhat.com>; Thomas Gleixner
> <tglx@linutronix.de>
> Subject: Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware
> mitigations
> 
> On Sun, Dec 11, 2022 at 12:00:42AM +0800, Zhang Chen wrote:
> > From: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> >
> > Guests that have different family/model than the host may not be aware
> > of hardware mitigations(such as RRSBA_DIS_S) available on host. This
> > is particularly true when guests migrate. To solve this problem Intel
> > processors have added a virtual MSR interface through which guests can
> > report their mitigation status and request VMM to deploy relevant
> > hardware mitigations.
> >
> > Use this virtualized MSR interface to request relevant hardware
> > controls for retpoline mitigation.
> >
> > Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> > ---
> >  arch/x86/include/asm/msr-index.h | 23 +++++++++++++++++++++++
> >  arch/x86/kernel/cpu/bugs.c       | 24 ++++++++++++++++++++++++
> >  2 files changed, 47 insertions(+)
> >
> > diff --git a/arch/x86/include/asm/msr-index.h
> > b/arch/x86/include/asm/msr-index.h
> > index 1143ac9400c3..1166b472377c 100644
> > --- a/arch/x86/include/asm/msr-index.h
> > +++ b/arch/x86/include/asm/msr-index.h
> > @@ -165,6 +165,7 @@
> >  						 *
> IA32_XAPIC_DISABLE_STATUS MSR
> >  						 * supported
> >  						 */
> > +#define ARCH_CAP_VIRTUAL_ENUM		BIT(63)	/*
> MSR_VIRTUAL_ENUMERATION supported */
> >
> >  #define MSR_IA32_FLUSH_CMD		0x0000010b
> >  #define L1D_FLUSH			BIT(0)	/*
> > @@ -1062,6 +1063,28 @@
> >  #define MSR_IA32_VMX_MISC_INTEL_PT                 (1ULL << 14)
> >  #define MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS (1ULL <<
> 29)
> >  #define MSR_IA32_VMX_MISC_PREEMPTION_TIMER_SCALE   0x1F
> > +
> > +/* Intel virtual MSRs */
> > +#define MSR_VIRTUAL_ENUMERATION			0x50000000
> > +#define VIRT_ENUM_MITIGATION_CTRL_SUPPORT	BIT(0)	/*
> > +							 * Mitigation ctrl via
> virtual
> > +							 * MSRs supported
> > +							 */
> > +
> > +#define MSR_VIRTUAL_MITIGATION_ENUM		0x50000001
> > +#define MITI_ENUM_BHB_CLEAR_SEQ_S_SUPPORT	BIT(0)	/* VMM
> supports BHI_DIS_S */
> > +#define MITI_ENUM_RETPOLINE_S_SUPPORT		BIT(1)	/*
> VMM supports RRSBA_DIS_S */
> > +
> > +#define MSR_VIRTUAL_MITIGATION_CTRL		0x50000002
> > +#define MITI_CTRL_BHB_CLEAR_SEQ_S_USED		BIT(0)	/*
> > +							 * Request VMM to
> deploy
> > +							 * BHI_DIS_S
> mitigation
> > +							 */
> > +#define MITI_CTRL_RETPOLINE_S_USED		BIT(1)	/*
> > +							 * Request VMM to
> deploy
> > +							 * RRSBA_DIS_S
> mitigation
> > +							 */
> > +
> >  /* AMD-V MSRs */
> >
> >  #define MSR_VM_CR                       0xc0010114
> > diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> > index 3e3230cccaa7..a9e869f568ee 100644
> > --- a/arch/x86/kernel/cpu/bugs.c
> > +++ b/arch/x86/kernel/cpu/bugs.c
> > @@ -1379,6 +1379,28 @@ static void __init
> spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_
> >  	dump_stack();
> >  }
> >
> > +/* Speculation control using virtualized MSRs */ static void __init
> > +spec_ctrl_setup_virtualized_msr(void)
> > +{
> > +	u64 msr_virt_enum, msr_mitigation_enum, msr_mitigation_ctrl;
> > +
> > +	if (!(x86_read_arch_cap_msr() & ARCH_CAP_VIRTUAL_ENUM))
> > +		return;
> > +
> > +	rdmsrl(MSR_VIRTUAL_ENUMERATION, msr_virt_enum);
> > +	if (!(msr_virt_enum & VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
> > +		return;
> > +
> > +	rdmsrl(MSR_VIRTUAL_MITIGATION_ENUM, msr_mitigation_enum);
> > +	/* When retpoline is being used, request relevant hardware controls
> */
> > +	if (boot_cpu_has(X86_FEATURE_RETPOLINE) &&
> > +	    msr_mitigation_enum & MITI_ENUM_RETPOLINE_S_SUPPORT) {
> > +		rdmsrl(MSR_VIRTUAL_MITIGATION_CTRL,
> msr_mitigation_ctrl);
> > +		msr_mitigation_ctrl |= MITI_CTRL_RETPOLINE_S_USED;
> > +		wrmsrl(MSR_VIRTUAL_MITIGATION_CTRL,
> msr_mitigation_ctrl);
> > +	}
> > +}
> > +
> >  static void __init spectre_v2_select_mitigation(void)
> >  {
> >  	enum spectre_v2_mitigation_cmd cmd = spectre_v2_parse_cmdline();
> @@
> > -1485,6 +1507,8 @@ static void __init spectre_v2_select_mitigation(void)
> >  	    mode == SPECTRE_V2_RETPOLINE)
> >  		spec_ctrl_disable_kernel_rrsba();
> >
> > +	spec_ctrl_setup_virtualized_msr();
> 
> I think this also needs to be called during secondary CPU initialization.

Yes, we need setup virtual MSRs for each CPU.
I will add the " spec_ctrl_setup_virtualized_msr()" to the "start_secondary()" in the arch/x86/kernel/smpboot.c  next version.

Thanks
Chen

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

[Sean Christopherson]

On Sun, Dec 11, 2022, Zhang Chen wrote:
> From: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
> 
> Guests that have different family/model than the host may not be aware
> of hardware mitigations(such as RRSBA_DIS_S) available on host. This is
> particularly true when guests migrate. To solve this problem Intel
> processors have added a virtual MSR interface

Is there any actual "processor" support here?  To me, this looks like Intel is
foisting a paravirt interface on KVM and other hypervisors without collaborating
with said hypervisors' developers and maintainers.

I get that some of the mitigations are vendor specific, but things like RETPOLINE
aren't vendor specific.  I haven't followed all of the mitigation stuff very
closely, but I wouldn't be surprised if there are mitigations now or in the future
that are common across architectures, e.g. arm64 and x86-64.  Intel doing its own
thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting
multiple paravirt interfaces for very similar things, without having any control
over the APIs.  That's all kinds of backwards.

And having to wait for Intel to roll out new documentation when software inevitably
comes up with some clever new mitigation doesn't exactly fill my heart with joy.

Re: [RFC PATCH 2/9] KVM: x86: Add a kvm-only leaf for RRSBA_CTRL

[Ricardo Neri]

On Sun, Dec 11, 2022 at 12:00:39AM +0800, Zhang Chen wrote:
> KVM needs to check if guests can see RRSBA_CTRL. If a guest is using
> retpoline and cannot see RRSBA_CTRL and the host enumerates RRSBA,
> KVM is responsible for setting RRSBA_DIS_S for the guest.
> 
> This allows VM migration from parts doesn't enumerates RRSBA to those

perhaps s/parts doesn't enumerates/parts that do not enumerate/ reads
better?

> that enumerate RRSBA.

and here maybe s/that enumerate RRSBA/do/ ?

Thanks and BR,
Ricardo

Re: [RFC PATCH 3/9] KVM: x86: Add a kvm-only leaf for BHI_CTRL

[Ricardo Neri]

On Sun, Dec 11, 2022 at 12:00:40AM +0800, Zhang Chen wrote:
> KVM needs to check if guests can see BHI_CTRL. If a guest is using
> BHB-clearing sequence and cannot see BHI_CTRL and the host enumerates BHI,
> KVM is responsible for setting BHI_DIS_S for the guest.
> 
> This allows VM migration from parts doesn't enumerate BHI to those
> that enumerate BHI.

Similar comment as in patch 2.

RE: [RFC PATCH 2/9] KVM: x86: Add a kvm-only leaf for RRSBA_CTRL

[Zhang, Chen]

> -----Original Message-----
> From: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
> Sent: Thursday, December 15, 2022 5:34 AM
> To: Zhang, Chen <chen.zhang@intel.com>
> Cc: x86@kernel.org; linux-kernel@vger.kernel.org; kvm@vger.kernel.org; Gao,
> Chao <chao.gao@intel.com>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Paolo Bonzini
> <pbonzini@redhat.com>; Christopherson,, Sean <seanjc@google.com>; H.
> Peter Anvin <hpa@zytor.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> Borislav Petkov <bp@alien8.de>; Ingo Molnar <mingo@redhat.com>; Thomas
> Gleixner <tglx@linutronix.de>
> Subject: Re: [RFC PATCH 2/9] KVM: x86: Add a kvm-only leaf for RRSBA_CTRL
> 
> On Sun, Dec 11, 2022 at 12:00:39AM +0800, Zhang Chen wrote:
> > KVM needs to check if guests can see RRSBA_CTRL. If a guest is using
> > retpoline and cannot see RRSBA_CTRL and the host enumerates RRSBA, KVM
> > is responsible for setting RRSBA_DIS_S for the guest.
> >
> > This allows VM migration from parts doesn't enumerates RRSBA to those
> 
> perhaps s/parts doesn't enumerates/parts that do not enumerate/ reads
> better?

OK.

> 
> > that enumerate RRSBA.
> 
> and here maybe s/that enumerate RRSBA/do/ ?

Looks good to me. Thanks for your comments.
I will fix patch2/3 in the next version.

Thanks
Chen

> 
> Thanks and BR,
> Ricardo

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

[Chao Gao]

On Wed, Dec 14, 2022 at 08:18:17PM +0000, Sean Christopherson wrote:
>On Sun, Dec 11, 2022, Zhang Chen wrote:
>> From: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
>> 
>> Guests that have different family/model than the host may not be aware
>> of hardware mitigations(such as RRSBA_DIS_S) available on host. This is
>> particularly true when guests migrate. To solve this problem Intel
>> processors have added a virtual MSR interface
>
>Is there any actual "processor" support here?

No.

>  To me, this looks like Intel is
>foisting a paravirt interface on KVM and other hypervisors without collaborating
>with said hypervisors' developers and maintainers.
>
>I get that some of the mitigations are vendor specific, but things like RETPOLINE
>aren't vendor specific.  I haven't followed all of the mitigation stuff very
>closely, but I wouldn't be surprised if there are mitigations now or in the future
>that are common across architectures, e.g. arm64 and x86-64.  Intel doing its own
>thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting
>multiple paravirt interfaces for very similar things, without having any control
>over the APIs.  That's all kinds of backwards.

But if the interface is defined by KVM rather than Intel, it will likely end up
with different interfaces for different VMMs, then Linux guest needs to support
all of them. And KVM needs to implement Hyper-V's and Xen's interface to support
Hyper-V enlightened and Xen enlightened guest. This is a _real_ problem and
complicates KVM/Linux in a similar way as multiple paravirt interfaces.

The use case of this paravirt interface is specific to Intel CPU microarchitecture.
Supporting multiple paravirt interfaces may not happen in the near future if there
is no use case for AMD and arm64.

>
>And having to wait for Intel to roll out new documentation when software inevitably
>comes up with some clever new mitigation doesn't exactly fill my heart with joy.

There should be some misunderstanding.

A bit for a software mitigation will be added if
1. the software mitigation works well on existing processors. And
2. there will be a new processor on which the software mitigation won't be
   fully effective (due to some microarchiecture change).

IOW, the new documentation comes along with a new microarchitecture (new
processor) rather than software mitigations.

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

[Sean Christopherson]

On Mon, Dec 19, 2022, Chao Gao wrote:
> On Wed, Dec 14, 2022 at 08:18:17PM +0000, Sean Christopherson wrote:
> > To me, this looks like Intel is foisting a paravirt interface on KVM and other
> > hypervisors without collaborating with said hypervisors' developers and maintainers.
> >
> >I get that some of the mitigations are vendor specific, but things like RETPOLINE
> >aren't vendor specific.  I haven't followed all of the mitigation stuff very
> >closely, but I wouldn't be surprised if there are mitigations now or in the future
> >that are common across architectures, e.g. arm64 and x86-64.  Intel doing its own
> >thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting
> >multiple paravirt interfaces for very similar things, without having any control
> >over the APIs.  That's all kinds of backwards.
> 
> But if the interface is defined by KVM rather than Intel, it will likely end up
> with different interfaces for different VMMs, then Linux guest needs to support
> all of them. And KVM needs to implement Hyper-V's and Xen's interface to support
> Hyper-V enlightened and Xen enlightened guest. This is a _real_ problem and
> complicates KVM/Linux in a similar way as multiple paravirt interfaces.

I never said the PV interfaces should be defined by KVM.  I 100% agree that any
one hypervisor defining its own interface will suffer the same problem.

I think having a PV interface for coordinating mitigations between host and guest
is a great idea.  What I don't like is tying the interface to "hardware" and defining
the interface without even trying to collaborate with others.

> The use case of this paravirt interface is specific to Intel CPU microarchitecture.

Well yeah, because the interface was designed only to work for Intel CPUs.

> Supporting multiple paravirt interfaces may not happen in the near future if there
> is no use case for AMD and arm64.

I'll take that bet.  The vast majority of problems that are solved by PV interfaces
are common to all architectures and vendors, e.g. steal time, PV spinlocks, async
page faults, directed yield, confidential VMs (GHCB vs. GHCI), etc.  I highly doubt
Intel is the only hardware vendor that will ever benefit from paravirtualizing
mitigations.

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

[Chao Gao]

On Mon, Dec 19, 2022 at 05:14:00PM +0000, Sean Christopherson wrote:
>On Mon, Dec 19, 2022, Chao Gao wrote:
>> On Wed, Dec 14, 2022 at 08:18:17PM +0000, Sean Christopherson wrote:
>> > To me, this looks like Intel is foisting a paravirt interface on KVM and other
>> > hypervisors without collaborating with said hypervisors' developers and maintainers.
>> >
>> >I get that some of the mitigations are vendor specific, but things like RETPOLINE
>> >aren't vendor specific.  I haven't followed all of the mitigation stuff very
>> >closely, but I wouldn't be surprised if there are mitigations now or in the future
>> >that are common across architectures, e.g. arm64 and x86-64.  Intel doing its own
>> >thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting
>> >multiple paravirt interfaces for very similar things, without having any control
>> >over the APIs.  That's all kinds of backwards.
>> 
>> But if the interface is defined by KVM rather than Intel, it will likely end up
>> with different interfaces for different VMMs, then Linux guest needs to support
>> all of them. And KVM needs to implement Hyper-V's and Xen's interface to support
>> Hyper-V enlightened and Xen enlightened guest. This is a _real_ problem and
>> complicates KVM/Linux in a similar way as multiple paravirt interfaces.
>
>I never said the PV interfaces should be defined by KVM.  I 100% agree that any
>one hypervisor defining its own interface will suffer the same problem.

I am thinking there are only two options:

1. Intel defines the interface.
2. Every VMM defines its own interface.

Any middle ground between the two options?

>
>I think having a PV interface for coordinating mitigations between host and guest
>is a great idea.  What I don't like is tying the interface to "hardware" and defining

Do you think something below is better than the intel-defined interface?
add a new KVM_CAP_* and a KVM_FEATURE_* in hypervisor CPUID leaf to enumerate
the interface. And add a new virtual MSR 0x4b564dxx for guest to report in-use
software mitigations and assign one bit for each software mitigation. On MSR
write emulation, call into vmx code to enable some hardware mitigations if
the corresponding software mitigations are not effective on host.

I am afraid it is late to switch to above approach because e.g., if Hyper-V
decides to support the intel-defined interface, KVM probably needs to support it
for Hyper-V guests.

Can we emulate IA32_ARCH_CAPABILITIES for AMD guests so that AMD can re-use
intel-defined interface?

>the interface without even trying to collaborate with others.

I will pass on this to related Intel folks and ask them to collaborate with
KVM community on future PV interfaces.

Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

[Yang, Weijiang]

On 12/11/2022 12:00 AM, Zhang Chen wrote:
> Add the 63 bit in MSR_IA32_ARCH_CAPABILITIES for enable the virtual MSRs.
> Virtual MSRs can allow guests to notify VMM whether or not
> they are using specific software mitigation, allowing a VMM
> to enable there hardware control only where necessary.
> As Intel spec defination, expose virtual MSR for guest.
> Make guest have ability to check virtual MSR 0x50000000.
>
> Signed-off-by: Zhang Chen <chen.zhang@intel.com>
> ---
>   arch/x86/kvm/vmx/vmx.c | 15 +++++++++++++++
>   arch/x86/kvm/vmx/vmx.h |  1 +
>   arch/x86/kvm/x86.c     | 16 +++++++++++++++-
>   3 files changed, 31 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> index 407061b369b4..6ed6b743be0e 100644
> --- a/arch/x86/kvm/vmx/vmx.c
> +++ b/arch/x86/kvm/vmx/vmx.c
> @@ -2001,6 +2001,12 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
>   	case MSR_IA32_DEBUGCTLMSR:
>   		msr_info->data = vmcs_read64(GUEST_IA32_DEBUGCTL);
>   		break;
> +	case MSR_VIRTUAL_ENUMERATION:
> +		if (!msr_info->host_initiated &&
> +		    !(vcpu->arch.arch_capabilities & ARCH_CAP_VIRTUAL_ENUM))
> +			return 1;
> +		msr_info->data = vmx->msr_virtual_enumeration;
> +		break;
>   	default:
>   	find_uret_msr:
>   		msr = vmx_find_uret_msr(vmx, msr_info->index);
> @@ -2375,6 +2381,15 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
>   		}
>   		ret = kvm_set_msr_common(vcpu, msr_info);
>   		break;
> +	case MSR_VIRTUAL_ENUMERATION:
> +		if (msr_info->host_initiated &&
> +		    !(vcpu->arch.arch_capabilities & ARCH_CAP_VIRTUAL_ENUM))
> +			return 1;
> +		if (data & ~VIRT_ENUM_MITIGATION_CTRL_SUPPORT)
> +			return 1;
> +		vmx->msr_virtual_enumeration = data &
> +						VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
> +		break;
>   
>   	default:
>   	find_uret_msr:
> diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
> index c5a41ae14237..fc873cf45f70 100644
> --- a/arch/x86/kvm/vmx/vmx.h
> +++ b/arch/x86/kvm/vmx/vmx.h
> @@ -298,6 +298,7 @@ struct vcpu_vmx {
>   	 * IA32_SPEC_CTRL_MSR.
>   	 */
>   	u64		      spec_ctrl_mask;
> +	u64		      msr_virtual_enumeration;
>   	u32		      msr_ia32_umwait_control;
>   
>   	/*
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 2835bd796639..6be0a3f1281f 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -1532,6 +1532,8 @@ static const u32 emulated_msrs_all[] = {
>   	MSR_IA32_VMX_EPT_VPID_CAP,
>   	MSR_IA32_VMX_VMFUNC,
>   
> +	MSR_VIRTUAL_ENUMERATION,
> +
>   	MSR_K7_HWCR,
>   	MSR_KVM_POLL_CONTROL,
>   };
> @@ -1567,6 +1569,7 @@ static const u32 msr_based_features_all[] = {
>   	MSR_IA32_UCODE_REV,
>   	MSR_IA32_ARCH_CAPABILITIES,
>   	MSR_IA32_PERF_CAPABILITIES,
> +	MSR_VIRTUAL_ENUMERATION,
>   };
>   
>   static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all)];
> @@ -1588,7 +1591,8 @@ static unsigned int num_msr_based_features;
>   	 ARCH_CAP_SKIP_VMENTRY_L1DFLUSH | ARCH_CAP_SSB_NO | ARCH_CAP_MDS_NO | \
>   	 ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \
>   	 ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \
> -	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO)
> +	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | \
> +	 ARCH_CAP_VIRTUAL_ENUM)
>   
>   static u64 kvm_get_arch_capabilities(void)
>   {
> @@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
>   	 */
>   	data |= ARCH_CAP_PSCHANGE_MC_NO;
>   
> +	/*
> +	 * Virtual MSRs can allow guests to notify VMM whether or not
> +	 * they are using specific software mitigation, allowing a VMM
> +	 * to enable there hardware control only where necessary.
> +	 */
> +	data |= ARCH_CAP_VIRTUAL_ENUM;


IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires 
platform support.


> +
>   	/*
>   	 * If we're doing cache flushes (either "always" or "cond")
>   	 * we will do one whenever the guest does a vmlaunch/vmresume.
> @@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct kvm_msr_entry *msr)
>   	case MSR_IA32_UCODE_REV:
>   		rdmsrl_safe(msr->index, &msr->data);
>   		break;
> +	case MSR_VIRTUAL_ENUMERATION:
> +		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;


Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the 
feature.


> +		break;
>   	default:
>   		return static_call(kvm_x86_get_msr_feature)(msr);
>   	}

Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

[Sean Christopherson]

On Tue, Dec 20, 2022, Chao Gao wrote:
> On Mon, Dec 19, 2022 at 05:14:00PM +0000, Sean Christopherson wrote:
> >On Mon, Dec 19, 2022, Chao Gao wrote:
> >> On Wed, Dec 14, 2022 at 08:18:17PM +0000, Sean Christopherson wrote:
> >> > To me, this looks like Intel is foisting a paravirt interface on KVM and other
> >> > hypervisors without collaborating with said hypervisors' developers and maintainers.
> >> >
> >> >I get that some of the mitigations are vendor specific, but things like RETPOLINE
> >> >aren't vendor specific.  I haven't followed all of the mitigation stuff very
> >> >closely, but I wouldn't be surprised if there are mitigations now or in the future
> >> >that are common across architectures, e.g. arm64 and x86-64.  Intel doing its own
> >> >thing means AMD and arm64 will likely follow suit, and suddenly KVM is supporting
> >> >multiple paravirt interfaces for very similar things, without having any control
> >> >over the APIs.  That's all kinds of backwards.
> >> 
> >> But if the interface is defined by KVM rather than Intel, it will likely end up
> >> with different interfaces for different VMMs, then Linux guest needs to support
> >> all of them. And KVM needs to implement Hyper-V's and Xen's interface to support
> >> Hyper-V enlightened and Xen enlightened guest. This is a _real_ problem and
> >> complicates KVM/Linux in a similar way as multiple paravirt interfaces.
> >
> >I never said the PV interfaces should be defined by KVM.  I 100% agree that any
> >one hypervisor defining its own interface will suffer the same problem.
> 
> I am thinking there are only two options:
> 
> 1. Intel defines the interface.
> 2. Every VMM defines its own interface.
> 
> Any middle ground between the two options?

Work with other x86 hardware vendors to define a common interface?  Ask hypervisor
developers to define a common, extensible interface?

Maybe it turns out that it's impossible to abstract anything away and everything
ends up being vendor-specific anyways, but not even trying to provide a common
interace is extremely frustrating, especially since all this mitigation stuff has
been going on for ~5 years.  There's been plenty of time to establish relationships
and points of contact.

> >I think having a PV interface for coordinating mitigations between host and guest
> >is a great idea.  What I don't like is tying the interface to "hardware" and defining
> 
> Do you think something below is better than the intel-defined interface?

No, KVM doing its own thing would only exacerbate the issue.

> add a new KVM_CAP_* and a KVM_FEATURE_* in hypervisor CPUID leaf to enumerate
> the interface. And add a new virtual MSR 0x4b564dxx for guest to report in-use
> software mitigations and assign one bit for each software mitigation. On MSR
> write emulation, call into vmx code to enable some hardware mitigations if
> the corresponding software mitigations are not effective on host.
> 
> I am afraid it is late to switch to above approach because e.g., if Hyper-V
> decides to support the intel-defined interface, KVM probably needs to support it
> for Hyper-V guests.

Hence my frustration.

RE: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

[Zhang, Chen]

> -----Original Message-----
> From: Yang, Weijiang <weijiang.yang@intel.com>
> Sent: Wednesday, December 21, 2022 12:04 PM
> To: Zhang, Chen <chen.zhang@intel.com>
> Cc: Gao, Chao <chao.gao@intel.com>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Paolo Bonzini
> <pbonzini@redhat.com>; Christopherson,, Sean <seanjc@google.com>; H.
> Peter Anvin <hpa@zytor.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> Borislav Petkov <bp@alien8.de>; Ingo Molnar <mingo@redhat.com>; Thomas
> Gleixner <tglx@linutronix.de>; x86@kernel.org; linux-kernel@vger.kernel.org;
> kvm@vger.kernel.org
> Subject: Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for
> guest MSR_IA32_ARCH_CAPABILITIES
> 
> 
> On 12/11/2022 12:00 AM, Zhang Chen wrote:
> > Add the 63 bit in MSR_IA32_ARCH_CAPABILITIES for enable the virtual MSRs.
> > Virtual MSRs can allow guests to notify VMM whether or not they are
> > using specific software mitigation, allowing a VMM to enable there
> > hardware control only where necessary.
> > As Intel spec defination, expose virtual MSR for guest.
> > Make guest have ability to check virtual MSR 0x50000000.
> >
> > Signed-off-by: Zhang Chen <chen.zhang@intel.com>
> > ---
> >   arch/x86/kvm/vmx/vmx.c | 15 +++++++++++++++
> >   arch/x86/kvm/vmx/vmx.h |  1 +
> >   arch/x86/kvm/x86.c     | 16 +++++++++++++++-
> >   3 files changed, 31 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index
> > 407061b369b4..6ed6b743be0e 100644
> > --- a/arch/x86/kvm/vmx/vmx.c
> > +++ b/arch/x86/kvm/vmx/vmx.c
> > @@ -2001,6 +2001,12 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu,
> struct msr_data *msr_info)
> >   	case MSR_IA32_DEBUGCTLMSR:
> >   		msr_info->data = vmcs_read64(GUEST_IA32_DEBUGCTL);
> >   		break;
> > +	case MSR_VIRTUAL_ENUMERATION:
> > +		if (!msr_info->host_initiated &&
> > +		    !(vcpu->arch.arch_capabilities &
> ARCH_CAP_VIRTUAL_ENUM))
> > +			return 1;
> > +		msr_info->data = vmx->msr_virtual_enumeration;
> > +		break;
> >   	default:
> >   	find_uret_msr:
> >   		msr = vmx_find_uret_msr(vmx, msr_info->index); @@ -2375,6
> +2381,15
> > @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data
> *msr_info)
> >   		}
> >   		ret = kvm_set_msr_common(vcpu, msr_info);
> >   		break;
> > +	case MSR_VIRTUAL_ENUMERATION:
> > +		if (msr_info->host_initiated &&
> > +		    !(vcpu->arch.arch_capabilities &
> ARCH_CAP_VIRTUAL_ENUM))
> > +			return 1;
> > +		if (data & ~VIRT_ENUM_MITIGATION_CTRL_SUPPORT)
> > +			return 1;
> > +		vmx->msr_virtual_enumeration = data &
> > +
> 	VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
> > +		break;
> >
> >   	default:
> >   	find_uret_msr:
> > diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index
> > c5a41ae14237..fc873cf45f70 100644
> > --- a/arch/x86/kvm/vmx/vmx.h
> > +++ b/arch/x86/kvm/vmx/vmx.h
> > @@ -298,6 +298,7 @@ struct vcpu_vmx {
> >   	 * IA32_SPEC_CTRL_MSR.
> >   	 */
> >   	u64		      spec_ctrl_mask;
> > +	u64		      msr_virtual_enumeration;
> >   	u32		      msr_ia32_umwait_control;
> >
> >   	/*
> > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index
> > 2835bd796639..6be0a3f1281f 100644
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> > @@ -1532,6 +1532,8 @@ static const u32 emulated_msrs_all[] = {
> >   	MSR_IA32_VMX_EPT_VPID_CAP,
> >   	MSR_IA32_VMX_VMFUNC,
> >
> > +	MSR_VIRTUAL_ENUMERATION,
> > +
> >   	MSR_K7_HWCR,
> >   	MSR_KVM_POLL_CONTROL,
> >   };
> > @@ -1567,6 +1569,7 @@ static const u32 msr_based_features_all[] = {
> >   	MSR_IA32_UCODE_REV,
> >   	MSR_IA32_ARCH_CAPABILITIES,
> >   	MSR_IA32_PERF_CAPABILITIES,
> > +	MSR_VIRTUAL_ENUMERATION,
> >   };
> >
> >   static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all)];
> > @@ -1588,7 +1591,8 @@ static unsigned int num_msr_based_features;
> >   	 ARCH_CAP_SKIP_VMENTRY_L1DFLUSH | ARCH_CAP_SSB_NO |
> ARCH_CAP_MDS_NO | \
> >   	 ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR |
> ARCH_CAP_TAA_NO | \
> >   	 ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO |
> ARCH_CAP_PSDP_NO | \
> > -	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO)
> > +	 ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO |
> \
> > +	 ARCH_CAP_VIRTUAL_ENUM)
> >
> >   static u64 kvm_get_arch_capabilities(void)
> >   {
> > @@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
> >   	 */
> >   	data |= ARCH_CAP_PSCHANGE_MC_NO;
> >
> > +	/*
> > +	 * Virtual MSRs can allow guests to notify VMM whether or not
> > +	 * they are using specific software mitigation, allowing a VMM
> > +	 * to enable there hardware control only where necessary.
> > +	 */
> > +	data |= ARCH_CAP_VIRTUAL_ENUM;
> 
> 
> IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires
> platform support.

Intel defined the virtual MSRs for software mitigations for all platforms.
KVM should be unconditionally opened it for the software mitigation in migration pools.
For example migration from the old platform to the new platform.
Please check the Software Mitigations in Migration Pools section in documents:
https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html

> 
> 
> > +
> >   	/*
> >   	 * If we're doing cache flushes (either "always" or "cond")
> >   	 * we will do one whenever the guest does a vmlaunch/vmresume.
> > @@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct
> kvm_msr_entry *msr)
> >   	case MSR_IA32_UCODE_REV:
> >   		rdmsrl_safe(msr->index, &msr->data);
> >   		break;
> > +	case MSR_VIRTUAL_ENUMERATION:
> > +		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
> 
> 
> Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the
> feature.

Refer to the above comments.

Thanks
Chen

> 
> 
> > +		break;
> >   	default:
> >   		return static_call(kvm_x86_get_msr_feature)(msr);
> >   	}

Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

[Yang, Weijiang]

On 12/29/2022 10:58 AM, Zhang, Chen wrote:

[...]

>> \
>>> +	 ARCH_CAP_VIRTUAL_ENUM)
>>>
>>>    static u64 kvm_get_arch_capabilities(void)
>>>    {
>>> @@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
>>>    	 */
>>>    	data |= ARCH_CAP_PSCHANGE_MC_NO;
>>>
>>> +	/*
>>> +	 * Virtual MSRs can allow guests to notify VMM whether or not
>>> +	 * they are using specific software mitigation, allowing a VMM
>>> +	 * to enable there hardware control only where necessary.
>>> +	 */
>>> +	data |= ARCH_CAP_VIRTUAL_ENUM;
>>
>> IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires
>> platform support.
> Intel defined the virtual MSRs for software mitigations for all platforms.
> KVM should be unconditionally opened it for the software mitigation in migration pools.
> For example migration from the old platform to the new platform.
> Please check the Software Mitigations in Migration Pools section in documents:
> https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/branch-history-injection.html

If this series running on old platforms, how VMM can set specific vmcs 
fields,

e.g., “virtualize IA32_SPEC_CTRL” VM-execution control, to mitigate 
guest issues?


>
>>
>>> +
>>>    	/*
>>>    	 * If we're doing cache flushes (either "always" or "cond")
>>>    	 * we will do one whenever the guest does a vmlaunch/vmresume.
>>> @@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct
>> kvm_msr_entry *msr)
>>>    	case MSR_IA32_UCODE_REV:
>>>    		rdmsrl_safe(msr->index, &msr->data);
>>>    		break;
>>> +	case MSR_VIRTUAL_ENUMERATION:
>>> +		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
>>
>> Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the
>> feature.
> Refer to the above comments.
>
> Thanks
> Chen
>
>>
>>> +		break;
>>>    	default:
>>>    		return static_call(kvm_x86_get_msr_feature)(msr);
>>>    	}

RE: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

[Zhang, Chen]

> -----Original Message-----
> From: Yang, Weijiang <weijiang.yang@intel.com>
> Sent: Thursday, December 29, 2022 3:03 PM
> To: Zhang, Chen <chen.zhang@intel.com>
> Cc: Gao, Chao <chao.gao@intel.com>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Paolo Bonzini
> <pbonzini@redhat.com>; Christopherson,, Sean <seanjc@google.com>; H.
> Peter Anvin <hpa@zytor.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> Borislav Petkov <bp@alien8.de>; Ingo Molnar <mingo@redhat.com>; Thomas
> Gleixner <tglx@linutronix.de>; x86@kernel.org; linux-kernel@vger.kernel.org;
> kvm@vger.kernel.org
> Subject: Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for
> guest MSR_IA32_ARCH_CAPABILITIES
> 
> 
> On 12/29/2022 10:58 AM, Zhang, Chen wrote:
> 
> [...]
> 
> >> \
> >>> +	 ARCH_CAP_VIRTUAL_ENUM)
> >>>
> >>>    static u64 kvm_get_arch_capabilities(void)
> >>>    {
> >>> @@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
> >>>    	 */
> >>>    	data |= ARCH_CAP_PSCHANGE_MC_NO;
> >>>
> >>> +	/*
> >>> +	 * Virtual MSRs can allow guests to notify VMM whether or not
> >>> +	 * they are using specific software mitigation, allowing a VMM
> >>> +	 * to enable there hardware control only where necessary.
> >>> +	 */
> >>> +	data |= ARCH_CAP_VIRTUAL_ENUM;
> >>
> >> IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires
> >> platform support.
> > Intel defined the virtual MSRs for software mitigations for all platforms.
> > KVM should be unconditionally opened it for the software mitigation in
> migration pools.
> > For example migration from the old platform to the new platform.
> > Please check the Software Mitigations in Migration Pools section in
> documents:
> > https://www.intel.com/content/www/us/en/developer/articles/technical/s
> > oftware-security-guidance/technical-documentation/branch-history-injec
> > tion.html
> 
> If this series running on old platforms, how VMM can set specific vmcs fields,
> 
> e.g., “virtualize IA32_SPEC_CTRL” VM-execution control, to mitigate guest
> issues?

Enable the virtual MSRs does not means to enable the “virtualize IA32_SPEC_CTRL”.
KVM will check "cpu_has_virt_spec_ctrl()" before set specific VMCS.

Thanks
Chen

> 
> 
> >
> >>
> >>> +
> >>>    	/*
> >>>    	 * If we're doing cache flushes (either "always" or "cond")
> >>>    	 * we will do one whenever the guest does a vmlaunch/vmresume.
> >>> @@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct
> >> kvm_msr_entry *msr)
> >>>    	case MSR_IA32_UCODE_REV:
> >>>    		rdmsrl_safe(msr->index, &msr->data);
> >>>    		break;
> >>> +	case MSR_VIRTUAL_ENUMERATION:
> >>> +		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
> >>
> >> Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the
> >> feature.
> > Refer to the above comments.
> >
> > Thanks
> > Chen
> >
> >>
> >>> +		break;
> >>>    	default:
> >>>    		return static_call(kvm_x86_get_msr_feature)(msr);
> >>>    	}

Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

[Yang, Weijiang]

On 12/29/2022 3:41 PM, Zhang, Chen wrote:
>
>> -----Original Message-----
>> From: Yang, Weijiang <weijiang.yang@intel.com>
>> Sent: Thursday, December 29, 2022 3:03 PM
>> To: Zhang, Chen <chen.zhang@intel.com>
>> Cc: Gao, Chao <chao.gao@intel.com>; Pawan Gupta
>> <pawan.kumar.gupta@linux.intel.com>; Paolo Bonzini
>> <pbonzini@redhat.com>; Christopherson,, Sean <seanjc@google.com>; H.
>> Peter Anvin <hpa@zytor.com>; Dave Hansen <dave.hansen@linux.intel.com>;
>> Borislav Petkov <bp@alien8.de>; Ingo Molnar <mingo@redhat.com>; Thomas
>> Gleixner <tglx@linutronix.de>; x86@kernel.org; linux-kernel@vger.kernel.org;
>> kvm@vger.kernel.org
>> Subject: Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for
>> guest MSR_IA32_ARCH_CAPABILITIES
>>
>>
>> On 12/29/2022 10:58 AM, Zhang, Chen wrote:
>>
>> [...]
>>
>>>> \
>>>>> +	 ARCH_CAP_VIRTUAL_ENUM)
>>>>>
>>>>>     static u64 kvm_get_arch_capabilities(void)
>>>>>     {
>>>>> @@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
>>>>>     	 */
>>>>>     	data |= ARCH_CAP_PSCHANGE_MC_NO;
>>>>>
>>>>> +	/*
>>>>> +	 * Virtual MSRs can allow guests to notify VMM whether or not
>>>>> +	 * they are using specific software mitigation, allowing a VMM
>>>>> +	 * to enable there hardware control only where necessary.
>>>>> +	 */
>>>>> +	data |= ARCH_CAP_VIRTUAL_ENUM;
>>>> IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires
>>>> platform support.
>>> Intel defined the virtual MSRs for software mitigations for all platforms.
>>> KVM should be unconditionally opened it for the software mitigation in
>> migration pools.
>>> For example migration from the old platform to the new platform.
>>> Please check the Software Mitigations in Migration Pools section in
>> documents:
>>> https://www.intel.com/content/www/us/en/developer/articles/technical/s
>>> oftware-security-guidance/technical-documentation/branch-history-injec
>>> tion.html
>> If this series running on old platforms, how VMM can set specific vmcs fields,
>>
>> e.g., “virtualize IA32_SPEC_CTRL” VM-execution control, to mitigate guest
>> issues?
> Enable the virtual MSRs does not means to enable the “virtualize IA32_SPEC_CTRL”.
> KVM will check "cpu_has_virt_spec_ctrl()" before set specific VMCS.


What I'm concerned is, if the feature is not supported on some 
platforms, unconditionally

exposing to guest would make it wrongly "think"  the feature is 
supported on HW, but actually

WMM can do nothing to mitigate issues. Read through the patch set, looks 
like guest cannot

even get any errors during feature configuration in this case. So I 
suggest add more

sanity checks. Did I  miss something?


>
> Thanks
> Chen
>
>>
>>>>> +
>>>>>     	/*
>>>>>     	 * If we're doing cache flushes (either "always" or "cond")
>>>>>     	 * we will do one whenever the guest does a vmlaunch/vmresume.
>>>>> @@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct
>>>> kvm_msr_entry *msr)
>>>>>     	case MSR_IA32_UCODE_REV:
>>>>>     		rdmsrl_safe(msr->index, &msr->data);
>>>>>     		break;
>>>>> +	case MSR_VIRTUAL_ENUMERATION:
>>>>> +		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
>>>> Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose the
>>>> feature.
>>> Refer to the above comments.
>>>
>>> Thanks
>>> Chen
>>>
>>>>> +		break;
>>>>>     	default:
>>>>>     		return static_call(kvm_x86_get_msr_feature)(msr);
>>>>>     	}

RE: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for guest MSR_IA32_ARCH_CAPABILITIES

[Zhang, Chen]

> -----Original Message-----
> From: Yang, Weijiang <weijiang.yang@intel.com>
> Sent: Thursday, December 29, 2022 4:39 PM
> To: Zhang, Chen <chen.zhang@intel.com>
> Cc: Gao, Chao <chao.gao@intel.com>; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Paolo Bonzini
> <pbonzini@redhat.com>; Christopherson,, Sean <seanjc@google.com>; H.
> Peter Anvin <hpa@zytor.com>; Dave Hansen <dave.hansen@linux.intel.com>;
> Borislav Petkov <bp@alien8.de>; Ingo Molnar <mingo@redhat.com>; Thomas
> Gleixner <tglx@linutronix.de>; x86@kernel.org; linux-kernel@vger.kernel.org;
> kvm@vger.kernel.org
> Subject: Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for
> guest MSR_IA32_ARCH_CAPABILITIES
> 
> 
> On 12/29/2022 3:41 PM, Zhang, Chen wrote:
> >
> >> -----Original Message-----
> >> From: Yang, Weijiang <weijiang.yang@intel.com>
> >> Sent: Thursday, December 29, 2022 3:03 PM
> >> To: Zhang, Chen <chen.zhang@intel.com>
> >> Cc: Gao, Chao <chao.gao@intel.com>; Pawan Gupta
> >> <pawan.kumar.gupta@linux.intel.com>; Paolo Bonzini
> >> <pbonzini@redhat.com>; Christopherson,, Sean <seanjc@google.com>; H.
> >> Peter Anvin <hpa@zytor.com>; Dave Hansen
> >> <dave.hansen@linux.intel.com>; Borislav Petkov <bp@alien8.de>; Ingo
> >> Molnar <mingo@redhat.com>; Thomas Gleixner <tglx@linutronix.de>;
> >> x86@kernel.org; linux-kernel@vger.kernel.org; kvm@vger.kernel.org
> >> Subject: Re: [RFC PATCH 6/9] kvm/x86: Add ARCH_CAP_VIRTUAL_ENUM for
> >> guest MSR_IA32_ARCH_CAPABILITIES
> >>
> >>
> >> On 12/29/2022 10:58 AM, Zhang, Chen wrote:
> >>
> >> [...]
> >>
> >>>> \
> >>>>> +	 ARCH_CAP_VIRTUAL_ENUM)
> >>>>>
> >>>>>     static u64 kvm_get_arch_capabilities(void)
> >>>>>     {
> >>>>> @@ -1607,6 +1611,13 @@ static u64 kvm_get_arch_capabilities(void)
> >>>>>     	 */
> >>>>>     	data |= ARCH_CAP_PSCHANGE_MC_NO;
> >>>>>
> >>>>> +	/*
> >>>>> +	 * Virtual MSRs can allow guests to notify VMM whether or not
> >>>>> +	 * they are using specific software mitigation, allowing a VMM
> >>>>> +	 * to enable there hardware control only where necessary.
> >>>>> +	 */
> >>>>> +	data |= ARCH_CAP_VIRTUAL_ENUM;
> >>>> IMO, this is:  data &= ARCH_CAP_VIRTUAL_ENUM; because it requires
> >>>> platform support.
> >>> Intel defined the virtual MSRs for software mitigations for all platforms.
> >>> KVM should be unconditionally opened it for the software mitigation
> >>> in
> >> migration pools.
> >>> For example migration from the old platform to the new platform.
> >>> Please check the Software Mitigations in Migration Pools section in
> >> documents:
> >>> https://www.intel.com/content/www/us/en/developer/articles/technical
> >>> /s
> >>> oftware-security-guidance/technical-documentation/branch-history-inj
> >>> ec
> >>> tion.html
> >> If this series running on old platforms, how VMM can set specific
> >> vmcs fields,
> >>
> >> e.g., “virtualize IA32_SPEC_CTRL” VM-execution control, to mitigate
> >> guest issues?
> > Enable the virtual MSRs does not means to enable the “virtualize
> IA32_SPEC_CTRL”.
> > KVM will check "cpu_has_virt_spec_ctrl()" before set specific VMCS.
> 
> 
> What I'm concerned is, if the feature is not supported on some platforms,
> unconditionally
> 
> exposing to guest would make it wrongly "think"  the feature is supported on
> HW, but actually
> 
> WMM can do nothing to mitigate issues. Read through the patch set, looks like
> guest cannot
> 
> even get any errors during feature configuration in this case. So I suggest add
> more
> 
> sanity checks. Did I  miss something?

Understand your concern, but I think no need to concern about it.
Although we have unconditionally turned on virtual MSRs in all platforms,
The detailed virtual MSR bits still can tell guest what feature is supported on HW.
QEMU will setup the MSR_VIRTUAL_MITIGATION_ENUM(0x50000001) bits
Like the BHB_CLEAR_SEQ_S_SUPPORT/RETPOLINE_S_SUPPORT before boot VMs.

Thanks
Chen

> 
> 
> >
> > Thanks
> > Chen
> >
> >>
> >>>>> +
> >>>>>     	/*
> >>>>>     	 * If we're doing cache flushes (either "always" or "cond")
> >>>>>     	 * we will do one whenever the guest does a
> vmlaunch/vmresume.
> >>>>> @@ -1657,6 +1668,9 @@ static int kvm_get_msr_feature(struct
> >>>> kvm_msr_entry *msr)
> >>>>>     	case MSR_IA32_UCODE_REV:
> >>>>>     		rdmsrl_safe(msr->index, &msr->data);
> >>>>>     		break;
> >>>>> +	case MSR_VIRTUAL_ENUMERATION:
> >>>>> +		msr->data = VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
> >>>> Need to check bit 63 of host MSR_ARCH_CAPABILITIES before expose
> >>>> the feature.
> >>> Refer to the above comments.
> >>>
> >>> Thanks
> >>> Chen
> >>>
> >>>>> +		break;
> >>>>>     	default:
> >>>>>     		return static_call(kvm_x86_get_msr_feature)(msr);
> >>>>>     	}

RE: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware mitigations

[Zhang, Chen]

> -----Original Message-----
> From: Sean Christopherson <seanjc@google.com>
> Sent: Friday, December 23, 2022 2:32 AM
> To: Gao, Chao <chao.gao@intel.com>
> Cc: Zhang, Chen <chen.zhang@intel.com>; x86@kernel.org; linux-
> kernel@vger.kernel.org; kvm@vger.kernel.org; Pawan Gupta
> <pawan.kumar.gupta@linux.intel.com>; Paolo Bonzini
> <pbonzini@redhat.com>; H. Peter Anvin <hpa@zytor.com>; Dave Hansen
> <dave.hansen@linux.intel.com>; Borislav Petkov <bp@alien8.de>; Ingo
> Molnar <mingo@redhat.com>; Thomas Gleixner <tglx@linutronix.de>
> Subject: Re: [RFC PATCH 5/9] x86/bugs: Use Virtual MSRs to request hardware
> mitigations
> 
> On Tue, Dec 20, 2022, Chao Gao wrote:
> > On Mon, Dec 19, 2022 at 05:14:00PM +0000, Sean Christopherson wrote:
> > >On Mon, Dec 19, 2022, Chao Gao wrote:
> > >> On Wed, Dec 14, 2022 at 08:18:17PM +0000, Sean Christopherson wrote:
> > >> > To me, this looks like Intel is foisting a paravirt interface on
> > >> > KVM and other hypervisors without collaborating with said hypervisors'
> developers and maintainers.
> > >> >
> > >> >I get that some of the mitigations are vendor specific, but things
> > >> >like RETPOLINE aren't vendor specific.  I haven't followed all of
> > >> >the mitigation stuff very closely, but I wouldn't be surprised if
> > >> >there are mitigations now or in the future that are common across
> > >> >architectures, e.g. arm64 and x86-64.  Intel doing its own thing
> > >> >means AMD and arm64 will likely follow suit, and suddenly KVM is
> > >> >supporting multiple paravirt interfaces for very similar things, without
> having any control over the APIs.  That's all kinds of backwards.
> > >>
> > >> But if the interface is defined by KVM rather than Intel, it will
> > >> likely end up with different interfaces for different VMMs, then
> > >> Linux guest needs to support all of them. And KVM needs to
> > >> implement Hyper-V's and Xen's interface to support Hyper-V
> > >> enlightened and Xen enlightened guest. This is a _real_ problem and
> complicates KVM/Linux in a similar way as multiple paravirt interfaces.
> > >
> > >I never said the PV interfaces should be defined by KVM.  I 100%
> > >agree that any one hypervisor defining its own interface will suffer the same
> problem.
> >
> > I am thinking there are only two options:
> >
> > 1. Intel defines the interface.
> > 2. Every VMM defines its own interface.
> >
> > Any middle ground between the two options?
> 
> Work with other x86 hardware vendors to define a common interface?  Ask
> hypervisor developers to define a common, extensible interface?
> 
> Maybe it turns out that it's impossible to abstract anything away and
> everything ends up being vendor-specific anyways, but not even trying to
> provide a common interace is extremely frustrating, especially since all this
> mitigation stuff has been going on for ~5 years.  There's been plenty of time to
> establish relationships and points of contact.
> 
> > >I think having a PV interface for coordinating mitigations between
> > >host and guest is a great idea.  What I don't like is tying the
> > >interface to "hardware" and defining
> >
> > Do you think something below is better than the intel-defined interface?
> 
> No, KVM doing its own thing would only exacerbate the issue.

Hi Sean,

Rethink about it and synced with Chao, we didn't think of a better solution than
Intel-defined interface. If you have no more suggestions/comments here,
please review the rest of patches from this series when you have time.
I will try to address comments and push this series to RFC V2.

Thanks
Chen

> 
> > add a new KVM_CAP_* and a KVM_FEATURE_* in hypervisor CPUID leaf to
> > enumerate the interface. And add a new virtual MSR 0x4b564dxx for
> > guest to report in-use software mitigations and assign one bit for
> > each software mitigation. On MSR write emulation, call into vmx code
> > to enable some hardware mitigations if the corresponding software
> mitigations are not effective on host.
> >
> > I am afraid it is late to switch to above approach because e.g., if
> > Hyper-V decides to support the intel-defined interface, KVM probably
> > needs to support it for Hyper-V guests.
> 
> Hence my frustration.

Re: [RFC PATCH 8/9] x86/kvm/vmx: Initialize SPEC_CTRL MASK for RRSBA

[Chao Gao]

On Sun, Dec 11, 2022 at 12:00:45AM +0800, Zhang Chen wrote:
>VMMs can address mitigations issues in migration pool by applying
>the needed controls whenever the guest is operating on a newer
>processor. If a guest is using retpoline to mitigate intra-mode BTI
>in CPL0, the VMM can set RRSBA_DIS_S when the guest runs on hardware
>which enumerates RRSBA.
>
>Signed-off-by: Zhang Chen <chen.zhang@intel.com>
>---
> arch/x86/kvm/vmx/vmx.c | 57 +++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 56 insertions(+), 1 deletion(-)
>
>diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
>index 6ed6b743be0e..fb0f3b1639b9 100644
>--- a/arch/x86/kvm/vmx/vmx.c
>+++ b/arch/x86/kvm/vmx/vmx.c
>@@ -2007,6 +2007,20 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
> 			return 1;
> 		msr_info->data = vmx->msr_virtual_enumeration;
> 		break;
>+	case MSR_VIRTUAL_MITIGATION_ENUM:
>+		if (!msr_info->host_initiated &&
>+		    !(vmx->msr_virtual_enumeration &
>+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
>+			return 1;
>+		msr_info->data = vmx->msr_virtual_mitigation_enum;
>+		break;
>+	case MSR_VIRTUAL_MITIGATION_CTRL:
>+		if (!msr_info->host_initiated &&
>+		    !(vmx->msr_virtual_enumeration &
>+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
>+			return 1;
>+		msr_info->data = vmx->msr_virtual_mitigation_ctrl;
>+		break;
> 	default:
> 	find_uret_msr:
> 		msr = vmx_find_uret_msr(vmx, msr_info->index);
>@@ -2056,7 +2070,7 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
> 	struct vmx_uret_msr *msr;
> 	int ret = 0;
> 	u32 msr_index = msr_info->index;
>-	u64 data = msr_info->data;
>+	u64 data = msr_info->data, arch_msr;
> 	u32 index;
> 
> 	switch (msr_index) {
>@@ -2390,6 +2404,46 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
> 		vmx->msr_virtual_enumeration = data &
> 						VIRT_ENUM_MITIGATION_CTRL_SUPPORT;
> 		break;
>+	case MSR_VIRTUAL_MITIGATION_ENUM:
>+		if (msr_info->host_initiated &&
>+		    !(vmx->msr_virtual_enumeration &
>+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
>+			return 1;
>+		if (data & ~MITI_ENUM_SUPPORTED)
>+			return 1;
>+		vmx->msr_virtual_mitigation_enum = data;
>+		break;
>+	case MSR_VIRTUAL_MITIGATION_CTRL:
>+		if (!msr_info->host_initiated &&
>+		    !(vmx->msr_virtual_enumeration &
>+		    VIRT_ENUM_MITIGATION_CTRL_SUPPORT))
>+			return 1;
>+		if (data & ~MITI_CTRL_USED)
>+			return 1;
>+


>+		if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES))
>+			rdmsrl(MSR_IA32_ARCH_CAPABILITIES, arch_msr);
>+
>+		if (data & MITI_CTRL_RETPOLINE_S_USED &&
>+		    boot_cpu_has(X86_FEATURE_RRSBA_CTRL) &&
>+		    arch_msr & ARCH_CAP_RRSBA)
>+			vmx->spec_ctrl_mask |= SPEC_CTRL_RRSBA_DIS_S;
>+		else
>+			vmx->spec_ctrl_mask &= ~SPEC_CTRL_RRSBA_DIS_S;

IIUC, only above chunk belongs to this patch. Other changes should be moved
to patch 7, which deals with MSRs emulation.