[RFC PATCH v2 0/6] SVM guest shadow stack support

[RFC PATCH v2 0/6] SVM guest shadow stack support

[John Allen]

AMD Zen3 and newer processors support shadow stack, a feature designed to
protect against ROP (return-oriented programming) attacks in which an attacker
manipulates return addresses on the call stack in order to execute arbitrary
code. To prevent this, shadow stacks can be allocated that are only used by
control transfer and return instructions. When a CALL instruction is issued, it
writes the return address to both the program stack and the shadow stack. When
the subsequent RET instruction is issued, it pops the return address from both
stacks and compares them. If the addresses don't match, a control-protection
exception is raised.

Shadow stack and a related feature, Indirect Branch Tracking (IBT), are
collectively referred to as Control-flow Enforcement Technology (CET). However,
current AMD processors only support shadow stack and not IBT.

This series adds support for shadow stack in SVM guests and builds upon
the support added in the CET guest support patch series [1]. Additional
patches are required to support shadow stack enabled guests in qemu [2]
and glibc [3].

[1]: CET guest support patches
https://lore.kernel.org/all/20230511040857.6094-1-weijiang.yang@intel.com/

[2]: CET qemu patches
https://patchwork.ozlabs.org/project/qemu-devel/patch/20201013051935.6052-2-weijiang.yang@intel.com/

[3]: glibc tree containing necessary updates
https://gitlab.com/x86-glibc/glibc/-/tree/users/hjl/cet/master/

---

v2:
  - Rebased on v3 of the Intel CET virtualization series, dropping the
    patch that moved cet_is_msr_accessible to common code as that has
    been pulled into the Intel series.
  - Minor change removing curly brackets around if statement introduced
    in patch 6/6.

---

Note to maintainers:

This series is split into two parts. This series pertains to KVM and
should apply to the KVM tree. Another patch which I will be submitting
separately entitled, "[RFC PATCH v2] x86/sev-es: Include XSS value in
GHCB CPUID request", pertains to the guest kernel and should apply to
the tip tree. However, I realized that both series depend on the
following line from patch 5/6 in this series:

diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index e7c7379d6ac7..8f91376273e0 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -677,5 +677,6 @@ DEFINE_GHCB_ACCESSORS(sw_exit_info_1)
 DEFINE_GHCB_ACCESSORS(sw_exit_info_2)
 DEFINE_GHCB_ACCESSORS(sw_scratch)
 DEFINE_GHCB_ACCESSORS(xcr0)
+DEFINE_GHCB_ACCESSORS(xss)

 #endif

Please advise on how to resolve this situation.

Thanks,
John

John Allen (6):
  KVM: x86: SVM: Emulate reads and writes to shadow stack MSRs
  KVM: x86: SVM: Update dump_vmcb with shadow stack save area additions
  KVM: x86: SVM: Pass through shadow stack MSRs
  KVM: SVM: Save shadow stack host state on VMRUN
  KVM: SVM: Add MSR_IA32_XSS to the GHCB for hypervisor kernel
  KVM: SVM: Add CET features to supported_xss

 arch/x86/include/asm/svm.h |  1 +
 arch/x86/kvm/svm/sev.c     | 25 +++++++++++-
 arch/x86/kvm/svm/svm.c     | 84 ++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/svm/svm.h     |  2 +-
 4 files changed, 109 insertions(+), 3 deletions(-)

-- 
2.39.1

[RFC PATCH v2 1/6] KVM: x86: SVM: Emulate reads and writes to shadow stack MSRs

[John Allen]

Set up interception of shadow stack MSRs. In the event that shadow stack
is unsupported on the host or the MSRs are otherwise inaccessible, the
interception code will return an error. In certain circumstances such as
host initiated MSR reads or writes, the interception code will get or
set the requested MSR value.

Signed-off-by: John Allen <john.allen@amd.com>
---
 arch/x86/kvm/svm/svm.c | 58 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index eb308c9994f9..822d7a65e92a 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2800,6 +2800,31 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		if (guest_cpuid_is_intel(vcpu))
 			msr_info->data |= (u64)svm->sysenter_esp_hi << 32;
 		break;
+	case MSR_IA32_S_CET:
+		if (!kvm_cet_is_msr_accessible(vcpu, msr_info))
+			return 1;
+		msr_info->data = svm->vmcb->save.s_cet;
+		break;
+	case MSR_IA32_U_CET:
+		if (!kvm_cet_is_msr_accessible(vcpu, msr_info))
+			return 1;
+		kvm_get_xsave_msr(msr_info);
+		break;
+	case MSR_IA32_INT_SSP_TAB:
+		if (!kvm_cet_is_msr_accessible(vcpu, msr_info))
+			return 1;
+		msr_info->data = svm->vmcb->save.isst_addr;
+		break;
+	case MSR_KVM_GUEST_SSP:
+		if (!kvm_cet_is_msr_accessible(vcpu, msr_info))
+			return 1;
+		msr_info->data = svm->vmcb->save.ssp;
+		break;
+	case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP:
+		if (!kvm_cet_is_msr_accessible(vcpu, msr_info))
+			return 1;
+		kvm_get_xsave_msr(msr_info);
+		break;
 	case MSR_TSC_AUX:
 		msr_info->data = svm->tsc_aux;
 		break;
@@ -3016,6 +3041,39 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 		svm->vmcb01.ptr->save.sysenter_esp = (u32)data;
 		svm->sysenter_esp_hi = guest_cpuid_is_intel(vcpu) ? (data >> 32) : 0;
 		break;
+	case MSR_IA32_S_CET:
+		if (!kvm_cet_is_msr_accessible(vcpu, msr))
+			return 1;
+		svm->vmcb->save.s_cet = data;
+		break;
+	case MSR_IA32_U_CET:
+		if (!kvm_cet_is_msr_accessible(vcpu, msr))
+			return 1;
+		kvm_set_xsave_msr(msr);
+		break;
+	case MSR_IA32_INT_SSP_TAB:
+		if (!kvm_cet_is_msr_accessible(vcpu, msr))
+			return 1;
+		if (is_noncanonical_address(data, vcpu))
+			return 1;
+		svm->vmcb->save.isst_addr = data;
+		break;
+	case MSR_KVM_GUEST_SSP:
+		if (!kvm_cet_is_msr_accessible(vcpu, msr))
+			return 1;
+		/* SSP MSR values should be a 4-byte aligned canonical addresses */
+		if ((data & GENMASK(1, 0)) || is_noncanonical_address(data, vcpu))
+			return 1;
+		svm->vmcb->save.ssp = data;
+		break;
+	case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP:
+		if (!kvm_cet_is_msr_accessible(vcpu, msr))
+			return 1;
+		/* SSP MSR values should be a 4-byte aligned canonical addresses */
+		if ((data & GENMASK(1, 0)) || is_noncanonical_address(data, vcpu))
+			return 1;
+		kvm_set_xsave_msr(msr);
+		break;
 	case MSR_TSC_AUX:
 		/*
 		 * TSC_AUX is usually changed only during boot and never read
-- 
2.39.1

[RFC PATCH v2 2/6] KVM: x86: SVM: Update dump_vmcb with shadow stack save area additions

[John Allen]

Add shadow stack VMCB save area fields to dump_vmcb. Only include S_CET,
SSP, and ISST_ADDR. Since there currently isn't support to decrypt and
dump the SEV-ES save area, exclude PL0_SSP, PL1_SSP, PL2_SSP, PL3_SSP, and
U_CET which are only inlcuded in the SEV-ES save area.

Signed-off-by: John Allen <john.allen@amd.com>
---
 arch/x86/kvm/svm/svm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 822d7a65e92a..6df486bb1ac4 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3392,6 +3392,10 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
 	       "rip:", save->rip, "rflags:", save->rflags);
 	pr_err("%-15s %016llx %-13s %016llx\n",
 	       "rsp:", save->rsp, "rax:", save->rax);
+	pr_err("%-15s %016llx %-13s %016llx\n",
+	       "s_cet:", save->s_cet, "ssp:", save->ssp);
+	pr_err("%-15s %016llx\n",
+	       "isst_addr:", save->isst_addr);
 	pr_err("%-15s %016llx %-13s %016llx\n",
 	       "star:", save01->star, "lstar:", save01->lstar);
 	pr_err("%-15s %016llx %-13s %016llx\n",
-- 
2.39.1

[RFC PATCH v2 3/6] KVM: x86: SVM: Pass through shadow stack MSRs

[John Allen]

If kvm supports shadow stack, pass through shadow stack MSRs to improve
guest performance.

Signed-off-by: John Allen <john.allen@amd.com>
---
 arch/x86/kvm/svm/svm.c | 17 +++++++++++++++++
 arch/x86/kvm/svm/svm.h |  2 +-
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 6df486bb1ac4..cdbce20989b8 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -136,6 +136,13 @@ static const struct svm_direct_access_msrs {
 	{ .index = X2APIC_MSR(APIC_TMICT),		.always = false },
 	{ .index = X2APIC_MSR(APIC_TMCCT),		.always = false },
 	{ .index = X2APIC_MSR(APIC_TDCR),		.always = false },
+	{ .index = MSR_IA32_U_CET,                      .always = false },
+	{ .index = MSR_IA32_S_CET,                      .always = false },
+	{ .index = MSR_IA32_INT_SSP_TAB,                .always = false },
+	{ .index = MSR_IA32_PL0_SSP,                    .always = false },
+	{ .index = MSR_IA32_PL1_SSP,                    .always = false },
+	{ .index = MSR_IA32_PL2_SSP,                    .always = false },
+	{ .index = MSR_IA32_PL3_SSP,                    .always = false },
 	{ .index = MSR_INVALID,				.always = false },
 };
 
@@ -1181,6 +1188,16 @@ static inline void init_vmcb_after_set_cpuid(struct kvm_vcpu *vcpu)
 		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_EIP, 1, 1);
 		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_ESP, 1, 1);
 	}
+
+	if (kvm_cet_user_supported() && guest_cpuid_has(vcpu, X86_FEATURE_SHSTK)) {
+		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_U_CET, 1, 1);
+		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_S_CET, 1, 1);
+		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_INT_SSP_TAB, 1, 1);
+		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL0_SSP, 1, 1);
+		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL1_SSP, 1, 1);
+		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL2_SSP, 1, 1);
+		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL3_SSP, 1, 1);
+	}
 }
 
 static void init_vmcb(struct kvm_vcpu *vcpu)
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index f44751dd8d5d..dad977747a15 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -29,7 +29,7 @@
 #define	IOPM_SIZE PAGE_SIZE * 3
 #define	MSRPM_SIZE PAGE_SIZE * 2
 
-#define MAX_DIRECT_ACCESS_MSRS	46
+#define MAX_DIRECT_ACCESS_MSRS	53
 #define MSRPM_OFFSETS	32
 extern u32 msrpm_offsets[MSRPM_OFFSETS] __read_mostly;
 extern bool npt_enabled;
-- 
2.39.1

[RFC PATCH v2 4/6] KVM: SVM: Save shadow stack host state on VMRUN

[John Allen]

When running as an SEV-ES guest, the PL0_SSP, PL1_SSP, PL2_SSP, PL3_SSP,
and U_CET fields in the VMCB save area are type B, meaning the host
state is automatically loaded on a VMEXIT, but is not saved on a VMRUN.
The other shadow stack MSRs, S_CET, SSP, and ISST_ADDR are type A,
meaning they are loaded on VMEXIT and saved on VMRUN. Manually save the
type B host MSR values before VMRUN.

Signed-off-by: John Allen <john.allen@amd.com>
---
 arch/x86/kvm/svm/sev.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index c25aeb550cd9..03dd68bddd51 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3028,6 +3028,19 @@ void sev_es_prepare_switch_to_guest(struct sev_es_save_area *hostsa)
 
 	/* MSR_IA32_XSS is restored on VMEXIT, save the currnet host value */
 	hostsa->xss = host_xss;
+
+	if (boot_cpu_has(X86_FEATURE_SHSTK)) {
+		/*
+		 * MSR_IA32_U_CET, MSR_IA32_PL0_SSP, MSR_IA32_PL1_SSP,
+		 * MSR_IA32_PL2_SSP, and MSR_IA32_PL3_SSP are restored on
+		 * VMEXIT, save the current host values.
+		 */
+		rdmsrl(MSR_IA32_U_CET, hostsa->u_cet);
+		rdmsrl(MSR_IA32_PL0_SSP, hostsa->vmpl0_ssp);
+		rdmsrl(MSR_IA32_PL1_SSP, hostsa->vmpl1_ssp);
+		rdmsrl(MSR_IA32_PL2_SSP, hostsa->vmpl2_ssp);
+		rdmsrl(MSR_IA32_PL3_SSP, hostsa->vmpl3_ssp);
+	}
 }
 
 void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector)
-- 
2.39.1

[RFC PATCH v2 5/6] KVM: SVM: Add MSR_IA32_XSS to the GHCB for hypervisor kernel

[John Allen]

When a guest issues a cpuid instruction for Fn0000000D_x0B
(CetUserOffset), KVM will intercept and need to access the guest
MSR_IA32_XSS value. For SEV-ES, this is encrypted and needs to be
included in the GHCB to be visible to the hypervisor.

Signed-off-by: John Allen <john.allen@amd.com>
---
 arch/x86/include/asm/svm.h |  1 +
 arch/x86/kvm/svm/sev.c     | 12 ++++++++++--
 arch/x86/kvm/svm/svm.c     |  1 +
 arch/x86/kvm/svm/svm.h     |  2 +-
 4 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index e7c7379d6ac7..8f91376273e0 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -677,5 +677,6 @@ DEFINE_GHCB_ACCESSORS(sw_exit_info_1)
 DEFINE_GHCB_ACCESSORS(sw_exit_info_2)
 DEFINE_GHCB_ACCESSORS(sw_scratch)
 DEFINE_GHCB_ACCESSORS(xcr0)
+DEFINE_GHCB_ACCESSORS(xss)
 
 #endif
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 03dd68bddd51..92a7c77bc66b 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -2419,8 +2419,13 @@ static void sev_es_sync_from_ghcb(struct vcpu_svm *svm)
 
 	svm->vmcb->save.cpl = ghcb_get_cpl_if_valid(ghcb);
 
-	if (ghcb_xcr0_is_valid(ghcb)) {
-		vcpu->arch.xcr0 = ghcb_get_xcr0(ghcb);
+	if (ghcb_xcr0_is_valid(ghcb) || ghcb_xss_is_valid(ghcb)) {
+		if (ghcb_xcr0_is_valid(ghcb))
+			vcpu->arch.xcr0 = ghcb_get_xcr0(ghcb);
+
+		if (ghcb_xss_is_valid(ghcb))
+			vcpu->arch.ia32_xss = ghcb_get_xss(ghcb);
+
 		kvm_update_cpuid_runtime(vcpu);
 	}
 
@@ -2989,6 +2994,9 @@ static void sev_es_init_vmcb(struct vcpu_svm *svm)
 		if (guest_cpuid_has(&svm->vcpu, X86_FEATURE_RDTSCP))
 			svm_clr_intercept(svm, INTERCEPT_RDTSCP);
 	}
+
+	if (kvm_caps.supported_xss)
+		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_XSS, 1, 1);
 }
 
 void sev_init_vmcb(struct vcpu_svm *svm)
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index cdbce20989b8..6afd2c44fdb6 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -143,6 +143,7 @@ static const struct svm_direct_access_msrs {
 	{ .index = MSR_IA32_PL1_SSP,                    .always = false },
 	{ .index = MSR_IA32_PL2_SSP,                    .always = false },
 	{ .index = MSR_IA32_PL3_SSP,                    .always = false },
+	{ .index = MSR_IA32_XSS,                        .always = false },
 	{ .index = MSR_INVALID,				.always = false },
 };
 
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index dad977747a15..92ac1aefe640 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -29,7 +29,7 @@
 #define	IOPM_SIZE PAGE_SIZE * 3
 #define	MSRPM_SIZE PAGE_SIZE * 2
 
-#define MAX_DIRECT_ACCESS_MSRS	53
+#define MAX_DIRECT_ACCESS_MSRS	54
 #define MSRPM_OFFSETS	32
 extern u32 msrpm_offsets[MSRPM_OFFSETS] __read_mostly;
 extern bool npt_enabled;
-- 
2.39.1

[RFC PATCH v2 6/6] KVM: SVM: Add CET features to supported_xss

[John Allen]

If the CPU supports CET, add CET XSAVES feature bits to the
supported_xss mask.

Signed-off-by: John Allen <john.allen@amd.com>
---
v2:
  - Remove curly braces around if statement
---
 arch/x86/kvm/svm/svm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 6afd2c44fdb6..cee496bee0a9 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5070,6 +5070,10 @@ static __init void svm_set_cpu_caps(void)
 	    boot_cpu_has(X86_FEATURE_AMD_SSBD))
 		kvm_cpu_cap_set(X86_FEATURE_VIRT_SSBD);
 
+	if (kvm_cpu_cap_has(X86_FEATURE_SHSTK))
+		kvm_caps.supported_xss |= XFEATURE_MASK_CET_USER |
+					  XFEATURE_MASK_CET_KERNEL;
+
 	/* AMD PMU PERFCTR_CORE CPUID */
 	if (enable_pmu && boot_cpu_has(X86_FEATURE_PERFCTR_CORE))
 		kvm_cpu_cap_set(X86_FEATURE_PERFCTR_CORE);
-- 
2.39.1

Re: [RFC PATCH v2 6/6] KVM: SVM: Add CET features to supported_xss

[Edgecombe, Rick P]

On Wed, 2023-05-24 at 15:53 +0000, John Allen wrote:
> If the CPU supports CET, add CET XSAVES feature bits to the
> supported_xss mask.
> 
> Signed-off-by: John Allen <john.allen@amd.com>
> ---
> v2:
>   - Remove curly braces around if statement
> ---
>  arch/x86/kvm/svm/svm.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 6afd2c44fdb6..cee496bee0a9 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -5070,6 +5070,10 @@ static __init void svm_set_cpu_caps(void)
>             boot_cpu_has(X86_FEATURE_AMD_SSBD))
>                 kvm_cpu_cap_set(X86_FEATURE_VIRT_SSBD);
>  
> +       if (kvm_cpu_cap_has(X86_FEATURE_SHSTK))
> +               kvm_caps.supported_xss |= XFEATURE_MASK_CET_USER |
> +                                         XFEATURE_MASK_CET_KERNEL;
> +

Is setting XFEATURE_MASK_CET_KERNEL here ok? The host kernel will not
support XFEATURE_MASK_CET_KERNEL. I guess after this there is a small
window of time where host IA32_XSS could have non-host supported
supervisor state.

Sort of separately, how does SVM work with respect to saving and
restoring guest supervisor CET state (I mean the CET_S stuff)?

I'm not sure there is any problem, but just wondering how it all works.
Thanks.

Re: [RFC PATCH v2 6/6] KVM: SVM: Add CET features to supported_xss

[John Allen]

On Wed, May 24, 2023 at 05:24:23PM +0000, Edgecombe, Rick P wrote:
> On Wed, 2023-05-24 at 15:53 +0000, John Allen wrote:
> > If the CPU supports CET, add CET XSAVES feature bits to the
> > supported_xss mask.
> > 
> > Signed-off-by: John Allen <john.allen@amd.com>
> > ---
> > v2:
> >   - Remove curly braces around if statement
> > ---
> >  arch/x86/kvm/svm/svm.c | 4 ++++
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> > index 6afd2c44fdb6..cee496bee0a9 100644
> > --- a/arch/x86/kvm/svm/svm.c
> > +++ b/arch/x86/kvm/svm/svm.c
> > @@ -5070,6 +5070,10 @@ static __init void svm_set_cpu_caps(void)
> >             boot_cpu_has(X86_FEATURE_AMD_SSBD))
> >                 kvm_cpu_cap_set(X86_FEATURE_VIRT_SSBD);
> >  
> > +       if (kvm_cpu_cap_has(X86_FEATURE_SHSTK))
> > +               kvm_caps.supported_xss |= XFEATURE_MASK_CET_USER |
> > +                                         XFEATURE_MASK_CET_KERNEL;
> > +
> 
> Is setting XFEATURE_MASK_CET_KERNEL here ok? The host kernel will not
> support XFEATURE_MASK_CET_KERNEL. I guess after this there is a small
> window of time where host IA32_XSS could have non-host supported
> supervisor state.
> 
> Sort of separately, how does SVM work with respect to saving and
> restoring guest supervisor CET state (I mean the CET_S stuff)?

Apart from a minor exception involving SEV-ES, we are piggybacking on
the state saving/restoring in Yang Weijiang's x86/VMX series. So by
inspection, it looks like guest supervisor support is broken as the
supervisor XSAVES state and MSRs are not included in that series. I
currently don't have a way to test this case, but I think there are
operating systems that support it. I'll work on getting a guest set up
that can actually test this and hopefully have working guest supervisor
support in the next version of the series.

Thanks,
John

> 
> I'm not sure there is any problem, but just wondering how it all works.
> Thanks.

Re: [RFC PATCH v2 6/6] KVM: SVM: Add CET features to supported_xss

[Edgecombe, Rick P]

On Fri, 2023-06-09 at 10:34 -0500, John Allen wrote:
> > Is setting XFEATURE_MASK_CET_KERNEL here ok? The host kernel will
> > not
> > support XFEATURE_MASK_CET_KERNEL. I guess after this there is a
> > small
> > window of time where host IA32_XSS could have non-host supported
> > supervisor state.
> > 
> > Sort of separately, how does SVM work with respect to saving and
> > restoring guest supervisor CET state (I mean the CET_S stuff)?
> 
> Apart from a minor exception involving SEV-ES, we are piggybacking on
> the state saving/restoring in Yang Weijiang's x86/VMX series. So by
> inspection, it looks like guest supervisor support is broken as the
> supervisor XSAVES state and MSRs are not included in that series. I
> currently don't have a way to test this case, but I think there are
> operating systems that support it. I'll work on getting a guest set
> up
> that can actually test this and hopefully have working guest
> supervisor
> support in the next version of the series.

Hmm, interesting. VMX has some separate non-xsaves thing to save and
restore the guests supervisor CET state, so Weijiang's series doesn't
use the xsaves supervisor CET support. Also, since the host might have
CR4.CET set for its own reasons, if the host handled an exit with the
the guests MSR_IA32_S_CET set it could suddenly be subjected to CET
enforcement that it doesn't expect. Waiting to restore it until
returning to the guest is too late.

At least that's the reasoning on the VMX side as I understand it. If
you need to add XFEATURE_CET_KERNEL generally, we'll have to think
about how it works with host IBT. Probably easiest to leave it disabled
on the Intel side.

Re: [RFC PATCH v2 4/6] KVM: SVM: Save shadow stack host state on VMRUN

[Sean Christopherson]

On Wed, May 24, 2023, John Allen wrote:
> When running as an SEV-ES guest, the PL0_SSP, PL1_SSP, PL2_SSP, PL3_SSP,
> and U_CET fields in the VMCB save area are type B, meaning the host
> state is automatically loaded on a VMEXIT, but is not saved on a VMRUN.
> The other shadow stack MSRs, S_CET, SSP, and ISST_ADDR are type A,
> meaning they are loaded on VMEXIT and saved on VMRUN. Manually save the
> type B host MSR values before VMRUN.
> 
> Signed-off-by: John Allen <john.allen@amd.com>
> ---
>  arch/x86/kvm/svm/sev.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> index c25aeb550cd9..03dd68bddd51 100644
> --- a/arch/x86/kvm/svm/sev.c
> +++ b/arch/x86/kvm/svm/sev.c
> @@ -3028,6 +3028,19 @@ void sev_es_prepare_switch_to_guest(struct sev_es_save_area *hostsa)
>  
>  	/* MSR_IA32_XSS is restored on VMEXIT, save the currnet host value */
>  	hostsa->xss = host_xss;
> +
> +	if (boot_cpu_has(X86_FEATURE_SHSTK)) {
> +		/*
> +		 * MSR_IA32_U_CET, MSR_IA32_PL0_SSP, MSR_IA32_PL1_SSP,
> +		 * MSR_IA32_PL2_SSP, and MSR_IA32_PL3_SSP are restored on
> +		 * VMEXIT, save the current host values.
> +		 */
> +		rdmsrl(MSR_IA32_U_CET, hostsa->u_cet);
> +		rdmsrl(MSR_IA32_PL0_SSP, hostsa->vmpl0_ssp);
> +		rdmsrl(MSR_IA32_PL1_SSP, hostsa->vmpl1_ssp);
> +		rdmsrl(MSR_IA32_PL2_SSP, hostsa->vmpl2_ssp);
> +		rdmsrl(MSR_IA32_PL3_SSP, hostsa->vmpl3_ssp);

Heh, can you send a patch to fix the names for the PLx_SSP fields?  They should
be ->plN_ssp, not ->vmplN_ssp.

As for the values themselves, the kernel doesn't support Supervisor Shadow Stacks
(SSS), so PL0-2_SSP are guaranteed to be zero.  And if/when SSS support is added,
I doubt the kernel will ever use PL1_SSP or PL2_SSP, so those can probably be
ignored entirely, and PL0_SSP might be constant per task?  In other words, I don't
see any reason to try and track the host values for support that doesn't exist,
just do what VMX does for BNDCFGS and yell if the MSRs are non-zero.  Though for
SSS it probably makes sense for KVM to refuse to load (KVM continues on for BNDCFGS
because it's a pretty safe assumption that the kernel won't regain MPX supported).

E.g. in rough pseudocode

	if (boot_cpu_has(X86_FEATURE_SHSTK)) {
		rdmsrl(MSR_IA32_PLx_SSP, host_plx_ssp);

		if (WARN_ON_ONCE(host_pl0_ssp || host_pl1_ssp || host_pl2_ssp))
			return -EIO;
	}

Re: [RFC PATCH v2 6/6] KVM: SVM: Add CET features to supported_xss

[Sean Christopherson]

On Fri, Jun 09, 2023, Rick P Edgecombe wrote:
> On Fri, 2023-06-09 at 10:34 -0500, John Allen wrote:
> > > Is setting XFEATURE_MASK_CET_KERNEL here ok? The host kernel will not
> > > support XFEATURE_MASK_CET_KERNEL. I guess after this there is a small
> > > window of time where host IA32_XSS could have non-host supported
> > > supervisor state.
> > > 
> > > Sort of separately, how does SVM work with respect to saving and
> > > restoring guest supervisor CET state (I mean the CET_S stuff)?
> > 
> > Apart from a minor exception involving SEV-ES, we are piggybacking on the
> > state saving/restoring in Yang Weijiang's x86/VMX series. So by inspection,
> > it looks like guest supervisor support is broken as the supervisor XSAVES
> > state and MSRs are not included in that series. I currently don't have a
> > way to test this case, but I think there are operating systems that support
> > it. I'll work on getting a guest set up that can actually test this and
> > hopefully have working guest supervisor support in the next version of the
> > series.
> 
> Hmm, interesting. VMX has some separate non-xsaves thing to save and
> restore the guests supervisor CET state, so Weijiang's series doesn't
> use the xsaves supervisor CET support.

Heh, that and Weijiang's series is a wee bit incomplete.

> Also, since the host might have CR4.CET set for its own reasons, if the host
> handled an exit with the the guests MSR_IA32_S_CET set it could suddenly be
> subjected to CET enforcement that it doesn't expect. Waiting to restore it
> until returning to the guest is too late.
>
> At least that's the reasoning on the VMX side as I understand it

The APM doesn't come right out and say it, but I assume/hope that S_CET is saved
on VMRUN and loaded on #VMEXIT, i.e. is the same as VMX for all intents and
purposes.

The host save state definitely has a field for S_CET, and VMRUN documents that the
guest values are loaded, I just can't find anything in the APM that explicitly states
how host S_CET and friends are handled.  E.g. in theory, they could have been
shoved into VMSAVE+VMLOAD, though I very much doubt that's the case.

John?

Re: [RFC PATCH v2 3/6] KVM: x86: SVM: Pass through shadow stack MSRs

[Sean Christopherson]

On Wed, May 24, 2023, John Allen wrote:
> If kvm supports shadow stack, pass through shadow stack MSRs to improve
> guest performance.
> 
> Signed-off-by: John Allen <john.allen@amd.com>
> ---
>  arch/x86/kvm/svm/svm.c | 17 +++++++++++++++++
>  arch/x86/kvm/svm/svm.h |  2 +-
>  2 files changed, 18 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 6df486bb1ac4..cdbce20989b8 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -136,6 +136,13 @@ static const struct svm_direct_access_msrs {
>  	{ .index = X2APIC_MSR(APIC_TMICT),		.always = false },
>  	{ .index = X2APIC_MSR(APIC_TMCCT),		.always = false },
>  	{ .index = X2APIC_MSR(APIC_TDCR),		.always = false },
> +	{ .index = MSR_IA32_U_CET,                      .always = false },
> +	{ .index = MSR_IA32_S_CET,                      .always = false },
> +	{ .index = MSR_IA32_INT_SSP_TAB,                .always = false },
> +	{ .index = MSR_IA32_PL0_SSP,                    .always = false },
> +	{ .index = MSR_IA32_PL1_SSP,                    .always = false },
> +	{ .index = MSR_IA32_PL2_SSP,                    .always = false },
> +	{ .index = MSR_IA32_PL3_SSP,                    .always = false },
>  	{ .index = MSR_INVALID,				.always = false },
>  };
>  
> @@ -1181,6 +1188,16 @@ static inline void init_vmcb_after_set_cpuid(struct kvm_vcpu *vcpu)
>  		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_EIP, 1, 1);
>  		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_ESP, 1, 1);
>  	}
> +
> +	if (kvm_cet_user_supported() && guest_cpuid_has(vcpu, X86_FEATURE_SHSTK)) {
> +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_U_CET, 1, 1);
> +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_S_CET, 1, 1);
> +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_INT_SSP_TAB, 1, 1);
> +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL0_SSP, 1, 1);
> +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL1_SSP, 1, 1);
> +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL2_SSP, 1, 1);
> +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL3_SSP, 1, 1);
> +	}

This is wrong, KVM needs to set/clear interception based on SHSKT, i.e. it can't
be a one-way street.  Userspace *probably* won't toggle SHSTK in guest CPUID, but
weirder things have happened.

Re: [RFC PATCH v2 6/6] KVM: SVM: Add CET features to supported_xss

[Tom Lendacky]

On 6/23/23 17:18, Sean Christopherson wrote:
> On Fri, Jun 09, 2023, Rick P Edgecombe wrote:
>> On Fri, 2023-06-09 at 10:34 -0500, John Allen wrote:
>>>> Is setting XFEATURE_MASK_CET_KERNEL here ok? The host kernel will not
>>>> support XFEATURE_MASK_CET_KERNEL. I guess after this there is a small
>>>> window of time where host IA32_XSS could have non-host supported
>>>> supervisor state.
>>>>
>>>> Sort of separately, how does SVM work with respect to saving and
>>>> restoring guest supervisor CET state (I mean the CET_S stuff)?
>>>
>>> Apart from a minor exception involving SEV-ES, we are piggybacking on the
>>> state saving/restoring in Yang Weijiang's x86/VMX series. So by inspection,
>>> it looks like guest supervisor support is broken as the supervisor XSAVES
>>> state and MSRs are not included in that series. I currently don't have a
>>> way to test this case, but I think there are operating systems that support
>>> it. I'll work on getting a guest set up that can actually test this and
>>> hopefully have working guest supervisor support in the next version of the
>>> series.
>>
>> Hmm, interesting. VMX has some separate non-xsaves thing to save and
>> restore the guests supervisor CET state, so Weijiang's series doesn't
>> use the xsaves supervisor CET support.
> 
> Heh, that and Weijiang's series is a wee bit incomplete.
> 
>> Also, since the host might have CR4.CET set for its own reasons, if the host
>> handled an exit with the the guests MSR_IA32_S_CET set it could suddenly be
>> subjected to CET enforcement that it doesn't expect. Waiting to restore it
>> until returning to the guest is too late.
>>
>> At least that's the reasoning on the VMX side as I understand it
> 
> The APM doesn't come right out and say it, but I assume/hope that S_CET is saved
> on VMRUN and loaded on #VMEXIT, i.e. is the same as VMX for all intents and
> purposes.
> 
> The host save state definitely has a field for S_CET, and VMRUN documents that the
> guest values are loaded, I just can't find anything in the APM that explicitly states
> how host S_CET and friends are handled.  E.g. in theory, they could have been
> shoved into VMSAVE+VMLOAD, though I very much doubt that's the case.

Yes, the host value is saved/restored on VMRUN/#VMEXIT. Anything that is 
in the VMCB Save Area (the non-SEV-ES save area) is fully virtualized 
(unless noted otherwise) and doesn't require special processing to 
save/restore the host values.

S_CET is list in the SVM/SEV VMCB save area. Similarly, for 
SEV-ES/SEV-SNP, S_CET is swap type A and is saved/restored on VMRUN/#VMEXIT.

Thanks,
Tom

> 
> John?

Re: [RFC PATCH v2 6/6] KVM: SVM: Add CET features to supported_xss

[Sean Christopherson]

On Mon, Jun 26, 2023, Tom Lendacky wrote:
> On 6/23/23 17:18, Sean Christopherson wrote:
> > On Fri, Jun 09, 2023, Rick P Edgecombe wrote:
> > > Also, since the host might have CR4.CET set for its own reasons, if the host
> > > handled an exit with the the guests MSR_IA32_S_CET set it could suddenly be
> > > subjected to CET enforcement that it doesn't expect. Waiting to restore it
> > > until returning to the guest is too late.
> > > 
> > > At least that's the reasoning on the VMX side as I understand it
> > 
> > The APM doesn't come right out and say it, but I assume/hope that S_CET is saved
> > on VMRUN and loaded on #VMEXIT, i.e. is the same as VMX for all intents and
> > purposes.
> > 
> > The host save state definitely has a field for S_CET, and VMRUN documents that the
> > guest values are loaded, I just can't find anything in the APM that explicitly states
> > how host S_CET and friends are handled.  E.g. in theory, they could have been
> > shoved into VMSAVE+VMLOAD, though I very much doubt that's the case.
> 
> Yes, the host value is saved/restored on VMRUN/#VMEXIT. Anything that is in
> the VMCB Save Area (the non-SEV-ES save area) is fully virtualized (unless
> noted otherwise) and doesn't require special processing to save/restore the
> host values.

Would it makes sense to add a column in "Table B-2. VMCB Layout, State Save Area"
to specify whether a field is handled by VMRUN+#VMEXIT vs. VMLOAD+VMSAVE?  I can't
find anywhere in the APM where it explicitly states that VMRUN+#VMEXIT context
switches everything in the Save Area except the fields listed in "15.5.2 VMSAVE
and VMLOAD Instructions".

"15.5 VMRUN Instruction" kinda sorta covers that behavior, but the information is
either incomplete or stale, e.g. for host state it says "at least the following"

  Saving Host State. To ensure that the host can resume operation after #VMEXIT,
  VMRUN saves at least the following host state information:

but for guest state it says "the following"

  Loading Guest State. After saving host state, VMRUN loads the following guest
  state from the VMCB:

and then both provide incomplete lists of state.  A pedantic reading of the guest
case suggests that there's a large pile of state that *isn't* loaded, and the host
case isn't all that helpful because it's way too handwavy.

Re: [RFC PATCH v2 6/6] KVM: SVM: Add CET features to supported_xss

[Tom Lendacky]

On 6/26/23 11:28, Sean Christopherson wrote:
> On Mon, Jun 26, 2023, Tom Lendacky wrote:
>> On 6/23/23 17:18, Sean Christopherson wrote:
>>> On Fri, Jun 09, 2023, Rick P Edgecombe wrote:
>>>> Also, since the host might have CR4.CET set for its own reasons, if the host
>>>> handled an exit with the the guests MSR_IA32_S_CET set it could suddenly be
>>>> subjected to CET enforcement that it doesn't expect. Waiting to restore it
>>>> until returning to the guest is too late.
>>>>
>>>> At least that's the reasoning on the VMX side as I understand it
>>>
>>> The APM doesn't come right out and say it, but I assume/hope that S_CET is saved
>>> on VMRUN and loaded on #VMEXIT, i.e. is the same as VMX for all intents and
>>> purposes.
>>>
>>> The host save state definitely has a field for S_CET, and VMRUN documents that the
>>> guest values are loaded, I just can't find anything in the APM that explicitly states
>>> how host S_CET and friends are handled.  E.g. in theory, they could have been
>>> shoved into VMSAVE+VMLOAD, though I very much doubt that's the case.
>>
>> Yes, the host value is saved/restored on VMRUN/#VMEXIT. Anything that is in
>> the VMCB Save Area (the non-SEV-ES save area) is fully virtualized (unless
>> noted otherwise) and doesn't require special processing to save/restore the
>> host values.
> 
> Would it makes sense to add a column in "Table B-2. VMCB Layout, State Save Area"
> to specify whether a field is handled by VMRUN+#VMEXIT vs. VMLOAD+VMSAVE?  I can't
> find anywhere in the APM where it explicitly states that VMRUN+#VMEXIT context
> switches everything in the Save Area except the fields listed in "15.5.2 VMSAVE
> and VMLOAD Instructions".
> 
> "15.5 VMRUN Instruction" kinda sorta covers that behavior, but the information is
> either incomplete or stale, e.g. for host state it says "at least the following"
> 
>    Saving Host State. To ensure that the host can resume operation after #VMEXIT,
>    VMRUN saves at least the following host state information:
> 
> but for guest state it says "the following"
> 
>    Loading Guest State. After saving host state, VMRUN loads the following guest
>    state from the VMCB:
> 
> and then both provide incomplete lists of state.  A pedantic reading of the guest
> case suggests that there's a large pile of state that *isn't* loaded, and the host
> case isn't all that helpful because it's way too handwavy.

I'll communicate this feedback to the folks that update the APM volumes 
and see what can be done.

Thanks,
Tom

Re: [RFC PATCH v2 6/6] KVM: SVM: Add CET features to supported_xss

[Sean Christopherson]

On Mon, Jun 26, 2023, Tom Lendacky wrote:
> On 6/26/23 11:28, Sean Christopherson wrote:
> > On Mon, Jun 26, 2023, Tom Lendacky wrote:
> > > On 6/23/23 17:18, Sean Christopherson wrote:
> > > > On Fri, Jun 09, 2023, Rick P Edgecombe wrote:
> > > > > Also, since the host might have CR4.CET set for its own reasons, if the host
> > > > > handled an exit with the the guests MSR_IA32_S_CET set it could suddenly be
> > > > > subjected to CET enforcement that it doesn't expect. Waiting to restore it
> > > > > until returning to the guest is too late.
> > > > > 
> > > > > At least that's the reasoning on the VMX side as I understand it
> > > > 
> > > > The APM doesn't come right out and say it, but I assume/hope that S_CET is saved
> > > > on VMRUN and loaded on #VMEXIT, i.e. is the same as VMX for all intents and
> > > > purposes.
> > > > 
> > > > The host save state definitely has a field for S_CET, and VMRUN documents that the
> > > > guest values are loaded, I just can't find anything in the APM that explicitly states
> > > > how host S_CET and friends are handled.  E.g. in theory, they could have been
> > > > shoved into VMSAVE+VMLOAD, though I very much doubt that's the case.
> > > 
> > > Yes, the host value is saved/restored on VMRUN/#VMEXIT. Anything that is in
> > > the VMCB Save Area (the non-SEV-ES save area) is fully virtualized (unless
> > > noted otherwise) and doesn't require special processing to save/restore the
> > > host values.
> > 
> > Would it makes sense to add a column in "Table B-2. VMCB Layout, State Save Area"
> > to specify whether a field is handled by VMRUN+#VMEXIT vs. VMLOAD+VMSAVE?  I can't
> > find anywhere in the APM where it explicitly states that VMRUN+#VMEXIT context
> > switches everything in the Save Area except the fields listed in "15.5.2 VMSAVE
> > and VMLOAD Instructions".
> > 
> > "15.5 VMRUN Instruction" kinda sorta covers that behavior, but the information is
> > either incomplete or stale, e.g. for host state it says "at least the following"
> > 
> >    Saving Host State. To ensure that the host can resume operation after #VMEXIT,
> >    VMRUN saves at least the following host state information:
> > 
> > but for guest state it says "the following"
> > 
> >    Loading Guest State. After saving host state, VMRUN loads the following guest
> >    state from the VMCB:
> > 
> > and then both provide incomplete lists of state.  A pedantic reading of the guest
> > case suggests that there's a large pile of state that *isn't* loaded, and the host
> > case isn't all that helpful because it's way too handwavy.
> 
> I'll communicate this feedback to the folks that update the APM volumes and
> see what can be done.

Thanks, much appreciated!

Re: [RFC PATCH v2 4/6] KVM: SVM: Save shadow stack host state on VMRUN

[John Allen]

On Fri, Jun 23, 2023 at 02:11:46PM -0700, Sean Christopherson wrote:
> On Wed, May 24, 2023, John Allen wrote:
> > When running as an SEV-ES guest, the PL0_SSP, PL1_SSP, PL2_SSP, PL3_SSP,
> > and U_CET fields in the VMCB save area are type B, meaning the host
> > state is automatically loaded on a VMEXIT, but is not saved on a VMRUN.
> > The other shadow stack MSRs, S_CET, SSP, and ISST_ADDR are type A,
> > meaning they are loaded on VMEXIT and saved on VMRUN. Manually save the
> > type B host MSR values before VMRUN.
> > 
> > Signed-off-by: John Allen <john.allen@amd.com>
> > ---
> >  arch/x86/kvm/svm/sev.c | 13 +++++++++++++
> >  1 file changed, 13 insertions(+)
> > 
> > diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> > index c25aeb550cd9..03dd68bddd51 100644
> > --- a/arch/x86/kvm/svm/sev.c
> > +++ b/arch/x86/kvm/svm/sev.c
> > @@ -3028,6 +3028,19 @@ void sev_es_prepare_switch_to_guest(struct sev_es_save_area *hostsa)
> >  
> >  	/* MSR_IA32_XSS is restored on VMEXIT, save the currnet host value */
> >  	hostsa->xss = host_xss;
> > +
> > +	if (boot_cpu_has(X86_FEATURE_SHSTK)) {
> > +		/*
> > +		 * MSR_IA32_U_CET, MSR_IA32_PL0_SSP, MSR_IA32_PL1_SSP,
> > +		 * MSR_IA32_PL2_SSP, and MSR_IA32_PL3_SSP are restored on
> > +		 * VMEXIT, save the current host values.
> > +		 */
> > +		rdmsrl(MSR_IA32_U_CET, hostsa->u_cet);
> > +		rdmsrl(MSR_IA32_PL0_SSP, hostsa->vmpl0_ssp);
> > +		rdmsrl(MSR_IA32_PL1_SSP, hostsa->vmpl1_ssp);
> > +		rdmsrl(MSR_IA32_PL2_SSP, hostsa->vmpl2_ssp);
> > +		rdmsrl(MSR_IA32_PL3_SSP, hostsa->vmpl3_ssp);
> 
> Heh, can you send a patch to fix the names for the PLx_SSP fields?  They should
> be ->plN_ssp, not ->vmplN_ssp.

Yes, I will include a patch to address this in the next version of the
series.

> 
> As for the values themselves, the kernel doesn't support Supervisor Shadow Stacks
> (SSS), so PL0-2_SSP are guaranteed to be zero.  And if/when SSS support is added,
> I doubt the kernel will ever use PL1_SSP or PL2_SSP, so those can probably be
> ignored entirely, and PL0_SSP might be constant per task?  In other words, I don't
> see any reason to try and track the host values for support that doesn't exist,
> just do what VMX does for BNDCFGS and yell if the MSRs are non-zero.  Though for
> SSS it probably makes sense for KVM to refuse to load (KVM continues on for BNDCFGS
> because it's a pretty safe assumption that the kernel won't regain MPX supported).
> 
> E.g. in rough pseudocode
> 
> 	if (boot_cpu_has(X86_FEATURE_SHSTK)) {
> 		rdmsrl(MSR_IA32_PLx_SSP, host_plx_ssp);
> 
> 		if (WARN_ON_ONCE(host_pl0_ssp || host_pl1_ssp || host_pl2_ssp))
> 			return -EIO;
> 	}

The function in question returns void and wouldn't be able to return a
failure code to callers. We would have to rework this path in order to
fail in this way. Is it sufficient to just WARN_ON_ONCE here or is there
some other way we can cause KVM to fail to load here?

Re: [RFC PATCH v2 3/6] KVM: x86: SVM: Pass through shadow stack MSRs

[John Allen]

On Fri, Jun 23, 2023 at 05:05:18PM -0700, Sean Christopherson wrote:
> On Wed, May 24, 2023, John Allen wrote:
> > If kvm supports shadow stack, pass through shadow stack MSRs to improve
> > guest performance.
> > 
> > Signed-off-by: John Allen <john.allen@amd.com>
> > ---
> >  arch/x86/kvm/svm/svm.c | 17 +++++++++++++++++
> >  arch/x86/kvm/svm/svm.h |  2 +-
> >  2 files changed, 18 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> > index 6df486bb1ac4..cdbce20989b8 100644
> > --- a/arch/x86/kvm/svm/svm.c
> > +++ b/arch/x86/kvm/svm/svm.c
> > @@ -136,6 +136,13 @@ static const struct svm_direct_access_msrs {
> >  	{ .index = X2APIC_MSR(APIC_TMICT),		.always = false },
> >  	{ .index = X2APIC_MSR(APIC_TMCCT),		.always = false },
> >  	{ .index = X2APIC_MSR(APIC_TDCR),		.always = false },
> > +	{ .index = MSR_IA32_U_CET,                      .always = false },
> > +	{ .index = MSR_IA32_S_CET,                      .always = false },
> > +	{ .index = MSR_IA32_INT_SSP_TAB,                .always = false },
> > +	{ .index = MSR_IA32_PL0_SSP,                    .always = false },
> > +	{ .index = MSR_IA32_PL1_SSP,                    .always = false },
> > +	{ .index = MSR_IA32_PL2_SSP,                    .always = false },
> > +	{ .index = MSR_IA32_PL3_SSP,                    .always = false },
> >  	{ .index = MSR_INVALID,				.always = false },
> >  };
> >  
> > @@ -1181,6 +1188,16 @@ static inline void init_vmcb_after_set_cpuid(struct kvm_vcpu *vcpu)
> >  		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_EIP, 1, 1);
> >  		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_ESP, 1, 1);
> >  	}
> > +
> > +	if (kvm_cet_user_supported() && guest_cpuid_has(vcpu, X86_FEATURE_SHSTK)) {
> > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_U_CET, 1, 1);
> > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_S_CET, 1, 1);
> > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_INT_SSP_TAB, 1, 1);
> > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL0_SSP, 1, 1);
> > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL1_SSP, 1, 1);
> > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL2_SSP, 1, 1);
> > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL3_SSP, 1, 1);
> > +	}
> 
> This is wrong, KVM needs to set/clear interception based on SHSKT, i.e. it can't
> be a one-way street.  Userspace *probably* won't toggle SHSTK in guest CPUID, but
> weirder things have happened.

Can you clarify what you mean by that? Do you mean that we need to check
both guest_cpuid_has and kvm_cpu_cap_has like the guest_can_use function
that is used in Weijiang Yang's series? Or is there something else I'm
omitting here?

static __always_inline bool guest_can_use(struct kvm_vcpu *vcpu,
                                          unsigned int feature)
{
        return kvm_cpu_cap_has(feature) && guest_cpuid_has(vcpu, feature);
}

Re: [RFC PATCH v2 4/6] KVM: SVM: Save shadow stack host state on VMRUN

[Sean Christopherson]

On Tue, Aug 01, 2023, John Allen wrote:
> On Fri, Jun 23, 2023 at 02:11:46PM -0700, Sean Christopherson wrote:
> > On Wed, May 24, 2023, John Allen wrote:
> > As for the values themselves, the kernel doesn't support Supervisor Shadow Stacks
> > (SSS), so PL0-2_SSP are guaranteed to be zero.  And if/when SSS support is added,
> > I doubt the kernel will ever use PL1_SSP or PL2_SSP, so those can probably be
> > ignored entirely, and PL0_SSP might be constant per task?  In other words, I don't
> > see any reason to try and track the host values for support that doesn't exist,
> > just do what VMX does for BNDCFGS and yell if the MSRs are non-zero.  Though for
> > SSS it probably makes sense for KVM to refuse to load (KVM continues on for BNDCFGS
> > because it's a pretty safe assumption that the kernel won't regain MPX supported).
> > 
> > E.g. in rough pseudocode
> > 
> > 	if (boot_cpu_has(X86_FEATURE_SHSTK)) {
> > 		rdmsrl(MSR_IA32_PLx_SSP, host_plx_ssp);
> > 
> > 		if (WARN_ON_ONCE(host_pl0_ssp || host_pl1_ssp || host_pl2_ssp))
> > 			return -EIO;
> > 	}
> 
> The function in question returns void and wouldn't be able to return a
> failure code to callers. We would have to rework this path in order to
> fail in this way. Is it sufficient to just WARN_ON_ONCE here or is there
> some other way we can cause KVM to fail to load here?

Sorry, I should have been more explicit than "it probably make sense for KVM to
refuse to load".  The above would go somewhere in __kvm_x86_vendor_init().

Re: [RFC PATCH v2 3/6] KVM: x86: SVM: Pass through shadow stack MSRs

[Sean Christopherson]

On Tue, Aug 01, 2023, John Allen wrote:
> On Fri, Jun 23, 2023 at 05:05:18PM -0700, Sean Christopherson wrote:
> > On Wed, May 24, 2023, John Allen wrote:
> > > If kvm supports shadow stack, pass through shadow stack MSRs to improve
> > > guest performance.
> > > 
> > > Signed-off-by: John Allen <john.allen@amd.com>
> > > ---
> > >  arch/x86/kvm/svm/svm.c | 17 +++++++++++++++++
> > >  arch/x86/kvm/svm/svm.h |  2 +-
> > >  2 files changed, 18 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> > > index 6df486bb1ac4..cdbce20989b8 100644
> > > --- a/arch/x86/kvm/svm/svm.c
> > > +++ b/arch/x86/kvm/svm/svm.c
> > > @@ -136,6 +136,13 @@ static const struct svm_direct_access_msrs {
> > >  	{ .index = X2APIC_MSR(APIC_TMICT),		.always = false },
> > >  	{ .index = X2APIC_MSR(APIC_TMCCT),		.always = false },
> > >  	{ .index = X2APIC_MSR(APIC_TDCR),		.always = false },
> > > +	{ .index = MSR_IA32_U_CET,                      .always = false },
> > > +	{ .index = MSR_IA32_S_CET,                      .always = false },
> > > +	{ .index = MSR_IA32_INT_SSP_TAB,                .always = false },
> > > +	{ .index = MSR_IA32_PL0_SSP,                    .always = false },
> > > +	{ .index = MSR_IA32_PL1_SSP,                    .always = false },
> > > +	{ .index = MSR_IA32_PL2_SSP,                    .always = false },
> > > +	{ .index = MSR_IA32_PL3_SSP,                    .always = false },
> > >  	{ .index = MSR_INVALID,				.always = false },
> > >  };
> > >  
> > > @@ -1181,6 +1188,16 @@ static inline void init_vmcb_after_set_cpuid(struct kvm_vcpu *vcpu)
> > >  		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_EIP, 1, 1);
> > >  		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_ESP, 1, 1);
> > >  	}
> > > +
> > > +	if (kvm_cet_user_supported() && guest_cpuid_has(vcpu, X86_FEATURE_SHSTK)) {
> > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_U_CET, 1, 1);
> > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_S_CET, 1, 1);
> > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_INT_SSP_TAB, 1, 1);
> > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL0_SSP, 1, 1);
> > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL1_SSP, 1, 1);
> > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL2_SSP, 1, 1);
> > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL3_SSP, 1, 1);
> > > +	}
> > 
> > This is wrong, KVM needs to set/clear interception based on SHSKT, i.e. it can't
> > be a one-way street.  Userspace *probably* won't toggle SHSTK in guest CPUID, but
> > weirder things have happened.
> 
> Can you clarify what you mean by that? Do you mean that we need to check
> both guest_cpuid_has and kvm_cpu_cap_has like the guest_can_use function
> that is used in Weijiang Yang's series? Or is there something else I'm
> omitting here?

When init_vmcb_after_set_cpuid() is called, KVM must not assume that the MSRs are
currently intercepted, i.e. KVM can't just handle the case where userspace enables
SHSTK, KVM must also handle the case where userspace disables SHSTK.

Using guest_can_use() is also a good idea, but it would likely lead to extra work
on CPUs that don't support CET/SHSTK.  This isn't a fastpath, but toggling
interception for MSRs that don't exist would be odd.  It's probably better to
effectively open code guest_can_use(), which the KVM check gating the MSR toggling.

E.g. something like

	if (kvm_cpu_cap_has(X86_FEATURE_SHSTK)) {
		bool shstk_enabled = guest_cpuid_has(vcpu, X86_FEATURE_SHSTK);

		set_msr_inteception(vcpu, svm->msrpm, MSR_IA32_BLAH,
				    shstk_enabled, shstk_enabled);
	}

Re: [RFC PATCH v2 3/6] KVM: x86: SVM: Pass through shadow stack MSRs

[John Allen]

On Tue, Aug 01, 2023 at 09:42:09AM -0700, Sean Christopherson wrote:
> On Tue, Aug 01, 2023, John Allen wrote:
> > On Fri, Jun 23, 2023 at 05:05:18PM -0700, Sean Christopherson wrote:
> > > On Wed, May 24, 2023, John Allen wrote:
> > > > If kvm supports shadow stack, pass through shadow stack MSRs to improve
> > > > guest performance.
> > > > 
> > > > Signed-off-by: John Allen <john.allen@amd.com>
> > > > ---
> > > >  arch/x86/kvm/svm/svm.c | 17 +++++++++++++++++
> > > >  arch/x86/kvm/svm/svm.h |  2 +-
> > > >  2 files changed, 18 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> > > > index 6df486bb1ac4..cdbce20989b8 100644
> > > > --- a/arch/x86/kvm/svm/svm.c
> > > > +++ b/arch/x86/kvm/svm/svm.c
> > > > @@ -136,6 +136,13 @@ static const struct svm_direct_access_msrs {
> > > >  	{ .index = X2APIC_MSR(APIC_TMICT),		.always = false },
> > > >  	{ .index = X2APIC_MSR(APIC_TMCCT),		.always = false },
> > > >  	{ .index = X2APIC_MSR(APIC_TDCR),		.always = false },
> > > > +	{ .index = MSR_IA32_U_CET,                      .always = false },
> > > > +	{ .index = MSR_IA32_S_CET,                      .always = false },
> > > > +	{ .index = MSR_IA32_INT_SSP_TAB,                .always = false },
> > > > +	{ .index = MSR_IA32_PL0_SSP,                    .always = false },
> > > > +	{ .index = MSR_IA32_PL1_SSP,                    .always = false },
> > > > +	{ .index = MSR_IA32_PL2_SSP,                    .always = false },
> > > > +	{ .index = MSR_IA32_PL3_SSP,                    .always = false },
> > > >  	{ .index = MSR_INVALID,				.always = false },
> > > >  };
> > > >  
> > > > @@ -1181,6 +1188,16 @@ static inline void init_vmcb_after_set_cpuid(struct kvm_vcpu *vcpu)
> > > >  		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_EIP, 1, 1);
> > > >  		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_SYSENTER_ESP, 1, 1);
> > > >  	}
> > > > +
> > > > +	if (kvm_cet_user_supported() && guest_cpuid_has(vcpu, X86_FEATURE_SHSTK)) {
> > > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_U_CET, 1, 1);
> > > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_S_CET, 1, 1);
> > > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_INT_SSP_TAB, 1, 1);
> > > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL0_SSP, 1, 1);
> > > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL1_SSP, 1, 1);
> > > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL2_SSP, 1, 1);
> > > > +		set_msr_interception(vcpu, svm->msrpm, MSR_IA32_PL3_SSP, 1, 1);
> > > > +	}
> > > 
> > > This is wrong, KVM needs to set/clear interception based on SHSKT, i.e. it can't
> > > be a one-way street.  Userspace *probably* won't toggle SHSTK in guest CPUID, but
> > > weirder things have happened.
> > 
> > Can you clarify what you mean by that? Do you mean that we need to check
> > both guest_cpuid_has and kvm_cpu_cap_has like the guest_can_use function
> > that is used in Weijiang Yang's series? Or is there something else I'm
> > omitting here?
> 
> When init_vmcb_after_set_cpuid() is called, KVM must not assume that the MSRs are
> currently intercepted, i.e. KVM can't just handle the case where userspace enables
> SHSTK, KVM must also handle the case where userspace disables SHSTK.
> 
> Using guest_can_use() is also a good idea, but it would likely lead to extra work
> on CPUs that don't support CET/SHSTK.  This isn't a fastpath, but toggling
> interception for MSRs that don't exist would be odd.  It's probably better to
> effectively open code guest_can_use(), which the KVM check gating the MSR toggling.
> 
> E.g. something like
> 
> 	if (kvm_cpu_cap_has(X86_FEATURE_SHSTK)) {
> 		bool shstk_enabled = guest_cpuid_has(vcpu, X86_FEATURE_SHSTK);
> 
> 		set_msr_inteception(vcpu, svm->msrpm, MSR_IA32_BLAH,
> 				    shstk_enabled, shstk_enabled);
> 	}

Thanks for the clarification. I will use the above method in the next
version of the series.

Re: [RFC PATCH v2 4/6] KVM: SVM: Save shadow stack host state on VMRUN

[John Allen]

On Tue, Aug 01, 2023 at 09:28:11AM -0700, Sean Christopherson wrote:
> On Tue, Aug 01, 2023, John Allen wrote:
> > On Fri, Jun 23, 2023 at 02:11:46PM -0700, Sean Christopherson wrote:
> > > On Wed, May 24, 2023, John Allen wrote:
> > > As for the values themselves, the kernel doesn't support Supervisor Shadow Stacks
> > > (SSS), so PL0-2_SSP are guaranteed to be zero.  And if/when SSS support is added,
> > > I doubt the kernel will ever use PL1_SSP or PL2_SSP, so those can probably be
> > > ignored entirely, and PL0_SSP might be constant per task?  In other words, I don't
> > > see any reason to try and track the host values for support that doesn't exist,
> > > just do what VMX does for BNDCFGS and yell if the MSRs are non-zero.  Though for
> > > SSS it probably makes sense for KVM to refuse to load (KVM continues on for BNDCFGS
> > > because it's a pretty safe assumption that the kernel won't regain MPX supported).
> > > 
> > > E.g. in rough pseudocode
> > > 
> > > 	if (boot_cpu_has(X86_FEATURE_SHSTK)) {
> > > 		rdmsrl(MSR_IA32_PLx_SSP, host_plx_ssp);
> > > 
> > > 		if (WARN_ON_ONCE(host_pl0_ssp || host_pl1_ssp || host_pl2_ssp))
> > > 			return -EIO;
> > > 	}
> > 
> > The function in question returns void and wouldn't be able to return a
> > failure code to callers. We would have to rework this path in order to
> > fail in this way. Is it sufficient to just WARN_ON_ONCE here or is there
> > some other way we can cause KVM to fail to load here?
> 
> Sorry, I should have been more explicit than "it probably make sense for KVM to
> refuse to load".  The above would go somewhere in __kvm_x86_vendor_init().

I see, in that case that change should probably go up with:
"KVM:x86: Enable CET virtualization for VMX and advertise to userspace"
in Weijiang Yang's series with the rest of the changes to
__kvm_x86_vendor_init(). Though I can tack it on in my series if
needed.

Re: [RFC PATCH v2 4/6] KVM: SVM: Save shadow stack host state on VMRUN

[Yang, Weijiang]

On 8/2/2023 1:03 AM, John Allen wrote:
> On Tue, Aug 01, 2023 at 09:28:11AM -0700, Sean Christopherson wrote:
>> On Tue, Aug 01, 2023, John Allen wrote:
>>> On Fri, Jun 23, 2023 at 02:11:46PM -0700, Sean Christopherson wrote:
>>>> On Wed, May 24, 2023, John Allen wrote:
>>>> As for the values themselves, the kernel doesn't support Supervisor Shadow Stacks
>>>> (SSS), so PL0-2_SSP are guaranteed to be zero.  And if/when SSS support is added,
>>>> I doubt the kernel will ever use PL1_SSP or PL2_SSP, so those can probably be
>>>> ignored entirely, and PL0_SSP might be constant per task?  In other words, I don't
>>>> see any reason to try and track the host values for support that doesn't exist,
>>>> just do what VMX does for BNDCFGS and yell if the MSRs are non-zero.  Though for
>>>> SSS it probably makes sense for KVM to refuse to load (KVM continues on for BNDCFGS
>>>> because it's a pretty safe assumption that the kernel won't regain MPX supported).
>>>>
>>>> E.g. in rough pseudocode
>>>>
>>>> 	if (boot_cpu_has(X86_FEATURE_SHSTK)) {
>>>> 		rdmsrl(MSR_IA32_PLx_SSP, host_plx_ssp);
>>>>
>>>> 		if (WARN_ON_ONCE(host_pl0_ssp || host_pl1_ssp || host_pl2_ssp))
>>>> 			return -EIO;
>>>> 	}
>>> The function in question returns void and wouldn't be able to return a
>>> failure code to callers. We would have to rework this path in order to
>>> fail in this way. Is it sufficient to just WARN_ON_ONCE here or is there
>>> some other way we can cause KVM to fail to load here?
>> Sorry, I should have been more explicit than "it probably make sense for KVM to
>> refuse to load".  The above would go somewhere in __kvm_x86_vendor_init().
> I see, in that case that change should probably go up with:
> "KVM:x86: Enable CET virtualization for VMX and advertise to userspace"
> in Weijiang Yang's series with the rest of the changes to
> __kvm_x86_vendor_init(). Though I can tack it on in my series if
> needed.

The downside with above WARN_ON check is, KVM has to clear PL{0,1,2}_SSP 
for all CPUs when

SVM/VMX module is unloaded given guest would use them, otherwise, it may 
hit the check next

time the module is reloaded.

Can we add  check as below to make it easier?

@@ -9616,6 +9618,24 @@ static int __kvm_x86_vendor_init(struct 
kvm_x86_init_ops *ops)
                 return -EIO;
         }

+       if (boot_cpu_has(X86_FEATURE_SHSTK)) {
+               rdmsrl(MSR_IA32_S_CET, host_s_cet);
+               if (host_s_cet & CET_SHSTK_EN) {
+                       /*
+                        * Current CET KVM solution assumes host supervisor
+                        * shadow stack is always disable. If it's enabled
+                        * on host side, the guest supervisor states would
+                        * conflict with that of host's. When host 
supervisor
+                        * shadow stack is enabled one day, part of 
guest CET
+                        * enabling code should be refined to make both 
parties
+                        * work properly. Right now stop KVM module loading
+                        * once host supervisor shadow stack is detected on.
+                        */
+                       pr_err("Host supervisor shadow stack is not 
compatible with KVM!\n");
+                       return -EIO;
+               }
+       }
+
         x86_emulator_cache = kvm_alloc_emulator_cache();
         if (!x86_emulator_cache) {
                 pr_err("failed to allocate cache for x86 emulator\n");

Anyway, these PLx_SSP only takes effect when SSS is enabled on host, 
otherwise,

they can used as scratch registers when SHSTK is available.

Re: [RFC PATCH v2 4/6] KVM: SVM: Save shadow stack host state on VMRUN

[Sean Christopherson]

On Wed, Aug 02, 2023, Weijiang Yang wrote:
> 
> On 8/2/2023 1:03 AM, John Allen wrote:
> > On Tue, Aug 01, 2023 at 09:28:11AM -0700, Sean Christopherson wrote:
> > > On Tue, Aug 01, 2023, John Allen wrote:
> > > > On Fri, Jun 23, 2023 at 02:11:46PM -0700, Sean Christopherson wrote:
> > > > > On Wed, May 24, 2023, John Allen wrote:
> > > > > As for the values themselves, the kernel doesn't support Supervisor Shadow Stacks
> > > > > (SSS), so PL0-2_SSP are guaranteed to be zero.  And if/when SSS support is added,
> > > > > I doubt the kernel will ever use PL1_SSP or PL2_SSP, so those can probably be
> > > > > ignored entirely, and PL0_SSP might be constant per task?  In other words, I don't
> > > > > see any reason to try and track the host values for support that doesn't exist,
> > > > > just do what VMX does for BNDCFGS and yell if the MSRs are non-zero.  Though for
> > > > > SSS it probably makes sense for KVM to refuse to load (KVM continues on for BNDCFGS
> > > > > because it's a pretty safe assumption that the kernel won't regain MPX supported).
> > > > > 
> > > > > E.g. in rough pseudocode
> > > > > 
> > > > > 	if (boot_cpu_has(X86_FEATURE_SHSTK)) {
> > > > > 		rdmsrl(MSR_IA32_PLx_SSP, host_plx_ssp);
> > > > > 
> > > > > 		if (WARN_ON_ONCE(host_pl0_ssp || host_pl1_ssp || host_pl2_ssp))
> > > > > 			return -EIO;
> > > > > 	}
> > > > The function in question returns void and wouldn't be able to return a
> > > > failure code to callers. We would have to rework this path in order to
> > > > fail in this way. Is it sufficient to just WARN_ON_ONCE here or is there
> > > > some other way we can cause KVM to fail to load here?
> > > Sorry, I should have been more explicit than "it probably make sense for KVM to
> > > refuse to load".  The above would go somewhere in __kvm_x86_vendor_init().
> > I see, in that case that change should probably go up with:
> > "KVM:x86: Enable CET virtualization for VMX and advertise to userspace"
> > in Weijiang Yang's series with the rest of the changes to
> > __kvm_x86_vendor_init(). Though I can tack it on in my series if
> > needed.
> 
> The downside with above WARN_ON check is, KVM has to clear PL{0,1,2}_SSP for
> all CPUs when SVM/VMX module is unloaded given guest would use them, otherwise,
> it may hit the check next time the module is reloaded.

Off topic, can you please try to fix your mail client?  Almost of your replies
have extra newlines.  I'm guessing something is auto-wrapping at 80 chars, and
doing it poorly.

> Can we add  check as below to make it easier?

Hmm, yeah, that makes sense.  I based my suggestion off of what KVM does for MPX,
but I forgot that KVM clears MSR_IA32_BNDCFGS on VM-Exit via the VMCS, i.e.
effectively does preserve the host value so long as the host value is zero.

Not clearing the MSRs on module exit is a bit uncouth, but this is more or less
the same situation/argument for not doing INVEPT on module exit.  It's unsafe for
a module to assume that there aren't TLB entries for a given EP4TA, because even
if all sources of EPTPs (hypervisor/KVM modules) are well-intentioned and *try*
to clean up after themselves, it's always possible that a module crashed or was
buggy.  I.e. asserting the the PLx_SSP MSRs are zero is simply wrong, whereas
asserting that SSS is not enabled is correct.

> @@ -9616,6 +9618,24 @@ static int __kvm_x86_vendor_init(struct
> kvm_x86_init_ops *ops)
>                 return -EIO;
>         }
> 
> +       if (boot_cpu_has(X86_FEATURE_SHSTK)) {
> +               rdmsrl(MSR_IA32_S_CET, host_s_cet);
> +               if (host_s_cet & CET_SHSTK_EN) {

Make this a WARN_ON_ONCE() and drop the pr_err().  Hitting this would very much
be a kernel bug, e.g. either someone added SSS support and neglected to update
KVM, or the kernel's MSR_IA32_S_CET is corrupted.

> +                       /*
> +                        * Current CET KVM solution assumes host supervisor
> +                        * shadow stack is always disable. If it's enabled
> +                        * on host side, the guest supervisor states would
> +                        * conflict with that of host's. When host
> supervisor
> +                        * shadow stack is enabled one day, part of guest
> CET
> +                        * enabling code should be refined to make both
> parties
> +                        * work properly. Right now stop KVM module loading
> +                        * once host supervisor shadow stack is detected on.

I don't think we need to have a super elaborate comment, stating that SSS isn't
support and so KVM doesn't save/restore PLx_SSP MSRs should suffice.

> +                        */

Put the comment above the if-statment that has the WARN.  That reduces the
indentation, and avoids the question of whether or not a comment above a single
line is supposed to have curly braces.

E.g. something like this, though I think even the below comment is probably
unnecessarily verbose.

		/*
		 * Linux doesn't yet support supervisor shadow stacks (SSS), so
		 * so KVM doesn't save/restore the associated MSRs, i.e. KVM
		 * may clobber the host values.  Yell and refuse to load if SSS
		 * is unexpectedly enabled, e.g. to avoid crashing the host.
		 */
		if (WARN_ON_ONCE(host_s_cet & CET_SHSTK_EN))

Thanks!

Re: [RFC PATCH v2 4/6] KVM: SVM: Save shadow stack host state on VMRUN

[Yang, Weijiang]

On 8/3/2023 12:38 AM, Sean Christopherson wrote:
> On Wed, Aug 02, 2023, Weijiang Yang wrote:
>> On 8/2/2023 1:03 AM, John Allen wrote:
>>> On Tue, Aug 01, 2023 at 09:28:11AM -0700, Sean Christopherson wrote:
>>>> On Tue, Aug 01, 2023, John Allen wrote:
>>>>> On Fri, Jun 23, 2023 at 02:11:46PM -0700, Sean Christopherson wrote:
>>>>>> On Wed, May 24, 2023, John Allen wrote:
>>>>>> As for the values themselves, the kernel doesn't support Supervisor Shadow Stacks
>>>>>> (SSS), so PL0-2_SSP are guaranteed to be zero.  And if/when SSS support is added,
>>>>>> I doubt the kernel will ever use PL1_SSP or PL2_SSP, so those can probably be
>>>>>> ignored entirely, and PL0_SSP might be constant per task?  In other words, I don't
>>>>>> see any reason to try and track the host values for support that doesn't exist,
>>>>>> just do what VMX does for BNDCFGS and yell if the MSRs are non-zero.  Though for
>>>>>> SSS it probably makes sense for KVM to refuse to load (KVM continues on for BNDCFGS
>>>>>> because it's a pretty safe assumption that the kernel won't regain MPX supported).
>>>>>>
>>>>>> E.g. in rough pseudocode
>>>>>>
>>>>>> 	if (boot_cpu_has(X86_FEATURE_SHSTK)) {
>>>>>> 		rdmsrl(MSR_IA32_PLx_SSP, host_plx_ssp);
>>>>>>
>>>>>> 		if (WARN_ON_ONCE(host_pl0_ssp || host_pl1_ssp || host_pl2_ssp))
>>>>>> 			return -EIO;
>>>>>> 	}
>>>>> The function in question returns void and wouldn't be able to return a
>>>>> failure code to callers. We would have to rework this path in order to
>>>>> fail in this way. Is it sufficient to just WARN_ON_ONCE here or is there
>>>>> some other way we can cause KVM to fail to load here?
>>>> Sorry, I should have been more explicit than "it probably make sense for KVM to
>>>> refuse to load".  The above would go somewhere in __kvm_x86_vendor_init().
>>> I see, in that case that change should probably go up with:
>>> "KVM:x86: Enable CET virtualization for VMX and advertise to userspace"
>>> in Weijiang Yang's series with the rest of the changes to
>>> __kvm_x86_vendor_init(). Though I can tack it on in my series if
>>> needed.
>> The downside with above WARN_ON check is, KVM has to clear PL{0,1,2}_SSP for
>> all CPUs when SVM/VMX module is unloaded given guest would use them, otherwise,
>> it may hit the check next time the module is reloaded.
> Off topic, can you please try to fix your mail client?  Almost of your replies
> have extra newlines.  I'm guessing something is auto-wrapping at 80 chars, and
> doing it poorly.
Sorry for that. Some of the blank lines are added by me, and some are auto-added by the
editor. I changed the settings and will avoid to do so.
>> Can we add  check as below to make it easier?
> Hmm, yeah, that makes sense.  I based my suggestion off of what KVM does for MPX,
> but I forgot that KVM clears MSR_IA32_BNDCFGS on VM-Exit via the VMCS, i.e.
> effectively does preserve the host value so long as the host value is zero.
>
> Not clearing the MSRs on module exit is a bit uncouth, but this is more or less
> the same situation/argument for not doing INVEPT on module exit.  It's unsafe for
> a module to assume that there aren't TLB entries for a given EP4TA, because even
> if all sources of EPTPs (hypervisor/KVM modules) are well-intentioned and *try*
> to clean up after themselves, it's always possible that a module crashed or was
> buggy.  I.e. asserting the the PLx_SSP MSRs are zero is simply wrong, whereas
> asserting that SSS is not enabled is correct.
OK.
>> @@ -9616,6 +9618,24 @@ static int __kvm_x86_vendor_init(struct
>> kvm_x86_init_ops *ops)
>>                  return -EIO;
>>          }
>>
>> +       if (boot_cpu_has(X86_FEATURE_SHSTK)) {
>> +               rdmsrl(MSR_IA32_S_CET, host_s_cet);
>> +               if (host_s_cet & CET_SHSTK_EN) {
> Make this a WARN_ON_ONCE() and drop the pr_err().  Hitting this would very much
> be a kernel bug, e.g. either someone added SSS support and neglected to update
> KVM, or the kernel's MSR_IA32_S_CET is corrupted.
OK, will change it.
>> +                       /*
>> +                        * Current CET KVM solution assumes host supervisor
>> +                        * shadow stack is always disable. If it's enabled
>> +                        * on host side, the guest supervisor states would
>> +                        * conflict with that of host's. When host
>> supervisor
>> +                        * shadow stack is enabled one day, part of guest
>> CET
>> +                        * enabling code should be refined to make both
>> parties
>> +                        * work properly. Right now stop KVM module loading
>> +                        * once host supervisor shadow stack is detected on.
> I don't think we need to have a super elaborate comment, stating that SSS isn't
> support and so KVM doesn't save/restore PLx_SSP MSRs should suffice.
>
>> +                        */
> Put the comment above the if-statment that has the WARN.  That reduces the
> indentation, and avoids the question of whether or not a comment above a single
> line is supposed to have curly braces.
>
> E.g. something like this, though I think even the below comment is probably
> unnecessarily verbose.
>
> 		/*
> 		 * Linux doesn't yet support supervisor shadow stacks (SSS), so
> 		 * so KVM doesn't save/restore the associated MSRs, i.e. KVM
> 		 * may clobber the host values.  Yell and refuse to load if SSS
> 		 * is unexpectedly enabled, e.g. to avoid crashing the host.
> 		 */
> 		if (WARN_ON_ONCE(host_s_cet & CET_SHSTK_EN))
I will keep these comments as some hints to end users when something unexpected happens!
Thanks a lot!