[PATCH 0/5] Introduce support for PSF mitigation

[PATCH 0/5] Introduce support for PSF mitigation

[Ramakrishna Saripalli]

From: Ramakrishna Saripalli <rk.saripalli@amd.com>

Predictive Store Forwarding:
AMD Zen3 processors feature a new technology called
Predictive Store Forwarding (PSF).

<TODO:Insert link to AMD PSF whitepaper>

PSF is a hardware-based micro-architectural optimization designed
to improve the performance of code execution by predicting address
dependencies between loads and stores.

How PSF works:

It is very common for a CPU to execute a load instruction to an address
that was recently written by a store. Modern CPUs implement a technique
known as Store-To-Load-Forwarding (STLF) to improve performance in such
cases. With STLF, data from the store is forwarded directly to the load
without having to wait for it to be written to memory. In a typical CPU,
STLF occurs after the address of both the load and store are calculated
and determined to match.

PSF expands on this by speculating on the relationship between loads and
stores without waiting for the address calculation to complete. With PSF,
the CPU learns over time the relationship between loads and stores.
If STLF typically occurs between a particular store and load, the CPU will
remember this.

In typical code, PSF provides a performance benefit by speculating on
the load result and allowing later instructions to begin execution
sooner than they otherwise would be able to.

Causes of Incorrect PSF:

Incorrect PSF predictions can occur due to two reasons.

First, it is possible that the store/load pair had a dependency for a
while but later stops having a dependency.  This can occur if the address
of either the store or load changes during the execution of the program.

The second source of incorrect PSF predictions can occur if there is an
alias in the PSF predictor structure.  The PSF predictor tracks
store-load pairs based on portions of their RIP. It is possible that a
store-load pair which does have a dependency may alias in the predictor
with another store-load pair which does not.

This can result in incorrect speculation when the second store/load pair
is executed.

Security Analysis:

Previous research has shown that when CPUs speculate on non-architectural
paths it can lead to the potential of side channel attacks.
In particular, programs that implement isolation, also known as
‘sandboxing’, entirely in software may need to be concerned with incorrect
CPU speculation as they can occur due to bad PSF predictions.

Because PSF speculation is limited to the current program context,
the impact of bad PSF speculation is very similar to that of
Speculative Store Bypass (Spectre v4)

Predictive Store Forwarding controls:
There are two hardware control bits which influence the PSF feature:
- MSR 48h bit 2 – Speculative Store Bypass (SSBD)
- MSR 48h bit 7 – Predictive Store Forwarding Disable (PSFD)

The PSF feature is disabled if either of these bits are set.  These bits
are controllable on a per-thread basis in an SMT system. By default, both
SSBD and PSFD are 0 meaning that the speculation features are enabled.

While the SSBD bit disables PSF and speculative store bypass, PSFD only
disables PSF.

PSFD may be desirable for software which is concerned with the
speculative behavior of PSF but desires a smaller performance impact than
setting SSBD.

Support for PSFD is indicated in CPUID Fn8000_0008 EBX[28].
All processors that support PSF will also support PSFD.

Ramakrishna Saripalli (5):
  x86/cpufeatures: Define feature bits to support mitigation of PSF
  x86/speculation: Implement support for PSFD detection and reporting
  x86/speculation: Introduce SPEC_CTRL_MSR bit for PSFD
  x86/speculation: Implement PSF mitigation support
  x86/speculation: Add PSF mitigation kernel parameters

 .../admin-guide/kernel-parameters.txt         |  45 +++++
 arch/x86/include/asm/cpufeatures.h            |   4 +-
 arch/x86/include/asm/msr-index.h              |   2 +
 arch/x86/include/asm/nospec-branch.h          |   8 +
 arch/x86/include/asm/spec-ctrl.h              |  12 ++
 arch/x86/include/asm/thread_info.h            |   2 +
 arch/x86/kernel/cpu/bugs.c                    | 191 ++++++++++++++++++
 arch/x86/kernel/cpu/common.c                  |   6 +
 arch/x86/kernel/process.c                     |  23 +++
 include/linux/sched.h                         |  15 ++
 include/uapi/linux/prctl.h                    |   2 +
 11 files changed, 309 insertions(+), 1 deletion(-)


base-commit: 0e16f466004d7f04296b9676a712a32a12367d1f
-- 
2.25.1

[PATCH 1/5] x86/cpufeatures: Define feature bits to support mitigation of PSF

[Ramakrishna Saripalli]

From: Ramakrishna Saripalli <rk.saripalli@amd.com>

Certain AMD processors feature a new technology called Predictive Store
Forwarding (PSF).

PSF is a micro-architectural optimization designed to improve the
performance of code execution by predicting dependencies between
loads and stores.

Incorrect PSF predictions can occur due to two reasons.

- It is possible that the load/store pair may have had dependency for
  a while but the dependency has stopped because the address in the
  load/store pair has changed.

- Second source of incorrect PSF prediction can occur because of an alias
  in the PSF predictor structure stored in the microarchitectural state.
  PSF predictor tracks load/store pair based on portions of instruction
  pointer. It is possible that a load/store pair which does have a
  dependency may be aliased by another load/store pair which does not have
  the same dependency. This can result in incorrect speculation.

  Software may be able to detect this aliasing and perform side-channel
  attacks.

These features are being introduced to support mitigation from these attacks.

All CPUs that implement PSF provide one bit to disable this feature.
If the bit to disable this feature is available, it means that the CPU
implements PSF feature and is therefore vulnerable to PSF risks.

The bits that are introduced

X86_FEATURE_AMD_PSFD: CPUID_Fn80000008_EBX[28] ("PSF disable")
	If this bit is 1, CPU implements PSF and PSF mitigation is
	supported.

X86_FEATURE_PSFD: Generic Linux feature to indicate the CPU supports
	controls to mitigate PSF.

 	X86_FEATURE_PSFD is set if and only if X86_FEATURE_AMD_PSFD
	is set

X86_BUG_PSF: Generic Linux feature to indicate that the CPU is affected
	by the PSF feature.

Signed-off-by: Ramakrishna Saripalli<rk.saripalli@amd.com>
---
 arch/x86/include/asm/cpufeatures.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index cc96e26d69f7..21e7f8d0d7d9 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -201,7 +201,7 @@
 #define X86_FEATURE_INVPCID_SINGLE	( 7*32+ 7) /* Effectively INVPCID && CR4.PCIDE=1 */
 #define X86_FEATURE_HW_PSTATE		( 7*32+ 8) /* AMD HW-PState */
 #define X86_FEATURE_PROC_FEEDBACK	( 7*32+ 9) /* AMD ProcFeedbackInterface */
-/* FREE!                                ( 7*32+10) */
+#define X86_FEATURE_PSFD		( 7*32+10) /* Predictive Store Forward Disable */
 #define X86_FEATURE_PTI			( 7*32+11) /* Kernel Page Table Isolation enabled */
 #define X86_FEATURE_RETPOLINE		( 7*32+12) /* "" Generic Retpoline mitigation for Spectre variant 2 */
 #define X86_FEATURE_RETPOLINE_AMD	( 7*32+13) /* "" AMD Retpoline mitigation for Spectre variant 2 */
@@ -309,6 +309,7 @@
 #define X86_FEATURE_AMD_SSBD		(13*32+24) /* "" Speculative Store Bypass Disable */
 #define X86_FEATURE_VIRT_SSBD		(13*32+25) /* Virtualized Speculative Store Bypass Disable */
 #define X86_FEATURE_AMD_SSB_NO		(13*32+26) /* "" Speculative Store Bypass is fixed in hardware. */
+#define X86_FEATURE_AMD_PSFD		(13*32+28) /* "" Predictive Store Forward Disable */
 
 /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */
 #define X86_FEATURE_DTHERM		(14*32+ 0) /* Digital Thermal Sensor */
@@ -428,5 +429,6 @@
 #define X86_BUG_TAA			X86_BUG(22) /* CPU is affected by TSX Async Abort(TAA) */
 #define X86_BUG_ITLB_MULTIHIT		X86_BUG(23) /* CPU may incur MCE during certain page attribute changes */
 #define X86_BUG_SRBDS			X86_BUG(24) /* CPU may leak RNG bits if not mitigated */
+#define X86_BUG_PSF			X86_BUG(25) /* CPU is affected by Predictive Store Forwarding attack */
 
 #endif /* _ASM_X86_CPUFEATURES_H */
-- 
2.25.1

Re: [PATCH 1/5] x86/cpufeatures: Define feature bits to support mitigation of PSF

[Borislav Petkov]

On Tue, Apr 06, 2021 at 10:50:00AM -0500, Ramakrishna Saripalli wrote:
> diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
> index cc96e26d69f7..21e7f8d0d7d9 100644
> --- a/arch/x86/include/asm/cpufeatures.h
> +++ b/arch/x86/include/asm/cpufeatures.h
> @@ -201,7 +201,7 @@
>  #define X86_FEATURE_INVPCID_SINGLE	( 7*32+ 7) /* Effectively INVPCID && CR4.PCIDE=1 */
>  #define X86_FEATURE_HW_PSTATE		( 7*32+ 8) /* AMD HW-PState */
>  #define X86_FEATURE_PROC_FEEDBACK	( 7*32+ 9) /* AMD ProcFeedbackInterface */
> -/* FREE!                                ( 7*32+10) */
> +#define X86_FEATURE_PSFD		( 7*32+10) /* Predictive Store Forward Disable */

You don't need this one...

>  #define X86_FEATURE_PTI			( 7*32+11) /* Kernel Page Table Isolation enabled */
>  #define X86_FEATURE_RETPOLINE		( 7*32+12) /* "" Generic Retpoline mitigation for Spectre variant 2 */
>  #define X86_FEATURE_RETPOLINE_AMD	( 7*32+13) /* "" AMD Retpoline mitigation for Spectre variant 2 */
> @@ -309,6 +309,7 @@
>  #define X86_FEATURE_AMD_SSBD		(13*32+24) /* "" Speculative Store Bypass Disable */
>  #define X86_FEATURE_VIRT_SSBD		(13*32+25) /* Virtualized Speculative Store Bypass Disable */
>  #define X86_FEATURE_AMD_SSB_NO		(13*32+26) /* "" Speculative Store Bypass is fixed in hardware. */
> +#define X86_FEATURE_AMD_PSFD		(13*32+28) /* "" Predictive Store Forward Disable */

... when you have this one. And this one is AMD-specific so you can just
as well call it X86_FEATURE_PSFD and remove the "".

>  
>  /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */
>  #define X86_FEATURE_DTHERM		(14*32+ 0) /* Digital Thermal Sensor */
> @@ -428,5 +429,6 @@
>  #define X86_BUG_TAA			X86_BUG(22) /* CPU is affected by TSX Async Abort(TAA) */
>  #define X86_BUG_ITLB_MULTIHIT		X86_BUG(23) /* CPU may incur MCE during certain page attribute changes */
>  #define X86_BUG_SRBDS			X86_BUG(24) /* CPU may leak RNG bits if not mitigated */
> +#define X86_BUG_PSF			X86_BUG(25) /* CPU is affected by Predictive Store Forwarding attack */

And I think you don't need this one either if we do a "light" controls
thing but lemme look at the rest first.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 1/5] x86/cpufeatures: Define feature bits to support mitigation of PSF

[Saripalli, RK]

On 4/9/2021 12:41 PM, Borislav Petkov wrote:
> On Tue, Apr 06, 2021 at 10:50:00AM -0500, Ramakrishna Saripalli wrote:
>> diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
>> index cc96e26d69f7..21e7f8d0d7d9 100644
>> --- a/arch/x86/include/asm/cpufeatures.h
>> +++ b/arch/x86/include/asm/cpufeatures.h
>> @@ -201,7 +201,7 @@
>>  #define X86_FEATURE_INVPCID_SINGLE	( 7*32+ 7) /* Effectively INVPCID && CR4.PCIDE=1 */
>>  #define X86_FEATURE_HW_PSTATE		( 7*32+ 8) /* AMD HW-PState */
>>  #define X86_FEATURE_PROC_FEEDBACK	( 7*32+ 9) /* AMD ProcFeedbackInterface */
>> -/* FREE!                                ( 7*32+10) */
>> +#define X86_FEATURE_PSFD		( 7*32+10) /* Predictive Store Forward Disable */
> 
> You don't need this one...

Boris, I added this bit so we could detect later that PSF is supported.
But since PSF is AMD specific for now, I guess I will go along with your suggestions.
> 
>>  #define X86_FEATURE_PTI			( 7*32+11) /* Kernel Page Table Isolation enabled */
>>  #define X86_FEATURE_RETPOLINE		( 7*32+12) /* "" Generic Retpoline mitigation for Spectre variant 2 */
>>  #define X86_FEATURE_RETPOLINE_AMD	( 7*32+13) /* "" AMD Retpoline mitigation for Spectre variant 2 */
>> @@ -309,6 +309,7 @@
>>  #define X86_FEATURE_AMD_SSBD		(13*32+24) /* "" Speculative Store Bypass Disable */
>>  #define X86_FEATURE_VIRT_SSBD		(13*32+25) /* Virtualized Speculative Store Bypass Disable */
>>  #define X86_FEATURE_AMD_SSB_NO		(13*32+26) /* "" Speculative Store Bypass is fixed in hardware. */
>> +#define X86_FEATURE_AMD_PSFD		(13*32+28) /* "" Predictive Store Forward Disable */
> 
> ... when you have this one. And this one is AMD-specific so you can just
> as well call it X86_FEATURE_PSFD and remove the "".
Ok
> 
>>  
>>  /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */
>>  #define X86_FEATURE_DTHERM		(14*32+ 0) /* Digital Thermal Sensor */
>> @@ -428,5 +429,6 @@
>>  #define X86_BUG_TAA			X86_BUG(22) /* CPU is affected by TSX Async Abort(TAA) */
>>  #define X86_BUG_ITLB_MULTIHIT		X86_BUG(23) /* CPU may incur MCE during certain page attribute changes */
>>  #define X86_BUG_SRBDS			X86_BUG(24) /* CPU may leak RNG bits if not mitigated */
>> +#define X86_BUG_PSF			X86_BUG(25) /* CPU is affected by Predictive Store Forwarding attack */
> 
> And I think you don't need this one either if we do a "light" controls
> thing but lemme look at the rest first.
> 
Ok.
> Thx.
> 

Re: [PATCH 1/5] x86/cpufeatures: Define feature bits to support mitigation of PSF

[Borislav Petkov]

On Fri, Apr 09, 2021 at 01:22:49PM -0500, Saripalli, RK wrote:
> > And I think you don't need this one either if we do a "light" controls
> > thing but lemme look at the rest first.

Ok, and what I mean with "lite" version is something like this below
which needs finishing and testing.

Initially, it could support the cmdline params:

predict_store_fwd={on,off,auto}

to give people the opportunity to experiment with the feature.

If it turns out that prctl and seccomp per-task toggling is needed then
sure, we can extend but I don't see the reason for a whole separate set
of options yet. Especially is ssbd already controls this.

AFAICT, of course and if I'm not missing some other aspect here.

Thx.

---
diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
index 2d11384dc9ab..226b73700f88 100644
--- a/arch/x86/kernel/cpu/amd.c
+++ b/arch/x86/kernel/cpu/amd.c
@@ -1165,3 +1165,22 @@ void set_dr_addr_mask(unsigned long mask, int dr)
 		break;
 	}
 }
+
+static int __init psf_cmdline(char *str)
+{
+	if (!boot_cpu_has(X86_FEATURE_PSFD))
+		return 0;
+
+	if (!str)
+		return -EINVAL;
+
+	if (!strcmp(str, "off")) {
+		x86_spec_ctrl_base |= SPEC_CTRL_PSFD;
+		setup_clear_cpu_cap(X86_FEATURE_PSFD);
+	}
+
+	return 0;
+}
+early_param("predict_store_fwd", psf_cmdline);
+
+

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 1/5] x86/cpufeatures: Define feature bits to support mitigation of PSF

[Saripalli, RK]

Boris, thank you.

On 4/9/2021 2:39 PM, Borislav Petkov wrote:
> On Fri, Apr 09, 2021 at 01:22:49PM -0500, Saripalli, RK wrote:
>>> And I think you don't need this one either if we do a "light" controls
>>> thing but lemme look at the rest first.
> 
> Ok, and what I mean with "lite" version is something like this below
> which needs finishing and testing.
> 
> Initially, it could support the cmdline params:
> 
> predict_store_fwd={on,off,auto}
> 
> to give people the opportunity to experiment with the feature.
> 
> If it turns out that prctl and seccomp per-task toggling is needed then
> sure, we can extend but I don't see the reason for a whole separate set
> of options yet. Especially is ssbd already controls this.
> 
> AFAICT, of course and if I'm not missing some other aspect here.
> 
> Thx.

Yes, these options should be fine for now.
Like you said, if we get the need to add prctl and seccomp, I can always do that later.

What do you think auto should default to?. 
In SSBD case, I believe auto defaults to prctl or seccomp.
Since we will not have that here, we should choose something for auto.


> 
> ---
> diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
> index 2d11384dc9ab..226b73700f88 100644
> --- a/arch/x86/kernel/cpu/amd.c
> +++ b/arch/x86/kernel/cpu/amd.c
> @@ -1165,3 +1165,22 @@ void set_dr_addr_mask(unsigned long mask, int dr)
>  		break;
>  	}
>  }
> +
> +static int __init psf_cmdline(char *str)
> +{
> +	if (!boot_cpu_has(X86_FEATURE_PSFD))
> +		return 0;
> +
> +	if (!str)
> +		return -EINVAL;
> +
> +	if (!strcmp(str, "off")) {
> +		x86_spec_ctrl_base |= SPEC_CTRL_PSFD;
> +		setup_clear_cpu_cap(X86_FEATURE_PSFD);
> +	}
> +
> +	return 0;
> +}
> +early_param("predict_store_fwd", psf_cmdline);
> +
> +
> 

All the other mitigation x86 mitigation code goes into kernel/cpu/bugs.c.
I think psf_cmdline() or equivalent also belongs there and not in kernel/cpu/amd.c.

Looking forward to your feedback.

Thanks,
RK

Re: [PATCH 1/5] x86/cpufeatures: Define feature bits to support mitigation of PSF

[Borislav Petkov]

On Fri, Apr 09, 2021 at 02:45:23PM -0500, Saripalli, RK wrote:
> Yes, these options should be fine for now.
> Like you said, if we get the need to add prctl and seccomp, I can always do that later.
> 
> What do you think auto should default to?. 
> In SSBD case, I believe auto defaults to prctl or seccomp.
> Since we will not have that here, we should choose something for auto.

Or not add it yet. Just have "on" and "off" for now.

Which begs the question should this be controllable by the mitigations=
switch too?

I wanna say, let's have people evaluate and play with it first and
we can add it to that switch later. As long as we don't change the
user-visible controls - if anything we'll be extending them later,
potentially - we should be fine usage-wise and from user visibility POV.

> All the other mitigation x86 mitigation code goes into kernel/cpu/bugs.c.
> I think psf_cmdline() or equivalent also belongs there and not in kernel/cpu/amd.c.

It being AMD-specific, it can dwell in amd.c initially.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 1/5] x86/cpufeatures: Define feature bits to support mitigation of PSF

[Saripalli, RK]

On 4/9/2021 3:19 PM, Borislav Petkov wrote:
> On Fri, Apr 09, 2021 at 02:45:23PM -0500, Saripalli, RK wrote:
>> Yes, these options should be fine for now.
>> Like you said, if we get the need to add prctl and seccomp, I can always do that later.
>>
>> What do you think auto should default to?. 
>> In SSBD case, I believe auto defaults to prctl or seccomp.
>> Since we will not have that here, we should choose something for auto.
> 
> Or not add it yet. Just have "on" and "off" for now.

OK

> 
> Which begs the question should this be controllable by the mitigations=
> switch too?

> 
> I wanna say, let's have people evaluate and play with it first and
> we can add it to that switch later. As long as we don't change the
> user-visible controls - if anything we'll be extending them later,
> potentially - we should be fine usage-wise and from user visibility POV.

Agreed. We can add this later.
> 
>> All the other mitigation x86 mitigation code goes into kernel/cpu/bugs.c.
>> I think psf_cmdline() or equivalent also belongs there and not in kernel/cpu/amd.c.
> 
> It being AMD-specific, it can dwell in amd.c initially.

OK
> 
> Thx.
> 

[PATCH 2/5] x86/speculation: Implement support for PSFD detection and reporting

[Ramakrishna Saripalli]

From: Ramakrishna Saripalli <rk.saripalli@amd.com>

All AMD processors that support Predictive Store Forwarding (PSF)
provide a CPUID bit to detect support for mitigation of this
feature. This bit is referred to as PSFD (Predictive Store Forwarding
Disable)

If CPU advertises PSFD:
- Advertise a generic feature (X86_FEATURE_PSFD) in Linux capability
flags to indicate PSF mitigation is available.

- Advertise X86_BUG_PSF in Linux bug bits to indicate that CPU is
affected by PSF security risks.

Signed-off-by: Ramakrishna Saripalli<rk.saripalli@amd.com>
---
 arch/x86/kernel/cpu/common.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index ab640abe26b6..4e604d58f3b3 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -893,6 +893,9 @@ static void init_speculation_control(struct cpuinfo_x86 *c)
 		set_cpu_cap(c, X86_FEATURE_MSR_SPEC_CTRL);
 		clear_cpu_cap(c, X86_FEATURE_VIRT_SSBD);
 	}
+
+	if (cpu_has(c, X86_FEATURE_AMD_PSFD))
+		set_cpu_cap(c, X86_FEATURE_PSFD);
 }
 
 void get_cpu_cap(struct cpuinfo_x86 *c)
@@ -1154,6 +1157,9 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
 	   !cpu_has(c, X86_FEATURE_AMD_SSB_NO))
 		setup_force_cpu_bug(X86_BUG_SPEC_STORE_BYPASS);
 
+	if (cpu_has(c, X86_FEATURE_AMD_PSFD))
+		setup_force_cpu_bug(X86_BUG_PSF);
+
 	if (ia32_cap & ARCH_CAP_IBRS_ALL)
 		setup_force_cpu_cap(X86_FEATURE_IBRS_ENHANCED);
 
-- 
2.25.1

[PATCH 3/5] x86/speculation: Introduce SPEC_CTRL_MSR bit for PSFD

[Ramakrishna Saripalli]

From: Ramakrishna Saripalli <rk.saripalli@amd.com>

All AMD processors that support PSF implement a bit in
SPEC_CTRL MSR (0x48) to disable or enable Predictive Store
Forwarding.

Signed-off-by: Ramakrishna Saripalli<rk.saripalli@amd.com>
---
 arch/x86/include/asm/msr-index.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 546d6ecf0a35..f569918c8754 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -51,6 +51,8 @@
 #define SPEC_CTRL_STIBP			BIT(SPEC_CTRL_STIBP_SHIFT)	/* STIBP mask */
 #define SPEC_CTRL_SSBD_SHIFT		2	   /* Speculative Store Bypass Disable bit */
 #define SPEC_CTRL_SSBD			BIT(SPEC_CTRL_SSBD_SHIFT)	/* Speculative Store Bypass Disable */
+#define SPEC_CTRL_PSFD_SHIFT		7
+#define SPEC_CTRL_PSFD			BIT(SPEC_CTRL_PSFD_SHIFT)	/* Predictive Store Forwarding Disable */
 
 #define MSR_IA32_PRED_CMD		0x00000049 /* Prediction Command */
 #define PRED_CMD_IBPB			BIT(0)	   /* Indirect Branch Prediction Barrier */
-- 
2.25.1

[PATCH 4/5] x86/speculation: Implement PSF mitigation support

[Ramakrishna Saripalli]

From: Ramakrishna Saripalli <rk.saripalli@amd.com>

Implement support for PSF mitigation via kernel parameters.

On AMD processors that support Predictive Store Forwarding(PSF),
the disabling of Speculative Store Bypass(SSB) will also disable PSF.

Two kernel parameters are being introduced to enable PSF
mitigation.

Behavior and implementation of these parameters is similar to
spec_control_bypass_disable kernel parameter.

nopsfd: Disable all mitigations for Predictive Store
		Forwarding.

	This parameter has no values.

psfd: This parameter has five possible values.
	on
		Unconditionally disable Predictive Store Forwarding.

	off
		Unconditionally enable Predictive Store Forwarding.

	auto
		Kernel detects whether CPU is vulnerable
		If CPU is not vulnerable, off is selected
		If CPU is vulnerable, default mitigation is
			KConfig dependent.

	prctl
		Control Predictive Store Forwarding per thread
		via prctl.

	seccomp
		Same as prctl above but all seccomp threads will
		disable PSF unless they opt out

	Default mitigation:
	If CONFIG_SECCOMP=y "seccomp" else "prctl"

Signed-off-by: Ramakrishna Saripalli<rk.saripalli@amd.com>
---
 arch/x86/include/asm/nospec-branch.h |   8 ++
 arch/x86/include/asm/spec-ctrl.h     |  12 ++
 arch/x86/include/asm/thread_info.h   |   2 +
 arch/x86/kernel/cpu/bugs.c           | 191 +++++++++++++++++++++++++++
 arch/x86/kernel/process.c            |  23 ++++
 include/linux/sched.h                |  15 +++
 include/uapi/linux/prctl.h           |   2 +
 7 files changed, 253 insertions(+)

diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
index cb9ad6b73973..7ac27aaef110 100644
--- a/arch/x86/include/asm/nospec-branch.h
+++ b/arch/x86/include/asm/nospec-branch.h
@@ -198,6 +198,14 @@ enum ssb_mitigation {
 	SPEC_STORE_BYPASS_SECCOMP,
 };
 
+/* The Predictive Store Forwarding disable variants */
+enum psf_mitigation {
+	PREDICTIVE_STORE_FORWARD_NONE,
+	PREDICTIVE_STORE_FORWARD_DISABLE,
+	PREDICTIVE_STORE_FORWARD_PRCTL,
+	PREDICTIVE_STORE_FORWARD_SECCOMP,
+};
+
 extern char __indirect_thunk_start[];
 extern char __indirect_thunk_end[];
 
diff --git a/arch/x86/include/asm/spec-ctrl.h b/arch/x86/include/asm/spec-ctrl.h
index 5393babc0598..eefab0707931 100644
--- a/arch/x86/include/asm/spec-ctrl.h
+++ b/arch/x86/include/asm/spec-ctrl.h
@@ -76,6 +76,18 @@ static inline u64 ssbd_tif_to_amd_ls_cfg(u64 tifn)
 	return (tifn & _TIF_SSBD) ? x86_amd_ls_cfg_ssbd_mask : 0ULL;
 }
 
+static inline u64 psfd_tif_to_spec_ctrl(u64 tifn)
+{
+	BUILD_BUG_ON(TIF_PSFD < SPEC_CTRL_PSFD_SHIFT);
+	return (tifn & _TIF_PSFD) >> (TIF_PSFD - SPEC_CTRL_PSFD_SHIFT);
+}
+
+static inline unsigned long psfd_spec_ctrl_to_tif(u64 spec_ctrl)
+{
+	BUILD_BUG_ON(TIF_PSFD < SPEC_CTRL_PSFD_SHIFT);
+	return (spec_ctrl & SPEC_CTRL_PSFD) << (TIF_PSFD - SPEC_CTRL_PSFD_SHIFT);
+}
+
 #ifdef CONFIG_SMP
 extern void speculative_store_bypass_ht_init(void);
 #else
diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thread_info.h
index 0d751d5da702..d65bf27956f3 100644
--- a/arch/x86/include/asm/thread_info.h
+++ b/arch/x86/include/asm/thread_info.h
@@ -80,6 +80,7 @@ struct thread_info {
 #define TIF_NEED_RESCHED	3	/* rescheduling necessary */
 #define TIF_SINGLESTEP		4	/* reenable singlestep on user return*/
 #define TIF_SSBD		5	/* Speculative store bypass disable */
+#define TIF_PSFD		8	/* Predictive Store Forward disable */
 #define TIF_SPEC_IB		9	/* Indirect branch speculation mitigation */
 #define TIF_SPEC_FORCE_UPDATE	10	/* Force speculation MSR update in context switch */
 #define TIF_USER_RETURN_NOTIFY	11	/* notify kernel of userspace return */
@@ -103,6 +104,7 @@ struct thread_info {
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
 #define _TIF_SINGLESTEP		(1 << TIF_SINGLESTEP)
 #define _TIF_SSBD		(1 << TIF_SSBD)
+#define _TIF_PSFD		(1 << TIF_PSFD)
 #define _TIF_SPEC_IB		(1 << TIF_SPEC_IB)
 #define _TIF_SPEC_FORCE_UPDATE	(1 << TIF_SPEC_FORCE_UPDATE)
 #define _TIF_USER_RETURN_NOTIFY	(1 << TIF_USER_RETURN_NOTIFY)
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index d41b70fe4918..1e0eca761759 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -38,6 +38,7 @@
 static void __init spectre_v1_select_mitigation(void);
 static void __init spectre_v2_select_mitigation(void);
 static void __init ssb_select_mitigation(void);
+static void __init psf_select_mitigation(void);
 static void __init l1tf_select_mitigation(void);
 static void __init mds_select_mitigation(void);
 static void __init mds_print_mitigation(void);
@@ -107,6 +108,7 @@ void __init check_bugs(void)
 	spectre_v1_select_mitigation();
 	spectre_v2_select_mitigation();
 	ssb_select_mitigation();
+	psf_select_mitigation();
 	l1tf_select_mitigation();
 	mds_select_mitigation();
 	taa_select_mitigation();
@@ -1195,6 +1197,129 @@ static void ssb_select_mitigation(void)
 		pr_info("%s\n", ssb_strings[ssb_mode]);
 }
 
+#undef pr_fmt
+#define pr_fmt(fmt)     "Predictive Store Forwarding: " fmt
+
+static enum psf_mitigation psf_mode __ro_after_init = PREDICTIVE_STORE_FORWARD_NONE;
+
+enum psf_mitigation_cmd {
+	PREDICTIVE_STORE_FORWARD_CMD_NONE,
+	PREDICTIVE_STORE_FORWARD_CMD_AUTO,
+	PREDICTIVE_STORE_FORWARD_CMD_ON,
+	PREDICTIVE_STORE_FORWARD_CMD_PRCTL,
+	PREDICTIVE_STORE_FORWARD_CMD_SECCOMP,
+};
+
+static const char * const psf_strings[] = {
+	[PREDICTIVE_STORE_FORWARD_NONE]		= "Vulnerable",
+	[PREDICTIVE_STORE_FORWARD_DISABLE]	= "Mitigation: Predictive Store Forwarding disabled",
+	[PREDICTIVE_STORE_FORWARD_PRCTL]	= "Mitigation: Predictive Store Forwarding disabled via prctl",
+	[PREDICTIVE_STORE_FORWARD_SECCOMP]	= "Mitigation: Predictive Store Forwarding disabled via prctl and seccomp",
+};
+
+static const struct {
+	const char *option;
+	enum psf_mitigation_cmd cmd;
+} psf_mitigation_options[]  __initconst = {
+	{ "auto",	PREDICTIVE_STORE_FORWARD_CMD_AUTO },    /* Platform decides */
+	{ "on",		PREDICTIVE_STORE_FORWARD_CMD_ON },      /* Disable Predictive Store Forwarding*/
+	{ "off",	PREDICTIVE_STORE_FORWARD_CMD_NONE },    /* Don't touch Predictive Store Forwarding */
+	{ "prctl",	PREDICTIVE_STORE_FORWARD_CMD_PRCTL },   /* Disable Predictive Store Forwarding via prctl */
+	{ "seccomp",	PREDICTIVE_STORE_FORWARD_CMD_SECCOMP }, /* Disable Predictive Store Forwarding via prctl and seccomp */
+};
+
+static enum psf_mitigation_cmd __init psf_parse_cmdline(void)
+{
+	enum psf_mitigation_cmd cmd = PREDICTIVE_STORE_FORWARD_CMD_AUTO;
+	char arg[20];
+	int ret, i;
+
+	if (cmdline_find_option_bool(boot_command_line, "nopsfd") ||
+	    cpu_mitigations_off()) {
+		return PREDICTIVE_STORE_FORWARD_CMD_NONE;
+	} else {
+		ret = cmdline_find_option(boot_command_line, "psfd",
+					  arg, sizeof(arg));
+		if (ret < 0)
+			return PREDICTIVE_STORE_FORWARD_CMD_AUTO;
+
+		for (i = 0; i < ARRAY_SIZE(psf_mitigation_options); i++) {
+			if (!match_option(arg, ret, psf_mitigation_options[i].option))
+				continue;
+
+			cmd = psf_mitigation_options[i].cmd;
+			break;
+		}
+
+		if (i >= ARRAY_SIZE(psf_mitigation_options)) {
+			pr_err("unknown option (%s). Switching to AUTO select\n", arg);
+			return PREDICTIVE_STORE_FORWARD_CMD_AUTO;
+		}
+	}
+
+	return cmd;
+}
+
+static enum psf_mitigation __init __psf_select_mitigation(void)
+{
+	enum psf_mitigation mode = PREDICTIVE_STORE_FORWARD_NONE;
+	enum psf_mitigation_cmd cmd;
+
+	if (!boot_cpu_has(X86_FEATURE_PSFD))
+		return mode;
+
+	cmd = psf_parse_cmdline();
+	if (!boot_cpu_has_bug(X86_BUG_PSF) &&
+	    (cmd == PREDICTIVE_STORE_FORWARD_CMD_NONE ||
+	     cmd == PREDICTIVE_STORE_FORWARD_CMD_AUTO))
+		return mode;
+
+	switch (cmd) {
+	case PREDICTIVE_STORE_FORWARD_CMD_AUTO:
+	case PREDICTIVE_STORE_FORWARD_CMD_SECCOMP:
+		/*
+		 * Choose prctl+seccomp as the default mode if seccomp is
+		 * enabled.
+		 */
+		if (IS_ENABLED(CONFIG_SECCOMP))
+			mode = PREDICTIVE_STORE_FORWARD_SECCOMP;
+		else
+			mode = PREDICTIVE_STORE_FORWARD_PRCTL;
+		break;
+	case PREDICTIVE_STORE_FORWARD_CMD_ON:
+		mode = PREDICTIVE_STORE_FORWARD_DISABLE;
+		break;
+	case PREDICTIVE_STORE_FORWARD_CMD_PRCTL:
+		mode = PREDICTIVE_STORE_FORWARD_PRCTL;
+		break;
+	case PREDICTIVE_STORE_FORWARD_CMD_NONE:
+		break;
+	}
+
+	x86_spec_ctrl_mask |= SPEC_CTRL_PSFD;
+
+	/*
+	 * Disabling SSB automatically disables PSF on AMD processors.
+	 */
+	if (ssb_mode == SPEC_STORE_BYPASS_DISABLE &&
+	    boot_cpu_has(X86_FEATURE_AMD_PSFD))
+		mode = PREDICTIVE_STORE_FORWARD_DISABLE;
+
+	if (mode == PREDICTIVE_STORE_FORWARD_DISABLE) {
+		x86_spec_ctrl_base |= SPEC_CTRL_PSFD;
+		wrmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base);
+	}
+
+	return mode;
+}
+
+static void psf_select_mitigation(void)
+{
+	psf_mode = __psf_select_mitigation();
+	if (boot_cpu_has_bug(X86_BUG_PSF))
+		pr_info("%s\n", psf_strings[psf_mode]);
+}
+
 #undef pr_fmt
 #define pr_fmt(fmt)     "Speculation prctl: " fmt
 
@@ -1215,6 +1340,45 @@ static void task_update_spec_tif(struct task_struct *tsk)
 		speculation_ctrl_update_current();
 }
 
+static int psf_prctl_set(struct task_struct *task, unsigned long ctrl)
+{
+	if (psf_mode != PREDICTIVE_STORE_FORWARD_PRCTL &&
+	    psf_mode != PREDICTIVE_STORE_FORWARD_SECCOMP)
+		return -ENXIO;
+
+	switch (ctrl) {
+	case PR_SPEC_ENABLE:
+		/* If speculation is force disabled, enable is not allowed */
+		if (task_spec_psf_force_disable(task))
+			return -EPERM;
+		task_clear_spec_psf_disable(task);
+		task_clear_spec_psf_noexec(task);
+		task_update_spec_tif(task);
+		break;
+	case PR_SPEC_DISABLE:
+		task_set_spec_psf_disable(task);
+		task_clear_spec_psf_noexec(task);
+		task_update_spec_tif(task);
+		break;
+	case PR_SPEC_FORCE_DISABLE:
+		task_set_spec_psf_disable(task);
+		task_set_spec_psf_force_disable(task);
+		task_clear_spec_psf_noexec(task);
+		task_update_spec_tif(task);
+		break;
+	case PR_SPEC_DISABLE_NOEXEC:
+		if (task_spec_psf_force_disable(task))
+			return -EPERM;
+		task_set_spec_psf_disable(task);
+		task_set_spec_psf_noexec(task);
+		task_update_spec_tif(task);
+		break;
+	default:
+		return -ERANGE;
+	}
+	return 0;
+}
+
 static int ssb_prctl_set(struct task_struct *task, unsigned long ctrl)
 {
 	if (ssb_mode != SPEC_STORE_BYPASS_PRCTL &&
@@ -1324,6 +1488,8 @@ int arch_prctl_spec_ctrl_set(struct task_struct *task, unsigned long which,
 		return ssb_prctl_set(task, ctrl);
 	case PR_SPEC_INDIRECT_BRANCH:
 		return ib_prctl_set(task, ctrl);
+	case PR_SPEC_PSF:
+		return psf_prctl_set(task, ctrl);
 	default:
 		return -ENODEV;
 	}
@@ -1334,12 +1500,35 @@ void arch_seccomp_spec_mitigate(struct task_struct *task)
 {
 	if (ssb_mode == SPEC_STORE_BYPASS_SECCOMP)
 		ssb_prctl_set(task, PR_SPEC_FORCE_DISABLE);
+	if (psf_mode == PREDICTIVE_STORE_FORWARD_SECCOMP)
+		psf_prctl_set(task, PR_SPEC_FORCE_DISABLE);
 	if (spectre_v2_user_ibpb == SPECTRE_V2_USER_SECCOMP ||
 	    spectre_v2_user_stibp == SPECTRE_V2_USER_SECCOMP)
 		ib_prctl_set(task, PR_SPEC_FORCE_DISABLE);
 }
 #endif
 
+static int psf_prctl_get(struct task_struct *task)
+{
+	switch (psf_mode) {
+	case PREDICTIVE_STORE_FORWARD_DISABLE:
+		return PR_SPEC_DISABLE;
+	case PREDICTIVE_STORE_FORWARD_SECCOMP:
+	case PREDICTIVE_STORE_FORWARD_PRCTL:
+		if (task_spec_psf_force_disable(task))
+			return PR_SPEC_PRCTL | PR_SPEC_FORCE_DISABLE;
+		if (task_spec_psf_noexec(task))
+			return PR_SPEC_PRCTL | PR_SPEC_DISABLE_NOEXEC;
+		if (task_spec_psf_disable(task))
+			return PR_SPEC_PRCTL | PR_SPEC_DISABLE;
+		return PR_SPEC_PRCTL | PR_SPEC_ENABLE;
+	default:
+		if (boot_cpu_has_bug(X86_BUG_PSF))
+			return PR_SPEC_ENABLE;
+		return PR_SPEC_NOT_AFFECTED;
+	}
+}
+
 static int ssb_prctl_get(struct task_struct *task)
 {
 	switch (ssb_mode) {
@@ -1390,6 +1579,8 @@ int arch_prctl_spec_ctrl_get(struct task_struct *task, unsigned long which)
 		return ssb_prctl_get(task);
 	case PR_SPEC_INDIRECT_BRANCH:
 		return ib_prctl_get(task);
+	case PR_SPEC_PSF:
+		return psf_prctl_get(task);
 	default:
 		return -ENODEV;
 	}
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index 9c214d7085a4..5916b02e57f7 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -322,6 +322,18 @@ void arch_setup_new_exec(void)
 		task_clear_spec_ssb_noexec(current);
 		speculation_ctrl_update(task_thread_info(current)->flags);
 	}
+
+	/*
+	 * Don't inherit TIF_PSFD across exec boundary when
+	 * PR_SPEC_DISABLE_NOEXEC is used.
+	 */
+	if (test_thread_flag(TIF_PSFD) &&
+	    task_spec_psf_noexec(current)) {
+		clear_thread_flag(TIF_PSFD);
+		task_clear_spec_psf_disable(current);
+		task_clear_spec_psf_noexec(current);
+		speculation_ctrl_update(task_thread_info(current)->flags);
+	}
 }
 
 #ifdef CONFIG_X86_IOPL_IOPERM
@@ -547,6 +559,12 @@ static __always_inline void __speculation_ctrl_update(unsigned long tifp,
 		msr |= ssbd_tif_to_spec_ctrl(tifn);
 	}
 
+	/* Handle change of TIF_PSFD if CPU supports PSFD */
+	if (static_cpu_has(X86_FEATURE_PSFD)) {
+		updmsr |= !!(tif_diff & _TIF_PSFD);
+		msr |= psfd_tif_to_spec_ctrl(tifn);
+	}
+
 	/* Only evaluate TIF_SPEC_IB if conditional STIBP is enabled. */
 	if (IS_ENABLED(CONFIG_SMP) &&
 	    static_branch_unlikely(&switch_to_cond_stibp)) {
@@ -570,6 +588,11 @@ static unsigned long speculation_ctrl_update_tif(struct task_struct *tsk)
 			set_tsk_thread_flag(tsk, TIF_SPEC_IB);
 		else
 			clear_tsk_thread_flag(tsk, TIF_SPEC_IB);
+
+		if (task_spec_psf_disable(tsk))
+			set_tsk_thread_flag(tsk, TIF_PSFD);
+		else
+			clear_tsk_thread_flag(tsk, TIF_PSFD);
 	}
 	/* Return the updated threadinfo flags*/
 	return task_thread_info(tsk)->flags;
diff --git a/include/linux/sched.h b/include/linux/sched.h
index ef00bb22164c..54ac4cfb1086 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1622,6 +1622,10 @@ static inline bool is_percpu_thread(void)
 #define PFA_SPEC_IB_DISABLE		5	/* Indirect branch speculation restricted */
 #define PFA_SPEC_IB_FORCE_DISABLE	6	/* Indirect branch speculation permanently restricted */
 #define PFA_SPEC_SSB_NOEXEC		7	/* Speculative Store Bypass clear on execve() */
+#define PFA_SPEC_PSF_DISABLE		8	/* Predictive Store Forwarding disabled */
+#define PFA_SPEC_PSF_FORCE_DISABLE	9	/* Predictive Store Forwarding force disabled */
+#define PFA_SPEC_PSF_NOEXEC		10	/* Predictive Store Forwarding clear on execve */
+
 
 #define TASK_PFA_TEST(name, func)					\
 	static inline bool task_##func(struct task_struct *p)		\
@@ -1664,6 +1668,17 @@ TASK_PFA_CLEAR(SPEC_IB_DISABLE, spec_ib_disable)
 TASK_PFA_TEST(SPEC_IB_FORCE_DISABLE, spec_ib_force_disable)
 TASK_PFA_SET(SPEC_IB_FORCE_DISABLE, spec_ib_force_disable)
 
+TASK_PFA_TEST(SPEC_PSF_DISABLE, spec_psf_disable)
+TASK_PFA_SET(SPEC_PSF_DISABLE, spec_psf_disable)
+TASK_PFA_CLEAR(SPEC_PSF_DISABLE, spec_psf_disable)
+
+TASK_PFA_TEST(SPEC_PSF_NOEXEC, spec_psf_noexec)
+TASK_PFA_SET(SPEC_PSF_NOEXEC, spec_psf_noexec)
+TASK_PFA_CLEAR(SPEC_PSF_NOEXEC, spec_psf_noexec)
+
+TASK_PFA_TEST(SPEC_PSF_FORCE_DISABLE, spec_psf_force_disable)
+TASK_PFA_SET(SPEC_PSF_FORCE_DISABLE, spec_psf_force_disable)
+
 static inline void
 current_restore_flags(unsigned long orig_flags, unsigned long flags)
 {
diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h
index 667f1aed091c..eb8378ccf580 100644
--- a/include/uapi/linux/prctl.h
+++ b/include/uapi/linux/prctl.h
@@ -213,6 +213,8 @@ struct prctl_mm_map {
 /* Speculation control variants */
 # define PR_SPEC_STORE_BYPASS		0
 # define PR_SPEC_INDIRECT_BRANCH	1
+# define PR_SPEC_PSF			2 /* Predictive Store Forwarding */
+
 /* Return and control values for PR_SET/GET_SPECULATION_CTRL */
 # define PR_SPEC_NOT_AFFECTED		0
 # define PR_SPEC_PRCTL			(1UL << 0)
-- 
2.25.1

[PATCH 5/5] x86/speculation: Add PSF mitigation kernel parameters

[Ramakrishna Saripalli]

From: Ramakrishna Saripalli <rk.saripalli@amd.com>

PSF mitigation introduces new kernel parameters.

The kernel parameters for PSF mitigation are modeled
after spec_store_bypass_disable.

Signed-off-by: Ramakrishna Saripalli<rk.saripalli@amd.com>
---
 .../admin-guide/kernel-parameters.txt         | 45 +++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 04545725f187..68dfde77a87d 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2876,6 +2876,7 @@
 					       nospectre_v2 [X86,PPC,S390,ARM64]
 					       spectre_v2_user=off [X86]
 					       spec_store_bypass_disable=off [X86,PPC]
+					       psfd=off [X86]
 					       ssbd=force-off [ARM64]
 					       l1tf=off [X86]
 					       mds=off [X86]
@@ -3243,6 +3244,8 @@
 
 	nohugeiomap	[KNL,X86,PPC,ARM64] Disable kernel huge I/O mappings.
 
+	nopsfd          [HW,X86] Disable mitigation for Predictive Store Forwarding.
+
 	nosmt		[KNL,S390] Disable symmetric multithreading (SMT).
 			Equivalent to smt=1.
 
@@ -4002,6 +4005,48 @@
 			that).
 			Format: <bool>
 
+        psfd=		[HW,X86]
+                        Predictive Store Forwarding Disable control
+
+                        Certain AMD processors feature a new technology called Predictive
+                        Store Forwarding. This feature is designed to improve the
+                        performance of code execution by predicting dependencies
+                        between loads and stores.
+
+                        Modern processors implement techniques to optimize the
+                        execution of a load instruction to an address that was
+                        recently written by a store instruction.
+
+                        PSF expands on the above by speculating on the relationship
+                        between loads and stores without waiting for address
+                        calculation to complete. With PSF, CPU learns over time the
+                        relationship between loads and stores.
+
+                        Incorrect PSF predictions can occur for various reasons.
+                        Please see the AMD PSF whitepaper for more information.
+
+                        All AMD processors that implement PSF also provide ability
+                        to control mitigation of PSF.
+
+                        Following options are provided to control PSF mitigation.
+
+                        The options are:
+                        on      - Unconditionally disable Speculative Store Bypass
+                        off     - Unconditionally enable Speculative Store Bypass
+                        auto    - Kernel detects whether the CPU is vulnerable.
+                                  If the CPU is not vulnerable, off is selected.
+                                  If the CPU is vulnerable, default mitigation is
+                                  KConfig dependent.
+                        prctl   - Control Predictive Store Forwarding per thread
+                                  via prctl. Predictive Store Forwarding is enabled
+                                  per process by default. The state of the control
+                                  is inherited on fork.
+                        seccomp - Same as prctl above but all seccomp threads will
+                                  disable PSF unless they opt out.
+
+                        Default mitigations:
+                        [X86] If CONFIG_SECCOMP=y "seccomp" else "prctl"
+
 	psi=		[KNL] Enable or disable pressure stall information
 			tracking.
 			Format: <bool>
-- 
2.25.1

Re: [PATCH 0/5] Introduce support for PSF mitigation

[Borislav Petkov]

On Tue, Apr 06, 2021 at 10:49:59AM -0500, Ramakrishna Saripalli wrote:
> From: Ramakrishna Saripalli <rk.saripalli@amd.com>
> 
> Predictive Store Forwarding:
> AMD Zen3 processors feature a new technology called
> Predictive Store Forwarding (PSF).
> 
> <TODO:Insert link to AMD PSF whitepaper>

You probably should upload it here:

https://bugzilla.kernel.org/show_bug.cgi?id=206537

instead and then point to it in the docs. And by "docs" I mean, putting
all that nice text below which I've snipped in here:

Documentation/admin-guide/hw-vuln/

for future reference. Look at the docs there for an idea how to
structure it.

I have come to appreciate our documentation a lot these days when I've
forgotten all about those nightmares but needed to refresh them back in
my L1.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 0/5] Introduce support for PSF mitigation

[Josh Poimboeuf]

On Tue, Apr 06, 2021 at 10:49:59AM -0500, Ramakrishna Saripalli wrote:
> Because PSF speculation is limited to the current program context,
> the impact of bad PSF speculation is very similar to that of
> Speculative Store Bypass (Spectre v4)
> 
> Predictive Store Forwarding controls:
> There are two hardware control bits which influence the PSF feature:
> - MSR 48h bit 2 – Speculative Store Bypass (SSBD)
> - MSR 48h bit 7 – Predictive Store Forwarding Disable (PSFD)
> 
> The PSF feature is disabled if either of these bits are set.  These bits
> are controllable on a per-thread basis in an SMT system. By default, both
> SSBD and PSFD are 0 meaning that the speculation features are enabled.
> 
> While the SSBD bit disables PSF and speculative store bypass, PSFD only
> disables PSF.
> 
> PSFD may be desirable for software which is concerned with the
> speculative behavior of PSF but desires a smaller performance impact than
> setting SSBD.

Hi Ramakrishna,

Is there a realistic scenario where an application would want to disable
PSF, but not disable SSB?

Maybe I'm missing something, but I'd presume an application would either
care about this class of attacks, or not.

-- 
Josh

Re: [PATCH 0/5] Introduce support for PSF mitigation

[Saripalli, RK]

Josh, thank you for taking the time to review the patches.

On 4/7/2021 5:39 PM, Josh Poimboeuf wrote:
> On Tue, Apr 06, 2021 at 10:49:59AM -0500, Ramakrishna Saripalli wrote:
>> Because PSF speculation is limited to the current program context,
>> the impact of bad PSF speculation is very similar to that of
>> Speculative Store Bypass (Spectre v4)
>>
>> Predictive Store Forwarding controls:
>> There are two hardware control bits which influence the PSF feature:
>> - MSR 48h bit 2 – Speculative Store Bypass (SSBD)
>> - MSR 48h bit 7 – Predictive Store Forwarding Disable (PSFD)
>>
>> The PSF feature is disabled if either of these bits are set.  These bits
>> are controllable on a per-thread basis in an SMT system. By default, both
>> SSBD and PSFD are 0 meaning that the speculation features are enabled.
>>
>> While the SSBD bit disables PSF and speculative store bypass, PSFD only
>> disables PSF.
>>
>> PSFD may be desirable for software which is concerned with the
>> speculative behavior of PSF but desires a smaller performance impact than
>> setting SSBD.
> 
> Hi Ramakrishna,
> 
> Is there a realistic scenario where an application would want to disable
> PSF, but not disable SSB?

It is possible most applications have been reviewed and scrubbed for SSB-type attacks but PSF-type issues may not have been looked at yet.
This may be one of the cases where SSB is enabled but PSF is disabled until the application(s) are scrubbed for the same.

In certain cases, disabling PSF may have less impact performance-wise than disabling SSB.


> 
> Maybe I'm missing something, but I'd presume an application would either
> care about this class of attacks, or not.
> 

Re: [PATCH 0/5] Introduce support for PSF mitigation

[Borislav Petkov]

On Thu, Apr 08, 2021 at 09:56:47AM -0500, Saripalli, RK wrote:
> It is possible most applications have been reviewed and scrubbed for
> SSB-type attacks but PSF-type issues may not have been looked at yet.
> This may be one of the cases where SSB is enabled but PSF is disabled
> until the application(s) are scrubbed for the same.

Right, and for that I think we could do a slimmer version of the psfd=
toggle - no prctl and seccomp stuff - just the cmdline disable thing to
keep this simpler.

Btw "psfd=" is maybe too short and cryptic. It would probably be more
user-friendly if it were called:

predict_store_fwd={on,off,...}

or so.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 0/5] Introduce support for PSF mitigation

[Josh Poimboeuf]

On Thu, Apr 08, 2021 at 09:56:47AM -0500, Saripalli, RK wrote:
> Josh, thank you for taking the time to review the patches.
> 
> On 4/7/2021 5:39 PM, Josh Poimboeuf wrote:
> > On Tue, Apr 06, 2021 at 10:49:59AM -0500, Ramakrishna Saripalli wrote:
> >> Because PSF speculation is limited to the current program context,
> >> the impact of bad PSF speculation is very similar to that of
> >> Speculative Store Bypass (Spectre v4)
> >>
> >> Predictive Store Forwarding controls:
> >> There are two hardware control bits which influence the PSF feature:
> >> - MSR 48h bit 2 – Speculative Store Bypass (SSBD)
> >> - MSR 48h bit 7 – Predictive Store Forwarding Disable (PSFD)
> >>
> >> The PSF feature is disabled if either of these bits are set.  These bits
> >> are controllable on a per-thread basis in an SMT system. By default, both
> >> SSBD and PSFD are 0 meaning that the speculation features are enabled.
> >>
> >> While the SSBD bit disables PSF and speculative store bypass, PSFD only
> >> disables PSF.
> >>
> >> PSFD may be desirable for software which is concerned with the
> >> speculative behavior of PSF but desires a smaller performance impact than
> >> setting SSBD.
> > 
> > Hi Ramakrishna,
> > 
> > Is there a realistic scenario where an application would want to disable
> > PSF, but not disable SSB?
> 
> It is possible most applications have been reviewed and scrubbed for
> SSB-type attacks but PSF-type issues may not have been looked at yet.

It's "possible", but is it realistic?  As far as I know, SSB is
impractical to scrub an application for.

Do we know of any real-world cases where this option is needed?

-- 
Josh

Re: [PATCH 0/5] Introduce support for PSF mitigation

[Saripalli, RK]

Josh, PSF being new, may be in the future someone will find something new.

This is really extra precaution. 



On 4/9/2021 11:45 AM, Josh Poimboeuf wrote:
> On Thu, Apr 08, 2021 at 09:56:47AM -0500, Saripalli, RK wrote:
>> Josh, thank you for taking the time to review the patches.
>>
>> On 4/7/2021 5:39 PM, Josh Poimboeuf wrote:
>>> On Tue, Apr 06, 2021 at 10:49:59AM -0500, Ramakrishna Saripalli wrote:
>>>> Because PSF speculation is limited to the current program context,
>>>> the impact of bad PSF speculation is very similar to that of
>>>> Speculative Store Bypass (Spectre v4)
>>>>
>>>> Predictive Store Forwarding controls:
>>>> There are two hardware control bits which influence the PSF feature:
>>>> - MSR 48h bit 2 – Speculative Store Bypass (SSBD)
>>>> - MSR 48h bit 7 – Predictive Store Forwarding Disable (PSFD)
>>>>
>>>> The PSF feature is disabled if either of these bits are set.  These bits
>>>> are controllable on a per-thread basis in an SMT system. By default, both
>>>> SSBD and PSFD are 0 meaning that the speculation features are enabled.
>>>>
>>>> While the SSBD bit disables PSF and speculative store bypass, PSFD only
>>>> disables PSF.
>>>>
>>>> PSFD may be desirable for software which is concerned with the
>>>> speculative behavior of PSF but desires a smaller performance impact than
>>>> setting SSBD.
>>>
>>> Hi Ramakrishna,
>>>
>>> Is there a realistic scenario where an application would want to disable
>>> PSF, but not disable SSB?
>>
>> It is possible most applications have been reviewed and scrubbed for
>> SSB-type attacks but PSF-type issues may not have been looked at yet.
> 
> It's "possible", but is it realistic?  As far as I know, SSB is
> impractical to scrub an application for.
> 
> Do we know of any real-world cases where this option is needed?
> 

[PATCH 5/5] x86/speculation: Add PSF mitigation kernel parameters

[Ramakrishna Saripalli]

From: Ramakrishna Saripalli <rk.saripalli@amd.com>

PSF mitigation introduces new kernel parameters.

The kernel parameters for PSF mitigation are modeled
after spec_store_bypass_disable.

Signed-off-by: Ramakrishna Saripalli<rk.saripalli@amd.com>
---
 .../admin-guide/kernel-parameters.txt         | 45 +++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 04545725f187..68dfde77a87d 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2876,6 +2876,7 @@
 					       nospectre_v2 [X86,PPC,S390,ARM64]
 					       spectre_v2_user=off [X86]
 					       spec_store_bypass_disable=off [X86,PPC]
+					       psfd=off [X86]
 					       ssbd=force-off [ARM64]
 					       l1tf=off [X86]
 					       mds=off [X86]
@@ -3243,6 +3244,8 @@
 
 	nohugeiomap	[KNL,X86,PPC,ARM64] Disable kernel huge I/O mappings.
 
+	nopsfd          [HW,X86] Disable mitigation for Predictive Store Forwarding.
+
 	nosmt		[KNL,S390] Disable symmetric multithreading (SMT).
 			Equivalent to smt=1.
 
@@ -4002,6 +4005,48 @@
 			that).
 			Format: <bool>
 
+        psfd=		[HW,X86]
+                        Predictive Store Forwarding Disable control
+
+                        Certain AMD processors feature a new technology called Predictive
+                        Store Forwarding. This feature is designed to improve the
+                        performance of code execution by predicting dependencies
+                        between loads and stores.
+
+                        Modern processors implement techniques to optimize the
+                        execution of a load instruction to an address that was
+                        recently written by a store instruction.
+
+                        PSF expands on the above by speculating on the relationship
+                        between loads and stores without waiting for address
+                        calculation to complete. With PSF, CPU learns over time the
+                        relationship between loads and stores.
+
+                        Incorrect PSF predictions can occur for various reasons.
+                        Please see the AMD PSF whitepaper for more information.
+
+                        All AMD processors that implement PSF also provide ability
+                        to control mitigation of PSF.
+
+                        Following options are provided to control PSF mitigation.
+
+                        The options are:
+                        on      - Unconditionally disable Speculative Store Bypass
+                        off     - Unconditionally enable Speculative Store Bypass
+                        auto    - Kernel detects whether the CPU is vulnerable.
+                                  If the CPU is not vulnerable, off is selected.
+                                  If the CPU is vulnerable, default mitigation is
+                                  KConfig dependent.
+                        prctl   - Control Predictive Store Forwarding per thread
+                                  via prctl. Predictive Store Forwarding is enabled
+                                  per process by default. The state of the control
+                                  is inherited on fork.
+                        seccomp - Same as prctl above but all seccomp threads will
+                                  disable PSF unless they opt out.
+
+                        Default mitigations:
+                        [X86] If CONFIG_SECCOMP=y "seccomp" else "prctl"
+
 	psi=		[KNL] Enable or disable pressure stall information
 			tracking.
 			Format: <bool>
-- 
2.25.1

Re: [PATCH 5/5] x86/speculation: Add PSF mitigation kernel parameters

[Randy Dunlap]

On 4/7/21 5:50 AM, Ramakrishna Saripalli wrote:
> From: Ramakrishna Saripalli <rk.saripalli@amd.com>
> 
> PSF mitigation introduces new kernel parameters.
> 
> The kernel parameters for PSF mitigation are modeled
> after spec_store_bypass_disable.

Maybe too much copy-pasta. See below.

> 
> Signed-off-by: Ramakrishna Saripalli<rk.saripalli@amd.com>
> ---
>  .../admin-guide/kernel-parameters.txt         | 45 +++++++++++++++++++
>  1 file changed, 45 insertions(+)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 04545725f187..68dfde77a87d 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -2876,6 +2876,7 @@
>  					       nospectre_v2 [X86,PPC,S390,ARM64]
>  					       spectre_v2_user=off [X86]
>  					       spec_store_bypass_disable=off [X86,PPC]
> +					       psfd=off [X86]
>  					       ssbd=force-off [ARM64]
>  					       l1tf=off [X86]
>  					       mds=off [X86]
> @@ -3243,6 +3244,8 @@
>  
>  	nohugeiomap	[KNL,X86,PPC,ARM64] Disable kernel huge I/O mappings.
>  
> +	nopsfd          [HW,X86] Disable mitigation for Predictive Store Forwarding.
> +
>  	nosmt		[KNL,S390] Disable symmetric multithreading (SMT).
>  			Equivalent to smt=1.
>  
> @@ -4002,6 +4005,48 @@
>  			that).
>  			Format: <bool>
>  
> +        psfd=		[HW,X86]
> +                        Predictive Store Forwarding Disable control
> +
> +                        Certain AMD processors feature a new technology called Predictive
> +                        Store Forwarding. This feature is designed to improve the
> +                        performance of code execution by predicting dependencies
> +                        between loads and stores.
> +
> +                        Modern processors implement techniques to optimize the
> +                        execution of a load instruction to an address that was
> +                        recently written by a store instruction.
> +
> +                        PSF expands on the above by speculating on the relationship
> +                        between loads and stores without waiting for address
> +                        calculation to complete. With PSF, CPU learns over time the
> +                        relationship between loads and stores.
> +
> +                        Incorrect PSF predictions can occur for various reasons.
> +                        Please see the AMD PSF whitepaper for more information.
> +
> +                        All AMD processors that implement PSF also provide ability
> +                        to control mitigation of PSF.
> +
> +                        Following options are provided to control PSF mitigation.
> +
> +                        The options are:
> +                        on      - Unconditionally disable Speculative Store Bypass

			                                     PSF.

> +                        off     - Unconditionally enable Speculative Store Bypass

			                                    PSF.

> +                        auto    - Kernel detects whether the CPU is vulnerable.
> +                                  If the CPU is not vulnerable, off is selected.
> +                                  If the CPU is vulnerable, default mitigation is
> +                                  KConfig dependent.
> +                        prctl   - Control Predictive Store Forwarding per thread
> +                                  via prctl. Predictive Store Forwarding is enabled
> +                                  per process by default. The state of the control
> +                                  is inherited on fork.
> +                        seccomp - Same as prctl above but all seccomp threads will
> +                                  disable PSF unless they opt out.
> +
> +                        Default mitigations:
> +                        [X86] If CONFIG_SECCOMP=y "seccomp" else "prctl"
> +
>  	psi=		[KNL] Enable or disable pressure stall information
>  			tracking.
>  			Format: <bool>
> 


-- 
~Randy