[PATCH v5] x86/mce: Check for writes ignored in MCA_STATUS register

[PATCH v5] x86/mce: Check for writes ignored in MCA_STATUS register

[Smita Koralahalli]

According to Section 2.1.16.3 under HWCR[McStatusWrEn] in "PPR for AMD
Family 19h, Model 01h, Revision B1 Processors - 55898 Rev 0.35 - Feb 5,
2021", the status register may sometimes enforce write ignored behavior
independent of the value of HWCR[McStatusWrEn] depending on the platform
settings.

Hence, evaluate for writes ignored for MCA_STATUS to determine if hardware
error injection is possible. Perform this evaluation early during module
init and store the result in the static "hw_injection_possible" variable.
Query this variable for subsequent error injections and limit checking for
MCA_STATUS only once on driver init.

Avoid exporting mce_flags into modules and check for smca feature from
'X86_FEATURE_SMCA' flag directly.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
Signed-off-by: Smita Koralahalli <Smita.KoralahalliChannabasappa@amd.com>
---
Link:
https://lkml.kernel.org/r/20220214233640.70510-2-Smita.KoralahalliChannabasappa@amd.com
v2:
	msr_ops -> mca_msr_reg().
	simulation -> injection.
	pr_info() -> pr_err().
	Aligned on ",".
v3:
	Removed "x86/mce: Use mca_msr_reg() in prepare_msrs()" patch.
	and made changes on the existing MCx_{STATUS, ADDR, MISC} macros.
v4:
	Simplified the code by just checking for writes ignored behavior in
	MCA_STATUS register.
	Introduced prepare_mca_status() and performed writes ignored checks
	inside the function.
	Rephrased error message.
v5:
	Evaluated for writes ignored early and only once during module init.
	Introduced "hw_injection_possible" variable to store the result of
	writes ignored behavior of MCA_STATUS. This variable is checked
	before performing subsequent hw error injections.
---
 arch/x86/kernel/cpu/mce/inject.c   | 50 +++++++++++++++++++++++++++++-
 arch/x86/kernel/cpu/mce/internal.h |  2 +-
 2 files changed, 50 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/mce/inject.c b/arch/x86/kernel/cpu/mce/inject.c
index 5fbd7ffb3233..ac5582691530 100644
--- a/arch/x86/kernel/cpu/mce/inject.c
+++ b/arch/x86/kernel/cpu/mce/inject.c
@@ -33,6 +33,8 @@
 
 #include "internal.h"
 
+static bool hw_injection_possible = true;
+
 /*
  * Collect all the MCi_XXX settings
  */
@@ -339,6 +341,8 @@ static int __set_inj(const char *buf)
 
 	for (i = 0; i < N_INJ_TYPES; i++) {
 		if (!strncmp(flags_options[i], buf, strlen(flags_options[i]))) {
+			if (i > SW_INJ && !hw_injection_possible)
+				continue;
 			inj_type = i;
 			return 0;
 		}
@@ -376,7 +380,11 @@ static ssize_t flags_write(struct file *filp, const char __user *ubuf,
 
 	err = __set_inj(__buf);
 	if (err) {
-		pr_err("%s: Invalid flags value: %s\n", __func__, __buf);
+		pr_err("%s: Invalid flags value: %s: %s\n", __func__,
+			(!hw_injection_possible
+			  ? " (SW-only injection possible on this platform)"
+			  : ""),
+		       __buf);
 		return err;
 	}
 
@@ -501,6 +509,8 @@ static void do_inject(void)
 	unsigned int cpu = i_mce.extcpu;
 	u8 b = i_mce.bank;
 
+	pr_info("Using '%s' error injection method", flags_options[inj_type]);
+
 	i_mce.tsc = rdtsc_ordered();
 
 	i_mce.status |= MCI_STATUS_VAL;
@@ -717,11 +727,49 @@ static void __init debugfs_init(void)
 				    &i_mce, dfs_fls[i].fops);
 }
 
+static void check_hw_inj_possible(void)
+{
+	u8 bank;
+
+	/*
+	 * This behavior exists only on SMCA systems though its not directly
+	 * related to SMCA.
+	 */
+	if (!cpu_feature_enabled(X86_FEATURE_SMCA))
+		return;
+
+	get_cpu();
+
+	for (bank = 0; bank < MAX_NR_BANKS; ++bank) {
+		u64 status = MCI_STATUS_VAL, ipid;
+
+		rdmsrl(MSR_AMD64_SMCA_MCx_IPID(bank), ipid);
+
+		if (!ipid)
+			continue;
+
+		wrmsrl(mca_msr_reg(bank, MCA_STATUS), status);
+		rdmsrl(mca_msr_reg(bank, MCA_STATUS), status);
+
+		if (!status) {
+			hw_injection_possible = false;
+			pr_warn("Platform does not allow error injection, try using APEI EINJ instead.\n");
+		}
+
+		/* Exit after the check for first available MCA bank */
+		break;
+	}
+
+	put_cpu();
+}
+
 static int __init inject_init(void)
 {
 	if (!alloc_cpumask_var(&mce_inject_cpumask, GFP_KERNEL))
 		return -ENOMEM;
 
+	check_hw_inj_possible();
+
 	debugfs_init();
 
 	register_nmi_handler(NMI_LOCAL, mce_raise_notify, 0, "mce_notify");
diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h
index 4ae0e603f7fa..7e03f5b7f6bd 100644
--- a/arch/x86/kernel/cpu/mce/internal.h
+++ b/arch/x86/kernel/cpu/mce/internal.h
@@ -211,7 +211,7 @@ noinstr u64 mce_rdmsrl(u32 msr);
 
 static __always_inline u32 mca_msr_reg(int bank, enum mca_msr reg)
 {
-	if (mce_flags.smca) {
+	if (cpu_feature_enabled(X86_FEATURE_SMCA)) {
 		switch (reg) {
 		case MCA_CTL:	 return MSR_AMD64_SMCA_MCx_CTL(bank);
 		case MCA_ADDR:	 return MSR_AMD64_SMCA_MCx_ADDR(bank);
-- 
2.17.1

Re: [PATCH v5] x86/mce: Check for writes ignored in MCA_STATUS register

[Borislav Petkov]

On Fri, Jun 10, 2022 at 07:25:15PM +0000, Smita Koralahalli wrote:
>  arch/x86/kernel/cpu/mce/inject.c   | 50 +++++++++++++++++++++++++++++-
>  arch/x86/kernel/cpu/mce/internal.h |  2 +-
>  2 files changed, 50 insertions(+), 2 deletions(-)

I've simplified, improved and removed some stuff. See if that still
works on that platform which ignores writes.

Thx.

---
From: Smita Koralahalli <Smita.KoralahalliChannabasappa@amd.com>
Date: Fri, 10 Jun 2022 19:25:15 +0000
Subject: [PATCH] x86/mce: Check whether writes to MCA_STATUS are getting
 ignored

The platform can sometimes - depending on its settings - cause writes
to MCA_STATUS MSRs to get ignored, regardless of HWCR[McStatusWrEn]'s
value.

For further info see

  PPR for AMD Family 19h, Model 01h, Revision B1 Processors, doc ID 55898

at https://bugzilla.kernel.org/show_bug.cgi?id=206537.

Therefore, probe for ignored writes to MCA_STATUS to determine if hardware
error injection is at all possible.

  [ bp: Heavily massage commit message and patch. ]

Signed-off-by: Smita Koralahalli <Smita.KoralahalliChannabasappa@amd.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lore.kernel.org/r/20220214233640.70510-2-Smita.KoralahalliChannabasappa@amd.com
---
 arch/x86/kernel/cpu/mce/inject.c   | 44 ++++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/mce/internal.h |  2 +-
 2 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/cpu/mce/inject.c b/arch/x86/kernel/cpu/mce/inject.c
index 5fbd7ffb3233..35ac8bf5bee6 100644
--- a/arch/x86/kernel/cpu/mce/inject.c
+++ b/arch/x86/kernel/cpu/mce/inject.c
@@ -33,6 +33,8 @@
 
 #include "internal.h"
 
+static bool hw_injection_possible = true;
+
 /*
  * Collect all the MCi_XXX settings
  */
@@ -339,6 +341,8 @@ static int __set_inj(const char *buf)
 
 	for (i = 0; i < N_INJ_TYPES; i++) {
 		if (!strncmp(flags_options[i], buf, strlen(flags_options[i]))) {
+			if (i > SW_INJ && !hw_injection_possible)
+				continue;
 			inj_type = i;
 			return 0;
 		}
@@ -717,11 +721,51 @@ static void __init debugfs_init(void)
 				    &i_mce, dfs_fls[i].fops);
 }
 
+static void check_hw_inj_possible(void)
+{
+	int cpu;
+	u8 bank;
+
+	/*
+	 * This behavior exists only on SMCA systems though its not directly
+	 * related to SMCA.
+	 */
+	if (!cpu_feature_enabled(X86_FEATURE_SMCA))
+		return;
+
+	cpu = get_cpu();
+
+	for (bank = 0; bank < MAX_NR_BANKS; ++bank) {
+		u64 status = MCI_STATUS_VAL, ipid;
+
+		/* Check whether bank is populated */
+		rdmsrl(MSR_AMD64_SMCA_MCx_IPID(bank), ipid);
+		if (!ipid)
+			continue;
+
+		toggle_hw_mce_inject(cpu, true);
+
+		if (wrmsrl_safe(mca_msr_reg(bank, MCA_STATUS), status)) {
+			hw_injection_possible = false;
+			pr_warn("Platform does not allow *hardware* error injection."
+				"Try using APEI EINJ instead.\n");
+		}
+
+		toggle_hw_mce_inject(cpu, false);
+
+		break;
+	}
+
+	put_cpu();
+}
+
 static int __init inject_init(void)
 {
 	if (!alloc_cpumask_var(&mce_inject_cpumask, GFP_KERNEL))
 		return -ENOMEM;
 
+	check_hw_inj_possible();
+
 	debugfs_init();
 
 	register_nmi_handler(NMI_LOCAL, mce_raise_notify, 0, "mce_notify");
diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h
index 4ae0e603f7fa..7e03f5b7f6bd 100644
--- a/arch/x86/kernel/cpu/mce/internal.h
+++ b/arch/x86/kernel/cpu/mce/internal.h
@@ -211,7 +211,7 @@ noinstr u64 mce_rdmsrl(u32 msr);
 
 static __always_inline u32 mca_msr_reg(int bank, enum mca_msr reg)
 {
-	if (mce_flags.smca) {
+	if (cpu_feature_enabled(X86_FEATURE_SMCA)) {
 		switch (reg) {
 		case MCA_CTL:	 return MSR_AMD64_SMCA_MCx_CTL(bank);
 		case MCA_ADDR:	 return MSR_AMD64_SMCA_MCx_ADDR(bank);
-- 
2.35.1

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH v5] x86/mce: Check for writes ignored in MCA_STATUS register

[Smita Koralahalli]

Hi Boris,

On 6/22/2022 10:29 AM, Borislav Petkov wrote:
> On Fri, Jun 10, 2022 at 07:25:15PM +0000, Smita Koralahalli wrote:
>>   arch/x86/kernel/cpu/mce/inject.c   | 50 +++++++++++++++++++++++++++++-
>>   arch/x86/kernel/cpu/mce/internal.h |  2 +-
>>   2 files changed, 50 insertions(+), 2 deletions(-)
> I've simplified, improved and removed some stuff. See if that still
> works on that platform which ignores writes.

I tested out the patch. Its not behaving as expected. The module fails 
silently
again.
> +	cpu = get_cpu();
> +
> +	for (bank = 0; bank < MAX_NR_BANKS; ++bank) {
> +		u64 status = MCI_STATUS_VAL, ipid;
> +
> +		/* Check whether bank is populated */
> +		rdmsrl(MSR_AMD64_SMCA_MCx_IPID(bank), ipid);
> +		if (!ipid)
> +			continue;
> +
> +		toggle_hw_mce_inject(cpu, true);
> +
> +		if (wrmsrl_safe(mca_msr_reg(bank, MCA_STATUS), status)) {

The wrmsrl_safe() doesn't throw an error here.
I think we need to read back the written value and check for it whatsoever.
What do you think?

Thanks,
Smita
> +			hw_injection_possible = false;
> +			pr_warn("Platform does not allow *hardware* error injection."
> +				"Try using APEI EINJ instead.\n");
> +		}
> +
> +		toggle_hw_mce_inject(cpu, false);
> +
> +		break;
> +	}
> +
> +	put_cpu();
> +}

Re: [PATCH v5] x86/mce: Check for writes ignored in MCA_STATUS register

[Borislav Petkov]

On Fri, Jun 24, 2022 at 03:34:42PM -0700, Smita Koralahalli wrote:
> The wrmsrl_safe() doesn't throw an error here.
> I think we need to read back the written value and check for it whatsoever.
> What do you think?

Ah, that's the write-ignored thing. Not the #GP-generating thing when
McStatusWrEn=0.

Sorry, I got confused.

Can you pls change that part back to reading the previously written
value and send me a tested version?

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH v5] x86/mce: Check for writes ignored in MCA_STATUS register

[Smita Koralahalli]

On 6/25/2022 6:10 AM, Borislav Petkov wrote:
> On Fri, Jun 24, 2022 at 03:34:42PM -0700, Smita Koralahalli wrote:
>> The wrmsrl_safe() doesn't throw an error here.
>> I think we need to read back the written value and check for it whatsoever.
>> What do you think?
> Ah, that's the write-ignored thing. Not the #GP-generating thing when
> McStatusWrEn=0.
>
> Sorry, I got confused.
>
> Can you pls change that part back to reading the previously written
> value and send me a tested version?
>
> Thx.
Thanks, I have sent them.
>