[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Geoff Levand]

Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
a work-around for HPE ProLiant m400 APEI firmware problems.

The work-around disables APEI when CONFIG_ACPI_APEI is set and
m400 firmware is detected.  Without this fixup m400 systems
experience errors like these on startup:

  [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
  [Hardware Error]: event severity: fatal
  [Hardware Error]:  Error 0, type: fatal
  [Hardware Error]:   section_type: memory error
  [Hardware Error]:   error_status: 0x0000000000001300
  [Hardware Error]:   error_type: 10, invalid address
  Kernel panic - not syncing: Fatal hardware error!

Signed-off-by: Geoff Levand <geoff@infradead.org>
---
Hi,

It seems unlikely there will be any m400 firmware updates to fix
this problem.  APEI support is desired for new ARM64 servers coming
to market.  Distros are now forced to have their own fixes, not
enable APEI, or let m400 users fend for themselves.

Please consider.

-Geoff

 arch/arm64/kernel/acpi.c | 40 ++++++++++++++++++++++++++++++++++++----
 1 file changed, 36 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 7b09487ff8fb..3c315c2c7476 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -31,6 +31,8 @@
 #include <asm/cpu_ops.h>
 #include <asm/smp_plat.h>
 
+#include <acpi/apei.h>
+
 #ifdef CONFIG_ACPI_APEI
 # include <linux/efi.h>
 # include <asm/pgtable.h>
@@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void)
 	return ret;
 }
 
+/*
+ * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware
+ * problems.
+ */
+static void __init acpi_fixup_m400_quirks(void)
+{
+	acpi_status status;
+	struct acpi_table_header *header;
+#if !defined(CONFIG_ACPI_APEI)
+	int hest_disable = HEST_DISABLED;
+#endif
+
+	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
+		return;
+
+	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
+
+	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
+		!strncmp(header->oem_table_id, "ProLiant", 8) &&
+		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
+		hest_disable = HEST_DISABLED;
+		pr_info("Disabled APEI for m400.\n");
+	}
+
+	acpi_put_table(header);
+}
+
 /*
  * acpi_boot_table_init() called from setup_arch(), always.
  *	1. find RSDP and get its address, and then find XSDT
@@ -232,11 +261,14 @@ void __init acpi_boot_table_init(void)
 	if (acpi_disabled) {
 		if (earlycon_acpi_spcr_enable)
 			early_init_dt_scan_chosen_stdout();
-	} else {
-		acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
-		if (IS_ENABLED(CONFIG_ACPI_BGRT))
-			acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
+		return;
 	}
+
+	acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
+	if (IS_ENABLED(CONFIG_ACPI_BGRT))
+		acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
+
+	acpi_fixup_m400_quirks();
 }
 
 #ifdef CONFIG_ACPI_APEI
-- 
2.14.1

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Geoff Levand]

Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
a work-around for HPE ProLiant m400 APEI firmware problems.

The work-around disables APEI when CONFIG_ACPI_APEI is set and
m400 firmware is detected.  Without this fixup m400 systems
experience errors like these on startup:

  [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
  [Hardware Error]: event severity: fatal
  [Hardware Error]:  Error 0, type: fatal
  [Hardware Error]:   section_type: memory error
  [Hardware Error]:   error_status: 0x0000000000001300
  [Hardware Error]:   error_type: 10, invalid address
  Kernel panic - not syncing: Fatal hardware error!

Signed-off-by: Geoff Levand <geoff@infradead.org>
---
Hi,

It seems unlikely there will be any m400 firmware updates to fix
this problem.  APEI support is desired for new ARM64 servers coming
to market.  Distros are now forced to have their own fixes, not
enable APEI, or let m400 users fend for themselves.

Please consider.

-Geoff

 arch/arm64/kernel/acpi.c | 40 ++++++++++++++++++++++++++++++++++++----
 1 file changed, 36 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 7b09487ff8fb..3c315c2c7476 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -31,6 +31,8 @@
 #include <asm/cpu_ops.h>
 #include <asm/smp_plat.h>
 
+#include <acpi/apei.h>
+
 #ifdef CONFIG_ACPI_APEI
 # include <linux/efi.h>
 # include <asm/pgtable.h>
@@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void)
 	return ret;
 }
 
+/*
+ * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware
+ * problems.
+ */
+static void __init acpi_fixup_m400_quirks(void)
+{
+	acpi_status status;
+	struct acpi_table_header *header;
+#if !defined(CONFIG_ACPI_APEI)
+	int hest_disable = HEST_DISABLED;
+#endif
+
+	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
+		return;
+
+	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
+
+	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
+		!strncmp(header->oem_table_id, "ProLiant", 8) &&
+		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
+		hest_disable = HEST_DISABLED;
+		pr_info("Disabled APEI for m400.\n");
+	}
+
+	acpi_put_table(header);
+}
+
 /*
  * acpi_boot_table_init() called from setup_arch(), always.
  *	1. find RSDP and get its address, and then find XSDT
@@ -232,11 +261,14 @@ void __init acpi_boot_table_init(void)
 	if (acpi_disabled) {
 		if (earlycon_acpi_spcr_enable)
 			early_init_dt_scan_chosen_stdout();
-	} else {
-		acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
-		if (IS_ENABLED(CONFIG_ACPI_BGRT))
-			acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
+		return;
 	}
+
+	acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
+	if (IS_ENABLED(CONFIG_ACPI_BGRT))
+		acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
+
+	acpi_fixup_m400_quirks();
 }
 
 #ifdef CONFIG_ACPI_APEI
-- 
2.14.1

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Riku Voipio]

On Wed, 13 Jun 2018 at 21:22, Geoff Levand <geoff@infradead.org> wrote:
>
> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
> a work-around for HPE ProLiant m400 APEI firmware problems.
>
> The work-around disables APEI when CONFIG_ACPI_APEI is set and
> m400 firmware is detected.  Without this fixup m400 systems
> experience errors like these on startup:
>
>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>   [Hardware Error]: event severity: fatal
>   [Hardware Error]:  Error 0, type: fatal
>   [Hardware Error]:   section_type: memory error
>   [Hardware Error]:   error_status: 0x0000000000001300
>   [Hardware Error]:   error_type: 10, invalid address
>   Kernel panic - not syncing: Fatal hardware error!
>
> Signed-off-by: Geoff Levand <geoff@infradead.org>
> ---
> Hi,
>
> It seems unlikely there will be any m400 firmware updates to fix
> this problem.  APEI support is desired for new ARM64 servers coming
> to market.  Distros are now forced to have their own fixes, not
> enable APEI, or let m400 users fend for themselves.

Thanks for working on this. This patch works well on m400 with APEI enabled.

Tested-by: Riku Voipio <riku.voipio@linaro.org>

> -Geoff
>
>  arch/arm64/kernel/acpi.c | 40 ++++++++++++++++++++++++++++++++++++----
>  1 file changed, 36 insertions(+), 4 deletions(-)
>
> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index 7b09487ff8fb..3c315c2c7476 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -31,6 +31,8 @@
>  #include <asm/cpu_ops.h>
>  #include <asm/smp_plat.h>
>
> +#include <acpi/apei.h>
> +
>  #ifdef CONFIG_ACPI_APEI
>  # include <linux/efi.h>
>  # include <asm/pgtable.h>
> @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void)
>         return ret;
>  }
>
> +/*
> + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware
> + * problems.
> + */
> +static void __init acpi_fixup_m400_quirks(void)
> +{
> +       acpi_status status;
> +       struct acpi_table_header *header;
> +#if !defined(CONFIG_ACPI_APEI)
> +       int hest_disable = HEST_DISABLED;
> +#endif
> +
> +       if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
> +               return;
> +
> +       status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
> +
> +       if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
> +               !strncmp(header->oem_table_id, "ProLiant", 8) &&
> +               MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> +               hest_disable = HEST_DISABLED;
> +               pr_info("Disabled APEI for m400.\n");
> +       }
> +
> +       acpi_put_table(header);
> +}
> +
>  /*
>   * acpi_boot_table_init() called from setup_arch(), always.
>   *     1. find RSDP and get its address, and then find XSDT
> @@ -232,11 +261,14 @@ void __init acpi_boot_table_init(void)
>         if (acpi_disabled) {
>                 if (earlycon_acpi_spcr_enable)
>                         early_init_dt_scan_chosen_stdout();
> -       } else {
> -               acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
> -               if (IS_ENABLED(CONFIG_ACPI_BGRT))
> -                       acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
> +               return;
>         }
> +
> +       acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
> +       if (IS_ENABLED(CONFIG_ACPI_BGRT))
> +               acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
> +
> +       acpi_fixup_m400_quirks();
>  }
>
>  #ifdef CONFIG_ACPI_APEI
> --
> 2.14.1
>

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Riku Voipio]

On Wed, 13 Jun 2018 at 21:22, Geoff Levand <geoff@infradead.org> wrote:
>
> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
> a work-around for HPE ProLiant m400 APEI firmware problems.
>
> The work-around disables APEI when CONFIG_ACPI_APEI is set and
> m400 firmware is detected.  Without this fixup m400 systems
> experience errors like these on startup:
>
>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>   [Hardware Error]: event severity: fatal
>   [Hardware Error]:  Error 0, type: fatal
>   [Hardware Error]:   section_type: memory error
>   [Hardware Error]:   error_status: 0x0000000000001300
>   [Hardware Error]:   error_type: 10, invalid address
>   Kernel panic - not syncing: Fatal hardware error!
>
> Signed-off-by: Geoff Levand <geoff@infradead.org>
> ---
> Hi,
>
> It seems unlikely there will be any m400 firmware updates to fix
> this problem.  APEI support is desired for new ARM64 servers coming
> to market.  Distros are now forced to have their own fixes, not
> enable APEI, or let m400 users fend for themselves.

Thanks for working on this. This patch works well on m400 with APEI enabled.

Tested-by: Riku Voipio <riku.voipio@linaro.org>

> -Geoff
>
>  arch/arm64/kernel/acpi.c | 40 ++++++++++++++++++++++++++++++++++++----
>  1 file changed, 36 insertions(+), 4 deletions(-)
>
> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index 7b09487ff8fb..3c315c2c7476 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -31,6 +31,8 @@
>  #include <asm/cpu_ops.h>
>  #include <asm/smp_plat.h>
>
> +#include <acpi/apei.h>
> +
>  #ifdef CONFIG_ACPI_APEI
>  # include <linux/efi.h>
>  # include <asm/pgtable.h>
> @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void)
>         return ret;
>  }
>
> +/*
> + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware
> + * problems.
> + */
> +static void __init acpi_fixup_m400_quirks(void)
> +{
> +       acpi_status status;
> +       struct acpi_table_header *header;
> +#if !defined(CONFIG_ACPI_APEI)
> +       int hest_disable = HEST_DISABLED;
> +#endif
> +
> +       if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
> +               return;
> +
> +       status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
> +
> +       if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
> +               !strncmp(header->oem_table_id, "ProLiant", 8) &&
> +               MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> +               hest_disable = HEST_DISABLED;
> +               pr_info("Disabled APEI for m400.\n");
> +       }
> +
> +       acpi_put_table(header);
> +}
> +
>  /*
>   * acpi_boot_table_init() called from setup_arch(), always.
>   *     1. find RSDP and get its address, and then find XSDT
> @@ -232,11 +261,14 @@ void __init acpi_boot_table_init(void)
>         if (acpi_disabled) {
>                 if (earlycon_acpi_spcr_enable)
>                         early_init_dt_scan_chosen_stdout();
> -       } else {
> -               acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
> -               if (IS_ENABLED(CONFIG_ACPI_BGRT))
> -                       acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
> +               return;
>         }
> +
> +       acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
> +       if (IS_ENABLED(CONFIG_ACPI_BGRT))
> +               acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
> +
> +       acpi_fixup_m400_quirks();
>  }
>
>  #ifdef CONFIG_ACPI_APEI
> --
> 2.14.1
>

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Graeme Gregory]

On Wed, Jun 13, 2018 at 11:22:30AM -0700, Geoff Levand wrote:
> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
> a work-around for HPE ProLiant m400 APEI firmware problems.
> 
> The work-around disables APEI when CONFIG_ACPI_APEI is set and
> m400 firmware is detected.  Without this fixup m400 systems
> experience errors like these on startup:
> 
>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>   [Hardware Error]: event severity: fatal
>   [Hardware Error]:  Error 0, type: fatal
>   [Hardware Error]:   section_type: memory error
>   [Hardware Error]:   error_status: 0x0000000000001300
>   [Hardware Error]:   error_type: 10, invalid address
>   Kernel panic - not syncing: Fatal hardware error!
> 
> Signed-off-by: Geoff Levand <geoff@infradead.org>
> ---
> Hi,
> 
> It seems unlikely there will be any m400 firmware updates to fix
> this problem.  APEI support is desired for new ARM64 servers coming
> to market.  Distros are now forced to have their own fixes, not
> enable APEI, or let m400 users fend for themselves.
> 
> Please consider.
> 
> -Geoff
> 
>  arch/arm64/kernel/acpi.c | 40 ++++++++++++++++++++++++++++++++++++----
>  1 file changed, 36 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index 7b09487ff8fb..3c315c2c7476 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -31,6 +31,8 @@
>  #include <asm/cpu_ops.h>
>  #include <asm/smp_plat.h>
>  
> +#include <acpi/apei.h>
> +
>  #ifdef CONFIG_ACPI_APEI
>  # include <linux/efi.h>
>  # include <asm/pgtable.h>
> @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void)
>  	return ret;
>  }
>  
> +/*
> + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware
> + * problems.
> + */
> +static void __init acpi_fixup_m400_quirks(void)
> +{
> +	acpi_status status;
> +	struct acpi_table_header *header;
> +#if !defined(CONFIG_ACPI_APEI)
> +	int hest_disable = HEST_DISABLED;
> +#endif
> +
> +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
> +		return;
> +
> +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
> +
> +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
> +		!strncmp(header->oem_table_id, "ProLiant", 8) &&
> +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> +		hest_disable = HEST_DISABLED;
> +		pr_info("Disabled APEI for m400.\n");
> +	}
> +
> +	acpi_put_table(header);
> +}
> +
>  /*
>   * acpi_boot_table_init() called from setup_arch(), always.
>   *	1. find RSDP and get its address, and then find XSDT
> @@ -232,11 +261,14 @@ void __init acpi_boot_table_init(void)
>  	if (acpi_disabled) {
>  		if (earlycon_acpi_spcr_enable)
>  			early_init_dt_scan_chosen_stdout();
> -	} else {
> -		acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
> -		if (IS_ENABLED(CONFIG_ACPI_BGRT))
> -			acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
> +		return;
>  	}
> +
> +	acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
> +	if (IS_ENABLED(CONFIG_ACPI_BGRT))
> +		acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
> +
> +	acpi_fixup_m400_quirks();
>  }
>  
>  #ifdef CONFIG_ACPI_APEI

Quirk actually make code nicer to read.

Reviewed-by: Graeme Gregory <graeme.gregory@linaro.org>

Thanks we needed this having 90+ m400s

Graeme

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Graeme Gregory]

On Wed, Jun 13, 2018 at 11:22:30AM -0700, Geoff Levand wrote:
> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
> a work-around for HPE ProLiant m400 APEI firmware problems.
> 
> The work-around disables APEI when CONFIG_ACPI_APEI is set and
> m400 firmware is detected.  Without this fixup m400 systems
> experience errors like these on startup:
> 
>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>   [Hardware Error]: event severity: fatal
>   [Hardware Error]:  Error 0, type: fatal
>   [Hardware Error]:   section_type: memory error
>   [Hardware Error]:   error_status: 0x0000000000001300
>   [Hardware Error]:   error_type: 10, invalid address
>   Kernel panic - not syncing: Fatal hardware error!
> 
> Signed-off-by: Geoff Levand <geoff@infradead.org>
> ---
> Hi,
> 
> It seems unlikely there will be any m400 firmware updates to fix
> this problem.  APEI support is desired for new ARM64 servers coming
> to market.  Distros are now forced to have their own fixes, not
> enable APEI, or let m400 users fend for themselves.
> 
> Please consider.
> 
> -Geoff
> 
>  arch/arm64/kernel/acpi.c | 40 ++++++++++++++++++++++++++++++++++++----
>  1 file changed, 36 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index 7b09487ff8fb..3c315c2c7476 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -31,6 +31,8 @@
>  #include <asm/cpu_ops.h>
>  #include <asm/smp_plat.h>
>  
> +#include <acpi/apei.h>
> +
>  #ifdef CONFIG_ACPI_APEI
>  # include <linux/efi.h>
>  # include <asm/pgtable.h>
> @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void)
>  	return ret;
>  }
>  
> +/*
> + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware
> + * problems.
> + */
> +static void __init acpi_fixup_m400_quirks(void)
> +{
> +	acpi_status status;
> +	struct acpi_table_header *header;
> +#if !defined(CONFIG_ACPI_APEI)
> +	int hest_disable = HEST_DISABLED;
> +#endif
> +
> +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
> +		return;
> +
> +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
> +
> +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
> +		!strncmp(header->oem_table_id, "ProLiant", 8) &&
> +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> +		hest_disable = HEST_DISABLED;
> +		pr_info("Disabled APEI for m400.\n");
> +	}
> +
> +	acpi_put_table(header);
> +}
> +
>  /*
>   * acpi_boot_table_init() called from setup_arch(), always.
>   *	1. find RSDP and get its address, and then find XSDT
> @@ -232,11 +261,14 @@ void __init acpi_boot_table_init(void)
>  	if (acpi_disabled) {
>  		if (earlycon_acpi_spcr_enable)
>  			early_init_dt_scan_chosen_stdout();
> -	} else {
> -		acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
> -		if (IS_ENABLED(CONFIG_ACPI_BGRT))
> -			acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
> +		return;
>  	}
> +
> +	acpi_parse_spcr(earlycon_acpi_spcr_enable, true);
> +	if (IS_ENABLED(CONFIG_ACPI_BGRT))
> +		acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
> +
> +	acpi_fixup_m400_quirks();
>  }
>  
>  #ifdef CONFIG_ACPI_APEI

Quirk actually make code nicer to read.

Reviewed-by: Graeme Gregory <graeme.gregory@linaro.org>

Thanks we needed this having 90+ m400s

Graeme

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[James Morse]

Hi Geoff,

On 13/06/18 19:22, Geoff Levand wrote:
> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
> a work-around for HPE ProLiant m400 APEI firmware problems.
> 
> The work-around disables APEI when CONFIG_ACPI_APEI is set and
> m400 firmware is detected.  Without this fixup m400 systems
> experience errors like these on startup:
> 
>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>   [Hardware Error]: event severity: fatal
>   [Hardware Error]:  Error 0, type: fatal
>   [Hardware Error]:   section_type: memory error
>   [Hardware Error]:   error_status: 0x0000000000001300

"Access to a memory address which is not mapped to any component"


>   [Hardware Error]:   error_type: 10, invalid address
>   Kernel panic - not syncing: Fatal hardware error!

Why is this a problem?

Surely this is a valid description of an error.
(okay its not particularly useful without the physical address, but the address
is optional in that structure)

When does this happen during boot? This looks like a driver mapping some
non-existent physical address space to see if its device is present...
unsurprisingly this doesn't go well.
(might also be a typo in the DSDT)

Can't we pin down the driver that does this and fix it. Its either wrong for
everyone, or still broken after you disable APEI.


> It seems unlikely there will be any m400 firmware updates to fix
> this problem.

What is the problem? This patch looks like it shoots the messenger for bringing
bad news.


> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index 7b09487ff8fb..3c315c2c7476 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -31,6 +31,8 @@
>  #include <asm/cpu_ops.h>
>  #include <asm/smp_plat.h>
>  
> +#include <acpi/apei.h>
> +
>  #ifdef CONFIG_ACPI_APEI
>  # include <linux/efi.h>
>  # include <asm/pgtable.h>
> @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void)
>  	return ret;
>  }
>  
> +/*
> + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware
> + * problems.
> + */
> +static void __init acpi_fixup_m400_quirks(void)
> +{
> +	acpi_status status;
> +	struct acpi_table_header *header;
> +#if !defined(CONFIG_ACPI_APEI)
> +	int hest_disable = HEST_DISABLED;
> +#endif

Yuck.


> +
> +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
> +		return;
> +
> +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
> +
> +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
> +		!strncmp(header->oem_table_id, "ProLiant", 8) &&

You should match the affected range of OEM table revisions too, that way a
firmware upgrade should start working, instead of being permanently disabled
because we think its unlikely.


> +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {

How is the CPU implementer relevant?

You suggest a firmware-update would make this issue go away...


> +		hest_disable = HEST_DISABLED;
> +		pr_info("Disabled APEI for m400.\n");
> +	}
> +
> +	acpi_put_table(header);
> +}
> +
>  /*
>   * acpi_boot_table_init() called from setup_arch(), always.
>   *	1. find RSDP and get its address, and then find XSDT

Nothing arch-specific here. You're adding this to arch/arm64 because
drivers/acpi/apei doesn't have an existing quirks table?


Thanks,

James

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[James Morse]

Hi Geoff,

On 13/06/18 19:22, Geoff Levand wrote:
> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
> a work-around for HPE ProLiant m400 APEI firmware problems.
> 
> The work-around disables APEI when CONFIG_ACPI_APEI is set and
> m400 firmware is detected.  Without this fixup m400 systems
> experience errors like these on startup:
> 
>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>   [Hardware Error]: event severity: fatal
>   [Hardware Error]:  Error 0, type: fatal
>   [Hardware Error]:   section_type: memory error
>   [Hardware Error]:   error_status: 0x0000000000001300

"Access to a memory address which is not mapped to any component"


>   [Hardware Error]:   error_type: 10, invalid address
>   Kernel panic - not syncing: Fatal hardware error!

Why is this a problem?

Surely this is a valid description of an error.
(okay its not particularly useful without the physical address, but the address
is optional in that structure)

When does this happen during boot? This looks like a driver mapping some
non-existent physical address space to see if its device is present...
unsurprisingly this doesn't go well.
(might also be a typo in the DSDT)

Can't we pin down the driver that does this and fix it. Its either wrong for
everyone, or still broken after you disable APEI.


> It seems unlikely there will be any m400 firmware updates to fix
> this problem.

What is the problem? This patch looks like it shoots the messenger for bringing
bad news.


> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index 7b09487ff8fb..3c315c2c7476 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -31,6 +31,8 @@
>  #include <asm/cpu_ops.h>
>  #include <asm/smp_plat.h>
>  
> +#include <acpi/apei.h>
> +
>  #ifdef CONFIG_ACPI_APEI
>  # include <linux/efi.h>
>  # include <asm/pgtable.h>
> @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void)
>  	return ret;
>  }
>  
> +/*
> + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware
> + * problems.
> + */
> +static void __init acpi_fixup_m400_quirks(void)
> +{
> +	acpi_status status;
> +	struct acpi_table_header *header;
> +#if !defined(CONFIG_ACPI_APEI)
> +	int hest_disable = HEST_DISABLED;
> +#endif

Yuck.


> +
> +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
> +		return;
> +
> +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
> +
> +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
> +		!strncmp(header->oem_table_id, "ProLiant", 8) &&

You should match the affected range of OEM table revisions too, that way a
firmware upgrade should start working, instead of being permanently disabled
because we think its unlikely.


> +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {

How is the CPU implementer relevant?

You suggest a firmware-update would make this issue go away...


> +		hest_disable = HEST_DISABLED;
> +		pr_info("Disabled APEI for m400.\n");
> +	}
> +
> +	acpi_put_table(header);
> +}
> +
>  /*
>   * acpi_boot_table_init() called from setup_arch(), always.
>   *	1. find RSDP and get its address, and then find XSDT

Nothing arch-specific here. You're adding this to arch/arm64 because
drivers/acpi/apei doesn't have an existing quirks table?


Thanks,

James

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Geoff Levand]

Hi James,

Just for background, this is a well known bug in the m400's AEPI/HEST
firmware.  There are a number of fixes out there the different distros
have.  I just put together this patch to unify things and have a
common 'upstream' fix.

On 06/15/2018 04:14 AM, James Morse wrote:
> On 13/06/18 19:22, Geoff Levand wrote:
>> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
>> a work-around for HPE ProLiant m400 APEI firmware problems.
>>
>> The work-around disables APEI when CONFIG_ACPI_APEI is set and
>> m400 firmware is detected.  Without this fixup m400 systems
>> experience errors like these on startup:
>>
>>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>>   [Hardware Error]: event severity: fatal
>>   [Hardware Error]:  Error 0, type: fatal
>>   [Hardware Error]:   section_type: memory error
>>   [Hardware Error]:   error_status: 0x0000000000001300
> 
> "Access to a memory address which is not mapped to any component"
> 
> 
>>   [Hardware Error]:   error_type: 10, invalid address
>>   Kernel panic - not syncing: Fatal hardware error!
> 
> Why is this a problem?
> 
> Surely this is a valid description of an error.

The firmware bug causes this failure, not bad hardware.

> (okay its not particularly useful without the physical address, but the address
> is optional in that structure)
> 
> When does this happen during boot? This looks like a driver mapping some
> non-existent physical address space to see if its device is present...
> unsurprisingly this doesn't go well.
> (might also be a typo in the DSDT)
> 
> Can't we pin down the driver that does this and fix it. Its either wrong for
> everyone, or still broken after you disable APEI.
> 
> 
>> It seems unlikely there will be any m400 firmware updates to fix
>> this problem.
> 
> What is the problem? This patch looks like it shoots the messenger for bringing
> bad news.
 
The news is incorrect, so this patch disables the source (APEI code).

>> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
>> index 7b09487ff8fb..3c315c2c7476 100644
>> --- a/arch/arm64/kernel/acpi.c
>> +++ b/arch/arm64/kernel/acpi.c
>> @@ -31,6 +31,8 @@
>>  #include <asm/cpu_ops.h>
>>  #include <asm/smp_plat.h>
>>  
>> +#include <acpi/apei.h>
>> +
>>  #ifdef CONFIG_ACPI_APEI
>>  # include <linux/efi.h>
>>  # include <asm/pgtable.h>
>> @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void)
>>  	return ret;
>>  }
>>  
>> +/*
>> + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware
>> + * problems.
>> + */
>> +static void __init acpi_fixup_m400_quirks(void)
>> +{
>> +	acpi_status status;
>> +	struct acpi_table_header *header;
>> +#if !defined(CONFIG_ACPI_APEI)
>> +	int hest_disable = HEST_DISABLED;
>> +#endif
> 
> Yuck.

Yes, unfortunately, the hest code conditionally defines hest_disable.

>> +
>> +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
>> +		return;
>> +
>> +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
>> +
>> +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
>> +		!strncmp(header->oem_table_id, "ProLiant", 8) &&
> 
> You should match the affected range of OEM table revisions too, that way a
> firmware upgrade should start working, instead of being permanently disabled
> because we think its unlikely.

The m400 has reached end of life. No one really expects to see any firmware
update.  I don't know what the effected OEM table revisions are, and I don't
think there is an active platform maintainer who could give that info either.

If someone can provide the info. I'll update the fix.

>> +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> 
> How is the CPU implementer relevant?

That was just a copy of what other fixes had.  Should I remove it?

> You suggest a firmware-update would make this issue go away...
> 
> 
>> +		hest_disable = HEST_DISABLED;
>> +		pr_info("Disabled APEI for m400.\n");
>> +	}
>> +
>> +	acpi_put_table(header);
>> +}
>> +
>>  /*
>>   * acpi_boot_table_init() called from setup_arch(), always.
>>   *	1. find RSDP and get its address, and then find XSDT
> 
> Nothing arch-specific here. You're adding this to arch/arm64 because
> drivers/acpi/apei doesn't have an existing quirks table?

There was a fix submitted that had it in drivers/acpi/scan.c, but the
ACPI maintainer said he didn't want the fix in the main ACPI code.
See:

  https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs)

The m400 is an arm64 platform, so it seems most appropriate to
have it in arch/arm64/kernel/acpi.c.  I followed what was done
for x86 quirks (into arch/x86/kernel/acpi/boot.c), and what was
suggested here: 

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900581 (linux: Enable Buster kernel features for newer ARM64 servers)

Thanks for the review.

-Geoff

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Geoff Levand]

Hi James,

Just for background, this is a well known bug in the m400's AEPI/HEST
firmware.  There are a number of fixes out there the different distros
have.  I just put together this patch to unify things and have a
common 'upstream' fix.

On 06/15/2018 04:14 AM, James Morse wrote:
> On 13/06/18 19:22, Geoff Levand wrote:
>> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
>> a work-around for HPE ProLiant m400 APEI firmware problems.
>>
>> The work-around disables APEI when CONFIG_ACPI_APEI is set and
>> m400 firmware is detected.  Without this fixup m400 systems
>> experience errors like these on startup:
>>
>>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>>   [Hardware Error]: event severity: fatal
>>   [Hardware Error]:  Error 0, type: fatal
>>   [Hardware Error]:   section_type: memory error
>>   [Hardware Error]:   error_status: 0x0000000000001300
> 
> "Access to a memory address which is not mapped to any component"
> 
> 
>>   [Hardware Error]:   error_type: 10, invalid address
>>   Kernel panic - not syncing: Fatal hardware error!
> 
> Why is this a problem?
> 
> Surely this is a valid description of an error.

The firmware bug causes this failure, not bad hardware.

> (okay its not particularly useful without the physical address, but the address
> is optional in that structure)
> 
> When does this happen during boot? This looks like a driver mapping some
> non-existent physical address space to see if its device is present...
> unsurprisingly this doesn't go well.
> (might also be a typo in the DSDT)
> 
> Can't we pin down the driver that does this and fix it. Its either wrong for
> everyone, or still broken after you disable APEI.
> 
> 
>> It seems unlikely there will be any m400 firmware updates to fix
>> this problem.
> 
> What is the problem? This patch looks like it shoots the messenger for bringing
> bad news.
 
The news is incorrect, so this patch disables the source (APEI code).

>> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
>> index 7b09487ff8fb..3c315c2c7476 100644
>> --- a/arch/arm64/kernel/acpi.c
>> +++ b/arch/arm64/kernel/acpi.c
>> @@ -31,6 +31,8 @@
>>  #include <asm/cpu_ops.h>
>>  #include <asm/smp_plat.h>
>>  
>> +#include <acpi/apei.h>
>> +
>>  #ifdef CONFIG_ACPI_APEI
>>  # include <linux/efi.h>
>>  # include <asm/pgtable.h>
>> @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void)
>>  	return ret;
>>  }
>>  
>> +/*
>> + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware
>> + * problems.
>> + */
>> +static void __init acpi_fixup_m400_quirks(void)
>> +{
>> +	acpi_status status;
>> +	struct acpi_table_header *header;
>> +#if !defined(CONFIG_ACPI_APEI)
>> +	int hest_disable = HEST_DISABLED;
>> +#endif
> 
> Yuck.

Yes, unfortunately, the hest code conditionally defines hest_disable.

>> +
>> +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
>> +		return;
>> +
>> +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
>> +
>> +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
>> +		!strncmp(header->oem_table_id, "ProLiant", 8) &&
> 
> You should match the affected range of OEM table revisions too, that way a
> firmware upgrade should start working, instead of being permanently disabled
> because we think its unlikely.

The m400 has reached end of life. No one really expects to see any firmware
update.  I don't know what the effected OEM table revisions are, and I don't
think there is an active platform maintainer who could give that info either.

If someone can provide the info. I'll update the fix.

>> +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> 
> How is the CPU implementer relevant?

That was just a copy of what other fixes had.  Should I remove it?

> You suggest a firmware-update would make this issue go away...
> 
> 
>> +		hest_disable = HEST_DISABLED;
>> +		pr_info("Disabled APEI for m400.\n");
>> +	}
>> +
>> +	acpi_put_table(header);
>> +}
>> +
>>  /*
>>   * acpi_boot_table_init() called from setup_arch(), always.
>>   *	1. find RSDP and get its address, and then find XSDT
> 
> Nothing arch-specific here. You're adding this to arch/arm64 because
> drivers/acpi/apei doesn't have an existing quirks table?

There was a fix submitted that had it in drivers/acpi/scan.c, but the
ACPI maintainer said he didn't want the fix in the main ACPI code.
See:

  https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs)

The m400 is an arm64 platform, so it seems most appropriate to
have it in arch/arm64/kernel/acpi.c.  I followed what was done
for x86 quirks (into arch/x86/kernel/acpi/boot.c), and what was
suggested here: 

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900581 (linux: Enable Buster kernel features for newer ARM64 servers)

Thanks for the review.

-Geoff

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Fri, 2018-06-15 at 10:17 -0700, Geoff Levand wrote:
> > > +    if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
> > > +            !strncmp(header->oem_table_id, "ProLiant", 8) &&
> > 
> > You should match the affected range of OEM table revisions too, that way a
> > firmware upgrade should start working, instead of being permanently disabled
> > because we think its unlikely.
> 
> The m400 has reached end of life. No one really expects to see any firmware
> update.  I don't know what the effected OEM table revisions are, and I don't
> think there is an active platform maintainer who could give that info either.
> 
> If someone can provide the info. I'll update the fix.
> 
> > > +            MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> > 
> > How is the CPU implementer relevant?
> 
> That was just a copy of what other fixes had.  Should I remove it?

It was there because HPE ProLiant strings are generic and you may end up
disabling platforms which would otherwise work. It is the ProLiant system
based on the APM chipset which is the problem. Thus the check for cpu
implementor.

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Fri, 2018-06-15 at 10:17 -0700, Geoff Levand wrote:
> > > +    if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
> > > +            !strncmp(header->oem_table_id, "ProLiant", 8) &&
> > 
> > You should match the affected range of OEM table revisions too, that way a
> > firmware upgrade should start working, instead of being permanently disabled
> > because we think its unlikely.
> 
> The m400 has reached end of life. No one really expects to see any firmware
> update.  I don't know what the effected OEM table revisions are, and I don't
> think there is an active platform maintainer who could give that info either.
> 
> If someone can provide the info. I'll update the fix.
> 
> > > +            MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> > 
> > How is the CPU implementer relevant?
> 
> That was just a copy of what other fixes had.  Should I remove it?

It was there because HPE ProLiant strings are generic and you may end up
disabling platforms which would otherwise work. It is the ProLiant system
based on the APM chipset which is the problem. Thus the check for cpu
implementor.

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Geoff Levand]

Hi Mark,

On 06/15/2018 10:33 AM, Mark Salter wrote:
> On Fri, 2018-06-15 at 10:17 -0700, Geoff Levand wrote:
>>>> +            MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
>>>
>>> How is the CPU implementer relevant?
>>
>> That was just a copy of what other fixes had.  Should I remove it?
> 
> It was there because HPE ProLiant strings are generic and you may end up
> disabling platforms which would otherwise work. It is the ProLiant system
> based on the APM chipset which is the problem. Thus the check for cpu
> implementor.

Your original fix that had this cpu implementor check was in the main
ACPI code, so would be built for other arches.  This is now in
arch/arm64/kernel/acpi.c, which will only be built for arm64.  Is that
enough to limit it, or do we still need the check?

-Geoff 

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Geoff Levand]

Hi Mark,

On 06/15/2018 10:33 AM, Mark Salter wrote:
> On Fri, 2018-06-15 at 10:17 -0700, Geoff Levand wrote:
>>>> +            MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
>>>
>>> How is the CPU implementer relevant?
>>
>> That was just a copy of what other fixes had.  Should I remove it?
> 
> It was there because HPE ProLiant strings are generic and you may end up
> disabling platforms which would otherwise work. It is the ProLiant system
> based on the APM chipset which is the problem. Thus the check for cpu
> implementor.

Your original fix that had this cpu implementor check was in the main
ACPI code, so would be built for other arches.  This is now in
arch/arm64/kernel/acpi.c, which will only be built for arm64.  Is that
enough to limit it, or do we still need the check?

-Geoff 

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Fri, 2018-06-15 at 11:15 -0700, Geoff Levand wrote:
> Hi Mark,
> 
> On 06/15/2018 10:33 AM, Mark Salter wrote:
> > On Fri, 2018-06-15 at 10:17 -0700, Geoff Levand wrote:
> > > > > +            MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> > > > 
> > > > How is the CPU implementer relevant?
> > > 
> > > That was just a copy of what other fixes had.  Should I remove it?
> > 
> > It was there because HPE ProLiant strings are generic and you may end up
> > disabling platforms which would otherwise work. It is the ProLiant system
> > based on the APM chipset which is the problem. Thus the check for cpu
> > implementor.
> 
> Your original fix that had this cpu implementor check was in the main
> ACPI code, so would be built for other arches.  This is now in
> arch/arm64/kernel/acpi.c, which will only be built for arm64.  Is that
> enough to limit it, or do we still need the check?

The original code was protected by #ifdef ARM64. But yes, HPE has announced
another aarch64 ProLiant system based on Cavium ThunderX2. So we need to
allow everything but the APM XGene based ProLiant products.

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Fri, 2018-06-15 at 11:15 -0700, Geoff Levand wrote:
> Hi Mark,
> 
> On 06/15/2018 10:33 AM, Mark Salter wrote:
> > On Fri, 2018-06-15 at 10:17 -0700, Geoff Levand wrote:
> > > > > +            MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> > > > 
> > > > How is the CPU implementer relevant?
> > > 
> > > That was just a copy of what other fixes had.  Should I remove it?
> > 
> > It was there because HPE ProLiant strings are generic and you may end up
> > disabling platforms which would otherwise work. It is the ProLiant system
> > based on the APM chipset which is the problem. Thus the check for cpu
> > implementor.
> 
> Your original fix that had this cpu implementor check was in the main
> ACPI code, so would be built for other arches.  This is now in
> arch/arm64/kernel/acpi.c, which will only be built for arm64.  Is that
> enough to limit it, or do we still need the check?

The original code was protected by #ifdef ARM64. But yes, HPE has announced
another aarch64 ProLiant system based on Cavium ThunderX2. So we need to
allow everything but the APM XGene based ProLiant products.

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[James Morse]

Hi Geoff,

On 15/06/18 18:17, Geoff Levand wrote:
> Just for background, this is a well known bug in the m400's AEPI/HEST
> firmware.  There are a number of fixes out there the different distros
> have.

Thanks,

From:
https://bugzilla.redhat.com/show_bug.cgi?id=1574718

This is tripped by the ghes_probe() call, it finds an error has already occurred
before ghes code is initialized.

Does this still happen if you compile without CONFIG_PCI? (that is easily half
the dmesg that has happened before this point).
Disabling CONFIG_EFIVAR_FS would be interesting too, as that is firmware-code we
can't fix bugs in.


Your argument here is the that the firmware-vendor only build-tested their code,
and never noticed it notifies a fatal exception on startup. I find this hard to
believe, especially as these systems don't have EL3.

It's much more likely a driver is causing this, possibly because of bad data in
the firmware tables. I'd like to quirk the driver, so we can fix the next error
like this, as opposed to blindly continuing.

If it really is firmware, what is it doing that causes this error, and where
does it run? Disabling APEI is just reacting after the error has occurred, when
we could prevent it happening.

I can't reproduce this on a Mustang. I assume its the different ACPI tables, not
the kernel-config.


> I just put together this patch to unify things and have a
> common 'upstream' fix.

Wouldn't passing 'hest_disable' on the cmdline do the same for all kernel versions?


> On 06/15/2018 04:14 AM, James Morse wrote:
>> On 13/06/18 19:22, Geoff Levand wrote:
>>> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
>>> a work-around for HPE ProLiant m400 APEI firmware problems.
>>>
>>> The work-around disables APEI when CONFIG_ACPI_APEI is set and
>>> m400 firmware is detected.  Without this fixup m400 systems
>>> experience errors like these on startup:
>>>
>>>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>>>   [Hardware Error]: event severity: fatal
>>>   [Hardware Error]:  Error 0, type: fatal
>>>   [Hardware Error]:   section_type: memory error
>>>   [Hardware Error]:   error_status: 0x0000000000001300
>>
>> "Access to a memory address which is not mapped to any component"
>>
>>
>>>   [Hardware Error]:   error_type: 10, invalid address
>>>   Kernel panic - not syncing: Fatal hardware error!
>>
>> Why is this a problem?
>>
>> Surely this is a valid description of an error.
> 
> The firmware bug causes this failure, not bad hardware.

I'm not talking about bad hardware here. What I think is happening is a software
bug is causing the CPU to make a bad access, which the RAS mechanism is
reporting like this because software would never be stupid enough to access an
address which is not mapped to any component! The RAS stuff believes this must
be address corruption.

I think this is a linux-driver bug, or a typo in the firmware tables, that cause
$non_existent_address to be mapped and probed.


>>> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
>>> index 7b09487ff8fb..3c315c2c7476 100644
>>> --- a/arch/arm64/kernel/acpi.c
>>> +++ b/arch/arm64/kernel/acpi.c

>>> +
>>> +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
>>> +		return;
>>> +
>>> +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
>>> +
>>> +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
>>> +		!strncmp(header->oem_table_id, "ProLiant", 8) &&

>> You should match the affected range of OEM table revisions too, that way a
>> firmware upgrade should start working, instead of being permanently disabled
>> because we think its unlikely.
> 
> The m400 has reached end of life. No one really expects to see any firmware
> update.  I don't know what the effected OEM table revisions are, and I don't
> think there is an active platform maintainer who could give that info either.
> 
> If someone can provide the info. I'll update the fix.

We can start with the version you have. You mention distro's have their own
fixes, hopefully they can supply the missing range of values.


>>> +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
>>
>> How is the CPU implementer relevant?
> 
> That was just a copy of what other fixes had.  Should I remove it?

The conclusion in the rest of this thread was HPE also produces ProLiant boxes
with a (presumably) identical HEST, but a totally different architecture.

Matching the DMI platform name as well would work round this. (or somewhere only
arm64 builds, with an appropriate comment in case we move it).


>> Nothing arch-specific here. You're adding this to arch/arm64 because
>> drivers/acpi/apei doesn't have an existing quirks table?
> 
> There was a fix submitted that had it in drivers/acpi/scan.c, but the
> ACPI maintainer said he didn't want the fix in the main ACPI code.

Specifically about a HID-hack in the core device enumeration code, as opposed to
in the driver that claims it.


> See:
> 
>   https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs)

| 'some X-Gene based platforms (Mustang and M400) with invalid DSDT.'

... sounds familiar ...

And:
https://bugzilla.redhat.com/attachment.cgi?id=1144903&action=diff

Fixes a typo in firmware description of the GIC.

Why do we think this is a new kind of firmware bug, and not a repeat of the last
two? The only difference is this is being caught by the RAS code.


> The m400 is an arm64 platform, so it seems most appropriate to
> have it in arch/arm64/kernel/acpi.c.

We don't keep all drivers under arch/arm64, I'd really like to find the driver
that is causing this, and quirk it there.

If we have to bolt the hest-stable-door, drivers/acpi/apei/hest.c looks better,
it at least doesn't have to bodge around hest_disabled not being exposed in a
compatible way. If the maintainer objects, (as x86 hasn't had to do this yet),
then we can try drivers/acpi/arm64/hest_quirks.c.

When only once architecture had to quirk stuff based on the ACPI tables, the
arch-code was a suitable dumping ground. Now there is more than one, we should
do things in a way they are useful to both architectures.


Thanks,

James

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[James Morse]

Hi Geoff,

On 15/06/18 18:17, Geoff Levand wrote:
> Just for background, this is a well known bug in the m400's AEPI/HEST
> firmware.  There are a number of fixes out there the different distros
> have.

Thanks,

From:
https://bugzilla.redhat.com/show_bug.cgi?id=1574718

This is tripped by the ghes_probe() call, it finds an error has already occurred
before ghes code is initialized.

Does this still happen if you compile without CONFIG_PCI? (that is easily half
the dmesg that has happened before this point).
Disabling CONFIG_EFIVAR_FS would be interesting too, as that is firmware-code we
can't fix bugs in.


Your argument here is the that the firmware-vendor only build-tested their code,
and never noticed it notifies a fatal exception on startup. I find this hard to
believe, especially as these systems don't have EL3.

It's much more likely a driver is causing this, possibly because of bad data in
the firmware tables. I'd like to quirk the driver, so we can fix the next error
like this, as opposed to blindly continuing.

If it really is firmware, what is it doing that causes this error, and where
does it run? Disabling APEI is just reacting after the error has occurred, when
we could prevent it happening.

I can't reproduce this on a Mustang. I assume its the different ACPI tables, not
the kernel-config.


> I just put together this patch to unify things and have a
> common 'upstream' fix.

Wouldn't passing 'hest_disable' on the cmdline do the same for all kernel versions?


> On 06/15/2018 04:14 AM, James Morse wrote:
>> On 13/06/18 19:22, Geoff Levand wrote:
>>> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
>>> a work-around for HPE ProLiant m400 APEI firmware problems.
>>>
>>> The work-around disables APEI when CONFIG_ACPI_APEI is set and
>>> m400 firmware is detected.  Without this fixup m400 systems
>>> experience errors like these on startup:
>>>
>>>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>>>   [Hardware Error]: event severity: fatal
>>>   [Hardware Error]:  Error 0, type: fatal
>>>   [Hardware Error]:   section_type: memory error
>>>   [Hardware Error]:   error_status: 0x0000000000001300
>>
>> "Access to a memory address which is not mapped to any component"
>>
>>
>>>   [Hardware Error]:   error_type: 10, invalid address
>>>   Kernel panic - not syncing: Fatal hardware error!
>>
>> Why is this a problem?
>>
>> Surely this is a valid description of an error.
> 
> The firmware bug causes this failure, not bad hardware.

I'm not talking about bad hardware here. What I think is happening is a software
bug is causing the CPU to make a bad access, which the RAS mechanism is
reporting like this because software would never be stupid enough to access an
address which is not mapped to any component! The RAS stuff believes this must
be address corruption.

I think this is a linux-driver bug, or a typo in the firmware tables, that cause
$non_existent_address to be mapped and probed.


>>> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
>>> index 7b09487ff8fb..3c315c2c7476 100644
>>> --- a/arch/arm64/kernel/acpi.c
>>> +++ b/arch/arm64/kernel/acpi.c

>>> +
>>> +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
>>> +		return;
>>> +
>>> +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
>>> +
>>> +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
>>> +		!strncmp(header->oem_table_id, "ProLiant", 8) &&

>> You should match the affected range of OEM table revisions too, that way a
>> firmware upgrade should start working, instead of being permanently disabled
>> because we think its unlikely.
> 
> The m400 has reached end of life. No one really expects to see any firmware
> update.  I don't know what the effected OEM table revisions are, and I don't
> think there is an active platform maintainer who could give that info either.
> 
> If someone can provide the info. I'll update the fix.

We can start with the version you have. You mention distro's have their own
fixes, hopefully they can supply the missing range of values.


>>> +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
>>
>> How is the CPU implementer relevant?
> 
> That was just a copy of what other fixes had.  Should I remove it?

The conclusion in the rest of this thread was HPE also produces ProLiant boxes
with a (presumably) identical HEST, but a totally different architecture.

Matching the DMI platform name as well would work round this. (or somewhere only
arm64 builds, with an appropriate comment in case we move it).


>> Nothing arch-specific here. You're adding this to arch/arm64 because
>> drivers/acpi/apei doesn't have an existing quirks table?
> 
> There was a fix submitted that had it in drivers/acpi/scan.c, but the
> ACPI maintainer said he didn't want the fix in the main ACPI code.

Specifically about a HID-hack in the core device enumeration code, as opposed to
in the driver that claims it.


> See:
> 
>   https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs)

| 'some X-Gene based platforms (Mustang and M400) with invalid DSDT.'

... sounds familiar ...

And:
https://bugzilla.redhat.com/attachment.cgi?id=1144903&action=diff

Fixes a typo in firmware description of the GIC.

Why do we think this is a new kind of firmware bug, and not a repeat of the last
two? The only difference is this is being caught by the RAS code.


> The m400 is an arm64 platform, so it seems most appropriate to
> have it in arch/arm64/kernel/acpi.c.

We don't keep all drivers under arch/arm64, I'd really like to find the driver
that is causing this, and quirk it there.

If we have to bolt the hest-stable-door, drivers/acpi/apei/hest.c looks better,
it at least doesn't have to bodge around hest_disabled not being exposed in a
compatible way. If the maintainer objects, (as x86 hasn't had to do this yet),
then we can try drivers/acpi/arm64/hest_quirks.c.

When only once architecture had to quirk stuff based on the ACPI tables, the
arch-code was a suitable dumping ground. Now there is more than one, we should
do things in a way they are useful to both architectures.


Thanks,

James

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Geoff Levand]

Hi James,

Thanks for all the comments, but my lack of access to an m400 platform, and
my lack of knowledge about the m400 limits what I can comment on and what I
can do.  

My motivation for submitting this fix was to enable CONFIG_ACPI_APEI in the
Debian 10 kernel (for the new Cavium ThunderX2 based systems).  The Debian
maintainers wanted to either have an upstream kernel fix for this m400 APEI
problem or to disable CONFIG_ACPI_APEI.  The later would be unfortunate.  It
means users would need to have custom kernels to get APEI support or go
without.

Comments follow.

On 06/18/2018 09:18 AM, James Morse wrote:
> On 15/06/18 18:17, Geoff Levand wrote:
> 
> From:
> https://bugzilla.redhat.com/show_bug.cgi?id=1574718
> 
> This is tripped by the ghes_probe() call, it finds an error has already occurred
> before ghes code is initialized.
> 
> Does this still happen if you compile without CONFIG_PCI? (that is easily half
> the dmesg that has happened before this point).
> Disabling CONFIG_EFIVAR_FS would be interesting too, as that is firmware-code we
> can't fix bugs in.

Sorry, no m400 to test these on, but looking at the code, I would say it doesn't
occur if CONFIG_PCI=n.

> Your argument here is the that the firmware-vendor only build-tested their code,
> and never noticed it notifies a fatal exception on startup. I find this hard to
> believe, especially as these systems don't have EL3.
> 
> It's much more likely a driver is causing this, possibly because of bad data in
> the firmware tables. I'd like to quirk the driver, so we can fix the next error
> like this, as opposed to blindly continuing.

That sounds OK, but which driver?

> If it really is firmware, what is it doing that causes this error, and where
> does it run? Disabling APEI is just reacting after the error has occurred, when
> we could prevent it happening.
> 
> I can't reproduce this on a Mustang. I assume its the different ACPI tables, not
> the kernel-config.

>From what I understand this is only on the m400, not the Mustang.

>> I just put together this patch to unify things and have a
>> common 'upstream' fix.
> 
> Wouldn't passing 'hest_disable' on the cmdline do the same for all kernel versions?

Yes, the current patch essentially just sets hest_disable when an m400 is
detected.  The cmdline work-around is what some have been using, but is
not an acceptable solution for the Debian maintainers.  See

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900581

>> On 06/15/2018 04:14 AM, James Morse wrote:
>>> On 13/06/18 19:22, Geoff Levand wrote:
>>>> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
>>>> a work-around for HPE ProLiant m400 APEI firmware problems.
>>>>
>>>> The work-around disables APEI when CONFIG_ACPI_APEI is set and
>>>> m400 firmware is detected.  Without this fixup m400 systems
>>>> experience errors like these on startup:
>>>>
>>>>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>>>>   [Hardware Error]: event severity: fatal
>>>>   [Hardware Error]:  Error 0, type: fatal
>>>>   [Hardware Error]:   section_type: memory error
>>>>   [Hardware Error]:   error_status: 0x0000000000001300
>>>
>>> "Access to a memory address which is not mapped to any component"
>>>
>>>
>>>>   [Hardware Error]:   error_type: 10, invalid address
>>>>   Kernel panic - not syncing: Fatal hardware error!
>>>
>>> Why is this a problem?
>>>
>>> Surely this is a valid description of an error.
>>
>> The firmware bug causes this failure, not bad hardware.
> 
> I'm not talking about bad hardware here. What I think is happening is a software
> bug is causing the CPU to make a bad access, which the RAS mechanism is
> reporting like this because software would never be stupid enough to access an
> address which is not mapped to any component! The RAS stuff believes this must
> be address corruption.
> 
> I think this is a linux-driver bug, or a typo in the firmware tables, that cause
> $non_existent_address to be mapped and probed.

>From what I know about it the problem is in the access of the firmware tables,
and that corrupted data is retrieved.  But I am not sure, as there are so many
reported m400 firmware problems it is hard to tell what exactly is what.

>>>> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
>>>> index 7b09487ff8fb..3c315c2c7476 100644
>>>> --- a/arch/arm64/kernel/acpi.c
>>>> +++ b/arch/arm64/kernel/acpi.c
> 
>>>> +
>>>> +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
>>>> +		return;
>>>> +
>>>> +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
>>>> +
>>>> +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
>>>> +		!strncmp(header->oem_table_id, "ProLiant", 8) &&
> 
>>> You should match the affected range of OEM table revisions too, that way a
>>> firmware upgrade should start working, instead of being permanently disabled
>>> because we think its unlikely.
>>
>> The m400 has reached end of life. No one really expects to see any firmware
>> update.  I don't know what the effected OEM table revisions are, and I don't
>> think there is an active platform maintainer who could give that info either.
>>
>> If someone can provide the info. I'll update the fix.
> 
> We can start with the version you have. You mention distro's have their own
> fixes, hopefully they can supply the missing range of values.

It seems you are living in a dream world...

>>>> +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
>>>
>>> How is the CPU implementer relevant?
>>
>> That was just a copy of what other fixes had.  Should I remove it?
> 
> The conclusion in the rest of this thread was HPE also produces ProLiant boxes
> with a (presumably) identical HEST, but a totally different architecture.
> 
> Matching the DMI platform name as well would work round this. (or somewhere only
> arm64 builds, with an appropriate comment in case we move it).
> 
> 
>>> Nothing arch-specific here. You're adding this to arch/arm64 because
>>> drivers/acpi/apei doesn't have an existing quirks table?
>>
>> There was a fix submitted that had it in drivers/acpi/scan.c, but the
>> ACPI maintainer said he didn't want the fix in the main ACPI code.
> 
> Specifically about a HID-hack in the core device enumeration code, as opposed to
> in the driver that claims it.
> 
> 
>> See:
>>
>>   https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs)
> 
> | 'some X-Gene based platforms (Mustang and M400) with invalid DSDT.'
> 
> ... sounds familiar ...
> 
> And:
> https://bugzilla.redhat.com/attachment.cgi?id=1144903&action=diff
> 
> Fixes a typo in firmware description of the GIC.
> 
> Why do we think this is a new kind of firmware bug, and not a repeat of the last
> two? The only difference is this is being caught by the RAS code.

It could very well be.

>> The m400 is an arm64 platform, so it seems most appropriate to
>> have it in arch/arm64/kernel/acpi.c.
> 
> We don't keep all drivers under arch/arm64, I'd really like to find the driver
> that is causing this, and quirk it there.
> 
> If we have to bolt the hest-stable-door, drivers/acpi/apei/hest.c looks better,
> it at least doesn't have to bodge around hest_disabled not being exposed in a
> compatible way. If the maintainer objects, (as x86 hasn't had to do this yet),
> then we can try drivers/acpi/arm64/hest_quirks.c.

We need to call acpi_fixup_m400_quirks early enough to get hest_disable set
before it is used.  To do that from drivers/acpi/ code we would need have
it as arch_initcall, or still call it from acpi_boot_table_init.  I think
having it as a static routine and the call in arch/arm64/kernel/acpi.c is
cleaner.

> When only once architecture had to quirk stuff based on the ACPI tables, the
> arch-code was a suitable dumping ground. Now there is more than one, we should
> do things in a way they are useful to both architectures.

As I mentioned in the opening, I don't have access to m400 equipment.  There is
little I can do.  I could move acpi_fixup_m400_quirks into
drivers/acpi/apei/hest.c and submit that to the ACPI maintainer, but based on
https://lkml.org/lkml/2018/4/19/1020, I don't think it would be accepted.

-Geoff

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Geoff Levand]

Hi James,

Thanks for all the comments, but my lack of access to an m400 platform, and
my lack of knowledge about the m400 limits what I can comment on and what I
can do.  

My motivation for submitting this fix was to enable CONFIG_ACPI_APEI in the
Debian 10 kernel (for the new Cavium ThunderX2 based systems).  The Debian
maintainers wanted to either have an upstream kernel fix for this m400 APEI
problem or to disable CONFIG_ACPI_APEI.  The later would be unfortunate.  It
means users would need to have custom kernels to get APEI support or go
without.

Comments follow.

On 06/18/2018 09:18 AM, James Morse wrote:
> On 15/06/18 18:17, Geoff Levand wrote:
> 
> From:
> https://bugzilla.redhat.com/show_bug.cgi?id=1574718
> 
> This is tripped by the ghes_probe() call, it finds an error has already occurred
> before ghes code is initialized.
> 
> Does this still happen if you compile without CONFIG_PCI? (that is easily half
> the dmesg that has happened before this point).
> Disabling CONFIG_EFIVAR_FS would be interesting too, as that is firmware-code we
> can't fix bugs in.

Sorry, no m400 to test these on, but looking at the code, I would say it doesn't
occur if CONFIG_PCI=n.

> Your argument here is the that the firmware-vendor only build-tested their code,
> and never noticed it notifies a fatal exception on startup. I find this hard to
> believe, especially as these systems don't have EL3.
> 
> It's much more likely a driver is causing this, possibly because of bad data in
> the firmware tables. I'd like to quirk the driver, so we can fix the next error
> like this, as opposed to blindly continuing.

That sounds OK, but which driver?

> If it really is firmware, what is it doing that causes this error, and where
> does it run? Disabling APEI is just reacting after the error has occurred, when
> we could prevent it happening.
> 
> I can't reproduce this on a Mustang. I assume its the different ACPI tables, not
> the kernel-config.

>From what I understand this is only on the m400, not the Mustang.

>> I just put together this patch to unify things and have a
>> common 'upstream' fix.
> 
> Wouldn't passing 'hest_disable' on the cmdline do the same for all kernel versions?

Yes, the current patch essentially just sets hest_disable when an m400 is
detected.  The cmdline work-around is what some have been using, but is
not an acceptable solution for the Debian maintainers.  See

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900581

>> On 06/15/2018 04:14 AM, James Morse wrote:
>>> On 13/06/18 19:22, Geoff Levand wrote:
>>>> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
>>>> a work-around for HPE ProLiant m400 APEI firmware problems.
>>>>
>>>> The work-around disables APEI when CONFIG_ACPI_APEI is set and
>>>> m400 firmware is detected.  Without this fixup m400 systems
>>>> experience errors like these on startup:
>>>>
>>>>   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
>>>>   [Hardware Error]: event severity: fatal
>>>>   [Hardware Error]:  Error 0, type: fatal
>>>>   [Hardware Error]:   section_type: memory error
>>>>   [Hardware Error]:   error_status: 0x0000000000001300
>>>
>>> "Access to a memory address which is not mapped to any component"
>>>
>>>
>>>>   [Hardware Error]:   error_type: 10, invalid address
>>>>   Kernel panic - not syncing: Fatal hardware error!
>>>
>>> Why is this a problem?
>>>
>>> Surely this is a valid description of an error.
>>
>> The firmware bug causes this failure, not bad hardware.
> 
> I'm not talking about bad hardware here. What I think is happening is a software
> bug is causing the CPU to make a bad access, which the RAS mechanism is
> reporting like this because software would never be stupid enough to access an
> address which is not mapped to any component! The RAS stuff believes this must
> be address corruption.
> 
> I think this is a linux-driver bug, or a typo in the firmware tables, that cause
> $non_existent_address to be mapped and probed.

>From what I know about it the problem is in the access of the firmware tables,
and that corrupted data is retrieved.  But I am not sure, as there are so many
reported m400 firmware problems it is hard to tell what exactly is what.

>>>> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
>>>> index 7b09487ff8fb..3c315c2c7476 100644
>>>> --- a/arch/arm64/kernel/acpi.c
>>>> +++ b/arch/arm64/kernel/acpi.c
> 
>>>> +
>>>> +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
>>>> +		return;
>>>> +
>>>> +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
>>>> +
>>>> +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
>>>> +		!strncmp(header->oem_table_id, "ProLiant", 8) &&
> 
>>> You should match the affected range of OEM table revisions too, that way a
>>> firmware upgrade should start working, instead of being permanently disabled
>>> because we think its unlikely.
>>
>> The m400 has reached end of life. No one really expects to see any firmware
>> update.  I don't know what the effected OEM table revisions are, and I don't
>> think there is an active platform maintainer who could give that info either.
>>
>> If someone can provide the info. I'll update the fix.
> 
> We can start with the version you have. You mention distro's have their own
> fixes, hopefully they can supply the missing range of values.

It seems you are living in a dream world...

>>>> +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
>>>
>>> How is the CPU implementer relevant?
>>
>> That was just a copy of what other fixes had.  Should I remove it?
> 
> The conclusion in the rest of this thread was HPE also produces ProLiant boxes
> with a (presumably) identical HEST, but a totally different architecture.
> 
> Matching the DMI platform name as well would work round this. (or somewhere only
> arm64 builds, with an appropriate comment in case we move it).
> 
> 
>>> Nothing arch-specific here. You're adding this to arch/arm64 because
>>> drivers/acpi/apei doesn't have an existing quirks table?
>>
>> There was a fix submitted that had it in drivers/acpi/scan.c, but the
>> ACPI maintainer said he didn't want the fix in the main ACPI code.
> 
> Specifically about a HID-hack in the core device enumeration code, as opposed to
> in the driver that claims it.
> 
> 
>> See:
>>
>>   https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs)
> 
> | 'some X-Gene based platforms (Mustang and M400) with invalid DSDT.'
> 
> ... sounds familiar ...
> 
> And:
> https://bugzilla.redhat.com/attachment.cgi?id=1144903&action=diff
> 
> Fixes a typo in firmware description of the GIC.
> 
> Why do we think this is a new kind of firmware bug, and not a repeat of the last
> two? The only difference is this is being caught by the RAS code.

It could very well be.

>> The m400 is an arm64 platform, so it seems most appropriate to
>> have it in arch/arm64/kernel/acpi.c.
> 
> We don't keep all drivers under arch/arm64, I'd really like to find the driver
> that is causing this, and quirk it there.
> 
> If we have to bolt the hest-stable-door, drivers/acpi/apei/hest.c looks better,
> it at least doesn't have to bodge around hest_disabled not being exposed in a
> compatible way. If the maintainer objects, (as x86 hasn't had to do this yet),
> then we can try drivers/acpi/arm64/hest_quirks.c.

We need to call acpi_fixup_m400_quirks early enough to get hest_disable set
before it is used.  To do that from drivers/acpi/ code we would need have
it as arch_initcall, or still call it from acpi_boot_table_init.  I think
having it as a static routine and the call in arch/arm64/kernel/acpi.c is
cleaner.

> When only once architecture had to quirk stuff based on the ACPI tables, the
> arch-code was a suitable dumping ground. Now there is more than one, we should
> do things in a way they are useful to both architectures.

As I mentioned in the opening, I don't have access to m400 equipment.  There is
little I can do.  I could move acpi_fixup_m400_quirks into
drivers/acpi/apei/hest.c and submit that to the ACPI maintainer, but based on
https://lkml.org/lkml/2018/4/19/1020, I don't think it would be accepted.

-Geoff

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Mon, 2018-06-18 at 11:04 -0700, Geoff Levand wrote:
> Hi James,
> 
> Thanks for all the comments, but my lack of access to an m400 platform, and
> my lack of knowledge about the m400 limits what I can comment on and what I
> can do.  

I can take another look at this on an m400 here. I don't believe it is a
memory access to physical space with nothing attached to it. I seem to recall
an errata with xgene-1 where such accesses cause the cpu to halt. But I could
be misremembering that. I have no trouble believing the firmware ras code was
untested. It is probably some boilerplate code built in before ras was supported
in kernel. But the problem occurs early enough in boot where there can't be
that many things that would cause a problem on m400 and not mustang so I'll
look again.

> 
> My motivation for submitting this fix was to enable CONFIG_ACPI_APEI in the
> Debian 10 kernel (for the new Cavium ThunderX2 based systems).  The Debian
> maintainers wanted to either have an upstream kernel fix for this m400 APEI
> problem or to disable CONFIG_ACPI_APEI.  The later would be unfortunate.  It
> means users would need to have custom kernels to get APEI support or go
> without.
> 
> Comments follow.
> 
> On 06/18/2018 09:18 AM, James Morse wrote:
> > On 15/06/18 18:17, Geoff Levand wrote:
> > 
> > From:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1574718
> > 
> > This is tripped by the ghes_probe() call, it finds an error has already occurred
> > before ghes code is initialized.
> > 
> > Does this still happen if you compile without CONFIG_PCI? (that is easily half
> > the dmesg that has happened before this point).
> > Disabling CONFIG_EFIVAR_FS would be interesting too, as that is firmware-code we
> > can't fix bugs in.
> 
> Sorry, no m400 to test these on, but looking at the code, I would say it doesn't
> occur if CONFIG_PCI=n.
> 
> > Your argument here is the that the firmware-vendor only build-tested their code,
> > and never noticed it notifies a fatal exception on startup. I find this hard to
> > believe, especially as these systems don't have EL3.
> > 
> > It's much more likely a driver is causing this, possibly because of bad data in
> > the firmware tables. I'd like to quirk the driver, so we can fix the next error
> > like this, as opposed to blindly continuing.
> 
> That sounds OK, but which driver?
> 
> > If it really is firmware, what is it doing that causes this error, and where
> > does it run? Disabling APEI is just reacting after the error has occurred, when
> > we could prevent it happening.
> > 
> > I can't reproduce this on a Mustang. I assume its the different ACPI tables, not
> > the kernel-config.
> 
> From what I understand this is only on the m400, not the Mustang.
> 
> > > I just put together this patch to unify things and have a
> > > common 'upstream' fix.
> > 
> > Wouldn't passing 'hest_disable' on the cmdline do the same for all kernel versions?
> 
> Yes, the current patch essentially just sets hest_disable when an m400 is
> detected.  The cmdline work-around is what some have been using, but is
> not an acceptable solution for the Debian maintainers.  See
> 
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900581
> 
> > > On 06/15/2018 04:14 AM, James Morse wrote:
> > > > On 13/06/18 19:22, Geoff Levand wrote:
> > > > > Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
> > > > > a work-around for HPE ProLiant m400 APEI firmware problems.
> > > > > 
> > > > > The work-around disables APEI when CONFIG_ACPI_APEI is set and
> > > > > m400 firmware is detected.  Without this fixup m400 systems
> > > > > experience errors like these on startup:
> > > > > 
> > > > >   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
> > > > >   [Hardware Error]: event severity: fatal
> > > > >   [Hardware Error]:  Error 0, type: fatal
> > > > >   [Hardware Error]:   section_type: memory error
> > > > >   [Hardware Error]:   error_status: 0x0000000000001300
> > > > 
> > > > "Access to a memory address which is not mapped to any component"
> > > > 
> > > > 
> > > > >   [Hardware Error]:   error_type: 10, invalid address
> > > > >   Kernel panic - not syncing: Fatal hardware error!
> > > > 
> > > > Why is this a problem?
> > > > 
> > > > Surely this is a valid description of an error.
> > > 
> > > The firmware bug causes this failure, not bad hardware.
> > 
> > I'm not talking about bad hardware here. What I think is happening is a software
> > bug is causing the CPU to make a bad access, which the RAS mechanism is
> > reporting like this because software would never be stupid enough to access an
> > address which is not mapped to any component! The RAS stuff believes this must
> > be address corruption.
> > 
> > I think this is a linux-driver bug, or a typo in the firmware tables, that cause
> > $non_existent_address to be mapped and probed.
> 
> From what I know about it the problem is in the access of the firmware tables,
> and that corrupted data is retrieved.  But I am not sure, as there are so many
> reported m400 firmware problems it is hard to tell what exactly is what.
> 
> > > > > diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> > > > > index 7b09487ff8fb..3c315c2c7476 100644
> > > > > --- a/arch/arm64/kernel/acpi.c
> > > > > +++ b/arch/arm64/kernel/acpi.c
> > > > > +
> > > > > +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
> > > > > +		return;
> > > > > +
> > > > > +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
> > > > > +
> > > > > +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
> > > > > +		!strncmp(header->oem_table_id, "ProLiant", 8) &&
> > > > You should match the affected range of OEM table revisions too, that way a
> > > > firmware upgrade should start working, instead of being permanently disabled
> > > > because we think its unlikely.
> > > 
> > > The m400 has reached end of life. No one really expects to see any firmware
> > > update.  I don't know what the effected OEM table revisions are, and I don't
> > > think there is an active platform maintainer who could give that info either.
> > > 
> > > If someone can provide the info. I'll update the fix.
> > 
> > We can start with the version you have. You mention distro's have their own
> > fixes, hopefully they can supply the missing range of values.
> 
> It seems you are living in a dream world...
> 
> > > > > +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> > > > 
> > > > How is the CPU implementer relevant?
> > > 
> > > That was just a copy of what other fixes had.  Should I remove it?
> > 
> > The conclusion in the rest of this thread was HPE also produces ProLiant boxes
> > with a (presumably) identical HEST, but a totally different architecture.
> > 
> > Matching the DMI platform name as well would work round this. (or somewhere only
> > arm64 builds, with an appropriate comment in case we move it).
> > 
> > 
> > > > Nothing arch-specific here. You're adding this to arch/arm64 because
> > > > drivers/acpi/apei doesn't have an existing quirks table?
> > > 
> > > There was a fix submitted that had it in drivers/acpi/scan.c, but the
> > > ACPI maintainer said he didn't want the fix in the main ACPI code.
> > 
> > Specifically about a HID-hack in the core device enumeration code, as opposed to
> > in the driver that claims it.
> > 
> > 
> > > See:
> > > 
> > >   https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs)
> > > 'some X-Gene based platforms (Mustang and M400) with invalid DSDT.'
> > 
> > ... sounds familiar ...
> > 
> > And:
> > https://bugzilla.redhat.com/attachment.cgi?id=1144903&action=diff
> > 
> > Fixes a typo in firmware description of the GIC.
> > 
> > Why do we think this is a new kind of firmware bug, and not a repeat of the last
> > two? The only difference is this is being caught by the RAS code.
> 
> It could very well be.
> 
> > > The m400 is an arm64 platform, so it seems most appropriate to
> > > have it in arch/arm64/kernel/acpi.c.
> > 
> > We don't keep all drivers under arch/arm64, I'd really like to find the driver
> > that is causing this, and quirk it there.
> > 
> > If we have to bolt the hest-stable-door, drivers/acpi/apei/hest.c looks better,
> > it at least doesn't have to bodge around hest_disabled not being exposed in a
> > compatible way. If the maintainer objects, (as x86 hasn't had to do this yet),
> > then we can try drivers/acpi/arm64/hest_quirks.c.
> 
> We need to call acpi_fixup_m400_quirks early enough to get hest_disable set
> before it is used.  To do that from drivers/acpi/ code we would need have
> it as arch_initcall, or still call it from acpi_boot_table_init.  I think
> having it as a static routine and the call in arch/arm64/kernel/acpi.c is
> cleaner.
> 
> > When only once architecture had to quirk stuff based on the ACPI tables, the
> > arch-code was a suitable dumping ground. Now there is more than one, we should
> > do things in a way they are useful to both architectures.
> 
> As I mentioned in the opening, I don't have access to m400 equipment.  There is
> little I can do.  I could move acpi_fixup_m400_quirks into
> drivers/acpi/apei/hest.c and submit that to the ACPI maintainer, but based on
> https://lkml.org/lkml/2018/4/19/1020, I don't think it would be accepted.
> 
> -Geoff
> 

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Mon, 2018-06-18 at 11:04 -0700, Geoff Levand wrote:
> Hi James,
> 
> Thanks for all the comments, but my lack of access to an m400 platform, and
> my lack of knowledge about the m400 limits what I can comment on and what I
> can do.  

I can take another look at this on an m400 here. I don't believe it is a
memory access to physical space with nothing attached to it. I seem to recall
an errata with xgene-1 where such accesses cause the cpu to halt. But I could
be misremembering that. I have no trouble believing the firmware ras code was
untested. It is probably some boilerplate code built in before ras was supported
in kernel. But the problem occurs early enough in boot where there can't be
that many things that would cause a problem on m400 and not mustang so I'll
look again.

> 
> My motivation for submitting this fix was to enable CONFIG_ACPI_APEI in the
> Debian 10 kernel (for the new Cavium ThunderX2 based systems).  The Debian
> maintainers wanted to either have an upstream kernel fix for this m400 APEI
> problem or to disable CONFIG_ACPI_APEI.  The later would be unfortunate.  It
> means users would need to have custom kernels to get APEI support or go
> without.
> 
> Comments follow.
> 
> On 06/18/2018 09:18 AM, James Morse wrote:
> > On 15/06/18 18:17, Geoff Levand wrote:
> > 
> > From:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1574718
> > 
> > This is tripped by the ghes_probe() call, it finds an error has already occurred
> > before ghes code is initialized.
> > 
> > Does this still happen if you compile without CONFIG_PCI? (that is easily half
> > the dmesg that has happened before this point).
> > Disabling CONFIG_EFIVAR_FS would be interesting too, as that is firmware-code we
> > can't fix bugs in.
> 
> Sorry, no m400 to test these on, but looking at the code, I would say it doesn't
> occur if CONFIG_PCI=n.
> 
> > Your argument here is the that the firmware-vendor only build-tested their code,
> > and never noticed it notifies a fatal exception on startup. I find this hard to
> > believe, especially as these systems don't have EL3.
> > 
> > It's much more likely a driver is causing this, possibly because of bad data in
> > the firmware tables. I'd like to quirk the driver, so we can fix the next error
> > like this, as opposed to blindly continuing.
> 
> That sounds OK, but which driver?
> 
> > If it really is firmware, what is it doing that causes this error, and where
> > does it run? Disabling APEI is just reacting after the error has occurred, when
> > we could prevent it happening.
> > 
> > I can't reproduce this on a Mustang. I assume its the different ACPI tables, not
> > the kernel-config.
> 
> From what I understand this is only on the m400, not the Mustang.
> 
> > > I just put together this patch to unify things and have a
> > > common 'upstream' fix.
> > 
> > Wouldn't passing 'hest_disable' on the cmdline do the same for all kernel versions?
> 
> Yes, the current patch essentially just sets hest_disable when an m400 is
> detected.  The cmdline work-around is what some have been using, but is
> not an acceptable solution for the Debian maintainers.  See
> 
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900581
> 
> > > On 06/15/2018 04:14 AM, James Morse wrote:
> > > > On 13/06/18 19:22, Geoff Levand wrote:
> > > > > Adds a new ACPI init routine acpi_fixup_m400_quirks that adds
> > > > > a work-around for HPE ProLiant m400 APEI firmware problems.
> > > > > 
> > > > > The work-around disables APEI when CONFIG_ACPI_APEI is set and
> > > > > m400 firmware is detected.  Without this fixup m400 systems
> > > > > experience errors like these on startup:
> > > > > 
> > > > >   [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
> > > > >   [Hardware Error]: event severity: fatal
> > > > >   [Hardware Error]:  Error 0, type: fatal
> > > > >   [Hardware Error]:   section_type: memory error
> > > > >   [Hardware Error]:   error_status: 0x0000000000001300
> > > > 
> > > > "Access to a memory address which is not mapped to any component"
> > > > 
> > > > 
> > > > >   [Hardware Error]:   error_type: 10, invalid address
> > > > >   Kernel panic - not syncing: Fatal hardware error!
> > > > 
> > > > Why is this a problem?
> > > > 
> > > > Surely this is a valid description of an error.
> > > 
> > > The firmware bug causes this failure, not bad hardware.
> > 
> > I'm not talking about bad hardware here. What I think is happening is a software
> > bug is causing the CPU to make a bad access, which the RAS mechanism is
> > reporting like this because software would never be stupid enough to access an
> > address which is not mapped to any component! The RAS stuff believes this must
> > be address corruption.
> > 
> > I think this is a linux-driver bug, or a typo in the firmware tables, that cause
> > $non_existent_address to be mapped and probed.
> 
> From what I know about it the problem is in the access of the firmware tables,
> and that corrupted data is retrieved.  But I am not sure, as there are so many
> reported m400 firmware problems it is hard to tell what exactly is what.
> 
> > > > > diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> > > > > index 7b09487ff8fb..3c315c2c7476 100644
> > > > > --- a/arch/arm64/kernel/acpi.c
> > > > > +++ b/arch/arm64/kernel/acpi.c
> > > > > +
> > > > > +	if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED)
> > > > > +		return;
> > > > > +
> > > > > +	status = acpi_get_table(ACPI_SIG_HEST, 0, &header);
> > > > > +
> > > > > +	if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE   ", 6) &&
> > > > > +		!strncmp(header->oem_table_id, "ProLiant", 8) &&
> > > > You should match the affected range of OEM table revisions too, that way a
> > > > firmware upgrade should start working, instead of being permanently disabled
> > > > because we think its unlikely.
> > > 
> > > The m400 has reached end of life. No one really expects to see any firmware
> > > update.  I don't know what the effected OEM table revisions are, and I don't
> > > think there is an active platform maintainer who could give that info either.
> > > 
> > > If someone can provide the info. I'll update the fix.
> > 
> > We can start with the version you have. You mention distro's have their own
> > fixes, hopefully they can supply the missing range of values.
> 
> It seems you are living in a dream world...
> 
> > > > > +		MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) {
> > > > 
> > > > How is the CPU implementer relevant?
> > > 
> > > That was just a copy of what other fixes had.  Should I remove it?
> > 
> > The conclusion in the rest of this thread was HPE also produces ProLiant boxes
> > with a (presumably) identical HEST, but a totally different architecture.
> > 
> > Matching the DMI platform name as well would work round this. (or somewhere only
> > arm64 builds, with an appropriate comment in case we move it).
> > 
> > 
> > > > Nothing arch-specific here. You're adding this to arch/arm64 because
> > > > drivers/acpi/apei doesn't have an existing quirks table?
> > > 
> > > There was a fix submitted that had it in drivers/acpi/scan.c, but the
> > > ACPI maintainer said he didn't want the fix in the main ACPI code.
> > 
> > Specifically about a HID-hack in the core device enumeration code, as opposed to
> > in the driver that claims it.
> > 
> > 
> > > See:
> > > 
> > >   https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs)
> > > 'some X-Gene based platforms (Mustang and M400) with invalid DSDT.'
> > 
> > ... sounds familiar ...
> > 
> > And:
> > https://bugzilla.redhat.com/attachment.cgi?id=1144903&action=diff
> > 
> > Fixes a typo in firmware description of the GIC.
> > 
> > Why do we think this is a new kind of firmware bug, and not a repeat of the last
> > two? The only difference is this is being caught by the RAS code.
> 
> It could very well be.
> 
> > > The m400 is an arm64 platform, so it seems most appropriate to
> > > have it in arch/arm64/kernel/acpi.c.
> > 
> > We don't keep all drivers under arch/arm64, I'd really like to find the driver
> > that is causing this, and quirk it there.
> > 
> > If we have to bolt the hest-stable-door, drivers/acpi/apei/hest.c looks better,
> > it at least doesn't have to bodge around hest_disabled not being exposed in a
> > compatible way. If the maintainer objects, (as x86 hasn't had to do this yet),
> > then we can try drivers/acpi/arm64/hest_quirks.c.
> 
> We need to call acpi_fixup_m400_quirks early enough to get hest_disable set
> before it is used.  To do that from drivers/acpi/ code we would need have
> it as arch_initcall, or still call it from acpi_boot_table_init.  I think
> having it as a static routine and the call in arch/arm64/kernel/acpi.c is
> cleaner.
> 
> > When only once architecture had to quirk stuff based on the ACPI tables, the
> > arch-code was a suitable dumping ground. Now there is more than one, we should
> > do things in a way they are useful to both architectures.
> 
> As I mentioned in the opening, I don't have access to m400 equipment.  There is
> little I can do.  I could move acpi_fixup_m400_quirks into
> drivers/acpi/apei/hest.c and submit that to the ACPI maintainer, but based on
> https://lkml.org/lkml/2018/4/19/1020, I don't think it would be accepted.
> 
> -Geoff
> 

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[James Morse]

Hi Mark,

On 18/06/18 23:18, Mark Salter wrote:
> On Mon, 2018-06-18 at 11:04 -0700, Geoff Levand wrote:
>> Thanks for all the comments, but my lack of access to an m400 platform, and
>> my lack of knowledge about the m400 limits what I can comment on and what I
>> can do.  
> 
> I can take another look at this on an m400 here.

Thanks!


> I don't believe it is a
> memory access to physical space with nothing attached to it.

That is what the CPER records are describing though.


> I seem to recall
> an errata with xgene-1 where such accesses cause the cpu to halt. But I could
> be misremembering that. I have no trouble believing the firmware ras code was
> untested. It is probably some boilerplate code built in before ras was supported
> in kernel.

It would be interesting to know which GHES this error is being found in, and
whether the Error Status Block points anywhere (or at an empty block) when Linux
is started from UEFI.

If there is something in the Error Status Block out of UEFI, then this must be
something triggered by UEFI, or a bug that can be fixed by UEFI clearing out the
CPER records.

https://bugzilla.redhat.com/show_bug.cgi?id=1285107
suggests redhat can rebuild the UEFI firmware for this box.


If there is nothing in the Error Status Block when Linux is started, surely
Linux is doing something to cause this to happen. I'd like to find out what, as
its probably a software bug.


(The case where disabling HEST would be the right thing to do is if there is a
bogus GHES->GAS entry in GHES.0, the access to which causes GHES.1 to be
populated with 'Access to an address not mapped to any component', which we find
next. If this is the case it would be better to check GHES entries against the
UEFI memory map to check this is memory, and it was reserved.)


> But the problem occurs early enough in boot where there can't be
> that many things that would cause a problem on m400 and not mustang so I'll
> look again.

Playing spot the difference in the dmesg, I'd check for smoke coming out of:
| acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
| xgene-gpio APMC0D14:00: X-Gene GPIO driver registered.
| pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22

If the firmware description of the GIC is wrong in someway, disabling KVM may be
worth testing too.


Thanks,

James

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[James Morse]

Hi Mark,

On 18/06/18 23:18, Mark Salter wrote:
> On Mon, 2018-06-18 at 11:04 -0700, Geoff Levand wrote:
>> Thanks for all the comments, but my lack of access to an m400 platform, and
>> my lack of knowledge about the m400 limits what I can comment on and what I
>> can do.  
> 
> I can take another look at this on an m400 here.

Thanks!


> I don't believe it is a
> memory access to physical space with nothing attached to it.

That is what the CPER records are describing though.


> I seem to recall
> an errata with xgene-1 where such accesses cause the cpu to halt. But I could
> be misremembering that. I have no trouble believing the firmware ras code was
> untested. It is probably some boilerplate code built in before ras was supported
> in kernel.

It would be interesting to know which GHES this error is being found in, and
whether the Error Status Block points anywhere (or at an empty block) when Linux
is started from UEFI.

If there is something in the Error Status Block out of UEFI, then this must be
something triggered by UEFI, or a bug that can be fixed by UEFI clearing out the
CPER records.

https://bugzilla.redhat.com/show_bug.cgi?id=1285107
suggests redhat can rebuild the UEFI firmware for this box.


If there is nothing in the Error Status Block when Linux is started, surely
Linux is doing something to cause this to happen. I'd like to find out what, as
its probably a software bug.


(The case where disabling HEST would be the right thing to do is if there is a
bogus GHES->GAS entry in GHES.0, the access to which causes GHES.1 to be
populated with 'Access to an address not mapped to any component', which we find
next. If this is the case it would be better to check GHES entries against the
UEFI memory map to check this is memory, and it was reserved.)


> But the problem occurs early enough in boot where there can't be
> that many things that would cause a problem on m400 and not mustang so I'll
> look again.

Playing spot the difference in the dmesg, I'd check for smoke coming out of:
| acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
| xgene-gpio APMC0D14:00: X-Gene GPIO driver registered.
| pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22

If the firmware description of the GIC is wrong in someway, disabling KVM may be
worth testing too.


Thanks,

James

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Tue, 2018-06-19 at 11:21 +0100, James Morse wrote:
> Hi Mark,
> 
> On 18/06/18 23:18, Mark Salter wrote:
> > On Mon, 2018-06-18 at 11:04 -0700, Geoff Levand wrote:
> > > Thanks for all the comments, but my lack of access to an m400 platform, and
> > > my lack of knowledge about the m400 limits what I can comment on and what I
> > > can do.  
> > 
> > I can take another look at this on an m400 here.
> 
> Thanks!
> 
> 
> > I don't believe it is a
> > memory access to physical space with nothing attached to it.
> 
> That is what the CPER records are describing though.

Yes.

> 
> 
> > I seem to recall
> > an errata with xgene-1 where such accesses cause the cpu to halt. But I could
> > be misremembering that. I have no trouble believing the firmware ras code was
> > untested. It is probably some boilerplate code built in before ras was supported
> > in kernel.
> 
> It would be interesting to know which GHES this error is being found in, and
> whether the Error Status Block points anywhere (or at an empty block) when Linux
> is started from UEFI.
> 
> If there is something in the Error Status Block out of UEFI, then this must be
> something triggered by UEFI, or a bug that can be fixed by UEFI clearing out the
> CPER records.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1285107
> suggests redhat can rebuild the UEFI firmware for this box.
> 
> 
> If there is nothing in the Error Status Block when Linux is started, surely
> Linux is doing something to cause this to happen. I'd like to find out what, as
> its probably a software bug.
> 
> 
> (The case where disabling HEST would be the right thing to do is if there is a
> bogus GHES->GAS entry in GHES.0, the access to which causes GHES.1 to be
> populated with 'Access to an address not mapped to any component', which we find
> next. If this is the case it would be better to check GHES entries against the
> UEFI memory map to check this is memory, and it was reserved.)
> 
> 
> > But the problem occurs early enough in boot where there can't be
> > that many things that would cause a problem on m400 and not mustang so I'll
> > look again.
> 
> Playing spot the difference in the dmesg, I'd check for smoke coming out of:
> > acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> > xgene-gpio APMC0D14:00: X-Gene GPIO driver registered.
> > pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22

I've eliminated these by building a kernel with minimalized config and hacks (ACPI
requires PCI, so I added code to prevent the root complexe from being probed). I
also eliminated all the xgene-specific devices from the config (network, sata,
etc). Still hit the ghes panic.

I'm going to hack something to get to the ghes info earlier in boot and
check the things you mention above wrt Error Status Block and GHES.0.


> 
> If the firmware description of the GIC is wrong in someway, disabling KVM may be
> worth testing too.
> 
> 
> Thanks,
> 
> James

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Tue, 2018-06-19 at 11:21 +0100, James Morse wrote:
> Hi Mark,
> 
> On 18/06/18 23:18, Mark Salter wrote:
> > On Mon, 2018-06-18 at 11:04 -0700, Geoff Levand wrote:
> > > Thanks for all the comments, but my lack of access to an m400 platform, and
> > > my lack of knowledge about the m400 limits what I can comment on and what I
> > > can do.  
> > 
> > I can take another look at this on an m400 here.
> 
> Thanks!
> 
> 
> > I don't believe it is a
> > memory access to physical space with nothing attached to it.
> 
> That is what the CPER records are describing though.

Yes.

> 
> 
> > I seem to recall
> > an errata with xgene-1 where such accesses cause the cpu to halt. But I could
> > be misremembering that. I have no trouble believing the firmware ras code was
> > untested. It is probably some boilerplate code built in before ras was supported
> > in kernel.
> 
> It would be interesting to know which GHES this error is being found in, and
> whether the Error Status Block points anywhere (or at an empty block) when Linux
> is started from UEFI.
> 
> If there is something in the Error Status Block out of UEFI, then this must be
> something triggered by UEFI, or a bug that can be fixed by UEFI clearing out the
> CPER records.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1285107
> suggests redhat can rebuild the UEFI firmware for this box.
> 
> 
> If there is nothing in the Error Status Block when Linux is started, surely
> Linux is doing something to cause this to happen. I'd like to find out what, as
> its probably a software bug.
> 
> 
> (The case where disabling HEST would be the right thing to do is if there is a
> bogus GHES->GAS entry in GHES.0, the access to which causes GHES.1 to be
> populated with 'Access to an address not mapped to any component', which we find
> next. If this is the case it would be better to check GHES entries against the
> UEFI memory map to check this is memory, and it was reserved.)
> 
> 
> > But the problem occurs early enough in boot where there can't be
> > that many things that would cause a problem on m400 and not mustang so I'll
> > look again.
> 
> Playing spot the difference in the dmesg, I'd check for smoke coming out of:
> > acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> > xgene-gpio APMC0D14:00: X-Gene GPIO driver registered.
> > pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22

I've eliminated these by building a kernel with minimalized config and hacks (ACPI
requires PCI, so I added code to prevent the root complexe from being probed). I
also eliminated all the xgene-specific devices from the config (network, sata,
etc). Still hit the ghes panic.

I'm going to hack something to get to the ghes info earlier in boot and
check the things you mention above wrt Error Status Block and GHES.0.


> 
> If the firmware description of the GIC is wrong in someway, disabling KVM may be
> worth testing too.
> 
> 
> Thanks,
> 
> James

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
> I'm going to hack something to get to the ghes info earlier in boot and
> check the things you mention above wrt Error Status Block and GHES.0.

So I had to end up instrumenting the EFI stub to see where the error came
from. At the start of the stub, there is no GHES.2 error. The error first
shows up after the stub's call to ExitBootServices returns. So it looks
like the firmware itself is causing the error. There's still a chance that
the stub is doing something wrong with the memory map passed to the
firmware, so I'll try to eliminate that as well.

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
> I'm going to hack something to get to the ghes info earlier in boot and
> check the things you mention above wrt Error Status Block and GHES.0.

So I had to end up instrumenting the EFI stub to see where the error came
from. At the start of the stub, there is no GHES.2 error. The error first
shows up after the stub's call to ExitBootServices returns. So it looks
like the firmware itself is causing the error. There's still a chance that
the stub is doing something wrong with the memory map passed to the
firmware, so I'll try to eliminate that as well.

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[James Morse]

Hi Mark,

Thanks for shed-ing some light on what is going on here!

On 25/06/18 16:34, Mark Salter wrote:
> On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
>> I'm going to hack something to get to the ghes info earlier in boot and
>> check the things you mention above wrt Error Status Block and GHES.0.
> 
> So I had to end up instrumenting the EFI stub to see where the error came
> from. At the start of the stub, there is no GHES.2 error. The error first
> shows up after the stub's call to ExitBootServices returns.

What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.

These systems don't have EL3, so the CPU must continue running while something
external generates the CPER records. The records being visible is the last point
the faulty-access could have been made, with the window of time depending on how
fast this external-thing receives and processes the error.


> So it looks
> like the firmware itself is causing the error. There's still a chance that
> the stub is doing something wrong with the memory map passed to the
> firmware, so I'll try to eliminate that as well.

adding delay loops will help prove the EFIStub is innocent.

Are there any optional drivers being loaded by UEFI? (can you remove any USB
mass storage drives for instance).

Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)

https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
description of the GIC, comments 15 and 16 show a UEFI patch to something other
than the upstream platforms tree[0], and new firmware being tested.
(although this may be wishful thinking)

It looks like quirking this based on the DMI platform name and UEFI version will
be what we need. We could discard anything in the error status block areas at
ghes_probe() time based on this quirk, but we may have missed other problems
during boot, giving a false sense of security.


Thanks,

James


[0] Might be wrong, but this is where I look:
https://github.com/tianocore/edk2-platforms.git

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[James Morse]

Hi Mark,

Thanks for shed-ing some light on what is going on here!

On 25/06/18 16:34, Mark Salter wrote:
> On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
>> I'm going to hack something to get to the ghes info earlier in boot and
>> check the things you mention above wrt Error Status Block and GHES.0.
> 
> So I had to end up instrumenting the EFI stub to see where the error came
> from. At the start of the stub, there is no GHES.2 error. The error first
> shows up after the stub's call to ExitBootServices returns.

What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.

These systems don't have EL3, so the CPU must continue running while something
external generates the CPER records. The records being visible is the last point
the faulty-access could have been made, with the window of time depending on how
fast this external-thing receives and processes the error.


> So it looks
> like the firmware itself is causing the error. There's still a chance that
> the stub is doing something wrong with the memory map passed to the
> firmware, so I'll try to eliminate that as well.

adding delay loops will help prove the EFIStub is innocent.

Are there any optional drivers being loaded by UEFI? (can you remove any USB
mass storage drives for instance).

Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)

https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
description of the GIC, comments 15 and 16 show a UEFI patch to something other
than the upstream platforms tree[0], and new firmware being tested.
(although this may be wishful thinking)

It looks like quirking this based on the DMI platform name and UEFI version will
be what we need. We could discard anything in the error status block areas at
ghes_probe() time based on this quirk, but we may have missed other problems
during boot, giving a false sense of security.


Thanks,

James


[0] Might be wrong, but this is where I look:
https://github.com/tianocore/edk2-platforms.git

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote:
> Hi Mark,
> 
> Thanks for shed-ing some light on what is going on here!
> 
> On 25/06/18 16:34, Mark Salter wrote:
> > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
> > > I'm going to hack something to get to the ghes info earlier in boot and
> > > check the things you mention above wrt Error Status Block and GHES.0.
> > 
> > So I had to end up instrumenting the EFI stub to see where the error came
> > from. At the start of the stub, there is no GHES.2 error. The error first
> > shows up after the stub's call to ExitBootServices returns.
> 
> What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.

SCI

Here's the HEST entry:

[028h 0040   2]                Subtable Type : 0009 [Generic Hardware Error Source]
[02Ah 0042   2]                    Source Id : 0002
[02Ch 0044   2]            Related Source Id : FFFF
[02Eh 0046   1]                     Reserved : 00
[02Fh 0047   1]                      Enabled : 01
[030h 0048   4]       Records To Preallocate : 00000001
[034h 0052   4]      Max Sections Per Record : 00000001
[038h 0056   4]          Max Raw Data Length : 00000AEC

[03Ch 0060  12]         Error Status Address : [Generic Address Structure]
[03Ch 0060   1]                     Space ID : 00 [SystemMemory]
[03Dh 0061   1]                    Bit Width : 40
[03Eh 0062   1]                   Bit Offset : 00
[03Fh 0063   1]         Encoded Access Width : 04 [QWord Access:64]
[040h 0064   8]                      Address : 0000004FF7E9F0E0

There are 9 others all identical except for Source ID and address.

> These systems don't have EL3, so the CPU must continue running while something
> external generates the CPER records. The records being visible is the last point
> the faulty-access could have been made, with the window of time depending on how
> fast this external-thing receives and processes the error.

There's a System Control Processor (slimpro) on the SoC which can interact with
the CPU in various ways and which has access to memory and other hw.

> 
> 
> > So it looks
> > like the firmware itself is causing the error. There's still a chance that
> > the stub is doing something wrong with the memory map passed to the
> > firmware, so I'll try to eliminate that as well.
> 
> adding delay loops will help prove the EFIStub is innocent.

Didn't change anything.

> 
> Are there any optional drivers being loaded by UEFI? (can you remove any USB
> mass storage drives for instance).

The only storage is pci based. There is a USB port but doesn't look like
anything is attached to it. I don't have physical access to it. It is one on
many moonshot cartridges in a chassis several hundred miles away.

> 
> Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)

No.

> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
> description of the GIC, comments 15 and 16 show a UEFI patch to something other
> than the upstream platforms tree[0], and new firmware being tested.
> (although this may be wishful thinking)

HPe would respond to bug reports until m400 reached EOL. They have been pretty
clear that no more firmware updates will be done.

> 
> It looks like quirking this based on the DMI platform name and UEFI version will
> be what we need. We could discard anything in the error status block areas at
> ghes_probe() time based on this quirk, but we may have missed other problems
> during boot, giving a false sense of security.
> 
> 
> Thanks,
> 
> James
> 
> 
> [0] Might be wrong, but this is where I look:
> https://github.com/tianocore/edk2-platforms.git

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote:
> Hi Mark,
> 
> Thanks for shed-ing some light on what is going on here!
> 
> On 25/06/18 16:34, Mark Salter wrote:
> > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
> > > I'm going to hack something to get to the ghes info earlier in boot and
> > > check the things you mention above wrt Error Status Block and GHES.0.
> > 
> > So I had to end up instrumenting the EFI stub to see where the error came
> > from. At the start of the stub, there is no GHES.2 error. The error first
> > shows up after the stub's call to ExitBootServices returns.
> 
> What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.

SCI

Here's the HEST entry:

[028h 0040   2]                Subtable Type : 0009 [Generic Hardware Error Source]
[02Ah 0042   2]                    Source Id : 0002
[02Ch 0044   2]            Related Source Id : FFFF
[02Eh 0046   1]                     Reserved : 00
[02Fh 0047   1]                      Enabled : 01
[030h 0048   4]       Records To Preallocate : 00000001
[034h 0052   4]      Max Sections Per Record : 00000001
[038h 0056   4]          Max Raw Data Length : 00000AEC

[03Ch 0060  12]         Error Status Address : [Generic Address Structure]
[03Ch 0060   1]                     Space ID : 00 [SystemMemory]
[03Dh 0061   1]                    Bit Width : 40
[03Eh 0062   1]                   Bit Offset : 00
[03Fh 0063   1]         Encoded Access Width : 04 [QWord Access:64]
[040h 0064   8]                      Address : 0000004FF7E9F0E0

There are 9 others all identical except for Source ID and address.

> These systems don't have EL3, so the CPU must continue running while something
> external generates the CPER records. The records being visible is the last point
> the faulty-access could have been made, with the window of time depending on how
> fast this external-thing receives and processes the error.

There's a System Control Processor (slimpro) on the SoC which can interact with
the CPU in various ways and which has access to memory and other hw.

> 
> 
> > So it looks
> > like the firmware itself is causing the error. There's still a chance that
> > the stub is doing something wrong with the memory map passed to the
> > firmware, so I'll try to eliminate that as well.
> 
> adding delay loops will help prove the EFIStub is innocent.

Didn't change anything.

> 
> Are there any optional drivers being loaded by UEFI? (can you remove any USB
> mass storage drives for instance).

The only storage is pci based. There is a USB port but doesn't look like
anything is attached to it. I don't have physical access to it. It is one on
many moonshot cartridges in a chassis several hundred miles away.

> 
> Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)

No.

> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
> description of the GIC, comments 15 and 16 show a UEFI patch to something other
> than the upstream platforms tree[0], and new firmware being tested.
> (although this may be wishful thinking)

HPe would respond to bug reports until m400 reached EOL. They have been pretty
clear that no more firmware updates will be done.

> 
> It looks like quirking this based on the DMI platform name and UEFI version will
> be what we need. We could discard anything in the error status block areas at
> ghes_probe() time based on this quirk, but we may have missed other problems
> during boot, giving a false sense of security.
> 
> 
> Thanks,
> 
> James
> 
> 
> [0] Might be wrong, but this is where I look:
> https://github.com/tianocore/edk2-platforms.git

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Ard Biesheuvel]

On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote:
> On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote:
>> Hi Mark,
>>
>> Thanks for shed-ing some light on what is going on here!
>>
>> On 25/06/18 16:34, Mark Salter wrote:
>> > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
>> > > I'm going to hack something to get to the ghes info earlier in boot and
>> > > check the things you mention above wrt Error Status Block and GHES.0.
>> >
>> > So I had to end up instrumenting the EFI stub to see where the error came
>> > from. At the start of the stub, there is no GHES.2 error. The error first
>> > shows up after the stub's call to ExitBootServices returns.
>>
>> What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.
>
> SCI
>
> Here's the HEST entry:
>
> [028h 0040   2]                Subtable Type : 0009 [Generic Hardware Error Source]
> [02Ah 0042   2]                    Source Id : 0002
> [02Ch 0044   2]            Related Source Id : FFFF
> [02Eh 0046   1]                     Reserved : 00
> [02Fh 0047   1]                      Enabled : 01
> [030h 0048   4]       Records To Preallocate : 00000001
> [034h 0052   4]      Max Sections Per Record : 00000001
> [038h 0056   4]          Max Raw Data Length : 00000AEC
>
> [03Ch 0060  12]         Error Status Address : [Generic Address Structure]
> [03Ch 0060   1]                     Space ID : 00 [SystemMemory]
> [03Dh 0061   1]                    Bit Width : 40
> [03Eh 0062   1]                   Bit Offset : 00
> [03Fh 0063   1]         Encoded Access Width : 04 [QWord Access:64]
> [040h 0064   8]                      Address : 0000004FF7E9F0E0
>

This is a reserved region in the memory map. Does that apply to the
other occurrences as well?

> There are 9 others all identical except for Source ID and address.
>
>> These systems don't have EL3, so the CPU must continue running while something
>> external generates the CPER records. The records being visible is the last point
>> the faulty-access could have been made, with the window of time depending on how
>> fast this external-thing receives and processes the error.
>
> There's a System Control Processor (slimpro) on the SoC which can interact with
> the CPU in various ways and which has access to memory and other hw.
>
>>
>>
>> > So it looks
>> > like the firmware itself is causing the error. There's still a chance that
>> > the stub is doing something wrong with the memory map passed to the
>> > firmware, so I'll try to eliminate that as well.
>>
>> adding delay loops will help prove the EFIStub is innocent.
>
> Didn't change anything.
>
>>
>> Are there any optional drivers being loaded by UEFI? (can you remove any USB
>> mass storage drives for instance).
>
> The only storage is pci based. There is a USB port but doesn't look like
> anything is attached to it. I don't have physical access to it. It is one on
> many moonshot cartridges in a chassis several hundred miles away.
>
>>
>> Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)
>
> No.
>
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
>> description of the GIC, comments 15 and 16 show a UEFI patch to something other
>> than the upstream platforms tree[0], and new firmware being tested.
>> (although this may be wishful thinking)
>
> HPe would respond to bug reports until m400 reached EOL. They have been pretty
> clear that no more firmware updates will be done.
>
>>
>> It looks like quirking this based on the DMI platform name and UEFI version will
>> be what we need. We could discard anything in the error status block areas at
>> ghes_probe() time based on this quirk, but we may have missed other problems
>> during boot, giving a false sense of security.
>>
>>
>> Thanks,
>>
>> James
>>
>>
>> [0] Might be wrong, but this is where I look:
>> https://github.com/tianocore/edk2-platforms.git
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Ard Biesheuvel]

On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote:
> On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote:
>> Hi Mark,
>>
>> Thanks for shed-ing some light on what is going on here!
>>
>> On 25/06/18 16:34, Mark Salter wrote:
>> > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
>> > > I'm going to hack something to get to the ghes info earlier in boot and
>> > > check the things you mention above wrt Error Status Block and GHES.0.
>> >
>> > So I had to end up instrumenting the EFI stub to see where the error came
>> > from. At the start of the stub, there is no GHES.2 error. The error first
>> > shows up after the stub's call to ExitBootServices returns.
>>
>> What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.
>
> SCI
>
> Here's the HEST entry:
>
> [028h 0040   2]                Subtable Type : 0009 [Generic Hardware Error Source]
> [02Ah 0042   2]                    Source Id : 0002
> [02Ch 0044   2]            Related Source Id : FFFF
> [02Eh 0046   1]                     Reserved : 00
> [02Fh 0047   1]                      Enabled : 01
> [030h 0048   4]       Records To Preallocate : 00000001
> [034h 0052   4]      Max Sections Per Record : 00000001
> [038h 0056   4]          Max Raw Data Length : 00000AEC
>
> [03Ch 0060  12]         Error Status Address : [Generic Address Structure]
> [03Ch 0060   1]                     Space ID : 00 [SystemMemory]
> [03Dh 0061   1]                    Bit Width : 40
> [03Eh 0062   1]                   Bit Offset : 00
> [03Fh 0063   1]         Encoded Access Width : 04 [QWord Access:64]
> [040h 0064   8]                      Address : 0000004FF7E9F0E0
>

This is a reserved region in the memory map. Does that apply to the
other occurrences as well?

> There are 9 others all identical except for Source ID and address.
>
>> These systems don't have EL3, so the CPU must continue running while something
>> external generates the CPER records. The records being visible is the last point
>> the faulty-access could have been made, with the window of time depending on how
>> fast this external-thing receives and processes the error.
>
> There's a System Control Processor (slimpro) on the SoC which can interact with
> the CPU in various ways and which has access to memory and other hw.
>
>>
>>
>> > So it looks
>> > like the firmware itself is causing the error. There's still a chance that
>> > the stub is doing something wrong with the memory map passed to the
>> > firmware, so I'll try to eliminate that as well.
>>
>> adding delay loops will help prove the EFIStub is innocent.
>
> Didn't change anything.
>
>>
>> Are there any optional drivers being loaded by UEFI? (can you remove any USB
>> mass storage drives for instance).
>
> The only storage is pci based. There is a USB port but doesn't look like
> anything is attached to it. I don't have physical access to it. It is one on
> many moonshot cartridges in a chassis several hundred miles away.
>
>>
>> Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)
>
> No.
>
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
>> description of the GIC, comments 15 and 16 show a UEFI patch to something other
>> than the upstream platforms tree[0], and new firmware being tested.
>> (although this may be wishful thinking)
>
> HPe would respond to bug reports until m400 reached EOL. They have been pretty
> clear that no more firmware updates will be done.
>
>>
>> It looks like quirking this based on the DMI platform name and UEFI version will
>> be what we need. We could discard anything in the error status block areas at
>> ghes_probe() time based on this quirk, but we may have missed other problems
>> during boot, giving a false sense of security.
>>
>>
>> Thanks,
>>
>> James
>>
>>
>> [0] Might be wrong, but this is where I look:
>> https://github.com/tianocore/edk2-platforms.git
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Wed, 2018-06-27 at 10:48 +0200, Ard Biesheuvel wrote:
> On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote:
> > On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote:
> > > Hi Mark,
> > > 
> > > Thanks for shed-ing some light on what is going on here!
> > > 
> > > On 25/06/18 16:34, Mark Salter wrote:
> > > > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
> > > > > I'm going to hack something to get to the ghes info earlier in boot and
> > > > > check the things you mention above wrt Error Status Block and GHES.0.
> > > > 
> > > > So I had to end up instrumenting the EFI stub to see where the error came
> > > > from. At the start of the stub, there is no GHES.2 error. The error first
> > > > shows up after the stub's call to ExitBootServices returns.
> > > 
> > > What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.
> > 
> > SCI
> > 
> > Here's the HEST entry:
> > 
> > [028h 0040   2]                Subtable Type : 0009 [Generic Hardware Error Source]
> > [02Ah 0042   2]                    Source Id : 0002
> > [02Ch 0044   2]            Related Source Id : FFFF
> > [02Eh 0046   1]                     Reserved : 00
> > [02Fh 0047   1]                      Enabled : 01
> > [030h 0048   4]       Records To Preallocate : 00000001
> > [034h 0052   4]      Max Sections Per Record : 00000001
> > [038h 0056   4]          Max Raw Data Length : 00000AEC
> > 
> > [03Ch 0060  12]         Error Status Address : [Generic Address Structure]
> > [03Ch 0060   1]                     Space ID : 00 [SystemMemory]
> > [03Dh 0061   1]                    Bit Width : 40
> > [03Eh 0062   1]                   Bit Offset : 00
> > [03Fh 0063   1]         Encoded Access Width : 04 [QWord Access:64]
> > [040h 0064   8]                      Address : 0000004FF7E9F0E0
> > 
> 
> This is a reserved region in the memory map. Does that apply to the
> other occurrences as well?

Yes, they are all in the same reserved region.

> 
> > There are 9 others all identical except for Source ID and address.
> > 
> > > These systems don't have EL3, so the CPU must continue running while something
> > > external generates the CPER records. The records being visible is the last point
> > > the faulty-access could have been made, with the window of time depending on how
> > > fast this external-thing receives and processes the error.
> > 
> > There's a System Control Processor (slimpro) on the SoC which can interact with
> > the CPU in various ways and which has access to memory and other hw.
> > 
> > > 
> > > 
> > > > So it looks
> > > > like the firmware itself is causing the error. There's still a chance that
> > > > the stub is doing something wrong with the memory map passed to the
> > > > firmware, so I'll try to eliminate that as well.
> > > 
> > > adding delay loops will help prove the EFIStub is innocent.
> > 
> > Didn't change anything.
> > 
> > > 
> > > Are there any optional drivers being loaded by UEFI? (can you remove any USB
> > > mass storage drives for instance).
> > 
> > The only storage is pci based. There is a USB port but doesn't look like
> > anything is attached to it. I don't have physical access to it. It is one on
> > many moonshot cartridges in a chassis several hundred miles away.
> > 
> > > 
> > > Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)
> > 
> > No.
> > 
> > > 
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
> > > description of the GIC, comments 15 and 16 show a UEFI patch to something other
> > > than the upstream platforms tree[0], and new firmware being tested.
> > > (although this may be wishful thinking)
> > 
> > HPe would respond to bug reports until m400 reached EOL. They have been pretty
> > clear that no more firmware updates will be done.
> > 
> > > 
> > > It looks like quirking this based on the DMI platform name and UEFI version will
> > > be what we need. We could discard anything in the error status block areas at
> > > ghes_probe() time based on this quirk, but we may have missed other problems
> > > during boot, giving a false sense of security.
> > > 
> > > 
> > > Thanks,
> > > 
> > > James
> > > 
> > > 
> > > [0] Might be wrong, but this is where I look:
> > > https://github.com/tianocore/edk2-platforms.git
> > 
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Wed, 2018-06-27 at 10:48 +0200, Ard Biesheuvel wrote:
> On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote:
> > On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote:
> > > Hi Mark,
> > > 
> > > Thanks for shed-ing some light on what is going on here!
> > > 
> > > On 25/06/18 16:34, Mark Salter wrote:
> > > > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
> > > > > I'm going to hack something to get to the ghes info earlier in boot and
> > > > > check the things you mention above wrt Error Status Block and GHES.0.
> > > > 
> > > > So I had to end up instrumenting the EFI stub to see where the error came
> > > > from. At the start of the stub, there is no GHES.2 error. The error first
> > > > shows up after the stub's call to ExitBootServices returns.
> > > 
> > > What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.
> > 
> > SCI
> > 
> > Here's the HEST entry:
> > 
> > [028h 0040   2]                Subtable Type : 0009 [Generic Hardware Error Source]
> > [02Ah 0042   2]                    Source Id : 0002
> > [02Ch 0044   2]            Related Source Id : FFFF
> > [02Eh 0046   1]                     Reserved : 00
> > [02Fh 0047   1]                      Enabled : 01
> > [030h 0048   4]       Records To Preallocate : 00000001
> > [034h 0052   4]      Max Sections Per Record : 00000001
> > [038h 0056   4]          Max Raw Data Length : 00000AEC
> > 
> > [03Ch 0060  12]         Error Status Address : [Generic Address Structure]
> > [03Ch 0060   1]                     Space ID : 00 [SystemMemory]
> > [03Dh 0061   1]                    Bit Width : 40
> > [03Eh 0062   1]                   Bit Offset : 00
> > [03Fh 0063   1]         Encoded Access Width : 04 [QWord Access:64]
> > [040h 0064   8]                      Address : 0000004FF7E9F0E0
> > 
> 
> This is a reserved region in the memory map. Does that apply to the
> other occurrences as well?

Yes, they are all in the same reserved region.

> 
> > There are 9 others all identical except for Source ID and address.
> > 
> > > These systems don't have EL3, so the CPU must continue running while something
> > > external generates the CPER records. The records being visible is the last point
> > > the faulty-access could have been made, with the window of time depending on how
> > > fast this external-thing receives and processes the error.
> > 
> > There's a System Control Processor (slimpro) on the SoC which can interact with
> > the CPU in various ways and which has access to memory and other hw.
> > 
> > > 
> > > 
> > > > So it looks
> > > > like the firmware itself is causing the error. There's still a chance that
> > > > the stub is doing something wrong with the memory map passed to the
> > > > firmware, so I'll try to eliminate that as well.
> > > 
> > > adding delay loops will help prove the EFIStub is innocent.
> > 
> > Didn't change anything.
> > 
> > > 
> > > Are there any optional drivers being loaded by UEFI? (can you remove any USB
> > > mass storage drives for instance).
> > 
> > The only storage is pci based. There is a USB port but doesn't look like
> > anything is attached to it. I don't have physical access to it. It is one on
> > many moonshot cartridges in a chassis several hundred miles away.
> > 
> > > 
> > > Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)
> > 
> > No.
> > 
> > > 
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
> > > description of the GIC, comments 15 and 16 show a UEFI patch to something other
> > > than the upstream platforms tree[0], and new firmware being tested.
> > > (although this may be wishful thinking)
> > 
> > HPe would respond to bug reports until m400 reached EOL. They have been pretty
> > clear that no more firmware updates will be done.
> > 
> > > 
> > > It looks like quirking this based on the DMI platform name and UEFI version will
> > > be what we need. We could discard anything in the error status block areas at
> > > ghes_probe() time based on this quirk, but we may have missed other problems
> > > during boot, giving a false sense of security.
> > > 
> > > 
> > > Thanks,
> > > 
> > > James
> > > 
> > > 
> > > [0] Might be wrong, but this is where I look:
> > > https://github.com/tianocore/edk2-platforms.git
> > 
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel at lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Ian Campbell]

On Wed, 2018-06-27 at 08:25 -0400, Mark Salter wrote:
> On Wed, 2018-06-27 at 10:48 +0200, Ard Biesheuvel wrote:
> > On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote:
> > > [040h 0064   8]                      Address : 0000004FF7E9F0E0
> > > 
> > 
> > This is a reserved region in the memory map. Does that apply to the
> > other occurrences as well?
> 
> Yes, they are all in the same reserved region.

I wonder if it corresponds to some actual peripheral on a different X-
gene based platform (e.g. perhaps a reference design)?

Not sure what one would do with that information, just curious.

Ian.

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Ian Campbell]

On Wed, 2018-06-27 at 08:25 -0400, Mark Salter wrote:
> On Wed, 2018-06-27 at 10:48 +0200, Ard Biesheuvel wrote:
> > On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote:
> > > [040h 0064   8]                      Address : 0000004FF7E9F0E0
> > > 
> > 
> > This is a reserved region in the memory map. Does that apply to the
> > other occurrences as well?
> 
> Yes, they are all in the same reserved region.

I wonder if it corresponds to some actual peripheral on a different X-
gene based platform (e.g. perhaps a reference design)?

Not sure what one would do with that information, just curious.

Ian.

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Tue, 2018-07-03 at 10:30 +0100, Ian Campbell wrote:
> On Wed, 2018-06-27 at 08:25 -0400, Mark Salter wrote:
> > On Wed, 2018-06-27 at 10:48 +0200, Ard Biesheuvel wrote:
> > > On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote:
> > > > [040h 0064   8]                      Address : 0000004FF7E9F0E0
> > > > 
> > > 
> > > This is a reserved region in the memory map. Does that apply to the
> > > other occurrences as well?
> > 
> > Yes, they are all in the same reserved region.
> 
> I wonder if it corresponds to some actual peripheral on a different X-
> gene based platform (e.g. perhaps a reference design)?
> 
> Not sure what one would do with that information, just curious.
> 
> Ian.

It's a reserved region of system RAM.

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Tue, 2018-07-03 at 10:30 +0100, Ian Campbell wrote:
> On Wed, 2018-06-27 at 08:25 -0400, Mark Salter wrote:
> > On Wed, 2018-06-27 at 10:48 +0200, Ard Biesheuvel wrote:
> > > On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote:
> > > > [040h 0064   8]                      Address : 0000004FF7E9F0E0
> > > > 
> > > 
> > > This is a reserved region in the memory map. Does that apply to the
> > > other occurrences as well?
> > 
> > Yes, they are all in the same reserved region.
> 
> I wonder if it corresponds to some actual peripheral on a different X-
> gene based platform (e.g. perhaps a reference design)?
> 
> Not sure what one would do with that information, just curious.
> 
> Ian.

It's a reserved region of system RAM.

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[James Morse]

Hi Mark,

On 26/06/18 21:20, Mark Salter wrote:
> On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote:
>> On 25/06/18 16:34, Mark Salter wrote:
>>> On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
>>>> I'm going to hack something to get to the ghes info earlier in boot and
>>>> check the things you mention above wrt Error Status Block and GHES.0.
>>>
>>> So I had to end up instrumenting the EFI stub to see where the error came
>>> from. At the start of the stub, there is no GHES.2 error. The error first
>>> shows up after the stub's call to ExitBootServices returns.
>>
>> What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.

>> These systems don't have EL3, so the CPU must continue running while something
>> external generates the CPER records. The records being visible is the last point
>> the faulty-access could have been made, with the window of time depending on how
>> fast this external-thing receives and processes the error.
> 
> There's a System Control Processor (slimpro) on the SoC which can interact with
> the CPU in various ways and which has access to memory and other hw.

Thanks, saves me guessing!


>>> So it looks
>>> like the firmware itself is causing the error. There's still a chance that
>>> the stub is doing something wrong with the memory map passed to the
>>> firmware, so I'll try to eliminate that as well.
>>
>> adding delay loops will help prove the EFIStub is innocent.
> 
> Didn't change anything.

Okay, so just to clarify, a delay before ExitBootServices doesn't cause the
error to show up before ExitBootServices, so the error hasn't occurred prior to
this point.
And a delay after ExitBootServices allows us to see the error before we exit
into head.S. (this rules out a bug in head.S)
The delays should be long enough to tell us this slimpro isn't generating the
error records N seconds after reset.

Given this I agree we should disable_hest based on the DMI platform name and the
UEFI version number. (it may be earlier firmware didn't have this bug).


I don't have anything to test this on, so I've picked the DMI strings out the
demsg output on that bugzilla entry. Any chance you could give it a test?


>> Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)

>> https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
>> description of the GIC, comments 15 and 16 show a UEFI patch to something other
>> than the upstream platforms tree[0], and new firmware being tested.
>> (although this may be wishful thinking)
> 
> HPe would respond to bug reports until m400 reached EOL. They have been pretty
> clear that no more firmware updates will be done.

Thanks, it was a bit murky from that ticket...


Thanks for doing this!

James

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[James Morse]

Hi Mark,

On 26/06/18 21:20, Mark Salter wrote:
> On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote:
>> On 25/06/18 16:34, Mark Salter wrote:
>>> On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
>>>> I'm going to hack something to get to the ghes info earlier in boot and
>>>> check the things you mention above wrt Error Status Block and GHES.0.
>>>
>>> So I had to end up instrumenting the EFI stub to see where the error came
>>> from. At the start of the stub, there is no GHES.2 error. The error first
>>> shows up after the stub's call to ExitBootServices returns.
>>
>> What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.

>> These systems don't have EL3, so the CPU must continue running while something
>> external generates the CPER records. The records being visible is the last point
>> the faulty-access could have been made, with the window of time depending on how
>> fast this external-thing receives and processes the error.
> 
> There's a System Control Processor (slimpro) on the SoC which can interact with
> the CPU in various ways and which has access to memory and other hw.

Thanks, saves me guessing!


>>> So it looks
>>> like the firmware itself is causing the error. There's still a chance that
>>> the stub is doing something wrong with the memory map passed to the
>>> firmware, so I'll try to eliminate that as well.
>>
>> adding delay loops will help prove the EFIStub is innocent.
> 
> Didn't change anything.

Okay, so just to clarify, a delay before ExitBootServices doesn't cause the
error to show up before ExitBootServices, so the error hasn't occurred prior to
this point.
And a delay after ExitBootServices allows us to see the error before we exit
into head.S. (this rules out a bug in head.S)
The delays should be long enough to tell us this slimpro isn't generating the
error records N seconds after reset.

Given this I agree we should disable_hest based on the DMI platform name and the
UEFI version number. (it may be earlier firmware didn't have this bug).


I don't have anything to test this on, so I've picked the DMI strings out the
demsg output on that bugzilla entry. Any chance you could give it a test?


>> Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)

>> https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
>> description of the GIC, comments 15 and 16 show a UEFI patch to something other
>> than the upstream platforms tree[0], and new firmware being tested.
>> (although this may be wishful thinking)
> 
> HPe would respond to bug reports until m400 reached EOL. They have been pretty
> clear that no more firmware updates will be done.

Thanks, it was a bit murky from that ticket...


Thanks for doing this!

James

Re: [PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Thu, 2018-06-28 at 11:06 +0100, James Morse wrote:
> Hi Mark,
> 
> On 26/06/18 21:20, Mark Salter wrote:
> > On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote:
> > > On 25/06/18 16:34, Mark Salter wrote:
> > > > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
> > > > > I'm going to hack something to get to the ghes info earlier in boot and
> > > > > check the things you mention above wrt Error Status Block and GHES.0.
> > > > 
> > > > So I had to end up instrumenting the EFI stub to see where the error came
> > > > from. At the start of the stub, there is no GHES.2 error. The error first
> > > > shows up after the stub's call to ExitBootServices returns.
> > > 
> > > What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.
> > > These systems don't have EL3, so the CPU must continue running while something
> > > external generates the CPER records. The records being visible is the last point
> > > the faulty-access could have been made, with the window of time depending on how
> > > fast this external-thing receives and processes the error.
> > 
> > There's a System Control Processor (slimpro) on the SoC which can interact with
> > the CPU in various ways and which has access to memory and other hw.
> 
> Thanks, saves me guessing!
> 
> 
> > > > So it looks
> > > > like the firmware itself is causing the error. There's still a chance that
> > > > the stub is doing something wrong with the memory map passed to the
> > > > firmware, so I'll try to eliminate that as well.
> > > 
> > > adding delay loops will help prove the EFIStub is innocent.
> > 
> > Didn't change anything.
> 

Just closing the loop on this...

> Okay, so just to clarify, a delay before ExitBootServices doesn't cause the
> error to show up before ExitBootServices, so the error hasn't occurred prior to
> this point.

Correct. I have never seen the error before ExitBootServices.

> And a delay after ExitBootServices allows us to see the error before we exit
> into head.S. (this rules out a bug in head.S)
> The delays should be long enough to tell us this slimpro isn't generating the
> error records N seconds after reset.

No delay needed after ExitBootServices. The error would be there right after the
call returns back to stub.


> 
> Given this I agree we should disable_hest based on the DMI platform name and the
> UEFI version number. (it may be earlier firmware didn't have this bug).
> 
> 
> I don't have anything to test this on, so I've picked the DMI strings out the
> demsg output on that bugzilla entry. Any chance you could give it a test?
> 
> 
> > > Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
> > > description of the GIC, comments 15 and 16 show a UEFI patch to something other
> > > than the upstream platforms tree[0], and new firmware being tested.
> > > (although this may be wishful thinking)
> > 
> > HPe would respond to bug reports until m400 reached EOL. They have been pretty
> > clear that no more firmware updates will be done.
> 
> Thanks, it was a bit murky from that ticket...
> 
> 
> Thanks for doing this!
> 
> James

[PATCH] arm64/acpi: Add fixup for HPE m400 quirks

[Mark Salter]

On Thu, 2018-06-28 at 11:06 +0100, James Morse wrote:
> Hi Mark,
> 
> On 26/06/18 21:20, Mark Salter wrote:
> > On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote:
> > > On 25/06/18 16:34, Mark Salter wrote:
> > > > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote:
> > > > > I'm going to hack something to get to the ghes info earlier in boot and
> > > > > check the things you mention above wrt Error Status Block and GHES.0.
> > > > 
> > > > So I had to end up instrumenting the EFI stub to see where the error came
> > > > from. At the start of the stub, there is no GHES.2 error. The error first
> > > > shows up after the stub's call to ExitBootServices returns.
> > > 
> > > What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ.
> > > These systems don't have EL3, so the CPU must continue running while something
> > > external generates the CPER records. The records being visible is the last point
> > > the faulty-access could have been made, with the window of time depending on how
> > > fast this external-thing receives and processes the error.
> > 
> > There's a System Control Processor (slimpro) on the SoC which can interact with
> > the CPU in various ways and which has access to memory and other hw.
> 
> Thanks, saves me guessing!
> 
> 
> > > > So it looks
> > > > like the firmware itself is causing the error. There's still a chance that
> > > > the stub is doing something wrong with the memory map passed to the
> > > > firmware, so I'll try to eliminate that as well.
> > > 
> > > adding delay loops will help prove the EFIStub is innocent.
> > 
> > Didn't change anything.
> 

Just closing the loop on this...

> Okay, so just to clarify, a delay before ExitBootServices doesn't cause the
> error to show up before ExitBootServices, so the error hasn't occurred prior to
> this point.

Correct. I have never seen the error before ExitBootServices.

> And a delay after ExitBootServices allows us to see the error before we exit
> into head.S. (this rules out a bug in head.S)
> The delays should be long enough to tell us this slimpro isn't generating the
> error records N seconds after reset.

No delay needed after ExitBootServices. The error would be there right after the
call returns back to stub.


> 
> Given this I agree we should disable_hest based on the DMI platform name and the
> UEFI version number. (it may be earlier firmware didn't have this bug).
> 
> 
> I don't have anything to test this on, so I've picked the DMI strings out the
> demsg output on that bugzilla entry. Any chance you could give it a test?
> 
> 
> > > Are redhat able to rebuild UEFI on these systems? (Can it be fixed?)
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400
> > > description of the GIC, comments 15 and 16 show a UEFI patch to something other
> > > than the upstream platforms tree[0], and new firmware being tested.
> > > (although this may be wishful thinking)
> > 
> > HPe would respond to bug reports until m400 reached EOL. They have been pretty
> > clear that no more firmware updates will be done.
> 
> Thanks, it was a bit murky from that ticket...
> 
> 
> Thanks for doing this!
> 
> James