[PATCH] x86: Add an option to disable decoding of MCE

[PATCH] x86: Add an option to disable decoding of MCE

[Mike Waychison]

This patch applies to v2.6.37.
---

On our systems, we do not want to have any "decoders" called on machine
check events.  These decoders can easily spam our logs and cause space
problems on machines that have a lot of correctable error events.  We
_do_ however want to get the messages delivered via /dev/mcelog for
userland processing.

Introduce an interface "dont_decode" that allows us to skip the
decoders.  We always call the decoders by default.

Google-Bug-Id: 3289142
Signed-off-by: Mike Waychison <mikew@google.com>
---
 arch/x86/kernel/cpu/mcheck/mce.c |   24 ++++++++++++++++++------
 1 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index 7a35b72..3c30057 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -82,6 +82,7 @@ static int			mce_bootlog		__read_mostly = -1;
 static int			monarch_timeout		__read_mostly = -1;
 static int			mce_panic_timeout	__read_mostly;
 static int			mce_dont_log_ce		__read_mostly;
+static int			mce_dont_decode		__read_mostly;
 int				mce_cmci_disabled	__read_mostly;
 int				mce_ignore_ce		__read_mostly;
 int				mce_ser			__read_mostly;
@@ -209,6 +210,17 @@ void mce_log(struct mce *mce)
 	set_bit(0, &mce_need_notify);
 }
 
+static void call_decoders(struct mce *m)
+{
+	if (mce_dont_decode)
+		return;
+	/*
+	 * Print out human-readable details about the MCE error,
+	 * (if the CPU has an implementation for that)
+	 */
+	atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
+}
+
 static void print_mce(struct mce *m)
 {
 	pr_emerg(HW_ERR "CPU %d: Machine Check Exception: %Lx Bank %d: %016Lx\n",
@@ -234,11 +246,7 @@ static void print_mce(struct mce *m)
 	pr_emerg(HW_ERR "PROCESSOR %u:%x TIME %llu SOCKET %u APIC %x\n",
 		m->cpuvendor, m->cpuid, m->time, m->socketid, m->apicid);
 
-	/*
-	 * Print out human-readable details about the MCE error,
-	 * (if the CPU has an implementation for that)
-	 */
-	atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
+	call_decoders(m);
 }
 
 #define PANIC_TIMEOUT 5 /* 5 seconds */
@@ -588,7 +596,7 @@ void machine_check_poll(enum mcp_flags flags, mce_banks_t *b)
 		 */
 		if (!(flags & MCP_DONTLOG) && !mce_dont_log_ce) {
 			mce_log(&m);
-			atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, &m);
+			call_decoders(&m);
 			add_taint(TAINT_MACHINE_CHECK);
 		}
 
@@ -1700,6 +1708,8 @@ static int __init mcheck_enable(char *str)
 		mce_cmci_disabled = 1;
 	else if (!strcmp(str, "dont_log_ce"))
 		mce_dont_log_ce = 1;
+	else if (!strcmp(str, "dont_decode"))
+		mce_dont_decode = 1;
 	else if (!strcmp(str, "ignore_ce"))
 		mce_ignore_ce = 1;
 	else if (!strcmp(str, "bootlog") || !strcmp(str, "nobootlog"))
@@ -1926,6 +1936,7 @@ static SYSDEV_ATTR(trigger, 0644, show_trigger, set_trigger);
 static SYSDEV_INT_ATTR(tolerant, 0644, tolerant);
 static SYSDEV_INT_ATTR(monarch_timeout, 0644, monarch_timeout);
 static SYSDEV_INT_ATTR(dont_log_ce, 0644, mce_dont_log_ce);
+static SYSDEV_INT_ATTR(dont_decode, 0644, mce_dont_decode);
 
 static struct sysdev_ext_attribute attr_check_interval = {
 	_SYSDEV_ATTR(check_interval, 0644, sysdev_show_int,
@@ -1949,6 +1960,7 @@ static struct sysdev_attribute *mce_attrs[] = {
 	&attr_trigger,
 	&attr_monarch_timeout.attr,
 	&attr_dont_log_ce.attr,
+	&attr_dont_decode.attr,
 	&attr_ignore_ce.attr,
 	&attr_cmci_disabled.attr,
 	NULL
-- 
1.7.3.1

Re: [PATCH] x86: Add an option to disable decoding of MCE

[Randy Dunlap]

On Mon, January 10, 2011 2:46 pm, Mike Waychison wrote:
> This patch applies to v2.6.37.
> ---
>
>
> On our systems, we do not want to have any "decoders" called on machine
> check events.  These decoders can easily spam our logs and cause space
> problems on machines that have a lot of correctable error events.  We _do_
> however want to get the messages delivered via /dev/mcelog for userland
> processing.
>
> Introduce an interface "dont_decode" that allows us to skip the
> decoders.  We always call the decoders by default.
>
> Google-Bug-Id: 3289142
> Signed-off-by: Mike Waychison <mikew@google.com>
> ---
> arch/x86/kernel/cpu/mcheck/mce.c |   24 ++++++++++++++++++------ 1 files
> changed, 18 insertions(+), 6 deletions(-)


Hi,

Please update Documentation/x86/x86_64/boot-options.txt
with this new option.

-- 
~Randy

[PATCH] x86: Add an option to disable decoding of MCE

[Mike Waychison]

This patch applies to v2.6.37.

Updated with documentation of the new option.
---

On our systems, we do not want to have any "decoders" called on machine
check events.  These decoders can easily spam our logs and cause space
problems on machines that have a lot of correctable error events.  We
_do_ however want to get the messages delivered via /dev/mcelog for
userland processing.

Introduce an interface "dont_decode" that allows us to skip the
decoders.  We always call the decoders by default.

Google-Bug-Id: 3289142
Signed-off-by: Mike Waychison <mikew@google.com>
---
 Documentation/x86/x86_64/boot-options.txt |    5 +++++
 Documentation/x86/x86_64/machinecheck     |    6 ++++++
 arch/x86/kernel/cpu/mcheck/mce.c          |   24 ++++++++++++++++++------
 3 files changed, 29 insertions(+), 6 deletions(-)

diff --git a/Documentation/x86/x86_64/boot-options.txt b/Documentation/x86/x86_64/boot-options.txt
index 7fbbaf8..dd7145a 100644
--- a/Documentation/x86/x86_64/boot-options.txt
+++ b/Documentation/x86/x86_64/boot-options.txt
@@ -22,6 +22,11 @@ Machine check
 		as corrected are silently cleared by OS.
 		This option will be useful if you have no interest in any
 		of corrected errors.
+   mce=dont_decode
+		Disable in-kernel decoding of errors.  Setting this boot
+		option will cause EDAC to be skipped (if enabled) and no
+		messages to be printed into the logs.  Events will still
+		be available via /dev/mcelog however.
    mce=ignore_ce
 		Disable features for corrected errors, e.g. polling timer
 		and CMCI.  All events reported as corrected are not cleared
diff --git a/Documentation/x86/x86_64/machinecheck b/Documentation/x86/x86_64/machinecheck
index b1fb302..7ef7003 100644
--- a/Documentation/x86/x86_64/machinecheck
+++ b/Documentation/x86/x86_64/machinecheck
@@ -65,6 +65,12 @@ tolerant
 	Note this only makes a difference if the CPU allows recovery
 	from a machine check exception. Current x86 CPUs generally do not.
 
+dont_decode
+	Disable in-kernel decoding of any errors.  Setting this boot
+	option will cause EDAC to be skipped (if enabled) and no
+	messages to be printed into the logs.  Events will still be
+	available via /dev/mcelog however.
+
 trigger
 	Program to run when a machine check event is detected.
 	This is an alternative to running mcelog regularly from cron
diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index 7a35b72..3c30057 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -82,6 +82,7 @@ static int			mce_bootlog		__read_mostly = -1;
 static int			monarch_timeout		__read_mostly = -1;
 static int			mce_panic_timeout	__read_mostly;
 static int			mce_dont_log_ce		__read_mostly;
+static int			mce_dont_decode		__read_mostly;
 int				mce_cmci_disabled	__read_mostly;
 int				mce_ignore_ce		__read_mostly;
 int				mce_ser			__read_mostly;
@@ -209,6 +210,17 @@ void mce_log(struct mce *mce)
 	set_bit(0, &mce_need_notify);
 }
 
+static void call_decoders(struct mce *m)
+{
+	if (mce_dont_decode)
+		return;
+	/*
+	 * Print out human-readable details about the MCE error,
+	 * (if the CPU has an implementation for that)
+	 */
+	atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
+}
+
 static void print_mce(struct mce *m)
 {
 	pr_emerg(HW_ERR "CPU %d: Machine Check Exception: %Lx Bank %d: %016Lx\n",
@@ -234,11 +246,7 @@ static void print_mce(struct mce *m)
 	pr_emerg(HW_ERR "PROCESSOR %u:%x TIME %llu SOCKET %u APIC %x\n",
 		m->cpuvendor, m->cpuid, m->time, m->socketid, m->apicid);
 
-	/*
-	 * Print out human-readable details about the MCE error,
-	 * (if the CPU has an implementation for that)
-	 */
-	atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
+	call_decoders(m);
 }
 
 #define PANIC_TIMEOUT 5 /* 5 seconds */
@@ -588,7 +596,7 @@ void machine_check_poll(enum mcp_flags flags, mce_banks_t *b)
 		 */
 		if (!(flags & MCP_DONTLOG) && !mce_dont_log_ce) {
 			mce_log(&m);
-			atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, &m);
+			call_decoders(&m);
 			add_taint(TAINT_MACHINE_CHECK);
 		}
 
@@ -1700,6 +1708,8 @@ static int __init mcheck_enable(char *str)
 		mce_cmci_disabled = 1;
 	else if (!strcmp(str, "dont_log_ce"))
 		mce_dont_log_ce = 1;
+	else if (!strcmp(str, "dont_decode"))
+		mce_dont_decode = 1;
 	else if (!strcmp(str, "ignore_ce"))
 		mce_ignore_ce = 1;
 	else if (!strcmp(str, "bootlog") || !strcmp(str, "nobootlog"))
@@ -1926,6 +1936,7 @@ static SYSDEV_ATTR(trigger, 0644, show_trigger, set_trigger);
 static SYSDEV_INT_ATTR(tolerant, 0644, tolerant);
 static SYSDEV_INT_ATTR(monarch_timeout, 0644, monarch_timeout);
 static SYSDEV_INT_ATTR(dont_log_ce, 0644, mce_dont_log_ce);
+static SYSDEV_INT_ATTR(dont_decode, 0644, mce_dont_decode);
 
 static struct sysdev_ext_attribute attr_check_interval = {
 	_SYSDEV_ATTR(check_interval, 0644, sysdev_show_int,
@@ -1949,6 +1960,7 @@ static struct sysdev_attribute *mce_attrs[] = {
 	&attr_trigger,
 	&attr_monarch_timeout.attr,
 	&attr_dont_log_ce.attr,
+	&attr_dont_decode.attr,
 	&attr_ignore_ce.attr,
 	&attr_cmci_disabled.attr,
 	NULL
-- 
1.7.3.1

Re: [PATCH] x86: Add an option to disable decoding of MCE

[Borislav Petkov]

On Mon, Jan 10, 2011 at 06:03:17PM -0500, Mike Waychison wrote:
> This patch applies to v2.6.37.
> 
> Updated with documentation of the new option.
> ---
> 
> On our systems, we do not want to have any "decoders" called on machine
> check events.  These decoders can easily spam our logs and cause space
> problems on machines that have a lot of correctable error events.  We
> _do_ however want to get the messages delivered via /dev/mcelog for
> userland processing.

Ok, question: how do you guys process DRAM ECCs? And more specifically,
with a large number of machines, how do you do the mapping from the DRAM
ECC error address reported by MCA to a DIMM that's failing in userspace
on a particular machine?

Also, I've worked on trimming down all that decoding output to 3-5
lines. Now it looks like this:

[  521.677316] [Hardware Error]: MC4_STATUS[Over|UE|MiscV|PCC|AddrV|UECC]: 0xfe00200000080a0f
[  521.686467] [Hardware Error]: Northbridge Error (node 0): DRAM ECC error detected on the NB.
[  521.686498] EDAC MC0: UE page 0x0, offset 0x0, grain 0, row 0, labels ":": amd64_edac
[  521.686501] EDAC MC0: UE - no information available: UE bit is set
[  521.686503] [Hardware Error]: cache level: L3/GEN, mem/io: GEN, mem-tx: GEN, part-proc: RES (no timeout)

and the two lines starting with "EDAC MC0" will get trimmed even
more with time. I'm assuming this is not a lot but if you get a lot
of correctable error events, then output like that accumulates over
time. How about an error thresholding scheme in software then which
accumulates the error events and reports only when some configurable
thresholds per DRAM device in error have been reached?

> Introduce an interface "dont_decode" that allows us to skip the
> decoders.  We always call the decoders by default.
> 
> Google-Bug-Id: 3289142
> Signed-off-by: Mike Waychison <mikew@google.com>
> ---
>  Documentation/x86/x86_64/boot-options.txt |    5 +++++
>  Documentation/x86/x86_64/machinecheck     |    6 ++++++
>  arch/x86/kernel/cpu/mcheck/mce.c          |   24 ++++++++++++++++++------
>  3 files changed, 29 insertions(+), 6 deletions(-)
> 
> diff --git a/Documentation/x86/x86_64/boot-options.txt b/Documentation/x86/x86_64/boot-options.txt
> index 7fbbaf8..dd7145a 100644
> --- a/Documentation/x86/x86_64/boot-options.txt
> +++ b/Documentation/x86/x86_64/boot-options.txt
> @@ -22,6 +22,11 @@ Machine check
>  		as corrected are silently cleared by OS.
>  		This option will be useful if you have no interest in any
>  		of corrected errors.
> +   mce=dont_decode
> +		Disable in-kernel decoding of errors.  Setting this boot
> +		option will cause EDAC to be skipped (if enabled) and no
> +		messages to be printed into the logs.  Events will still
> +		be available via /dev/mcelog however.
>     mce=ignore_ce
>  		Disable features for corrected errors, e.g. polling timer
>  		and CMCI.  All events reported as corrected are not cleared
> diff --git a/Documentation/x86/x86_64/machinecheck b/Documentation/x86/x86_64/machinecheck
> index b1fb302..7ef7003 100644
> --- a/Documentation/x86/x86_64/machinecheck
> +++ b/Documentation/x86/x86_64/machinecheck
> @@ -65,6 +65,12 @@ tolerant
>  	Note this only makes a difference if the CPU allows recovery
>  	from a machine check exception. Current x86 CPUs generally do not.
>  
> +dont_decode
> +	Disable in-kernel decoding of any errors.  Setting this boot
> +	option will cause EDAC to be skipped (if enabled) and no
> +	messages to be printed into the logs.  Events will still be
> +	available via /dev/mcelog however.
> +
>  trigger
>  	Program to run when a machine check event is detected.
>  	This is an alternative to running mcelog regularly from cron
> diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
> index 7a35b72..3c30057 100644
> --- a/arch/x86/kernel/cpu/mcheck/mce.c
> +++ b/arch/x86/kernel/cpu/mcheck/mce.c
> @@ -82,6 +82,7 @@ static int			mce_bootlog		__read_mostly = -1;
>  static int			monarch_timeout		__read_mostly = -1;
>  static int			mce_panic_timeout	__read_mostly;
>  static int			mce_dont_log_ce		__read_mostly;
> +static int			mce_dont_decode		__read_mostly;
>  int				mce_cmci_disabled	__read_mostly;
>  int				mce_ignore_ce		__read_mostly;
>  int				mce_ser			__read_mostly;
> @@ -209,6 +210,17 @@ void mce_log(struct mce *mce)
>  	set_bit(0, &mce_need_notify);
>  }
>  
> +static void call_decoders(struct mce *m)

Yeah, let's call this decode_mce().

> +{
> +	if (mce_dont_decode)
> +		return;
> +	/*
> +	 * Print out human-readable details about the MCE error,
> +	 * (if the CPU has an implementation for that)
> +	 */
> +	atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
> +}
> +
>  static void print_mce(struct mce *m)
>  {
>  	pr_emerg(HW_ERR "CPU %d: Machine Check Exception: %Lx Bank %d: %016Lx\n",
> @@ -234,11 +246,7 @@ static void print_mce(struct mce *m)
>  	pr_emerg(HW_ERR "PROCESSOR %u:%x TIME %llu SOCKET %u APIC %x\n",
>  		m->cpuvendor, m->cpuid, m->time, m->socketid, m->apicid);
>  
> -	/*
> -	 * Print out human-readable details about the MCE error,
> -	 * (if the CPU has an implementation for that)
> -	 */
> -	atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
> +	call_decoders(m);
>  }
>  
>  #define PANIC_TIMEOUT 5 /* 5 seconds */
> @@ -588,7 +596,7 @@ void machine_check_poll(enum mcp_flags flags, mce_banks_t *b)
>  		 */
>  		if (!(flags & MCP_DONTLOG) && !mce_dont_log_ce) {
>  			mce_log(&m);

Also, there's another hook in the function above that does
edac_mce_parse(mce) (which shouldnt've been there actually) which is
used by the Nehalem driver i7core_edac which does also decode DRAM ECCs.

@Mauro: how about dropping the whole <drivers/edac/edac_mce.c> and using
a simple notifier which is much smaller in code and does the same thing?

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: [PATCH] x86: Add an option to disable decoding of MCE

[Mike Waychison]

On Mon, Jan 10, 2011 at 10:55 PM, Borislav Petkov <bp@amd64.org> wrote:
> On Mon, Jan 10, 2011 at 06:03:17PM -0500, Mike Waychison wrote:
>> This patch applies to v2.6.37.
>>
>> Updated with documentation of the new option.
>> ---
>>
>> On our systems, we do not want to have any "decoders" called on machine
>> check events.  These decoders can easily spam our logs and cause space
>> problems on machines that have a lot of correctable error events.  We
>> _do_ however want to get the messages delivered via /dev/mcelog for
>> userland processing.
>
> Ok, question: how do you guys process DRAM ECCs? And more specifically,
> with a large number of machines, how do you do the mapping from the DRAM
> ECC error address reported by MCA to a DIMM that's failing in userspace
> on a particular machine?

We process machine checks in userland, using carnal knowledge of the
memory controller and the board specific addressing of the SPDs on the
various i2c busses to deswizzle and make sense of the addresses and
symptoms.  We then expose this digested data on the network, which is
dealt with at the cluster level.

>
> Also, I've worked on trimming down all that decoding output to 3-5
> lines. Now it looks like this:
>
> [  521.677316] [Hardware Error]: MC4_STATUS[Over|UE|MiscV|PCC|AddrV|UECC]: 0xfe00200000080a0f
> [  521.686467] [Hardware Error]: Northbridge Error (node 0): DRAM ECC error detected on the NB.
> [  521.686498] EDAC MC0: UE page 0x0, offset 0x0, grain 0, row 0, labels ":": amd64_edac
> [  521.686501] EDAC MC0: UE - no information available: UE bit is set
> [  521.686503] [Hardware Error]: cache level: L3/GEN, mem/io: GEN, mem-tx: GEN, part-proc: RES (no timeout)
>
> and the two lines starting with "EDAC MC0" will get trimmed even
> more with time. I'm assuming this is not a lot but if you get a lot
> of correctable error events, then output like that accumulates over
> time.

This decoded information is great, but it's only really consumable by
a human.  We'd _much_ rather have structured data that we know isn't
as volatile (in terms of regexes that break with development churn)
nor susceptible to corruption (as the logs are not transactional and
are shared by all CPUs concurrently).   For the most part, we rarely
rely on the anything automated consuming kernel printk logs, and the
bits that do are there only for historical reasons (read: we haven't
figured out how to replace them yet).  With almost every single kernel
version bump we've had, we've found numerous places where strings in
the logs have subtly changed, breaking our infrastructure :(  Finding
these is a painful process and often shows up late in the deployment
process.  This is disastrous to our deployment schedules as rebooting
our number of machines, well, it takes a while...

I'm sorry if this is tangential to your comment above, but I feel the
need to have folks recognize that printk() is _not_ a good foundation
on which to build automation.

As for using EDAC, I know we tried using it a few years ago, but have
since reverted to processing MCE ourselves.  I don't know all the
details as to why, however Duncan Laurie might be able to share more
(CCed).

> How about an error thresholding scheme in software then which
> accumulates the error events and reports only when some configurable
> thresholds per DRAM device in error have been reached?

This is pretty much the opposite of what we'd want.   We have no use
for anything printed in the logs, though emitting these messages
wouldn't be a problem except for those machines which very frequently
have MCA events to report.  On those machines, the chatter in the logs
can potentially wedge our root filesystem (which is tiny) and has the
downside that it forces logs to rotate quickly (which means there is
less useful data for a human to consume when analyzing what happened
on the host).  Our policy has been to neuter bits of the kernel that
don't know when to stfu :)


>>
>> +static void call_decoders(struct mce *m)
>
> Yeah, let's call this decode_mce().

OK.

>
>> +{
>> +     if (mce_dont_decode)
>> +             return;
>> +     /*
>> +      * Print out human-readable details about the MCE error,
>> +      * (if the CPU has an implementation for that)
>> +      */
>> +     atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
>> +}
>> +
>>  static void print_mce(struct mce *m)
>>  {
>>       pr_emerg(HW_ERR "CPU %d: Machine Check Exception: %Lx Bank %d: %016Lx\n",
>> @@ -234,11 +246,7 @@ static void print_mce(struct mce *m)
>>       pr_emerg(HW_ERR "PROCESSOR %u:%x TIME %llu SOCKET %u APIC %x\n",
>>               m->cpuvendor, m->cpuid, m->time, m->socketid, m->apicid);
>>
>> -     /*
>> -      * Print out human-readable details about the MCE error,
>> -      * (if the CPU has an implementation for that)
>> -      */
>> -     atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
>> +     call_decoders(m);
>>  }
>>
>>  #define PANIC_TIMEOUT 5 /* 5 seconds */
>> @@ -588,7 +596,7 @@ void machine_check_poll(enum mcp_flags flags, mce_banks_t *b)
>>                */
>>               if (!(flags & MCP_DONTLOG) && !mce_dont_log_ce) {
>>                       mce_log(&m);
>
> Also, there's another hook in the function above that does
> edac_mce_parse(mce) (which shouldnt've been there actually) which is
> used by the Nehalem driver i7core_edac which does also decode DRAM ECCs.

What should we do with this guy?  I'd be happy to send a patch in, but
as I mentioned above, we don't use EDAC at all (which is probably why
I didn't notice this guy).

>
> @Mauro: how about dropping the whole <drivers/edac/edac_mce.c> and using
> a simple notifier which is much smaller in code and does the same thing?

Re: [PATCH] x86: Add an option to disable decoding of MCE

[Borislav Petkov]

Ok, let me preface this with an even easier suggestion: Can you simply
not compile EDAC (which includes CONFIG_EDAC_DECODE_MCE) in your kernels
and the whole issue with decoding disappears simply because no module
registers as a decoder...?

On Tue, Jan 11, 2011 at 02:56:50PM -0500, Mike Waychison wrote:
> >> On our systems, we do not want to have any "decoders" called on machine
> >> check events.  These decoders can easily spam our logs and cause space
> >> problems on machines that have a lot of correctable error events.  We
> >> _do_ however want to get the messages delivered via /dev/mcelog for
> >> userland processing.
> >
> > Ok, question: how do you guys process DRAM ECCs? And more specifically,
> > with a large number of machines, how do you do the mapping from the DRAM
> > ECC error address reported by MCA to a DIMM that's failing in userspace
> > on a particular machine?
> 
> We process machine checks in userland, using carnal knowledge of the
> memory controller and the board specific addressing of the SPDs on the
> various i2c busses to deswizzle and make sense of the addresses and
> symptoms.  We then expose this digested data on the network, which is
> dealt with at the cluster level.

Right, and this means that you need to know all the memory controller
topologies of all the different architectures and also the SPD accessing
based on a board type could be a pain. One of the main reasons for
fleshing out MCE decoding in the kernel was to avoid needless trouble
like that.

> > Also, I've worked on trimming down all that decoding output to 3-5
> > lines. Now it looks like this:
> >
> > [  521.677316] [Hardware Error]: MC4_STATUS[Over|UE|MiscV|PCC|AddrV|UECC]: 0xfe00200000080a0f
> > [  521.686467] [Hardware Error]: Northbridge Error (node 0): DRAM ECC error detected on the NB.
> > [  521.686498] EDAC MC0: UE page 0x0, offset 0x0, grain 0, row 0, labels ":": amd64_edac
> > [  521.686501] EDAC MC0: UE - no information available: UE bit is set
> > [  521.686503] [Hardware Error]: cache level: L3/GEN, mem/io: GEN, mem-tx: GEN, part-proc: RES (no timeout)
> >
> > and the two lines starting with "EDAC MC0" will get trimmed even
> > more with time. I'm assuming this is not a lot but if you get a lot
> > of correctable error events, then output like that accumulates over
> > time.
> 
> This decoded information is great, but it's only really consumable by
> a human.  We'd _much_ rather have structured data that we know isn't
> as volatile (in terms of regexes that break with development churn)
> nor susceptible to corruption (as the logs are not transactional and
> are shared by all CPUs concurrently).   For the most part, we rarely
> rely on the anything automated consuming kernel printk logs, and the
> bits that do are there only for historical reasons (read: we haven't
> figured out how to replace them yet).  With almost every single kernel
> version bump we've had, we've found numerous places where strings in
> the logs have subtly changed, breaking our infrastructure :(  Finding
> these is a painful process and often shows up late in the deployment
> process.  This is disastrous to our deployment schedules as rebooting
> our number of machines, well, it takes a while...

I know exactly what you mean, maybe I should say that the error format
is continuously changing because development is still ongoing. But
also, I don't think that parsing dmesg is the correct approach; I've
heard similar troubles reported by other big server farm people and
what I'm currently working on is a RAS daemon that hooks into perf thus
enabling persistent performance events. This way, you could open a
debugfs file (this'll move to sysfs someday) and read the same decoded
data by mmaping the perf ringbuffer.

This is still in an alpha stage, though, but once we have something
working we could freeze the exported format (think stable tracepoints)
so that you can plug your tools into it and forget all dmesg
parsing. Here's a link to give you an impression of what I mean:
http://lwn.net/Articles/413260/

This is -v3 and I'm currently working on -v4 which should be better.

> I'm sorry if this is tangential to your comment above, but I feel the
> need to have folks recognize that printk() is _not_ a good foundation
> on which to build automation.

Agreed.

> As for using EDAC, I know we tried using it a few years ago, but have
> since reverted to processing MCE ourselves.  I don't know all the
> details as to why, however Duncan Laurie might be able to share more
> (CCed).

It should be in much better shape now :).

> > How about an error thresholding scheme in software then which
> > accumulates the error events and reports only when some configurable
> > thresholds per DRAM device in error have been reached?
> 
> This is pretty much the opposite of what we'd want.   We have no use
> for anything printed in the logs, though emitting these messages
> wouldn't be a problem except for those machines which very frequently
> have MCA events to report.  On those machines, the chatter in the logs
> can potentially wedge our root filesystem (which is tiny) and has the
> downside that it forces logs to rotate quickly (which means there is
> less useful data for a human to consume when analyzing what happened
> on the host).  Our policy has been to neuter bits of the kernel that
> don't know when to stfu :)

Ok, see above. I think with the RAS daemon you could easily make
it even send _decoded_ error records over the network to a central
collecting machine instead of parsing the logs. Then you can simplify
your userspace post-processing too.

[..]

> > Also, there's another hook in the function above that does
> > edac_mce_parse(mce) (which shouldnt've been there actually) which is
> > used by the Nehalem driver i7core_edac which does also decode DRAM ECCs.
> 
> What should we do with this guy?  I'd be happy to send a patch in, but
> as I mentioned above, we don't use EDAC at all (which is probably why
> I didn't notice this guy).

I think this is also easy disabled by not configuring EDAC, as I said
above. Basically, if you don't enable EDAC, you can drop that patch
too and run your kernels without any modification, or am I missing
something..?

Thoughts?

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: [PATCH] x86: Add an option to disable decoding of MCE

[Mike Waychison]

On Tue, Jan 11, 2011 at 12:49 PM, Borislav Petkov <bp@amd64.org> wrote:
> Ok, let me preface this with an even easier suggestion: Can you simply
> not compile EDAC (which includes CONFIG_EDAC_DECODE_MCE) in your kernels
> and the whole issue with decoding disappears simply because no module
> registers as a decoder...?

The trouble here is that default_decode_mce() is still getting called
no matter what :(  It didn't really cause problems until you added
atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, &m) to
machine_check_poll().

> On Tue, Jan 11, 2011 at 02:56:50PM -0500, Mike Waychison wrote:
>> >> On our systems, we do not want to have any "decoders" called on machine
>> >> check events.  These decoders can easily spam our logs and cause space
>> >> problems on machines that have a lot of correctable error events.  We
>> >> _do_ however want to get the messages delivered via /dev/mcelog for
>> >> userland processing.
>> >
>> > Ok, question: how do you guys process DRAM ECCs? And more specifically,
>> > with a large number of machines, how do you do the mapping from the DRAM
>> > ECC error address reported by MCA to a DIMM that's failing in userspace
>> > on a particular machine?
>>
>> We process machine checks in userland, using carnal knowledge of the
>> memory controller and the board specific addressing of the SPDs on the
>> various i2c busses to deswizzle and make sense of the addresses and
>> symptoms.  We then expose this digested data on the network, which is
>> dealt with at the cluster level.
>
> Right, and this means that you need to know all the memory controller
> topologies of all the different architectures and also the SPD accessing
> based on a board type could be a pain. One of the main reasons for
> fleshing out MCE decoding in the kernel was to avoid needless trouble
> like that.

It's not that painful for us.  Our firmware guys own this userland
code :)  I can see it being a pain for others however.

Doing this all in userland does have it's upsides as well fwiw.  For
example, MC4 is usually kept around across a reset, which means that
the firmware can pick it up when the system goes down due to a
processor context corruption.  We rely on these libraries as well to
decode egregious uncorrectable memory errors as well as bus errors
(like hypertransport sync floods).

*snip*

> I've
> heard similar troubles reported by other big server farm people and
> what I'm currently working on is a RAS daemon that hooks into perf thus
> enabling persistent performance events. This way, you could open a
> debugfs file (this'll move to sysfs someday) and read the same decoded
> data by mmaping the perf ringbuffer.

I'll definitely keep an eye out for your developments with RAS :)

>
> I think this is also easy disabled by not configuring EDAC, as I said
> above. Basically, if you don't enable EDAC, you can drop that patch
> too and run your kernels without any modification, or am I missing
> something..?

See the comment above.  It's the spamming for the default decoder that
I wanted to disable specifically, though my approach was a knob to
disable decoders generally.

Re: [PATCH] x86: Add an option to disable decoding of MCE

[Duncan Laurie]

On Tue, Jan 11, 2011 at 11:56 AM, Mike Waychison <mikew@google.com> wrote:
>
> As for using EDAC, I know we tried using it a few years ago, but have
> since reverted to processing MCE ourselves.  I don't know all the
> details as to why, however Duncan Laurie might be able to share more
> (CCed).
>


The short answer is that we needed access to the raw chipset error
registers rather than the parsed and collated counters (or printk
output) that we got with EDAC.

The long answer is that our goals have changed over time.  For a long
time our goal with ECC was simply identifying the actual bad DIMM in a
system so it can be replaced.  For systems which report ECC via
machine checks this is done by collecting the physical addresses and
converting it with an external utility.  For other systems which did
not report memory errors via machine checks EDAC was used to poll and
decode the chipset error registers and export row+channel sysfs
counters because there was not a physical memory address supplied with
each event.

However as the scale of our operations grows so too does the number of
DIMMs that have to be replaced.  By gathering and analyzing complete
error data (row, column, bank, rank, etc) we are able to identify if
there is a single chip on a module that is bad, which could
potentially get repaired rather than replacing the whole module.

Gathering this data is easy enough with machine checks since you have
the physical address on which it occurred and doing the translation of
that address into a particular DIMM ends up providing the other
relevant geometry information along the way.  It also has a convenient
interface via mcelog that makes it easy to consume the event and do
the translation in userspace (as long as you run with a high mce
tolerance level) and has other advantages of being able to develop and
maintain the utility outside of our internal kernel release schedule.

For platforms that were reporting counters via EDAC there are several
values that need to be read from different chipset registers and
exported.  Because parsing printk output for this data is messy we
decided that rather than invent another mcelog type interface (or
subvert an existing interface) we might as well do the polling for
events in userspace and simply save all the data we care about at that
point.

As has been noted this requires us to have intimate knowledge of the
hardware that we're running on.  It isn't an approach that I would
recommend to many and your efforts to do decoding of errors with EDAC
is something that most people will benefit from.

-duncan

Re: [PATCH] x86: Add an option to disable decoding of MCE

[Borislav Petkov]

On Tue, Jan 11, 2011 at 04:51:57PM -0500, Mike Waychison wrote:
> On Tue, Jan 11, 2011 at 12:49 PM, Borislav Petkov <bp@amd64.org> wrote:
> > Ok, let me preface this with an even easier suggestion: Can you simply
> > not compile EDAC (which includes CONFIG_EDAC_DECODE_MCE) in your kernels
> > and the whole issue with decoding disappears simply because no module
> > registers as a decoder...?
> 
> The trouble here is that default_decode_mce() is still getting called
> no matter what :(  It didn't really cause problems until you added
> atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, &m) to
> machine_check_poll().

Yeah, I figured as much... after hitting send :(. Feel free to add my

Acked-by: Borislav Petkov <borislav.petkov@amd.com>

to your patch.

> > Right, and this means that you need to know all the memory controller
> > topologies of all the different architectures and also the SPD accessing
> > based on a board type could be a pain. One of the main reasons for
> > fleshing out MCE decoding in the kernel was to avoid needless trouble
> > like that.
> 
> It's not that painful for us.  Our firmware guys own this userland
> code :)  I can see it being a pain for others however.

Yeah, what we actually need is a chipset-agnostic way (ACPI maybe?)
of mapping chip selects to the actual DIMMs so all that above can be
avoided...

> Doing this all in userland does have it's upsides as well fwiw. For
> example, MC4 is usually kept around across a reset, which means
> that the firmware can pick it up when the system goes down due to a
> processor context corruption.

True, although not always reliable due to that context corruption. Tony
Luck is adding code for writing the error information to a persistent
storage _before_ you reset so on platforms with such a device you'll
be able to look at MC4 (or any of the MCE regs, for that matter) after
reset, which, I think, would make error analysis much more convenient.
See http://marc.info/?l=linux-arch&m=129356804427008&w=2.

> We rely on these libraries as well to decode egregious uncorrectable
> memory errors as well as bus errors (like hypertransport sync floods).

Yeah, I don't think error decoding would help, at least on AMD, with
uncorrectable errors causing syncflood. We might do something about it
though :).

> > I've
> > heard similar troubles reported by other big server farm people and
> > what I'm currently working on is a RAS daemon that hooks into perf thus
> > enabling persistent performance events. This way, you could open a
> > debugfs file (this'll move to sysfs someday) and read the same decoded
> > data by mmaping the perf ringbuffer.
> 
> I'll definitely keep an eye out for your developments with RAS :)

Cool, I think I have the bulk of your requirements now, from your and
Duncan's mails. I'm always open for other suggestions too.

Thanks.

-- 
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

[PATCH] x86: Add an option to disable decoding of MCE

[Mike Waychison]

v3

This patch applies to v2.6.37.

Updated with documentation of the new option.

Renamed call_decoders() -> decode_mce()
---

On our systems, we do not want to have any "decoders" called on machine
check events.  These decoders can easily spam our logs and cause space
problems on machines that have a lot of correctable error events.  We
_do_ however want to get the messages delivered via /dev/mcelog for
userland processing.

Introduce an interface "dont_decode" that allows us to skip the
decoders.  We always call the decoders by default.

Google-Bug-Id: 3289142
Signed-off-by: Mike Waychison <mikew@google.com>
Acked-by: Borislav Petkov <borislav.petkov@amd.com>
---
 Documentation/x86/x86_64/boot-options.txt |    5 +++++
 Documentation/x86/x86_64/machinecheck     |    6 ++++++
 arch/x86/kernel/cpu/mcheck/mce.c          |   24 ++++++++++++++++++------
 3 files changed, 29 insertions(+), 6 deletions(-)

diff --git a/Documentation/x86/x86_64/boot-options.txt b/Documentation/x86/x86_64/boot-options.txt
index 7fbbaf8..dd7145a 100644
--- a/Documentation/x86/x86_64/boot-options.txt
+++ b/Documentation/x86/x86_64/boot-options.txt
@@ -22,6 +22,11 @@ Machine check
 		as corrected are silently cleared by OS.
 		This option will be useful if you have no interest in any
 		of corrected errors.
+   mce=dont_decode
+		Disable in-kernel decoding of errors.  Setting this boot
+		option will cause EDAC to be skipped (if enabled) and no
+		messages to be printed into the logs.  Events will still
+		be available via /dev/mcelog however.
    mce=ignore_ce
 		Disable features for corrected errors, e.g. polling timer
 		and CMCI.  All events reported as corrected are not cleared
diff --git a/Documentation/x86/x86_64/machinecheck b/Documentation/x86/x86_64/machinecheck
index b1fb302..7ef7003 100644
--- a/Documentation/x86/x86_64/machinecheck
+++ b/Documentation/x86/x86_64/machinecheck
@@ -65,6 +65,12 @@ tolerant
 	Note this only makes a difference if the CPU allows recovery
 	from a machine check exception. Current x86 CPUs generally do not.
 
+dont_decode
+	Disable in-kernel decoding of any errors.  Setting this boot
+	option will cause EDAC to be skipped (if enabled) and no
+	messages to be printed into the logs.  Events will still be
+	available via /dev/mcelog however.
+
 trigger
 	Program to run when a machine check event is detected.
 	This is an alternative to running mcelog regularly from cron
diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index 7a35b72..ee91e96 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -82,6 +82,7 @@ static int			mce_bootlog		__read_mostly = -1;
 static int			monarch_timeout		__read_mostly = -1;
 static int			mce_panic_timeout	__read_mostly;
 static int			mce_dont_log_ce		__read_mostly;
+static int			mce_dont_decode		__read_mostly;
 int				mce_cmci_disabled	__read_mostly;
 int				mce_ignore_ce		__read_mostly;
 int				mce_ser			__read_mostly;
@@ -209,6 +210,17 @@ void mce_log(struct mce *mce)
 	set_bit(0, &mce_need_notify);
 }
 
+static void decode_mce(struct mce *m)
+{
+	if (mce_dont_decode)
+		return;
+	/*
+	 * Print out human-readable details about the MCE error,
+	 * (if the CPU has an implementation for that)
+	 */
+	atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
+}
+
 static void print_mce(struct mce *m)
 {
 	pr_emerg(HW_ERR "CPU %d: Machine Check Exception: %Lx Bank %d: %016Lx\n",
@@ -234,11 +246,7 @@ static void print_mce(struct mce *m)
 	pr_emerg(HW_ERR "PROCESSOR %u:%x TIME %llu SOCKET %u APIC %x\n",
 		m->cpuvendor, m->cpuid, m->time, m->socketid, m->apicid);
 
-	/*
-	 * Print out human-readable details about the MCE error,
-	 * (if the CPU has an implementation for that)
-	 */
-	atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
+	decode_mce(m);
 }
 
 #define PANIC_TIMEOUT 5 /* 5 seconds */
@@ -588,7 +596,7 @@ void machine_check_poll(enum mcp_flags flags, mce_banks_t *b)
 		 */
 		if (!(flags & MCP_DONTLOG) && !mce_dont_log_ce) {
 			mce_log(&m);
-			atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, &m);
+			decode_mce(&m);
 			add_taint(TAINT_MACHINE_CHECK);
 		}
 
@@ -1700,6 +1708,8 @@ static int __init mcheck_enable(char *str)
 		mce_cmci_disabled = 1;
 	else if (!strcmp(str, "dont_log_ce"))
 		mce_dont_log_ce = 1;
+	else if (!strcmp(str, "dont_decode"))
+		mce_dont_decode = 1;
 	else if (!strcmp(str, "ignore_ce"))
 		mce_ignore_ce = 1;
 	else if (!strcmp(str, "bootlog") || !strcmp(str, "nobootlog"))
@@ -1926,6 +1936,7 @@ static SYSDEV_ATTR(trigger, 0644, show_trigger, set_trigger);
 static SYSDEV_INT_ATTR(tolerant, 0644, tolerant);
 static SYSDEV_INT_ATTR(monarch_timeout, 0644, monarch_timeout);
 static SYSDEV_INT_ATTR(dont_log_ce, 0644, mce_dont_log_ce);
+static SYSDEV_INT_ATTR(dont_decode, 0644, mce_dont_decode);
 
 static struct sysdev_ext_attribute attr_check_interval = {
 	_SYSDEV_ATTR(check_interval, 0644, sysdev_show_int,
@@ -1949,6 +1960,7 @@ static struct sysdev_attribute *mce_attrs[] = {
 	&attr_trigger,
 	&attr_monarch_timeout.attr,
 	&attr_dont_log_ce.attr,
+	&attr_dont_decode.attr,
 	&attr_ignore_ce.attr,
 	&attr_cmci_disabled.attr,
 	NULL
-- 
1.7.3.1

Re: [PATCH] x86: Add an option to disable decoding of MCE

[Tony Luck]

On Tue, Jan 11, 2011 at 2:48 PM, Borislav Petkov <bp@amd64.org> wrote:
> True, although not always reliable due to that context corruption. Tony
> Luck is adding code for writing the error information to a persistent
> storage _before_ you reset so on platforms with such a device you'll
> be able to look at MC4 (or any of the MCE regs, for that matter) after
> reset, which, I think, would make error analysis much more convenient.
> See http://marc.info/?l=linux-arch&m=129356804427008&w=2.

This code is also in linux-next (any next-2011* tag).  I'd be very happy
to get some testing reports, feedback on the code, etc.

-Tony

Re: [PATCH] x86: Add an option to disable decoding of MCE

[Tony Luck]

On Tue, Jan 11, 2011 at 11:56 AM, Mike Waychison <mikew@google.com> wrote:
> figured out how to replace them yet).  With almost every single kernel
> version bump we've had, we've found numerous places where strings in
> the logs have subtly changed, breaking our infrastructure :(  Finding
> these is a painful process and often shows up late in the deployment
> process.  This is disastrous to our deployment schedules as rebooting
> our number of machines, well, it takes a while...
>
> I'm sorry if this is tangential to your comment above, but I feel the
> need to have folks recognize that printk() is _not_ a good foundation
> on which to build automation.

Congratulations .... you made the lwn.net "quote of the week"
  http://lwn.net/Articles/421784

 - wider distribution of this wisdom is a very good thing.

-Tony