MCE reports errors that can't be verified

MCE reports errors that can't be verified

[Daniel Aberger - Profihost AG]

Hello,

We currently have several servers reporting faulty memory through MCE.

Example dmesg output:

[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: Machine check events logged
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: CPU 12: Machine Check:
0 Bank 7: cc027c0000010091
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: TSC 0 ADDR 70fc337d80
MISC 50202086
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: PROCESSOR 0:306f2 TIME
1534938887 SOCKET 1 APIC 20 microcode 3d
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: Machine check events logged
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: CPU 14: Machine Check:
0 Bank 7: 8c00004000010091
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: TSC 0 ADDR 70fb117d40
MISC 4268e886
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: PROCESSOR 0:306f2 TIME
1534938887 SOCKET 1 APIC 24 microcode 3d
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: CPU 15: Machine Check:
0 Bank 7: cc00008000010091
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: TSC 0 ADDR 70fb1b3ec0
MISC 142189886
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: PROCESSOR 0:306f2 TIME
1534938887 SOCKET 1 APIC 26 microcode 3d

Normally we verify those errors by checking IPMIs event log, but no
errors are showing there.

Neither IPMI nor ras-mc-ctl report any errors.

We encountered this problem running Kernel 4.12.0 based on openSUSE
SLE15 on commit a906b62b3f80679eac4f38373492a871c5f3568e.

Is this an MCE Kernel bug?


-- 
Mit freundlichen Grüßen
  Daniel Aberger
Ihr Profihost Team

-------------------------------
Profihost AG
Expo Plaza 1
30539 Hannover
Deutschland

Tel.: +49 (511) 5151 8181     | Fax.: +49 (511) 5151 8282
URL: http://www.profihost.com | E-Mail: info@profihost.com

Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827
Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350
Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe
Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)

Re: MCE reports errors that can't be verified

[Borislav Petkov]

On Mon, Oct 29, 2018 at 11:45:04AM +0100, Daniel Aberger - Profihost AG wrote:

...

> [Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: CPU 15: Machine Check:
> 0 Bank 7: cc00008000010091
> [Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: TSC 0 ADDR 70fb1b3ec0
> MISC 142189886
> [Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: PROCESSOR 0:306f2 TIME
> 1534938887 SOCKET 1 APIC 26 microcode 3d

The fact that you see this, means, the error has reached the last
notifier. So the EDAC notifier must've run too and handed the error to
the EDAC driver.

Can you send a full dmesg from that machine, privately to Tony and me is
fine too.

Thx.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

RE: MCE reports errors that can't be verified

[Luck, Tony]

> The fact that you see this, means, the error has reached the last
> notifier. So the EDAC notifier must've run too and handed the error to
> the EDAC driver.
>
> Can you send a full dmesg from that machine, privately to Tony and me is
> fine too.

Some system configuration information would be helpful. I can see from the logs
that this on a Haswell Xeon, but not which model.  The addresses of the errors
are around 448GB, how much memory does the system have? Output of
"dmidecode -t 17" would help.

There are some strange values in the snip you sent. First logged error shows a
corrected error count of 2544. Next shows just count=1 (with overflow bit set to 0).
Third has a count of 2, with overflow set.

So I'm guessing that you may have some storms of corrected errors and Linux
has switched to polling mode.

The three errors in the sample are all on the same channel (1) of the same memory
controller (0). But the addresses are different. Indicates that something serious has
happened to a DIMM on that channel.

The EDAC driver printed out those messages, but then silently failed to give any
extra detail.  Not really sure how that happened. Perhaps I'm missing some error
path where it just gives up. Can you run (as root) "lspci -xxxx" and send the output
from that. (Compress and send off list).

-Tony

Re: MCE reports errors that can't be verified

[Borislav Petkov]

On Mon, Oct 29, 2018 at 04:59:44PM +0000, Luck, Tony wrote:
> The EDAC driver printed out those messages,

I don't think so - that's __print_mce() in mce.c which dumps the three
lines under "... events logged". Which means, that's the lowest prio,
fallback notifier which runs when nothing else before it has done any
logging.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

Re: MCE reports errors that can't be verified

[Luck, Tony]

On Mon, Oct 29, 2018 at 06:51:29PM +0100, Borislav Petkov wrote:
> On Mon, Oct 29, 2018 at 04:59:44PM +0000, Luck, Tony wrote:
> > The EDAC driver printed out those messages,
> 
> I don't think so - that's __print_mce() in mce.c which dumps the three
> lines under "... events logged". Which means, that's the lowest prio,
> fallback notifier which runs when nothing else before it has done any
> logging.

Ah yes. In which case the EDAC driver wasn't loaded (or gave up
silently) to let us arrive at the end of the decode chain.

Another question. Does /proc/interrupts show a lot of CMCI (for
historical reasons this is the THR: line).

-Tony

Re: MCE reports errors that can't be verified

[Daniel Aberger - Profihost AG]

Am 29.10.18 um 19:25 schrieb Luck, Tony:
> On Mon, Oct 29, 2018 at 06:51:29PM +0100, Borislav Petkov wrote:
>> On Mon, Oct 29, 2018 at 04:59:44PM +0000, Luck, Tony wrote:
>>> The EDAC driver printed out those messages,
>>
>> I don't think so - that's __print_mce() in mce.c which dumps the three
>> lines under "... events logged". Which means, that's the lowest prio,
>> fallback notifier which runs when nothing else before it has done any
>> logging.
> 
> Ah yes. In which case the EDAC driver wasn't loaded (or gave up
> silently) to let us arrive at the end of the decode chain.
> 
> Another question. Does /proc/interrupts show a lot of CMCI (for
> historical reasons this is the THR: line).
> 
> -Tony
> 

I mailed you and Boris the requested information privately. Thanks!

- Daniel