* MCE reports errors that can't be verified
@ 2018-10-29 10:45 Daniel Aberger - Profihost AG
2018-10-29 16:15 ` Borislav Petkov
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Aberger - Profihost AG @ 2018-10-29 10:45 UTC (permalink / raw)
To: tony.luck, bp, mingo, hpa, x86, linux-edac, linux-kernel
Cc: s.priebe, n.fahldieck, p.kramme
Hello,
We currently have several servers reporting faulty memory through MCE.
Example dmesg output:
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: Machine check events logged
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: CPU 12: Machine Check:
0 Bank 7: cc027c0000010091
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: TSC 0 ADDR 70fc337d80
MISC 50202086
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: PROCESSOR 0:306f2 TIME
1534938887 SOCKET 1 APIC 20 microcode 3d
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: Machine check events logged
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: CPU 14: Machine Check:
0 Bank 7: 8c00004000010091
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: TSC 0 ADDR 70fb117d40
MISC 4268e886
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: PROCESSOR 0:306f2 TIME
1534938887 SOCKET 1 APIC 24 microcode 3d
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: CPU 15: Machine Check:
0 Bank 7: cc00008000010091
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: TSC 0 ADDR 70fb1b3ec0
MISC 142189886
[Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: PROCESSOR 0:306f2 TIME
1534938887 SOCKET 1 APIC 26 microcode 3d
Normally we verify those errors by checking IPMIs event log, but no
errors are showing there.
Neither IPMI nor ras-mc-ctl report any errors.
We encountered this problem running Kernel 4.12.0 based on openSUSE
SLE15 on commit a906b62b3f80679eac4f38373492a871c5f3568e.
Is this an MCE Kernel bug?
--
Mit freundlichen Grüßen
Daniel Aberger
Ihr Profihost Team
-------------------------------
Profihost AG
Expo Plaza 1
30539 Hannover
Deutschland
Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282
URL: http://www.profihost.com | E-Mail: info@profihost.com
Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827
Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350
Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe
Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: MCE reports errors that can't be verified
2018-10-29 10:45 MCE reports errors that can't be verified Daniel Aberger - Profihost AG
@ 2018-10-29 16:15 ` Borislav Petkov
2018-10-29 16:59 ` Luck, Tony
0 siblings, 1 reply; 6+ messages in thread
From: Borislav Petkov @ 2018-10-29 16:15 UTC (permalink / raw)
To: Daniel Aberger - Profihost AG
Cc: tony.luck, mingo, hpa, x86, linux-edac, linux-kernel, s.priebe,
n.fahldieck, p.kramme
On Mon, Oct 29, 2018 at 11:45:04AM +0100, Daniel Aberger - Profihost AG wrote:
...
> [Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: CPU 15: Machine Check:
> 0 Bank 7: cc00008000010091
> [Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: TSC 0 ADDR 70fb1b3ec0
> MISC 142189886
> [Mi Aug 22 13:54:47 2018] mce: [Hardware Error]: PROCESSOR 0:306f2 TIME
> 1534938887 SOCKET 1 APIC 26 microcode 3d
The fact that you see this, means, the error has reached the last
notifier. So the EDAC notifier must've run too and handed the error to
the EDAC driver.
Can you send a full dmesg from that machine, privately to Tony and me is
fine too.
Thx.
--
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.
^ permalink raw reply [flat|nested] 6+ messages in thread* RE: MCE reports errors that can't be verified
2018-10-29 16:15 ` Borislav Petkov
@ 2018-10-29 16:59 ` Luck, Tony
2018-10-29 17:51 ` Borislav Petkov
0 siblings, 1 reply; 6+ messages in thread
From: Luck, Tony @ 2018-10-29 16:59 UTC (permalink / raw)
To: Borislav Petkov, Daniel Aberger - Profihost AG
Cc: mingo, hpa, x86, linux-edac, linux-kernel, s.priebe, n.fahldieck,
p.kramme
> The fact that you see this, means, the error has reached the last
> notifier. So the EDAC notifier must've run too and handed the error to
> the EDAC driver.
>
> Can you send a full dmesg from that machine, privately to Tony and me is
> fine too.
Some system configuration information would be helpful. I can see from the logs
that this on a Haswell Xeon, but not which model. The addresses of the errors
are around 448GB, how much memory does the system have? Output of
"dmidecode -t 17" would help.
There are some strange values in the snip you sent. First logged error shows a
corrected error count of 2544. Next shows just count=1 (with overflow bit set to 0).
Third has a count of 2, with overflow set.
So I'm guessing that you may have some storms of corrected errors and Linux
has switched to polling mode.
The three errors in the sample are all on the same channel (1) of the same memory
controller (0). But the addresses are different. Indicates that something serious has
happened to a DIMM on that channel.
The EDAC driver printed out those messages, but then silently failed to give any
extra detail. Not really sure how that happened. Perhaps I'm missing some error
path where it just gives up. Can you run (as root) "lspci -xxxx" and send the output
from that. (Compress and send off list).
-Tony
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: MCE reports errors that can't be verified
2018-10-29 16:59 ` Luck, Tony
@ 2018-10-29 17:51 ` Borislav Petkov
2018-10-29 18:25 ` Luck, Tony
0 siblings, 1 reply; 6+ messages in thread
From: Borislav Petkov @ 2018-10-29 17:51 UTC (permalink / raw)
To: Luck, Tony
Cc: Daniel Aberger - Profihost AG, mingo, hpa, x86, linux-edac,
linux-kernel, s.priebe, n.fahldieck, p.kramme
On Mon, Oct 29, 2018 at 04:59:44PM +0000, Luck, Tony wrote:
> The EDAC driver printed out those messages,
I don't think so - that's __print_mce() in mce.c which dumps the three
lines under "... events logged". Which means, that's the lowest prio,
fallback notifier which runs when nothing else before it has done any
logging.
--
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: MCE reports errors that can't be verified
2018-10-29 17:51 ` Borislav Petkov
@ 2018-10-29 18:25 ` Luck, Tony
2018-10-30 7:22 ` Daniel Aberger - Profihost AG
0 siblings, 1 reply; 6+ messages in thread
From: Luck, Tony @ 2018-10-29 18:25 UTC (permalink / raw)
To: Borislav Petkov
Cc: Daniel Aberger - Profihost AG, mingo, hpa, x86, linux-edac,
linux-kernel, s.priebe, n.fahldieck, p.kramme
On Mon, Oct 29, 2018 at 06:51:29PM +0100, Borislav Petkov wrote:
> On Mon, Oct 29, 2018 at 04:59:44PM +0000, Luck, Tony wrote:
> > The EDAC driver printed out those messages,
>
> I don't think so - that's __print_mce() in mce.c which dumps the three
> lines under "... events logged". Which means, that's the lowest prio,
> fallback notifier which runs when nothing else before it has done any
> logging.
Ah yes. In which case the EDAC driver wasn't loaded (or gave up
silently) to let us arrive at the end of the decode chain.
Another question. Does /proc/interrupts show a lot of CMCI (for
historical reasons this is the THR: line).
-Tony
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: MCE reports errors that can't be verified
2018-10-29 18:25 ` Luck, Tony
@ 2018-10-30 7:22 ` Daniel Aberger - Profihost AG
0 siblings, 0 replies; 6+ messages in thread
From: Daniel Aberger - Profihost AG @ 2018-10-30 7:22 UTC (permalink / raw)
To: Luck, Tony, Borislav Petkov
Cc: mingo, hpa, x86, linux-edac, linux-kernel, s.priebe, n.fahldieck,
p.kramme
Am 29.10.18 um 19:25 schrieb Luck, Tony:
> On Mon, Oct 29, 2018 at 06:51:29PM +0100, Borislav Petkov wrote:
>> On Mon, Oct 29, 2018 at 04:59:44PM +0000, Luck, Tony wrote:
>>> The EDAC driver printed out those messages,
>>
>> I don't think so - that's __print_mce() in mce.c which dumps the three
>> lines under "... events logged". Which means, that's the lowest prio,
>> fallback notifier which runs when nothing else before it has done any
>> logging.
>
> Ah yes. In which case the EDAC driver wasn't loaded (or gave up
> silently) to let us arrive at the end of the decode chain.
>
> Another question. Does /proc/interrupts show a lot of CMCI (for
> historical reasons this is the THR: line).
>
> -Tony
>
I mailed you and Boris the requested information privately. Thanks!
- Daniel
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-10-30 7:22 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-29 10:45 MCE reports errors that can't be verified Daniel Aberger - Profihost AG
2018-10-29 16:15 ` Borislav Petkov
2018-10-29 16:59 ` Luck, Tony
2018-10-29 17:51 ` Borislav Petkov
2018-10-29 18:25 ` Luck, Tony
2018-10-30 7:22 ` Daniel Aberger - Profihost AG
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).