linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: ira.weiny@intel.com
To: Rik van Riel <riel@surriel.com>, Borislav Petkov <bp@alien8.de>,
	Dave Hansen <dave.hansen@intel.com>
Cc: Dave Jones <dsj@fb.com>,
	x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com
Subject: [RFC PATCH 4/5] x86,mm: print likely CPU at segfault time
Date: Fri,  5 Aug 2022 10:30:08 -0700	[thread overview]
Message-ID: <20220805173009.3128098-5-ira.weiny@intel.com> (raw)
In-Reply-To: <20220805173009.3128098-1-ira.weiny@intel.com>

From: Rik van Riel <riel@surriel.com>

In a large enough fleet of computers, it is common to have a few bad CPUs.
Those can often be identified by seeing that some commonly run kernel code,
which runs fine everywhere else, keeps crashing on the same CPU core on one
particular bad system.

However, the failure modes in CPUs that have gone bad over the years are
often oddly specific, and the only bad behavior seen might be segfaults
in programs like bash, python, or various system daemons that run fine
everywhere else.

Add a printk() to show_signal_msg() to print the CPU, core, and socket
at segfault time. This is not perfect, since the task might get rescheduled
on another CPU between when the fault hit, and when the message is printed,
but in practice this has been good enough to help us identify several bad
CPU cores.

segfault[1349]: segfault at 0 ip 000000000040113a sp 00007ffc6d32e360 error 4 in segfault[401000+1000] on CPU 0 (core 0, socket 0)

This printk can be controlled through /proc/sys/debug/exception-trace

Signed-off-by: Rik van Riel <riel@surriel.com>
CC: Dave Jones <dsj@fb.com>
---
 arch/x86/mm/fault.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 971977c438fc..82cf23975aa1 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -769,6 +769,8 @@ show_signal_msg(struct pt_regs *regs, unsigned long error_code,
 		unsigned long address, struct task_struct *tsk)
 {
 	const char *loglvl = task_pid_nr(tsk) > 1 ? KERN_INFO : KERN_EMERG;
+	/* This is a racy snapshot, but it's better than nothing. */
+	int cpu = raw_smp_processor_id();
 
 	if (!unhandled_signal(tsk, SIGSEGV))
 		return;
@@ -782,6 +784,14 @@ show_signal_msg(struct pt_regs *regs, unsigned long error_code,
 
 	print_vma_addr(KERN_CONT " in ", regs->ip);
 
+	/*
+	 * Dump the likely CPU where the fatal segfault happened.
+	 * This can help identify faulty hardware.
+	 */
+	printk(KERN_CONT " on CPU %d (core %d, socket %d)", cpu,
+	       topology_core_id(cpu), topology_physical_package_id(cpu));
+
+
 	printk(KERN_CONT "\n");
 
 	show_opcodes(regs, loglvl);
-- 
2.35.3


  parent reply	other threads:[~2022-08-05 17:31 UTC|newest]

Thread overview: 37+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-08-05 17:30 [RFC PATCH 0/5] Print CPU at segfault time ira.weiny
2022-08-05 17:30 ` [RFC PATCH 1/5] entry: Pass pt_regs to irqentry_exit_cond_resched() ira.weiny
2022-08-05 18:33   ` Rik van Riel
2022-08-08 10:38   ` Borislav Petkov
2022-08-08 17:34     ` Ira Weiny
2022-08-08 17:38       ` Borislav Petkov
2022-08-08 17:43         ` Dave Hansen
2022-08-08 17:52           ` Borislav Petkov
2022-08-09 23:18     ` Thomas Gleixner
2022-08-10  7:25       ` Thomas Gleixner
2022-08-05 17:30 ` [RFC PATCH 2/5] entry: Add calls for save/restore auxiliary pt_regs ira.weiny
2022-08-05 18:34   ` Rik van Riel
2022-08-09 12:05   ` Borislav Petkov
2022-08-09 18:38     ` Ira Weiny
2022-08-09 18:49       ` Borislav Petkov
2022-08-09 21:14         ` Thomas Gleixner
2022-08-09 21:38           ` Borislav Petkov
2022-08-09 22:33             ` Ira Weiny
2022-08-09 21:49         ` Ira Weiny
2022-08-09 23:53           ` Thomas Gleixner
2022-08-05 17:30 ` [RFC PATCH 3/5] x86/entry: Add auxiliary pt_regs space ira.weiny
2022-08-05 18:45   ` Rik van Riel
2022-08-05 17:30 ` ira.weiny [this message]
2022-08-05 17:30 ` [RFC PATCH 5/5] x86/entry: Store CPU info on exception entry ira.weiny
2022-08-05 18:46   ` Rik van Riel
2022-08-05 18:47   ` Dave Hansen
2022-08-06  9:01     ` Ingo Molnar
2022-08-06  9:11       ` Borislav Petkov
2022-08-07 10:02         ` Ingo Molnar
2022-08-07 10:35           ` Borislav Petkov
2022-08-07 20:02             ` Ira Weiny
2022-08-08 11:03               ` Ingo Molnar
2022-08-08 12:01                 ` Borislav Petkov
2022-08-09 20:06                   ` Thomas Gleixner
2022-08-08 16:16                 ` Dave Hansen
2022-08-08 17:24                   ` Rik van Riel
2022-08-09 21:19                   ` Andy Lutomirski

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220805173009.3128098-5-ira.weiny@intel.com \
    --to=ira.weiny@intel.com \
    --cc=bp@alien8.de \
    --cc=dave.hansen@intel.com \
    --cc=dsj@fb.com \
    --cc=kernel-team@fb.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=riel@surriel.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).