linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Stephen Brennan <stephen.s.brennan@oracle.com>
To: Petr Mladek <pmladek@suse.com>,
	Sergey Senozhatsky <senozhatsky@chromium.org>,
	Steven Rostedt <rostedt@goodmis.org>,
	John Ogness <john.ogness@linutronix.de>
Cc: Stephen Brennan <stephen.s.brennan@oracle.com>,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	linux-kernel@vger.kernel.org
Subject: [PATCH 3/4] printk: Avoid livelock with heavy printk during panic
Date: Fri, 21 Jan 2022 11:02:21 -0800	[thread overview]
Message-ID: <20220121190222.572694-4-stephen.s.brennan@oracle.com> (raw)
In-Reply-To: <20220121190222.572694-1-stephen.s.brennan@oracle.com>

During a panic(), if another CPU is writing heavily the kernel log (e.g.
via /dev/kmsg), then the panic CPU may livelock writing out its messages
to the console. Note when too many messages are dropped during panic and
suppress further printk, except from the panic CPU.

Suggested-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
---
 kernel/printk/printk.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 20b4b71a1a07..ca253ac07615 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -93,6 +93,12 @@ EXPORT_SYMBOL_GPL(console_drivers);
  */
 int __read_mostly suppress_printk;
 
+/*
+ * During panic, heavy printk by other CPUs can delay the
+ * panic and risk deadlock on console resources.
+ */
+int __read_mostly suppress_panic_printk;
+
 #ifdef CONFIG_LOCKDEP
 static struct lockdep_map console_lock_dep_map = {
 	.name = "console_lock"
@@ -2228,6 +2234,10 @@ asmlinkage int vprintk_emit(int facility, int level,
 	if (unlikely(suppress_printk))
 		return 0;
 
+	if (unlikely(suppress_panic_printk) &&
+	    atomic_read(&panic_cpu) != raw_smp_processor_id())
+		return 0;
+
 	if (level == LOGLEVEL_SCHED) {
 		level = LOGLEVEL_DEFAULT;
 		in_sched = true;
@@ -2613,6 +2623,7 @@ void console_unlock(void)
 {
 	static char ext_text[CONSOLE_EXT_LOG_MAX];
 	static char text[CONSOLE_LOG_MAX];
+	static int panic_console_dropped;
 	unsigned long flags;
 	bool do_cond_resched, retry;
 	struct printk_info info;
@@ -2667,6 +2678,8 @@ void console_unlock(void)
 		if (console_seq != r.info->seq) {
 			console_dropped += r.info->seq - console_seq;
 			console_seq = r.info->seq;
+			if (panic_in_progress() && panic_console_dropped++ > 10)
+				suppress_panic_printk = 1;
 		}
 
 		if (suppress_message_printing(r.info->level)) {
-- 
2.30.2


  parent reply	other threads:[~2022-01-21 19:03 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-01-21 19:02 [PATCH 0/4] printk: reduce deadlocks during panic Stephen Brennan
2022-01-21 19:02 ` [PATCH 1/4] panic: Add panic_in_progress helper Stephen Brennan
2022-01-25 11:48   ` Petr Mladek
2022-01-26 17:37     ` Stephen Brennan
2022-01-21 19:02 ` [PATCH 2/4] printk: disable optimistic spin during panic Stephen Brennan
2022-01-25 12:42   ` Petr Mladek
2022-01-26  9:18   ` Sergey Senozhatsky
2022-01-26  9:45     ` John Ogness
2022-01-26 10:06       ` Sergey Senozhatsky
2022-01-26 18:15         ` Stephen Brennan
2022-01-27  7:11           ` Sergey Senozhatsky
2022-01-27  9:09             ` John Ogness
2022-01-27 11:38             ` Petr Mladek
2022-01-27 12:43               ` John Ogness
2022-01-27 14:25                 ` Petr Mladek
2022-01-21 19:02 ` Stephen Brennan [this message]
2022-01-25 14:25   ` [PATCH 3/4] printk: Avoid livelock with heavy printk " Petr Mladek
2022-01-21 19:02 ` [PATCH 4/4] printk: Drop console_sem " Stephen Brennan
2022-01-24 16:12   ` John Ogness
2022-01-24 16:26     ` John Ogness
2022-01-25 15:04     ` Petr Mladek

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220121190222.572694-4-stephen.s.brennan@oracle.com \
    --to=stephen.s.brennan@oracle.com \
    --cc=john.ogness@linutronix.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=pmladek@suse.com \
    --cc=rostedt@goodmis.org \
    --cc=senozhatsky@chromium.org \
    --cc=sergey.senozhatsky@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).