From: Petr Mladek <pmladek@suse.com>
To: Muchun Song <songmuchun@bytedance.com>
Cc: sergey.senozhatsky@gmail.com, rostedt@goodmis.org,
john.ogness@linutronix.de, akpm@linux-foundation.org,
linux-kernel@vger.kernel.org, stable@vger.kernel.org
Subject: Re: [PATCH v3] printk: fix deadlock when kernel panic
Date: Wed, 10 Feb 2021 18:16:59 +0100 [thread overview]
Message-ID: <YCQVC6oBKKjtM/yg@alley> (raw)
In-Reply-To: <20210210034823.64867-1-songmuchun@bytedance.com>
On Wed 2021-02-10 11:48:23, Muchun Song wrote:
> printk_safe_flush_on_panic() caused the following deadlock on our
> server:
>
> CPU0: CPU1:
> panic rcu_dump_cpu_stacks
> kdump_nmi_shootdown_cpus nmi_trigger_cpumask_backtrace
> register_nmi_handler(crash_nmi_callback) printk_safe_flush
> __printk_safe_flush
> raw_spin_lock_irqsave(&read_lock)
> // send NMI to other processors
> apic_send_IPI_allbutself(NMI_VECTOR)
> // NMI interrupt, dead loop
> crash_nmi_callback
> printk_safe_flush_on_panic
> printk_safe_flush
> __printk_safe_flush
> // deadlock
> raw_spin_lock_irqsave(&read_lock)
>
> DEADLOCK: read_lock is taken on CPU1 and will never get released.
>
> It happens when panic() stops a CPU by NMI while it has been in
> the middle of printk_safe_flush().
>
> Handle the lock the same way as logbuf_lock. The printk_safe buffers
> are flushed only when both locks can be safely taken. It can avoid
> the deadlock _in this particular case_ at expense of losing contents
> of printk_safe buffers.
>
> Note: It would actually be safe to re-init the locks when all CPUs were
> stopped by NMI. But it would require passing this information
> from arch-specific code. It is not worth the complexity.
> Especially because logbuf_lock and printk_safe buffers have been
> obsoleted by the lockless ring buffer.
>
> Fixes: cf9b1106c81c ("printk/nmi: flush NMI messages on the system panic")
> Signed-off-by: Muchun Song <songmuchun@bytedance.com>
> Reviewed-by: Petr Mladek <pmladek@suse.com>
> Cc: <stable@vger.kernel.org>
The patch is committed on printk/linux.git, branch for-5.12.
Best Regards,
Petr
prev parent reply other threads:[~2021-02-10 17:19 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-10 3:48 [PATCH v3] printk: fix deadlock when kernel panic Muchun Song
2021-02-10 6:33 ` Sergey Senozhatsky
2021-02-10 17:16 ` Petr Mladek [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YCQVC6oBKKjtM/yg@alley \
--to=pmladek@suse.com \
--cc=akpm@linux-foundation.org \
--cc=john.ogness@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=sergey.senozhatsky@gmail.com \
--cc=songmuchun@bytedance.com \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.