From: Petr Mladek <pmladek@suse.com>
To: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Cc: linux-kernel@vger.kernel.org,
Steven Rostedt <rostedt@goodmis.org>,
Daniel Wang <wonderfly@google.com>,
Peter Zijlstra <peterz@infradead.org>,
Andrew Morton <akpm@linux-foundation.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Alan Cox <gnomes@lxorguk.ukuu.org.uk>,
Jiri Slaby <jslaby@suse.com>, Peter Feiner <pfeiner@google.com>,
linux-serial@vger.kernel.org,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Subject: Re: [RFC][PATCHv2 1/4] panic: avoid deadlocks in re-entrant console drivers
Date: Tue, 23 Oct 2018 13:07:51 +0200 [thread overview]
Message-ID: <20181023110751.un2u67bc7dpo4ska@pathway.suse.cz> (raw)
In-Reply-To: <20181016050428.17966-2-sergey.senozhatsky@gmail.com>
On Tue 2018-10-16 14:04:25, Sergey Senozhatsky wrote:
> >From printk()/serial console point of view panic() is special, because
> it may force CPU to re-enter printk() or/and serial console driver.
> Therefore, some of serial consoles drivers are re-entrant. E.g. 8250:
>
> serial8250_console_write()
> {
> if (port->sysrq)
> locked = 0;
> else if (oops_in_progress)
> locked = spin_trylock_irqsave(&port->lock, flags);
> else
> spin_lock_irqsave(&port->lock, flags);
> ...
> }
>
> panic() does set oops_in_progress via bust_spinlocks(1), so in theory
> we should be able to re-enter serial console driver from panic():
>
> CPU0
> <NMI>
> uart_console_write()
> serial8250_console_write() // if (oops_in_progress)
> // spin_trylock_irqsave()
> call_console_drivers()
> console_unlock()
> console_flush_on_panic()
> bust_spinlocks(1) // oops_in_progress++
> panic()
> <NMI/>
> spin_lock_irqsave(&port->lock, flags) // spin_lock_irqsave()
> serial8250_console_write()
> call_console_drivers()
> console_unlock()
> printk()
> ...
>
> However, this does not happen and we deadlock in serial console on
> port->lock spinlock. And the problem is that console_flush_on_panic()
> called after bust_spinlocks(0):
>
> void panic(const char *fmt, ...)
> {
> bust_spinlocks(1);
> ...
> bust_spinlocks(0);
> console_flush_on_panic();
> ...
> }
>
> bust_spinlocks(0) decrements oops_in_progress, so oops_in_progress
> can go back to zero. Thus even re-entrant console drivers will simply
> spin on port->lock spinlock. Given that port->lock may already be
> locked either by a stopped CPU, or by the very same CPU we execute
> panic() on (for instance, NMI panic() on printing CPU) the system
> deadlocks and does not reboot.
The idea makes sense to me. You are right that we already called
printk/console with busted spinlock many times in panic().
Therefore it should not be worse.
> Fix this by setting oops_in_progress before console_flush_on_panic(),
> so re-entrant console drivers will trylock the port->lock instead of
> spinning on it forever.
>
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
> ---
> kernel/panic.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/kernel/panic.c b/kernel/panic.c
> index f6d549a29a5c..a0e60ccf3031 100644
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -237,7 +237,13 @@ void panic(const char *fmt, ...)
> if (_crash_kexec_post_notifiers)
> __crash_kexec(NULL);
>
> + /*
> + * Decrement oops_in_progress and let bust_spinlocks() to
> + * unblank_screen(), console_unblank() and wake_up_klogd()
> + */
> bust_spinlocks(0);
> + /* Set oops_in_progress, so we can reenter serial console driver */
> + bust_spinlocks(1);
Though this looks a bit weird.
I have just realized that console_unblank() is called by
bust_spinlocks(0) and does basically the same as
console_flush_on_panic(). Also it does not make much
sense wake_up_klogd() there. Finally, it seems to be
too late to disable lockdep there.
I would suggest something like:
diff --git a/kernel/panic.c b/kernel/panic.c
index 8b2e002d52eb..c78e3df8dd58 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -233,17 +233,14 @@ void panic(const char *fmt, ...)
if (_crash_kexec_post_notifiers)
__crash_kexec(NULL);
- bust_spinlocks(0);
-
/*
* We may have ended up stopping the CPU holding the lock (in
- * smp_send_stop()) while still having some valuable data in the console
- * buffer. Try to acquire the lock then release it regardless of the
- * result. The release will also print the buffers out. Locks debug
- * should be disabled to avoid reporting bad unlock balance when
- * panic() is not being callled from OOPS.
+ * smp_send_stop()) while still having some valuable data in
+ * the console buffer. Try hard to see them.
*/
- debug_locks_off();
+#ifdef CONFIG_VT
+ unblank_screen();
+#endif
console_flush_on_panic();
if (!panic_blink)
diff --git a/lib/bust_spinlocks.c b/lib/bust_spinlocks.c
index ab719495e2cb..e42d2fcd6453 100644
--- a/lib/bust_spinlocks.c
+++ b/lib/bust_spinlocks.c
@@ -20,6 +20,12 @@
void __attribute__((weak)) bust_spinlocks(int yes)
{
if (yes) {
+ /*
+ * Some locks might get ignored in the Oops situation
+ * to get an important work done. Locks debug should
+ * be disabled to avoid reporting bad unlock balance.
+ */
+ debug_locks_off();
++oops_in_progress;
} else {
#ifdef CONFIG_VT
next prev parent reply other threads:[~2018-10-23 11:07 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-16 5:04 [RFC][PATCHv2 0/4] less deadlock prone serial consoles Sergey Senozhatsky
2018-10-16 5:04 ` [RFC][PATCHv2 1/4] panic: avoid deadlocks in re-entrant console drivers Sergey Senozhatsky
2018-10-17 4:48 ` Sergey Senozhatsky
2018-10-23 11:07 ` Petr Mladek [this message]
2018-10-23 11:54 ` Sergey Senozhatsky
2018-10-23 12:04 ` Sergey Senozhatsky
2018-10-23 12:12 ` Sergey Senozhatsky
2018-10-25 9:06 ` Petr Mladek
2018-10-25 9:31 ` Sergey Senozhatsky
2018-10-25 8:29 ` Petr Mladek
2018-10-25 9:05 ` Sergey Senozhatsky
2018-10-25 10:10 ` [PATCHv3] " Sergey Senozhatsky
2018-10-25 10:51 ` kbuild test robot
2018-10-25 11:56 ` Sergey Senozhatsky
2018-10-31 12:27 ` Petr Mladek
2018-11-01 1:48 ` Sergey Senozhatsky
2018-11-01 8:08 ` Petr Mladek
2018-11-22 13:12 ` Petr Mladek
2018-12-12 0:53 ` Daniel Wang
2018-12-12 5:23 ` Sergey Senozhatsky
2018-12-12 5:59 ` Daniel Wang
2018-12-12 6:06 ` Sergey Senozhatsky
2018-12-12 6:09 ` Daniel Wang
2018-10-16 5:04 ` [RFC][PATCHv2 2/4] printk: move printk_safe macros to printk header Sergey Senozhatsky
2018-10-16 7:27 ` Peter Zijlstra
2018-10-16 11:40 ` Petr Mladek
2018-10-16 12:17 ` Peter Zijlstra
2018-10-17 10:50 ` Petr Mladek
2018-10-17 14:00 ` Peter Zijlstra
2018-10-22 14:30 ` Petr Mladek
2018-10-16 12:27 ` Sergey Senozhatsky
2018-10-16 12:38 ` Peter Zijlstra
2018-10-16 12:54 ` Peter Zijlstra
2018-10-16 14:21 ` Peter Zijlstra
2018-10-17 4:32 ` Sergey Senozhatsky
2018-10-17 7:57 ` Peter Zijlstra
2018-10-17 13:36 ` Sergey Senozhatsky
2018-10-23 6:25 ` Sergey Senozhatsky
2018-10-16 5:04 ` [RFC][PATCHv2 3/4] serial: introduce uart_port locking helpers Sergey Senozhatsky
2018-12-08 3:12 ` Sergey Senozhatsky
2018-12-12 11:08 ` Greg Kroah-Hartman
2018-10-16 5:04 ` [RFC][PATCHv2 4/4] tty: 8250: switch to " Sergey Senozhatsky
2018-10-16 7:23 ` [RFC][PATCHv2 0/4] less deadlock prone serial consoles Peter Zijlstra
2018-10-16 8:12 ` Sergey Senozhatsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181023110751.un2u67bc7dpo4ska@pathway.suse.cz \
--to=pmladek@suse.com \
--cc=akpm@linux-foundation.org \
--cc=gnomes@lxorguk.ukuu.org.uk \
--cc=gregkh@linuxfoundation.org \
--cc=jslaby@suse.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-serial@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=pfeiner@google.com \
--cc=rostedt@goodmis.org \
--cc=sergey.senozhatsky.work@gmail.com \
--cc=sergey.senozhatsky@gmail.com \
--cc=torvalds@linux-foundation.org \
--cc=wonderfly@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).