From: Petr Mladek <pmladek@suse.com>
To: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Cc: linux-kernel@vger.kernel.org,
Steven Rostedt <rostedt@goodmis.org>,
Daniel Wang <wonderfly@google.com>,
Peter Zijlstra <peterz@infradead.org>,
Andrew Morton <akpm@linux-foundation.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Alan Cox <gnomes@lxorguk.ukuu.org.uk>,
Jiri Slaby <jslaby@suse.com>, Peter Feiner <pfeiner@google.com>,
linux-serial@vger.kernel.org,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Subject: Re: [PATCHv3] panic: avoid deadlocks in re-entrant console drivers
Date: Wed, 31 Oct 2018 13:27:36 +0100 [thread overview]
Message-ID: <20181031122736.epbxu424l3xagge5@pathway.suse.cz> (raw)
In-Reply-To: <20181025101036.6823-1-sergey.senozhatsky@gmail.com>
On Thu 2018-10-25 19:10:36, Sergey Senozhatsky wrote:
> >From printk()/serial console point of view panic() is special, because
> it may force CPU to re-enter printk() or/and serial console driver.
> Therefore, some of serial consoles drivers are re-entrant. E.g. 8250:
>
> serial8250_console_write()
> {
> if (port->sysrq)
> locked = 0;
> else if (oops_in_progress)
> locked = spin_trylock_irqsave(&port->lock, flags);
> else
> spin_lock_irqsave(&port->lock, flags);
> ...
> }
>
> panic() does set oops_in_progress via bust_spinlocks(1), so in theory
> we should be able to re-enter serial console driver from panic():
>
> CPU0
> <NMI>
> uart_console_write()
> serial8250_console_write() // if (oops_in_progress)
> // spin_trylock_irqsave()
> call_console_drivers()
> console_unlock()
> console_flush_on_panic()
> bust_spinlocks(1) // oops_in_progress++
> panic()
> <NMI/>
> spin_lock_irqsave(&port->lock, flags) // spin_lock_irqsave()
> serial8250_console_write()
> call_console_drivers()
> console_unlock()
> printk()
> ...
>
> However, this does not happen and we deadlock in serial console on
> port->lock spinlock. And the problem is that console_flush_on_panic()
> called after bust_spinlocks(0):
>
> void panic(const char *fmt, ...)
> {
> bust_spinlocks(1);
> ...
> bust_spinlocks(0);
> console_flush_on_panic();
> ...
> }
>
> bust_spinlocks(0) decrements oops_in_progress, so oops_in_progress
> can go back to zero. Thus even re-entrant console drivers will simply
> spin on port->lock spinlock. Given that port->lock may already be
> locked either by a stopped CPU, or by the very same CPU we execute
> panic() on (for instance, NMI panic() on printing CPU) the system
> deadlocks and does not reboot.
>
> Fix this by removing bust_spinlocks(0), so oops_in_progress is always
> set in panic() now and, thus, re-entrant console drivers will trylock
> the port->lock instead of spinning on it forever, when we call them
> from console_flush_on_panic().
>
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
The patch makes sense to me. The locks should stay busted also for
console_flush_on_panic().
With the added #include <linux/vt_kern.h>:
Reviewed-by: Petr Mladek <pmladek@suse.com>
Best Regards,
Petr
next prev parent reply other threads:[~2018-10-31 12:27 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-16 5:04 [RFC][PATCHv2 0/4] less deadlock prone serial consoles Sergey Senozhatsky
2018-10-16 5:04 ` [RFC][PATCHv2 1/4] panic: avoid deadlocks in re-entrant console drivers Sergey Senozhatsky
2018-10-17 4:48 ` Sergey Senozhatsky
2018-10-23 11:07 ` Petr Mladek
2018-10-23 11:54 ` Sergey Senozhatsky
2018-10-23 12:04 ` Sergey Senozhatsky
2018-10-23 12:12 ` Sergey Senozhatsky
2018-10-25 9:06 ` Petr Mladek
2018-10-25 9:31 ` Sergey Senozhatsky
2018-10-25 8:29 ` Petr Mladek
2018-10-25 9:05 ` Sergey Senozhatsky
2018-10-25 10:10 ` [PATCHv3] " Sergey Senozhatsky
2018-10-25 10:51 ` kbuild test robot
2018-10-25 11:56 ` Sergey Senozhatsky
2018-10-31 12:27 ` Petr Mladek [this message]
2018-11-01 1:48 ` Sergey Senozhatsky
2018-11-01 8:08 ` Petr Mladek
2018-11-22 13:12 ` Petr Mladek
2018-12-12 0:53 ` Daniel Wang
2018-12-12 5:23 ` Sergey Senozhatsky
2018-12-12 5:59 ` Daniel Wang
2018-12-12 6:06 ` Sergey Senozhatsky
2018-12-12 6:09 ` Daniel Wang
2018-10-16 5:04 ` [RFC][PATCHv2 2/4] printk: move printk_safe macros to printk header Sergey Senozhatsky
2018-10-16 7:27 ` Peter Zijlstra
2018-10-16 11:40 ` Petr Mladek
2018-10-16 12:17 ` Peter Zijlstra
2018-10-17 10:50 ` Petr Mladek
2018-10-17 14:00 ` Peter Zijlstra
2018-10-22 14:30 ` Petr Mladek
2018-10-16 12:27 ` Sergey Senozhatsky
2018-10-16 12:38 ` Peter Zijlstra
2018-10-16 12:54 ` Peter Zijlstra
2018-10-16 14:21 ` Peter Zijlstra
2018-10-17 4:32 ` Sergey Senozhatsky
2018-10-17 7:57 ` Peter Zijlstra
2018-10-17 13:36 ` Sergey Senozhatsky
2018-10-23 6:25 ` Sergey Senozhatsky
2018-10-16 5:04 ` [RFC][PATCHv2 3/4] serial: introduce uart_port locking helpers Sergey Senozhatsky
2018-12-08 3:12 ` Sergey Senozhatsky
2018-12-12 11:08 ` Greg Kroah-Hartman
2018-10-16 5:04 ` [RFC][PATCHv2 4/4] tty: 8250: switch to " Sergey Senozhatsky
2018-10-16 7:23 ` [RFC][PATCHv2 0/4] less deadlock prone serial consoles Peter Zijlstra
2018-10-16 8:12 ` Sergey Senozhatsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181031122736.epbxu424l3xagge5@pathway.suse.cz \
--to=pmladek@suse.com \
--cc=akpm@linux-foundation.org \
--cc=gnomes@lxorguk.ukuu.org.uk \
--cc=gregkh@linuxfoundation.org \
--cc=jslaby@suse.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-serial@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=pfeiner@google.com \
--cc=rostedt@goodmis.org \
--cc=sergey.senozhatsky.work@gmail.com \
--cc=sergey.senozhatsky@gmail.com \
--cc=torvalds@linux-foundation.org \
--cc=wonderfly@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).