linux-serial.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Petr Mladek <pmladek@suse.com>
To: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
Cc: linux-kernel@vger.kernel.org,
	Steven Rostedt <rostedt@goodmis.org>,
	Daniel Wang <wonderfly@google.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Alan Cox <gnomes@lxorguk.ukuu.org.uk>,
	Jiri Slaby <jslaby@suse.com>, Peter Feiner <pfeiner@google.com>,
	linux-serial@vger.kernel.org,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Subject: Re: [PATCHv3] panic: avoid deadlocks in re-entrant console drivers
Date: Wed, 31 Oct 2018 13:27:36 +0100	[thread overview]
Message-ID: <20181031122736.epbxu424l3xagge5@pathway.suse.cz> (raw)
In-Reply-To: <20181025101036.6823-1-sergey.senozhatsky@gmail.com>

On Thu 2018-10-25 19:10:36, Sergey Senozhatsky wrote:
> >From printk()/serial console point of view panic() is special, because
> it may force CPU to re-enter printk() or/and serial console driver.
> Therefore, some of serial consoles drivers are re-entrant. E.g. 8250:
> 
> serial8250_console_write()
> {
>         if (port->sysrq)
>                 locked = 0;
>         else if (oops_in_progress)
>                 locked = spin_trylock_irqsave(&port->lock, flags);
>         else
>                 spin_lock_irqsave(&port->lock, flags);
> 	...
> }
> 
> panic() does set oops_in_progress via bust_spinlocks(1), so in theory
> we should be able to re-enter serial console driver from panic():
> 
> 	CPU0
> 	<NMI>
> 	uart_console_write()
> 	serial8250_console_write()		// if (oops_in_progress)
> 						//    spin_trylock_irqsave()
> 	call_console_drivers()
> 	console_unlock()
> 	console_flush_on_panic()
> 	bust_spinlocks(1)			// oops_in_progress++
> 	panic()
> 	<NMI/>
> 	spin_lock_irqsave(&port->lock, flags)   // spin_lock_irqsave()
> 	serial8250_console_write()
> 	call_console_drivers()
> 	console_unlock()
> 	printk()
> 	...
> 
> However, this does not happen and we deadlock in serial console on
> port->lock spinlock. And the problem is that console_flush_on_panic()
> called after bust_spinlocks(0):
> 
> void panic(const char *fmt, ...)
> {
>         bust_spinlocks(1);
> ...
>         bust_spinlocks(0);
>         console_flush_on_panic();
> ...
> }
> 
> bust_spinlocks(0) decrements oops_in_progress, so oops_in_progress
> can go back to zero. Thus even re-entrant console drivers will simply
> spin on port->lock spinlock. Given that port->lock may already be
> locked either by a stopped CPU, or by the very same CPU we execute
> panic() on (for instance, NMI panic() on printing CPU) the system
> deadlocks and does not reboot.
> 
> Fix this by removing bust_spinlocks(0), so oops_in_progress is always
> set in panic() now and, thus, re-entrant console drivers will trylock
> the port->lock instead of spinning on it forever, when we call them
> from console_flush_on_panic().
> 
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>

The patch makes sense to me. The locks should stay busted also for
console_flush_on_panic().

With the added #include <linux/vt_kern.h>:

Reviewed-by: Petr Mladek <pmladek@suse.com>

Best Regards,
Petr

  parent reply	other threads:[~2018-10-31 12:27 UTC|newest]

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-10-16  5:04 [RFC][PATCHv2 0/4] less deadlock prone serial consoles Sergey Senozhatsky
2018-10-16  5:04 ` [RFC][PATCHv2 1/4] panic: avoid deadlocks in re-entrant console drivers Sergey Senozhatsky
2018-10-17  4:48   ` Sergey Senozhatsky
2018-10-23 11:07   ` Petr Mladek
2018-10-23 11:54     ` Sergey Senozhatsky
2018-10-23 12:04       ` Sergey Senozhatsky
2018-10-23 12:12         ` Sergey Senozhatsky
2018-10-25  9:06           ` Petr Mladek
2018-10-25  9:31             ` Sergey Senozhatsky
2018-10-25  8:29       ` Petr Mladek
2018-10-25  9:05         ` Sergey Senozhatsky
2018-10-25 10:10   ` [PATCHv3] " Sergey Senozhatsky
2018-10-25 10:51     ` kbuild test robot
2018-10-25 11:56       ` Sergey Senozhatsky
2018-10-31 12:27     ` Petr Mladek [this message]
2018-11-01  1:48       ` Sergey Senozhatsky
2018-11-01  8:08         ` Petr Mladek
2018-11-22 13:12           ` Petr Mladek
2018-12-12  0:53             ` Daniel Wang
2018-12-12  5:23               ` Sergey Senozhatsky
2018-12-12  5:59                 ` Daniel Wang
2018-12-12  6:06                   ` Sergey Senozhatsky
2018-12-12  6:09                     ` Daniel Wang
2018-10-16  5:04 ` [RFC][PATCHv2 2/4] printk: move printk_safe macros to printk header Sergey Senozhatsky
2018-10-16  7:27   ` Peter Zijlstra
2018-10-16 11:40     ` Petr Mladek
2018-10-16 12:17       ` Peter Zijlstra
2018-10-17 10:50         ` Petr Mladek
2018-10-17 14:00           ` Peter Zijlstra
2018-10-22 14:30             ` Petr Mladek
2018-10-16 12:27     ` Sergey Senozhatsky
2018-10-16 12:38       ` Peter Zijlstra
2018-10-16 12:54       ` Peter Zijlstra
2018-10-16 14:21         ` Peter Zijlstra
2018-10-17  4:32         ` Sergey Senozhatsky
2018-10-17  7:57           ` Peter Zijlstra
2018-10-17 13:36             ` Sergey Senozhatsky
2018-10-23  6:25         ` Sergey Senozhatsky
2018-10-16  5:04 ` [RFC][PATCHv2 3/4] serial: introduce uart_port locking helpers Sergey Senozhatsky
2018-12-08  3:12   ` Sergey Senozhatsky
2018-12-12 11:08     ` Greg Kroah-Hartman
2018-10-16  5:04 ` [RFC][PATCHv2 4/4] tty: 8250: switch to " Sergey Senozhatsky
2018-10-16  7:23 ` [RFC][PATCHv2 0/4] less deadlock prone serial consoles Peter Zijlstra
2018-10-16  8:12   ` Sergey Senozhatsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20181031122736.epbxu424l3xagge5@pathway.suse.cz \
    --to=pmladek@suse.com \
    --cc=akpm@linux-foundation.org \
    --cc=gnomes@lxorguk.ukuu.org.uk \
    --cc=gregkh@linuxfoundation.org \
    --cc=jslaby@suse.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-serial@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=pfeiner@google.com \
    --cc=rostedt@goodmis.org \
    --cc=sergey.senozhatsky.work@gmail.com \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=torvalds@linux-foundation.org \
    --cc=wonderfly@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).