[PATCH v2 0/4] printk: reduce deadlocks during panic

* [PATCH v2 0/4] printk: reduce deadlocks during panic
@ 2022-01-26 23:02 Stephen Brennan
  2022-01-26 23:02 ` [PATCH v2 1/4] panic: Add panic_in_progress helper Stephen Brennan
                   ` (3 more replies)
  0 siblings, 4 replies; 14+ messages in thread
From: Stephen Brennan @ 2022-01-26 23:02 UTC (permalink / raw)
  To: Andrew Morton, Sergey Senozhatsky, Petr Mladek
  Cc: Arnd Bergmann, Steven Rostedt, Stephen Brennan,
	Sebastian Reichel, John Ogness, Andy Shevchenko,
	Luis Chamberlain, linux-kernel

When a caller writes heavily to the kernel log (e.g. writing to
/dev/kmsg in a loop) while another panics, there's currently a high
likelihood of a deadlock (see patch 2 for the full description of this
deadlock).

The principle fix is to disable the optimistic spin once panic_cpu is
set, so the panic CPU doesn't spin waiting for a halted CPU to hand over
the console_sem.

However, this exposed us to a livelock situation, where the panic CPU
holds the console_sem, and another CPU could fill up the log buffer
faster than the consoles could drain it, preventing the panic from
progressing and halting the other CPUs. To avoid this, patch 3 adds a
mechanism to suppress printk (from non-panic-CPU) during panic, if we
reach a threshold of dropped messages.

A major goal with all of these patches is to try to decrease the
likelihood that another CPU is holding the console_sem when we halt it
in panic(). This reduces the odds of needing to break locks and
potentially encountering further deadlocks with the console drivers.

To test, I use the following script, kmsg_panic.sh:

    #!/bin/bash
    date
    # 991 chars (based on log buffer size):
    chars="$(printf 'a%.0s' {1..991})"
    while :; do
        echo $chars > /dev/kmsg
    done &
    echo c > /proc/sysrq-trigger &
    date
    exit

I defined a hang as any time the system did not reboot to a login prompt
on the serial console within 60 seconds. Here are the statistics on
hangs using this script, before and after the patch.

before:  776 hangs / 1484 trials - 52.3%
after :    0 hangs /  408 trials -  0.0%

v2:
    Each patch has minor updates from code reviews. I've re-done testing and
    updated the above statistics. Exact changes are in each patch.

Stephen Brennan (4):
  panic: Add panic_in_progress helper
  printk: disable optimistic spin during panic
  printk: Avoid livelock with heavy printk during panic
  printk: Drop console_sem during panic

 include/linux/panic.h  |  6 +++++
 kernel/printk/printk.c | 51 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 56 insertions(+), 1 deletion(-)

-- 
2.30.2

^ permalink raw reply	[flat|nested] 14+ messages in thread