Re: [PATCH v4 0/4] printk: reduce deadlocks during panic

[Petr Mladek <pmladek@suse.com>]

On Wed 2022-02-02 09:18:17, Stephen Brennan wrote:
> When a caller writes heavily to the kernel log (e.g. writing to
> /dev/kmsg in a loop) while another panics, there's currently a high
> likelihood of a deadlock (see patch 2 for the full description of this
> deadlock).
> 
> The principle fix is to disable the optimistic spin once panic_cpu is
> set, so the panic CPU doesn't spin waiting for a halted CPU to hand over
> the console_sem.
> 
> However, this exposed us to a livelock situation, where the panic CPU
> holds the console_sem, and another CPU could fill up the log buffer
> faster than the consoles could drain it, preventing the panic from
> progressing and halting the other CPUs. To avoid this, patch 3 adds a
> mechanism to suppress printk (from non-panic-CPU) during panic, if we
> reach a threshold of dropped messages.
> 
> A major goal with all of these patches is to try to decrease the
> likelihood that another CPU is holding the console_sem when we halt it
> in panic(). This reduces the odds of needing to break locks and
> potentially encountering further deadlocks with the console drivers.
> 
> To test, I use the following script, kmsg_panic.sh:
> 
>     #!/bin/bash
>     date
>     # 991 chars (based on log buffer size):
>     chars="$(printf 'a%.0s' {1..991})"
>     while :; do
>         echo $chars > /dev/kmsg
>     done &
>     echo c > /proc/sysrq-trigger &
>     date
>     exit
> 
> I defined a hang as any time the system did not reboot to a login prompt
> on the serial console within 60 seconds. Here are the statistics on
> hangs using this script, before and after the patch.
> 
> before:  776 hangs / 1484 trials - 52.3%
> after :    0 hangs /  15k trials -  0.0%
> 
> Stephen Brennan (4):
>   printk: Add panic_in_progress helper
>   printk: disable optimistic spin during panic
>   printk: Avoid livelock with heavy printk during panic
>   printk: Drop console_sem during panic
> 
>  kernel/printk/printk.c | 55 +++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 54 insertions(+), 1 deletion(-)

For the entire patchset:

Reviewed-by: Petr Mladek <pmladek@suse.com>

It looks ready for linux-next from my POV. I am going to push it early
next week unless anyone complains in the meantime.

Best Regards,
Petr Mladek