[PATCH v2] kernel/printk: prevent deadlock at unexpected call kmsg_dump in NMI context

[PATCH v2] kernel/printk: prevent deadlock at unexpected call kmsg_dump in NMI context

[Konstantin Khlebnikov]

Kernel message dumper - function kmsg_dump() is called on various oops or
panic paths which could happen in unpredictable context including NMI.

Panic in NMI is handled especially by stopping all other cpus with
smp_send_stop() and busting locks in printk_safe_flush_on_panic().

Other less-fatal cases shouldn't happen in NMI and cannot be handled.
But this might happen for example on oops in nmi context. In this case
dumper could deadlock on lockbuf_lock or break internal structures.

This patch catches kmsg_dump() called in NMI context except panic and
prints warning once.

Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Link: https://lore.kernel.org/lkml/156294329676.1745.2620297516210526183.stgit@buzz/ (v1)
---
 kernel/printk/printk.c |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 1888f6a3b694..e711f64a1843 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -3104,6 +3104,13 @@ void kmsg_dump(enum kmsg_dump_reason reason)
 	struct kmsg_dumper *dumper;
 	unsigned long flags;
 
+	/*
+	 * In NMI context only panic could be handled safely:
+	 * it stops other cpus and busts logbuf lock.
+	 */
+	if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi()))
+		return;
+
 	if ((reason > KMSG_DUMP_OOPS) && !always_kmsg_dump)
 		return;
 

Re: [PATCH v2] kernel/printk: prevent deadlock at unexpected call kmsg_dump in NMI context

[Petr Mladek]

On Mon 2019-07-15 11:04:55, Konstantin Khlebnikov wrote:
> Kernel message dumper - function kmsg_dump() is called on various oops or
> panic paths which could happen in unpredictable context including NMI.
> 
> Panic in NMI is handled especially by stopping all other cpus with
> smp_send_stop() and busting locks in printk_safe_flush_on_panic().
> 
> Other less-fatal cases shouldn't happen in NMI and cannot be handled.
> But this might happen for example on oops in nmi context. In this case
> dumper could deadlock on lockbuf_lock or break internal structures.

If I get it correctly than this patch could really prevent a deadlock
in at least:

  + oops_end()
    + oops_exit()
      + kmsg_dump(KMSG_DUMP_OOPS)

If it is called in NMI, it should end up with panic(). Then the dump
will be called later after stopping CPUs...

Or am I wrong?

Otherwise, the patch looks good to me. I would just mention
the above scenario if it is correct.

Best Regards,
Petr

> This patch catches kmsg_dump() called in NMI context except panic and
> prints warning once.
> 
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> Link: https://lore.kernel.org/lkml/156294329676.1745.2620297516210526183.stgit@buzz/ (v1)
> ---
>  kernel/printk/printk.c |    7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index 1888f6a3b694..e711f64a1843 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -3104,6 +3104,13 @@ void kmsg_dump(enum kmsg_dump_reason reason)
>  	struct kmsg_dumper *dumper;
>  	unsigned long flags;
>  
> +	/*
> +	 * In NMI context only panic could be handled safely:
> +	 * it stops other cpus and busts logbuf lock.
> +	 */
> +	if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi()))
> +		return;
> +
>  	if ((reason > KMSG_DUMP_OOPS) && !always_kmsg_dump)
>  		return;
>  
> 

Re: [PATCH v2] kernel/printk: prevent deadlock at unexpected call kmsg_dump in NMI context

[Konstantin Khlebnikov]

On 16.07.2019 10:41, Petr Mladek wrote:
> On Mon 2019-07-15 11:04:55, Konstantin Khlebnikov wrote:
>> Kernel message dumper - function kmsg_dump() is called on various oops or
>> panic paths which could happen in unpredictable context including NMI.
>>
>> Panic in NMI is handled especially by stopping all other cpus with
>> smp_send_stop() and busting locks in printk_safe_flush_on_panic().
>>
>> Other less-fatal cases shouldn't happen in NMI and cannot be handled.
>> But this might happen for example on oops in nmi context. In this case
>> dumper could deadlock on lockbuf_lock or break internal structures.
> 
> If I get it correctly than this patch could really prevent a deadlock
> in at least:
> 
>    + oops_end()
>      + oops_exit()
>        + kmsg_dump(KMSG_DUMP_OOPS)
> 
> If it is called in NMI, it should end up with panic(). Then the dump
> will be called later after stopping CPUs...
> 
> Or am I wrong?

Yep. Under 'oops in nmi context' I mean exactly that case.

> 
> Otherwise, the patch looks good to me. I would just mention
> the above scenario if it is correct.
> 
> Best Regards,
> Petr
> 
>> This patch catches kmsg_dump() called in NMI context except panic and
>> prints warning once.
>>
>> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
>> Link: https://lore.kernel.org/lkml/156294329676.1745.2620297516210526183.stgit@buzz/ (v1)
>> ---
>>   kernel/printk/printk.c |    7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
>> index 1888f6a3b694..e711f64a1843 100644
>> --- a/kernel/printk/printk.c
>> +++ b/kernel/printk/printk.c
>> @@ -3104,6 +3104,13 @@ void kmsg_dump(enum kmsg_dump_reason reason)
>>   	struct kmsg_dumper *dumper;
>>   	unsigned long flags;
>>   
>> +	/*
>> +	 * In NMI context only panic could be handled safely:
>> +	 * it stops other cpus and busts logbuf lock.
>> +	 */
>> +	if (WARN_ON_ONCE(reason != KMSG_DUMP_PANIC && in_nmi()))
>> +		return;
>> +
>>   	if ((reason > KMSG_DUMP_OOPS) && !always_kmsg_dump)
>>   		return;
>>   
>>