* printk deadlock due to double lock attempt on current CPU's runqueue
@ 2021-11-09 20:06 ` Sultan Alsawaf
0 siblings, 0 replies; 29+ messages in thread
From: Sultan Alsawaf @ 2021-11-09 20:06 UTC (permalink / raw)
Cc: Anton Vorontsov, Ben Segall, Colin Cross,
Daniel Bristot de Oliveira, Daniel Vetter, David Airlie,
Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness,
Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst,
Maxime Ripard, Mel Gorman, Peter Zijlstra, Petr Mladek,
Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck,
Vincent Guittot
Hi,
I encountered a printk deadlock on 5.13 which appears to still affect the latest
kernel. The deadlock occurs due to printk being used while having the current
CPU's runqueue locked, and the underlying framebuffer console attempting to lock
the same runqueue when printk tries to flush the log buffer.
Here's the backtrace corresponding to the deadlock, with some notes I've added:
-----------------------8<-----------------------
PID: 0 TASK: ffff888100d832c0 CPU: 11 COMMAND: "swapper/11"
[exception RIP: native_halt+22]
RIP: ffffffff8320eb97 RSP: ffffc900005b8540 RFLAGS: 00000002
RAX: 0000000000000001 RBX: ffff88821c534280 RCX: ffffed10438a6850
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff88821c534280
RBP: 0000000000000003 R8: 0000000000000001 R9: ffffffff81302973
R10: ffff88821c534280 R11: ffffed10438a6850 R12: ffff88821c535080
R13: ffff88825c0b32c0 R14: 0000000000000000 R15: ffff88821c534280
CS: 0010 SS: 0018
#0 [ffffc900005b8540] kvm_wait at ffffffff81125cc1
#1 [ffffc900005b8558] pv_wait_head_or_lock at ffffffff8130294a
#2 [ffffc900005b8580] __pv_queued_spin_lock_slowpath at ffffffff81302d2e
#3 [ffffc900005b8600] do_raw_spin_lock at ffffffff81303a33
#4 [ffffc900005b8668] _raw_spin_lock at ffffffff8320f118 <-- ***DEADLOCK***
#5 [ffffc900005b8680] ttwu_queue at ffffffff8125cff9 <-- tries to lock this CPU's runqueue again...
#6 [ffffc900005b86f8] try_to_wake_up at ffffffff8125d778
#7 [ffffc900005b8780] wake_up_process at ffffffff8125d924
#8 [ffffc900005b8788] wake_up_worker at ffffffff8121268b
#9 [ffffc900005b8798] insert_work at ffffffff812199aa
#10 [ffffc900005b87d0] __queue_work at ffffffff8121dcde
#11 [ffffc900005b8810] queue_work_on at ffffffff8121e1ca
#12 [ffffc900005b8838] drm_fb_helper_damage at ffffffffc079b0a0 [drm_kms_helper]
#13 [ffffc900005b8878] drm_fb_helper_sys_imageblit at ffffffffc079b613 [drm_kms_helper]
#14 [ffffc900005b88a8] drm_fbdev_fb_imageblit at ffffffffc079b9e1 [drm_kms_helper]
#15 [ffffc900005b88c0] soft_cursor at ffffffff8236ffc9
#16 [ffffc900005b8928] bit_cursor at ffffffff8236f207
#17 [ffffc900005b8a50] fbcon_cursor at ffffffff82362c0f
#18 [ffffc900005b8a88] hide_cursor at ffffffff8253be1c
#19 [ffffc900005b8aa0] vt_console_print at ffffffff82543094
#20 [ffffc900005b8af0] call_console_drivers at ffffffff81319b32
#21 [ffffc900005b8b30] console_unlock at ffffffff8131bd50
#22 [ffffc900005b8c68] vprintk_emit at ffffffff8131e0f5
#23 [ffffc900005b8ca8] vprintk_default at ffffffff8131e4ff
#24 [ffffc900005b8cb0] vprintk at ffffffff8131eee6
#25 [ffffc900005b8cd0] printk at ffffffff81318ce0
#26 [ffffc900005b8d78] __warn_printk at ffffffff811c7b9d
#27 [ffffc900005b8e28] enqueue_task_fair at ffffffff8129774a <-- SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list);
#28 [ffffc900005b8ec0] activate_task at ffffffff8125625d
#29 [ffffc900005b8ef0] ttwu_do_activate at ffffffff81257943
#30 [ffffc900005b8f28] sched_ttwu_pending at ffffffff8125c71f <-- locks this CPU's runqueue
#31 [ffffc900005b8fa0] flush_smp_call_function_queue at ffffffff813c6833
#32 [ffffc900005b8fd8] generic_smp_call_function_single_interrupt at ffffffff813c7f58
#33 [ffffc900005b8fe0] __sysvec_call_function_single at ffffffff810f1456
#34 [ffffc900005b8ff0] sysvec_call_function_single at ffffffff831ec1bc
--- <IRQ stack> ---
#35 [ffffc9000019fda8] sysvec_call_function_single at ffffffff831ec1bc
RIP: ffffffff831ed06e RSP: ffffed10438a6a49 RFLAGS: 00000001
RAX: ffff888100d832c0 RBX: 0000000000000000 RCX: 1ffff92000033fd7
RDX: 0000000000000000 RSI: ffff888100d832c0 RDI: ffffed10438a6a49
RBP: ffffffff831ec166 R8: dffffc0000000000 R9: 0000000000000000
R10: ffffffff83400e22 R11: 0000000000000000 R12: ffffffff831ed83e
R13: 0000000000000000 R14: ffffc9000019fde8 R15: ffffffff814d4d9d
ORIG_RAX: ffff88821c53524b CS: 0001 SS: ef073a2
WARNING: possibly bogus exception frame
----------------------->8-----------------------
The catalyst is that CONFIG_SCHED_DEBUG is enabled and the tmp_alone_branch
assertion fails (Peter, is this bad?). The current CPU's runqueue is locked when
the warning is printed, and then drm_fb_helper_damage() triggers the deadlock
when it tries to schedule a worker and thus attempts to re-lock the runqueue.
I'm not sure what the *correct* solution is here (don't use printk while having
a runqueue locked? don't use schedule_work() from the fbcon path? tell printk
to use one of its lock-less backends?), so I've cc'd all the relevant folks.
Thanks,
Sultan
^ permalink raw reply [flat|nested] 29+ messages in thread* printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-09 20:06 ` Sultan Alsawaf 0 siblings, 0 replies; 29+ messages in thread From: Sultan Alsawaf @ 2021-11-09 20:06 UTC (permalink / raw) Cc: Juri Lelli, David Airlie, dri-devel, Ben Segall, Vincent Guittot, Anton Vorontsov, Peter Zijlstra, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, John Ogness, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira Hi, I encountered a printk deadlock on 5.13 which appears to still affect the latest kernel. The deadlock occurs due to printk being used while having the current CPU's runqueue locked, and the underlying framebuffer console attempting to lock the same runqueue when printk tries to flush the log buffer. Here's the backtrace corresponding to the deadlock, with some notes I've added: -----------------------8<----------------------- PID: 0 TASK: ffff888100d832c0 CPU: 11 COMMAND: "swapper/11" [exception RIP: native_halt+22] RIP: ffffffff8320eb97 RSP: ffffc900005b8540 RFLAGS: 00000002 RAX: 0000000000000001 RBX: ffff88821c534280 RCX: ffffed10438a6850 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff88821c534280 RBP: 0000000000000003 R8: 0000000000000001 R9: ffffffff81302973 R10: ffff88821c534280 R11: ffffed10438a6850 R12: ffff88821c535080 R13: ffff88825c0b32c0 R14: 0000000000000000 R15: ffff88821c534280 CS: 0010 SS: 0018 #0 [ffffc900005b8540] kvm_wait at ffffffff81125cc1 #1 [ffffc900005b8558] pv_wait_head_or_lock at ffffffff8130294a #2 [ffffc900005b8580] __pv_queued_spin_lock_slowpath at ffffffff81302d2e #3 [ffffc900005b8600] do_raw_spin_lock at ffffffff81303a33 #4 [ffffc900005b8668] _raw_spin_lock at ffffffff8320f118 <-- ***DEADLOCK*** #5 [ffffc900005b8680] ttwu_queue at ffffffff8125cff9 <-- tries to lock this CPU's runqueue again... #6 [ffffc900005b86f8] try_to_wake_up at ffffffff8125d778 #7 [ffffc900005b8780] wake_up_process at ffffffff8125d924 #8 [ffffc900005b8788] wake_up_worker at ffffffff8121268b #9 [ffffc900005b8798] insert_work at ffffffff812199aa #10 [ffffc900005b87d0] __queue_work at ffffffff8121dcde #11 [ffffc900005b8810] queue_work_on at ffffffff8121e1ca #12 [ffffc900005b8838] drm_fb_helper_damage at ffffffffc079b0a0 [drm_kms_helper] #13 [ffffc900005b8878] drm_fb_helper_sys_imageblit at ffffffffc079b613 [drm_kms_helper] #14 [ffffc900005b88a8] drm_fbdev_fb_imageblit at ffffffffc079b9e1 [drm_kms_helper] #15 [ffffc900005b88c0] soft_cursor at ffffffff8236ffc9 #16 [ffffc900005b8928] bit_cursor at ffffffff8236f207 #17 [ffffc900005b8a50] fbcon_cursor at ffffffff82362c0f #18 [ffffc900005b8a88] hide_cursor at ffffffff8253be1c #19 [ffffc900005b8aa0] vt_console_print at ffffffff82543094 #20 [ffffc900005b8af0] call_console_drivers at ffffffff81319b32 #21 [ffffc900005b8b30] console_unlock at ffffffff8131bd50 #22 [ffffc900005b8c68] vprintk_emit at ffffffff8131e0f5 #23 [ffffc900005b8ca8] vprintk_default at ffffffff8131e4ff #24 [ffffc900005b8cb0] vprintk at ffffffff8131eee6 #25 [ffffc900005b8cd0] printk at ffffffff81318ce0 #26 [ffffc900005b8d78] __warn_printk at ffffffff811c7b9d #27 [ffffc900005b8e28] enqueue_task_fair at ffffffff8129774a <-- SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); #28 [ffffc900005b8ec0] activate_task at ffffffff8125625d #29 [ffffc900005b8ef0] ttwu_do_activate at ffffffff81257943 #30 [ffffc900005b8f28] sched_ttwu_pending at ffffffff8125c71f <-- locks this CPU's runqueue #31 [ffffc900005b8fa0] flush_smp_call_function_queue at ffffffff813c6833 #32 [ffffc900005b8fd8] generic_smp_call_function_single_interrupt at ffffffff813c7f58 #33 [ffffc900005b8fe0] __sysvec_call_function_single at ffffffff810f1456 #34 [ffffc900005b8ff0] sysvec_call_function_single at ffffffff831ec1bc --- <IRQ stack> --- #35 [ffffc9000019fda8] sysvec_call_function_single at ffffffff831ec1bc RIP: ffffffff831ed06e RSP: ffffed10438a6a49 RFLAGS: 00000001 RAX: ffff888100d832c0 RBX: 0000000000000000 RCX: 1ffff92000033fd7 RDX: 0000000000000000 RSI: ffff888100d832c0 RDI: ffffed10438a6a49 RBP: ffffffff831ec166 R8: dffffc0000000000 R9: 0000000000000000 R10: ffffffff83400e22 R11: 0000000000000000 R12: ffffffff831ed83e R13: 0000000000000000 R14: ffffc9000019fde8 R15: ffffffff814d4d9d ORIG_RAX: ffff88821c53524b CS: 0001 SS: ef073a2 WARNING: possibly bogus exception frame ----------------------->8----------------------- The catalyst is that CONFIG_SCHED_DEBUG is enabled and the tmp_alone_branch assertion fails (Peter, is this bad?). The current CPU's runqueue is locked when the warning is printed, and then drm_fb_helper_damage() triggers the deadlock when it tries to schedule a worker and thus attempts to re-lock the runqueue. I'm not sure what the *correct* solution is here (don't use printk while having a runqueue locked? don't use schedule_work() from the fbcon path? tell printk to use one of its lock-less backends?), so I've cc'd all the relevant folks. Thanks, Sultan ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-09 20:06 ` Sultan Alsawaf @ 2021-11-09 21:38 ` Peter Zijlstra -1 siblings, 0 replies; 29+ messages in thread From: Peter Zijlstra @ 2021-11-09 21:38 UTC (permalink / raw) To: Sultan Alsawaf Cc: Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, Daniel Vetter, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, Vincent Guittot, mkoutny On Tue, Nov 09, 2021 at 12:06:48PM -0800, Sultan Alsawaf wrote: > Hi, > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > kernel. The deadlock occurs due to printk being used while having the current > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > the same runqueue when printk tries to flush the log buffer. Yes, that's a known 'feature' of some consoles. printk() is in the process of being reworked to not call con->write() from the printk() calling context, which would go a long way towards fixing this. > #27 [ffffc900005b8e28] enqueue_task_fair at ffffffff8129774a <-- SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); > #28 [ffffc900005b8ec0] activate_task at ffffffff8125625d > #29 [ffffc900005b8ef0] ttwu_do_activate at ffffffff81257943 > #30 [ffffc900005b8f28] sched_ttwu_pending at ffffffff8125c71f <-- locks this CPU's runqueue > #31 [ffffc900005b8fa0] flush_smp_call_function_queue at ffffffff813c6833 > #32 [ffffc900005b8fd8] generic_smp_call_function_single_interrupt at ffffffff813c7f58 > #33 [ffffc900005b8fe0] __sysvec_call_function_single at ffffffff810f1456 > #34 [ffffc900005b8ff0] sysvec_call_function_single at ffffffff831ec1bc > --- <IRQ stack> --- > #35 [ffffc9000019fda8] sysvec_call_function_single at ffffffff831ec1bc > RIP: ffffffff831ed06e RSP: ffffed10438a6a49 RFLAGS: 00000001 > RAX: ffff888100d832c0 RBX: 0000000000000000 RCX: 1ffff92000033fd7 > RDX: 0000000000000000 RSI: ffff888100d832c0 RDI: ffffed10438a6a49 > RBP: ffffffff831ec166 R8: dffffc0000000000 R9: 0000000000000000 > R10: ffffffff83400e22 R11: 0000000000000000 R12: ffffffff831ed83e > R13: 0000000000000000 R14: ffffc9000019fde8 R15: ffffffff814d4d9d > ORIG_RAX: ffff88821c53524b CS: 0001 SS: ef073a2 > WARNING: possibly bogus exception frame > ----------------------->8----------------------- > > The catalyst is that CONFIG_SCHED_DEBUG is enabled and the tmp_alone_branch > assertion fails (Peter, is this bad?). Yes, that's not good. IIRC Vincent and Michal were looking at that code recently. > I'm not sure what the *correct* solution is here (don't use printk while having > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > to use one of its lock-less backends?), so I've cc'd all the relevant folks. I'm a firm believer in early_printk serial consoles. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-09 21:38 ` Peter Zijlstra 0 siblings, 0 replies; 29+ messages in thread From: Peter Zijlstra @ 2021-11-09 21:38 UTC (permalink / raw) To: Sultan Alsawaf Cc: Juri Lelli, David Airlie, dri-devel, Ben Segall, Vincent Guittot, Anton Vorontsov, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, John Ogness, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, mkoutny, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Tue, Nov 09, 2021 at 12:06:48PM -0800, Sultan Alsawaf wrote: > Hi, > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > kernel. The deadlock occurs due to printk being used while having the current > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > the same runqueue when printk tries to flush the log buffer. Yes, that's a known 'feature' of some consoles. printk() is in the process of being reworked to not call con->write() from the printk() calling context, which would go a long way towards fixing this. > #27 [ffffc900005b8e28] enqueue_task_fair at ffffffff8129774a <-- SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); > #28 [ffffc900005b8ec0] activate_task at ffffffff8125625d > #29 [ffffc900005b8ef0] ttwu_do_activate at ffffffff81257943 > #30 [ffffc900005b8f28] sched_ttwu_pending at ffffffff8125c71f <-- locks this CPU's runqueue > #31 [ffffc900005b8fa0] flush_smp_call_function_queue at ffffffff813c6833 > #32 [ffffc900005b8fd8] generic_smp_call_function_single_interrupt at ffffffff813c7f58 > #33 [ffffc900005b8fe0] __sysvec_call_function_single at ffffffff810f1456 > #34 [ffffc900005b8ff0] sysvec_call_function_single at ffffffff831ec1bc > --- <IRQ stack> --- > #35 [ffffc9000019fda8] sysvec_call_function_single at ffffffff831ec1bc > RIP: ffffffff831ed06e RSP: ffffed10438a6a49 RFLAGS: 00000001 > RAX: ffff888100d832c0 RBX: 0000000000000000 RCX: 1ffff92000033fd7 > RDX: 0000000000000000 RSI: ffff888100d832c0 RDI: ffffed10438a6a49 > RBP: ffffffff831ec166 R8: dffffc0000000000 R9: 0000000000000000 > R10: ffffffff83400e22 R11: 0000000000000000 R12: ffffffff831ed83e > R13: 0000000000000000 R14: ffffc9000019fde8 R15: ffffffff814d4d9d > ORIG_RAX: ffff88821c53524b CS: 0001 SS: ef073a2 > WARNING: possibly bogus exception frame > ----------------------->8----------------------- > > The catalyst is that CONFIG_SCHED_DEBUG is enabled and the tmp_alone_branch > assertion fails (Peter, is this bad?). Yes, that's not good. IIRC Vincent and Michal were looking at that code recently. > I'm not sure what the *correct* solution is here (don't use printk while having > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > to use one of its lock-less backends?), so I've cc'd all the relevant folks. I'm a firm believer in early_printk serial consoles. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-09 21:38 ` Peter Zijlstra @ 2021-11-10 9:00 ` Vincent Guittot -1 siblings, 0 replies; 29+ messages in thread From: Vincent Guittot @ 2021-11-10 9:00 UTC (permalink / raw) To: Peter Zijlstra Cc: Juri Lelli, David Airlie, dri-devel, Ben Segall, Sultan Alsawaf, Anton Vorontsov, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, John Ogness, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, mkoutny, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Tue, 9 Nov 2021 at 22:38, Peter Zijlstra <peterz@infradead.org> wrote: > > On Tue, Nov 09, 2021 at 12:06:48PM -0800, Sultan Alsawaf wrote: > > Hi, > > > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > > kernel. The deadlock occurs due to printk being used while having the current > > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > > the same runqueue when printk tries to flush the log buffer. > > Yes, that's a known 'feature' of some consoles. printk() is in the > process of being reworked to not call con->write() from the printk() > calling context, which would go a long way towards fixing this. > > > #27 [ffffc900005b8e28] enqueue_task_fair at ffffffff8129774a <-- SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); > > #28 [ffffc900005b8ec0] activate_task at ffffffff8125625d > > #29 [ffffc900005b8ef0] ttwu_do_activate at ffffffff81257943 > > #30 [ffffc900005b8f28] sched_ttwu_pending at ffffffff8125c71f <-- locks this CPU's runqueue > > #31 [ffffc900005b8fa0] flush_smp_call_function_queue at ffffffff813c6833 > > #32 [ffffc900005b8fd8] generic_smp_call_function_single_interrupt at ffffffff813c7f58 > > #33 [ffffc900005b8fe0] __sysvec_call_function_single at ffffffff810f1456 > > #34 [ffffc900005b8ff0] sysvec_call_function_single at ffffffff831ec1bc > > --- <IRQ stack> --- > > #35 [ffffc9000019fda8] sysvec_call_function_single at ffffffff831ec1bc > > RIP: ffffffff831ed06e RSP: ffffed10438a6a49 RFLAGS: 00000001 > > RAX: ffff888100d832c0 RBX: 0000000000000000 RCX: 1ffff92000033fd7 > > RDX: 0000000000000000 RSI: ffff888100d832c0 RDI: ffffed10438a6a49 > > RBP: ffffffff831ec166 R8: dffffc0000000000 R9: 0000000000000000 > > R10: ffffffff83400e22 R11: 0000000000000000 R12: ffffffff831ed83e > > R13: 0000000000000000 R14: ffffc9000019fde8 R15: ffffffff814d4d9d > > ORIG_RAX: ffff88821c53524b CS: 0001 SS: ef073a2 > > WARNING: possibly bogus exception frame > > ----------------------->8----------------------- > > > > The catalyst is that CONFIG_SCHED_DEBUG is enabled and the tmp_alone_branch > > assertion fails (Peter, is this bad?). > > Yes, that's not good. IIRC Vincent and Michal were looking at that code > recently. Is it the same SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); that generates the deadlock on v5.15 too ? one remaining tmp_alone_branch warning has been fixed in v5.15 with 2630cde26711 ("sched/fair: Add ancestors of unthrottled undecayed cfs_rq") > > > I'm not sure what the *correct* solution is here (don't use printk while having > > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > > to use one of its lock-less backends?), so I've cc'd all the relevant folks. > > I'm a firm believer in early_printk serial consoles. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-10 9:00 ` Vincent Guittot 0 siblings, 0 replies; 29+ messages in thread From: Vincent Guittot @ 2021-11-10 9:00 UTC (permalink / raw) To: Peter Zijlstra Cc: Sultan Alsawaf, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, Daniel Vetter, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, mkoutny On Tue, 9 Nov 2021 at 22:38, Peter Zijlstra <peterz@infradead.org> wrote: > > On Tue, Nov 09, 2021 at 12:06:48PM -0800, Sultan Alsawaf wrote: > > Hi, > > > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > > kernel. The deadlock occurs due to printk being used while having the current > > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > > the same runqueue when printk tries to flush the log buffer. > > Yes, that's a known 'feature' of some consoles. printk() is in the > process of being reworked to not call con->write() from the printk() > calling context, which would go a long way towards fixing this. > > > #27 [ffffc900005b8e28] enqueue_task_fair at ffffffff8129774a <-- SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); > > #28 [ffffc900005b8ec0] activate_task at ffffffff8125625d > > #29 [ffffc900005b8ef0] ttwu_do_activate at ffffffff81257943 > > #30 [ffffc900005b8f28] sched_ttwu_pending at ffffffff8125c71f <-- locks this CPU's runqueue > > #31 [ffffc900005b8fa0] flush_smp_call_function_queue at ffffffff813c6833 > > #32 [ffffc900005b8fd8] generic_smp_call_function_single_interrupt at ffffffff813c7f58 > > #33 [ffffc900005b8fe0] __sysvec_call_function_single at ffffffff810f1456 > > #34 [ffffc900005b8ff0] sysvec_call_function_single at ffffffff831ec1bc > > --- <IRQ stack> --- > > #35 [ffffc9000019fda8] sysvec_call_function_single at ffffffff831ec1bc > > RIP: ffffffff831ed06e RSP: ffffed10438a6a49 RFLAGS: 00000001 > > RAX: ffff888100d832c0 RBX: 0000000000000000 RCX: 1ffff92000033fd7 > > RDX: 0000000000000000 RSI: ffff888100d832c0 RDI: ffffed10438a6a49 > > RBP: ffffffff831ec166 R8: dffffc0000000000 R9: 0000000000000000 > > R10: ffffffff83400e22 R11: 0000000000000000 R12: ffffffff831ed83e > > R13: 0000000000000000 R14: ffffc9000019fde8 R15: ffffffff814d4d9d > > ORIG_RAX: ffff88821c53524b CS: 0001 SS: ef073a2 > > WARNING: possibly bogus exception frame > > ----------------------->8----------------------- > > > > The catalyst is that CONFIG_SCHED_DEBUG is enabled and the tmp_alone_branch > > assertion fails (Peter, is this bad?). > > Yes, that's not good. IIRC Vincent and Michal were looking at that code > recently. Is it the same SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); that generates the deadlock on v5.15 too ? one remaining tmp_alone_branch warning has been fixed in v5.15 with 2630cde26711 ("sched/fair: Add ancestors of unthrottled undecayed cfs_rq") > > > I'm not sure what the *correct* solution is here (don't use printk while having > > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > > to use one of its lock-less backends?), so I've cc'd all the relevant folks. > > I'm a firm believer in early_printk serial consoles. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-10 9:00 ` Vincent Guittot @ 2021-11-10 10:45 ` Michal Koutný -1 siblings, 0 replies; 29+ messages in thread From: Michal Koutný @ 2021-11-10 10:45 UTC (permalink / raw) To: Vincent Guittot Cc: Peter Zijlstra, Sultan Alsawaf, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, Daniel Vetter, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck On Wed, Nov 10, 2021 at 10:00:35AM +0100, Vincent Guittot <vincent.guittot@linaro.org> wrote: > Is it the same SCHED_WARN_ON(rq->tmp_alone_branch != > &rq->leaf_cfs_rq_list); that generates the deadlock on v5.15 too ? > > one remaining tmp_alone_branch warning has been fixed in v5.15 with > 2630cde26711 ("sched/fair: Add ancestors of unthrottled undecayed cfs_rq") AFAIK, the warning is typically triggered on v5.13 when CPU throttling is employeed. The cfs_rq list is mis-ordered but it shouldn't be fatal. But the throttling currently (since v5.13 too) also exposes a race on task_group removal that can lead to corrupting the cfs_rq list (another thread [1], I'll leave this thread for printk (non)-locking). HTH, Michal [1] https://lore.kernel.org/r/20211011172236.11223-1-mkoutny@suse.com/ ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-10 10:45 ` Michal Koutný 0 siblings, 0 replies; 29+ messages in thread From: Michal Koutný @ 2021-11-10 10:45 UTC (permalink / raw) To: Vincent Guittot Cc: Juri Lelli, Peter Zijlstra, dri-devel, Ben Segall, Sultan Alsawaf, Anton Vorontsov, David Airlie, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, John Ogness, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Wed, Nov 10, 2021 at 10:00:35AM +0100, Vincent Guittot <vincent.guittot@linaro.org> wrote: > Is it the same SCHED_WARN_ON(rq->tmp_alone_branch != > &rq->leaf_cfs_rq_list); that generates the deadlock on v5.15 too ? > > one remaining tmp_alone_branch warning has been fixed in v5.15 with > 2630cde26711 ("sched/fair: Add ancestors of unthrottled undecayed cfs_rq") AFAIK, the warning is typically triggered on v5.13 when CPU throttling is employeed. The cfs_rq list is mis-ordered but it shouldn't be fatal. But the throttling currently (since v5.13 too) also exposes a race on task_group removal that can lead to corrupting the cfs_rq list (another thread [1], I'll leave this thread for printk (non)-locking). HTH, Michal [1] https://lore.kernel.org/r/20211011172236.11223-1-mkoutny@suse.com/ ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-10 9:00 ` Vincent Guittot @ 2021-11-10 19:50 ` Sultan Alsawaf -1 siblings, 0 replies; 29+ messages in thread From: Sultan Alsawaf @ 2021-11-10 19:50 UTC (permalink / raw) To: Vincent Guittot Cc: Peter Zijlstra, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, Daniel Vetter, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, mkoutny On Wed, Nov 10, 2021 at 10:00:35AM +0100, Vincent Guittot wrote: > Is it the same SCHED_WARN_ON(rq->tmp_alone_branch != > &rq->leaf_cfs_rq_list); that generates the deadlock on v5.15 too ? > > one remaining tmp_alone_branch warning has been fixed in v5.15 with > 2630cde26711 ("sched/fair: Add ancestors of unthrottled undecayed cfs_rq") I should clarify that I didn't actually reproduce the issue on v5.15; I just saw that the call chain leading to the deadlock still existed in v5.15 after looking through the code. Failing the SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); assert is extremely rare in my experience, and I don't have a reproducer. It has only happened once after months of heavy usage (with lots of reboots too, so not with crazy high uptime). Sultan ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-10 19:50 ` Sultan Alsawaf 0 siblings, 0 replies; 29+ messages in thread From: Sultan Alsawaf @ 2021-11-10 19:50 UTC (permalink / raw) To: Vincent Guittot Cc: Juri Lelli, Peter Zijlstra, dri-devel, Ben Segall, Anton Vorontsov, David Airlie, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, John Ogness, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, mkoutny, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Wed, Nov 10, 2021 at 10:00:35AM +0100, Vincent Guittot wrote: > Is it the same SCHED_WARN_ON(rq->tmp_alone_branch != > &rq->leaf_cfs_rq_list); that generates the deadlock on v5.15 too ? > > one remaining tmp_alone_branch warning has been fixed in v5.15 with > 2630cde26711 ("sched/fair: Add ancestors of unthrottled undecayed cfs_rq") I should clarify that I didn't actually reproduce the issue on v5.15; I just saw that the call chain leading to the deadlock still existed in v5.15 after looking through the code. Failing the SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); assert is extremely rare in my experience, and I don't have a reproducer. It has only happened once after months of heavy usage (with lots of reboots too, so not with crazy high uptime). Sultan ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-10 19:50 ` Sultan Alsawaf @ 2021-11-12 7:50 ` Vincent Guittot -1 siblings, 0 replies; 29+ messages in thread From: Vincent Guittot @ 2021-11-12 7:50 UTC (permalink / raw) To: Sultan Alsawaf Cc: Peter Zijlstra, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, Daniel Vetter, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, mkoutny On Wed, 10 Nov 2021 at 20:50, Sultan Alsawaf <sultan@kerneltoast.com> wrote: > > On Wed, Nov 10, 2021 at 10:00:35AM +0100, Vincent Guittot wrote: > > Is it the same SCHED_WARN_ON(rq->tmp_alone_branch != > > &rq->leaf_cfs_rq_list); that generates the deadlock on v5.15 too ? > > > > one remaining tmp_alone_branch warning has been fixed in v5.15 with > > 2630cde26711 ("sched/fair: Add ancestors of unthrottled undecayed cfs_rq") > > I should clarify that I didn't actually reproduce the issue on v5.15; I just saw > that the call chain leading to the deadlock still existed in v5.15 after looking > through the code. Thanks for the clarification > > Failing the SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); assert > is extremely rare in my experience, and I don't have a reproducer. It has only > happened once after months of heavy usage (with lots of reboots too, so not with > crazy high uptime). > > Sultan ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-12 7:50 ` Vincent Guittot 0 siblings, 0 replies; 29+ messages in thread From: Vincent Guittot @ 2021-11-12 7:50 UTC (permalink / raw) To: Sultan Alsawaf Cc: Juri Lelli, Peter Zijlstra, dri-devel, Ben Segall, Anton Vorontsov, David Airlie, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, John Ogness, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, mkoutny, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Wed, 10 Nov 2021 at 20:50, Sultan Alsawaf <sultan@kerneltoast.com> wrote: > > On Wed, Nov 10, 2021 at 10:00:35AM +0100, Vincent Guittot wrote: > > Is it the same SCHED_WARN_ON(rq->tmp_alone_branch != > > &rq->leaf_cfs_rq_list); that generates the deadlock on v5.15 too ? > > > > one remaining tmp_alone_branch warning has been fixed in v5.15 with > > 2630cde26711 ("sched/fair: Add ancestors of unthrottled undecayed cfs_rq") > > I should clarify that I didn't actually reproduce the issue on v5.15; I just saw > that the call chain leading to the deadlock still existed in v5.15 after looking > through the code. Thanks for the clarification > > Failing the SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); assert > is extremely rare in my experience, and I don't have a reproducer. It has only > happened once after months of heavy usage (with lots of reboots too, so not with > crazy high uptime). > > Sultan ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-09 21:38 ` Peter Zijlstra @ 2021-11-10 9:37 ` Daniel Vetter -1 siblings, 0 replies; 29+ messages in thread From: Daniel Vetter @ 2021-11-10 9:37 UTC (permalink / raw) To: Peter Zijlstra Cc: Sultan Alsawaf, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, Daniel Vetter, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, Vincent Guittot, mkoutny On Tue, Nov 09, 2021 at 10:38:47PM +0100, Peter Zijlstra wrote: > On Tue, Nov 09, 2021 at 12:06:48PM -0800, Sultan Alsawaf wrote: > > Hi, > > > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > > kernel. The deadlock occurs due to printk being used while having the current > > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > > the same runqueue when printk tries to flush the log buffer. > > Yes, that's a known 'feature' of some consoles. printk() is in the > process of being reworked to not call con->write() from the printk() > calling context, which would go a long way towards fixing this. I'm a bit out of the loop but from lwn articles my understanding is that part of upstreaming from -rt we no longer have the explicit "I'm a safe console for direct printing" opt-in. Which I get from a backwards compat pov, but I still think for at least fbcon we really should never attempt a direct printk con->write, it's just all around terrible. And it's getting worse by the year: - direct scanout displays (i.e. just a few mmio writes and it will show up) are on the way out at least in laptops, everyone gets self-refresh (dp psr) under software control, so without being able to kick a kthread off nothing shows up except more oopses - because of the impendence mismatch between fbdev and drm-kms we even go ever more this direction for dumb framebuffers, including the firmware boot-up framebuffer simpledrm. This could perhaps be fixed with a new dedicate console driver directly on top of drm-kms, but that's on the wishlist for years and I don't see anyone typing that. So yeah for fbcon at least I think we really should throw out direct con->write from printk completely. Also adding John Ogness. -Daniel > > > #27 [ffffc900005b8e28] enqueue_task_fair at ffffffff8129774a <-- SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); > > #28 [ffffc900005b8ec0] activate_task at ffffffff8125625d > > #29 [ffffc900005b8ef0] ttwu_do_activate at ffffffff81257943 > > #30 [ffffc900005b8f28] sched_ttwu_pending at ffffffff8125c71f <-- locks this CPU's runqueue > > #31 [ffffc900005b8fa0] flush_smp_call_function_queue at ffffffff813c6833 > > #32 [ffffc900005b8fd8] generic_smp_call_function_single_interrupt at ffffffff813c7f58 > > #33 [ffffc900005b8fe0] __sysvec_call_function_single at ffffffff810f1456 > > #34 [ffffc900005b8ff0] sysvec_call_function_single at ffffffff831ec1bc > > --- <IRQ stack> --- > > #35 [ffffc9000019fda8] sysvec_call_function_single at ffffffff831ec1bc > > RIP: ffffffff831ed06e RSP: ffffed10438a6a49 RFLAGS: 00000001 > > RAX: ffff888100d832c0 RBX: 0000000000000000 RCX: 1ffff92000033fd7 > > RDX: 0000000000000000 RSI: ffff888100d832c0 RDI: ffffed10438a6a49 > > RBP: ffffffff831ec166 R8: dffffc0000000000 R9: 0000000000000000 > > R10: ffffffff83400e22 R11: 0000000000000000 R12: ffffffff831ed83e > > R13: 0000000000000000 R14: ffffc9000019fde8 R15: ffffffff814d4d9d > > ORIG_RAX: ffff88821c53524b CS: 0001 SS: ef073a2 > > WARNING: possibly bogus exception frame > > ----------------------->8----------------------- > > > > The catalyst is that CONFIG_SCHED_DEBUG is enabled and the tmp_alone_branch > > assertion fails (Peter, is this bad?). > > Yes, that's not good. IIRC Vincent and Michal were looking at that code > recently. > > > I'm not sure what the *correct* solution is here (don't use printk while having > > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > > to use one of its lock-less backends?), so I've cc'd all the relevant folks. > > I'm a firm believer in early_printk serial consoles. -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-10 9:37 ` Daniel Vetter 0 siblings, 0 replies; 29+ messages in thread From: Daniel Vetter @ 2021-11-10 9:37 UTC (permalink / raw) To: Peter Zijlstra Cc: Juri Lelli, David Airlie, dri-devel, Ben Segall, Sultan Alsawaf, Vincent Guittot, Anton Vorontsov, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, John Ogness, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, mkoutny, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Tue, Nov 09, 2021 at 10:38:47PM +0100, Peter Zijlstra wrote: > On Tue, Nov 09, 2021 at 12:06:48PM -0800, Sultan Alsawaf wrote: > > Hi, > > > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > > kernel. The deadlock occurs due to printk being used while having the current > > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > > the same runqueue when printk tries to flush the log buffer. > > Yes, that's a known 'feature' of some consoles. printk() is in the > process of being reworked to not call con->write() from the printk() > calling context, which would go a long way towards fixing this. I'm a bit out of the loop but from lwn articles my understanding is that part of upstreaming from -rt we no longer have the explicit "I'm a safe console for direct printing" opt-in. Which I get from a backwards compat pov, but I still think for at least fbcon we really should never attempt a direct printk con->write, it's just all around terrible. And it's getting worse by the year: - direct scanout displays (i.e. just a few mmio writes and it will show up) are on the way out at least in laptops, everyone gets self-refresh (dp psr) under software control, so without being able to kick a kthread off nothing shows up except more oopses - because of the impendence mismatch between fbdev and drm-kms we even go ever more this direction for dumb framebuffers, including the firmware boot-up framebuffer simpledrm. This could perhaps be fixed with a new dedicate console driver directly on top of drm-kms, but that's on the wishlist for years and I don't see anyone typing that. So yeah for fbcon at least I think we really should throw out direct con->write from printk completely. Also adding John Ogness. -Daniel > > > #27 [ffffc900005b8e28] enqueue_task_fair at ffffffff8129774a <-- SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list); > > #28 [ffffc900005b8ec0] activate_task at ffffffff8125625d > > #29 [ffffc900005b8ef0] ttwu_do_activate at ffffffff81257943 > > #30 [ffffc900005b8f28] sched_ttwu_pending at ffffffff8125c71f <-- locks this CPU's runqueue > > #31 [ffffc900005b8fa0] flush_smp_call_function_queue at ffffffff813c6833 > > #32 [ffffc900005b8fd8] generic_smp_call_function_single_interrupt at ffffffff813c7f58 > > #33 [ffffc900005b8fe0] __sysvec_call_function_single at ffffffff810f1456 > > #34 [ffffc900005b8ff0] sysvec_call_function_single at ffffffff831ec1bc > > --- <IRQ stack> --- > > #35 [ffffc9000019fda8] sysvec_call_function_single at ffffffff831ec1bc > > RIP: ffffffff831ed06e RSP: ffffed10438a6a49 RFLAGS: 00000001 > > RAX: ffff888100d832c0 RBX: 0000000000000000 RCX: 1ffff92000033fd7 > > RDX: 0000000000000000 RSI: ffff888100d832c0 RDI: ffffed10438a6a49 > > RBP: ffffffff831ec166 R8: dffffc0000000000 R9: 0000000000000000 > > R10: ffffffff83400e22 R11: 0000000000000000 R12: ffffffff831ed83e > > R13: 0000000000000000 R14: ffffc9000019fde8 R15: ffffffff814d4d9d > > ORIG_RAX: ffff88821c53524b CS: 0001 SS: ef073a2 > > WARNING: possibly bogus exception frame > > ----------------------->8----------------------- > > > > The catalyst is that CONFIG_SCHED_DEBUG is enabled and the tmp_alone_branch > > assertion fails (Peter, is this bad?). > > Yes, that's not good. IIRC Vincent and Michal were looking at that code > recently. > > > I'm not sure what the *correct* solution is here (don't use printk while having > > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > > to use one of its lock-less backends?), so I've cc'd all the relevant folks. > > I'm a firm believer in early_printk serial consoles. -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-10 9:37 ` Daniel Vetter @ 2021-11-10 10:07 ` John Ogness -1 siblings, 0 replies; 29+ messages in thread From: John Ogness @ 2021-11-10 10:07 UTC (permalink / raw) To: Daniel Vetter, Peter Zijlstra Cc: Sultan Alsawaf, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, Daniel Vetter, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, Vincent Guittot, mkoutny On 2021-11-10, Daniel Vetter <daniel@ffwll.ch> wrote: > I'm a bit out of the loop but from lwn articles my understanding is > that part of upstreaming from -rt we no longer have the explicit "I'm > a safe console for direct printing" opt-in. Which I get from a > backwards compat pov, but I still think for at least fbcon we really > should never attempt a direct printk con->write, it's just all around > terrible. Right now we don't have an explicit "I'm a safe console for direct printing" option. Right now all printing is direct. But it sounds to me that we should add this console flag when we introduce kthread printers. > So yeah for fbcon at least I think we really should throw out direct > con->write from printk completely. Even after we introduce kthread printers, there will still be situations where direct printing is used: booting (before kthreads exist) and shutdown/suspend/crash situations, when the kthreads may not be active. I will introduce a console flag so that consoles can opt-out for direct printing. (opt-out rather than opt-in is probably easier, since there are only a few that would need to opt-out). Since kthread printers do not yet exist (hoping to get them in for 5.17), I am not sure how we should address the reported bug for existing kernels. John Ogness ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-10 10:07 ` John Ogness 0 siblings, 0 replies; 29+ messages in thread From: John Ogness @ 2021-11-10 10:07 UTC (permalink / raw) To: Daniel Vetter, Peter Zijlstra Cc: Juri Lelli, David Airlie, dri-devel, Ben Segall, Sultan Alsawaf, Vincent Guittot, Anton Vorontsov, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, mkoutny, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On 2021-11-10, Daniel Vetter <daniel@ffwll.ch> wrote: > I'm a bit out of the loop but from lwn articles my understanding is > that part of upstreaming from -rt we no longer have the explicit "I'm > a safe console for direct printing" opt-in. Which I get from a > backwards compat pov, but I still think for at least fbcon we really > should never attempt a direct printk con->write, it's just all around > terrible. Right now we don't have an explicit "I'm a safe console for direct printing" option. Right now all printing is direct. But it sounds to me that we should add this console flag when we introduce kthread printers. > So yeah for fbcon at least I think we really should throw out direct > con->write from printk completely. Even after we introduce kthread printers, there will still be situations where direct printing is used: booting (before kthreads exist) and shutdown/suspend/crash situations, when the kthreads may not be active. I will introduce a console flag so that consoles can opt-out for direct printing. (opt-out rather than opt-in is probably easier, since there are only a few that would need to opt-out). Since kthread printers do not yet exist (hoping to get them in for 5.17), I am not sure how we should address the reported bug for existing kernels. John Ogness ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-10 10:07 ` John Ogness @ 2021-11-10 10:44 ` Daniel Vetter -1 siblings, 0 replies; 29+ messages in thread From: Daniel Vetter @ 2021-11-10 10:44 UTC (permalink / raw) To: John Ogness Cc: Daniel Vetter, Peter Zijlstra, Sultan Alsawaf, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, Vincent Guittot, mkoutny On Wed, Nov 10, 2021 at 11:13:37AM +0106, John Ogness wrote: > On 2021-11-10, Daniel Vetter <daniel@ffwll.ch> wrote: > > I'm a bit out of the loop but from lwn articles my understanding is > > that part of upstreaming from -rt we no longer have the explicit "I'm > > a safe console for direct printing" opt-in. Which I get from a > > backwards compat pov, but I still think for at least fbcon we really > > should never attempt a direct printk con->write, it's just all around > > terrible. > > Right now we don't have an explicit "I'm a safe console for direct > printing" option. Right now all printing is direct. But it sounds to me > that we should add this console flag when we introduce kthread printers. Ah I wasnt' clear whether this was merged already or not yet. > > So yeah for fbcon at least I think we really should throw out direct > > con->write from printk completely. > > Even after we introduce kthread printers, there will still be situations > where direct printing is used: booting (before kthreads exist) and > shutdown/suspend/crash situations, when the kthreads may not be > active. > > I will introduce a console flag so that consoles can opt-out for direct > printing. (opt-out rather than opt-in is probably easier, since there > are only a few that would need to opt-out). Yeah opt-out for fbcon sounds good to me, and then across the board (even when the kthread is not yet or no longer running, it really doesn't make anything better). Maybe with a fbcon module (or system wide force_direct_printk) option to disable that flag in case someone needs it. > Since kthread printers do not yet exist (hoping to get them in for > 5.17), I am not sure how we should address the reported bug for existing > kernels. I think we just can't. This thing keeps popping up in all kinds of places, and in some cases it's really not possible to fix it. Like in oops_in_progress we can just give up, but for normal printk from funny places we'd corrupt the console (plus there's no flag to even detect these cases), since we really need to launch these work_struct to get stuff to the screen. I think in the past we sometimes managed to work around the issue by moving the printk outside of locks, but that's rarely a good idea just because fbcon. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-10 10:44 ` Daniel Vetter 0 siblings, 0 replies; 29+ messages in thread From: Daniel Vetter @ 2021-11-10 10:44 UTC (permalink / raw) To: John Ogness Cc: Juri Lelli, Peter Zijlstra, dri-devel, Ben Segall, Sultan Alsawaf, Vincent Guittot, Anton Vorontsov, David Airlie, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, mkoutny, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Wed, Nov 10, 2021 at 11:13:37AM +0106, John Ogness wrote: > On 2021-11-10, Daniel Vetter <daniel@ffwll.ch> wrote: > > I'm a bit out of the loop but from lwn articles my understanding is > > that part of upstreaming from -rt we no longer have the explicit "I'm > > a safe console for direct printing" opt-in. Which I get from a > > backwards compat pov, but I still think for at least fbcon we really > > should never attempt a direct printk con->write, it's just all around > > terrible. > > Right now we don't have an explicit "I'm a safe console for direct > printing" option. Right now all printing is direct. But it sounds to me > that we should add this console flag when we introduce kthread printers. Ah I wasnt' clear whether this was merged already or not yet. > > So yeah for fbcon at least I think we really should throw out direct > > con->write from printk completely. > > Even after we introduce kthread printers, there will still be situations > where direct printing is used: booting (before kthreads exist) and > shutdown/suspend/crash situations, when the kthreads may not be > active. > > I will introduce a console flag so that consoles can opt-out for direct > printing. (opt-out rather than opt-in is probably easier, since there > are only a few that would need to opt-out). Yeah opt-out for fbcon sounds good to me, and then across the board (even when the kthread is not yet or no longer running, it really doesn't make anything better). Maybe with a fbcon module (or system wide force_direct_printk) option to disable that flag in case someone needs it. > Since kthread printers do not yet exist (hoping to get them in for > 5.17), I am not sure how we should address the reported bug for existing > kernels. I think we just can't. This thing keeps popping up in all kinds of places, and in some cases it's really not possible to fix it. Like in oops_in_progress we can just give up, but for normal printk from funny places we'd corrupt the console (plus there's no flag to even detect these cases), since we really need to launch these work_struct to get stuff to the screen. I think in the past we sometimes managed to work around the issue by moving the printk outside of locks, but that's rarely a good idea just because fbcon. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-10 10:07 ` John Ogness @ 2021-11-10 20:03 ` Sultan Alsawaf -1 siblings, 0 replies; 29+ messages in thread From: Sultan Alsawaf @ 2021-11-10 20:03 UTC (permalink / raw) To: John Ogness Cc: Daniel Vetter, Peter Zijlstra, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, Vincent Guittot, mkoutny On Wed, Nov 10, 2021 at 11:13:37AM +0106, John Ogness wrote: > Even after we introduce kthread printers, there will still be situations > where direct printing is used: booting (before kthreads exist) and > shutdown/suspend/crash situations, when the kthreads may not be > active. Although I'm unaware of any ongoing kthread printer work, I'm curious to know how a kthread approach wouldn't employ a try_to_wake_up() from directly inside printk(), since the try_to_wake_up() hit from inside the fbcon code is what caused my deadlock. Sultan ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-10 20:03 ` Sultan Alsawaf 0 siblings, 0 replies; 29+ messages in thread From: Sultan Alsawaf @ 2021-11-10 20:03 UTC (permalink / raw) To: John Ogness Cc: Juri Lelli, Peter Zijlstra, dri-devel, Ben Segall, Vincent Guittot, Anton Vorontsov, David Airlie, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, mkoutny, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Wed, Nov 10, 2021 at 11:13:37AM +0106, John Ogness wrote: > Even after we introduce kthread printers, there will still be situations > where direct printing is used: booting (before kthreads exist) and > shutdown/suspend/crash situations, when the kthreads may not be > active. Although I'm unaware of any ongoing kthread printer work, I'm curious to know how a kthread approach wouldn't employ a try_to_wake_up() from directly inside printk(), since the try_to_wake_up() hit from inside the fbcon code is what caused my deadlock. Sultan ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-10 20:03 ` Sultan Alsawaf @ 2021-11-11 8:28 ` John Ogness -1 siblings, 0 replies; 29+ messages in thread From: John Ogness @ 2021-11-11 8:28 UTC (permalink / raw) To: Sultan Alsawaf Cc: Daniel Vetter, Peter Zijlstra, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, Vincent Guittot, mkoutny On 2021-11-10, Sultan Alsawaf <sultan@kerneltoast.com> wrote: > On Wed, Nov 10, 2021 at 11:13:37AM +0106, John Ogness wrote: >> Even after we introduce kthread printers, there will still be >> situations where direct printing is used: booting (before kthreads >> exist) and shutdown/suspend/crash situations, when the kthreads may >> not be active. > > Although I'm unaware of any ongoing kthread printer work, I'm curious > to know how a kthread approach wouldn't employ a try_to_wake_up() from > directly inside printk(), since the try_to_wake_up() hit from inside > the fbcon code is what caused my deadlock. The kthread approach triggers irq_work from printk(). The kthread printer is then woken from the irq_work. John Ogness ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-11 8:28 ` John Ogness 0 siblings, 0 replies; 29+ messages in thread From: John Ogness @ 2021-11-11 8:28 UTC (permalink / raw) To: Sultan Alsawaf Cc: Juri Lelli, Peter Zijlstra, dri-devel, Ben Segall, Vincent Guittot, Anton Vorontsov, David Airlie, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, mkoutny, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On 2021-11-10, Sultan Alsawaf <sultan@kerneltoast.com> wrote: > On Wed, Nov 10, 2021 at 11:13:37AM +0106, John Ogness wrote: >> Even after we introduce kthread printers, there will still be >> situations where direct printing is used: booting (before kthreads >> exist) and shutdown/suspend/crash situations, when the kthreads may >> not be active. > > Although I'm unaware of any ongoing kthread printer work, I'm curious > to know how a kthread approach wouldn't employ a try_to_wake_up() from > directly inside printk(), since the try_to_wake_up() hit from inside > the fbcon code is what caused my deadlock. The kthread approach triggers irq_work from printk(). The kthread printer is then woken from the irq_work. John Ogness ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-10 9:37 ` Daniel Vetter (?) (?) @ 2021-11-11 9:27 ` Petr Mladek -1 siblings, 0 replies; 29+ messages in thread From: Petr Mladek @ 2021-11-11 9:27 UTC (permalink / raw) To: Peter Zijlstra, Sultan Alsawaf, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, Vincent Guittot, mkoutny On Wed 2021-11-10 10:37:26, Daniel Vetter wrote: > On Tue, Nov 09, 2021 at 10:38:47PM +0100, Peter Zijlstra wrote: > > On Tue, Nov 09, 2021 at 12:06:48PM -0800, Sultan Alsawaf wrote: > > > Hi, > > > > > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > > > kernel. The deadlock occurs due to printk being used while having the current > > > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > > > the same runqueue when printk tries to flush the log buffer. > > > > Yes, that's a known 'feature' of some consoles. printk() is in the > > process of being reworked to not call con->write() from the printk() > > calling context, which would go a long way towards fixing this. > > And it's getting worse by the year: > - direct scanout displays (i.e. just a few mmio writes and it will show > up) are on the way out at least in laptops, everyone gets self-refresh > (dp psr) under software control, so without being able to kick a kthread > off nothing shows up except more oopses printk() tries to avoid this problem by waking processes via irq_vork(). For example, it is used for waking userspace log daemons, see wake_up_klogd(). There is one hidden wake_up_process() in up() of @console_sem. It is "less" critical because it calls the wake_up() only when there is another process waiting for console_lock(). It must be a non-printk caller because printk() uses trylock and does not add waiters. So, there is a non-trivial chance to avoid the deadlock and the messages. I wonder if fbdev might use the same trick. > - because of the impendence mismatch between fbdev and drm-kms we even go > ever more this direction for dumb framebuffers, including the firmware > boot-up framebuffer simpledrm. This could perhaps be fixed with a new > dedicate console driver directly on top of drm-kms, but that's on the > wishlist for years and I don't see anyone typing that. > > So yeah for fbcon at least I think we really should throw out direct > con->write from printk completely. Yup, that would be great. Well, I guess that it would be available only for a panic mode when it will not race with other users? Best Regards, Petr ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-09 20:06 ` Sultan Alsawaf @ 2021-11-10 10:50 ` Petr Mladek -1 siblings, 0 replies; 29+ messages in thread From: Petr Mladek @ 2021-11-10 10:50 UTC (permalink / raw) To: Sultan Alsawaf Cc: Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, Daniel Vetter, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Peter Zijlstra, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, Vincent Guittot On Tue 2021-11-09 12:06:48, Sultan Alsawaf wrote: > Hi, > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > kernel. The deadlock occurs due to printk being used while having the current > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > the same runqueue when printk tries to flush the log buffer. > > I'm not sure what the *correct* solution is here (don't use printk while having > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > to use one of its lock-less backends?), so I've cc'd all the relevant folks. At the moment, printk_deferred() could be used here. It defers the console handling via irq_work(). There is no deferred variant for WARN() at the moment. The following might work: #define WARN_DEFERRED(condition, format...) ({ \ unsigned long flags; \ \ printk_safe_enter_irqsave(flags); \ WARN(condition, format...) \ printk_safe_exit_irqrestore(flags); \ }) , where printk_safe_enter_irqsave()/printk_safe_exit_irqrestore(flags) are currently used only internally by printk() code and defined in the local kernel/printk/internal.h Be ware that using the deferred variants is a whack a mole approach. There are many printk() callers that might be called indirectly and eventually cause deadlocks. As already said, the plan is to upstream -rt solution and offload the console work into kthreads. But it goes slowly. We want to make it a clean way and prevent regressions as much as possible. Best Regards, Petr ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-10 10:50 ` Petr Mladek 0 siblings, 0 replies; 29+ messages in thread From: Petr Mladek @ 2021-11-10 10:50 UTC (permalink / raw) To: Sultan Alsawaf Cc: Juri Lelli, David Airlie, dri-devel, Ben Segall, Vincent Guittot, Anton Vorontsov, Peter Zijlstra, Ingo Molnar, Mel Gorman, Kees Cook, John Ogness, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Tue 2021-11-09 12:06:48, Sultan Alsawaf wrote: > Hi, > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > kernel. The deadlock occurs due to printk being used while having the current > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > the same runqueue when printk tries to flush the log buffer. > > I'm not sure what the *correct* solution is here (don't use printk while having > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > to use one of its lock-less backends?), so I've cc'd all the relevant folks. At the moment, printk_deferred() could be used here. It defers the console handling via irq_work(). There is no deferred variant for WARN() at the moment. The following might work: #define WARN_DEFERRED(condition, format...) ({ \ unsigned long flags; \ \ printk_safe_enter_irqsave(flags); \ WARN(condition, format...) \ printk_safe_exit_irqrestore(flags); \ }) , where printk_safe_enter_irqsave()/printk_safe_exit_irqrestore(flags) are currently used only internally by printk() code and defined in the local kernel/printk/internal.h Be ware that using the deferred variants is a whack a mole approach. There are many printk() callers that might be called indirectly and eventually cause deadlocks. As already said, the plan is to upstream -rt solution and offload the console work into kthreads. But it goes slowly. We want to make it a clean way and prevent regressions as much as possible. Best Regards, Petr ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-10 10:50 ` Petr Mladek @ 2021-11-10 11:20 ` Peter Zijlstra -1 siblings, 0 replies; 29+ messages in thread From: Peter Zijlstra @ 2021-11-10 11:20 UTC (permalink / raw) To: Petr Mladek Cc: Sultan Alsawaf, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, Daniel Vetter, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, Vincent Guittot On Wed, Nov 10, 2021 at 11:50:38AM +0100, Petr Mladek wrote: > On Tue 2021-11-09 12:06:48, Sultan Alsawaf wrote: > > Hi, > > > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > > kernel. The deadlock occurs due to printk being used while having the current > > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > > the same runqueue when printk tries to flush the log buffer. > > > > I'm not sure what the *correct* solution is here (don't use printk while having > > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > > to use one of its lock-less backends?), so I've cc'd all the relevant folks. > > At the moment, printk_deferred() could be used here. It defers the > console handling via irq_work(). I think I've rejected that patch at least twice now :-) John's printk stuff will really land real soon now, right. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-10 11:20 ` Peter Zijlstra 0 siblings, 0 replies; 29+ messages in thread From: Peter Zijlstra @ 2021-11-10 11:20 UTC (permalink / raw) To: Petr Mladek Cc: Juri Lelli, David Airlie, dri-devel, Ben Segall, Sultan Alsawaf, Vincent Guittot, Anton Vorontsov, Ingo Molnar, Mel Gorman, Kees Cook, John Ogness, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Wed, Nov 10, 2021 at 11:50:38AM +0100, Petr Mladek wrote: > On Tue 2021-11-09 12:06:48, Sultan Alsawaf wrote: > > Hi, > > > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > > kernel. The deadlock occurs due to printk being used while having the current > > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > > the same runqueue when printk tries to flush the log buffer. > > > > I'm not sure what the *correct* solution is here (don't use printk while having > > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > > to use one of its lock-less backends?), so I've cc'd all the relevant folks. > > At the moment, printk_deferred() could be used here. It defers the > console handling via irq_work(). I think I've rejected that patch at least twice now :-) John's printk stuff will really land real soon now, right. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue 2021-11-10 11:20 ` Peter Zijlstra @ 2021-11-10 13:21 ` Daniel Vetter -1 siblings, 0 replies; 29+ messages in thread From: Daniel Vetter @ 2021-11-10 13:21 UTC (permalink / raw) To: Peter Zijlstra Cc: Petr Mladek, Sultan Alsawaf, Anton Vorontsov, Ben Segall, Colin Cross, Daniel Bristot de Oliveira, Daniel Vetter, David Airlie, Dietmar Eggemann, dri-devel, Ingo Molnar, John Ogness, Juri Lelli, Kees Cook, linux-kernel, Maarten Lankhorst, Maxime Ripard, Mel Gorman, Sergey Senozhatsky, Steven Rostedt, Thomas Zimmermann, Tony Luck, Vincent Guittot On Wed, Nov 10, 2021 at 12:20:05PM +0100, Peter Zijlstra wrote: > On Wed, Nov 10, 2021 at 11:50:38AM +0100, Petr Mladek wrote: > > On Tue 2021-11-09 12:06:48, Sultan Alsawaf wrote: > > > Hi, > > > > > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > > > kernel. The deadlock occurs due to printk being used while having the current > > > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > > > the same runqueue when printk tries to flush the log buffer. > > > > > > I'm not sure what the *correct* solution is here (don't use printk while having > > > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > > > to use one of its lock-less backends?), so I've cc'd all the relevant folks. > > > > At the moment, printk_deferred() could be used here. It defers the > > console handling via irq_work(). > > I think I've rejected that patch at least twice now :-) John's printk > stuff will really land real soon now, right. Yeah whacking all affected prinkt callers just because of fbcon does not sound like a good idea to me either. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: printk deadlock due to double lock attempt on current CPU's runqueue @ 2021-11-10 13:21 ` Daniel Vetter 0 siblings, 0 replies; 29+ messages in thread From: Daniel Vetter @ 2021-11-10 13:21 UTC (permalink / raw) To: Peter Zijlstra Cc: Juri Lelli, David Airlie, dri-devel, Ben Segall, Sultan Alsawaf, Vincent Guittot, Anton Vorontsov, Ingo Molnar, Mel Gorman, Petr Mladek, Kees Cook, John Ogness, Steven Rostedt, Dietmar Eggemann, Tony Luck, linux-kernel, Sergey Senozhatsky, Thomas Zimmermann, Colin Cross, Daniel Bristot de Oliveira On Wed, Nov 10, 2021 at 12:20:05PM +0100, Peter Zijlstra wrote: > On Wed, Nov 10, 2021 at 11:50:38AM +0100, Petr Mladek wrote: > > On Tue 2021-11-09 12:06:48, Sultan Alsawaf wrote: > > > Hi, > > > > > > I encountered a printk deadlock on 5.13 which appears to still affect the latest > > > kernel. The deadlock occurs due to printk being used while having the current > > > CPU's runqueue locked, and the underlying framebuffer console attempting to lock > > > the same runqueue when printk tries to flush the log buffer. > > > > > > I'm not sure what the *correct* solution is here (don't use printk while having > > > a runqueue locked? don't use schedule_work() from the fbcon path? tell printk > > > to use one of its lock-less backends?), so I've cc'd all the relevant folks. > > > > At the moment, printk_deferred() could be used here. It defers the > > console handling via irq_work(). > > I think I've rejected that patch at least twice now :-) John's printk > stuff will really land real soon now, right. Yeah whacking all affected prinkt callers just because of fbcon does not sound like a good idea to me either. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2021-11-12 7:51 UTC | newest] Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-11-09 20:06 printk deadlock due to double lock attempt on current CPU's runqueue Sultan Alsawaf 2021-11-09 20:06 ` Sultan Alsawaf 2021-11-09 21:38 ` Peter Zijlstra 2021-11-09 21:38 ` Peter Zijlstra 2021-11-10 9:00 ` Vincent Guittot 2021-11-10 9:00 ` Vincent Guittot 2021-11-10 10:45 ` Michal Koutný 2021-11-10 10:45 ` Michal Koutný 2021-11-10 19:50 ` Sultan Alsawaf 2021-11-10 19:50 ` Sultan Alsawaf 2021-11-12 7:50 ` Vincent Guittot 2021-11-12 7:50 ` Vincent Guittot 2021-11-10 9:37 ` Daniel Vetter 2021-11-10 9:37 ` Daniel Vetter 2021-11-10 10:07 ` John Ogness 2021-11-10 10:07 ` John Ogness 2021-11-10 10:44 ` Daniel Vetter 2021-11-10 10:44 ` Daniel Vetter 2021-11-10 20:03 ` Sultan Alsawaf 2021-11-10 20:03 ` Sultan Alsawaf 2021-11-11 8:28 ` John Ogness 2021-11-11 8:28 ` John Ogness 2021-11-11 9:27 ` Petr Mladek 2021-11-10 10:50 ` Petr Mladek 2021-11-10 10:50 ` Petr Mladek 2021-11-10 11:20 ` Peter Zijlstra 2021-11-10 11:20 ` Peter Zijlstra 2021-11-10 13:21 ` Daniel Vetter 2021-11-10 13:21 ` Daniel Vetter
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.