From: Peter Zijlstra <peterz@infradead.org>
To: Sultan Alsawaf <sultan@kerneltoast.com>
Cc: Anton Vorontsov <anton@enomsg.org>,
Ben Segall <bsegall@google.com>, Colin Cross <ccross@android.com>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Daniel Vetter <daniel@ffwll.ch>, David Airlie <airlied@linux.ie>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
dri-devel@lists.freedesktop.org, Ingo Molnar <mingo@redhat.com>,
John Ogness <john.ogness@linutronix.de>,
Juri Lelli <juri.lelli@redhat.com>,
Kees Cook <keescook@chromium.org>,
linux-kernel@vger.kernel.org,
Maarten Lankhorst <maarten.lankhorst@linux.intel.com>,
Maxime Ripard <mripard@kernel.org>, Mel Gorman <mgorman@suse.de>,
Petr Mladek <pmladek@suse.com>,
Sergey Senozhatsky <senozhatsky@chromium.org>,
Steven Rostedt <rostedt@goodmis.org>,
Thomas Zimmermann <tzimmermann@suse.de>,
Tony Luck <tony.luck@intel.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
mkoutny@suse.com
Subject: Re: printk deadlock due to double lock attempt on current CPU's runqueue
Date: Tue, 9 Nov 2021 22:38:47 +0100 [thread overview]
Message-ID: <20211109213847.GY174703@worktop.programming.kicks-ass.net> (raw)
In-Reply-To: <YYrU2PdmdNkulWSM@sultan-box.localdomain>
On Tue, Nov 09, 2021 at 12:06:48PM -0800, Sultan Alsawaf wrote:
> Hi,
>
> I encountered a printk deadlock on 5.13 which appears to still affect the latest
> kernel. The deadlock occurs due to printk being used while having the current
> CPU's runqueue locked, and the underlying framebuffer console attempting to lock
> the same runqueue when printk tries to flush the log buffer.
Yes, that's a known 'feature' of some consoles. printk() is in the
process of being reworked to not call con->write() from the printk()
calling context, which would go a long way towards fixing this.
> #27 [ffffc900005b8e28] enqueue_task_fair at ffffffff8129774a <-- SCHED_WARN_ON(rq->tmp_alone_branch != &rq->leaf_cfs_rq_list);
> #28 [ffffc900005b8ec0] activate_task at ffffffff8125625d
> #29 [ffffc900005b8ef0] ttwu_do_activate at ffffffff81257943
> #30 [ffffc900005b8f28] sched_ttwu_pending at ffffffff8125c71f <-- locks this CPU's runqueue
> #31 [ffffc900005b8fa0] flush_smp_call_function_queue at ffffffff813c6833
> #32 [ffffc900005b8fd8] generic_smp_call_function_single_interrupt at ffffffff813c7f58
> #33 [ffffc900005b8fe0] __sysvec_call_function_single at ffffffff810f1456
> #34 [ffffc900005b8ff0] sysvec_call_function_single at ffffffff831ec1bc
> --- <IRQ stack> ---
> #35 [ffffc9000019fda8] sysvec_call_function_single at ffffffff831ec1bc
> RIP: ffffffff831ed06e RSP: ffffed10438a6a49 RFLAGS: 00000001
> RAX: ffff888100d832c0 RBX: 0000000000000000 RCX: 1ffff92000033fd7
> RDX: 0000000000000000 RSI: ffff888100d832c0 RDI: ffffed10438a6a49
> RBP: ffffffff831ec166 R8: dffffc0000000000 R9: 0000000000000000
> R10: ffffffff83400e22 R11: 0000000000000000 R12: ffffffff831ed83e
> R13: 0000000000000000 R14: ffffc9000019fde8 R15: ffffffff814d4d9d
> ORIG_RAX: ffff88821c53524b CS: 0001 SS: ef073a2
> WARNING: possibly bogus exception frame
> ----------------------->8-----------------------
>
> The catalyst is that CONFIG_SCHED_DEBUG is enabled and the tmp_alone_branch
> assertion fails (Peter, is this bad?).
Yes, that's not good. IIRC Vincent and Michal were looking at that code
recently.
> I'm not sure what the *correct* solution is here (don't use printk while having
> a runqueue locked? don't use schedule_work() from the fbcon path? tell printk
> to use one of its lock-less backends?), so I've cc'd all the relevant folks.
I'm a firm believer in early_printk serial consoles.
next prev parent reply other threads:[~2021-11-09 21:39 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-09 20:06 printk deadlock due to double lock attempt on current CPU's runqueue Sultan Alsawaf
2021-11-09 21:38 ` Peter Zijlstra [this message]
2021-11-10 9:00 ` Vincent Guittot
2021-11-10 10:45 ` Michal Koutný
2021-11-10 19:50 ` Sultan Alsawaf
2021-11-12 7:50 ` Vincent Guittot
2021-11-10 9:37 ` Daniel Vetter
2021-11-10 10:07 ` John Ogness
2021-11-10 10:44 ` Daniel Vetter
2021-11-10 20:03 ` Sultan Alsawaf
2021-11-11 8:28 ` John Ogness
2021-11-11 9:27 ` Petr Mladek
2021-11-10 10:50 ` Petr Mladek
2021-11-10 11:20 ` Peter Zijlstra
2021-11-10 13:21 ` Daniel Vetter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211109213847.GY174703@worktop.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=airlied@linux.ie \
--cc=anton@enomsg.org \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=ccross@android.com \
--cc=daniel@ffwll.ch \
--cc=dietmar.eggemann@arm.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=john.ogness@linutronix.de \
--cc=juri.lelli@redhat.com \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=maarten.lankhorst@linux.intel.com \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=mkoutny@suse.com \
--cc=mripard@kernel.org \
--cc=pmladek@suse.com \
--cc=rostedt@goodmis.org \
--cc=senozhatsky@chromium.org \
--cc=sultan@kerneltoast.com \
--cc=tony.luck@intel.com \
--cc=tzimmermann@suse.de \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).