From: Petr Mladek <pmladek@suse.com>
To: Sultan Alsawaf <sultan@kerneltoast.com>
Cc: Anton Vorontsov <anton@enomsg.org>,
Ben Segall <bsegall@google.com>, Colin Cross <ccross@android.com>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Daniel Vetter <daniel@ffwll.ch>, David Airlie <airlied@linux.ie>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
dri-devel@lists.freedesktop.org, Ingo Molnar <mingo@redhat.com>,
John Ogness <john.ogness@linutronix.de>,
Juri Lelli <juri.lelli@redhat.com>,
Kees Cook <keescook@chromium.org>,
linux-kernel@vger.kernel.org,
Maarten Lankhorst <maarten.lankhorst@linux.intel.com>,
Maxime Ripard <mripard@kernel.org>, Mel Gorman <mgorman@suse.de>,
Peter Zijlstra <peterz@infradead.org>,
Sergey Senozhatsky <senozhatsky@chromium.org>,
Steven Rostedt <rostedt@goodmis.org>,
Thomas Zimmermann <tzimmermann@suse.de>,
Tony Luck <tony.luck@intel.com>,
Vincent Guittot <vincent.guittot@linaro.org>
Subject: Re: printk deadlock due to double lock attempt on current CPU's runqueue
Date: Wed, 10 Nov 2021 11:50:38 +0100 [thread overview]
Message-ID: <YYuj/rT+EO3K0LsK@alley> (raw)
In-Reply-To: <YYrU2PdmdNkulWSM@sultan-box.localdomain>
On Tue 2021-11-09 12:06:48, Sultan Alsawaf wrote:
> Hi,
>
> I encountered a printk deadlock on 5.13 which appears to still affect the latest
> kernel. The deadlock occurs due to printk being used while having the current
> CPU's runqueue locked, and the underlying framebuffer console attempting to lock
> the same runqueue when printk tries to flush the log buffer.
>
> I'm not sure what the *correct* solution is here (don't use printk while having
> a runqueue locked? don't use schedule_work() from the fbcon path? tell printk
> to use one of its lock-less backends?), so I've cc'd all the relevant folks.
At the moment, printk_deferred() could be used here. It defers the
console handling via irq_work().
There is no deferred variant for WARN() at the moment. The following
might work:
#define WARN_DEFERRED(condition, format...) ({ \
unsigned long flags; \
\
printk_safe_enter_irqsave(flags); \
WARN(condition, format...) \
printk_safe_exit_irqrestore(flags); \
})
, where printk_safe_enter_irqsave()/printk_safe_exit_irqrestore(flags)
are currently used only internally by printk() code and defined
in the local kernel/printk/internal.h
Be ware that using the deferred variants is a whack a mole approach.
There are many printk() callers that might be called indirectly
and eventually cause deadlocks.
As already said, the plan is to upstream -rt solution and offload
the console work into kthreads. But it goes slowly. We want to
make it a clean way and prevent regressions as much as possible.
Best Regards,
Petr
WARNING: multiple messages have this Message-ID (diff)
From: Petr Mladek <pmladek@suse.com>
To: Sultan Alsawaf <sultan@kerneltoast.com>
Cc: Juri Lelli <juri.lelli@redhat.com>,
David Airlie <airlied@linux.ie>,
dri-devel@lists.freedesktop.org, Ben Segall <bsegall@google.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Anton Vorontsov <anton@enomsg.org>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>, Mel Gorman <mgorman@suse.de>,
Kees Cook <keescook@chromium.org>,
John Ogness <john.ogness@linutronix.de>,
Steven Rostedt <rostedt@goodmis.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Tony Luck <tony.luck@intel.com>,
linux-kernel@vger.kernel.org,
Sergey Senozhatsky <senozhatsky@chromium.org>,
Thomas Zimmermann <tzimmermann@suse.de>,
Colin Cross <ccross@android.com>,
Daniel Bristot de Oliveira <bristot@redhat.com>
Subject: Re: printk deadlock due to double lock attempt on current CPU's runqueue
Date: Wed, 10 Nov 2021 11:50:38 +0100 [thread overview]
Message-ID: <YYuj/rT+EO3K0LsK@alley> (raw)
In-Reply-To: <YYrU2PdmdNkulWSM@sultan-box.localdomain>
On Tue 2021-11-09 12:06:48, Sultan Alsawaf wrote:
> Hi,
>
> I encountered a printk deadlock on 5.13 which appears to still affect the latest
> kernel. The deadlock occurs due to printk being used while having the current
> CPU's runqueue locked, and the underlying framebuffer console attempting to lock
> the same runqueue when printk tries to flush the log buffer.
>
> I'm not sure what the *correct* solution is here (don't use printk while having
> a runqueue locked? don't use schedule_work() from the fbcon path? tell printk
> to use one of its lock-less backends?), so I've cc'd all the relevant folks.
At the moment, printk_deferred() could be used here. It defers the
console handling via irq_work().
There is no deferred variant for WARN() at the moment. The following
might work:
#define WARN_DEFERRED(condition, format...) ({ \
unsigned long flags; \
\
printk_safe_enter_irqsave(flags); \
WARN(condition, format...) \
printk_safe_exit_irqrestore(flags); \
})
, where printk_safe_enter_irqsave()/printk_safe_exit_irqrestore(flags)
are currently used only internally by printk() code and defined
in the local kernel/printk/internal.h
Be ware that using the deferred variants is a whack a mole approach.
There are many printk() callers that might be called indirectly
and eventually cause deadlocks.
As already said, the plan is to upstream -rt solution and offload
the console work into kthreads. But it goes slowly. We want to
make it a clean way and prevent regressions as much as possible.
Best Regards,
Petr
next prev parent reply other threads:[~2021-11-10 10:50 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-09 20:06 printk deadlock due to double lock attempt on current CPU's runqueue Sultan Alsawaf
2021-11-09 20:06 ` Sultan Alsawaf
2021-11-09 21:38 ` Peter Zijlstra
2021-11-09 21:38 ` Peter Zijlstra
2021-11-10 9:00 ` Vincent Guittot
2021-11-10 9:00 ` Vincent Guittot
2021-11-10 10:45 ` Michal Koutný
2021-11-10 10:45 ` Michal Koutný
2021-11-10 19:50 ` Sultan Alsawaf
2021-11-10 19:50 ` Sultan Alsawaf
2021-11-12 7:50 ` Vincent Guittot
2021-11-12 7:50 ` Vincent Guittot
2021-11-10 9:37 ` Daniel Vetter
2021-11-10 9:37 ` Daniel Vetter
2021-11-10 10:07 ` John Ogness
2021-11-10 10:07 ` John Ogness
2021-11-10 10:44 ` Daniel Vetter
2021-11-10 10:44 ` Daniel Vetter
2021-11-10 20:03 ` Sultan Alsawaf
2021-11-10 20:03 ` Sultan Alsawaf
2021-11-11 8:28 ` John Ogness
2021-11-11 8:28 ` John Ogness
2021-11-11 9:27 ` Petr Mladek
2021-11-10 10:50 ` Petr Mladek [this message]
2021-11-10 10:50 ` Petr Mladek
2021-11-10 11:20 ` Peter Zijlstra
2021-11-10 11:20 ` Peter Zijlstra
2021-11-10 13:21 ` Daniel Vetter
2021-11-10 13:21 ` Daniel Vetter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YYuj/rT+EO3K0LsK@alley \
--to=pmladek@suse.com \
--cc=airlied@linux.ie \
--cc=anton@enomsg.org \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=ccross@android.com \
--cc=daniel@ffwll.ch \
--cc=dietmar.eggemann@arm.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=john.ogness@linutronix.de \
--cc=juri.lelli@redhat.com \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=maarten.lankhorst@linux.intel.com \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=mripard@kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=senozhatsky@chromium.org \
--cc=sultan@kerneltoast.com \
--cc=tony.luck@intel.com \
--cc=tzimmermann@suse.de \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.