From: Petr Mladek <pmladek@suse.com>
To: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: DRI Development <dri-devel@lists.freedesktop.org>,
Intel Graphics Development <intel-gfx@lists.freedesktop.org>,
Daniel Vetter <daniel.vetter@intel.com>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>, Will Deacon <will.deacon@arm.com>,
Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
Steven Rostedt <rostedt@goodmis.org>,
John Ogness <john.ogness@linutronix.de>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] RFC: console: hack up console_trylock more
Date: Mon, 6 May 2019 09:48:09 +0200 [thread overview]
Message-ID: <20190506074809.huawsdaynyci5kwz@pathway.suse.cz> (raw)
In-Reply-To: <CAKMK7uF8AD6033_tJw1Y7VsAXb6OD_syZtG3a-JM2g9eEb-P9g@mail.gmail.com>
On Mon 2019-05-06 09:11:37, Daniel Vetter wrote:
> On Fri, May 3, 2019 at 5:14 PM Petr Mladek <pmladek@suse.com> wrote:
> > On Thu 2019-05-02 16:16:43, Daniel Vetter wrote:
> > > console_trylock, called from within printk, can be called from pretty
> > > much anywhere. Including try_to_wake_up. Note that this isn't common,
> > > usually the box is in pretty bad shape at that point already. But it
> > > really doesn't help when then lockdep jumps in and spams the logs,
> > > potentially obscuring the real backtrace we're really interested in.
> > > One case I've seen (slightly simplified backtrace):
> > >
> > > Call Trace:
> > > <IRQ>
> > > console_trylock+0xe/0x60
> > > vprintk_emit+0xf1/0x320
> > > printk+0x4d/0x69
> > > __warn_printk+0x46/0x90
> > > native_smp_send_reschedule+0x2f/0x40
> > > check_preempt_curr+0x81/0xa0
> > > ttwu_do_wakeup+0x14/0x220
> > > try_to_wake_up+0x218/0x5f0
> > > pollwake+0x6f/0x90
> > > credit_entropy_bits+0x204/0x310
> > > add_interrupt_randomness+0x18f/0x210
> > > handle_irq+0x67/0x160
> > > do_IRQ+0x5e/0x130
> > > common_interrupt+0xf/0xf
> > > </IRQ>
> > >
> > > This alone isn't a problem, but the spinlock in the semaphore is also
> > > still held while waking up waiters (up() -> __up() -> try_to_wake_up()
> > > callchain), which then closes the runqueue vs. semaphore.lock loop,
> > > and upsets lockdep, which issues a circular locking splat to dmesg.
> > > Worse it upsets developers, since we don't want to spam dmesg with
> > > clutter when the machine is dying already.
> > >
> > > Fix this by creating a __down_trylock which only trylocks the
> > > semaphore.lock. This isn't correct in full generality, but good enough
> > > for console_lock:
> > >
> > > - there's only ever one console_lock holder, we won't fail spuriously
> > > because someone is doing a down() or up() while there's still room
> > > (unlike other semaphores with count > 1).
> > >
> > > - console_unlock() has one massive retry loop, which will catch anyone
> > > who races the trylock against the up(). This makes sure that no
> > > printk lines will get lost. Making the trylock more racy therefore
> > > has no further impact.
> >
> > To be honest, I do not see how this could solve the problem.
> >
> > The circular dependency is still there. If the new __down_trylock()
> > succeeds then console_unlock() will get called in the same context
> > and it will still need to call up() -> try_to_wake_up().
> >
> > Note that there are many other console_lock() callers that might
> > happen in parallel and might appear in the wait queue.
>
> Hm right. It's very rare we hit this in our CI and I don't know how to
> repro otherwise, so just threw this out at the wall to see if it
> sticks. I'll try and come up with a new trick then.
Single messages are printed from scheduler via printk_deferred().
WARN() might be solved by introducing printk deferred context,
see the per-cpu variable printk_context.
Best Regards,
Petr
next prev parent reply other threads:[~2019-05-06 7:48 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-02 14:16 [PATCH] RFC: console: hack up console_trylock more Daniel Vetter
2019-05-03 15:14 ` Petr Mladek
2019-05-06 7:11 ` Daniel Vetter
2019-05-06 7:48 ` Petr Mladek [this message]
2019-05-06 8:40 ` Daniel Vetter
2019-05-06 7:45 ` [PATCH] RFC: console: hack up console_lock more v2 Daniel Vetter
2019-05-06 8:16 ` Petr Mladek
2019-05-06 8:26 ` Petr Mladek
2019-05-06 9:38 ` Daniel Vetter
2019-05-06 11:24 ` Petr Mladek
2019-05-08 8:17 ` [Intel-gfx] " Daniel Vetter
2019-05-09 15:08 ` Petr Mladek
2019-05-06 8:20 ` Petr Mladek
2019-05-09 10:32 ` [Intel-gfx] " Chris Wilson
2019-05-09 13:05 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190506074809.huawsdaynyci5kwz@pathway.suse.cz \
--to=pmladek@suse.com \
--cc=daniel.vetter@ffwll.ch \
--cc=daniel.vetter@intel.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=intel-gfx@lists.freedesktop.org \
--cc=john.ogness@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=sergey.senozhatsky@gmail.com \
--cc=will.deacon@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).