From: Mike Galbraith <efault@gmx.de>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Jan Kara <jack@suse.cz>, Jeff Moyer <jmoyer@redhat.com>,
LKML <linux-kernel@vger.kernel.org>,
linux-fsdevel@vger.kernel.org, Tejun Heo <tj@kernel.org>,
Jens Axboe <jaxboe@fusionio.com>,
mgalbraith@suse.com, Steven Rostedt <rostedt@goodmis.org>
Subject: Re: Deadlocks due to per-process plugging
Date: Mon, 23 Jul 2012 11:46:48 +0200 [thread overview]
Message-ID: <1343036808.7336.80.camel@marge.simpson.net> (raw)
In-Reply-To: <1342982589.7210.25.camel@marge.simpson.net>
On Sun, 2012-07-22 at 20:43 +0200, Mike Galbraith wrote:
> On Sat, 2012-07-21 at 09:47 +0200, Mike Galbraith wrote:
> > On Wed, 2012-07-18 at 07:30 +0200, Mike Galbraith wrote:
> > > On Wed, 2012-07-18 at 06:44 +0200, Mike Galbraith wrote:
> > >
> > > > The patch in question for missing Cc. Maybe should be only mutex, but I
> > > > see no reason why IO dependency can only possibly exist for mutexes...
> > >
> > > Well that was easy, box quickly said "nope, mutex only does NOT cut it".
> >
> > And I also learned (ouch) that both doesn't cut it either. Ksoftirqd
> > (or sirq-blk) being nailed by q->lock in blk_done_softirq() is.. not
> > particularly wonderful. As long as that doesn't happen, IO deadlock
> > doesn't happen, troublesome filesystems just work. If it does happen
> > though, you've instantly got a problem.
>
> That problem being slab_lock in practice btw, though I suppose it could
> do the same with any number of others. In encountered case, ksoftirqd
> (or sirq-blk) blocks on slab_lock while holding q->queue_lock, while a
> userspace task (dbench) blocks on q->queue_lock while holding slab_lock
> on the same cpu. Game over.
Hello vacationing rt wizards' mail boxen (and others so bored they're
actually reading about obscure -rt IO troubles;).
ext4 is still alive, which is a positive sign, and box hasn't yet
deadlocked either, another sign. Now all I have to do is (sigh) grind
filesystems to fine powder for a few days.. again.
---
kernel/rtmutex.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
--- a/kernel/rtmutex.c
+++ b/kernel/rtmutex.c
@@ -649,7 +649,14 @@ static inline void rt_spin_lock_fastlock
if (likely(rt_mutex_cmpxchg(lock, NULL, current)))
rt_mutex_deadlock_account_lock(lock, current);
else {
- if (blk_needs_flush_plug(current))
+ /*
+ * We can't pull the plug if we're already holding a lock
+ * else we can deadlock. eg, if we're holding slab_lock,
+ * ksoftirqd can block while processing BLOCK_SOFTIRQ after
+ * having acquired q->queue_lock. If _we_ then block on
+ * that q->queue_lock while flushing our plug, deadlock.
+ */
+ if (__migrate_disabled(current) < 2 && blk_needs_flush_plug(current))
blk_schedule_flush_plug(current);
slowfn(lock);
}
next prev parent reply other threads:[~2012-07-23 9:47 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-07-11 13:37 Deadlocks due to per-process plugging Jan Kara
2012-07-11 16:05 ` Jeff Moyer
2012-07-11 20:16 ` Jan Kara
2012-07-11 22:12 ` Thomas Gleixner
2012-07-12 4:12 ` Mike Galbraith
2012-07-13 12:38 ` Jan Kara
2012-07-12 2:07 ` Mike Galbraith
2012-07-12 14:15 ` Thomas Gleixner
2012-07-13 12:33 ` Jan Kara
2012-07-13 14:25 ` Thomas Gleixner
2012-07-13 14:46 ` Jan Kara
2012-07-15 8:59 ` Thomas Gleixner
2012-07-15 9:14 ` Mike Galbraith
2012-07-15 9:51 ` Thomas Gleixner
2012-07-16 2:22 ` Mike Galbraith
2012-07-16 8:59 ` Thomas Gleixner
2012-07-16 9:48 ` Mike Galbraith
2012-07-16 9:59 ` Thomas Gleixner
2012-07-16 10:13 ` Mike Galbraith
2012-07-16 10:08 ` Mike Galbraith
2012-07-16 10:19 ` Thomas Gleixner
2012-07-16 10:30 ` Mike Galbraith
2012-07-16 11:24 ` Mike Galbraith
2012-07-16 14:35 ` Mike Galbraith
2012-07-17 13:10 ` Mike Galbraith
2012-07-18 4:44 ` Mike Galbraith
2012-07-18 5:30 ` Mike Galbraith
2012-07-21 7:47 ` Mike Galbraith
2012-07-22 18:43 ` Mike Galbraith
2012-07-23 9:46 ` Mike Galbraith [this message]
2012-07-14 11:00 ` Mike Galbraith
2012-07-14 11:06 ` Mike Galbraith
2012-07-15 7:14 ` Mike Galbraith
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1343036808.7336.80.camel@marge.simpson.net \
--to=efault@gmx.de \
--cc=jack@suse.cz \
--cc=jaxboe@fusionio.com \
--cc=jmoyer@redhat.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mgalbraith@suse.com \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).