From: Jens Axboe <jens.axboe@oracle.com>
To: Tejun Heo <htejun@gmail.com>
Cc: David Greaves <david@dgreaves.com>,
"Rafael J. Wysocki" <rjw@sisk.pl>,
Linus Torvalds <torvalds@linux-foundation.org>,
David Chinner <dgc@sgi.com>,
xfs@oss.sgi.com,
"'linux-kernel@vger.kernel.org'" <linux-kernel@vger.kernel.org>,
linux-pm <linux-pm@lists.osdl.org>, Neil Brown <neilb@suse.de>,
Jeff Garzik <jgarzik@pobox.com>
Subject: Re: [PATCH] block: always requeue !fs requests at the front
Date: Fri, 15 Jun 2007 13:05:44 +0200 [thread overview]
Message-ID: <20070615110544.GR6149@kernel.dk> (raw)
In-Reply-To: <20070615094246.GN29122@htj.dyndns.org>
On Fri, Jun 15 2007, Tejun Heo wrote:
> SCSI marks internal commands with REQ_PREEMPT and push it at the front
> of the request queue using blk_execute_rq(). When entering suspended
> or frozen state, SCSI devices are quiesced using
> scsi_device_quiesce(). In quiesced state, only REQ_PREEMPT requests
> are processed. This is how SCSI blocks other requests out while
> suspending and resuming. As all internal commands are pushed at the
> front of the queue, this usually works.
>
> Unfortunately, this interacts badly with ordered requeueing. To
> preserve request order on requeueing (due to busy device, active EH or
> other failures), requests are sorted according to ordered sequence on
> requeue if IO barrier is in progress.
>
> The following sequence deadlocks.
>
> 1. IO barrier sequence issues.
>
> 2. Suspend requested. Queue is quiesced with part of all of IO
> barrier sequence at the front.
>
> 3. During suspending or resuming, SCSI issues internal command which
> gets deferred and requeued for some reason. As the command is
> issued after the IO barrier in #1, ordered requeueing code puts the
> request after IO barrier sequence.
>
> 4. The device is ready to process requests again but still is in
> quiesced state and the first request of the queue isn't
> REQ_PREEMPT, so command processing is deadlocked -
> suspending/resuming waits for the issued request to complete while
> the request can't be processed till device is put back into
> running state by resuming.
>
> This can be fixed by always putting !fs requests at the front when
> requeueing.
>
> The following thread reports this deadlock.
>
> http://thread.gmane.org/gmane.linux.kernel/537473
>
> Signed-off-by: Tejun Heo <htejun@gmail.com>
> Cc: Jenn Axboe <jens.axboe@oracle.com>
> Cc: David Greaves <david@dgreaves.com>
> ---
> Okay, it took a lot of hours of debugging but boiled down to two liner
> fix. I feel so empty. :-) RAID6 triggers this reliably because it
> uses BIO_BARRIER heavily to update its superblock. The recent ATA
> suspend/resume rewrite is hit by this because it uses SCSI internal
> commands to spin down and up the drives for suspending and resuming.
>
> David, please test this. Jens, does it look okay?
Yep looks good, except for the bad multi-line comment style, but that's
minor stuff ;-)
Acked-by: Jens Axboe <jens.axboe@oracle.com>
--
Jens Axboe
next prev parent reply other threads:[~2007-06-15 11:05 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-06-01 21:23 2.6.22-rc3 hibernate(?) disables skge wol David Greaves
2007-06-01 21:42 ` Rafael J. Wysocki
2007-06-01 22:37 ` 2.6.22-rc3 hibernate(?) fails totally - regression David Greaves
2007-06-01 23:22 ` Rafael J. Wysocki
2007-06-02 22:31 ` David Greaves
2007-06-02 22:46 ` Linus Torvalds
2007-06-03 15:03 ` David Greaves
2007-06-06 8:33 ` Tejun Heo
2007-06-06 10:18 ` [PATCH] sata_promise: use TF interface for polling NODATA commands Tejun Heo
2007-06-06 10:19 ` Tejun Heo
2007-06-06 10:39 ` 2.6.22-rc3 hibernate(?) fails totally - regression (xfs on raid6) David Greaves
2007-06-07 5:53 ` Tejun Heo
2007-06-07 10:30 ` David Greaves
2007-06-07 11:07 ` David Chinner
2007-06-07 13:59 ` David Greaves
2007-06-07 22:28 ` David Chinner
2007-06-08 19:09 ` David Greaves
2007-06-12 18:43 ` Linus Torvalds
2007-06-13 11:16 ` David Greaves
2007-06-13 21:04 ` Linus Torvalds
2007-06-13 21:22 ` Jeff Garzik
2007-06-13 22:02 ` David Greaves
2007-06-13 22:12 ` Linus Torvalds
2007-06-13 23:15 ` Rafael J. Wysocki
2007-06-14 14:21 ` Tejun Heo
2007-06-14 15:10 ` Tejun Heo
2007-06-15 9:42 ` [PATCH] block: always requeue !fs requests at the front Tejun Heo
2007-06-15 11:05 ` Jens Axboe [this message]
2007-06-15 11:17 ` Tejun Heo
2007-06-15 11:21 ` Jens Axboe
2007-06-15 15:08 ` Jeff Garzik
2007-06-16 19:54 ` Christoph Hellwig
2007-06-17 7:29 ` Jens Axboe
2007-06-17 8:03 ` Tejun Heo
2007-06-15 13:58 ` David Greaves
2007-06-14 15:19 ` 2.6.22-rc3 hibernate(?) fails totally - regression (xfs on raid6) David Greaves
2007-06-14 0:28 ` David Chinner
2007-06-12 12:31 ` David Greaves
2007-06-10 18:43 ` Pavel Machek
2007-06-12 18:00 ` David Greaves
2007-06-12 21:31 ` Pavel Machek
2007-06-07 13:45 ` Duane Griffin
2007-06-07 14:00 ` David Greaves
2007-06-07 14:05 ` Tejun Heo
2007-06-07 14:36 ` Mark Lord
2007-06-07 15:20 ` David Greaves
2007-06-07 16:58 ` Rafael J. Wysocki
2007-06-07 20:12 ` Pavel Machek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20070615110544.GR6149@kernel.dk \
--to=jens.axboe@oracle.com \
--cc=david@dgreaves.com \
--cc=dgc@sgi.com \
--cc=htejun@gmail.com \
--cc=jgarzik@pobox.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pm@lists.osdl.org \
--cc=neilb@suse.de \
--cc=rjw@sisk.pl \
--cc=torvalds@linux-foundation.org \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).