From: Ming Lei <ming.lei@redhat.com>
To: Jens Axboe <axboe@fb.com>,
linux-block@vger.kernel.org,
Christoph Hellwig <hch@infradead.org>
Cc: Bart Van Assche <bart.vanassche@sandisk.com>,
Ming Lei <ming.lei@redhat.com>
Subject: [PATCH v2 0/8] blk-mq: fix & improve queue quiescing
Date: Sat, 27 May 2017 22:21:18 +0800 [thread overview]
Message-ID: <20170527142126.26079-1-ming.lei@redhat.com> (raw)
Hi,
There are some issues in current blk_mq_quiesce_queue():
- in case of direct issue or BLK_MQ_S_START_ON_RUN, dispatch won't
be prevented after blk_mq_quiesce_queue() is returned.
- in theory, new RCU read-side critical sections may begin while
synchronize_rcu() was waiting, and end after returning from
synchronize_rcu(), then dispatch still may be run after
synchronize_rcu() returns
It is observed that request double-free/use-after-free
can be triggered easily when canceling NVMe requests via
blk_mq_tagset_busy_iter(...nvme_cancel_request) in nvme_dev_disable().
The reason is that blk_mq_quiesce_queue() can't prevent
dispatching from being run during the period.
Actually we have to quiesce queue for canceling dispatched
requests via blk_mq_tagset_busy_iter(), otherwise use-after-free
can be made easily. This way of canceling dispatched requests
has been used in several drivers, only NVMe uses blk_mq_quiesce_queue()
to avoid the issue, and others need to be fixed too. And it
should be a common way for handling dead controller.
blk_mq_quiesce_queue() is implemented via stopping queue, which
limits its uses, and easy to casue race, because any queue restart in
other paths may break blk_mq_quiesce_queue(). For example, we sometimes
stops queue when hw can't handle too many ongoing requests and restarts
queue after requests are completed. Meantime when we want to cancel
requests if hardware is dead, quiescing has to be run first, then the
restarting in complete path can break the quiescing. This patch improves
this interface via removing stopping queue, then it can be easier
to use.
V2:
- split patch "blk-mq: fix blk_mq_quiesce_queue" into two and
fix one build issue when only applying the 1st two patches.
- add kernel oops and hang log into commit log
- add 'Revert "blk-mq: don't use sync workqueue flushing from drivers"'
Ming Lei (8):
blk-mq: introduce blk_mq_unquiesce_queue
block: introduce flag of QUEUE_FLAG_QUIESCED
blk-mq: use the introduced blk_mq_unquiesce_queue()
blk-mq: fix blk_mq_quiesce_queue
blk-mq: update comments on blk_mq_quiesce_queue()
blk-mq: don't stop queue for quiescing
blk-mq: clarify dispatch may not be drained/blocked by stopping queue
Revert "blk-mq: don't use sync workqueue flushing from drivers"
block/blk-mq.c | 89 +++++++++++++++++++++++++++++++++---------------
drivers/md/dm-rq.c | 2 +-
drivers/nvme/host/core.c | 2 +-
drivers/scsi/scsi_lib.c | 5 ++-
include/linux/blkdev.h | 3 ++
5 files changed, 71 insertions(+), 30 deletions(-)
--
2.9.4
next reply other threads:[~2017-05-27 14:21 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-27 14:21 Ming Lei [this message]
2017-05-27 14:21 ` [PATCH v2 1/8] blk-mq: introduce blk_mq_unquiesce_queue Ming Lei
2017-05-30 15:09 ` Bart Van Assche
2017-05-27 14:21 ` [PATCH v2 2/8] block: introduce flag of QUEUE_FLAG_QUIESCED Ming Lei
2017-05-30 15:11 ` Bart Van Assche
2017-05-27 14:21 ` [PATCH v2 3/8] blk-mq: use the introduced blk_mq_unquiesce_queue() Ming Lei
2017-05-27 14:21 ` Ming Lei
2017-05-30 15:12 ` Bart Van Assche
2017-05-30 15:12 ` Bart Van Assche
2017-05-30 15:12 ` Bart Van Assche
2017-05-31 2:29 ` Ming Lei
2017-05-31 2:29 ` Ming Lei
2017-05-30 19:04 ` Eduardo Valentin
2017-05-30 19:04 ` Eduardo Valentin
2017-05-30 19:04 ` Eduardo Valentin
2017-05-31 2:28 ` Ming Lei
2017-05-31 2:28 ` Ming Lei
2017-05-27 14:21 ` [PATCH v2 4/8] blk-mq: fix blk_mq_quiesce_queue Ming Lei
2017-05-27 21:46 ` Bart Van Assche
2017-05-28 10:44 ` Ming Lei
2017-05-28 16:10 ` Bart Van Assche
2017-05-30 0:22 ` Ming Lei
2017-05-30 16:54 ` Bart Van Assche
2017-05-31 2:38 ` Ming Lei
2017-05-30 19:23 ` Bart Van Assche
2017-05-31 2:52 ` Ming Lei
2017-05-27 14:21 ` [PATCH v2 5/8] blk-mq: update comments on blk_mq_quiesce_queue() Ming Lei
2017-05-30 17:14 ` Bart Van Assche
2017-05-31 9:51 ` Ming Lei
2017-05-27 14:21 ` [PATCH v2 6/8] blk-mq: don't stop queue for quiescing Ming Lei
2017-05-27 21:49 ` Bart Van Assche
2017-05-28 10:50 ` Ming Lei
2017-05-28 16:03 ` Bart Van Assche
2017-05-30 0:27 ` Ming Lei
2017-05-30 17:02 ` Bart Van Assche
2017-05-31 2:55 ` Ming Lei
2017-05-27 14:21 ` [PATCH v2 7/8] blk-mq: clarify dispatch may not be drained/blocked by stopping queue Ming Lei
2017-05-27 14:21 ` [PATCH v2 8/8] Revert "blk-mq: don't use sync workqueue flushing from drivers" Ming Lei
2017-05-27 21:32 ` [PATCH v2 0/8] blk-mq: fix & improve queue quiescing Bart Van Assche
2017-05-28 11:11 ` Ming Lei
2017-05-28 16:01 ` Bart Van Assche
2017-05-30 0:34 ` Ming Lei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170527142126.26079-1-ming.lei@redhat.com \
--to=ming.lei@redhat.com \
--cc=axboe@fb.com \
--cc=bart.vanassche@sandisk.com \
--cc=hch@infradead.org \
--cc=linux-block@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.