From: Daniel Wagner <dwagner@suse.de>
To: Christoph Hellwig <hch@lst.de>
Cc: Jens Axboe <axboe@kernel.dk>, Ming Lei <ming.lei@redhat.com>,
linux-block@vger.kernel.org, John Garry <john.garry@huawei.com>,
Bart Van Assche <bvanassche@acm.org>,
Hannes Reinecke <hare@suse.com>,
Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH 8/8] blk-mq: drain I/O when all CPUs in a hctx are offline
Date: Fri, 29 May 2020 16:41:39 +0200 [thread overview]
Message-ID: <20200529144139.wg7qe55kkicvi6vw@beryllium.lan> (raw)
In-Reply-To: <20200529135315.199230-9-hch@lst.de>
On Fri, May 29, 2020 at 03:53:15PM +0200, Christoph Hellwig wrote:
> From: Ming Lei <ming.lei@redhat.com>
>
> Most of blk-mq drivers depend on managed IRQ's auto-affinity to setup
> up queue mapping. Thomas mentioned the following point[1]:
>
> "That was the constraint of managed interrupts from the very beginning:
>
> The driver/subsystem has to quiesce the interrupt line and the associated
> queue _before_ it gets shutdown in CPU unplug and not fiddle with it
> until it's restarted by the core when the CPU is plugged in again."
>
> However, current blk-mq implementation doesn't quiesce hw queue before
> the last CPU in the hctx is shutdown. Even worse, CPUHP_BLK_MQ_DEAD is a
> cpuhp state handled after the CPU is down, so there isn't any chance to
> quiesce the hctx before shutting down the CPU.
>
> Add new CPUHP_AP_BLK_MQ_ONLINE state to stop allocating from blk-mq hctxs
> where the last CPU goes away, and wait for completion of in-flight
> requests. This guarantees that there is no inflight I/O before shutting
> down the managed IRQ.
>
> Add a BLK_MQ_F_STACKING and set it for dm-rq and loop, so we don't need
> to wait for completion of in-flight requests from these drivers to avoid
> a potential dead-lock. It is safe to do this for stacking drivers as those
> do not use interrupts at all and their I/O completions are triggered by
> underlying devices I/O completion.
>
> [1] https://lore.kernel.org/linux-block/alpine.DEB.2.21.1904051331270.1802@nanos.tec.linutronix.de/
>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> [hch: different retry mechanism, merged two patches, minor cleanups]
> Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
next prev parent reply other threads:[~2020-05-29 14:43 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-29 13:53 blk-mq: improvement CPU hotplug (simplified version) v4 Christoph Hellwig
2020-05-29 13:53 ` [PATCH 1/8] blk-mq: remove the bio argument to ->prepare_request Christoph Hellwig
2020-05-29 13:53 ` [PATCH 2/8] blk-mq: simplify the blk_mq_get_request calling convention Christoph Hellwig
2020-05-29 13:53 ` [PATCH 3/8] blk-mq: move more request initialization to blk_mq_rq_ctx_init Christoph Hellwig
2020-05-29 13:53 ` [PATCH 4/8] blk-mq: rename BLK_MQ_TAG_FAIL to BLK_MQ_NO_TAG Christoph Hellwig
2020-05-29 13:53 ` [PATCH 5/8] blk-mq: use BLK_MQ_NO_TAG in more places Christoph Hellwig
2020-05-29 13:53 ` [PATCH 6/8] blk-mq: open code __blk_mq_alloc_request in blk_mq_alloc_request_hctx Christoph Hellwig
2020-05-29 13:53 ` [PATCH 7/8] blk-mq: add blk_mq_all_tag_iter Christoph Hellwig
2020-05-29 16:14 ` Bart Van Assche
2020-05-29 13:53 ` [PATCH 8/8] blk-mq: drain I/O when all CPUs in a hctx are offline Christoph Hellwig
2020-05-29 14:34 ` Hannes Reinecke
2020-05-29 14:41 ` Daniel Wagner [this message]
2020-05-29 16:23 ` blk-mq: improvement CPU hotplug (simplified version) v4 Jens Axboe
-- strict thread matches above, loose matches on Subject: below --
2020-05-27 18:06 Christoph Hellwig
2020-05-27 18:06 ` [PATCH 8/8] blk-mq: drain I/O when all CPUs in a hctx are offline Christoph Hellwig
2020-05-27 18:26 ` Hannes Reinecke
2020-05-27 23:09 ` Bart Van Assche
2020-05-28 1:46 ` Ming Lei
2020-05-28 3:33 ` Bart Van Assche
2020-05-28 5:19 ` Ming Lei
2020-05-28 13:37 ` Bart Van Assche
2020-05-28 17:21 ` Paul E. McKenney
2020-05-29 1:53 ` Ming Lei
2020-05-29 3:07 ` Paul E. McKenney
2020-05-29 3:53 ` Ming Lei
2020-05-29 18:13 ` Paul E. McKenney
2020-05-29 19:55 ` Bart Van Assche
2020-05-29 21:12 ` Paul E. McKenney
2020-05-29 1:13 ` Ming Lei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200529144139.wg7qe55kkicvi6vw@beryllium.lan \
--to=dwagner@suse.de \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=hare@suse.com \
--cc=hch@lst.de \
--cc=john.garry@huawei.com \
--cc=linux-block@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.