From: Hannes Reinecke <hare@suse.de>
To: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>
Cc: linux-block@vger.kernel.org, John Garry <john.garry@huawei.com>,
Bart Van Assche <bvanassche@acm.org>,
Hannes Reinecke <hare@suse.com>, Christoph Hellwig <hch@lst.de>,
Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH V10 06/11] blk-mq: prepare for draining IO when hctx's all CPUs are offline
Date: Tue, 5 May 2020 08:14:00 +0200 [thread overview]
Message-ID: <210b7a08-91ed-be02-49bc-bf30acac223c@suse.de> (raw)
In-Reply-To: <20200505020930.1146281-7-ming.lei@redhat.com>
On 5/5/20 4:09 AM, Ming Lei wrote:
> Most of blk-mq drivers depend on managed IRQ's auto-affinity to setup
> up queue mapping. Thomas mentioned the following point[1]:
>
> "
> That was the constraint of managed interrupts from the very beginning:
>
> The driver/subsystem has to quiesce the interrupt line and the associated
> queue _before_ it gets shutdown in CPU unplug and not fiddle with it
> until it's restarted by the core when the CPU is plugged in again.
> "
>
> However, current blk-mq implementation doesn't quiesce hw queue before
> the last CPU in the hctx is shutdown. Even worse, CPUHP_BLK_MQ_DEAD is
> one cpuhp state handled after the CPU is down, so there isn't any chance
> to quiesce hctx for blk-mq wrt. CPU hotplug.
>
> Add new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE for blk-mq to stop queues
> and wait for completion of in-flight requests.
>
> We will stop hw queue and wait for completion of in-flight requests
> when one hctx is becoming dead in the following patch. This way may
> cause dead-lock for some stacking blk-mq drivers, such as dm-rq and
> loop.
>
> Add blk-mq flag of BLK_MQ_F_NO_MANAGED_IRQ and mark it for dm-rq and
> loop, so we needn't to wait for completion of in-flight requests from
> dm-rq & loop, then the potential dead-lock can be avoided.
>
> [1] https://lore.kernel.org/linux-block/alpine.DEB.2.21.1904051331270.1802@nanos.tec.linutronix.de/
>
> Cc: John Garry <john.garry@huawei.com>
> Cc: Bart Van Assche <bvanassche@acm.org>
> Cc: Hannes Reinecke <hare@suse.com>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Tested-by: John Garry <john.garry@huawei.com>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
> block/blk-mq-debugfs.c | 1 +
> block/blk-mq.c | 19 +++++++++++++++++++
> drivers/block/loop.c | 2 +-
> drivers/md/dm-rq.c | 2 +-
> include/linux/blk-mq.h | 3 +++
> include/linux/cpuhotplug.h | 1 +
> 6 files changed, 26 insertions(+), 2 deletions(-)
> Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
next prev parent reply other threads:[~2020-05-05 6:14 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-05 2:09 [PATCH V10 00/11] blk-mq: improvement CPU hotplug Ming Lei
2020-05-05 2:09 ` [PATCH V10 01/11] block: clone nr_integrity_segments and write_hint in blk_rq_prep_clone Ming Lei
2020-05-05 2:09 ` [PATCH V10 02/11] block: add helper for copying request Ming Lei
2020-05-05 2:09 ` [PATCH V10 03/11] blk-mq: mark blk_mq_get_driver_tag as static Ming Lei
2020-05-05 2:09 ` [PATCH V10 04/11] blk-mq: assign rq->tag in blk_mq_get_driver_tag Ming Lei
2020-05-05 2:09 ` [PATCH V10 05/11] blk-mq: support rq filter callback when iterating rqs Ming Lei
2020-05-08 23:32 ` Bart Van Assche
2020-05-09 0:18 ` Bart Van Assche
2020-05-09 2:05 ` Ming Lei
2020-05-09 3:08 ` Bart Van Assche
2020-05-09 3:52 ` Ming Lei
2020-05-05 2:09 ` [PATCH V10 06/11] blk-mq: prepare for draining IO when hctx's all CPUs are offline Ming Lei
2020-05-05 6:14 ` Hannes Reinecke [this message]
2020-05-08 23:26 ` Bart Van Assche
2020-05-09 2:09 ` Ming Lei
2020-05-09 3:11 ` Bart Van Assche
2020-05-09 3:56 ` Ming Lei
2020-05-05 2:09 ` [PATCH V10 07/11] blk-mq: stop to handle IO and drain IO before hctx becomes inactive Ming Lei
2020-05-08 23:39 ` Bart Van Assche
2020-05-09 2:20 ` Ming Lei
2020-05-09 3:24 ` Bart Van Assche
2020-05-09 4:10 ` Ming Lei
2020-05-09 14:18 ` Bart Van Assche
2020-05-11 1:45 ` Ming Lei
2020-05-11 3:20 ` Bart Van Assche
2020-05-11 3:48 ` Ming Lei
2020-05-11 20:56 ` Bart Van Assche
2020-05-12 1:25 ` Ming Lei
2020-05-05 2:09 ` [PATCH V10 08/11] block: add blk_end_flush_machinery Ming Lei
2020-05-05 2:09 ` [PATCH V10 09/11] blk-mq: add blk_mq_hctx_handle_dead_cpu for handling cpu dead Ming Lei
2020-05-05 2:09 ` [PATCH V10 10/11] blk-mq: re-submit IO in case that hctx is inactive Ming Lei
2020-05-05 2:09 ` [PATCH V10 11/11] block: deactivate hctx when the hctx is actually inactive Ming Lei
2020-05-09 14:07 ` Bart Van Assche
2020-05-11 2:11 ` Ming Lei
2020-05-11 3:30 ` Bart Van Assche
2020-05-11 4:08 ` Ming Lei
2020-05-11 20:52 ` Bart Van Assche
2020-05-12 1:43 ` Ming Lei
2020-05-12 2:08 ` Ming Lei
2020-05-08 21:49 ` [PATCH V10 00/11] blk-mq: improvement CPU hotplug Ming Lei
2020-05-09 3:17 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=210b7a08-91ed-be02-49bc-bf30acac223c@suse.de \
--to=hare@suse.de \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=hare@suse.com \
--cc=hch@lst.de \
--cc=john.garry@huawei.com \
--cc=linux-block@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.