From: Hannes Reinecke <hare@suse.de>
To: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>
Cc: linux-block@vger.kernel.org, John Garry <john.garry@huawei.com>,
Bart Van Assche <bvanassche@acm.org>,
Hannes Reinecke <hare@suse.com>, Christoph Hellwig <hch@lst.de>,
Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH V11 06/12] blk-mq: prepare for draining IO when hctx's all CPUs are offline
Date: Wed, 13 May 2020 08:35:57 +0200 [thread overview]
Message-ID: <3994d91e-9f17-8c8f-eb6b-6ad22db44cf9@suse.de> (raw)
In-Reply-To: <20200513034803.1844579-7-ming.lei@redhat.com>
On 5/13/20 5:47 AM, Ming Lei wrote:
> Most of blk-mq drivers depend on managed IRQ's auto-affinity to setup
> up queue mapping. Thomas mentioned the following point[1]:
>
> "
> That was the constraint of managed interrupts from the very beginning:
>
> The driver/subsystem has to quiesce the interrupt line and the associated
> queue _before_ it gets shutdown in CPU unplug and not fiddle with it
> until it's restarted by the core when the CPU is plugged in again.
> "
>
> However, current blk-mq implementation doesn't quiesce hw queue before
> the last CPU in the hctx is shutdown. Even worse, CPUHP_BLK_MQ_DEAD is
> one cpuhp state handled after the CPU is down, so there isn't any chance
> to quiesce hctx for blk-mq wrt. CPU hotplug.
>
> Add new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE for blk-mq to stop queues
> and wait for completion of in-flight requests.
>
> We will stop hw queue and wait for completion of in-flight requests
> when one hctx is becoming dead in the following patch. This way may
> cause dead-lock for some stacking blk-mq drivers, such as dm-rq and
> loop.
>
> Add blk-mq flag of BLK_MQ_F_NO_MANAGED_IRQ and mark it for dm-rq and
> loop, so we needn't to wait for completion of in-flight requests from
> dm-rq & loop, then the potential dead-lock can be avoided.
>
> [1] https://lore.kernel.org/linux-block/alpine.DEB.2.21.1904051331270.1802@nanos.tec.linutronix.de/
>
> Cc: John Garry <john.garry@huawei.com>
> Cc: Bart Van Assche <bvanassche@acm.org>
> Cc: Hannes Reinecke <hare@suse.com>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Signed-off-by: Ming Lei <ming.lei@redhat.com>
> ---
> block/blk-mq-debugfs.c | 1 +
> block/blk-mq.c | 19 +++++++++++++++++++
> drivers/block/loop.c | 2 +-
> drivers/md/dm-rq.c | 2 +-
> include/linux/blk-mq.h | 4 ++++
> include/linux/cpuhotplug.h | 1 +
> 6 files changed, 27 insertions(+), 2 deletions(-)
>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
next prev parent reply other threads:[~2020-05-13 6:36 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-13 3:47 [PATCH V11 00/12] blk-mq: improvement CPU hotplug Ming Lei
2020-05-13 3:47 ` [PATCH V11 01/12] block: clone nr_integrity_segments and write_hint in blk_rq_prep_clone Ming Lei
2020-05-13 3:47 ` [PATCH V11 02/12] block: add helper for copying request Ming Lei
2020-05-13 3:47 ` [PATCH V11 03/12] blk-mq: mark blk_mq_get_driver_tag as static Ming Lei
2020-05-13 3:47 ` [PATCH V11 04/12] blk-mq: assign rq->tag in blk_mq_get_driver_tag Ming Lei
2020-05-13 3:47 ` [PATCH V11 05/12] blk-mq: add blk_mq_all_tag_iter Ming Lei
2020-05-13 11:56 ` Christoph Hellwig
2020-05-13 3:47 ` [PATCH V11 06/12] blk-mq: prepare for draining IO when hctx's all CPUs are offline Ming Lei
2020-05-13 6:35 ` Hannes Reinecke [this message]
2020-05-13 11:58 ` Christoph Hellwig
2020-05-14 0:33 ` Ming Lei
2020-05-13 3:47 ` [PATCH V11 07/12] blk-mq: stop to handle IO and drain IO before hctx becomes inactive Ming Lei
2020-05-13 11:59 ` Christoph Hellwig
2020-05-14 0:36 ` Ming Lei
2020-05-14 1:12 ` Bart Van Assche
2020-05-14 3:10 ` Ming Lei
2020-05-13 3:47 ` [PATCH V11 08/12] block: add blk_end_flush_machinery Ming Lei
2020-05-13 12:00 ` Christoph Hellwig
2020-05-13 3:48 ` [PATCH V11 09/12] blk-mq: add blk_mq_hctx_handle_dead_cpu for handling cpu dead Ming Lei
2020-05-13 12:06 ` Christoph Hellwig
2020-05-13 3:48 ` [PATCH V11 10/12] block: add request allocation flag of BLK_MQ_REQ_FORCE Ming Lei
2020-05-13 10:34 ` [PATCH V12 " Ming Lei
2020-05-13 3:48 ` [PATCH V11 11/12] blk-mq: re-submit IO in case that hctx is inactive Ming Lei
2020-05-13 9:21 ` John Garry
2020-05-13 12:21 ` Christoph Hellwig
2020-05-13 15:03 ` Bart Van Assche
2020-05-14 0:45 ` Ming Lei
2020-05-14 0:40 ` Ming Lei
2020-05-13 3:48 ` [PATCH V11 12/12] block: deactivate hctx when the hctx is actually inactive Ming Lei
2020-05-13 7:34 ` [PATCH V11 00/12] blk-mq: improvement CPU hotplug John Garry
2020-05-13 10:37 ` Ming Lei
2020-05-13 11:33 ` John Garry
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3994d91e-9f17-8c8f-eb6b-6ad22db44cf9@suse.de \
--to=hare@suse.de \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=hare@suse.com \
--cc=hch@lst.de \
--cc=john.garry@huawei.com \
--cc=linux-block@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.