From: Hannes Reinecke <hare@suse.de>
To: Daniel Wagner <dwagner@suse.de>, linux-nvme@lists.infradead.org
Cc: linux-kernel@vger.kernel.org,
James Smart <james.smart@broadcom.com>,
Keith Busch <kbusch@kernel.org>, Ming Lei <ming.lei@redhat.com>,
Sagi Grimberg <sagi@grimberg.me>, Wen Xiong <wenxiong@us.ibm.com>,
James Smart <jsmart2021@gmail.com>
Subject: Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic
Date: Wed, 4 Aug 2021 09:23:49 +0200 [thread overview]
Message-ID: <79c89923-f586-79e7-6dfd-c15ceb21f569@suse.de> (raw)
In-Reply-To: <20210802112658.75875-7-dwagner@suse.de>
On 8/2/21 1:26 PM, Daniel Wagner wrote:
> From: James Smart <jsmart2021@gmail.com>
>
> commit fe35ec58f0d3 ("block: update hctx map when use multiple maps")
> exposed an issue where we may hang trying to wait for queue freeze
> during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze
> the queue. However we never started queue freeze when starting the
> reset, which means that we have inflight pending requests that entered the
> queue that we will not complete once the queue is quiesced.
>
> So start a freeze before we quiesce the queue, and unfreeze the queue
> after we successfully connected the I/O queues (the unfreeze is already
> present in the code). blk_mq_update_nr_hw_queues will be called only
> after we are sure that the queue was already frozen.
>
> This follows to how the pci driver handles resets.
>
> This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix
> controller reset hang during traffic".
>
> Signed-off-by: James Smart <jsmart2021@gmail.com>
> CC: Sagi Grimberg <sagi@grimberg.me>
> [dwagner: call nvme_unfreeze() unconditionally in
> nvme_fc_recreate_io_queues() to match the nvme_start_freeze()]
> Tested-by: Daniel Wagner <dwagner@suse.de>
> Reviewed-by: Daniel Wagner <dwagner@suse.de>
> ---
> drivers/nvme/host/fc.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index 133b87db4f1d..b292af0fd655 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -2486,6 +2486,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
> * (but with error status).
> */
> if (ctrl->ctrl.queue_count > 1) {
> + nvme_start_freeze(&ctrl->ctrl);
> nvme_stop_queues(&ctrl->ctrl);
> nvme_sync_io_queues(&ctrl->ctrl);
> blk_mq_tagset_busy_iter(&ctrl->tag_set,
> @@ -2966,8 +2967,8 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
> return -ENODEV;
> }
> blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
> - nvme_unfreeze(&ctrl->ctrl);
> }
> + nvme_unfreeze(&ctrl->ctrl);
>
> ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
> if (ret)
>
There still is now an imbalance, as we're always calling
'nvme_unfreeze()' (irrespective on the number of queues), but will only
call 'nvme_start_freeze()' if we have more than one queue.
This might lead to an imbalance on the mq_freeze_depth counter.
Wouldn't it be better to move the call to 'nvme_start_freeze()' out of
the if() condition to avoid the imbalance?
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
WARNING: multiple messages have this Message-ID (diff)
From: Hannes Reinecke <hare@suse.de>
To: Daniel Wagner <dwagner@suse.de>, linux-nvme@lists.infradead.org
Cc: linux-kernel@vger.kernel.org,
James Smart <james.smart@broadcom.com>,
Keith Busch <kbusch@kernel.org>, Ming Lei <ming.lei@redhat.com>,
Sagi Grimberg <sagi@grimberg.me>, Wen Xiong <wenxiong@us.ibm.com>,
James Smart <jsmart2021@gmail.com>
Subject: Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic
Date: Wed, 4 Aug 2021 09:23:49 +0200 [thread overview]
Message-ID: <79c89923-f586-79e7-6dfd-c15ceb21f569@suse.de> (raw)
In-Reply-To: <20210802112658.75875-7-dwagner@suse.de>
On 8/2/21 1:26 PM, Daniel Wagner wrote:
> From: James Smart <jsmart2021@gmail.com>
>
> commit fe35ec58f0d3 ("block: update hctx map when use multiple maps")
> exposed an issue where we may hang trying to wait for queue freeze
> during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze
> the queue. However we never started queue freeze when starting the
> reset, which means that we have inflight pending requests that entered the
> queue that we will not complete once the queue is quiesced.
>
> So start a freeze before we quiesce the queue, and unfreeze the queue
> after we successfully connected the I/O queues (the unfreeze is already
> present in the code). blk_mq_update_nr_hw_queues will be called only
> after we are sure that the queue was already frozen.
>
> This follows to how the pci driver handles resets.
>
> This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix
> controller reset hang during traffic".
>
> Signed-off-by: James Smart <jsmart2021@gmail.com>
> CC: Sagi Grimberg <sagi@grimberg.me>
> [dwagner: call nvme_unfreeze() unconditionally in
> nvme_fc_recreate_io_queues() to match the nvme_start_freeze()]
> Tested-by: Daniel Wagner <dwagner@suse.de>
> Reviewed-by: Daniel Wagner <dwagner@suse.de>
> ---
> drivers/nvme/host/fc.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
> index 133b87db4f1d..b292af0fd655 100644
> --- a/drivers/nvme/host/fc.c
> +++ b/drivers/nvme/host/fc.c
> @@ -2486,6 +2486,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
> * (but with error status).
> */
> if (ctrl->ctrl.queue_count > 1) {
> + nvme_start_freeze(&ctrl->ctrl);
> nvme_stop_queues(&ctrl->ctrl);
> nvme_sync_io_queues(&ctrl->ctrl);
> blk_mq_tagset_busy_iter(&ctrl->tag_set,
> @@ -2966,8 +2967,8 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)
> return -ENODEV;
> }
> blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues);
> - nvme_unfreeze(&ctrl->ctrl);
> }
> + nvme_unfreeze(&ctrl->ctrl);
>
> ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.sqsize + 1);
> if (ret)
>
There still is now an imbalance, as we're always calling
'nvme_unfreeze()' (irrespective on the number of queues), but will only
call 'nvme_start_freeze()' if we have more than one queue.
This might lead to an imbalance on the mq_freeze_depth counter.
Wouldn't it be better to move the call to 'nvme_start_freeze()' out of
the if() condition to avoid the imbalance?
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2021-08-04 7:23 UTC|newest]
Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-02 11:26 [PATCH RESEND v4 0/8] Handle update hardware queues and queue freeze more carefully Daniel Wagner
2021-08-02 11:26 ` Daniel Wagner
2021-08-02 11:26 ` [PATCH v4 1/8] nvme-fc: Update hardware queues before using them Daniel Wagner
2021-08-02 11:26 ` Daniel Wagner
2021-08-02 19:34 ` Himanshu Madhani
2021-08-02 19:34 ` Himanshu Madhani
2021-08-02 11:26 ` [PATCH v4 2/8] nvme-tcp: Update number of " Daniel Wagner
2021-08-02 11:26 ` Daniel Wagner
2021-08-06 19:57 ` Sagi Grimberg
2021-08-06 19:57 ` Sagi Grimberg
2021-08-09 8:52 ` Daniel Wagner
2021-08-09 8:52 ` Daniel Wagner
2021-08-11 1:00 ` Sagi Grimberg
2021-08-11 1:00 ` Sagi Grimberg
2021-08-11 1:07 ` Keith Busch
2021-08-11 1:07 ` Keith Busch
2021-08-11 5:57 ` Sagi Grimberg
2021-08-11 5:57 ` Sagi Grimberg
2021-08-11 10:25 ` Daniel Wagner
2021-08-11 10:25 ` Daniel Wagner
2021-08-02 11:26 ` [PATCH v4 3/8] nvme-rdma: " Daniel Wagner
2021-08-02 11:26 ` Daniel Wagner
2021-08-02 11:26 ` [PATCH v4 4/8] nvme-fc: Wait with a timeout for queue to freeze Daniel Wagner
2021-08-02 11:26 ` Daniel Wagner
2021-08-02 19:36 ` Himanshu Madhani
2021-08-02 19:36 ` Himanshu Madhani
2021-08-02 11:26 ` [PATCH v4 5/8] nvme-fc: avoid race between time out and tear down Daniel Wagner
2021-08-02 11:26 ` Daniel Wagner
2021-08-02 19:38 ` Himanshu Madhani
2021-08-02 19:38 ` Himanshu Madhani
2021-08-02 11:26 ` [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic Daniel Wagner
2021-08-02 11:26 ` Daniel Wagner
2021-08-02 19:39 ` Himanshu Madhani
2021-08-02 19:39 ` Himanshu Madhani
2021-08-04 7:23 ` Hannes Reinecke [this message]
2021-08-04 7:23 ` Hannes Reinecke
2021-08-04 8:08 ` Daniel Wagner
2021-08-04 8:08 ` Daniel Wagner
2021-08-11 1:05 ` Sagi Grimberg
2021-08-11 1:05 ` Sagi Grimberg
2021-08-11 10:30 ` Daniel Wagner
2021-08-11 10:30 ` Daniel Wagner
2021-08-12 20:03 ` James Smart
2021-08-12 20:03 ` James Smart
2021-08-18 11:43 ` Daniel Wagner
2021-08-18 11:43 ` Daniel Wagner
2021-08-18 11:49 ` Daniel Wagner
2021-08-18 11:49 ` Daniel Wagner
2021-08-02 11:26 ` [PATCH v4 7/8] nvme-tcp: Unfreeze queues on reconnect Daniel Wagner
2021-08-02 11:26 ` Daniel Wagner
2021-08-02 11:26 ` [PATCH v4 8/8] nvme-rdma: " Daniel Wagner
2021-08-02 11:26 ` Daniel Wagner
2021-08-04 7:25 ` Hannes Reinecke
2021-08-04 7:25 ` Hannes Reinecke
2021-08-06 19:59 ` Sagi Grimberg
2021-08-06 19:59 ` Sagi Grimberg
2021-08-09 8:58 ` Daniel Wagner
2021-08-09 8:58 ` Daniel Wagner
-- strict thread matches above, loose matches on Subject: below --
2021-08-02 9:14 [PATCH v4 0/8] Handle update hardware queues and queue freeze more carefully Daniel Wagner
2021-08-02 9:14 ` [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic Daniel Wagner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=79c89923-f586-79e7-6dfd-c15ceb21f569@suse.de \
--to=hare@suse.de \
--cc=dwagner@suse.de \
--cc=james.smart@broadcom.com \
--cc=jsmart2021@gmail.com \
--cc=kbusch@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=ming.lei@redhat.com \
--cc=sagi@grimberg.me \
--cc=wenxiong@us.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.