From: Daniel Wagner <dwagner@suse.de> To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, James Smart <james.smart@broadcom.com>, Keith Busch <kbusch@kernel.org>, Ming Lei <ming.lei@redhat.com>, Sagi Grimberg <sagi@grimberg.me>, Hannes Reinecke <hare@suse.de>, Wen Xiong <wenxiong@us.ibm.com>, Himanshu Madhani <himanshu.madhani@oracle.com>, James Smart <jsmart2021@gmail.com>, Daniel Wagner <dwagner@suse.de> Subject: [PATCH v5 3/3] nvme-fc: fix controller reset hang during traffic Date: Wed, 18 Aug 2021 14:05:30 +0200 [thread overview] Message-ID: <20210818120530.130501-4-dwagner@suse.de> (raw) In-Reply-To: <20210818120530.130501-1-dwagner@suse.de> From: James Smart <jsmart2021@gmail.com> commit fe35ec58f0d3 ("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced. So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (the unfreeze is already present in the code). blk_mq_update_nr_hw_queues will be called only after we are sure that the queue was already frozen. This follows to how the pci driver handles resets. This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix controller reset hang during traffic". Signed-off-by: James Smart <jsmart2021@gmail.com> CC: Sagi Grimberg <sagi@grimberg.me> Tested-by: Daniel Wagner <dwagner@suse.de> Reviewed-by: Daniel Wagner <dwagner@suse.de> --- drivers/nvme/host/fc.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 3ff783a2e9f7..99dadab2724c 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2974,9 +2974,10 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) return -ENODEV; } blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues); - nvme_unfreeze(&ctrl->ctrl); } + nvme_unfreeze(&ctrl->ctrl); + return 0; out_delete_hw_queues: @@ -3215,6 +3216,9 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl) ctrl->iocnt = 0; spin_unlock_irqrestore(&ctrl->lock, flags); + if (ctrl->ctrl.queue_count > 1) + nvme_start_freeze(&ctrl->ctrl); + __nvme_fc_abort_outstanding_ios(ctrl, false); /* kill the aens as they are a separate path */ -- 2.29.2
WARNING: multiple messages have this Message-ID (diff)
From: Daniel Wagner <dwagner@suse.de> To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, James Smart <james.smart@broadcom.com>, Keith Busch <kbusch@kernel.org>, Ming Lei <ming.lei@redhat.com>, Sagi Grimberg <sagi@grimberg.me>, Hannes Reinecke <hare@suse.de>, Wen Xiong <wenxiong@us.ibm.com>, Himanshu Madhani <himanshu.madhani@oracle.com>, James Smart <jsmart2021@gmail.com>, Daniel Wagner <dwagner@suse.de> Subject: [PATCH v5 3/3] nvme-fc: fix controller reset hang during traffic Date: Wed, 18 Aug 2021 14:05:30 +0200 [thread overview] Message-ID: <20210818120530.130501-4-dwagner@suse.de> (raw) In-Reply-To: <20210818120530.130501-1-dwagner@suse.de> From: James Smart <jsmart2021@gmail.com> commit fe35ec58f0d3 ("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced. So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (the unfreeze is already present in the code). blk_mq_update_nr_hw_queues will be called only after we are sure that the queue was already frozen. This follows to how the pci driver handles resets. This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix controller reset hang during traffic". Signed-off-by: James Smart <jsmart2021@gmail.com> CC: Sagi Grimberg <sagi@grimberg.me> Tested-by: Daniel Wagner <dwagner@suse.de> Reviewed-by: Daniel Wagner <dwagner@suse.de> --- drivers/nvme/host/fc.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 3ff783a2e9f7..99dadab2724c 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2974,9 +2974,10 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) return -ENODEV; } blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues); - nvme_unfreeze(&ctrl->ctrl); } + nvme_unfreeze(&ctrl->ctrl); + return 0; out_delete_hw_queues: @@ -3215,6 +3216,9 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl) ctrl->iocnt = 0; spin_unlock_irqrestore(&ctrl->lock, flags); + if (ctrl->ctrl.queue_count > 1) + nvme_start_freeze(&ctrl->ctrl); + __nvme_fc_abort_outstanding_ios(ctrl, false); /* kill the aens as they are a separate path */ -- 2.29.2 _______________________________________________ Linux-nvme mailing list Linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2021-08-18 12:05 UTC|newest] Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-08-18 12:05 [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully Daniel Wagner 2021-08-18 12:05 ` Daniel Wagner 2021-08-18 12:05 ` [PATCH v5 1/3] nvme-fc: Wait with a timeout for queue to freeze Daniel Wagner 2021-08-18 12:05 ` Daniel Wagner 2021-08-18 12:05 ` [PATCH v5 2/3] nvme-fc: avoid race between time out and tear down Daniel Wagner 2021-08-18 12:05 ` Daniel Wagner 2021-08-18 12:05 ` Daniel Wagner [this message] 2021-08-18 12:05 ` [PATCH v5 3/3] nvme-fc: fix controller reset hang during traffic Daniel Wagner 2021-08-18 12:21 ` Hannes Reinecke 2021-08-18 12:21 ` Hannes Reinecke 2021-08-20 8:48 ` [PATCH v5 0/3] Handle update hardware queues and queue freeze more carefully Daniel Wagner 2021-08-20 8:48 ` Daniel Wagner 2021-08-20 11:55 ` Daniel Wagner 2021-08-20 11:55 ` Daniel Wagner 2021-08-20 15:27 ` James Smart 2021-08-20 15:27 ` James Smart 2021-08-23 8:01 ` Christoph Hellwig 2021-08-23 8:01 ` Christoph Hellwig 2021-08-23 9:14 ` Daniel Wagner 2021-08-23 9:14 ` Daniel Wagner 2021-08-23 9:19 ` Christoph Hellwig 2021-08-23 9:19 ` Christoph Hellwig
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210818120530.130501-4-dwagner@suse.de \ --to=dwagner@suse.de \ --cc=hare@suse.de \ --cc=himanshu.madhani@oracle.com \ --cc=james.smart@broadcom.com \ --cc=jsmart2021@gmail.com \ --cc=kbusch@kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-nvme@lists.infradead.org \ --cc=ming.lei@redhat.com \ --cc=sagi@grimberg.me \ --cc=wenxiong@us.ibm.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.