From: Sagi Grimberg <sagi@grimberg.me> To: Daniel Wagner <dwagner@suse.de>, linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, James Smart <james.smart@broadcom.com>, Keith Busch <kbusch@kernel.org>, Ming Lei <ming.lei@redhat.com>, Hannes Reinecke <hare@suse.de>, Wen Xiong <wenxiong@us.ibm.com> Subject: Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them Date: Fri, 6 Aug 2021 12:57:17 -0700 [thread overview] Message-ID: <8373c07f-f5df-1ec6-9fda-d0262fc1b377@grimberg.me> (raw) In-Reply-To: <20210802112658.75875-3-dwagner@suse.de> > From: Hannes Reinecke <hare@suse.de> > > When the number of hardware queues changes during resetting we should > update the tagset first before using it. > > Signed-off-by: Hannes Reinecke <hare@suse.de> > Signed-off-by: Daniel Wagner <dwagner@suse.de> > --- > drivers/nvme/host/tcp.c | 14 ++++++-------- > 1 file changed, 6 insertions(+), 8 deletions(-) > > diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c > index 0a97ba02f61e..32268f24f62a 100644 > --- a/drivers/nvme/host/tcp.c > +++ b/drivers/nvme/host/tcp.c > @@ -1789,6 +1789,7 @@ static void nvme_tcp_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove) > static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) > { > int ret; > + u32 prior_q_cnt = ctrl->queue_count; > > ret = nvme_tcp_alloc_io_queues(ctrl); > if (ret) > @@ -1806,14 +1807,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) > ret = PTR_ERR(ctrl->connect_q); > goto out_free_tag_set; > } > - } > - > - ret = nvme_tcp_start_io_queues(ctrl); > - if (ret) > - goto out_cleanup_connect_q; > - > - if (!new) { > - nvme_start_queues(ctrl); > + } else if (prior_q_cnt != ctrl->queue_count) { So if the queue count did not change we don't wait to make sure the queue g_usage_counter ref made it to zero? What guarantees that it did? > if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) { > /* > * If we timed out waiting for freeze we are likely to > @@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) > nvme_unfreeze(ctrl); > } > > + ret = nvme_tcp_start_io_queues(ctrl); > + if (ret) > + goto out_cleanup_connect_q; > + Did you test this with both heavy I/O, reset loop and ifdown/ifup loop? If we unquiesce and unfreeze before we start the queues the pending I/Os may resume before the connect and not allow the connect to make forward progress. > return 0; > > out_wait_freeze_timed_out: >
WARNING: multiple messages have this Message-ID (diff)
From: Sagi Grimberg <sagi@grimberg.me> To: Daniel Wagner <dwagner@suse.de>, linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, James Smart <james.smart@broadcom.com>, Keith Busch <kbusch@kernel.org>, Ming Lei <ming.lei@redhat.com>, Hannes Reinecke <hare@suse.de>, Wen Xiong <wenxiong@us.ibm.com> Subject: Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them Date: Fri, 6 Aug 2021 12:57:17 -0700 [thread overview] Message-ID: <8373c07f-f5df-1ec6-9fda-d0262fc1b377@grimberg.me> (raw) In-Reply-To: <20210802112658.75875-3-dwagner@suse.de> > From: Hannes Reinecke <hare@suse.de> > > When the number of hardware queues changes during resetting we should > update the tagset first before using it. > > Signed-off-by: Hannes Reinecke <hare@suse.de> > Signed-off-by: Daniel Wagner <dwagner@suse.de> > --- > drivers/nvme/host/tcp.c | 14 ++++++-------- > 1 file changed, 6 insertions(+), 8 deletions(-) > > diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c > index 0a97ba02f61e..32268f24f62a 100644 > --- a/drivers/nvme/host/tcp.c > +++ b/drivers/nvme/host/tcp.c > @@ -1789,6 +1789,7 @@ static void nvme_tcp_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove) > static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) > { > int ret; > + u32 prior_q_cnt = ctrl->queue_count; > > ret = nvme_tcp_alloc_io_queues(ctrl); > if (ret) > @@ -1806,14 +1807,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) > ret = PTR_ERR(ctrl->connect_q); > goto out_free_tag_set; > } > - } > - > - ret = nvme_tcp_start_io_queues(ctrl); > - if (ret) > - goto out_cleanup_connect_q; > - > - if (!new) { > - nvme_start_queues(ctrl); > + } else if (prior_q_cnt != ctrl->queue_count) { So if the queue count did not change we don't wait to make sure the queue g_usage_counter ref made it to zero? What guarantees that it did? > if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) { > /* > * If we timed out waiting for freeze we are likely to > @@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) > nvme_unfreeze(ctrl); > } > > + ret = nvme_tcp_start_io_queues(ctrl); > + if (ret) > + goto out_cleanup_connect_q; > + Did you test this with both heavy I/O, reset loop and ifdown/ifup loop? If we unquiesce and unfreeze before we start the queues the pending I/Os may resume before the connect and not allow the connect to make forward progress. > return 0; > > out_wait_freeze_timed_out: > _______________________________________________ Linux-nvme mailing list Linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2021-08-06 19:57 UTC|newest] Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-08-02 11:26 [PATCH RESEND v4 0/8] Handle update hardware queues and queue freeze more carefully Daniel Wagner 2021-08-02 11:26 ` Daniel Wagner 2021-08-02 11:26 ` [PATCH v4 1/8] nvme-fc: Update hardware queues before using them Daniel Wagner 2021-08-02 11:26 ` Daniel Wagner 2021-08-02 19:34 ` Himanshu Madhani 2021-08-02 19:34 ` Himanshu Madhani 2021-08-02 11:26 ` [PATCH v4 2/8] nvme-tcp: Update number of " Daniel Wagner 2021-08-02 11:26 ` Daniel Wagner 2021-08-06 19:57 ` Sagi Grimberg [this message] 2021-08-06 19:57 ` Sagi Grimberg 2021-08-09 8:52 ` Daniel Wagner 2021-08-09 8:52 ` Daniel Wagner 2021-08-11 1:00 ` Sagi Grimberg 2021-08-11 1:00 ` Sagi Grimberg 2021-08-11 1:07 ` Keith Busch 2021-08-11 1:07 ` Keith Busch 2021-08-11 5:57 ` Sagi Grimberg 2021-08-11 5:57 ` Sagi Grimberg 2021-08-11 10:25 ` Daniel Wagner 2021-08-11 10:25 ` Daniel Wagner 2021-08-02 11:26 ` [PATCH v4 3/8] nvme-rdma: " Daniel Wagner 2021-08-02 11:26 ` Daniel Wagner 2021-08-02 11:26 ` [PATCH v4 4/8] nvme-fc: Wait with a timeout for queue to freeze Daniel Wagner 2021-08-02 11:26 ` Daniel Wagner 2021-08-02 19:36 ` Himanshu Madhani 2021-08-02 19:36 ` Himanshu Madhani 2021-08-02 11:26 ` [PATCH v4 5/8] nvme-fc: avoid race between time out and tear down Daniel Wagner 2021-08-02 11:26 ` Daniel Wagner 2021-08-02 19:38 ` Himanshu Madhani 2021-08-02 19:38 ` Himanshu Madhani 2021-08-02 11:26 ` [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic Daniel Wagner 2021-08-02 11:26 ` Daniel Wagner 2021-08-02 19:39 ` Himanshu Madhani 2021-08-02 19:39 ` Himanshu Madhani 2021-08-04 7:23 ` Hannes Reinecke 2021-08-04 7:23 ` Hannes Reinecke 2021-08-04 8:08 ` Daniel Wagner 2021-08-04 8:08 ` Daniel Wagner 2021-08-11 1:05 ` Sagi Grimberg 2021-08-11 1:05 ` Sagi Grimberg 2021-08-11 10:30 ` Daniel Wagner 2021-08-11 10:30 ` Daniel Wagner 2021-08-12 20:03 ` James Smart 2021-08-12 20:03 ` James Smart 2021-08-18 11:43 ` Daniel Wagner 2021-08-18 11:43 ` Daniel Wagner 2021-08-18 11:49 ` Daniel Wagner 2021-08-18 11:49 ` Daniel Wagner 2021-08-02 11:26 ` [PATCH v4 7/8] nvme-tcp: Unfreeze queues on reconnect Daniel Wagner 2021-08-02 11:26 ` Daniel Wagner 2021-08-02 11:26 ` [PATCH v4 8/8] nvme-rdma: " Daniel Wagner 2021-08-02 11:26 ` Daniel Wagner 2021-08-04 7:25 ` Hannes Reinecke 2021-08-04 7:25 ` Hannes Reinecke 2021-08-06 19:59 ` Sagi Grimberg 2021-08-06 19:59 ` Sagi Grimberg 2021-08-09 8:58 ` Daniel Wagner 2021-08-09 8:58 ` Daniel Wagner -- strict thread matches above, loose matches on Subject: below -- 2021-08-02 9:14 [PATCH v4 0/8] Handle update hardware queues and queue freeze more carefully Daniel Wagner 2021-08-02 9:14 ` [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them Daniel Wagner
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=8373c07f-f5df-1ec6-9fda-d0262fc1b377@grimberg.me \ --to=sagi@grimberg.me \ --cc=dwagner@suse.de \ --cc=hare@suse.de \ --cc=james.smart@broadcom.com \ --cc=kbusch@kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-nvme@lists.infradead.org \ --cc=ming.lei@redhat.com \ --cc=wenxiong@us.ibm.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.