From: James Smart <james.smart@broadcom.com>
To: Sagi Grimberg <sagi@grimberg.me>,
linux-nvme@lists.infradead.org, Christoph Hellwig <hch@lst.de>,
Keith Busch <kbusch@kernel.org>
Subject: Re: [PATCH v2 6/8] nvme-rdma: serialize controller teardown sequences
Date: Fri, 14 Aug 2020 14:12:12 -0700 [thread overview]
Message-ID: <7c3b10ab-6176-92d8-f2eb-45498f903778@broadcom.com> (raw)
In-Reply-To: <20200806191127.592062-7-sagi@grimberg.me>
On 8/6/2020 12:11 PM, Sagi Grimberg wrote:
> In the timeout handler we may need to complete a request because the
> request that timed out may be an I/O that is a part of a serial sequence
> of controller teardown or initialization. In order to complete the
> request, we need to fence any other context that may compete with us
> and complete the request that is timing out.
>
> In this case, we could have a potential double completion in case
> a hard-irq or a different competing context triggered error recovery
> and is running inflight request cancellation concurrently with the
> timeout handler.
>
> Protect using a ctrl teardown_lock to serialize contexts that may
> complete a cancelled request due to error recovery or a reset.
>
> Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
> ---
> drivers/nvme/host/rdma.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 44c76ffbb264..abc318737f35 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -122,6 +122,7 @@ struct nvme_rdma_ctrl {
> struct sockaddr_storage src_addr;
>
> struct nvme_ctrl ctrl;
> + struct mutex teardown_lock;
> bool use_inline_data;
> u32 io_queues[HCTX_MAX_TYPES];
> };
> @@ -997,6 +998,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
> static void nvme_rdma_teardown_admin_queue(struct nvme_rdma_ctrl *ctrl,
> bool remove)
> {
> + mutex_lock(&ctrl->teardown_lock);
> blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
> nvme_rdma_stop_queue(&ctrl->queues[0]);
> if (ctrl->ctrl.admin_tagset) {
> @@ -1007,11 +1009,13 @@ static void nvme_rdma_teardown_admin_queue(struct nvme_rdma_ctrl *ctrl,
> if (remove)
> blk_mq_unquiesce_queue(ctrl->ctrl.admin_q);
> nvme_rdma_destroy_admin_queue(ctrl, remove);
> + mutex_unlock(&ctrl->teardown_lock);
> }
>
> static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl,
> bool remove)
> {
> + mutex_lock(&ctrl->teardown_lock);
> if (ctrl->ctrl.queue_count > 1) {
> nvme_start_freeze(&ctrl->ctrl);
> nvme_stop_queues(&ctrl->ctrl);
> @@ -1025,6 +1029,7 @@ static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl,
> nvme_start_queues(&ctrl->ctrl);
> nvme_rdma_destroy_io_queues(ctrl, remove);
> }
> + mutex_unlock(&ctrl->teardown_lock);
> }
>
> static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl)
> @@ -2278,6 +2283,7 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
> return ERR_PTR(-ENOMEM);
> ctrl->ctrl.opts = opts;
> INIT_LIST_HEAD(&ctrl->list);
> + mutex_init(&ctrl->teardown_lock);
>
> if (!(opts->mask & NVMF_OPT_TRSVCID)) {
> opts->trsvcid =
Looks good - but....
I hit this same issue on FC - I will need to post a similar path. My
problem was that the reset/teardown path due to the timeout then raced
with the error that the connect path saw for its io that dropped into
the partial-teardown steps as connect backed-out. So I recommend
looking at nvme_rdma_setup_ctrl() and any of it's teardown paths that
don't have the mutex and may race with cases that are taking the mutex.
If it's all good - you can add my Reviewed-by for it.
-- james
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2020-08-14 21:12 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-06 19:11 [PATCH v2 0/8] fix possible controller reset hangs in nvme-tcp/nvme-rdma Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 1/8] nvme-fabrics: allow to queue requests for live queues Sagi Grimberg
2020-08-14 6:44 ` Christoph Hellwig
2020-08-14 7:08 ` Sagi Grimberg
2020-08-14 7:22 ` Christoph Hellwig
2020-08-14 15:55 ` James Smart
2020-08-14 17:49 ` Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 2/8] nvme: have nvme_wait_freeze_timeout return if it timed out Sagi Grimberg
2020-08-14 6:45 ` Christoph Hellwig
2020-08-14 7:09 ` Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 3/8] nvme-tcp: serialize controller teardown double completion Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 4/8] nvme-tcp: fix timeout handler Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 5/8] nvme-tcp: fix reset hang if controller died in the middle of a reset Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 6/8] nvme-rdma: serialize controller teardown sequences Sagi Grimberg
2020-08-14 6:45 ` Christoph Hellwig
2020-08-14 21:12 ` James Smart [this message]
2020-08-19 0:35 ` Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 7/8] nvme-rdma: fix timeout handler Sagi Grimberg
2020-08-14 6:52 ` Christoph Hellwig
2020-08-14 7:14 ` Sagi Grimberg
2020-08-14 23:19 ` James Smart
2020-08-19 0:26 ` Sagi Grimberg
2020-08-14 23:27 ` James Smart
2020-08-14 23:30 ` James Smart
2020-08-19 0:39 ` Sagi Grimberg
2020-08-19 0:38 ` Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 8/8] nvme-rdma: fix reset hang if controller died in the middle of a reset Sagi Grimberg
2020-08-14 6:53 ` Christoph Hellwig
2020-08-11 22:16 ` [PATCH v2 0/8] fix possible controller reset hangs in nvme-tcp/nvme-rdma Sagi Grimberg
2020-08-13 15:39 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7c3b10ab-6176-92d8-f2eb-45498f903778@broadcom.com \
--to=james.smart@broadcom.com \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).