From: Sagi Grimberg <sagi@grimberg.me>
To: James Smart <james.smart@broadcom.com>,
linux-nvme@lists.infradead.org, Christoph Hellwig <hch@lst.de>,
Keith Busch <kbusch@kernel.org>
Subject: Re: [PATCH v2 6/8] nvme-rdma: serialize controller teardown sequences
Date: Tue, 18 Aug 2020 17:35:30 -0700 [thread overview]
Message-ID: <5f022114-9924-3b71-8746-72cc8457ddf7@grimberg.me> (raw)
In-Reply-To: <7c3b10ab-6176-92d8-f2eb-45498f903778@broadcom.com>
On 8/14/20 2:12 PM, James Smart wrote:
>
>
> On 8/6/2020 12:11 PM, Sagi Grimberg wrote:
>> In the timeout handler we may need to complete a request because the
>> request that timed out may be an I/O that is a part of a serial sequence
>> of controller teardown or initialization. In order to complete the
>> request, we need to fence any other context that may compete with us
>> and complete the request that is timing out.
>>
>> In this case, we could have a potential double completion in case
>> a hard-irq or a different competing context triggered error recovery
>> and is running inflight request cancellation concurrently with the
>> timeout handler.
>>
>> Protect using a ctrl teardown_lock to serialize contexts that may
>> complete a cancelled request due to error recovery or a reset.
>>
>> Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
>> ---
>> drivers/nvme/host/rdma.c | 6 ++++++
>> 1 file changed, 6 insertions(+)
>>
>> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
>> index 44c76ffbb264..abc318737f35 100644
>> --- a/drivers/nvme/host/rdma.c
>> +++ b/drivers/nvme/host/rdma.c
>> @@ -122,6 +122,7 @@ struct nvme_rdma_ctrl {
>> struct sockaddr_storage src_addr;
>> struct nvme_ctrl ctrl;
>> + struct mutex teardown_lock;
>> bool use_inline_data;
>> u32 io_queues[HCTX_MAX_TYPES];
>> };
>> @@ -997,6 +998,7 @@ static int nvme_rdma_configure_io_queues(struct
>> nvme_rdma_ctrl *ctrl, bool new)
>> static void nvme_rdma_teardown_admin_queue(struct nvme_rdma_ctrl *ctrl,
>> bool remove)
>> {
>> + mutex_lock(&ctrl->teardown_lock);
>> blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
>> nvme_rdma_stop_queue(&ctrl->queues[0]);
>> if (ctrl->ctrl.admin_tagset) {
>> @@ -1007,11 +1009,13 @@ static void
>> nvme_rdma_teardown_admin_queue(struct nvme_rdma_ctrl *ctrl,
>> if (remove)
>> blk_mq_unquiesce_queue(ctrl->ctrl.admin_q);
>> nvme_rdma_destroy_admin_queue(ctrl, remove);
>> + mutex_unlock(&ctrl->teardown_lock);
>> }
>> static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl,
>> bool remove)
>> {
>> + mutex_lock(&ctrl->teardown_lock);
>> if (ctrl->ctrl.queue_count > 1) {
>> nvme_start_freeze(&ctrl->ctrl);
>> nvme_stop_queues(&ctrl->ctrl);
>> @@ -1025,6 +1029,7 @@ static void nvme_rdma_teardown_io_queues(struct
>> nvme_rdma_ctrl *ctrl,
>> nvme_start_queues(&ctrl->ctrl);
>> nvme_rdma_destroy_io_queues(ctrl, remove);
>> }
>> + mutex_unlock(&ctrl->teardown_lock);
>> }
>> static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl)
>> @@ -2278,6 +2283,7 @@ static struct nvme_ctrl
>> *nvme_rdma_create_ctrl(struct device *dev,
>> return ERR_PTR(-ENOMEM);
>> ctrl->ctrl.opts = opts;
>> INIT_LIST_HEAD(&ctrl->list);
>> + mutex_init(&ctrl->teardown_lock);
>> if (!(opts->mask & NVMF_OPT_TRSVCID)) {
>> opts->trsvcid =
>
> Looks good - but....
>
> I hit this same issue on FC - I will need to post a similar path. My
> problem was that the reset/teardown path due to the timeout then raced
> with the error that the connect path saw for its io that dropped into
> the partial-teardown steps as connect backed-out. So I recommend
> looking at nvme_rdma_setup_ctrl() and any of it's teardown paths that
> don't have the mutex and may race with cases that are taking the mutex.
Goof point.
The synchronization is not really required for the entire teardown path,
because the delete_work and flushing the connect_work, and state machine
doesn't allow reset and reconnect to compete. So this synchronization is
really just against the timeout handler.
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2020-08-19 0:35 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-06 19:11 [PATCH v2 0/8] fix possible controller reset hangs in nvme-tcp/nvme-rdma Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 1/8] nvme-fabrics: allow to queue requests for live queues Sagi Grimberg
2020-08-14 6:44 ` Christoph Hellwig
2020-08-14 7:08 ` Sagi Grimberg
2020-08-14 7:22 ` Christoph Hellwig
2020-08-14 15:55 ` James Smart
2020-08-14 17:49 ` Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 2/8] nvme: have nvme_wait_freeze_timeout return if it timed out Sagi Grimberg
2020-08-14 6:45 ` Christoph Hellwig
2020-08-14 7:09 ` Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 3/8] nvme-tcp: serialize controller teardown double completion Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 4/8] nvme-tcp: fix timeout handler Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 5/8] nvme-tcp: fix reset hang if controller died in the middle of a reset Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 6/8] nvme-rdma: serialize controller teardown sequences Sagi Grimberg
2020-08-14 6:45 ` Christoph Hellwig
2020-08-14 21:12 ` James Smart
2020-08-19 0:35 ` Sagi Grimberg [this message]
2020-08-06 19:11 ` [PATCH v2 7/8] nvme-rdma: fix timeout handler Sagi Grimberg
2020-08-14 6:52 ` Christoph Hellwig
2020-08-14 7:14 ` Sagi Grimberg
2020-08-14 23:19 ` James Smart
2020-08-19 0:26 ` Sagi Grimberg
2020-08-14 23:27 ` James Smart
2020-08-14 23:30 ` James Smart
2020-08-19 0:39 ` Sagi Grimberg
2020-08-19 0:38 ` Sagi Grimberg
2020-08-06 19:11 ` [PATCH v2 8/8] nvme-rdma: fix reset hang if controller died in the middle of a reset Sagi Grimberg
2020-08-14 6:53 ` Christoph Hellwig
2020-08-11 22:16 ` [PATCH v2 0/8] fix possible controller reset hangs in nvme-tcp/nvme-rdma Sagi Grimberg
2020-08-13 15:39 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5f022114-9924-3b71-8746-72cc8457ddf7@grimberg.me \
--to=sagi@grimberg.me \
--cc=hch@lst.de \
--cc=james.smart@broadcom.com \
--cc=kbusch@kernel.org \
--cc=linux-nvme@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).