From: Chao Leng <lengchao@huawei.com>
To: <linux-nvme@lists.infradead.org>
Cc: kbusch@kernel.org, axboe@fb.com, hch@lst.de, lengchao@huawei.com,
sagi@grimberg.me
Subject: [PATCH] nvme-core: fix deadlock when reconnect failed due to nvme_set_queue_count timeout
Date: Wed, 5 Aug 2020 14:33:35 +0800 [thread overview]
Message-ID: <20200805063335.19882-1-lengchao@huawei.com> (raw)
A deadlock happens When we test nvme over roce with link blink. The
reason: link blink will cause error recovery, and then reconnect.If
reconnect fail due to nvme_set_queue_count timeout, the reconnect
process will set the queue count as 0 and continue , and then
nvme_start_ctrl will call nvme_enable_aen, and deadlock happens
because the admin queue is quiesced.
log:
Aug 3 22:47:24 localhost kernel: nvme nvme2: I/O 22 QID 0 timeout
Aug 3 22:47:24 localhost kernel: nvme nvme2: Could not set queue count
(881)
stack:
root 23848 0.0 0.0 0 0 ? D Aug03 0:00
[kworker/u12:4+nvme-wq]
[<0>] blk_execute_rq+0x69/0xa0
[<0>] __nvme_submit_sync_cmd+0xaf/0x1b0 [nvme_core]
[<0>] nvme_features+0x73/0xb0 [nvme_core]
[<0>] nvme_start_ctrl+0xa4/0x100 [nvme_core]
[<0>] nvme_rdma_setup_ctrl+0x438/0x700 [nvme_rdma]
[<0>] nvme_rdma_reconnect_ctrl_work+0x22/0x30 [nvme_rdma]
[<0>] process_one_work+0x1a7/0x370
[<0>] worker_thread+0x30/0x380
[<0>] kthread+0x112/0x130
[<0>] ret_from_fork+0x35/0x40
Many functions which call __nvme_submit_sync_cmd treat error code in two
modes: If error code less than 0, treat as command failed. If erroe code
more than 0, treat as target not support or other. So cancel io with
error code: NVME_SC_HOST_PATH_ERROR or NVME_SC_HOST_ABORTED_CMD, we need
set the flag:NVME_REQ_CANCELLED. Thus __nvme_submit_sync_cmd will return
error(less than 0), nvme_set_queue_count will return error, reconnect
progress will fail instead of continue.
Signed-off-by: Chao Leng <lengchao@huawei.com>
---
drivers/nvme/host/core.c | 1 +
drivers/nvme/host/fabrics.c | 1 +
2 files changed, 2 insertions(+)
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index c2c5bc4fb702..865645577f2c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -303,6 +303,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)
if (blk_mq_request_completed(req))
return true;
+ nvme_req(req)->flags |= NVME_REQ_CANCELLED;
nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;
blk_mq_force_complete_rq(req);
return true;
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 2a6c8190eeb7..4e745603a3af 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -552,6 +552,7 @@ blk_status_t nvmf_fail_nonready_command(struct nvme_ctrl *ctrl,
!blk_noretry_request(rq) && !(rq->cmd_flags & REQ_NVME_MPATH))
return BLK_STS_RESOURCE;
+ nvme_req(rq)->flags |= NVME_REQ_CANCELLED;
nvme_req(rq)->status = NVME_SC_HOST_PATH_ERROR;
blk_mq_start_request(rq);
nvme_complete_rq(rq);
--
2.16.4
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
next reply other threads:[~2020-08-05 6:33 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-05 6:33 Chao Leng [this message]
2020-08-05 7:10 ` [PATCH] nvme-core: fix deadlock when reconnect failed due to nvme_set_queue_count timeout Sagi Grimberg
2020-08-05 7:27 ` Chao Leng
2020-08-05 8:22 ` Sagi Grimberg
2020-08-05 9:42 ` Chao Leng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200805063335.19882-1-lengchao@huawei.com \
--to=lengchao@huawei.com \
--cc=axboe@fb.com \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).