From: Jianchao Wang <jianchao.w.wang@oracle.com> To: keith.busch@intel.com, axboe@fb.com, hch@lst.de, sagi@grimberg.me, linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH] nvme: unquiesce the queue before cleaup it Date: Thu, 19 Apr 2018 16:29:13 +0800 [thread overview] Message-ID: <1524126553-16290-1-git-send-email-jianchao.w.wang@oracle.com> (raw) There is race between nvme_remove and nvme_reset_work that can lead to io hang. nvme_remove nvme_reset_work -> change state to DELETING -> fail to change state to LIVE -> nvme_remove_dead_ctrl -> nvme_dev_disable -> quiesce request_queue -> queue remove_work -> cancel_work_sync reset_work -> nvme_remove_namespaces -> splice ctrl->namespaces nvme_remove_dead_ctrl_work -> nvme_kill_queues -> nvme_ns_remove do nothing -> blk_cleanup_queue -> blk_freeze_queue Finally, the request_queue is quiesced state when wait freeze, we will get io hang here. To fix it, unquiesce the request_queue directly before nvme_ns_remove. We have spliced the ctrl->namespaces, so nobody could access them and quiesce the queue any more. Signed-off-by: Jianchao Wang <jianchao.w.wang@oracle.com> --- drivers/nvme/host/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 9df4f71..0e95082 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -3249,8 +3249,15 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl) list_splice_init(&ctrl->namespaces, &ns_list); up_write(&ctrl->namespaces_rwsem); - list_for_each_entry_safe(ns, next, &ns_list, list) + /* + * After splice the namespaces list from the ctrl->namespaces, + * nobody could get them anymore, let's unquiesce the request_queue + * forcibly to avoid io hang. + */ + list_for_each_entry_safe(ns, next, &ns_list, list) { + blk_mq_unquiesce_queue(ns->queue); nvme_ns_remove(ns); + } } EXPORT_SYMBOL_GPL(nvme_remove_namespaces); -- 2.7.4
WARNING: multiple messages have this Message-ID (diff)
From: jianchao.w.wang@oracle.com (Jianchao Wang) Subject: [PATCH] nvme: unquiesce the queue before cleaup it Date: Thu, 19 Apr 2018 16:29:13 +0800 [thread overview] Message-ID: <1524126553-16290-1-git-send-email-jianchao.w.wang@oracle.com> (raw) There is race between nvme_remove and nvme_reset_work that can lead to io hang. nvme_remove nvme_reset_work -> change state to DELETING -> fail to change state to LIVE -> nvme_remove_dead_ctrl -> nvme_dev_disable -> quiesce request_queue -> queue remove_work -> cancel_work_sync reset_work -> nvme_remove_namespaces -> splice ctrl->namespaces nvme_remove_dead_ctrl_work -> nvme_kill_queues -> nvme_ns_remove do nothing -> blk_cleanup_queue -> blk_freeze_queue Finally, the request_queue is quiesced state when wait freeze, we will get io hang here. To fix it, unquiesce the request_queue directly before nvme_ns_remove. We have spliced the ctrl->namespaces, so nobody could access them and quiesce the queue any more. Signed-off-by: Jianchao Wang <jianchao.w.wang at oracle.com> --- drivers/nvme/host/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 9df4f71..0e95082 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -3249,8 +3249,15 @@ void nvme_remove_namespaces(struct nvme_ctrl *ctrl) list_splice_init(&ctrl->namespaces, &ns_list); up_write(&ctrl->namespaces_rwsem); - list_for_each_entry_safe(ns, next, &ns_list, list) + /* + * After splice the namespaces list from the ctrl->namespaces, + * nobody could get them anymore, let's unquiesce the request_queue + * forcibly to avoid io hang. + */ + list_for_each_entry_safe(ns, next, &ns_list, list) { + blk_mq_unquiesce_queue(ns->queue); nvme_ns_remove(ns); + } } EXPORT_SYMBOL_GPL(nvme_remove_namespaces); -- 2.7.4
next reply other threads:[~2018-04-19 8:29 UTC|newest] Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-04-19 8:29 Jianchao Wang [this message] 2018-04-19 8:29 ` [PATCH] nvme: unquiesce the queue before cleaup it Jianchao Wang 2018-04-22 13:32 ` jianchao.wang 2018-04-22 13:32 ` jianchao.wang 2018-04-22 14:18 ` Max Gurtovoy 2018-04-22 14:25 ` jianchao.wang 2018-04-22 14:25 ` jianchao.wang 2018-04-22 14:48 ` Max Gurtovoy 2018-04-22 15:00 ` jianchao.wang 2018-04-22 15:00 ` jianchao.wang 2018-04-26 8:27 ` jianchao.wang 2018-04-26 8:27 ` jianchao.wang 2018-04-26 10:23 ` Max Gurtovoy 2018-04-27 8:51 ` jianchao.wang 2018-04-27 8:51 ` jianchao.wang 2018-04-28 15:38 ` jianchao.wang 2018-04-28 15:38 ` jianchao.wang 2018-04-29 15:47 ` Max Gurtovoy
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1524126553-16290-1-git-send-email-jianchao.w.wang@oracle.com \ --to=jianchao.w.wang@oracle.com \ --cc=axboe@fb.com \ --cc=hch@lst.de \ --cc=keith.busch@intel.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-nvme@lists.infradead.org \ --cc=sagi@grimberg.me \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.