All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ming Lei <ming.lei@redhat.com>
To: Christoph Hellwig <hch@lst.de>
Cc: linux-nvme@lists.infradead.org, Xiao Liang <xiliang@redhat.com>,
	Ming Lei <ming.lei@redhat.com>,
	"jianchao.wang" <jianchao.w.wang@oracle.com>,
	Sagi Grimberg <sagi@grimberg.me>,
	Keith Busch <keith.busch@intel.com>,
	stable@vger.kernel.org
Subject: [PATCH] nvme: don't retry request marked as NVME_REQ_CANCELLED
Date: Thu, 25 Jan 2018 16:10:23 +0800	[thread overview]
Message-ID: <20180125081023.13303-1-ming.lei@redhat.com> (raw)

If request is marked as NVME_REQ_CANCELLED, we don't need to retry for
requeuing it, and it should be completed immediately. Even simply from
the flag name, it needn't to be requeued.

Otherwise, it is easy to cause IO hang when IO is timed out in case of
PCI NVMe:

1) IO timeout is triggered, and nvme_timeout() tries to disable
device(nvme_dev_disable) and reset controller(nvme_reset_ctrl)

2) inside nvme_dev_disable(), queue is frozen and quiesced, and
try to cancel every request, but the timeout request can't be canceled
since it is completed by __blk_mq_complete_request() in blk_mq_rq_timed_out().

3) this timeout req is requeued via nvme_complete_rq(), but can't be
dispatched at all because queue is quiesced and hardware isn't ready,
finally nvme_wait_freeze() waits for ever in nvme_reset_work().

Cc: "jianchao.wang" <jianchao.w.wang@oracle.com>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Keith Busch <keith.busch@intel.com>
Cc: stable@vger.kernel.org
Reported-by: Xiao Liang <xiliang@redhat.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
 drivers/nvme/host/core.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 0ff03cf95f7f..5cd713a164cb 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -210,6 +210,8 @@ static inline bool nvme_req_needs_retry(struct request *req)
 		return false;
 	if (nvme_req(req)->retries >= nvme_max_retries)
 		return false;
+	if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
+		return false;
 	return true;
 }
 
-- 
2.9.5

WARNING: multiple messages have this Message-ID (diff)
From: ming.lei@redhat.com (Ming Lei)
Subject: [PATCH] nvme: don't retry request marked as NVME_REQ_CANCELLED
Date: Thu, 25 Jan 2018 16:10:23 +0800	[thread overview]
Message-ID: <20180125081023.13303-1-ming.lei@redhat.com> (raw)

If request is marked as NVME_REQ_CANCELLED, we don't need to retry for
requeuing it, and it should be completed immediately. Even simply from
the flag name, it needn't to be requeued.

Otherwise, it is easy to cause IO hang when IO is timed out in case of
PCI NVMe:

1) IO timeout is triggered, and nvme_timeout() tries to disable
device(nvme_dev_disable) and reset controller(nvme_reset_ctrl)

2) inside nvme_dev_disable(), queue is frozen and quiesced, and
try to cancel every request, but the timeout request can't be canceled
since it is completed by __blk_mq_complete_request() in blk_mq_rq_timed_out().

3) this timeout req is requeued via nvme_complete_rq(), but can't be
dispatched at all because queue is quiesced and hardware isn't ready,
finally nvme_wait_freeze() waits for ever in nvme_reset_work().

Cc: "jianchao.wang" <jianchao.w.wang at oracle.com>
Cc: Sagi Grimberg <sagi at grimberg.me>
Cc: Keith Busch <keith.busch at intel.com>
Cc: stable at vger.kernel.org
Reported-by: Xiao Liang <xiliang at redhat.com>
Signed-off-by: Ming Lei <ming.lei at redhat.com>
---
 drivers/nvme/host/core.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 0ff03cf95f7f..5cd713a164cb 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -210,6 +210,8 @@ static inline bool nvme_req_needs_retry(struct request *req)
 		return false;
 	if (nvme_req(req)->retries >= nvme_max_retries)
 		return false;
+	if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
+		return false;
 	return true;
 }
 
-- 
2.9.5

             reply	other threads:[~2018-01-25  8:10 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-01-25  8:10 Ming Lei [this message]
2018-01-25  8:10 ` [PATCH] nvme: don't retry request marked as NVME_REQ_CANCELLED Ming Lei
2018-01-25  8:52 ` jianchao.wang
2018-01-25  8:52   ` jianchao.wang
2018-01-25 10:15   ` Ming Lei
2018-01-25 10:15     ` Ming Lei
2018-01-27 12:33     ` jianchao.wang
2018-01-27 12:33       ` jianchao.wang
2018-01-27 13:31       ` Ming Lei
2018-01-27 13:31         ` Ming Lei
2018-01-27 14:29         ` jianchao.wang
2018-01-27 14:29           ` jianchao.wang
2018-01-27 15:44           ` Ming Lei
2018-01-27 15:44             ` Ming Lei
2018-01-28  9:01             ` jianchao.wang
2018-01-28  9:01               ` jianchao.wang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180125081023.13303-1-ming.lei@redhat.com \
    --to=ming.lei@redhat.com \
    --cc=hch@lst.de \
    --cc=jianchao.w.wang@oracle.com \
    --cc=keith.busch@intel.com \
    --cc=linux-nvme@lists.infradead.org \
    --cc=sagi@grimberg.me \
    --cc=stable@vger.kernel.org \
    --cc=xiliang@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.