From: Ming Lei <ming.lei@redhat.com> To: Keith Busch <keith.busch@intel.com> Cc: Jens Axboe <axboe@kernel.dk>, linux-block@vger.kernel.org, Ming Lei <ming.lei@redhat.com>, James Smart <james.smart@broadcom.com>, Jianchao Wang <jianchao.w.wang@oracle.com>, Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>, linux-nvme@lists.infradead.org, Laurence Oberman <loberman@redhat.com> Subject: [PATCH V5 2/9] nvme: pci: cover timeout for admin commands running in EH Date: Fri, 11 May 2018 20:29:26 +0800 [thread overview] Message-ID: <20180511122933.27155-3-ming.lei@redhat.com> (raw) In-Reply-To: <20180511122933.27155-1-ming.lei@redhat.com> When admin commands are used in EH for recovering controller, we have to cover their timeout and can't depend on block's timeout since deadlock may be caused when these commands are timed-out by block layer again. Cc: James Smart <james.smart@broadcom.com> Cc: Jianchao Wang <jianchao.w.wang@oracle.com> Cc: Christoph Hellwig <hch@lst.de> Cc: Sagi Grimberg <sagi@grimberg.me> Cc: linux-nvme@lists.infradead.org Cc: Laurence Oberman <loberman@redhat.com> Signed-off-by: Ming Lei <ming.lei@redhat.com> --- drivers/nvme/host/pci.c | 81 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 70 insertions(+), 11 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index fbc71fac6f1e..ff09b1c760ea 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1733,21 +1733,28 @@ static inline void nvme_release_cmb(struct nvme_dev *dev) } } -static int nvme_set_host_mem(struct nvme_dev *dev, u32 bits) +static void nvme_init_set_host_mem_cmd(struct nvme_dev *dev, + struct nvme_command *c, u32 bits) { u64 dma_addr = dev->host_mem_descs_dma; + + memset(c, 0, sizeof(*c)); + c->features.opcode = nvme_admin_set_features; + c->features.fid = cpu_to_le32(NVME_FEAT_HOST_MEM_BUF); + c->features.dword11 = cpu_to_le32(bits); + c->features.dword12 = cpu_to_le32(dev->host_mem_size >> + ilog2(dev->ctrl.page_size)); + c->features.dword13 = cpu_to_le32(lower_32_bits(dma_addr)); + c->features.dword14 = cpu_to_le32(upper_32_bits(dma_addr)); + c->features.dword15 = cpu_to_le32(dev->nr_host_mem_descs); +} + +static int nvme_set_host_mem(struct nvme_dev *dev, u32 bits) +{ struct nvme_command c; int ret; - memset(&c, 0, sizeof(c)); - c.features.opcode = nvme_admin_set_features; - c.features.fid = cpu_to_le32(NVME_FEAT_HOST_MEM_BUF); - c.features.dword11 = cpu_to_le32(bits); - c.features.dword12 = cpu_to_le32(dev->host_mem_size >> - ilog2(dev->ctrl.page_size)); - c.features.dword13 = cpu_to_le32(lower_32_bits(dma_addr)); - c.features.dword14 = cpu_to_le32(upper_32_bits(dma_addr)); - c.features.dword15 = cpu_to_le32(dev->nr_host_mem_descs); + nvme_init_set_host_mem_cmd(dev, &c, bits); ret = nvme_submit_sync_cmd(dev->ctrl.admin_q, &c, NULL, 0); if (ret) { @@ -1758,6 +1765,58 @@ static int nvme_set_host_mem(struct nvme_dev *dev, u32 bits) return ret; } +static void nvme_set_host_mem_end_io(struct request *rq, blk_status_t sts) +{ + struct completion *waiting = rq->end_io_data; + + rq->end_io_data = NULL; + + /* + * complete last, if this is a stack request the process (and thus + * the rq pointer) could be invalid right after this complete() + */ + complete(waiting); +} + +/* + * This function can only be used inside nvme_dev_disable() when timeout + * may not work, then this function has to cover the timeout by itself. + * + * When wait_for_completion_io_timeout() returns 0 and timeout happens, + * this request will be completed after controller is shutdown. + */ +static int nvme_set_host_mem_timeout(struct nvme_dev *dev, u32 bits) +{ + DECLARE_COMPLETION_ONSTACK(wait); + struct nvme_command c; + struct request_queue *q = dev->ctrl.admin_q; + struct request *req; + int ret; + + nvme_init_set_host_mem_cmd(dev, &c, bits); + + req = nvme_alloc_request(q, &c, 0, NVME_QID_ANY); + if (IS_ERR(req)) + return PTR_ERR(req); + + req->timeout = ADMIN_TIMEOUT; + req->end_io_data = &wait; + + blk_execute_rq_nowait(q, NULL, req, false, + nvme_set_host_mem_end_io); + ret = wait_for_completion_io_timeout(&wait, ADMIN_TIMEOUT); + if (ret > 0) { + if (nvme_req(req)->flags & NVME_REQ_CANCELLED) + ret = -EINTR; + else + ret = nvme_req(req)->status; + blk_mq_free_request(req); + } else + ret = -EINTR; + + return ret; +} + static void nvme_free_host_mem(struct nvme_dev *dev) { int i; @@ -2216,7 +2275,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) * but I'd rather be safe than sorry.. */ if (dev->host_mem_descs) - nvme_set_host_mem(dev, 0); + nvme_set_host_mem_timeout(dev, 0); nvme_disable_io_queues(dev); nvme_disable_admin_queue(dev, shutdown); } -- 2.9.5
WARNING: multiple messages have this Message-ID (diff)
From: ming.lei@redhat.com (Ming Lei) Subject: [PATCH V5 2/9] nvme: pci: cover timeout for admin commands running in EH Date: Fri, 11 May 2018 20:29:26 +0800 [thread overview] Message-ID: <20180511122933.27155-3-ming.lei@redhat.com> (raw) In-Reply-To: <20180511122933.27155-1-ming.lei@redhat.com> When admin commands are used in EH for recovering controller, we have to cover their timeout and can't depend on block's timeout since deadlock may be caused when these commands are timed-out by block layer again. Cc: James Smart <james.smart at broadcom.com> Cc: Jianchao Wang <jianchao.w.wang at oracle.com> Cc: Christoph Hellwig <hch at lst.de> Cc: Sagi Grimberg <sagi at grimberg.me> Cc: linux-nvme at lists.infradead.org Cc: Laurence Oberman <loberman at redhat.com> Signed-off-by: Ming Lei <ming.lei at redhat.com> --- drivers/nvme/host/pci.c | 81 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 70 insertions(+), 11 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index fbc71fac6f1e..ff09b1c760ea 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1733,21 +1733,28 @@ static inline void nvme_release_cmb(struct nvme_dev *dev) } } -static int nvme_set_host_mem(struct nvme_dev *dev, u32 bits) +static void nvme_init_set_host_mem_cmd(struct nvme_dev *dev, + struct nvme_command *c, u32 bits) { u64 dma_addr = dev->host_mem_descs_dma; + + memset(c, 0, sizeof(*c)); + c->features.opcode = nvme_admin_set_features; + c->features.fid = cpu_to_le32(NVME_FEAT_HOST_MEM_BUF); + c->features.dword11 = cpu_to_le32(bits); + c->features.dword12 = cpu_to_le32(dev->host_mem_size >> + ilog2(dev->ctrl.page_size)); + c->features.dword13 = cpu_to_le32(lower_32_bits(dma_addr)); + c->features.dword14 = cpu_to_le32(upper_32_bits(dma_addr)); + c->features.dword15 = cpu_to_le32(dev->nr_host_mem_descs); +} + +static int nvme_set_host_mem(struct nvme_dev *dev, u32 bits) +{ struct nvme_command c; int ret; - memset(&c, 0, sizeof(c)); - c.features.opcode = nvme_admin_set_features; - c.features.fid = cpu_to_le32(NVME_FEAT_HOST_MEM_BUF); - c.features.dword11 = cpu_to_le32(bits); - c.features.dword12 = cpu_to_le32(dev->host_mem_size >> - ilog2(dev->ctrl.page_size)); - c.features.dword13 = cpu_to_le32(lower_32_bits(dma_addr)); - c.features.dword14 = cpu_to_le32(upper_32_bits(dma_addr)); - c.features.dword15 = cpu_to_le32(dev->nr_host_mem_descs); + nvme_init_set_host_mem_cmd(dev, &c, bits); ret = nvme_submit_sync_cmd(dev->ctrl.admin_q, &c, NULL, 0); if (ret) { @@ -1758,6 +1765,58 @@ static int nvme_set_host_mem(struct nvme_dev *dev, u32 bits) return ret; } +static void nvme_set_host_mem_end_io(struct request *rq, blk_status_t sts) +{ + struct completion *waiting = rq->end_io_data; + + rq->end_io_data = NULL; + + /* + * complete last, if this is a stack request the process (and thus + * the rq pointer) could be invalid right after this complete() + */ + complete(waiting); +} + +/* + * This function can only be used inside nvme_dev_disable() when timeout + * may not work, then this function has to cover the timeout by itself. + * + * When wait_for_completion_io_timeout() returns 0 and timeout happens, + * this request will be completed after controller is shutdown. + */ +static int nvme_set_host_mem_timeout(struct nvme_dev *dev, u32 bits) +{ + DECLARE_COMPLETION_ONSTACK(wait); + struct nvme_command c; + struct request_queue *q = dev->ctrl.admin_q; + struct request *req; + int ret; + + nvme_init_set_host_mem_cmd(dev, &c, bits); + + req = nvme_alloc_request(q, &c, 0, NVME_QID_ANY); + if (IS_ERR(req)) + return PTR_ERR(req); + + req->timeout = ADMIN_TIMEOUT; + req->end_io_data = &wait; + + blk_execute_rq_nowait(q, NULL, req, false, + nvme_set_host_mem_end_io); + ret = wait_for_completion_io_timeout(&wait, ADMIN_TIMEOUT); + if (ret > 0) { + if (nvme_req(req)->flags & NVME_REQ_CANCELLED) + ret = -EINTR; + else + ret = nvme_req(req)->status; + blk_mq_free_request(req); + } else + ret = -EINTR; + + return ret; +} + static void nvme_free_host_mem(struct nvme_dev *dev) { int i; @@ -2216,7 +2275,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) * but I'd rather be safe than sorry.. */ if (dev->host_mem_descs) - nvme_set_host_mem(dev, 0); + nvme_set_host_mem_timeout(dev, 0); nvme_disable_io_queues(dev); nvme_disable_admin_queue(dev, shutdown); } -- 2.9.5
next prev parent reply other threads:[~2018-05-11 12:29 UTC|newest] Thread overview: 64+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-05-11 12:29 [PATCH V5 0/9] nvme: pci: fix & improve timeout handling Ming Lei 2018-05-11 12:29 ` Ming Lei 2018-05-11 12:29 ` [PATCH V5 1/9] block: introduce blk_quiesce_timeout() and blk_unquiesce_timeout() Ming Lei 2018-05-11 12:29 ` Ming Lei 2018-05-11 12:29 ` Ming Lei [this message] 2018-05-11 12:29 ` [PATCH V5 2/9] nvme: pci: cover timeout for admin commands running in EH Ming Lei 2018-05-11 12:29 ` [PATCH V5 3/9] nvme: pci: only wait freezing if queue is frozen Ming Lei 2018-05-11 12:29 ` Ming Lei 2018-05-11 12:29 ` [PATCH V5 4/9] nvme: pci: freeze queue in nvme_dev_disable() in case of error recovery Ming Lei 2018-05-11 12:29 ` Ming Lei 2018-05-11 12:29 ` [PATCH V5 5/9] nvme: pci: prepare for supporting error recovery from resetting context Ming Lei 2018-05-11 12:29 ` Ming Lei 2018-05-11 12:29 ` [PATCH V5 6/9] nvme: pci: move error handling out of nvme_reset_dev() Ming Lei 2018-05-11 12:29 ` Ming Lei 2018-05-11 12:29 ` [PATCH V5 7/9] nvme: pci: don't unfreeze queue until controller state updating succeeds Ming Lei 2018-05-11 12:29 ` Ming Lei 2018-05-11 12:29 ` [PATCH V5 8/9] nvme: core: introduce nvme_force_change_ctrl_state() Ming Lei 2018-05-11 12:29 ` Ming Lei 2018-05-11 12:29 ` [PATCH V5 9/9] nvme: pci: support nested EH Ming Lei 2018-05-11 12:29 ` Ming Lei 2018-05-15 10:02 ` jianchao.wang 2018-05-15 10:02 ` jianchao.wang 2018-05-15 12:39 ` Ming Lei 2018-05-15 12:39 ` Ming Lei 2018-05-11 20:50 ` [PATCH V5 0/9] nvme: pci: fix & improve timeout handling Keith Busch 2018-05-11 20:50 ` Keith Busch 2018-05-12 0:21 ` Ming Lei 2018-05-12 0:21 ` Ming Lei 2018-05-14 15:18 ` Keith Busch 2018-05-14 15:18 ` Keith Busch 2018-05-14 23:47 ` Ming Lei 2018-05-14 23:47 ` Ming Lei 2018-05-15 0:33 ` Keith Busch 2018-05-15 0:33 ` Keith Busch 2018-05-15 9:08 ` Ming Lei 2018-05-15 9:08 ` Ming Lei 2018-05-16 4:31 ` Ming Lei 2018-05-16 4:31 ` Ming Lei 2018-05-16 15:18 ` Keith Busch 2018-05-16 15:18 ` Keith Busch 2018-05-16 22:18 ` Ming Lei 2018-05-16 22:18 ` Ming Lei 2018-05-14 8:21 ` jianchao.wang 2018-05-14 8:21 ` jianchao.wang 2018-05-14 9:38 ` Ming Lei 2018-05-14 9:38 ` Ming Lei 2018-05-14 10:05 ` jianchao.wang 2018-05-14 10:05 ` jianchao.wang 2018-05-14 12:22 ` Ming Lei 2018-05-14 12:22 ` Ming Lei 2018-05-15 0:33 ` Ming Lei 2018-05-15 0:33 ` Ming Lei 2018-05-15 9:56 ` jianchao.wang 2018-05-15 9:56 ` jianchao.wang 2018-05-15 12:56 ` Ming Lei 2018-05-15 12:56 ` Ming Lei 2018-05-16 3:03 ` jianchao.wang 2018-05-16 3:03 ` jianchao.wang 2018-05-16 2:04 ` Ming Lei 2018-05-16 2:04 ` Ming Lei 2018-05-16 2:09 ` Ming Lei 2018-05-16 2:09 ` Ming Lei 2018-05-16 2:15 ` jianchao.wang 2018-05-16 2:15 ` jianchao.wang
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20180511122933.27155-3-ming.lei@redhat.com \ --to=ming.lei@redhat.com \ --cc=axboe@kernel.dk \ --cc=hch@lst.de \ --cc=james.smart@broadcom.com \ --cc=jianchao.w.wang@oracle.com \ --cc=keith.busch@intel.com \ --cc=linux-block@vger.kernel.org \ --cc=linux-nvme@lists.infradead.org \ --cc=loberman@redhat.com \ --cc=sagi@grimberg.me \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.