linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Keith Busch <kbusch@kernel.org>, Sagi Grimberg <sagi@grimberg.me>,
	Christoph Hellwig <hch@lst.de>, Jens Axboe <axboe@kernel.dk>,
	Sasha Levin <sashal@kernel.org>,
	linux-nvme@lists.infradead.org
Subject: [PATCH AUTOSEL 5.4 50/50] nvme/pci: move cqe check after device shutdown
Date: Sat, 22 Feb 2020 21:22:35 -0500	[thread overview]
Message-ID: <20200223022235.1404-50-sashal@kernel.org> (raw)
In-Reply-To: <20200223022235.1404-1-sashal@kernel.org>

From: Keith Busch <kbusch@kernel.org>

[ Upstream commit fa46c6fb5d61b1f17b06d7c6ef75478b576304c7 ]

Many users have reported nvme triggered irq_startup() warnings during
shutdown. The driver uses the nvme queue's irq to synchronize scanning
for completions, and enabling an interrupt affined to only offline CPUs
triggers the alarming warning.

Move the final CQE check to after disabling the device and all
registered interrupts have been torn down so that we do not have any
IRQ to synchronize.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=206509
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 drivers/nvme/host/pci.c | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 14d513087a14b..858805929b2cd 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1413,6 +1413,23 @@ static void nvme_disable_admin_queue(struct nvme_dev *dev, bool shutdown)
 	nvme_poll_irqdisable(nvmeq, -1);
 }
 
+/*
+ * Called only on a device that has been disabled and after all other threads
+ * that can check this device's completion queues have synced. This is the
+ * last chance for the driver to see a natural completion before
+ * nvme_cancel_request() terminates all incomplete requests.
+ */
+static void nvme_reap_pending_cqes(struct nvme_dev *dev)
+{
+	u16 start, end;
+	int i;
+
+	for (i = dev->ctrl.queue_count - 1; i > 0; i--) {
+		nvme_process_cq(&dev->queues[i], &start, &end, -1);
+		nvme_complete_cqes(&dev->queues[i], start, end);
+	}
+}
+
 static int nvme_cmb_qdepth(struct nvme_dev *dev, int nr_io_queues,
 				int entry_size)
 {
@@ -2248,11 +2265,6 @@ static bool __nvme_disable_io_queues(struct nvme_dev *dev, u8 opcode)
 		if (timeout == 0)
 			return false;
 
-		/* handle any remaining CQEs */
-		if (opcode == nvme_admin_delete_cq &&
-		    !test_bit(NVMEQ_DELETE_ERROR, &nvmeq->flags))
-			nvme_poll_irqdisable(nvmeq, -1);
-
 		sent--;
 		if (nr_queues)
 			goto retry;
@@ -2441,6 +2453,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)
 	nvme_suspend_io_queues(dev);
 	nvme_suspend_queue(&dev->queues[0]);
 	nvme_pci_disable(dev);
+	nvme_reap_pending_cqes(dev);
 
 	blk_mq_tagset_busy_iter(&dev->tagset, nvme_cancel_request, &dev->ctrl);
 	blk_mq_tagset_busy_iter(&dev->admin_tagset, nvme_cancel_request, &dev->ctrl);
-- 
2.20.1


      parent reply	other threads:[~2020-02-23  2:23 UTC|newest]

Thread overview: 50+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-23  2:21 [PATCH AUTOSEL 5.4 01/50] ipmi:ssif: Handle a possible NULL pointer reference Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 02/50] drm/msm: Set dma maximum segment size for mdss Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 03/50] sched/core: Don't skip remote tick for idle CPUs Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 04/50] timers/nohz: Update NOHZ load in remote tick Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 05/50] sched/fair: Prevent unlimited runtime on throttled group Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 06/50] dax: pass NOWAIT flag to iomap_apply Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 07/50] mac80211: consider more elements in parsing CRC Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 08/50] cfg80211: check wiphy driver existence for drvinfo report Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 09/50] s390/zcrypt: fix card and queue total counter wrap Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 10/50] qmi_wwan: re-add DW5821e pre-production variant Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 11/50] qmi_wwan: unconditionally reject 2 ep interfaces Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 12/50] NFSv4: Fix races between open and dentry revalidation Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 13/50] perf/smmuv3: Use platform_get_irq_optional() for wired interrupt Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 14/50] arm/ftrace: Fix BE text poking Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 15/50] sched/psi: Fix OOB write when writing 0 bytes to PSI files Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 16/50] perf/x86/intel: Add Elkhart Lake support Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 17/50] perf/x86/cstate: Add Tremont support Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 18/50] perf/x86/msr: " Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 19/50] ceph: do not execute direct write in parallel if O_APPEND is specified Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 20/50] ARM: dts: sti: fixup sound frame-inversion for stihxxx-b2120.dtsi Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 21/50] drm/amd/display: Do not set optimized_require to false after plane disable Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 22/50] RDMA/siw: Remove unwanted WARN_ON in siw_cm_llp_data_ready() Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 23/50] drm/amd/display: Check engine is not NULL before acquiring Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 24/50] drm/amd/display: Limit minimum DPPCLK to 100MHz Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 25/50] drm/amd/display: Add initialitions for PLL2 clock source Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 26/50] amdgpu: Prevent build errors regarding soft/hard-float FP ABI tags Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 27/50] soc/tegra: fuse: Fix build with Tegra194 configuration Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 28/50] i40e: Fix the conditional for i40e_vc_validate_vqs_bitmaps Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 29/50] net: ena: fix potential crash when rxfh key is NULL Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 30/50] net: ena: fix uses of round_jiffies() Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 31/50] net: ena: add missing ethtool TX timestamping indication Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 32/50] net: ena: fix incorrect default RSS key Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 33/50] net: ena: rss: do not allocate key when not supported Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 34/50] net: ena: rss: fix failure to get indirection table Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 35/50] net: ena: rss: store hash function as values and not bits Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 36/50] net: ena: fix incorrectly saving queue numbers when setting RSS indirection table Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 37/50] net: ena: fix corruption of dev_idx_to_host_tbl Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 38/50] net: ena: ethtool: use correct value for crc32 hash Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 39/50] net: ena: ena-com.c: prevent NULL pointer dereference Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 40/50] enic: prevent waking up stopped tx queues over watchdog reset Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 41/50] ice: update Unit Load Status bitmask to check after reset Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 42/50] cifs: Fix mode output in debugging statements Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 43/50] bcache: ignore pending signals when creating gc and allocator thread Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 44/50] cfg80211: add missing policy for NL80211_ATTR_STATUS_CODE Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 45/50] mac80211: fix wrong 160/80+80 MHz setting Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 46/50] net: hns3: add management table after IMP reset Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 47/50] net: hns3: fix a copying IPv6 address error in hclge_fd_get_flow_tuples() Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 48/50] nvme/tcp: fix bug on double requeue when send fails Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 49/50] nvme: prevent warning triggered by nvme_stop_keep_alive Sasha Levin
2020-02-23  2:22 ` Sasha Levin [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200223022235.1404-50-sashal@kernel.org \
    --to=sashal@kernel.org \
    --cc=axboe@kernel.dk \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=sagi@grimberg.me \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).