From: Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
To: <linux-pci@vger.kernel.org>, <linuxppc-dev@lists.ozlabs.org>
Cc: Sergey Miroshnichenko <s.miroshnichenko@yadro.com>,
Bjorn Helgaas <helgaas@kernel.org>,
Christoph Hellwig <hch@lst.de>,
linux-nvme@lists.infradead.org, linux@yadro.com
Subject: [PATCH v5 21/23] nvme-pci: Handle movable BARs
Date: Fri, 16 Aug 2019 19:50:59 +0300 [thread overview]
Message-ID: <20190816165101.911-22-s.miroshnichenko@yadro.com> (raw)
In-Reply-To: <20190816165101.911-1-s.miroshnichenko@yadro.com>
Hotplugged devices can affect the existing ones by moving their BARs. The
PCI subsystem will inform the NVME driver about this by invoking the
.rescan_prepare() and .rescan_done() hooks, so the BARs can by re-mapped.
Tested under the "randrw" mode of the fio tool. Before the hotplugging:
% sudo cat /proc/iomem
...
3fe800000000-3fe8007fffff : PCI Bus 0020:0b
3fe800000000-3fe8007fffff : PCI Bus 0020:18
3fe800000000-3fe8000fffff : 0020:18:00.0
3fe800000000-3fe8000fffff : nvme
3fe800100000-3fe80017ffff : 0020:18:00.0
...
, then another NVME drive was hot-added, so BARs of the 0020:18:00.0 are
moved:
% sudo cat /proc/iomem
...
3fe800000000-3fe800ffffff : PCI Bus 0020:0b
3fe800000000-3fe8007fffff : PCI Bus 0020:10
3fe800000000-3fe800003fff : 0020:10:00.0
3fe800000000-3fe800003fff : nvme
3fe800010000-3fe80001ffff : 0020:10:00.0
3fe800800000-3fe800ffffff : PCI Bus 0020:18
3fe800800000-3fe8008fffff : 0020:18:00.0
3fe800800000-3fe8008fffff : nvme
3fe800900000-3fe80097ffff : 0020:18:00.0
...
During the rescanning, both READ and WRITE speeds drop to zero for a while
due to driver's pause, then restore.
Cc: linux-nvme@lists.infradead.org
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
---
drivers/nvme/host/pci.c | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index db160cee42ad..a805d80082ca 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1645,7 +1645,7 @@ static int nvme_remap_bar(struct nvme_dev *dev, unsigned long size)
{
struct pci_dev *pdev = to_pci_dev(dev->dev);
- if (size <= dev->bar_mapped_size)
+ if (dev->bar && size <= dev->bar_mapped_size)
return 0;
if (size > pci_resource_len(pdev, 0))
return -ENOMEM;
@@ -2980,6 +2980,23 @@ static void nvme_error_resume(struct pci_dev *pdev)
flush_work(&dev->ctrl.reset_work);
}
+static void nvme_rescan_prepare(struct pci_dev *pdev)
+{
+ struct nvme_dev *dev = pci_get_drvdata(pdev);
+
+ nvme_dev_disable(dev, false);
+ nvme_dev_unmap(dev);
+ dev->bar = NULL;
+}
+
+static void nvme_rescan_done(struct pci_dev *pdev)
+{
+ struct nvme_dev *dev = pci_get_drvdata(pdev);
+
+ nvme_dev_map(dev);
+ nvme_reset_ctrl_sync(&dev->ctrl);
+}
+
static const struct pci_error_handlers nvme_err_handler = {
.error_detected = nvme_error_detected,
.slot_reset = nvme_slot_reset,
@@ -3049,6 +3066,8 @@ static struct pci_driver nvme_driver = {
#endif
.sriov_configure = pci_sriov_configure_simple,
.err_handler = &nvme_err_handler,
+ .rescan_prepare = nvme_rescan_prepare,
+ .rescan_done = nvme_rescan_done,
};
static int __init nvme_init(void)
--
2.21.0
next prev parent reply other threads:[~2019-08-16 17:40 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-16 16:50 [PATCH v5 00/23] PCI: Allow BAR movement during hotplug Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 01/23] PCI: Fix race condition in pci_enable/disable_device() Sergey Miroshnichenko
2019-08-22 12:37 ` Marta Rybczynska
2019-09-27 21:59 ` Bjorn Helgaas
2019-09-30 8:53 ` Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 02/23] PCI: Enable bridge's I/O and MEM access for hotplugged devices Sergey Miroshnichenko
2019-09-27 22:01 ` Bjorn Helgaas
2019-08-16 16:50 ` [PATCH v5 03/23] PCI: hotplug: Add a flag for the movable BARs feature Sergey Miroshnichenko
2019-09-27 22:02 ` Bjorn Helgaas
2019-09-30 8:44 ` David Laight
2019-09-30 16:17 ` Sergey Miroshnichenko
2019-09-30 12:59 ` Sergey Miroshnichenko
2019-10-15 22:14 ` Bjorn Helgaas
2019-10-16 15:50 ` Sergey Miroshnichenko
2019-10-16 17:29 ` Bjorn Helgaas
2019-08-16 16:50 ` [PATCH v5 04/23] PCI: Define PCI-specific version of the release_child_resources() Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 05/23] PCI: hotplug: movable BARs: Fix reassigning the released bridge windows Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 06/23] PCI: hotplug: movable BARs: Recalculate all bridge windows during rescan Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 07/23] PCI: hotplug: movable BARs: Don't allow added devices to steal resources Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 08/23] PCI: Include fixed and immovable BARs into the bus size calculating Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 09/23] PCI: Prohibit assigning BARs and bridge windows to non-direct parents Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 10/23] PCI: hotplug: movable BARs: Try to assign unassigned resources only once Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 11/23] PCI: hotplug: movable BARs: Calculate immovable parts of bridge windows Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 12/23] PCI: hotplug: movable BARs: Compute limits for relocated " Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 13/23] PCI: Make sure bridge windows include their fixed BARs Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 14/23] PCI: Fix assigning the fixed prefetchable resources Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 15/23] PCI: hotplug: movable BARs: Assign fixed and immovable BARs before others Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 16/23] PCI: hotplug: movable BARs: Don't reserve IO/mem bus space Sergey Miroshnichenko
2019-09-04 5:42 ` Oliver O'Halloran
2019-09-04 11:22 ` Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 17/23] powerpc/pci: Fix crash with enabled movable BARs Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 18/23] powerpc/pci: Handle BAR movement Sergey Miroshnichenko
2019-09-04 5:37 ` Oliver O'Halloran
2019-09-06 16:24 ` Sergey Miroshnichenko
2019-09-09 14:02 ` Oliver O'Halloran
2019-08-16 16:50 ` [PATCH v5 19/23] PCI: hotplug: Configure MPS for hot-added bridges during bus rescan Sergey Miroshnichenko
2019-08-16 16:50 ` [PATCH v5 20/23] PCI: hotplug: movable BARs: Enable the feature by default Sergey Miroshnichenko
2019-08-16 16:50 ` Sergey Miroshnichenko [this message]
2019-08-16 16:51 ` [PATCH v5 22/23] PCI/portdrv: Declare support of movable BARs Sergey Miroshnichenko
2019-08-16 16:51 ` [PATCH v5 23/23] PCI: pciehp: movable BARs: Trigger a domain rescan on hp events Sergey Miroshnichenko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190816165101.911-22-s.miroshnichenko@yadro.com \
--to=s.miroshnichenko@yadro.com \
--cc=hch@lst.de \
--cc=helgaas@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux@yadro.com \
--cc=linuxppc-dev@lists.ozlabs.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).