From: Sergei Miroshnichenko <s.miroshnichenko@yadro.com>
To: <linux-pci@vger.kernel.org>
Cc: Bjorn Helgaas <helgaas@kernel.org>, Stefan Roese <sr@denx.de>,
<linux@yadro.com>,
Sergei Miroshnichenko <s.miroshnichenko@yadro.com>,
<linux-nvme@lists.infradead.org>, Christoph Hellwig <hch@lst.de>
Subject: [PATCH v7 25/26] nvme-pci: Handle movable BARs
Date: Wed, 29 Jan 2020 18:29:36 +0300 [thread overview]
Message-ID: <20200129152937.311162-26-s.miroshnichenko@yadro.com> (raw)
In-Reply-To: <20200129152937.311162-1-s.miroshnichenko@yadro.com>
Hotplugged devices can affect the existing ones by moving their BARs. The
PCI subsystem will inform the NVME driver about this by invoking the
.rescan_prepare() and .rescan_done() hooks, so the BARs can by re-mapped.
Tested under the "randrw" mode of the fio tool, and when using an NVME
drive as a root filesystem storage. Before the hotplugging:
% sudo cat /proc/iomem
...
3fe800000000-3fe8007fffff : PCI Bus 0020:0b
3fe800000000-3fe8007fffff : PCI Bus 0020:18
3fe800000000-3fe8000fffff : 0020:18:00.0
3fe800000000-3fe8000fffff : nvme
^^^^^^^^^^^^^^^^^^^^^^^^^
3fe800100000-3fe80017ffff : 0020:18:00.0
...
Then another NVME drive was hot-added, so BARs of the 0020:18:00.0 are
moved:
% sudo cat /proc/iomem
...
3fe800000000-3fe800ffffff : PCI Bus 0020:0b
3fe800000000-3fe8007fffff : PCI Bus 0020:10
3fe800000000-3fe800003fff : 0020:10:00.0
3fe800000000-3fe800003fff : nvme
3fe800010000-3fe80001ffff : 0020:10:00.0
3fe800800000-3fe800ffffff : PCI Bus 0020:18
3fe800800000-3fe8008fffff : 0020:18:00.0
3fe800800000-3fe8008fffff : nvme
^^^^^^^^^^^^^^^^^^^^^^^^^
3fe800900000-3fe80097ffff : 0020:18:00.0
...
During the rescanning, both READ and WRITE speeds drop to zero for a while
due to driver's pause, then restore.
Cc: linux-nvme@lists.infradead.org
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Sergei Miroshnichenko <s.miroshnichenko@yadro.com>
---
drivers/nvme/host/pci.c | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 365a2ddbeaa7..42976c13d3f7 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1644,7 +1644,7 @@ static int nvme_remap_bar(struct nvme_dev *dev, unsigned long size)
{
struct pci_dev *pdev = to_pci_dev(dev->dev);
- if (size <= dev->bar_mapped_size)
+ if (dev->bar && size <= dev->bar_mapped_size)
return 0;
if (size > pci_resource_len(pdev, 0))
return -ENOMEM;
@@ -3049,6 +3049,23 @@ static void nvme_error_resume(struct pci_dev *pdev)
flush_work(&dev->ctrl.reset_work);
}
+static void nvme_rescan_prepare(struct pci_dev *pdev)
+{
+ struct nvme_dev *dev = pci_get_drvdata(pdev);
+
+ nvme_dev_disable(dev, false);
+ nvme_dev_unmap(dev);
+ dev->bar = NULL;
+}
+
+static void nvme_rescan_done(struct pci_dev *pdev)
+{
+ struct nvme_dev *dev = pci_get_drvdata(pdev);
+
+ nvme_dev_map(dev);
+ nvme_reset_ctrl_sync(&dev->ctrl);
+}
+
static const struct pci_error_handlers nvme_err_handler = {
.error_detected = nvme_error_detected,
.slot_reset = nvme_slot_reset,
@@ -3126,6 +3143,8 @@ static struct pci_driver nvme_driver = {
#endif
.sriov_configure = pci_sriov_configure_simple,
.err_handler = &nvme_err_handler,
+ .rescan_prepare = nvme_rescan_prepare,
+ .rescan_done = nvme_rescan_done,
};
static int __init nvme_init(void)
--
2.24.1
next prev parent reply other threads:[~2020-01-29 15:30 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-29 15:29 [PATCH v7 00/26] PCI: Allow BAR movement during boot and hotplug Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 01/26] PCI: Fix race condition in pci_enable/disable_device() Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 02/26] PCI: Enable bridge's I/O and MEM access for hotplugged devices Sergei Miroshnichenko
2020-01-30 23:12 ` Bjorn Helgaas
2020-01-29 15:29 ` [PATCH v7 03/26] PCI: hotplug: Initial support of the movable BARs feature Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 04/26] PCI: Add version of release_child_resources() aware of immovable BARs Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 05/26] PCI: hotplug: Fix reassigning the released BARs Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 06/26] PCI: hotplug: Recalculate every bridge window during rescan Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 07/26] PCI: hotplug: Don't allow hot-added devices to steal resources Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 08/26] PCI: hotplug: Try to reassign movable BARs only once Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 09/26] PCI: hotplug: Calculate immovable parts of bridge windows Sergei Miroshnichenko
2020-01-30 21:26 ` Bjorn Helgaas
2020-01-31 16:59 ` Sergei Miroshnichenko
2020-01-31 20:10 ` Bjorn Helgaas
2020-01-29 15:29 ` [PATCH v7 10/26] PCI: Include fixed and immovable BARs into the bus size calculating Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 11/26] PCI: hotplug: movable BARs: Compute limits for relocated bridge windows Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 12/26] PCI: Make sure bridge windows include their fixed BARs Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 13/26] PCI: hotplug: Add support of immovable BARs to pci_assign_resource() Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 14/26] PCI: hotplug: Sort immovable BARs before assignment Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 15/26] PCI: hotplug: Enable the movable BARs feature by default Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 16/26] PCI: Ignore PCIBIOS_MIN_MEM Sergei Miroshnichenko
2020-01-30 23:52 ` Bjorn Helgaas
2020-01-31 18:19 ` Sergei Miroshnichenko
2020-01-31 20:27 ` Bjorn Helgaas
2020-02-05 13:04 ` Sergei Miroshnichenko
2020-02-05 16:32 ` Bjorn Helgaas
2020-02-12 12:16 ` Sergei Miroshnichenko
2020-02-12 14:13 ` Bjorn Helgaas
2020-01-29 15:29 ` [PATCH v7 17/26] PCI: hotplug: Ignore the MEM BAR offsets from BIOS/bootloader Sergei Miroshnichenko
2020-01-31 20:31 ` Bjorn Helgaas
2020-02-05 11:01 ` Sergei Miroshnichenko
2020-02-05 16:42 ` Bjorn Helgaas
2020-02-12 12:29 ` Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 18/26] PCI: Treat VGA BARs as immovable Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 19/26] PCI: hotplug: Configure MPS for hot-added bridges during bus rescan Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 20/26] PNP: Don't reserve BARs for PCI when enabled movable BARs Sergei Miroshnichenko
2020-01-30 14:39 ` Rafael J. Wysocki
2020-01-30 21:14 ` Bjorn Helgaas
2020-01-31 15:48 ` Sergei Miroshnichenko
2020-01-31 19:39 ` Bjorn Helgaas
2020-01-29 15:29 ` [PATCH v7 21/26] PCI: hotplug: Don't disable the released bridge windows immediately Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 22/26] PCI: pciehp: Trigger a domain rescan on hp events when enabled movable BARs Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 23/26] PCI: Don't claim immovable BARs Sergei Miroshnichenko
2020-01-29 15:29 ` [PATCH v7 24/26] PCI: hotplug: Don't reserve bus space when enabled movable BARs Sergei Miroshnichenko
2020-01-29 15:29 ` Sergei Miroshnichenko [this message]
2020-01-29 15:29 ` [PATCH v7 26/26] PCI/portdrv: Declare support of " Sergei Miroshnichenko
2020-01-30 23:37 ` [PATCH v7 00/26] PCI: Allow BAR movement during boot and hotplug Bjorn Helgaas
2020-02-03 4:56 ` Oliver O'Halloran
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200129152937.311162-26-s.miroshnichenko@yadro.com \
--to=s.miroshnichenko@yadro.com \
--cc=hch@lst.de \
--cc=helgaas@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux@yadro.com \
--cc=sr@denx.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).