Linux-PCI Archive on lore.kernel.org
 help / color / Atom feed
From: Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
To: <linux-pci@vger.kernel.org>
Cc: Bjorn Helgaas <helgaas@kernel.org>,
	Rajat Jain <rajatja@google.com>, <linux@yadro.com>,
	Sergey Miroshnichenko <s.miroshnichenko@yadro.com>,
	<linuxppc-dev@lists.ozlabs.org>,
	Benjamin Herrenschmidt <benh@kernel.crashing.org>,
	Oliver <oohall@gmail.com>, Sam Bobroff <sbobroff@linux.ibm.com>,
	Lukas Wunner <lukas@wunner.de>
Subject: [PATCH RFC v3 00/21] PCI: Allow BAR movement during hotplug
Date: Mon, 4 Feb 2019 18:35:40 +0300
Message-ID: <20190204153601.7576-1-s.miroshnichenko@yadro.com> (raw)

If the firmware or kernel has arranged memory for PCIe devices in a way that
doesn't provide enough space for BARs of a new hotplugged device, the kernel
can pause the drivers of the "obstructing" devices and move their BARs, so
new BARs can fit into the freed spaces.

When a driver is un-paused by the kernel after the PCIe rescan, it should
check if its BARs had moved, and ioremap() them if needed.

Drivers indicate their support of the feature by implementing the new
rescan_prepare() and rescan_done() hooks in the struct pci_driver. If a
driver doesn't yet support the feature, BARs of its devices will be marked
as immovable by the IORESOURCE_PCI_FIXED flag.

To re-arrange the BARs and bridge windows this patch releases all of them
after a rescan and re-assigns in the same way as during the initial PCIe
topology scan at system boot.

Tested on:
 - x86_64 with "pci=realloc,assign-busses,use_crs pcie_movable_bars=force"
 - ppc64le POWER8 PowerNV (with extra arch-specific patches which will be
   introduced later) with "pci=realloc pcie_movable_bars=force"

Not so many platforms and test cases were covered, so all who are interested
are highly welcome to test on your setups - the more exotic the better!

Changes since v2:
 - Fixed double-assignment of bridge windows;
 - Fixed assignment of fixed prefetched resources;
 - Fixed releasing of fixed resources;
 - Fixed a debug message;
 - Removed auto-enabling the movable BARs for x86 - let's rely on the
   "pcie_movable_bars=force" option for now;
 - Reordered the patches - bugfixes first.

Changes since v1:
 - Add a "pcie_movable_bars={ off | force }" command line argument;
 - Handle the IORESOURCE_PCI_FIXED flag properly;
 - Don't move BARs of devices which don't support the feature;
 - Guarantee that new hotplugged devices will not steal memory from working
   devices by ignoring the failing new devices with the new PCI_DEV_IGNORE
   flag;
 - Add rescan_prepare()+rescan_done() to the struct pci_driver instead of
   using the reset_prepare()+reset_done() from struct pci_error_handlers;
 - Add a bugfix of a race condition;
 - Fixed hotplug in a non-pre-enabled (by BIOS/firmware) bridge;
 - Fix the compatibility of the feature with pm_runtime and D3-state;
 - Hotplug events from pciehp also can move BARs;
 - Add support of the feature to the NVME driver.

This patchset is a part of our work on adding support for hotplugging
bridges full of NVME and GPU devices (without special requirement such as
Hot-Plug Controller, reservation of bus numbers and memory regions by
firmware, etc.). Next patchset will implement the movable bus numbers.

Sergey Miroshnichenko (21):
  PCI: Fix writing invalid BARs during pci_restore_state()
  PCI: Fix race condition in pci_enable/disable_device()
  PCI: Enable bridge's I/O and MEM access for hotplugged devices
  PCI: Define PCI-specific version of the release_child_resources()
  PCI: hotplug: Add a flag for the movable BARs feature
  PCI: Pause the devices with movable BARs during rescan
  PCI: Wake up bridges during rescan when movable BARs enabled
  nvme-pci: Handle movable BARs
  PCI: Mark immovable BARs with PCI_FIXED
  PCI: Fix assigning of fixed prefetchable resources
  PCI: Release and reassign the root bridge resources during rescan
  PCI: Don't allow hotplugged devices to steal resources
  PCI: Include fixed BARs into the bus size calculating
  PCI: Don't reserve memory for hotplug when enabled movable BARs
  PCI: Allow the failed resources to be reassigned later
  PCI: Calculate fixed areas of bridge windows based on fixed BARs
  PCI: Calculate boundaries for bridge windows
  PCI: Make sure bridge windows include their fixed BARs
  PCI: Prioritize fixed BAR assigning over the movable ones
  PCI: pciehp: Add support for the movable BARs feature
  powerpc/pci: Fix crash with enabled movable BARs

 .../admin-guide/kernel-parameters.txt         |   7 +
 arch/powerpc/platforms/powernv/pci-ioda.c     |   3 +-
 drivers/nvme/host/pci.c                       |  29 +-
 drivers/pci/bus.c                             |   7 +-
 drivers/pci/hotplug/pciehp_pci.c              |  14 +-
 drivers/pci/pci.c                             |  60 +++-
 drivers/pci/pci.h                             |  26 ++
 drivers/pci/probe.c                           | 271 +++++++++++++++++-
 drivers/pci/setup-bus.c                       | 245 ++++++++++++++--
 drivers/pci/setup-res.c                       |  43 ++-
 include/linux/pci.h                           |  14 +
 11 files changed, 678 insertions(+), 41 deletions(-)

-- 
2.20.1


             reply index

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-02-04 15:35 Sergey Miroshnichenko [this message]
2019-02-04 15:35 ` [PATCH RFC v3 01/21] PCI: Fix writing invalid BARs during pci_restore_state() Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 02/21] PCI: Fix race condition in pci_enable/disable_device() Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 03/21] PCI: Enable bridge's I/O and MEM access for hotplugged devices Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 04/21] PCI: Define PCI-specific version of the release_child_resources() Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 05/21] PCI: hotplug: Add a flag for the movable BARs feature Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 06/21] PCI: Pause the devices with movable BARs during rescan Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 07/21] PCI: Wake up bridges during rescan when movable BARs enabled Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 08/21] nvme-pci: Handle movable BARs Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 09/21] PCI: Mark immovable BARs with PCI_FIXED Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 10/21] PCI: Fix assigning of fixed prefetchable resources Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 11/21] PCI: Release and reassign the root bridge resources during rescan Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 12/21] PCI: Don't allow hotplugged devices to steal resources Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 13/21] PCI: Include fixed BARs into the bus size calculating Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 14/21] PCI: Don't reserve memory for hotplug when enabled movable BARs Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 15/21] PCI: Allow the failed resources to be reassigned later Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 16/21] PCI: Calculate fixed areas of bridge windows based on fixed BARs Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 17/21] PCI: Calculate boundaries for bridge windows Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 18/21] PCI: Make sure bridge windows include their fixed BARs Sergey Miroshnichenko
2019-02-04 15:35 ` [PATCH RFC v3 19/21] PCI: Prioritize fixed BAR assigning over the movable ones Sergey Miroshnichenko
2019-02-04 15:36 ` [PATCH RFC v3 20/21] PCI: pciehp: Add support for the movable BARs feature Sergey Miroshnichenko
2019-02-04 15:36 ` [PATCH RFC v3 21/21] powerpc/pci: Fix crash with enabled movable BARs Sergey Miroshnichenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190204153601.7576-1-s.miroshnichenko@yadro.com \
    --to=s.miroshnichenko@yadro.com \
    --cc=benh@kernel.crashing.org \
    --cc=helgaas@kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux@yadro.com \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=lukas@wunner.de \
    --cc=oohall@gmail.com \
    --cc=rajatja@google.com \
    --cc=sbobroff@linux.ibm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-PCI Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-pci/0 linux-pci/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-pci linux-pci/ https://lore.kernel.org/linux-pci \
		linux-pci@vger.kernel.org
	public-inbox-index linux-pci

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-pci


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git