linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
To: <linux-pci@vger.kernel.org>, <linuxppc-dev@lists.ozlabs.org>
Cc: Bjorn Helgaas <helgaas@kernel.org>, <linux@yadro.com>,
	Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
Subject: [PATCH v6 00/30] PCI: Allow BAR movement during hotplug
Date: Thu, 24 Oct 2019 20:11:58 +0300	[thread overview]
Message-ID: <20191024171228.877974-1-s.miroshnichenko@yadro.com> (raw)

Currently PCI hotplug works on top of resources, which are usually reserved
not by the kernel, but by BIOS, bootloader, firmware, etc. These resources
are gaps in the address space where BARs of new devices may fit, and extra
bus number per port, so bridges can be hot-added. This series aim the
former problem: it shows the kernel how to redistribute on the run, so the
hotplug becomes predictable and cross-platform. A follow-up patchset will
propose a solution for bus numbers.

If the memory is arranged in a way that doesn't provide enough space for
BARs of a new hotplugged device, the kernel can pause the drivers of the
"obstructing" devices and move their BARs, so the new BARs can fit into the
freed spaces.

To rearrange the BARs and bridge windows these patches releases all of them
after a rescan and re-assigns in the same way as during the initial PCIe
topology scan at system boot.

When a driver is un-paused by the kernel after the PCIe rescan, it should
ioremap() the new addresses of its BARs.

Drivers indicate their support of the feature by implementing the new hooks
.rescan_prepare() and .rescan_done() in the struct pci_driver. If a driver
doesn't yet support the feature, BARs of its devices will be considered as
immovable (by checking the pci_dev_movable_bars_supported(dev)) and handled
in the same way as resources with the IORESOURCE_PCI_FIXED flag.

If a driver doesn't yet support the feature, its devices are guaranteed to
have their BARs remaining untouched.

Tested on:
 - x86_64 with "pci=pcie_bus_peer2peer"
 - POWER8 PowerNV+OPAL+PHB3 ppc64le with "pci=pcie_bus_peer2peer".

This patchset is a part of our work on adding support for hotplugging
bridges full of other bridges, NVME drives, SAS HBAs and GPUs without
special requirements such as Hot-Plug Controller, reservation of bus
numbers or memory regions by firmware, etc.

Changes since v5:
 - Simplified the disable flag, now it is "pci=no_movable_buses";
 - More deliberate marking the BARs as immovable;
 - Mark as immovable BARs which are used by unbound drivers;
 - Ignoring BAR assignment by non-kernel program components, so the kernel
   is able now to distribute BARs in optimal and predictable way;
 - Move here PowerNV-specific patches from the older "powerpc/powernv/pci:
   Make hotplug self-sufficient, independent of FW and DT" series;
 - Fix EEH cache rebuilding and PE allocation for PowerNV during rescan.

Changes since v4:
 - Feature is enabled by default (turned on by one of the latest patches);
 - Add pci_dev_movable_bars_supported(dev) instead of marking the immovable
   BARs with the IORESOURCE_PCI_FIXED flag;
 - Set up PCIe bridges during rescan via sysfs, so MPS settings are now
   configured not only during system boot or pcihp events;
 - Allow movement of switch's BARs if claimed by portdrv;
 - Update EEH address caches after rescan for powerpc;
 - Don't disable completely hot-added devices which can't have BARs being
   fit - just disable their BARs, so they are still visible in lspci etc;
 - Clearer names: fixed_range_hard -> immovable_range, fixed_range_soft ->
   realloc_range;
 - Drop the patch for pci_restore_config_space() - fixed by properly using
   the runtime PM.

Changes since v3:
 - Rebased to the upstream, so the patches apply cleanly again.

Changes since v2:
 - Fixed double-assignment of bridge windows;
 - Fixed assignment of fixed prefetched resources;
 - Fixed releasing of fixed resources;
 - Fixed a debug message;
 - Removed auto-enabling the movable BARs for x86 - let's rely on the
   "pcie_movable_bars=force" option for now;
 - Reordered the patches - bugfixes first.

Changes since v1:
 - Add a "pcie_movable_bars={ off | force }" command line argument;
 - Handle the IORESOURCE_PCI_FIXED flag properly;
 - Don't move BARs of devices which don't support the feature;
 - Guarantee that new hotplugged devices will not steal memory from working
   devices by ignoring the failing new devices with the new PCI_DEV_IGNORE
   flag;
 - Add rescan_prepare()+rescan_done() to the struct pci_driver instead of
   using the reset_prepare()+reset_done() from struct pci_error_handlers;
 - Add a bugfix of a race condition;
 - Fixed hotplug in a non-pre-enabled (by BIOS/firmware) bridge;
 - Fix the compatibility of the feature with pm_runtime and D3-state;
 - Hotplug events from pciehp also can move BARs;
 - Add support of the feature to the NVME driver.

Sergey Miroshnichenko (30):
  PCI: Fix race condition in pci_enable/disable_device()
  PCI: Enable bridge's I/O and MEM access for hotplugged devices
  PCI: hotplug: Add a flag for the movable BARs feature
  PCI: Define PCI-specific version of the release_child_resources()
  PCI: hotplug: movable BARs: Fix reassigning the released bridge
    windows
  PCI: hotplug: movable BARs: Recalculate all bridge windows during
    rescan
  PCI: hotplug: movable BARs: Don't disable the released bridge windows
  PCI: hotplug: movable BARs: Don't allow added devices to steal
    resources
  PCI: Include fixed and immovable BARs into the bus size calculating
  PCI: Prohibit assigning BARs and bridge windows to non-direct parents
  PCI: hotplug: movable BARs: Try to assign unassigned resources only
    once
  PCI: hotplug: movable BARs: Calculate immovable parts of bridge
    windows
  PCI: hotplug: movable BARs: Compute limits for relocated bridge
    windows
  PCI: Make sure bridge windows include their fixed BARs
  PCI: Fix assigning the fixed prefetchable resources
  PCI: hotplug: movable BARs: Assign fixed and immovable BARs before
    others
  PCI: hotplug: movable BARs: Don't reserve IO/mem bus space
  PCI: hotplug: Configure MPS for hot-added bridges during bus rescan
  PCI: hotplug: movable BARs: Ignore the MEM BAR offsets from bootloader
  powerpc/pci: Fix crash with enabled movable BARs
  powerpc/pci: Access PCI config space directly w/o pci_dn
  powerpc/pci: Create pci_dn on demand
  powerpc/pci: hotplug: Add support for movable BARs
  powerpc/powernv/pci: Suppress an EEH error when reading an empty slot
  PNP: Don't reserve BARs for PCI when enabled movable BARs
  PCI: hotplug: movable BARs: Enable the feature by default
  nvme-pci: Handle movable BARs
  PCI/portdrv: Declare support of movable BARs
  PCI: pciehp: movable BARs: Trigger a domain rescan on hp events
  Revert "powerpc/powernv/pci: Work around races in PCI bridge enabling"

 .../admin-guide/kernel-parameters.txt         |   1 +
 arch/powerpc/kernel/pci-hotplug.c             |  43 +++
 arch/powerpc/kernel/pci_dn.c                  |  88 ++++-
 arch/powerpc/kernel/rtas_pci.c                |  97 ++++--
 arch/powerpc/platforms/powernv/pci-ioda.c     |  40 +--
 arch/powerpc/platforms/powernv/pci.c          |  73 ++--
 arch/powerpc/platforms/pseries/setup.c        |   2 +
 drivers/nvme/host/pci.c                       |  21 +-
 drivers/pci/bus.c                             |   2 +-
 drivers/pci/hotplug/pciehp_pci.c              |   5 +
 drivers/pci/pci.c                             |  38 ++-
 drivers/pci/pci.h                             |  30 ++
 drivers/pci/pcie/portdrv_pci.c                |  11 +
 drivers/pci/probe.c                           | 315 +++++++++++++++++-
 drivers/pci/setup-bus.c                       | 277 +++++++++++++--
 drivers/pci/setup-res.c                       |  50 ++-
 drivers/pnp/system.c                          |   4 +
 include/linux/pci.h                           |  21 ++
 18 files changed, 965 insertions(+), 153 deletions(-)

-- 
2.23.0


             reply	other threads:[~2019-10-24 17:12 UTC|newest]

Thread overview: 35+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-24 17:11 Sergey Miroshnichenko [this message]
2019-10-24 17:11 ` [PATCH v6 01/30] PCI: Fix race condition in pci_enable/disable_device() Sergey Miroshnichenko
2019-10-25 14:33   ` Oxford Semiconductor Ltd OX16PCI954 - weird dmesg Carlo Pisani
2019-10-25 16:37     ` Bjorn Helgaas
2019-10-24 17:12 ` [PATCH v6 02/30] PCI: Enable bridge's I/O and MEM access for hotplugged devices Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 03/30] PCI: hotplug: Add a flag for the movable BARs feature Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 04/30] PCI: Define PCI-specific version of the release_child_resources() Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 05/30] PCI: hotplug: movable BARs: Fix reassigning the released bridge windows Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 06/30] PCI: hotplug: movable BARs: Recalculate all bridge windows during rescan Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 07/30] PCI: hotplug: movable BARs: Don't disable the released bridge windows Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 08/30] PCI: hotplug: movable BARs: Don't allow added devices to steal resources Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 09/30] PCI: Include fixed and immovable BARs into the bus size calculating Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 10/30] PCI: Prohibit assigning BARs and bridge windows to non-direct parents Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 11/30] PCI: hotplug: movable BARs: Try to assign unassigned resources only once Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 12/30] PCI: hotplug: movable BARs: Calculate immovable parts of bridge windows Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 13/30] PCI: hotplug: movable BARs: Compute limits for relocated " Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 14/30] PCI: Make sure bridge windows include their fixed BARs Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 15/30] PCI: Fix assigning the fixed prefetchable resources Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 16/30] PCI: hotplug: movable BARs: Assign fixed and immovable BARs before others Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 17/30] PCI: hotplug: movable BARs: Don't reserve IO/mem bus space Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 18/30] PCI: hotplug: Configure MPS for hot-added bridges during bus rescan Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 19/30] PCI: hotplug: movable BARs: Ignore the MEM BAR offsets from bootloader Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 20/30] powerpc/pci: Fix crash with enabled movable BARs Sergey Miroshnichenko
2019-10-25  1:22   ` Alexey Kardashevskiy
2019-10-24 17:12 ` [PATCH v6 21/30] powerpc/pci: Access PCI config space directly w/o pci_dn Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 22/30] powerpc/pci: Create pci_dn on demand Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 23/30] powerpc/pci: hotplug: Add support for movable BARs Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 24/30] powerpc/powernv/pci: Suppress an EEH error when reading an empty slot Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 25/30] PNP: Don't reserve BARs for PCI when enabled movable BARs Sergey Miroshnichenko
2019-10-27 17:40   ` kbuild test robot
2019-10-24 17:12 ` [PATCH v6 26/30] PCI: hotplug: movable BARs: Enable the feature by default Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 27/30] nvme-pci: Handle movable BARs Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 28/30] PCI/portdrv: Declare support of " Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 29/30] PCI: pciehp: movable BARs: Trigger a domain rescan on hp events Sergey Miroshnichenko
2019-10-24 17:12 ` [PATCH v6 30/30] Revert "powerpc/powernv/pci: Work around races in PCI bridge enabling" Sergey Miroshnichenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191024171228.877974-1-s.miroshnichenko@yadro.com \
    --to=s.miroshnichenko@yadro.com \
    --cc=helgaas@kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux@yadro.com \
    --cc=linuxppc-dev@lists.ozlabs.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).