[PATCH v2 0/7] PCIe Hotplug Slot Emulation driver

[PATCH v2 0/7] PCIe Hotplug Slot Emulation driver

[Jonathan Derrick]

Please see note [1] about v2

This set adds an emulation driver for PCIe Hotplug. There may be platforms with
specific configurations that can support hotplug but don't provide the logical
slot hotplug hardware. For instance, the platform may use an
electrically-tolerant interposer between the slot and the device.

This driver utilizes the pci-bridge-emul architecture to manage register reads
and writes. The underlying functionality of the hotplug emulation driver uses
the Data Link Layer Link Active Reporting mechanism in a polling loop, but can
tolerate other event sources such as AER or DPC.

When enabled and a slot is managed by the driver, all port services are managed
by the kernel. This is done to ensure that firmware hotplug and error
architecture does not (correctly) halt/machine check the system when hotplug is
performed on a non-hotplug slot.

The driver offers two active mode: Auto and Force.
auto: The driver will bind to non-hotplug slots
force: The driver will bind to all slots and overrides the slot's services

There are three kernel params:
pciehp.pciehp_emul_mode={off, auto, force}
pciehp.pciehp_emul_time=<msecs polling time> (def 1000, min 100, max 60000)
pciehp.pciehp_emul_ports=<PCI [S]BDF/ID format string>

The pciehp_emul_ports kernel parameter takes a semi-colon tokenized string
representing PCI [S]BDFs and IDs. The pciehp_emul_mode will then be applied to
only those slots, leaving other slots unmanaged by pciehp_emul.

The string follows the pci_dev_str_match() format:

  [<domain>:]<bus>:<device>.<func>[/<device>.<func>]*
  pci:<vendor>:<device>[:<subvendor>:<subdevice>]

When using the path format, the path for the device can be obtained
using 'lspci -t' and further specified using the upstream bridge and the
downstream port's device-function to be more robust against bus
renumbering.

When using the vendor-device format, a value of '0' in any field acts as
a wildcard for that field, matching all values.

The driver is enabled with CONFIG_HOTPLUG_PCI_PCIE_EMUL=y.

The driver should be considered 'use at own risk' unless the platform/hardware
vendor recommends this mode.

[1]
The main intent of posting v2 is to help Vidya further along with his
GPIO-based hotplug driver. I have no direct need for a DLLSC-based emulated
mode, and I'm not certain the model is appropriate. The GPIO-based model could
modify some of the logic in patch 6 to check for GPIO instead of DLLSC.

v1->v2: Indirects capacibility accessors instead of filtering pciehp slot operations

Jonathan Derrick (7):
  PCI: Allow for indirecting capability registers
  PCI: Add pcie_port_slot_emulated stub
  PCI: pciehp: Expose the poll loop to other drivers
  PCI: Move pci_dev_str_match to search.c
  PCI: pci-bridge-emul: Provide a helper to set behavior
  PCI: pciehp: Add hotplug slot emulation driver
  PCI: pciehp: Wire up pcie_port_emulate_slot and

 drivers/pci/access.c              |  29 +++
 drivers/pci/hotplug/Makefile      |   4 +
 drivers/pci/hotplug/pciehp.h      |  20 ++
 drivers/pci/hotplug/pciehp_emul.c | 380 ++++++++++++++++++++++++++++++
 drivers/pci/hotplug/pciehp_hpc.c  |  19 +-
 drivers/pci/pci-acpi.c            |   3 +
 drivers/pci/pci-bridge-emul.c     |  19 ++
 drivers/pci/pci-bridge-emul.h     |  10 +
 drivers/pci/pci.c                 | 163 -------------
 drivers/pci/pcie/Kconfig          |  13 +
 drivers/pci/pcie/portdrv_core.c   |  12 +-
 drivers/pci/probe.c               |   2 +-
 drivers/pci/search.c              | 162 +++++++++++++
 include/linux/pci.h               |  22 ++
 14 files changed, 688 insertions(+), 170 deletions(-)
 create mode 100644 drivers/pci/hotplug/pciehp_emul.c

-- 
2.30.2

[PATCH v2 1/7] PCI: Allow for indirecting capability registers

[Jonathan Derrick]

Allow another driver to provide alternative operations when doing
capability register reads and writes. The intention is to have
pcie_bridge_emul provide alternate handlers for the Slot Capabilities, Slot
Control, and Slot Status registers. Alternate handlers can return > 0 if
unhandled, errno on error, or 0 on success. This could potentially be
used to handle quirks in a different manner.

Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
 drivers/pci/access.c | 29 +++++++++++++++++++++++++++++
 include/linux/pci.h  | 11 +++++++++++
 2 files changed, 40 insertions(+)

diff --git a/drivers/pci/access.c b/drivers/pci/access.c
index 708c7529647f..dbfea6824bd4 100644
--- a/drivers/pci/access.c
+++ b/drivers/pci/access.c
@@ -424,6 +424,17 @@ int pcie_capability_read_word(struct pci_dev *dev, int pos, u16 *val)
 		return ret;
 	}
 
+	if (dev->caps_rw_ops) {
+		u32 reg;
+		ret = dev->caps_rw_ops->read(dev, pos, 4, &reg);
+		if (!ret) {
+			*val = reg & 0xffff;
+			return ret;
+		} else if (ret < 0) {
+			return ret;
+		}
+	}
+
 	/*
 	 * For Functions that do not implement the Slot Capabilities,
 	 * Slot Status, and Slot Control registers, these spaces must
@@ -459,6 +470,12 @@ int pcie_capability_read_dword(struct pci_dev *dev, int pos, u32 *val)
 		return ret;
 	}
 
+	if (dev->caps_rw_ops) {
+		ret = dev->caps_rw_ops->read(dev, pos, 4, val);
+		if (ret <= 0)
+			return ret;
+	}
+
 	if (pci_is_pcie(dev) && pcie_downstream_port(dev) &&
 	    pos == PCI_EXP_SLTSTA)
 		*val = PCI_EXP_SLTSTA_PDS;
@@ -475,6 +492,12 @@ int pcie_capability_write_word(struct pci_dev *dev, int pos, u16 val)
 	if (!pcie_capability_reg_implemented(dev, pos))
 		return 0;
 
+	if (dev->caps_rw_ops) {
+		int ret = dev->caps_rw_ops->write(dev, pos, 2, val);
+		if (ret <= 0)
+			return ret;
+	}
+
 	return pci_write_config_word(dev, pci_pcie_cap(dev) + pos, val);
 }
 EXPORT_SYMBOL(pcie_capability_write_word);
@@ -487,6 +510,12 @@ int pcie_capability_write_dword(struct pci_dev *dev, int pos, u32 val)
 	if (!pcie_capability_reg_implemented(dev, pos))
 		return 0;
 
+	if (dev->caps_rw_ops) {
+		int ret = dev->caps_rw_ops->write(dev, pos, 4, val);
+		if (ret <= 0)
+			return ret;
+	}
+
 	return pci_write_config_dword(dev, pci_pcie_cap(dev) + pos, val);
 }
 EXPORT_SYMBOL(pcie_capability_write_dword);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 2bda4a4e47e8..ff47ef83ab38 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -311,6 +311,15 @@ struct pci_vpd {
 	u8		cap;
 };
 
+/*
+ * Capability reads/write redirect
+ * Returns 0, errno, or > 0 if unhandled
+ */
+struct caps_rw_ops {
+	int (*read)(struct pci_dev *dev, int pos, int len, u32 *val);
+	int (*write)(struct pci_dev *dev, int pos, int len, u32 val);
+};
+
 struct irq_affinity;
 struct pcie_link_state;
 struct pci_sriov;
@@ -523,6 +532,8 @@ struct pci_dev {
 
 	/* These methods index pci_reset_fn_methods[] */
 	u8 reset_methods[PCI_NUM_RESET_METHODS]; /* In priority order */
+
+	struct caps_rw_ops *caps_rw_ops;
 };
 
 static inline struct pci_dev *pci_physfn(struct pci_dev *dev)
-- 
2.30.2

[PATCH v2 2/7] PCI: Add pcie_port_slot_emulated stub

[Jonathan Derrick]

Add the checks to allow an emulated slot. An emulated slot will use
native Hotplug, AER, and PME services. It also needs to specify itself
as a hotplug bridge in order for bridge sizing to account for hotplug
reserved windows.

Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
 drivers/pci/pci-acpi.c          | 3 +++
 drivers/pci/pcie/portdrv_core.c | 9 ++++++---
 drivers/pci/probe.c             | 2 +-
 include/linux/pci.h             | 2 ++
 4 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c
index a46fec776ad7..77a3c9e39966 100644
--- a/drivers/pci/pci-acpi.c
+++ b/drivers/pci/pci-acpi.c
@@ -798,6 +798,9 @@ bool pciehp_is_native(struct pci_dev *bridge)
 	if (!IS_ENABLED(CONFIG_HOTPLUG_PCI_PCIE))
 		return false;
 
+	if (pcie_port_slot_emulated(bridge))
+		return true;
+
 	pcie_capability_read_dword(bridge, PCI_EXP_SLTCAP, &slot_cap);
 	if (!(slot_cap & PCI_EXP_SLTCAP_HPC))
 		return false;
diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
index 1ac7fec47d6f..b3c1e7d4ff10 100644
--- a/drivers/pci/pcie/portdrv_core.c
+++ b/drivers/pci/pcie/portdrv_core.c
@@ -209,7 +209,8 @@ static int get_port_device_capability(struct pci_dev *dev)
 	int services = 0;
 
 	if (dev->is_hotplug_bridge &&
-	    (pcie_ports_native || host->native_pcie_hotplug)) {
+	    (pcie_ports_native || pcie_port_slot_emulated(dev) ||
+	     host->native_pcie_hotplug)) {
 		services |= PCIE_PORT_SERVICE_HP;
 
 		/*
@@ -222,14 +223,16 @@ static int get_port_device_capability(struct pci_dev *dev)
 
 #ifdef CONFIG_PCIEAER
 	if (dev->aer_cap && pci_aer_available() &&
-	    (pcie_ports_native || host->native_aer))
+	    (pcie_ports_native || pcie_port_slot_emulated(dev) ||
+	     host->native_aer))
 		services |= PCIE_PORT_SERVICE_AER;
 #endif
 
 	/* Root Ports and Root Complex Event Collectors may generate PMEs */
 	if ((pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
 	     pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC) &&
-	    (pcie_ports_native || host->native_pme)) {
+	    (pcie_ports_native || pcie_port_slot_emulated(dev) ||
+	     host->native_pme)) {
 		services |= PCIE_PORT_SERVICE_PME;
 
 		/*
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index b66fa42c4b1f..86ac4d223eba 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1574,7 +1574,7 @@ void set_pcie_hotplug_bridge(struct pci_dev *pdev)
 	u32 reg32;
 
 	pcie_capability_read_dword(pdev, PCI_EXP_SLTCAP, &reg32);
-	if (reg32 & PCI_EXP_SLTCAP_HPC)
+	if (reg32 & PCI_EXP_SLTCAP_HPC || pcie_port_slot_emulated(pdev))
 		pdev->is_hotplug_bridge = 1;
 }
 
diff --git a/include/linux/pci.h b/include/linux/pci.h
index ff47ef83ab38..09f704337955 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1655,6 +1655,8 @@ extern bool pcie_ports_native;
 #define pcie_ports_native	false
 #endif
 
+#define pcie_port_slot_emulated(dev) false
+
 #define PCIE_LINK_STATE_L0S		BIT(0)
 #define PCIE_LINK_STATE_L1		BIT(1)
 #define PCIE_LINK_STATE_CLKPM		BIT(2)
-- 
2.30.2

[PATCH v2 3/7] PCI: pciehp: Expose the poll loop to other drivers

[Jonathan Derrick]

Abstract the poll loop into an 'events pending' check and a 'handle
events' method in order to allow another driver to call the polling
loop.

Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
 drivers/pci/hotplug/pciehp.h     |  2 ++
 drivers/pci/hotplug/pciehp_hpc.c | 16 +++++++++++++---
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/hotplug/pciehp.h b/drivers/pci/hotplug/pciehp.h
index e0a614acee05..e221f5d12ad5 100644
--- a/drivers/pci/hotplug/pciehp.h
+++ b/drivers/pci/hotplug/pciehp.h
@@ -157,6 +157,8 @@ struct controller {
 #define NO_CMD_CMPL(ctrl)	((ctrl)->slot_cap & PCI_EXP_SLTCAP_NCCS)
 #define PSN(ctrl)		(((ctrl)->slot_cap & PCI_EXP_SLTCAP_PSN) >> 19)
 
+bool pciehp_events_pending(struct controller *ctrl);
+void pciehp_handle_events(struct controller *ctrl);
 void pciehp_request(struct controller *ctrl, int action);
 void pciehp_handle_button_press(struct controller *ctrl);
 void pciehp_handle_disable_request(struct controller *ctrl);
diff --git a/drivers/pci/hotplug/pciehp_hpc.c b/drivers/pci/hotplug/pciehp_hpc.c
index 040ae076ec0e..f711448e0a70 100644
--- a/drivers/pci/hotplug/pciehp_hpc.c
+++ b/drivers/pci/hotplug/pciehp_hpc.c
@@ -764,6 +764,17 @@ static irqreturn_t pciehp_ist(int irq, void *dev_id)
 	return ret;
 }
 
+bool pciehp_events_pending(struct controller *ctrl)
+{
+	return pciehp_isr(IRQ_NOTCONNECTED, ctrl) == IRQ_WAKE_THREAD ||
+	       atomic_read(&ctrl->pending_events);
+}
+
+void pciehp_handle_events(struct controller *ctrl)
+{
+	pciehp_ist(IRQ_NOTCONNECTED, ctrl);
+}
+
 static int pciehp_poll(void *data)
 {
 	struct controller *ctrl = data;
@@ -772,9 +783,8 @@ static int pciehp_poll(void *data)
 
 	while (!kthread_should_stop()) {
 		/* poll for interrupt events or user requests */
-		while (pciehp_isr(IRQ_NOTCONNECTED, ctrl) == IRQ_WAKE_THREAD ||
-		       atomic_read(&ctrl->pending_events))
-			pciehp_ist(IRQ_NOTCONNECTED, ctrl);
+		while (pciehp_events_pending(ctrl))
+			pciehp_handle_events(ctrl);
 
 		if (pciehp_poll_time <= 0 || pciehp_poll_time > 60)
 			pciehp_poll_time = 2; /* clamp to sane value */
-- 
2.30.2

[PATCH v2 4/7] PCI: Move pci_dev_str_match to search.c

[Jonathan Derrick]

The method which extracts a string descriptor of one or more PCI devices
and matches to a struct pci_dev is useful in general to other subsystems
needing to match parameters or sysfs strings. Move this function to
search.c for general use.

Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
 drivers/pci/pci.c    | 163 -------------------------------------------
 drivers/pci/search.c | 162 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h  |   5 ++
 3 files changed, 167 insertions(+), 163 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 2127aba3550b..b58a8f5a7654 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -250,169 +250,6 @@ void __iomem *pci_ioremap_wc_bar(struct pci_dev *pdev, int bar)
 EXPORT_SYMBOL_GPL(pci_ioremap_wc_bar);
 #endif
 
-/**
- * pci_dev_str_match_path - test if a path string matches a device
- * @dev: the PCI device to test
- * @path: string to match the device against
- * @endptr: pointer to the string after the match
- *
- * Test if a string (typically from a kernel parameter) formatted as a
- * path of device/function addresses matches a PCI device. The string must
- * be of the form:
- *
- *   [<domain>:]<bus>:<device>.<func>[/<device>.<func>]*
- *
- * A path for a device can be obtained using 'lspci -t'.  Using a path
- * is more robust against bus renumbering than using only a single bus,
- * device and function address.
- *
- * Returns 1 if the string matches the device, 0 if it does not and
- * a negative error code if it fails to parse the string.
- */
-static int pci_dev_str_match_path(struct pci_dev *dev, const char *path,
-				  const char **endptr)
-{
-	int ret;
-	unsigned int seg, bus, slot, func;
-	char *wpath, *p;
-	char end;
-
-	*endptr = strchrnul(path, ';');
-
-	wpath = kmemdup_nul(path, *endptr - path, GFP_ATOMIC);
-	if (!wpath)
-		return -ENOMEM;
-
-	while (1) {
-		p = strrchr(wpath, '/');
-		if (!p)
-			break;
-		ret = sscanf(p, "/%x.%x%c", &slot, &func, &end);
-		if (ret != 2) {
-			ret = -EINVAL;
-			goto free_and_exit;
-		}
-
-		if (dev->devfn != PCI_DEVFN(slot, func)) {
-			ret = 0;
-			goto free_and_exit;
-		}
-
-		/*
-		 * Note: we don't need to get a reference to the upstream
-		 * bridge because we hold a reference to the top level
-		 * device which should hold a reference to the bridge,
-		 * and so on.
-		 */
-		dev = pci_upstream_bridge(dev);
-		if (!dev) {
-			ret = 0;
-			goto free_and_exit;
-		}
-
-		*p = 0;
-	}
-
-	ret = sscanf(wpath, "%x:%x:%x.%x%c", &seg, &bus, &slot,
-		     &func, &end);
-	if (ret != 4) {
-		seg = 0;
-		ret = sscanf(wpath, "%x:%x.%x%c", &bus, &slot, &func, &end);
-		if (ret != 3) {
-			ret = -EINVAL;
-			goto free_and_exit;
-		}
-	}
-
-	ret = (seg == pci_domain_nr(dev->bus) &&
-	       bus == dev->bus->number &&
-	       dev->devfn == PCI_DEVFN(slot, func));
-
-free_and_exit:
-	kfree(wpath);
-	return ret;
-}
-
-/**
- * pci_dev_str_match - test if a string matches a device
- * @dev: the PCI device to test
- * @p: string to match the device against
- * @endptr: pointer to the string after the match
- *
- * Test if a string (typically from a kernel parameter) matches a specified
- * PCI device. The string may be of one of the following formats:
- *
- *   [<domain>:]<bus>:<device>.<func>[/<device>.<func>]*
- *   pci:<vendor>:<device>[:<subvendor>:<subdevice>]
- *
- * The first format specifies a PCI bus/device/function address which
- * may change if new hardware is inserted, if motherboard firmware changes,
- * or due to changes caused in kernel parameters. If the domain is
- * left unspecified, it is taken to be 0.  In order to be robust against
- * bus renumbering issues, a path of PCI device/function numbers may be used
- * to address the specific device.  The path for a device can be determined
- * through the use of 'lspci -t'.
- *
- * The second format matches devices using IDs in the configuration
- * space which may match multiple devices in the system. A value of 0
- * for any field will match all devices. (Note: this differs from
- * in-kernel code that uses PCI_ANY_ID which is ~0; this is for
- * legacy reasons and convenience so users don't have to specify
- * FFFFFFFFs on the command line.)
- *
- * Returns 1 if the string matches the device, 0 if it does not and
- * a negative error code if the string cannot be parsed.
- */
-static int pci_dev_str_match(struct pci_dev *dev, const char *p,
-			     const char **endptr)
-{
-	int ret;
-	int count;
-	unsigned short vendor, device, subsystem_vendor, subsystem_device;
-
-	if (strncmp(p, "pci:", 4) == 0) {
-		/* PCI vendor/device (subvendor/subdevice) IDs are specified */
-		p += 4;
-		ret = sscanf(p, "%hx:%hx:%hx:%hx%n", &vendor, &device,
-			     &subsystem_vendor, &subsystem_device, &count);
-		if (ret != 4) {
-			ret = sscanf(p, "%hx:%hx%n", &vendor, &device, &count);
-			if (ret != 2)
-				return -EINVAL;
-
-			subsystem_vendor = 0;
-			subsystem_device = 0;
-		}
-
-		p += count;
-
-		if ((!vendor || vendor == dev->vendor) &&
-		    (!device || device == dev->device) &&
-		    (!subsystem_vendor ||
-			    subsystem_vendor == dev->subsystem_vendor) &&
-		    (!subsystem_device ||
-			    subsystem_device == dev->subsystem_device))
-			goto found;
-	} else {
-		/*
-		 * PCI Bus, Device, Function IDs are specified
-		 * (optionally, may include a path of devfns following it)
-		 */
-		ret = pci_dev_str_match_path(dev, p, &p);
-		if (ret < 0)
-			return ret;
-		else if (ret)
-			goto found;
-	}
-
-	*endptr = p;
-	return 0;
-
-found:
-	*endptr = p;
-	return 1;
-}
-
 static u8 __pci_find_next_cap_ttl(struct pci_bus *bus, unsigned int devfn,
 				  u8 pos, int cap, int *ttl)
 {
diff --git a/drivers/pci/search.c b/drivers/pci/search.c
index b4c138a6ec02..059fc5b9db4c 100644
--- a/drivers/pci/search.c
+++ b/drivers/pci/search.c
@@ -390,3 +390,165 @@ int pci_dev_present(const struct pci_device_id *ids)
 	return 0;
 }
 EXPORT_SYMBOL(pci_dev_present);
+
+/**
+ * pci_dev_str_match_path - test if a path string matches a device
+ * @dev: the PCI device to test
+ * @path: string to match the device against
+ * @endptr: pointer to the string after the match
+ *
+ * Test if a string (typically from a kernel parameter) formatted as a
+ * path of device/function addresses matches a PCI device. The string must
+ * be of the form:
+ *
+ *   [<domain>:]<bus>:<device>.<func>[/<device>.<func>]*
+ *
+ * A path for a device can be obtained using 'lspci -t'.  Using a path
+ * is more robust against bus renumbering than using only a single bus,
+ * device and function address.
+ *
+ * Returns 1 if the string matches the device, 0 if it does not and
+ * a negative error code if it fails to parse the string.
+ */
+static int pci_dev_str_match_path(struct pci_dev *dev, const char *path,
+				  const char **endptr)
+{
+	int ret;
+	unsigned int seg, bus, slot, func;
+	char *wpath, *p;
+	char end;
+
+	*endptr = strchrnul(path, ';');
+
+	wpath = kmemdup_nul(path, *endptr - path, GFP_ATOMIC);
+	if (!wpath)
+		return -ENOMEM;
+
+	while (1) {
+		p = strrchr(wpath, '/');
+		if (!p)
+			break;
+		ret = sscanf(p, "/%x.%x%c", &slot, &func, &end);
+		if (ret != 2) {
+			ret = -EINVAL;
+			goto free_and_exit;
+		}
+
+		if (dev->devfn != PCI_DEVFN(slot, func)) {
+			ret = 0;
+			goto free_and_exit;
+		}
+
+		/*
+		 * Note: we don't need to get a reference to the upstream
+		 * bridge because we hold a reference to the top level
+		 * device which should hold a reference to the bridge,
+		 * and so on.
+		 */
+		dev = pci_upstream_bridge(dev);
+		if (!dev) {
+			ret = 0;
+			goto free_and_exit;
+		}
+
+		*p = 0;
+	}
+
+	ret = sscanf(wpath, "%x:%x:%x.%x%c", &seg, &bus, &slot,
+		     &func, &end);
+	if (ret != 4) {
+		seg = 0;
+		ret = sscanf(wpath, "%x:%x.%x%c", &bus, &slot, &func, &end);
+		if (ret != 3) {
+			ret = -EINVAL;
+			goto free_and_exit;
+		}
+	}
+
+	ret = (seg == pci_domain_nr(dev->bus) &&
+	       bus == dev->bus->number &&
+	       dev->devfn == PCI_DEVFN(slot, func));
+
+free_and_exit:
+	kfree(wpath);
+	return ret;
+}
+
+/**
+ * pci_dev_str_match - test if a string matches a device
+ * @dev: the PCI device to test
+ * @p: string to match the device against
+ * @endptr: pointer to the string after the match
+ *
+ * Test if a string (typically from a kernel parameter) matches a specified
+ * PCI device. The string may be of one of the following formats:
+ *
+ *   [<domain>:]<bus>:<device>.<func>[/<device>.<func>]*
+ *   pci:<vendor>:<device>[:<subvendor>:<subdevice>]
+ *
+ * The first format specifies a PCI bus/device/function address which
+ * may change if new hardware is inserted, if motherboard firmware changes,
+ * or due to changes caused in kernel parameters. If the domain is
+ * left unspecified, it is taken to be 0.  In order to be robust against
+ * bus renumbering issues, a path of PCI device/function numbers may be used
+ * to address the specific device.  The path for a device can be determined
+ * through the use of 'lspci -t'.
+ *
+ * The second format matches devices using IDs in the configuration
+ * space which may match multiple devices in the system. A value of 0
+ * for any field will match all devices. (Note: this differs from
+ * in-kernel code that uses PCI_ANY_ID which is ~0; this is for
+ * legacy reasons and convenience so users don't have to specify
+ * FFFFFFFFs on the command line.)
+ *
+ * Returns 1 if the string matches the device, 0 if it does not and
+ * a negative error code if the string cannot be parsed.
+ */
+int pci_dev_str_match(struct pci_dev *dev, const char *p, const char **endptr)
+{
+	int ret;
+	int count;
+	unsigned short vendor, device, subsystem_vendor, subsystem_device;
+
+	if (strncmp(p, "pci:", 4) == 0) {
+		/* PCI vendor/device (subvendor/subdevice) IDs are specified */
+		p += 4;
+		ret = sscanf(p, "%hx:%hx:%hx:%hx%n", &vendor, &device,
+			     &subsystem_vendor, &subsystem_device, &count);
+		if (ret != 4) {
+			ret = sscanf(p, "%hx:%hx%n", &vendor, &device, &count);
+			if (ret != 2)
+				return -EINVAL;
+
+			subsystem_vendor = 0;
+			subsystem_device = 0;
+		}
+
+		p += count;
+
+		if ((!vendor || vendor == dev->vendor) &&
+		    (!device || device == dev->device) &&
+		    (!subsystem_vendor ||
+			    subsystem_vendor == dev->subsystem_vendor) &&
+		    (!subsystem_device ||
+			    subsystem_device == dev->subsystem_device))
+			goto found;
+	} else {
+		/*
+		 * PCI Bus, Device, Function IDs are specified
+		 * (optionally, may include a path of devfns following it)
+		 */
+		ret = pci_dev_str_match_path(dev, p, &p);
+		if (ret < 0)
+			return ret;
+		else if (ret)
+			goto found;
+	}
+
+	*endptr = p;
+	return 0;
+
+found:
+	*endptr = p;
+	return 1;
+}
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 09f704337955..0c907f94bb61 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1182,6 +1182,7 @@ struct pci_dev *pci_get_domain_bus_and_slot(int domain, unsigned int bus,
 					    unsigned int devfn);
 struct pci_dev *pci_get_class(unsigned int class, struct pci_dev *from);
 int pci_dev_present(const struct pci_device_id *ids);
+int pci_dev_str_match(struct pci_dev *dev, const char *p, const char **endptr);
 
 int pci_bus_read_config_byte(struct pci_bus *bus, unsigned int devfn,
 			     int where, u8 *val);
@@ -1816,6 +1817,10 @@ static inline struct pci_dev *pci_get_class(unsigned int class,
 static inline int pci_dev_present(const struct pci_device_id *ids)
 { return 0; }
 
+static inline int pci_dev_str_match(struct pci_dev *dev, const char *p,
+				    const char **endptr)
+{ return 0; }
+
 #define no_pci_devices()	(1)
 #define pci_dev_put(dev)	do { } while (0)
 
-- 
2.30.2

[PATCH v2 5/7] PCI: pci-bridge-emul: Provide a helper to set behavior

[Jonathan Derrick]

Add a handler to set behavior of a PCI or PCIe register. Add the
appropriate enums to specify the register's Read-Only, Read-Write, and
Write-1-to-Clear behaviors.

Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
 drivers/pci/pci-bridge-emul.c | 19 +++++++++++++++++++
 drivers/pci/pci-bridge-emul.h | 10 ++++++++++
 2 files changed, 29 insertions(+)

diff --git a/drivers/pci/pci-bridge-emul.c b/drivers/pci/pci-bridge-emul.c
index 9334b2dd4764..3c1a683ece66 100644
--- a/drivers/pci/pci-bridge-emul.c
+++ b/drivers/pci/pci-bridge-emul.c
@@ -46,6 +46,25 @@ struct pci_bridge_reg_behavior {
 	u32 w1c;
 };
 
+void pci_bridge_emul_set_reg_behavior(struct pci_bridge_emul *bridge,
+				      bool pcie, int reg, u32 val,
+				      enum pci_bridge_emul_reg_behavior type)
+{
+	struct pci_bridge_reg_behavior *behavior;
+
+	if (pcie)
+		behavior = &bridge->pcie_cap_regs_behavior[reg / 4];
+	else
+		behavior = &bridge->pci_regs_behavior[reg / 4];
+
+	if (type == PCI_BRIDGE_EMUL_REG_BEHAVIOR_RO)
+		behavior->ro = val;
+	else if (type == PCI_BRIDGE_EMUL_REG_BEHAVIOR_RW)
+		behavior->rw = val;
+	else /* PCI_BRIDGE_EMUL_REG_BEHAVIOR_W1C */
+		behavior->w1c = val;
+}
+
 static const
 struct pci_bridge_reg_behavior pci_regs_behavior[PCI_STD_HEADER_SIZEOF / 4] = {
 	[PCI_VENDOR_ID / 4] = { .ro = ~0 },
diff --git a/drivers/pci/pci-bridge-emul.h b/drivers/pci/pci-bridge-emul.h
index 2a0e59c7f0d9..b2401d58518c 100644
--- a/drivers/pci/pci-bridge-emul.h
+++ b/drivers/pci/pci-bridge-emul.h
@@ -72,6 +72,12 @@ struct pci_bridge_emul;
 typedef enum { PCI_BRIDGE_EMUL_HANDLED,
 	       PCI_BRIDGE_EMUL_NOT_HANDLED } pci_bridge_emul_read_status_t;
 
+enum pci_bridge_emul_reg_behavior {
+	PCI_BRIDGE_EMUL_REG_BEHAVIOR_RO,
+	PCI_BRIDGE_EMUL_REG_BEHAVIOR_RW,
+	PCI_BRIDGE_EMUL_REG_BEHAVIOR_W1C,
+};
+
 struct pci_bridge_emul_ops {
 	/*
 	 * Called when reading from the regular PCI bridge
@@ -161,4 +167,8 @@ int pci_bridge_emul_conf_read(struct pci_bridge_emul *bridge, int where,
 int pci_bridge_emul_conf_write(struct pci_bridge_emul *bridge, int where,
 			       int size, u32 value);
 
+void pci_bridge_emul_set_reg_behavior(struct pci_bridge_emul *bridge,
+				      bool pcie, int reg, u32 val,
+				      enum pci_bridge_emul_reg_behavior type);
+
 #endif /* __PCI_BRIDGE_EMUL_H__ */
-- 
2.30.2

[PATCH v2 6/7] PCI: pciehp: Add hotplug slot emulation driver

[Jonathan Derrick]

This adds the PCIe Hotplug emulation driver. There may be platforms with
specific configurations that can support hotplug but don't provide the
logical slot hotplug hardware. For instance, the platform may use an
electrically-tolerant interposer between the slot and the device.

This driver utilizes the pci-bridge-emul architecture to manage register
reads and writes. The underlying functionality of the hotplug emulation
driver uses the Data Link Layer Link Active Reporting mechanism in a
polling loop, but can tolerate other event sources such as AER or DPC.

When enabled and a slot is managed by the driver, all PCIe Port services
are managed by the kernel. This is done to ensure that firmware hotplug
and error architecture does not (correctly) machine check the system
when hotplug is performed on a non-hotplug slot.

The driver offers two active mode: auto and force. auto: Bind to
non-hotplug slots force: Bind to all slots and overrides the slot's
services

There are three kernel parameters:
pciehp.pciehp_emul_mode={off, auto, force}
pciehp.pciehp_emul_time=<msecs poll time> (def 1000, min 100, max 60000)
pciehp.pciehp_emul_ports=<PCI [S]BDF/ID format string>

The pciehp_emul_ports parameter takes a semi-colon tokenized string
representing PCI [S]BDFs and IDs. The pciehp_emul_mode will then be
applied to only those slots, leaving other slots unmanaged by
pciehp_emul.

The string follows the pci_dev_str_match() format:

  [<domain>:]<bus>:<device>.<func>[/<device>.<func>]*
  pci:<vendor>:<device>[:<subvendor>:<subdevice>]

When using the path format, the path for the device can be obtained
using 'lspci -t' and further specified using the upstream bridge and the
downstream port's device-function to be more robust against bus
renumbering.

When using the vendor-device format, a value of '0' in any field acts as
a wildcard for that field, matching all values.

The driver is enabled with CONFIG_HOTPLUG_PCI_PCIE_EMUL=y.

Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
 drivers/pci/hotplug/Makefile      |   4 +
 drivers/pci/hotplug/pciehp_emul.c | 372 ++++++++++++++++++++++++++++++
 drivers/pci/pcie/Kconfig          |  13 ++
 3 files changed, 389 insertions(+)
 create mode 100644 drivers/pci/hotplug/pciehp_emul.c

diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
index 5196983220df..4c01ecaa1dde 100644
--- a/drivers/pci/hotplug/Makefile
+++ b/drivers/pci/hotplug/Makefile
@@ -65,6 +65,10 @@ pciehp-objs		:=	pciehp_core.o	\
 				pciehp_pci.o	\
 				pciehp_hpc.o
 
+ifdef CONFIG_HOTPLUG_PCI_PCIE_EMUL
+pciehp-objs		+=	pciehp_emul.o
+endif
+
 shpchp-objs		:=	shpchp_core.o	\
 				shpchp_ctrl.o	\
 				shpchp_pci.o	\
diff --git a/drivers/pci/hotplug/pciehp_emul.c b/drivers/pci/hotplug/pciehp_emul.c
new file mode 100644
index 000000000000..716ec57feb79
--- /dev/null
+++ b/drivers/pci/hotplug/pciehp_emul.c
@@ -0,0 +1,372 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCI Express PCI Hot Plug Slot Emulation Driver
+ * Author: Jon Derrick <jonathan.derrick@linux.dev>
+ *
+ * Copyright (C) 2022 Solidigm
+ */
+
+#define dev_fmt(fmt) "pciehp: " fmt
+
+#include <linux/moduleparam.h>
+#include <linux/kthread.h>
+
+#include "../pci.h"
+#include "../pci-bridge-emul.h"
+#include "pciehp.h"
+
+static char *pciehp_emul_ports;
+module_param(pciehp_emul_ports, charp, 0444);
+MODULE_PARM_DESC(pciehp_emul_ports, "list of port BDF/IDs (optional)");
+
+static unsigned int pciehp_emul_time = 1000;
+module_param(pciehp_emul_time, uint, 0644);
+MODULE_PARM_DESC(pciehp_emul_time, "link polling delay in msecs");
+
+enum {
+	PCIEHP_EMUL_OFF,
+	PCIEHP_EMUL_AUTO,
+	PCIEHP_EMUL_FORCE,
+};
+
+static const char *const pciehp_emul_mode_strings[] = {
+	"off",
+	"auto",
+	"force",
+};
+
+static int pciehp_emul_mode;
+MODULE_PARM_DESC(pciehp_emul_mode, "slot emulation mode");
+
+static int pciehp_emul_mode_set(const char *str, const struct kernel_param *kp)
+{
+	int i;
+
+	i = match_string(pciehp_emul_mode_strings,
+			 ARRAY_SIZE(pciehp_emul_mode_strings), str);
+	if (i < 0)
+		return i;
+
+	pciehp_emul_mode = i;
+	return 0;
+}
+
+static int pciehp_emul_mode_get(char *buffer, const struct kernel_param *kp)
+{
+	return sprintf(buffer, "%s\n", pciehp_emul_mode_strings[pciehp_emul_mode]);
+}
+
+static const struct kernel_param_ops pciehp_emul_mode_ops = {
+	.set = pciehp_emul_mode_set,
+	.get = pciehp_emul_mode_get,
+};
+
+module_param_cb(pciehp_emul_mode, &pciehp_emul_mode_ops, &pciehp_emul_mode, 0444);
+
+static LIST_HEAD(pciehp_emul_list);
+static DEFINE_MUTEX(list_lock);
+
+/**
+ * struct pciehp_emul - Emulated PCIe Hotplug Slot structure
+ * @node:		List of PCIe Hotplug Emulated PCI devices.
+ * @ctrl:		Slot's pciehp controller.
+ * @bridge:		PCI Bridge Emul descriptor. Emulates the common register
+ *			read/write behaviors.
+ * @poll_thread:	Per-slot Data Link Layer Link Active polling thread.
+ */
+struct pciehp_emul {
+	struct list_head node;
+	struct controller *ctrl;
+	struct pci_bridge_emul bridge;
+	struct task_struct *poll_thread;
+};
+
+/* Emul Bridge OPS */
+static struct pci_bridge_emul_ops pciehp_emul_pci_bridge_ops = {};
+
+static struct pciehp_emul *pci_dev_to_pciehp_emul(struct pci_dev *dev)
+{
+	struct pciehp_emul *emul;
+
+	list_for_each_entry(emul, &pciehp_emul_list, node)
+		if (emul->ctrl->pcie->port == dev)
+			return emul;
+
+	return NULL;
+}
+
+static int pciehp_emul_slot_rw(struct pci_dev *dev, int pos, int len,
+			       u32 *val, int dir)
+{
+	struct pciehp_emul *emul;
+	int ret;
+
+	mutex_lock(&list_lock);
+	emul = pci_dev_to_pciehp_emul(dev);
+	if (!emul) {
+		mutex_unlock(&list_lock);
+		return PCIBIOS_DEVICE_NOT_FOUND;
+	}
+
+	if (dir == READ)
+		ret = pci_bridge_emul_conf_read(&emul->bridge,
+						pos + PCI_STD_HEADER_SIZEOF,
+						len, val);
+	else
+		ret = pci_bridge_emul_conf_write(&emul->bridge,
+						 pos + PCI_STD_HEADER_SIZEOF,
+						 len, *val);
+	mutex_unlock(&list_lock);
+
+	return ret;
+}
+
+/* Assuming aligned accesses to slot registers */
+static int pciehp_emul_slot_read(struct pci_dev *dev, int pos, int len, u32 *val)
+{
+	if (pos >= PCI_EXP_SLTCAP && pos <= PCI_EXP_SLTSTA)
+		return pciehp_emul_slot_rw(dev, pos, len, val, READ);
+	return 1;
+}
+
+static int pciehp_emul_slot_write(struct pci_dev *dev, int pos, int len, u32 val)
+{
+	if (pos >= PCI_EXP_SLTCAP && pos <= PCI_EXP_SLTSTA)
+		return pciehp_emul_slot_rw(dev, pos, len, &val, WRITE);
+	return 1;
+}
+
+static struct caps_rw_ops pciehp_emul_slot_ops = {
+	.read = pciehp_emul_slot_read,
+	.write = pciehp_emul_slot_write,
+};
+
+static int pciehp_emul_thread(void *data)
+{
+	struct pciehp_emul *emul = data;
+	struct controller *ctrl = emul->ctrl;
+	struct pci_dev *dev = ctrl->pcie->port;
+	__le16 *emul_lnksta_p = &emul->bridge.pcie_conf.lnksta;
+	__le16 *emul_slotsta_p = &emul->bridge.pcie_conf.slotsta;
+	u16 emul_lnksta, port_lnksta;
+	int ret;
+
+	/* Store initial link status state */
+	ret = pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &port_lnksta);
+	if (!ret && port_lnksta != (u16)~0)
+		*emul_lnksta_p = cpu_to_le16(port_lnksta);
+
+	/* Start with 1 sec delay */
+	schedule_timeout_idle(1 * HZ);
+
+	/*
+	 * Data Link Layer Link Active (DLLLA) event sets Data Link Layer State
+	 * Changed (DLLSC) and updates internal link status tracking.
+	 */
+	set_current_state(TASK_INTERRUPTIBLE);
+	while (!kthread_should_stop()) {
+		ret = pcie_capability_read_word(dev, PCI_EXP_LNKSTA,
+						&port_lnksta);
+		if (ret || port_lnksta == (u16)~0)
+			goto cont;
+
+		emul_lnksta = le16_to_cpup(emul_lnksta_p);
+		if ((port_lnksta ^ emul_lnksta) & PCI_EXP_LNKSTA_DLLLA) {
+			ctrl_dbg(ctrl, "link change event (%04x->%04x)\n",
+				 emul_lnksta, port_lnksta);
+			*emul_lnksta_p = cpu_to_le16(port_lnksta);
+			*emul_slotsta_p |= cpu_to_le16(PCI_EXP_SLTSTA_DLLSC);
+
+			while (pciehp_events_pending(ctrl))
+				pciehp_handle_events(ctrl);
+		}
+
+cont:
+		/* Clamp to sane value */
+		pciehp_emul_time = clamp(pciehp_emul_time, 100U, 60000U);
+		schedule_timeout_idle(msecs_to_jiffies(pciehp_emul_time));
+	}
+
+	return 0;
+}
+
+/**
+ * pciehp_emul_ports_check - Matches pci_dev against pciehp_emul_ports filter
+ * @dev:	Struct pci_dev to match against string
+ *
+ * Matches a semi-colon tokenized string list of PCI addresses or vendor-device
+ * IDs to the struct pci_dev. Follows the pci_dev_str_match format in the form:
+ *
+ *   [<domain>:]<bus>:<device>.<func>[/<device>.<func>]*
+ *   pci:<vendor>:<device>[:<subvendor>:<subdevice>]
+ *
+ * When using the path format, the path for the device can be obtained using
+ * 'lspci -t' and further specified using the upstream bridge and the
+ * downstream port's device-function to be more robust against bus renumbering.
+ *
+ * When using the vendor-device format, a value of '0' in any field acts as a
+ * wildcard for that field, matching all values.
+ *
+ * Returns 0 on match or null/empty string.
+ */
+static int pciehp_emul_ports_check(struct pci_dev *dev)
+{
+	const char *p = pciehp_emul_ports;
+	int ret;
+
+	/* Null or empty string matches all devices */
+	if (!p || !*p)
+		return 0;
+
+	while (*p) {
+		ret = pci_dev_str_match(dev, p, &p);
+		if (ret < 0) {
+			pr_info_once("Can't parse pciehp_emul_ports parameter: %s\n",
+				     pciehp_emul_ports);
+			return ret;
+		}
+
+		/* Found a match */
+		if (ret == 1)
+			return 0;
+
+		if (*p != ';' && *p != ',') {
+			/* End of param or invalid format */
+			return -ENODEV;
+		}
+		p++;
+	}
+
+	return -ENODEV;
+}
+
+bool pciehp_emul_will_manage(struct pci_dev *dev)
+{
+	u32 lnkcap;
+	u16 sltcap;
+	int ret;
+
+	if (pciehp_emul_mode == PCIEHP_EMUL_OFF)
+		return false;
+
+	/* A physical slot must be present for emulation to be useful. */
+	if (!(pcie_caps_reg(dev) & PCI_EXP_FLAGS_SLOT))
+		return false;
+
+	/* Match a port BDF/ID descriptor if given */
+	if (pciehp_emul_ports_check(dev))
+		return false;
+
+	/* Port must support Data Link Layer Link Active (DLLLA) Reporting. */
+	ret = pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &lnkcap);
+	if (ret || lnkcap == (u32)~0)
+		return false;
+	if (!(lnkcap & PCI_EXP_LNKCAP_DLLLARC))
+		return false;
+
+	/* Require 'force' if the port is already hotplug-capable */
+	ret = pcie_capability_read_word(dev, PCI_EXP_SLTCAP, &sltcap);
+	if (ret || sltcap == (u16)~0)
+		return false;
+	if (sltcap & PCI_EXP_SLTCAP_HPC && pciehp_emul_mode != PCIEHP_EMUL_FORCE)
+		return false;
+
+	return true;
+}
+
+int pciehp_emul_attach(struct controller *ctrl)
+{
+	struct pciehp_emul *emul;
+	struct pci_bridge_emul *bridge;
+	struct pci_dev *dev = ctrl->pcie->port;
+	u32 slotcap = 0;
+	int ret;
+
+	emul = kzalloc(sizeof(*emul), GFP_KERNEL);
+	if (!emul)
+		return -ENOMEM;
+
+	ret = pcie_capability_read_dword(dev, PCI_EXP_SLTCAP, &slotcap);
+	if (ret)
+		goto free_emul;
+
+	emul->ctrl = ctrl;
+	bridge = &emul->bridge;
+	bridge->has_pcie = true;
+	bridge->data = emul;
+	bridge->ops = &pciehp_emul_pci_bridge_ops;
+	pci_bridge_emul_init(bridge, 0);
+
+	pci_bridge_emul_set_reg_behavior(bridge, true, PCI_EXP_SLTCTL,
+					 ~PCI_EXP_SLTCTL_DLLSCE,
+					 PCI_BRIDGE_EMUL_REG_BEHAVIOR_RO);
+
+	bridge->pcie_conf.slotcap = cpu_to_le32(PCI_EXP_SLTCAP_HPS |
+						PCI_EXP_SLTCAP_HPC |
+						PCI_EXP_SLTCAP_NCCS |
+						(slotcap & PCI_EXP_SLTCAP_PSN));
+	bridge->pcie_conf.slotsta = cpu_to_le16(PCI_EXP_SLTSTA_PDS);
+
+	mutex_lock(&list_lock);
+	list_add_tail(&emul->node, &pciehp_emul_list);
+
+	dev->caps_rw_ops = &pciehp_emul_slot_ops;
+	mutex_unlock(&list_lock);
+
+	emul->poll_thread = kthread_run(&pciehp_emul_thread, emul,
+					"emul-%05x:%04x",
+					pci_domain_nr(dev->bus),
+					pci_dev_id(dev));
+	ret = PTR_ERR_OR_ZERO(emul->poll_thread);
+	if (ret)
+		goto del_node;
+
+	ctrl_info(ctrl, "PCIe Hotplug Emulation active -- use at own risk!\n");
+
+	return 0;
+
+del_node:
+	mutex_lock(&list_lock);
+	dev->caps_rw_ops = NULL;
+	list_del(&emul->node);
+	mutex_unlock(&list_lock);
+free_emul:
+	kfree(emul);
+	return ret;
+}
+
+static void pciehp_emul_stop_child_threads(struct pci_dev *dev)
+{
+	struct pciehp_emul *emul;
+	struct pci_bus *bus = dev->subordinate;
+
+	if (!bus)
+		return;
+
+	list_for_each_entry(dev, &bus->devices, bus_list) {
+		pciehp_emul_stop_child_threads(dev);
+
+		emul = pci_dev_to_pciehp_emul(dev);
+		if (emul)
+			kthread_stop(emul->poll_thread);
+	}
+}
+
+void pciehp_emul_detach(struct controller *ctrl)
+{
+	struct pciehp_emul *emul;
+	struct pci_dev *dev = ctrl->pcie->port;
+
+	mutex_lock(&list_lock);
+	pciehp_emul_stop_child_threads(ctrl->pcie->port);
+
+	emul = pci_dev_to_pciehp_emul(ctrl->pcie->port);
+	if (emul) {
+		kthread_stop(emul->poll_thread);
+		dev->caps_rw_ops = NULL;
+		list_del(&emul->node);
+	}
+	mutex_unlock(&list_lock);
+
+	kfree(emul);
+}
diff --git a/drivers/pci/pcie/Kconfig b/drivers/pci/pcie/Kconfig
index 788ac8df3f9d..44901b7e30d5 100644
--- a/drivers/pci/pcie/Kconfig
+++ b/drivers/pci/pcie/Kconfig
@@ -21,6 +21,19 @@ config HOTPLUG_PCI_PCIE
 
 	  When in doubt, say N.
 
+config HOTPLUG_PCI_PCIE_EMUL
+	bool "PCI Express Hotplug Emulation driver"
+	depends on HOTPLUG_PCI_PCIE && PCIEAER
+	select PCI_BRIDGE_EMUL
+	help
+	  This enables a PCI Express Native Hotplug emulation driver for
+	  non-hotplug-capable PCI Express slots. Hotplug is determined using
+	  link events and this driver can be disabled, or manage either only
+	  non-hotplug-capable slots or both non-hotplug and hotplug slots.
+	  Specific slots may also be targeted.
+
+	  When in doubt, say N.
+
 config PCIEAER
 	bool "PCI Express Advanced Error Reporting support"
 	depends on PCIEPORTBUS
-- 
2.30.2

[PATCH v2 7/7] PCI: pciehp: Wire up pcie_port_emulate_slot and

[Jonathan Derrick]

Add pcie_port_slot_emulated hook and wire it up to pciehp_emul. Add
pciehp_emul_{attach,detach} calls in pciehp and move the management
check into the attach caller.

Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
---
 drivers/pci/hotplug/pciehp.h      | 18 ++++++++++++++++++
 drivers/pci/hotplug/pciehp_emul.c |  8 ++++++++
 drivers/pci/hotplug/pciehp_hpc.c  |  3 +++
 drivers/pci/pcie/portdrv_core.c   |  3 +++
 include/linux/pci.h               |  4 ++++
 5 files changed, 36 insertions(+)

diff --git a/drivers/pci/hotplug/pciehp.h b/drivers/pci/hotplug/pciehp.h
index e221f5d12ad5..db09705ecb46 100644
--- a/drivers/pci/hotplug/pciehp.h
+++ b/drivers/pci/hotplug/pciehp.h
@@ -196,6 +196,24 @@ int pciehp_get_raw_indicator_status(struct hotplug_slot *h_slot, u8 *status);
 
 int pciehp_slot_reset(struct pcie_device *dev);
 
+#ifdef CONFIG_HOTPLUG_PCI_PCIE_EMUL
+bool pciehp_emul_will_manage(struct pci_dev *dev);
+int pciehp_emul_attach(struct controller *ctrl);
+void pciehp_emul_detach(struct controller *ctrl);
+#else
+static inline bool pciehp_emul_will_manage(struct pci_dev *dev)
+{
+	return false;
+};
+
+static inline int pciehp_emul_attach(struct controller *ctrl)
+{
+	return 0;
+};
+
+static inline void pciehp_emul_detach(struct controller *ctrl) {};
+#endif
+
 static inline const char *slot_name(struct controller *ctrl)
 {
 	return hotplug_slot_name(&ctrl->hotplug_slot);
diff --git a/drivers/pci/hotplug/pciehp_emul.c b/drivers/pci/hotplug/pciehp_emul.c
index 716ec57feb79..81e39b0964b4 100644
--- a/drivers/pci/hotplug/pciehp_emul.c
+++ b/drivers/pci/hotplug/pciehp_emul.c
@@ -370,3 +370,11 @@ void pciehp_emul_detach(struct controller *ctrl)
 
 	kfree(emul);
 }
+
+static int __init pciehp_emul_init(void)
+{
+	pcie_port_slot_emulated = &pciehp_emul_will_manage;
+
+	return 0;
+}
+postcore_initcall(pciehp_emul_init);
diff --git a/drivers/pci/hotplug/pciehp_hpc.c b/drivers/pci/hotplug/pciehp_hpc.c
index f711448e0a70..fc5983e70997 100644
--- a/drivers/pci/hotplug/pciehp_hpc.c
+++ b/drivers/pci/hotplug/pciehp_hpc.c
@@ -1003,6 +1003,9 @@ struct controller *pcie_init(struct pcie_device *dev)
 
 	ctrl->pcie = dev;
 	ctrl->depth = pcie_hotplug_depth(dev->port);
+	if (pciehp_emul_will_manage(pdev) && pciehp_emul_attach(ctrl))
+		return NULL;
+
 	pcie_capability_read_dword(pdev, PCI_EXP_SLTCAP, &slot_cap);
 
 	if (pdev->hotplug_user_indicators)
diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c
index b3c1e7d4ff10..3df397c01f0e 100644
--- a/drivers/pci/pcie/portdrv_core.c
+++ b/drivers/pci/pcie/portdrv_core.c
@@ -19,6 +19,9 @@
 #include "../pci.h"
 #include "portdrv.h"
 
+/* Hook for hotplug emulation driver */
+bool (*pcie_port_slot_emulated)(struct pci_dev *dev);
+
 struct portdrv_service_data {
 	struct pcie_port_service_driver *drv;
 	struct device *dev;
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 0c907f94bb61..0011e7775735 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1656,7 +1656,11 @@ extern bool pcie_ports_native;
 #define pcie_ports_native	false
 #endif
 
+#ifdef CONFIG_HOTPLUG_PCI_PCIE_EMUL
+extern bool (*pcie_port_slot_emulated)(struct pci_dev *dev);
+#else
 #define pcie_port_slot_emulated(dev) false
+#endif
 
 #define PCIE_LINK_STATE_L0S		BIT(0)
 #define PCIE_LINK_STATE_L1		BIT(1)
-- 
2.30.2

Re: [PATCH v2 5/7] PCI: pci-bridge-emul: Provide a helper to set behavior

[Pali Rohár]

On Thursday 10 November 2022 12:50:13 Jonathan Derrick wrote:
> Add a handler to set behavior of a PCI or PCIe register. Add the
> appropriate enums to specify the register's Read-Only, Read-Write, and
> Write-1-to-Clear behaviors.
> 
> Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>

I do not think that this is the correct way. Drivers should not need to
tell bridge emulator that some register is read-only or read-write.
Bridge emulator has already all required information in its internal
structures.

If there is a need to tell bridge emulator to "emulate" some optional
feature, for example hot plug capabilities, then it would be better to
extend pci_bridge_emul_init() flags and let init function to correctly
fill all behavior bits. There is already PCI_BRIDGE_EMUL_NO_PREFMEM_FORWARD
flag which modify bridge prefetchable bits.

In my opinion, all standard PCI/PCIe behavior bits should be in
pci-bridge-emul.c source file and drivers should not modify them.
I think that this approach makes implementation lot of cleaner.

> ---
>  drivers/pci/pci-bridge-emul.c | 19 +++++++++++++++++++
>  drivers/pci/pci-bridge-emul.h | 10 ++++++++++
>  2 files changed, 29 insertions(+)
> 
> diff --git a/drivers/pci/pci-bridge-emul.c b/drivers/pci/pci-bridge-emul.c
> index 9334b2dd4764..3c1a683ece66 100644
> --- a/drivers/pci/pci-bridge-emul.c
> +++ b/drivers/pci/pci-bridge-emul.c
> @@ -46,6 +46,25 @@ struct pci_bridge_reg_behavior {
>  	u32 w1c;
>  };
>  
> +void pci_bridge_emul_set_reg_behavior(struct pci_bridge_emul *bridge,
> +				      bool pcie, int reg, u32 val,
> +				      enum pci_bridge_emul_reg_behavior type)
> +{
> +	struct pci_bridge_reg_behavior *behavior;
> +
> +	if (pcie)
> +		behavior = &bridge->pcie_cap_regs_behavior[reg / 4];
> +	else
> +		behavior = &bridge->pci_regs_behavior[reg / 4];
> +
> +	if (type == PCI_BRIDGE_EMUL_REG_BEHAVIOR_RO)
> +		behavior->ro = val;
> +	else if (type == PCI_BRIDGE_EMUL_REG_BEHAVIOR_RW)
> +		behavior->rw = val;
> +	else /* PCI_BRIDGE_EMUL_REG_BEHAVIOR_W1C */
> +		behavior->w1c = val;
> +}
> +
>  static const
>  struct pci_bridge_reg_behavior pci_regs_behavior[PCI_STD_HEADER_SIZEOF / 4] = {
>  	[PCI_VENDOR_ID / 4] = { .ro = ~0 },
> diff --git a/drivers/pci/pci-bridge-emul.h b/drivers/pci/pci-bridge-emul.h
> index 2a0e59c7f0d9..b2401d58518c 100644
> --- a/drivers/pci/pci-bridge-emul.h
> +++ b/drivers/pci/pci-bridge-emul.h
> @@ -72,6 +72,12 @@ struct pci_bridge_emul;
>  typedef enum { PCI_BRIDGE_EMUL_HANDLED,
>  	       PCI_BRIDGE_EMUL_NOT_HANDLED } pci_bridge_emul_read_status_t;
>  
> +enum pci_bridge_emul_reg_behavior {
> +	PCI_BRIDGE_EMUL_REG_BEHAVIOR_RO,
> +	PCI_BRIDGE_EMUL_REG_BEHAVIOR_RW,
> +	PCI_BRIDGE_EMUL_REG_BEHAVIOR_W1C,
> +};
> +
>  struct pci_bridge_emul_ops {
>  	/*
>  	 * Called when reading from the regular PCI bridge
> @@ -161,4 +167,8 @@ int pci_bridge_emul_conf_read(struct pci_bridge_emul *bridge, int where,
>  int pci_bridge_emul_conf_write(struct pci_bridge_emul *bridge, int where,
>  			       int size, u32 value);
>  
> +void pci_bridge_emul_set_reg_behavior(struct pci_bridge_emul *bridge,
> +				      bool pcie, int reg, u32 val,
> +				      enum pci_bridge_emul_reg_behavior type);
> +
>  #endif /* __PCI_BRIDGE_EMUL_H__ */
> -- 
> 2.30.2
> 

Re: [PATCH v2 1/7] PCI: Allow for indirecting capability registers

[Pali Rohár]

On Thursday 10 November 2022 12:50:09 Jonathan Derrick wrote:
> Allow another driver to provide alternative operations when doing
> capability register reads and writes. The intention is to have
> pcie_bridge_emul provide alternate handlers for the Slot Capabilities, Slot
> Control, and Slot Status registers. Alternate handlers can return > 0 if
> unhandled, errno on error, or 0 on success. This could potentially be
> used to handle quirks in a different manner.

I think that this change should not be needed. Controller drivers
pci-mvebu.c and pci-aardvark.c already provides those alternative
operations when doing capability read and write, and it is working
without need to touch pci/access.c file. They directly register
pci_bridge_emul_init() device and then callbacks of this emulated bridge
either touches config space HW registers or provide some emulation layer
(for capabilities which are not provided by HW config space).

This approach (which is already in use) has one big advantage: There is
no need to touch common pci/access.c code, it only modifies code for
specific HW - controller driver which needs that bridge emulator. All
other HW platforms are unaffected / untouched. Whole emulator code is
separated from the core pci access code.

This is just my opinion, maybe Bjorn has different idea. I just wanted
to show how it is implemented in existing drivers.

> Signed-off-by: Jonathan Derrick <jonathan.derrick@linux.dev>
> ---
>  drivers/pci/access.c | 29 +++++++++++++++++++++++++++++
>  include/linux/pci.h  | 11 +++++++++++
>  2 files changed, 40 insertions(+)
> 
> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
> index 708c7529647f..dbfea6824bd4 100644
> --- a/drivers/pci/access.c
> +++ b/drivers/pci/access.c
> @@ -424,6 +424,17 @@ int pcie_capability_read_word(struct pci_dev *dev, int pos, u16 *val)
>  		return ret;
>  	}
>  
> +	if (dev->caps_rw_ops) {
> +		u32 reg;
> +		ret = dev->caps_rw_ops->read(dev, pos, 4, &reg);
> +		if (!ret) {
> +			*val = reg & 0xffff;
> +			return ret;
> +		} else if (ret < 0) {
> +			return ret;
> +		}
> +	}
> +
>  	/*
>  	 * For Functions that do not implement the Slot Capabilities,
>  	 * Slot Status, and Slot Control registers, these spaces must
> @@ -459,6 +470,12 @@ int pcie_capability_read_dword(struct pci_dev *dev, int pos, u32 *val)
>  		return ret;
>  	}
>  
> +	if (dev->caps_rw_ops) {
> +		ret = dev->caps_rw_ops->read(dev, pos, 4, val);
> +		if (ret <= 0)
> +			return ret;
> +	}
> +
>  	if (pci_is_pcie(dev) && pcie_downstream_port(dev) &&
>  	    pos == PCI_EXP_SLTSTA)
>  		*val = PCI_EXP_SLTSTA_PDS;
> @@ -475,6 +492,12 @@ int pcie_capability_write_word(struct pci_dev *dev, int pos, u16 val)
>  	if (!pcie_capability_reg_implemented(dev, pos))
>  		return 0;
>  
> +	if (dev->caps_rw_ops) {
> +		int ret = dev->caps_rw_ops->write(dev, pos, 2, val);
> +		if (ret <= 0)
> +			return ret;
> +	}
> +
>  	return pci_write_config_word(dev, pci_pcie_cap(dev) + pos, val);
>  }
>  EXPORT_SYMBOL(pcie_capability_write_word);
> @@ -487,6 +510,12 @@ int pcie_capability_write_dword(struct pci_dev *dev, int pos, u32 val)
>  	if (!pcie_capability_reg_implemented(dev, pos))
>  		return 0;
>  
> +	if (dev->caps_rw_ops) {
> +		int ret = dev->caps_rw_ops->write(dev, pos, 4, val);
> +		if (ret <= 0)
> +			return ret;
> +	}
> +
>  	return pci_write_config_dword(dev, pci_pcie_cap(dev) + pos, val);
>  }
>  EXPORT_SYMBOL(pcie_capability_write_dword);
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 2bda4a4e47e8..ff47ef83ab38 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -311,6 +311,15 @@ struct pci_vpd {
>  	u8		cap;
>  };
>  
> +/*
> + * Capability reads/write redirect
> + * Returns 0, errno, or > 0 if unhandled
> + */
> +struct caps_rw_ops {
> +	int (*read)(struct pci_dev *dev, int pos, int len, u32 *val);
> +	int (*write)(struct pci_dev *dev, int pos, int len, u32 val);
> +};
> +
>  struct irq_affinity;
>  struct pcie_link_state;
>  struct pci_sriov;
> @@ -523,6 +532,8 @@ struct pci_dev {
>  
>  	/* These methods index pci_reset_fn_methods[] */
>  	u8 reset_methods[PCI_NUM_RESET_METHODS]; /* In priority order */
> +
> +	struct caps_rw_ops *caps_rw_ops;
>  };
>  
>  static inline struct pci_dev *pci_physfn(struct pci_dev *dev)
> -- 
> 2.30.2
> 

Re: [PATCH v2 0/7] PCIe Hotplug Slot Emulation driver

[Pali Rohár]

On Thursday 10 November 2022 12:50:08 Jonathan Derrick wrote:
> The main intent of posting v2 is to help Vidya further along with his
> GPIO-based hotplug driver. I have no direct need for a DLLSC-based emulated
> mode, and I'm not certain the model is appropriate. The GPIO-based model could
> modify some of the logic in patch 6 to check for GPIO instead of DLLSC.

This approach for GPIO-based model introduce also a new hotplug driver
HOTPLUG_PCI_PCIE_EMUL. But if you implement all callbacks of the
pci-bridge-emul.c for standard PCIe hotplug capability then you do not
need this new HOTPLUG_PCI_PCIE_EMUL at all. Existing HOTPLUG_PCI_PCIE
driver would work without any issue. I have already implemented DLLSC
basics for pci-mvebu.c and pci-aardvark.c controller drivers, patches
are on the mailing list. I can help implementing it in other PCIe
controller driver.