[PATCH v1 3/3] PCI: Consider alignment of hot-added bridges when distributing available resources

[PATCH v1 3/3] PCI: Consider alignment of hot-added bridges when distributing available resources

[Nicholas Johnson]

Change pci_bus_distribute_available_resources() to better handle bridges
with different resource alignment requirements.

The arguments io, mmio and mmio_pref represent the start and end
addresses of resource, in which we must fit the current bridge window.

The steps taken by pci_bus_distribute_available_resources():

	- For io, mmio and mmio_pref, increase .start to align with the
	  alignment of the current bridge window (otherwise the current
	  bridge window may not fit within the available range).

	- For io, mmio and mmio_pref, adjust the current bridge window
	  to the size after the above.

	- Count the number of hotplug bridges and normal bridges on this
	  bus.

	- If the total number of bridges is one, give that bridge all of
	  the resources and return.

	- If there are no hotplug bridges, return.

	- For io, mmio and mmio_pref, increase .start by the amount
	  required for each bridge resource on the bus for non hotplug
	  bridges, giving extra room to make up for alignment of those
	  resources.

	- For io, mmio and mmio_pref, calculate the resource size per
	  hotplug bridge which is available after the previous steps.

	- For io, mmio and mmio_pref, distribute the resources to each
	  hotplug bridge, with the sizes calculated above.

The motivation for fixing this is Thunderbolt with native PCI
enumeration, enabling external graphics cards and other devices with
bridge alignment higher than 1MB. This fixes the use case where the user
hot-adds Thunderbolt devices containing PCI devices with BAR
alignment >1M and having the resources fail to assign.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=199581
Reported-by: Mika Westerberg <mika.westerberg@linux.intel.com>

Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
---
 drivers/pci/setup-bus.c | 77 ++++++++++++++++++++++++-----------------
 1 file changed, 46 insertions(+), 31 deletions(-)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 269082261..8b39b9ebb 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1863,9 +1863,7 @@ static void pci_bus_distribute_available_resources(struct pci_bus *bus,
 					    struct resource mmio,
 					    struct resource mmio_pref)
 {
-	resource_size_t remaining_io, remaining_mmio, remaining_mmio_pref;
-	resource_size_t io_per_hp, mmio_per_hp, mmio_pref_per_hp;
-	resource_size_t avail_io, avail_mmio, avail_mmio_pref;
+	resource_size_t io_per_hp, mmio_per_hp, mmio_pref_per_hp, align;
 	unsigned int normal_bridges = 0, hotplug_bridges = 0;
 	struct resource *io_res, *mmio_res, *mmio_pref_res;
 	struct pci_dev *dev, *bridge = bus->self;
@@ -1874,6 +1872,23 @@ static void pci_bus_distribute_available_resources(struct pci_bus *bus,
 	mmio_res = &bridge->resource[PCI_BRIDGE_RESOURCES + 1];
 	mmio_pref_res = &bridge->resource[PCI_BRIDGE_RESOURCES + 2];
 
+	/*
+	 * The alignment of this bridge is yet to be considered, hence it must
+	 * be done now before extending its bridge window.
+	 */
+	align = pci_resource_alignment(bridge, io_res);
+	if (!io_res->parent && align)
+		io.start = min(ALIGN(io.start, align), io.end + 1);
+
+	align = pci_resource_alignment(bridge, mmio_res);
+	if (!mmio_res->parent && align)
+		mmio.start = min(ALIGN(mmio.start, align), mmio.end + 1);
+
+	align = pci_resource_alignment(bridge, mmio_pref_res);
+	if (!mmio_pref_res->parent && align)
+		mmio_pref.start = min(ALIGN(mmio_pref.start, align),
+			mmio_pref.end + 1);
+
 	/*
 	 * Update additional resource list (add_list) to fill all the
 	 * extra resource space available for this port except the space
@@ -1919,12 +1934,9 @@ static void pci_bus_distribute_available_resources(struct pci_bus *bus,
 	 * extra space reduced by the minimal required space for the
 	 * non-hotplug bridges.
 	 */
-	remaining_io = avail_io = resource_size(&io);
-	remaining_mmio = avail_mmio = resource_size(&mmio);
-	remaining_mmio_pref = avail_mmio_pref = resource_size(&mmio_pref);
-
 	for_each_pci_bridge(dev, bus) {
-		const struct resource *res;
+		resource_size_t used_size;
+		struct resource *res;
 
 		if (dev->is_hotplug_bridge)
 			continue;
@@ -1934,24 +1946,39 @@ static void pci_bus_distribute_available_resources(struct pci_bus *bus,
 		 * bridge and devices below it occupy.
 		 */
 		res = &dev->resource[PCI_BRIDGE_RESOURCES + 0];
-		if (!res->parent && avail_io > resource_size(res))
-			remaining_io -= resource_size(res);
+		align = pci_resource_alignment(dev, res);
+		align = align ? ALIGN(io.start, align) - io.start : 0;
+		used_size = align + resource_size(res);
+		if (!res->parent)
+			io.start = min(io.start + used_size, io.end + 1);
 
 		res = &dev->resource[PCI_BRIDGE_RESOURCES + 1];
-		if (!res->parent && avail_mmio > resource_size(res))
-			remaining_mmio -= resource_size(res);
+		align = pci_resource_alignment(dev, res);
+		align = align ? ALIGN(mmio.start, align) - mmio.start : 0;
+		used_size = align + resource_size(res);
+		if (!res->parent)
+			mmio.start = min(mmio.start + used_size, mmio.end + 1);
 
 		res = &dev->resource[PCI_BRIDGE_RESOURCES + 2];
-		if (!res->parent && avail_mmio_pref > resource_size(res))
-			remaining_mmio_pref -= resource_size(res);
+		align = pci_resource_alignment(dev, res);
+		align = align ? ALIGN(mmio_pref.start, align) -
+			mmio_pref.start : 0;
+		used_size = align + resource_size(res);
+		if (!res->parent)
+			mmio_pref.start = min(mmio_pref.start + used_size,
+				mmio_pref.end + 1);
 	}
 
+	io_per_hp = div64_ul(resource_size(&io), hotplug_bridges);
+	mmio_per_hp = div64_ul(resource_size(&mmio), hotplug_bridges);
+	mmio_pref_per_hp = div64_ul(resource_size(&mmio_pref),
+		hotplug_bridges);
+
 	/*
 	 * Go over devices on this bus and distribute the remaining
 	 * resource space between hotplug bridges.
 	 */
 	for_each_pci_bridge(dev, bus) {
-		resource_size_t align;
 		struct pci_bus *b;
 
 		b = dev->subordinate;
@@ -1963,28 +1990,16 @@ static void pci_bus_distribute_available_resources(struct pci_bus *bus,
 		 * hotplug-capable downstream ports taking alignment into
 		 * account.
 		 */
-		align = pci_resource_alignment(bridge, io_res);
-		io_per_hp = div64_ul(avail_io, hotplug_bridges);
-		io_per_hp = min(ALIGN(io_per_hp, align), remaining_io);
-		remaining_io -= io_per_hp;
-
-		align = pci_resource_alignment(bridge, mmio_res);
-		mmio_per_hp = div64_ul(avail_mmio, hotplug_bridges);
-		mmio_per_hp = min(ALIGN(mmio_per_hp, align), remaining_mmio);
-		remaining_mmio -= mmio_per_hp;
-
-		align = pci_resource_alignment(bridge, mmio_pref_res);
-		mmio_pref_per_hp = div64_ul(avail_mmio_pref, hotplug_bridges);
-		mmio_pref_per_hp = min(ALIGN(mmio_pref_per_hp, align),
-			remaining_mmio_pref);
-		remaining_mmio_pref -= mmio_pref_per_hp;
-
 		io.end = io.start + io_per_hp - 1;
 		mmio.end = mmio.start + mmio_per_hp - 1;
 		mmio_pref.end = mmio_pref.start + mmio_pref_per_hp - 1;
 
 		pci_bus_distribute_available_resources(b, add_list, io, mmio,
 						       mmio_pref);
+
+		io.start += io_per_hp;
+		mmio.start += mmio_per_hp;
+		mmio_pref.start += mmio_pref_per_hp;
 	}
 }
 
-- 
2.24.1

Re: [PATCH v1 3/3] PCI: Consider alignment of hot-added bridges when distributing available resources

[Mika Westerberg]

On Mon, Jan 06, 2020 at 03:46:13PM +0000, Nicholas Johnson wrote:
> Change pci_bus_distribute_available_resources() to better handle bridges
> with different resource alignment requirements.
> 
> The arguments io, mmio and mmio_pref represent the start and end
> addresses of resource, in which we must fit the current bridge window.
> 
> The steps taken by pci_bus_distribute_available_resources():
> 
> 	- For io, mmio and mmio_pref, increase .start to align with the
> 	  alignment of the current bridge window (otherwise the current
> 	  bridge window may not fit within the available range).
> 
> 	- For io, mmio and mmio_pref, adjust the current bridge window
> 	  to the size after the above.
> 
> 	- Count the number of hotplug bridges and normal bridges on this
> 	  bus.
> 
> 	- If the total number of bridges is one, give that bridge all of
> 	  the resources and return.
> 
> 	- If there are no hotplug bridges, return.
> 
> 	- For io, mmio and mmio_pref, increase .start by the amount
> 	  required for each bridge resource on the bus for non hotplug
> 	  bridges, giving extra room to make up for alignment of those
> 	  resources.
> 
> 	- For io, mmio and mmio_pref, calculate the resource size per
> 	  hotplug bridge which is available after the previous steps.
> 
> 	- For io, mmio and mmio_pref, distribute the resources to each
> 	  hotplug bridge, with the sizes calculated above.
> 
> The motivation for fixing this is Thunderbolt with native PCI
> enumeration, enabling external graphics cards and other devices with
> bridge alignment higher than 1MB. This fixes the use case where the user
> hot-adds Thunderbolt devices containing PCI devices with BAR
> alignment >1M and having the resources fail to assign.
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=199581
> Reported-by: Mika Westerberg <mika.westerberg@linux.intel.com>

Still solves the issue I reported above so,

Tested-by: Mika Westerberg <mika.westerberg@linux.intel.com>

Also looks good to me,

Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>