[PATCH V9 0/5] Minimal alignment for p2p bars

[PATCH V9 0/5] Minimal alignment for p2p bars

[Gavin Shan]

v1 -> v2:
	* Shorten the varaible names so that they looks more short.
        * Changelog adjustment so that they looks more meaningful.
v2 -> v3:
        * Rebase to 3.5.RC4
v3 -> v4:
        * Merge Yinghai's patches.
        * Split patch for easy review.
        * Add function to retrieve the minimal alignment of p2p bridge.
v4 -> v5:
        * Rebase to 3.5.RC7
        * Introduce weak function pcibios_window_alignment() to retrieve
          I/O and memory alignment for P2P bridges.
        * Introduce pcibios_window_alignment() for ppc to override the
          PCI function.
        * Add ppc_md.pcibios_window_alignment() for specific platform like
          powernv can override ppc's pcibios_window_alignment().
v5 -> v6:
        * Refactor pcibios_window_alignment() so the platform-specific
          implementation needn't return the default alignment according
          to Bjorn's suggestion.
        * Simplify pbus_size_mem() according to Bjorn's suggestion: Just
          check the platform required alignment at very end and adjust
          the "min_align" if necessary.
v6 -> v7:
        * Change "type" to "b_res->flags & mask" while retrieving the
          minimal alignment for memory window according to Ram's suggestion.
        * Refactor pbus_size_mem() according to Ram's suggestion.
        * ppc_md.pcibios_window_alignment returns 1 for those PCI bridges
          behind PCI bridges so that PCI core will use default alignment
          values.
v7 -> v8:
	* Rebase to 3.6.RC2, which starts to use "struct resource" to represent
	  the range of PCI bus numbers that specific p2p bridge covers.
	* Define macros for the default alignment of P2P bars according to
	  Richard's comments.
v8 -> v9:
	* Rebase to 3.6.RC4
	* Remove the original first 3 patches since they're irrevelant to the
	  intention.
	* Platform can override p2p I/O alignment even the p2p bridge explicitly
	  requires 1KiB I/O alignment according to Bjorn's comments.
	* In function pbus_size_io(), the maximal allowed I/O alignment was changed
	  from 4KiB to 1KiB for those p2p bridges that require 1KiB alignment, which
	  is traced by variable "io_align". I'm not sure that's correct for 100% since
	  the original implementation had 1KiB for "io_align" for the case.
	* Adjustment for pnv_pci_window_alignment() so that we can use the default
	  alignments (4KiB for I/O, 1MiB for memory) if the PCI bus isn't the top
	  level bus in the associated EEH segment.

Gavin Shan(5)
  pci: weak function returns alignment
  pci: resource assignment based on p2p alignment
  pci: refactor function pbus_size_mem
  ppc/pci: override pcibios_window_alignment
  ppc/pnv: I/O and memory alignment for p2p bridges

-----

arch/powerpc/include/asm/machdep.h        |    3 +
arch/powerpc/kernel/pci-common.c          |   20 +++++++
arch/powerpc/platforms/powernv/pci-ioda.c |   39 ++++++++++++++
drivers/pci/setup-bus.c                   |   81 +++++++++++++++++++++--------
include/linux/pci.h                       |    2 +
5 files changed, 124 insertions(+), 21 deletions(-)

Thanks,
Gavin

[PATCH 1/5] pci: weak function returns alignment

[Gavin Shan]

The patch implements the weak function to return the default I/O
or memory alignment for P2P bridge. Currently, I/O window has 4KiB
or 1KiB alignment and memory window is 4MiB aligned by default. On
the other hand, those platforms (e.g. powernv) that have special
requirements on the alignment could override the function by themselves.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 drivers/pci/setup-bus.c |   32 ++++++++++++++++++++++++++++++++
 include/linux/pci.h     |    2 ++
 2 files changed, 34 insertions(+), 0 deletions(-)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index fb50613..896f06e 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -697,6 +697,38 @@ static resource_size_t calculate_memsize(resource_size_t size,
 	return size;
 }
 
+resource_size_t __weak pcibios_window_alignment(struct pci_bus *bus,
+						unsigned long type)
+{
+	return 1;
+}
+
+#define PCI_P2P_DEFAULT_MEM_ALIGN	0x100000	/* 1MiB */
+#define PCI_P2P_DEFAULT_IO_ALIGN	0x1000		/* 4KiB */
+#define PCI_P2P_DEFAULT_IO_ALIGN_1K	0x400		/* 1KiB */
+
+static resource_size_t window_alignment(struct pci_bus *bus,
+					unsigned long type)
+{
+	resource_size_t align = 1, arch_align;
+
+	if (type & IORESOURCE_MEM)
+		align = PCI_P2P_DEFAULT_MEM_ALIGN;
+	else if (type & IORESOURCE_IO) {
+		/*
+		 * Per spec, I/O windows are 4K-aligned, but some
+		 * bridges have an extension to support 1K alignment.
+		 */
+		if (bus->self->io_window_1k)
+			align = PCI_P2P_DEFAULT_IO_ALIGN_1K;
+		else
+			align = PCI_P2P_DEFAULT_IO_ALIGN;
+	}
+
+	arch_align = pcibios_window_alignment(bus, type);
+	return max(align, arch_align);
+}
+
 /**
  * pbus_size_io() - size the io window of a given bus
  *
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 5faa831..e4e4794 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1031,6 +1031,8 @@ int pci_cfg_space_size_ext(struct pci_dev *dev);
 int pci_cfg_space_size(struct pci_dev *dev);
 unsigned char pci_bus_max_busnr(struct pci_bus *bus);
 void pci_setup_bridge(struct pci_bus *bus);
+resource_size_t pcibios_window_alignment(struct pci_bus *bus,
+					 unsigned long type);
 
 #define PCI_VGA_STATE_CHANGE_BRIDGE (1 << 0)
 #define PCI_VGA_STATE_CHANGE_DECODES (1 << 1)
-- 
1.7.5.4

[PATCH 2/5] pci: resource assignment based on p2p alignment

[Gavin Shan]

The patch changes function pbus_size_io() and pbus_size_mem() to
do resource (I/O, memory and prefetchable memory) reassignment
based on the minimal alignments for the p2p bridge, which was
retrieved by function window_alignment().

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 drivers/pci/setup-bus.c |   15 ++++++---------
 1 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 896f06e..a66cf09 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -749,17 +749,12 @@ static void pbus_size_io(struct pci_bus *bus, resource_size_t min_size,
 	struct resource *b_res = find_free_bus_resource(bus, IORESOURCE_IO);
 	unsigned long size = 0, size0 = 0, size1 = 0;
 	resource_size_t children_add_size = 0;
-	resource_size_t min_align = 4096, align;
+	resource_size_t min_align, io_align, align;
 
 	if (!b_res)
  		return;
 
-	/*
-	 * Per spec, I/O windows are 4K-aligned, but some bridges have an
-	 * extension to support 1K alignment.
-	 */
-	if (bus->self->io_window_1k)
-		min_align = 1024;
+	io_align = min_align = window_alignment(bus, IORESOURCE_IO);
 	list_for_each_entry(dev, &bus->devices, bus_list) {
 		int i;
 
@@ -786,8 +781,8 @@ static void pbus_size_io(struct pci_bus *bus, resource_size_t min_size,
 		}
 	}
 
-	if (min_align > 4096)
-		min_align = 4096;
+	if (min_align > io_align)
+		min_align = io_align;
 
 	size0 = calculate_iosize(size, min_size, size1,
 			resource_size(b_res), min_align);
@@ -909,6 +904,8 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 			min_align = align1 >> 1;
 		align += aligns[order];
 	}
+
+	min_align = max(min_align, window_alignment(bus, b_res->flags & mask));
 	size0 = calculate_memsize(size, min_size, 0, resource_size(b_res), min_align);
 	if (children_add_size > add_size)
 		add_size = children_add_size;
-- 
1.7.5.4

[PATCH 3/5] pci: refactor function pbus_size_mem

[Gavin Shan]

The original idea comes from Ram Pai. The patch puts the chunk of
code for calculating the minimal alignment of memory window into
separate inline function.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 drivers/pci/setup-bus.c |   36 +++++++++++++++++++++++-------------
 1 files changed, 23 insertions(+), 13 deletions(-)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index a66cf09..1e808ca 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -812,6 +812,28 @@ static void pbus_size_io(struct pci_bus *bus, resource_size_t min_size,
 	}
 }
 
+static inline resource_size_t calculate_mem_align(resource_size_t *aligns,
+						  int max_order)
+{
+	resource_size_t align = 0;
+	resource_size_t min_align = 0;
+	int order;
+
+	for (order = 0; order <= max_order; order++) {
+		resource_size_t align1 = 1;
+
+		align1 <<= (order + 20);
+
+		if (!align)
+			min_align = align1;
+		else if (ALIGN(align + min_align, min_align) < align1)
+			min_align = align1 >> 1;
+		align += aligns[order];
+	}
+
+	return min_align;
+}
+
 /**
  * pbus_size_mem() - size the memory window of a given bus
  *
@@ -891,20 +913,8 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 				children_add_size += get_res_add_size(realloc_head, r);
 		}
 	}
-	align = 0;
-	min_align = 0;
-	for (order = 0; order <= max_order; order++) {
-		resource_size_t align1 = 1;
-
-		align1 <<= (order + 20);
-
-		if (!align)
-			min_align = align1;
-		else if (ALIGN(align + min_align, min_align) < align1)
-			min_align = align1 >> 1;
-		align += aligns[order];
-	}
 
+	min_align = calculate_mem_align(aligns, max_order);
 	min_align = max(min_align, window_alignment(bus, b_res->flags & mask));
 	size0 = calculate_memsize(size, min_size, 0, resource_size(b_res), min_align);
 	if (children_add_size > add_size)
-- 
1.7.5.4

[PATCH 4/5] ppc/pci: override pcibios_window_alignment

[Gavin Shan]

Function pcibios_window_alignment() has been implemented as "weak"
in PCI core to return the default alignment of I/O and memory windows
for the associated p2p bridge. The patch adds same function to
override the weak one so that the default alignment could be changed
for those platforms (e.g. powernv).

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/machdep.h |    3 +++
 arch/powerpc/kernel/pci-common.c   |   20 ++++++++++++++++++++
 2 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/arch/powerpc/include/asm/machdep.h b/arch/powerpc/include/asm/machdep.h
index 42ce570..f7706d7 100644
--- a/arch/powerpc/include/asm/machdep.h
+++ b/arch/powerpc/include/asm/machdep.h
@@ -214,6 +214,9 @@ struct machdep_calls {
 	/* Called after scan and before resource survey */
 	void (*pcibios_fixup_phb)(struct pci_controller *hose);
 
+	/* Called during PCI resource reassignment */
+	resource_size_t (*pcibios_window_alignment)(struct pci_bus *, unsigned long type);
+
 	/* Called to shutdown machine specific hardware not already controlled
 	 * by other drivers.
 	 */
diff --git a/arch/powerpc/kernel/pci-common.c b/arch/powerpc/kernel/pci-common.c
index 2aa04f2..43fea54 100644
--- a/arch/powerpc/kernel/pci-common.c
+++ b/arch/powerpc/kernel/pci-common.c
@@ -99,6 +99,26 @@ void pcibios_free_controller(struct pci_controller *phb)
 		kfree(phb);
 }
 
+/*
+ * The function is used to return the minimal alignment
+ * for memory or I/O windows of the associated P2P bridge.
+ * By default, 4KiB alignment for I/O windows and 1MiB for
+ * memory windows.
+ */
+resource_size_t pcibios_window_alignment(struct pci_bus *bus,
+					 unsigned long type)
+{
+	if (ppc_md.pcibios_window_alignment)
+		return ppc_md.pcibios_window_alignment(bus, type);
+
+	/*
+	 * PCI core will figure out the default
+	 * alignment: 4KiB for I/O and 1MiB for
+	 * memory window.
+	 */
+	return 1;
+}
+
 static resource_size_t pcibios_io_size(const struct pci_controller *hose)
 {
 #ifdef CONFIG_PPC64
-- 
1.7.5.4

[PATCH 5/5] ppc/pnv: I/O and memory alignment for p2p bridges

[Gavin Shan]

The patch implements ppc_md.pcibios_window_alignment for powernv
platform so that the resource reassignment in PCI core will be
done according to the I/O and memory alignment returned from
powernv platform. The alignments returned from powernv platform
is closely depending on the scheme for PE segmenting. Besides,
the patch isn't useful for now, but the subsequent patches will
be working based on it.

Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/pci-ioda.c |   39 +++++++++++++++++++++++++++++
 1 files changed, 39 insertions(+), 0 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 9cda6a1..f5479fe 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1139,6 +1139,44 @@ static void __devinit pnv_pci_ioda_fixup_phb(struct pci_controller *hose)
 	}
 }
 
+/*
+ * Returns the alignment for I/O or memory windows for p2p
+ * bridges. That actually depends on how PEs are segmented.
+ * For now, we return I/O or M32 segment size for PE sensitive
+ * p2p bridges. Otherwise, the default values (4KiB for I/O,
+ * 1MiB for memory) will be returned.
+ *
+ * The current PCI bus might be put into one PE, which was
+ * create againt the parent PCI bridge. For that case, we
+ * needn't enlarge the alignment so that we can save some
+ * resources.
+ */
+static resource_size_t pnv_pci_window_alignment(struct pci_bus *bus,
+						unsigned long type)
+{
+	struct pci_dev *bridge;
+	struct pci_controller *hose = pci_bus_to_host(bus);
+	struct pnv_phb *phb = hose->private_data;
+	int num_pci_bridges = 0;
+
+	bridge = bus->self;
+	while (bridge) {
+		if (bridge->pcie_type == PCI_EXP_TYPE_PCI_BRIDGE) {
+			num_pci_bridges++;
+			if (num_pci_bridges >= 2)
+				return 1;
+		}
+
+		bridge = bridge->bus->self;
+	}
+
+	/* We need support prefetchable memory window later */
+	if (type & IORESOURCE_MEM)
+		return phb->ioda.m32_segsize;
+
+	return phb->ioda.io_segsize;
+}
+
 /* Prevent enabling devices for which we couldn't properly
  * assign a PE
  */
@@ -1306,6 +1344,7 @@ void __init pnv_pci_init_ioda1_phb(struct device_node *np)
 	 */
 	ppc_md.pcibios_fixup_phb = pnv_pci_ioda_fixup_phb;
 	ppc_md.pcibios_enable_device_hook = pnv_pci_enable_device_hook;
+	ppc_md.pcibios_window_alignment = pnv_pci_window_alignment;
 	pci_add_flags(PCI_PROBE_ONLY | PCI_REASSIGN_ALL_RSRC);
 
 	/* Reset IODA tables to a clean state */
-- 
1.7.5.4

Re: [PATCH V9 0/5] Minimal alignment for p2p bars

[Bjorn Helgaas]

On Thu, Sep 6, 2012 at 10:00 PM, Gavin Shan <shangw@linux.vnet.ibm.com> wrote:
> v1 -> v2:
>         * Shorten the varaible names so that they looks more short.
>         * Changelog adjustment so that they looks more meaningful.
> v2 -> v3:
>         * Rebase to 3.5.RC4
> v3 -> v4:
>         * Merge Yinghai's patches.
>         * Split patch for easy review.
>         * Add function to retrieve the minimal alignment of p2p bridge.
> v4 -> v5:
>         * Rebase to 3.5.RC7
>         * Introduce weak function pcibios_window_alignment() to retrieve
>           I/O and memory alignment for P2P bridges.
>         * Introduce pcibios_window_alignment() for ppc to override the
>           PCI function.
>         * Add ppc_md.pcibios_window_alignment() for specific platform like
>           powernv can override ppc's pcibios_window_alignment().
> v5 -> v6:
>         * Refactor pcibios_window_alignment() so the platform-specific
>           implementation needn't return the default alignment according
>           to Bjorn's suggestion.
>         * Simplify pbus_size_mem() according to Bjorn's suggestion: Just
>           check the platform required alignment at very end and adjust
>           the "min_align" if necessary.
> v6 -> v7:
>         * Change "type" to "b_res->flags & mask" while retrieving the
>           minimal alignment for memory window according to Ram's suggestion.
>         * Refactor pbus_size_mem() according to Ram's suggestion.
>         * ppc_md.pcibios_window_alignment returns 1 for those PCI bridges
>           behind PCI bridges so that PCI core will use default alignment
>           values.
> v7 -> v8:
>         * Rebase to 3.6.RC2, which starts to use "struct resource" to represent
>           the range of PCI bus numbers that specific p2p bridge covers.
>         * Define macros for the default alignment of P2P bars according to
>           Richard's comments.
> v8 -> v9:
>         * Rebase to 3.6.RC4
>         * Remove the original first 3 patches since they're irrevelant to the
>           intention.
>         * Platform can override p2p I/O alignment even the p2p bridge explicitly
>           requires 1KiB I/O alignment according to Bjorn's comments.
>         * In function pbus_size_io(), the maximal allowed I/O alignment was changed
>           from 4KiB to 1KiB for those p2p bridges that require 1KiB alignment, which
>           is traced by variable "io_align". I'm not sure that's correct for 100% since
>           the original implementation had 1KiB for "io_align" for the case.
>         * Adjustment for pnv_pci_window_alignment() so that we can use the default
>           alignments (4KiB for I/O, 1MiB for memory) if the PCI bus isn't the top
>           level bus in the associated EEH segment.
>
> Gavin Shan(5)
>   pci: weak function returns alignment
>   pci: resource assignment based on p2p alignment
>   pci: refactor function pbus_size_mem
>   ppc/pci: override pcibios_window_alignment
>   ppc/pnv: I/O and memory alignment for p2p bridges
>
> -----
>
> arch/powerpc/include/asm/machdep.h        |    3 +
> arch/powerpc/kernel/pci-common.c          |   20 +++++++
> arch/powerpc/platforms/powernv/pci-ioda.c |   39 ++++++++++++++
> drivers/pci/setup-bus.c                   |   81 +++++++++++++++++++++--------
> include/linux/pci.h                       |    2 +
> 5 files changed, 124 insertions(+), 21 deletions(-)

I applied these to:

http://git.kernel.org/?p=linux/kernel/git/helgaas/pci.git;a=shortlog;h=refs/heads/pci/gavin-window-alignment

Assuming they pass Fengguang's build/smoke test, I'll merge that to my
"next" branch soon.  Thanks!

Bjorn

Re: [PATCH V9 0/5] Minimal alignment for p2p bars

[Benjamin Herrenschmidt]

On Fri, 2012-09-07 at 15:49 -0700, Bjorn Helgaas wrote:
> 
> http://git.kernel.org/?p=linux/kernel/git/helgaas/pci.git;a=shortlog;h=refs/heads/pci/gavin-window-alignment
> 
> Assuming they pass Fengguang's build/smoke test, I'll merge that to my
> "next" branch soon.  Thanks!

Thanks !

Any chance you can poke me when you do so ? At that point I'll include
them as well along with the rest of the stuff that depends on them.

Cheers,
Ben.

Re: [PATCH V9 0/5] Minimal alignment for p2p bars

[Bjorn Helgaas]

On Fri, Sep 7, 2012 at 5:51 PM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> On Fri, 2012-09-07 at 15:49 -0700, Bjorn Helgaas wrote:
>>
>> http://git.kernel.org/?p=linux/kernel/git/helgaas/pci.git;a=shortlog;h=refs/heads/pci/gavin-window-alignment
>>
>> Assuming they pass Fengguang's build/smoke test, I'll merge that to my
>> "next" branch soon.  Thanks!
>
> Thanks !
>
> Any chance you can poke me when you do so ? At that point I'll include
> them as well along with the rest of the stuff that depends on them.

I merged these to my "next" branch and pushed it.  Are you sure you
want to pull that as opposed to just using the
pci/gavin-window-alignment branch?  Both "next" and
"pci/gavin-window-alignment" have identical commits, but
"pci/gavin-window-alignment" is less likely to be rebased.

Bjorn

Re: [PATCH V9 0/5] Minimal alignment for p2p bars

[Benjamin Herrenschmidt]

On Mon, 2012-09-10 at 16:42 -0600, Bjorn Helgaas wrote:
> > Any chance you can poke me when you do so ? At that point I'll
> include
> > them as well along with the rest of the stuff that depends on them.
> 
> I merged these to my "next" branch and pushed it.  Are you sure you
> want to pull that as opposed to just using the
> pci/gavin-window-alignment branch?  Both "next" and
> "pci/gavin-window-alignment" have identical commits, but
> "pci/gavin-window-alignment" is less likely to be rebased.

No I will pull gavin's branch, but I didn't want to do so before it was
in your -next as well, just in case a problem creeped up needing a
rebase (I want to avoid rebasing powerpc-next).

Thanks !

Cheers,
Ben.