[PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

[PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

[David Daney]

From: David Daney <david.daney@cavium.com>

The original patches are from Sean O. Stalley. I made a few tweaks,
but feel that it is substancially Sean's work, so I am keeping the
patch set version numbering scheme going.

Tested on Cavium ThunderX system with 4 Root Complexes containing 50
devices/bridges provisioned with EA.

Here is Sean's description of the patches:

PCI Enhanced Allocation is a new method of allocating MMIO & IO
resources for PCI devices & bridges. It can be used instead
of the traditional PCI method of using BARs.

EA entries are hardware-initialized to a fixed address.
Unlike BARs, regions described by EA are cannot be moved.
Because of this, only devices which are permanently connected to
the PCI bus can use EA. A removable PCI card must not use EA.

This patchset adds support for using EA entries instead of BARs
on Root Complex Integrated Endpoints.

The Enhanced Allocation ECN is publicly available here:
https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf


Changes from V1:
	- Use generic PCI resource claim functions (instead of EA-specific functions)
	- Only add support for RCiEPs (instead of all devices).
	- Removed some debugging messages leftover from early testing.

Changes from V2 (By David Daney):
	- Add ea_cap to struct pci_device, to aid in finding the EA capability.
	- Factored EA entity decoding into a separate function.
	- Add functions to find EA entities by BEI or Property.
	- Add handling of EA provisioned bridges.
	- Add handling of EA SRIOV BARs.
	- Try to assign proper resource parent so that SRIOV device creation can occur.

Changes from V3 (By David Daney):
	- Discarded V3 changes and started over fresh based on Sean's V2.
	- Add more support/checking for Entry Properties.
	- Allow EA behind bridges.
	- Rewrite some error messages.
	- Add patch 3/5 to prevent resizing, and better handle
          assigning, of fixed EA resources.
	- Add patch 4/5 to handle EA provisioned SRIOV devices.
	- Add patch 5/5 to handle EA provisioned bridges.

David Daney (3):
  PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.
  PCI: Handle Enhanced Allocation (EA) capability for SRIOV devices.
  PCI: Handle Enhanced Allocation (EA) capability for bridges

Sean O. Stalley (2):
  PCI: Add Enhanced Allocation register entries
  PCI: Add support for Enhanced Allocation devices

 drivers/pci/bus.c             |   7 ++
 drivers/pci/iov.c             |  11 ++-
 drivers/pci/pci.c             | 202 ++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci.h             |   1 +
 drivers/pci/probe.c           |  34 ++++++-
 drivers/pci/setup-bus.c       |  63 ++++++++++++-
 include/linux/pci.h           |   1 +
 include/uapi/linux/pci_regs.h |  44 ++++++++-
 8 files changed, 355 insertions(+), 8 deletions(-)

-- 
1.9.1

[PATCH v4 1/5] PCI: Add Enhanced Allocation register entries

[David Daney]

From: "Sean O. Stalley" <sean.stalley@intel.com>

Add registers defined in PCI-SIG's Enhanced allocation ECN.

Signed-off-by: Sean O. Stalley <sean.stalley@intel.com>
[david.daney@cavium.com: Added more definitions for PCI_EA_BEI_*]
Signed-off-by: Signed-off-by: David Daney <david.daney@cavium.com>
---
 include/uapi/linux/pci_regs.h | 44 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
index 413417f..352e418 100644
--- a/include/uapi/linux/pci_regs.h
+++ b/include/uapi/linux/pci_regs.h
@@ -216,7 +216,8 @@
 #define  PCI_CAP_ID_MSIX	0x11	/* MSI-X */
 #define  PCI_CAP_ID_SATA	0x12	/* SATA Data/Index Conf. */
 #define  PCI_CAP_ID_AF		0x13	/* PCI Advanced Features */
-#define  PCI_CAP_ID_MAX		PCI_CAP_ID_AF
+#define  PCI_CAP_ID_EA		0x14	/* PCI Enhanced Allocation */
+#define  PCI_CAP_ID_MAX		PCI_CAP_ID_EA
 #define PCI_CAP_LIST_NEXT	1	/* Next capability in the list */
 #define PCI_CAP_FLAGS		2	/* Capability defined flags (16 bits) */
 #define PCI_CAP_SIZEOF		4
@@ -353,6 +354,47 @@
 #define  PCI_AF_STATUS_TP	0x01
 #define PCI_CAP_AF_SIZEOF	6	/* size of AF registers */
 
+/* PCI Enhanced Allocation registers */
+
+#define PCI_EA_NUM_ENT		2	/* Number of Capability Entries */
+#define PCI_EA_NUM_ENT_MASK	0x3f	/* Num Entries Mask */
+#define PCI_EA_FIRST_ENT	4	/* First EA Entry in List */
+#define PCI_EA_FIRST_ENT_BRIDGE	8	/* First EA Entry for Bridges */
+#define PCI_EA_ES		0x7	/* Entry Size */
+#define PCI_EA_BEI(x)	(((x) >> 4) & 0xf) /* BAR Equivalent Indicator */
+/* 0-5 map to BARs 0-5 respectively */
+#define  PCI_EA_BEI_BAR0	0
+#define  PCI_EA_BEI_BAR5	5
+#define  PCI_EA_BEI_BRIDGE	6	/* Resource behind bridge */
+#define  PCI_EA_BEI_ENI		7	/* Equivalent Not Indicated */
+#define  PCI_EA_BEI_ROM		8	/* Expansion ROM */
+/* 9-14 map to VF BARs 0-5 respectively */
+#define  PCI_EA_BEI_VF_BAR0	9
+#define  PCI_EA_BEI_VF_BAR5	14
+#define  PCI_EA_BEI_RESERVED	15	/* Reserved - Treat like ENI */
+
+#define PCI_EA_PP(x)	(((x) >>  8) & 0xff)	/* Primary Properties */
+#define PCI_EA_SP(x)	(((x) >> 16) & 0xff)	/* Secondary Properties */
+#define  PCI_EA_P_MEM			0x00	/* Non-Prefetch Memory */
+#define  PCI_EA_P_MEM_PREFETCH		0x01	/* Prefetchable Memory */
+#define  PCI_EA_P_IO			0x02	/* I/O Space */
+#define  PCI_EA_P_VIRT_MEM_PREFETCH	0x03	/* VF Prefetchable Memory */
+#define  PCI_EA_P_VIRT_MEM		0x04	/* VF Non-Prefetch Memory */
+#define  PCI_EA_P_BRIDGE_MEM		0x05	/* Bridge Non-Prefetch Memory */
+#define  PCI_EA_P_BRIDGE_MEM_PREFETCH	0x06	/* Bridge Prefetchable Memory */
+#define  PCI_EA_P_BRIDGE_IO		0x07	/* Bridge I/O Space */
+/* 0x08-0xfc reserved */
+#define  PCI_EA_P_MEM_RESERVED		0xfd	/* Reserved Memory */
+#define  PCI_EA_P_IO_RESERVED		0xfe	/* Reserved I/O Space */
+#define  PCI_EA_P_UNAVAILABLE		0xff	/* Entry Unavailable */
+#define PCI_EA_WRITEABLE	BIT(30) /* Writable, 1 = RW, 0 = HwInit */
+#define PCI_EA_ENABLE		BIT(31) /* Enable for this entry */
+#define PCI_EA_BASE		4	/* Base Address Offset */
+#define PCI_EA_MAX_OFFSET	8	/* MaxOffset (resource length) */
+/* bit 0 is reserved */
+#define PCI_EA_IS_64		BIT(1)	/* 64-bit field flag */
+#define PCI_EA_FIELD_MASK	0xfffffffc	/* For Base & Max Offset */
+
 /* PCI-X registers (Type 0 (non-bridge) devices) */
 
 #define PCI_X_CMD		2	/* Modes & Features */
-- 
1.9.1

[PATCH v4 2/5] PCI: Add support for Enhanced Allocation devices

[David Daney]

From: "Sean O. Stalley" <sean.stalley@intel.com>

Add support for devices using Enhanced Allocation entries instead of BARs.
This patch allows the kernel to parse the EA Extended Capability structure
in PCI configspace and claim the BAR-equivalent resources.

Signed-off-by: Sean O. Stalley <sean.stalley@intel.com>
[david.daney@cavium.com: Add more support/checking for Entry Properties,
allow EA behind bridges, rewrite some error messages.]
Signed-off-by: David Daney <david.daney@cavium.com>
---
 drivers/pci/pci.c   | 184 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci.h   |   1 +
 drivers/pci/probe.c |   3 +
 3 files changed, 188 insertions(+)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 6a9a111..54a0a98 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2148,6 +2148,190 @@ void pci_pm_init(struct pci_dev *dev)
 	}
 }
 
+static unsigned long pci_ea_set_flags(struct pci_dev *dev, u8 prop)
+{
+	unsigned long flags = IORESOURCE_PCI_FIXED;
+
+	switch (prop) {
+	case PCI_EA_P_MEM:
+	case PCI_EA_P_VIRT_MEM:
+	case PCI_EA_P_BRIDGE_MEM:
+		flags |= IORESOURCE_MEM;
+		break;
+	case PCI_EA_P_MEM_PREFETCH:
+	case PCI_EA_P_VIRT_MEM_PREFETCH:
+	case PCI_EA_P_BRIDGE_MEM_PREFETCH:
+		flags |= IORESOURCE_MEM | IORESOURCE_PREFETCH;
+		break;
+	case PCI_EA_P_IO:
+	case PCI_EA_P_BRIDGE_IO:
+		flags |= IORESOURCE_IO;
+		break;
+	default:
+		return 0;
+	}
+
+	return flags;
+}
+
+static struct resource *pci_ea_get_resource(struct pci_dev *dev, u8 bei, u8 prop)
+{
+	if (bei <= PCI_EA_BEI_BAR5 && prop <= PCI_EA_P_IO)
+		return &dev->resource[bei];
+	else if (bei == PCI_EA_BEI_ROM)
+		return &dev->resource[PCI_ROM_RESOURCE];
+	else
+		return NULL;
+}
+
+/* Read an Enhanced Allocation (EA) entry */
+static int pci_ea_read(struct pci_dev *dev, int offset)
+{
+	struct resource *res;
+	int ent_offset = offset;
+	int ent_size;
+	resource_size_t start;
+	resource_size_t end;
+	unsigned long flags;
+	u32 dw0;
+	u32 base;
+	u32 max_offset;
+	u8 prop;
+	bool support_64 = (sizeof(resource_size_t) >= 8);
+
+	pci_read_config_dword(dev, ent_offset, &dw0);
+	ent_offset += 4;
+
+	/* Entry size field indicates DWORDs after 1st */
+	ent_size = ((dw0 & PCI_EA_ES) + 1) << 2;
+
+	if (!(dw0 & PCI_EA_ENABLE)) /* Entry not enabled */
+		goto out;
+
+	prop = PCI_EA_PP(dw0);
+	/*
+	 * If the Property is in the reserved range, try the Secondary
+	 * Property instead.
+	 */
+	if (prop > PCI_EA_P_BRIDGE_IO && prop < PCI_EA_P_MEM_RESERVED)
+		prop = PCI_EA_SP(dw0);
+	if (prop > PCI_EA_P_BRIDGE_IO)
+		goto out;
+
+	res = pci_ea_get_resource(dev, PCI_EA_BEI(dw0), prop);
+	if (!res) {
+		dev_err(&dev->dev, "Unsupported EA entry BEI: %u\n",
+			PCI_EA_BEI(dw0));
+		goto out;
+	}
+
+	flags = pci_ea_set_flags(dev, prop);
+	if (!flags) {
+		dev_err(&dev->dev, "Unsupported EA properties: %u\n", prop);
+		goto out;
+	}
+
+	/* Read Base */
+	pci_read_config_dword(dev, ent_offset, &base);
+	start = (base & PCI_EA_FIELD_MASK);
+	ent_offset += 4;
+
+	/* Read MaxOffset */
+	pci_read_config_dword(dev, ent_offset, &max_offset);
+	ent_offset += 4;
+
+	/* Read Base MSBs (if 64-bit entry) */
+	if (base & PCI_EA_IS_64) {
+		u32 base_upper;
+
+		pci_read_config_dword(dev, ent_offset, &base_upper);
+		ent_offset += 4;
+
+		flags |= IORESOURCE_MEM_64;
+
+		/* entry starts above 32-bit boundary, can't use */
+		if (!support_64 && base_upper)
+			goto out;
+
+		if (support_64)
+			start |= ((u64)base_upper << 32);
+	}
+
+	dev_dbg(&dev->dev,
+		"EA (%u,%u) start = %pa\n", PCI_EA_BEI(dw0), prop, &start);
+
+	end = start + (max_offset | 0x03);
+
+	/* Read MaxOffset MSBs (if 64-bit entry) */
+	if (max_offset & PCI_EA_IS_64) {
+		u32 max_offset_upper;
+
+		pci_read_config_dword(dev, ent_offset, &max_offset_upper);
+		ent_offset += 4;
+
+		flags |= IORESOURCE_MEM_64;
+
+		/* entry too big, can't use */
+		if (!support_64 && max_offset_upper)
+			goto out;
+
+		if (support_64)
+			end += ((u64)max_offset_upper << 32);
+	}
+
+	dev_dbg(&dev->dev,
+		"EA (%u,%u) end = %pa\n", PCI_EA_BEI(dw0), prop, &end);
+
+	if (end < start) {
+		dev_err(&dev->dev, "EA Entry crosses address boundary\n");
+		goto out;
+	}
+
+	if (ent_size != ent_offset - offset) {
+		dev_err(&dev->dev,
+			"EA Entry Size (%d) does not match length read (%d)\n",
+			ent_size, ent_offset - offset);
+		goto out;
+	}
+
+	res->name = pci_name(dev);
+	res->start = start;
+	res->end = end;
+	res->flags = flags;
+
+out:
+	return offset + ent_size;
+}
+
+/* Enhanced Allocation Initalization */
+void pci_ea_init(struct pci_dev *dev)
+{
+	int ea;
+	u8 num_ent;
+	int offset;
+	int i;
+
+	/* find PCI EA capability in list */
+	ea = pci_find_capability(dev, PCI_CAP_ID_EA);
+	if (!ea)
+		return;
+
+	/* determine the number of entries */
+	pci_bus_read_config_byte(dev->bus, dev->devfn, ea + PCI_EA_NUM_ENT,
+					&num_ent);
+	num_ent &= PCI_EA_NUM_ENT_MASK;
+
+	offset = ea + PCI_EA_FIRST_ENT;
+
+	/* Skip DWORD 2 for type 1 functions */
+	if (dev->hdr_type == PCI_HEADER_TYPE_BRIDGE)
+		offset += 4;
+
+	/* parse each EA entry */
+	for (i = 0; i < num_ent; ++i)
+		offset = pci_ea_read(dev, offset);
+}
+
 static void pci_add_saved_cap(struct pci_dev *pci_dev,
 	struct pci_cap_saved_state *new_cap)
 {
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 24ba9dc..a160733 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -78,6 +78,7 @@ bool pci_dev_keep_suspended(struct pci_dev *dev);
 void pci_config_pm_runtime_get(struct pci_dev *dev);
 void pci_config_pm_runtime_put(struct pci_dev *dev);
 void pci_pm_init(struct pci_dev *dev);
+void pci_ea_init(struct pci_dev *dev);
 void pci_allocate_cap_save_buffers(struct pci_dev *dev);
 void pci_free_cap_save_buffers(struct pci_dev *dev);
 
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 0b2be17..4293eec 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1598,6 +1598,9 @@ static struct pci_dev *pci_scan_device(struct pci_bus *bus, int devfn)
 
 static void pci_init_capabilities(struct pci_dev *dev)
 {
+	/* Enhanced Allocation */
+	pci_ea_init(dev);
+
 	/* MSI/MSI-X list */
 	pci_msi_init_pci_dev(dev);
 
-- 
1.9.1

[PATCH v4 3/5] PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.

[David Daney]

From: David Daney <david.daney@cavium.com>

The new Enhanced Allocation (EA) capability support creates resources
with the IORESOURCE_PCI_FIXED set.  This creates a couple of problems:

1) Since these resources cannot be relocated or resized, their
   alignment is not really defined, and it is therefore not specified.
   This causes a problem in pbus_size_mem() where resources with
   unspecified alignment are disabled.

2) During resource assignment in pci_bus_assign_resources(),
   IORESOURCE_PCI_FIXED resources are not given a parent.  This, in
   turn, causes pci_enable_resources() to fail with a "not claimed"
   error.

So, in pbus_size_mem() skip IORESOURCE_PCI_FIXED resources, instead of
disabling them.

In __pci_bus_assign_resources(), for IORESOURCE_PCI_FIXED resources,
try to request the resource from a parent bus.

Signed-off-by: David Daney <david.daney@cavium.com>
---
 drivers/pci/setup-bus.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 57 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 508cc56..8f9ed9b 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1037,9 +1037,10 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 			struct resource *r = &dev->resource[i];
 			resource_size_t r_size;
 
-			if (r->parent || ((r->flags & mask) != type &&
-					  (r->flags & mask) != type2 &&
-					  (r->flags & mask) != type3))
+			if (r->parent || (r->flags | IORESOURCE_PCI_FIXED) ||
+			    ((r->flags & mask) != type &&
+			     (r->flags & mask) != type2 &&
+			     (r->flags & mask) != type3))
 				continue;
 			r_size = resource_size(r);
 #ifdef CONFIG_PCI_IOV
@@ -1340,6 +1341,57 @@ void pci_bus_size_bridges(struct pci_bus *bus)
 }
 EXPORT_SYMBOL(pci_bus_size_bridges);
 
+static void assign_fixed_resource_on_bus(struct pci_bus *b, struct resource *r)
+{
+	int i;
+	struct resource *parent_r;
+	unsigned long mask = IORESOURCE_IO | IORESOURCE_MEM |
+		IORESOURCE_PREFETCH;
+
+	pci_bus_for_each_resource(b, parent_r, i) {
+		if (!parent_r)
+			continue;
+
+		if ((r->flags & mask) == (parent_r->flags & mask) &&
+		    resource_contains(parent_r, r))
+			request_resource(parent_r, r);
+	}
+}
+
+/*
+ * Try to assign any resources marked as IORESOURCE_PCI_FIXED, as they
+ * are skipped by pbus_assign_resources_sorted().
+ */
+static void pdev_assign_fixed_resources(struct pci_dev *dev)
+{
+	int i;
+
+	for (i = 0; i <  PCI_NUM_RESOURCES; i++) {
+		struct pci_bus *b;
+		struct resource *r = &dev->resource[i];
+
+		if (r->parent || !(r->flags & IORESOURCE_PCI_FIXED) ||
+		    !(r->flags & (IORESOURCE_IO | IORESOURCE_MEM)))
+			continue;
+
+		b = dev->bus;
+		while (b && !r->parent) {
+			assign_fixed_resource_on_bus(b, r);
+			b = b->parent;
+		}
+		if (!r->parent) {
+			/*
+			 * If that didn't work, try to fallback to the
+			 * ultimate parent.
+			 */
+			if (r->flags &  IORESOURCE_IO)
+				request_resource(&ioport_resource, r);
+			else
+				request_resource(&iomem_resource, r);
+		}
+	}
+}
+
 void __pci_bus_assign_resources(const struct pci_bus *bus,
 				struct list_head *realloc_head,
 				struct list_head *fail_head)
@@ -1350,6 +1402,8 @@ void __pci_bus_assign_resources(const struct pci_bus *bus,
 	pbus_assign_resources_sorted(bus, realloc_head, fail_head);
 
 	list_for_each_entry(dev, &bus->devices, bus_list) {
+		pdev_assign_fixed_resources(dev);
+
 		b = dev->subordinate;
 		if (!b)
 			continue;
-- 
1.9.1

[PATCH v4 4/5] PCI: Handle Enhanced Allocation (EA) capability for SRIOV devices.

[David Daney]

From: David Daney <david.daney@cavium.com>

SRIOV BARs can be specified via EA entries.  Extend the EA parser to
extract the SRIOV BAR resources, and modify sriov_init() to use
resources previously obtained via EA.

Signed-off-by: David Daney <david.daney@cavium.com>
---
 drivers/pci/iov.c | 11 +++++++++--
 drivers/pci/pci.c |  5 +++++
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index ee0ebff..c789e68 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -436,8 +436,15 @@ found:
 	nres = 0;
 	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
 		res = &dev->resource[i + PCI_IOV_RESOURCES];
-		bar64 = __pci_read_base(dev, pci_bar_unknown, res,
-					pos + PCI_SRIOV_BAR + i * 4);
+		/*
+		 * If it is already FIXED, don't change it, something
+		 * (perhaps EA or header fixups) wants it this way.
+		 */
+		if (res->flags & IORESOURCE_PCI_FIXED)
+			bar64 = (res->flags & IORESOURCE_MEM_64) ? 1 : 0;
+		else
+			bar64 = __pci_read_base(dev, pci_bar_unknown, res,
+						pos + PCI_SRIOV_BAR + i * 4);
 		if (!res->flags)
 			continue;
 		if (resource_size(res) & (PAGE_SIZE - 1)) {
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 54a0a98..6750edf 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2178,6 +2178,11 @@ static struct resource *pci_ea_get_resource(struct pci_dev *dev, u8 bei, u8 prop
 {
 	if (bei <= PCI_EA_BEI_BAR5 && prop <= PCI_EA_P_IO)
 		return &dev->resource[bei];
+#ifdef CONFIG_PCI_IOV
+	else if (bei >= PCI_EA_BEI_VF_BAR0 && bei <= PCI_EA_BEI_VF_BAR5 &&
+		 (prop == PCI_EA_P_VIRT_MEM || prop == PCI_EA_P_VIRT_MEM_PREFETCH))
+		return &dev->resource[PCI_IOV_RESOURCES + bei - PCI_EA_BEI_VF_BAR0];
+#endif
 	else if (bei == PCI_EA_BEI_ROM)
 		return &dev->resource[PCI_ROM_RESOURCE];
 	else
-- 
1.9.1

[PATCH v4 5/5] PCI: Handle Enhanced Allocation (EA) capability for bridges

[David Daney]

From: David Daney <david.daney@cavium.com>

PCI bridges may have their properties be specified via EA entries.

Extend the EA parser to extract the bridge resources, and modify
pci_read_bridge_{io,mmio,mmio_pref}() to use resources previously
obtained via EA.

Save the offset to the EA capability in struct pci_dev, and use it to
easily find the EA bridge subordinate and secondary bus numbers.

When assigning the bridge resources a couple of changes are required
so that the EA obtained IORESOURCE_PCI_FIXED are not resized, and
correctly linked into the resource tree.

1) In pbus_size_mem() do not attempt to resize the bridge resources if
   they are marked as IORESOURCE_PCI_FIXED.

2) In pci_bus_alloc_from_region()for IORESOURCE_PCI_FIXED resources, just
   try to request the resource as is, without attempting to resize it.

Signed-off-by: David Daney <david.daney@cavium.com>
---
 drivers/pci/bus.c       |  7 +++++++
 drivers/pci/pci.c       | 13 +++++++++++++
 drivers/pci/probe.c     | 31 +++++++++++++++++++++++++++++--
 drivers/pci/setup-bus.c |  3 +++
 include/linux/pci.h     |  1 +
 5 files changed, 53 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 6fbd3f2..0556b33 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -153,6 +153,13 @@ static int pci_bus_alloc_from_region(struct pci_bus *bus, struct resource *res,
 		    !(res->flags & IORESOURCE_PREFETCH))
 			continue;
 
+		if (res->flags & IORESOURCE_PCI_FIXED) {
+			/* Cannot change it, just try to claim it. */
+			if (request_resource(r, res))
+				continue;
+			return 0;
+		}
+
 		avail = *r;
 		pci_clip_resource_to_region(bus, &avail, region);
 
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 6750edf..c857632 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2183,6 +2183,17 @@ static struct resource *pci_ea_get_resource(struct pci_dev *dev, u8 bei, u8 prop
 		 (prop == PCI_EA_P_VIRT_MEM || prop == PCI_EA_P_VIRT_MEM_PREFETCH))
 		return &dev->resource[PCI_IOV_RESOURCES + bei - PCI_EA_BEI_VF_BAR0];
 #endif
+	else if (bei == PCI_EA_BEI_BRIDGE)
+		switch (prop) {
+		case PCI_EA_P_BRIDGE_IO:
+			return &dev->resource[PCI_BRIDGE_RESOURCES + 0];
+		case PCI_EA_P_BRIDGE_MEM:
+			return &dev->resource[PCI_BRIDGE_RESOURCES + 1];
+		case PCI_EA_P_BRIDGE_MEM_PREFETCH:
+			return &dev->resource[PCI_BRIDGE_RESOURCES + 2];
+		default:
+			return NULL;
+		}
 	else if (bei == PCI_EA_BEI_ROM)
 		return &dev->resource[PCI_ROM_RESOURCE];
 	else
@@ -2321,6 +2332,8 @@ void pci_ea_init(struct pci_dev *dev)
 	if (!ea)
 		return;
 
+	dev->ea_cap = ea;
+
 	/* determine the number of entries */
 	pci_bus_read_config_byte(dev->bus, dev->devfn, ea + PCI_EA_NUM_ENT,
 					&num_ent);
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 4293eec..e4bcb0b 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -348,6 +348,10 @@ static void pci_read_bridge_io(struct pci_bus *child)
 	}
 
 	res = child->resource[0];
+	if (res->flags & IORESOURCE_PCI_FIXED) {
+		dev_dbg(&dev->dev, "  bridge window %pR (fixed)\n", res);
+		return;
+	}
 	pci_read_config_byte(dev, PCI_IO_BASE, &io_base_lo);
 	pci_read_config_byte(dev, PCI_IO_LIMIT, &io_limit_lo);
 	base = (io_base_lo & io_mask) << 8;
@@ -380,6 +384,10 @@ static void pci_read_bridge_mmio(struct pci_bus *child)
 	struct resource *res;
 
 	res = child->resource[1];
+	if (res->flags & IORESOURCE_PCI_FIXED) {
+		dev_dbg(&dev->dev, "  bridge window %pR (fixed)\n", res);
+		return;
+	}
 	pci_read_config_word(dev, PCI_MEMORY_BASE, &mem_base_lo);
 	pci_read_config_word(dev, PCI_MEMORY_LIMIT, &mem_limit_lo);
 	base = ((unsigned long) mem_base_lo & PCI_MEMORY_RANGE_MASK) << 16;
@@ -403,6 +411,10 @@ static void pci_read_bridge_mmio_pref(struct pci_bus *child)
 	struct resource *res;
 
 	res = child->resource[2];
+	if (res->flags & IORESOURCE_PCI_FIXED) {
+		dev_dbg(&dev->dev, "  bridge window %pR (fixed)\n", res);
+		return;
+	}
 	pci_read_config_word(dev, PCI_PREF_MEMORY_BASE, &mem_base_lo);
 	pci_read_config_word(dev, PCI_PREF_MEMORY_LIMIT, &mem_limit_lo);
 	base64 = (mem_base_lo & PCI_PREF_RANGE_MASK) << 16;
@@ -801,8 +813,23 @@ int pci_scan_bridge(struct pci_bus *bus, struct pci_dev *dev, int max, int pass)
 
 	pci_read_config_dword(dev, PCI_PRIMARY_BUS, &buses);
 	primary = buses & 0xFF;
-	secondary = (buses >> 8) & 0xFF;
-	subordinate = (buses >> 16) & 0xFF;
+	if (dev->ea_cap) {
+		u32 dw1;
+
+		pci_read_config_dword(dev, dev->ea_cap + 4, &dw1);
+		if (dw1 & 0xFF)
+			secondary = dw1 & 0xFF;
+		else
+			secondary = (buses >> 8) & 0xFF;
+
+		if ((dw1 >> 8) & 0xFF)
+			subordinate = (dw1 >> 8) & 0xFF;
+		else
+			subordinate = (buses >> 16) & 0xFF;
+	} else {
+		secondary = (buses >> 8) & 0xFF;
+		subordinate = (buses >> 16) & 0xFF;
+	}
 
 	dev_dbg(&dev->dev, "scanning [bus %02x-%02x] behind bridge, pass %d\n",
 		secondary, subordinate, pass);
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 8f9ed9b..8b8cc32 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1086,6 +1086,9 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
 		}
 	}
 
+	if (b_res->flags & IORESOURCE_PCI_FIXED)
+		return 0;
+
 	min_align = calculate_mem_align(aligns, max_order);
 	min_align = max(min_align, window_alignment(bus, b_res->flags));
 	size0 = calculate_memsize(size, min_size, 0, resource_size(b_res), min_align);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index b505b50..a8a177c 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -384,6 +384,7 @@ struct pci_dev {
 	phys_addr_t rom; /* Physical address of ROM if it's not from the BAR */
 	size_t romlen; /* Length of ROM if it's not from the BAR */
 	char *driver_override; /* Driver name to force a match */
+	u8		ea_cap; /* Enhanced Allocation capability offset */
 };
 
 static inline struct pci_dev *pci_physfn(struct pci_dev *dev)
-- 
1.9.1

Re: [PATCH v4 3/5] PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.

[Yinghai Lu]

On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
> From: David Daney <david.daney@cavium.com>
>
> The new Enhanced Allocation (EA) capability support creates resources
> with the IORESOURCE_PCI_FIXED set.  This creates a couple of problems:
>
> 1) Since these resources cannot be relocated or resized, their
>    alignment is not really defined, and it is therefore not specified.
>    This causes a problem in pbus_size_mem() where resources with
>    unspecified alignment are disabled.
>
> 2) During resource assignment in pci_bus_assign_resources(),
>    IORESOURCE_PCI_FIXED resources are not given a parent.  This, in
>    turn, causes pci_enable_resources() to fail with a "not claimed"
>    error.
>
> So, in pbus_size_mem() skip IORESOURCE_PCI_FIXED resources, instead of
> disabling them.
>
> In __pci_bus_assign_resources(), for IORESOURCE_PCI_FIXED resources,
> try to request the resource from a parent bus.

Can you check if

https://patchwork.kernel.org/patch/7304971/
[v6,05/53] PCI: Don't release fixed resource for realloc

https://patchwork.kernel.org/patch/7304371/
[v6,06/53] PCI: Claim fixed resource during remove/rescan path

address the the problem that you met?

        Yinghai

Re: [PATCH v4 3/5] PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.

[David Daney]

On 10/02/2015 04:14 PM, Yinghai Lu wrote:
> On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
>> From: David Daney <david.daney@cavium.com>
>>
>> The new Enhanced Allocation (EA) capability support creates resources
>> with the IORESOURCE_PCI_FIXED set.  This creates a couple of problems:
>>
>> 1) Since these resources cannot be relocated or resized, their
>>     alignment is not really defined, and it is therefore not specified.
>>     This causes a problem in pbus_size_mem() where resources with
>>     unspecified alignment are disabled.
>>
>> 2) During resource assignment in pci_bus_assign_resources(),
>>     IORESOURCE_PCI_FIXED resources are not given a parent.  This, in
>>     turn, causes pci_enable_resources() to fail with a "not claimed"
>>     error.
>>
>> So, in pbus_size_mem() skip IORESOURCE_PCI_FIXED resources, instead of
>> disabling them.
>>
>> In __pci_bus_assign_resources(), for IORESOURCE_PCI_FIXED resources,
>> try to request the resource from a parent bus.
>
> Can you check if
>
> https://patchwork.kernel.org/patch/7304971/
> [v6,05/53] PCI: Don't release fixed resource for realloc


This one isn't relevant as the problem is seen when we are acquiring 
resources, not releasing them.

>
> https://patchwork.kernel.org/patch/7304371/
> [v6,06/53] PCI: Claim fixed resource during remove/rescan path

This one is interesting, but I don't think it will work.

pci_claim_resource() calls pci_find_parent_resource(), which will fail 
in important use cases.

It is perfectly legal for a bridge provisioned by EA to not specify any 
resources.  In this case we must walk up the bus tree until we find 
something that contains the device resource, and can thus be a parent.

That is a big part of what my patch is doing.

As for the merits of assigning fixed resources from the FINAL fixup, 
rather than in __pci_bus_assign_resources(), I am unsure.

Thanks,
David Daney


>
> address the the problem that you met?
>
>          Yinghai
>

Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

[Sean O. Stalley]

Hi David,

I did a quick look through & overall I what you have done.
I will try to find some time to do a full review early next week.

Thanks Again,
Sean

On Fri, Oct 02, 2015 at 03:37:51PM -0700, David Daney wrote:
> From: David Daney <david.daney@cavium.com>
> 
> The original patches are from Sean O. Stalley. I made a few tweaks,
> but feel that it is substancially Sean's work, so I am keeping the
> patch set version numbering scheme going.
> 
> Tested on Cavium ThunderX system with 4 Root Complexes containing 50
> devices/bridges provisioned with EA.
> 
> Here is Sean's description of the patches:
> 
> PCI Enhanced Allocation is a new method of allocating MMIO & IO
> resources for PCI devices & bridges. It can be used instead
> of the traditional PCI method of using BARs.
> 
> EA entries are hardware-initialized to a fixed address.
> Unlike BARs, regions described by EA are cannot be moved.
> Because of this, only devices which are permanently connected to
> the PCI bus can use EA. A removable PCI card must not use EA.
> 
> This patchset adds support for using EA entries instead of BARs
> on Root Complex Integrated Endpoints.
> 
> The Enhanced Allocation ECN is publicly available here:
> https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf
> 
> 
> Changes from V1:
> 	- Use generic PCI resource claim functions (instead of EA-specific functions)
> 	- Only add support for RCiEPs (instead of all devices).
> 	- Removed some debugging messages leftover from early testing.
> 
> Changes from V2 (By David Daney):
> 	- Add ea_cap to struct pci_device, to aid in finding the EA capability.
> 	- Factored EA entity decoding into a separate function.
> 	- Add functions to find EA entities by BEI or Property.
> 	- Add handling of EA provisioned bridges.
> 	- Add handling of EA SRIOV BARs.
> 	- Try to assign proper resource parent so that SRIOV device creation can occur.
> 
> Changes from V3 (By David Daney):
> 	- Discarded V3 changes and started over fresh based on Sean's V2.
> 	- Add more support/checking for Entry Properties.
> 	- Allow EA behind bridges.
> 	- Rewrite some error messages.
> 	- Add patch 3/5 to prevent resizing, and better handle
>           assigning, of fixed EA resources.
> 	- Add patch 4/5 to handle EA provisioned SRIOV devices.
> 	- Add patch 5/5 to handle EA provisioned bridges.
> 
> David Daney (3):
>   PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.
>   PCI: Handle Enhanced Allocation (EA) capability for SRIOV devices.
>   PCI: Handle Enhanced Allocation (EA) capability for bridges
> 
> Sean O. Stalley (2):
>   PCI: Add Enhanced Allocation register entries
>   PCI: Add support for Enhanced Allocation devices
> 
>  drivers/pci/bus.c             |   7 ++
>  drivers/pci/iov.c             |  11 ++-
>  drivers/pci/pci.c             | 202 ++++++++++++++++++++++++++++++++++++++++++
>  drivers/pci/pci.h             |   1 +
>  drivers/pci/probe.c           |  34 ++++++-
>  drivers/pci/setup-bus.c       |  63 ++++++++++++-
>  include/linux/pci.h           |   1 +
>  include/uapi/linux/pci_regs.h |  44 ++++++++-
>  8 files changed, 355 insertions(+), 8 deletions(-)
> 
> -- 
> 1.9.1
> 

Re: [PATCH v4 3/5] PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.

[Yinghai Lu]

On Fri, Oct 2, 2015 at 4:38 PM, David Daney <ddaney@caviumnetworks.com> wrote:
> On 10/02/2015 04:14 PM, Yinghai Lu wrote:
>>
>> https://patchwork.kernel.org/patch/7304371/
>> [v6,06/53] PCI: Claim fixed resource during remove/rescan path
>
>
> This one is interesting, but I don't think it will work.
>
> pci_claim_resource() calls pci_find_parent_resource(), which will fail in
> important use cases.
>
> It is perfectly legal for a bridge provisioned by EA to not specify any
> resources.  In this case we must walk up the bus tree until we find
> something that contains the device resource, and can thus be a parent.
>
> That is a big part of what my patch is doing.

looks we need another resource flags for EA in addition to
FIXED as it could out side of bridge MMIO range.

>
> As for the merits of assigning fixed resources from the FINAL fixup, rather
> than in __pci_bus_assign_resources(), I am unsure.

usually __pci_bus_assign_resources() is used for unassigned one. so
I don't want it mixed with request resource there.

Thanks

Yinghai

Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

[Yinghai Lu]

On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
> From: David Daney <david.daney@cavium.com>
>
> PCI Enhanced Allocation is a new method of allocating MMIO & IO
> resources for PCI devices & bridges. It can be used instead
> of the traditional PCI method of using BARs.
>
> EA entries are hardware-initialized to a fixed address.
> Unlike BARs, regions described by EA are cannot be moved.
> Because of this, only devices which are permanently connected to
> the PCI bus can use EA. A removable PCI card must not use EA.
>
> The Enhanced Allocation ECN is publicly available here:
> https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf

Looks like the EA will support more than just fixed address later.

"Enhanced Allocation is an optional Conventional PCI Capability that
may be implemented by
Functions to indicate fixed (non reprogrammable) I/O and memory ranges
assigned to the
Function, as well as supporting new resource “type” definitions and
future extensibility to also
support reprogrammable allocations."

so I would prefer to think more to make frame configurable to leave
space for that.

Bjorn,

I wonder if we need to revive the add-on resource support patchset
that i suggested couple years ago,
so we can extend it to support EA features.

URL: https://lkml.org/lkml/2012/3/19/86

Thanks

Yinghai

Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

[David Daney]

On 10/02/2015 08:16 PM, Yinghai Lu wrote:
> On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
>> From: David Daney <david.daney@cavium.com>
>>
>> PCI Enhanced Allocation is a new method of allocating MMIO & IO
>> resources for PCI devices & bridges. It can be used instead
>> of the traditional PCI method of using BARs.
>>
>> EA entries are hardware-initialized to a fixed address.
>> Unlike BARs, regions described by EA are cannot be moved.
>> Because of this, only devices which are permanently connected to
>> the PCI bus can use EA. A removable PCI card must not use EA.
>>
>> The Enhanced Allocation ECN is publicly available here:
>> https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf
>
> Looks like the EA will support more than just fixed address later.
>
> "Enhanced Allocation is an optional Conventional PCI Capability that
> may be implemented by
> Functions to indicate fixed (non reprogrammable) I/O and memory ranges
> assigned to the
> Function, as well as supporting new resource “type” definitions and
> future extensibility to also
> support reprogrammable allocations."
>
> so I would prefer to think more to make frame configurable to leave
> space for that.

I disagree.  Currently we know of no use case other than devices with 
fixed address BARs and bridges to buses containing such devices.

Since we have no idea what these future types of EA entries might look 
like, it is impossible to design a framework to handle them.  Since we 
cannot design the framework, it seems like insisting on the creation of 
a framework is the equivalent of refusing to handle EA.

We have today, actual hardware with configuration space that contains 
the EA entries defined by the specification.  We know how to handle 
these, it can be done with this patch set.

I have no objection to changing the patches so that they fit better with 
the core PCI code.  In fact, I fully expect to do so based on feedback I 
receive.

For the record the general idea is this:  IORESOURCE_PCI_FIXED already 
does almost exactly what we need for EA, there are only a couple of 
places where we need to fix things for it to work well with the 
pci-host-generic driver.  If we fix these few issues with 
IORESOURCE_PCI_FIXED, then EA is supported.  It also makes doing similar 
things by setting the IORESOURCE_PCI_FIXED in a "header" fixup.  I don't 
think it make sense to invent an additional flag for fixed EA resources, 
as it would just be a duplicate of IORESOURCE_PCI_FIXED.

David Daney

>
> Bjorn,
>
> I wonder if we need to revive the add-on resource support patchset
> that i suggested couple years ago,
> so we can extend it to support EA features.
>
> URL: https://lkml.org/lkml/2012/3/19/86
>
> Thanks
>
> Yinghai
>

Re: [PATCH v4 3/5] PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.

[Sean O. Stalley]

On Fri, Oct 02, 2015 at 03:37:54PM -0700, David Daney wrote:

...

> +/*
> + * Try to assign any resources marked as IORESOURCE_PCI_FIXED, as they
> + * are skipped by pbus_assign_resources_sorted().
> + */
> +static void pdev_assign_fixed_resources(struct pci_dev *dev)
> +{
> +	int i;
> +
> +	for (i = 0; i <  PCI_NUM_RESOURCES; i++) {
> +		struct pci_bus *b;
> +		struct resource *r = &dev->resource[i];
> +
> +		if (r->parent || !(r->flags & IORESOURCE_PCI_FIXED) ||
> +		    !(r->flags & (IORESOURCE_IO | IORESOURCE_MEM)))
> +			continue;
> +
> +		b = dev->bus;
> +		while (b && !r->parent) {
> +			assign_fixed_resource_on_bus(b, r);
> +			b = b->parent;
> +		}
> +		if (!r->parent) {
> +			/*
> +			 * If that didn't work, try to fallback to the
> +			 * ultimate parent.
> +			 */
> +			if (r->flags &  IORESOURCE_IO)
> +				request_resource(&ioport_resource, r);
> +			else
> +				request_resource(&iomem_resource, r);
> +		}

I don't think this check is necessary.
assign_fixed_resource_on_bus() should find these resources when it reaches the top bus.
(since since the root bus contains these resources).

> +	}
> +}

-Sean

Re: [PATCH v4 3/5] PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.

[Sean O. Stalley]

On Fri, Oct 02, 2015 at 08:00:20PM -0700, Yinghai Lu wrote:
> On Fri, Oct 2, 2015 at 4:38 PM, David Daney <ddaney@caviumnetworks.com> wrote:
> > On 10/02/2015 04:14 PM, Yinghai Lu wrote:
> >>
> >> https://patchwork.kernel.org/patch/7304371/
> >> [v6,06/53] PCI: Claim fixed resource during remove/rescan path
> >
> >
> > This one is interesting, but I don't think it will work.
> >
> > pci_claim_resource() calls pci_find_parent_resource(), which will fail in
> > important use cases.
> >
> > It is perfectly legal for a bridge provisioned by EA to not specify any
> > resources.  In this case we must walk up the bus tree until we find
> > something that contains the device resource, and can thus be a parent.
> >
> > That is a big part of what my patch is doing.
> 
> looks we need another resource flags for EA in addition to
> FIXED as it could out side of bridge MMIO range.

I think I get what you are saying, but let me check.

If a bridge gets reconfigured, it needs to be able to tell if this resource is EA or not.
If it is a fixed EA resource, it doesn't need to include it in its MMIO range.
If it is fixed for some other reason,
	we should make sure the bridges MMIO range includes the fixed region.

You are suggesting that we should add a flag for EA, so if a bridge tries to move,
the bridge knows weather or not the resource is EA,
and therefore if it needs to worry about the resource when setting it's range?

Is that what you are trying to say?

-Sean

Re: [PATCH v4 5/5] PCI: Handle Enhanced Allocation (EA) capability for bridges

[Sean O. Stalley]

On Fri, Oct 02, 2015 at 03:37:56PM -0700, David Daney wrote:
> From: David Daney <david.daney@cavium.com>
> 
> PCI bridges may have their properties be specified via EA entries.
> 
> Extend the EA parser to extract the bridge resources, and modify
> pci_read_bridge_{io,mmio,mmio_pref}() to use resources previously
> obtained via EA.
> 
> Save the offset to the EA capability in struct pci_dev, and use it to
> easily find the EA bridge subordinate and secondary bus numbers.
> 
> When assigning the bridge resources a couple of changes are required
> so that the EA obtained IORESOURCE_PCI_FIXED are not resized, and
> correctly linked into the resource tree.
> 
> 1) In pbus_size_mem() do not attempt to resize the bridge resources if
>    they are marked as IORESOURCE_PCI_FIXED.
> 
> 2) In pci_bus_alloc_from_region()for IORESOURCE_PCI_FIXED resources, just
>    try to request the resource as is, without attempting to resize it.
> 
> Signed-off-by: David Daney <david.daney@cavium.com>
> ---
>  drivers/pci/bus.c       |  7 +++++++
>  drivers/pci/pci.c       | 13 +++++++++++++
>  drivers/pci/probe.c     | 31 +++++++++++++++++++++++++++++--
>  drivers/pci/setup-bus.c |  3 +++
>  include/linux/pci.h     |  1 +
>  5 files changed, 53 insertions(+), 2 deletions(-)
> @@ -801,8 +813,23 @@ int pci_scan_bridge(struct pci_bus *bus, struct pci_dev *dev, int max, int pass)
>  
>  	pci_read_config_dword(dev, PCI_PRIMARY_BUS, &buses);
>  	primary = buses & 0xFF;
> -	secondary = (buses >> 8) & 0xFF;
> -	subordinate = (buses >> 16) & 0xFF;
> +	if (dev->ea_cap) {
> +		u32 dw1;
> +
> +		pci_read_config_dword(dev, dev->ea_cap + 4, &dw1);
> +		if (dw1 & 0xFF)
> +			secondary = dw1 & 0xFF;
> +		else
> +			secondary = (buses >> 8) & 0xFF;
> +
> +		if ((dw1 >> 8) & 0xFF)
> +			subordinate = (dw1 >> 8) & 0xFF;
> +		else
> +			subordinate = (buses >> 16) & 0xFF;
> +	} else {
> +		secondary = (buses >> 8) & 0xFF;
> +		subordinate = (buses >> 16) & 0xFF;
> +	}

We can refactor this to make it cleaner/more compact. from V3 review:


	secondary = (buses >> 8) & 0xFF;
	subordinate = (buses >> 16) & 0xFF;
	if (dev->ea_cap) {
		u32 sdw;
		
		pci_read_config_dword(dev, dev->ea_cap + 4, &sdw);
		
		if (sdw & 0xFF)
			secondary = sdw & 0xFF;
		
		sdw >>= 8;
		if (sdw & 0xFF)
			subordinate = sdw & 0xFF;
	}


-Sean

Re: [PATCH v4 5/5] PCI: Handle Enhanced Allocation (EA) capability for bridges

[David Daney]

On 10/05/2015 03:54 PM, Sean O. Stalley wrote:
> On Fri, Oct 02, 2015 at 03:37:56PM -0700, David Daney wrote:
>> From: David Daney <david.daney@cavium.com>
>>
>> PCI bridges may have their properties be specified via EA entries.
>>
>> Extend the EA parser to extract the bridge resources, and modify
>> pci_read_bridge_{io,mmio,mmio_pref}() to use resources previously
>> obtained via EA.
>>
>> Save the offset to the EA capability in struct pci_dev, and use it to
>> easily find the EA bridge subordinate and secondary bus numbers.
>>
>> When assigning the bridge resources a couple of changes are required
>> so that the EA obtained IORESOURCE_PCI_FIXED are not resized, and
>> correctly linked into the resource tree.
>>
>> 1) In pbus_size_mem() do not attempt to resize the bridge resources if
>>     they are marked as IORESOURCE_PCI_FIXED.
>>
>> 2) In pci_bus_alloc_from_region()for IORESOURCE_PCI_FIXED resources, just
>>     try to request the resource as is, without attempting to resize it.
>>
>> Signed-off-by: David Daney <david.daney@cavium.com>
>> ---
>>   drivers/pci/bus.c       |  7 +++++++
>>   drivers/pci/pci.c       | 13 +++++++++++++
>>   drivers/pci/probe.c     | 31 +++++++++++++++++++++++++++++--
>>   drivers/pci/setup-bus.c |  3 +++
>>   include/linux/pci.h     |  1 +
>>   5 files changed, 53 insertions(+), 2 deletions(-)
>> @@ -801,8 +813,23 @@ int pci_scan_bridge(struct pci_bus *bus, struct pci_dev *dev, int max, int pass)
>>
>>   	pci_read_config_dword(dev, PCI_PRIMARY_BUS, &buses);
>>   	primary = buses & 0xFF;
>> -	secondary = (buses >> 8) & 0xFF;
>> -	subordinate = (buses >> 16) & 0xFF;
>> +	if (dev->ea_cap) {
>> +		u32 dw1;
>> +
>> +		pci_read_config_dword(dev, dev->ea_cap + 4, &dw1);
>> +		if (dw1 & 0xFF)
>> +			secondary = dw1 & 0xFF;
>> +		else
>> +			secondary = (buses >> 8) & 0xFF;
>> +
>> +		if ((dw1 >> 8) & 0xFF)
>> +			subordinate = (dw1 >> 8) & 0xFF;
>> +		else
>> +			subordinate = (buses >> 16) & 0xFF;
>> +	} else {
>> +		secondary = (buses >> 8) & 0xFF;
>> +		subordinate = (buses >> 16) & 0xFF;
>> +	}
>
> We can refactor this to make it cleaner/more compact. from V3 review:
>
>
> 	secondary = (buses >> 8) & 0xFF;
> 	subordinate = (buses >> 16) & 0xFF;
> 	if (dev->ea_cap) {
> 		u32 sdw;
> 		
> 		pci_read_config_dword(dev, dev->ea_cap + 4, &sdw);
> 		
> 		if (sdw & 0xFF)
> 			secondary = sdw & 0xFF;
> 		
> 		sdw >>= 8;
> 		if (sdw & 0xFF)
> 			subordinate = sdw & 0xFF;
> 	}
>

Yes, that is cleaner.  I think I didn't read the comments on v3 closely 
enough.  I will switch to doing it this way.

Thanks,
David Daney


>
> -Sean
>

Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

[Sean O. Stalley]

On Fri, Oct 02, 2015 at 08:16:48PM -0700, Yinghai Lu wrote:
> On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
> > From: David Daney <david.daney@cavium.com>
> >
> > PCI Enhanced Allocation is a new method of allocating MMIO & IO
> > resources for PCI devices & bridges. It can be used instead
> > of the traditional PCI method of using BARs.
> >
> > EA entries are hardware-initialized to a fixed address.
> > Unlike BARs, regions described by EA are cannot be moved.
> > Because of this, only devices which are permanently connected to
> > the PCI bus can use EA. A removable PCI card must not use EA.
> >
> > The Enhanced Allocation ECN is publicly available here:
> > https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf
> 
> Looks like the EA will support more than just fixed address later.
> 
> "Enhanced Allocation is an optional Conventional PCI Capability that
> may be implemented by
> Functions to indicate fixed (non reprogrammable) I/O and memory ranges
> assigned to the
> Function, as well as supporting new resource “type” definitions and
> future extensibility to also
> support reprogrammable allocations."
> 
> so I would prefer to think more to make frame configurable to leave
> space for that.
> 
> Bjorn,
> 
> I wonder if we need to revive the add-on resource support patchset
> that i suggested couple years ago,
> so we can extend it to support EA features.
> 
> URL: https://lkml.org/lkml/2012/3/19/86
> 
> Thanks
> 
> Yinghai

This might be useful for fixed resources as well.

For some BEI values, EA allows for an arbitrary number of EA entries.
For PF & VF resource ranges, it allows 2 ranges.
(one below the 4GB boundry, and one above).
I don't think the current pci_dev struct can handle that many resources.

-Sean

Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

[David Daney]

On 10/05/2015 04:05 PM, Sean O. Stalley wrote:
> On Fri, Oct 02, 2015 at 08:16:48PM -0700, Yinghai Lu wrote:
>> On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
>>> From: David Daney <david.daney@cavium.com>
>>>
>>> PCI Enhanced Allocation is a new method of allocating MMIO & IO
>>> resources for PCI devices & bridges. It can be used instead
>>> of the traditional PCI method of using BARs.
>>>
>>> EA entries are hardware-initialized to a fixed address.
>>> Unlike BARs, regions described by EA are cannot be moved.
>>> Because of this, only devices which are permanently connected to
>>> the PCI bus can use EA. A removable PCI card must not use EA.
>>>
>>> The Enhanced Allocation ECN is publicly available here:
>>> https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf
>>
>> Looks like the EA will support more than just fixed address later.
>>
>> "Enhanced Allocation is an optional Conventional PCI Capability that
>> may be implemented by
>> Functions to indicate fixed (non reprogrammable) I/O and memory ranges
>> assigned to the
>> Function, as well as supporting new resource “type” definitions and
>> future extensibility to also
>> support reprogrammable allocations."
>>
>> so I would prefer to think more to make frame configurable to leave
>> space for that.
>>
>> Bjorn,
>>
>> I wonder if we need to revive the add-on resource support patchset
>> that i suggested couple years ago,
>> so we can extend it to support EA features.
>>
>> URL: https://lkml.org/lkml/2012/3/19/86
>>
>> Thanks
>>
>> Yinghai
>
> This might be useful for fixed resources as well.
>
> For some BEI values, EA allows for an arbitrary number of EA entries.

I think this is true only for BEI = 6 which is for type-1 config space 
only (i.e. for bridges)

I am thinking about splitting out the bridge part of the patch set, as 
my systems work fine without explicitly assigning bridge resources via 
EA.  That would allow us to more forward with the patches that are less 
controversial, and spend more time hashing out the proper approach to 
take with bridges.

> For PF & VF resource ranges, it allows 2 ranges.

I don't really understand what you are saying here.  My reading of the 
spec. is that BEI[0..5] are PF BARs and each may have any of the 
properties that are allowed for normal BARs (io, memory-nonprefetchable, 
memory-prefetchable).  BEI[9..14] are VF BARs, and likewise may have any 
of the properties that are alloed for normal VF bars 
(memory-nonprefetchable, memory-prefetchable)

I guess in theory EA allows you to allocate 6 64-bit BARs, where you 
would be limited to only 3 64-bit normal BARs


> (one below the 4GB boundry, and one above).
> I don't think the current pci_dev struct can handle that many resources.
>
> -Sean
>

Re: [PATCH v4 0/5] PCI: Add support for PCI Enhanced Allocation "BARs"

[Sean O. Stalley]

On Mon, Oct 05, 2015 at 06:17:20PM -0700, David Daney wrote:
> On 10/05/2015 04:05 PM, Sean O. Stalley wrote:
> >On Fri, Oct 02, 2015 at 08:16:48PM -0700, Yinghai Lu wrote:
> >>On Fri, Oct 2, 2015 at 3:37 PM, David Daney <ddaney.cavm@gmail.com> wrote:
> >>>From: David Daney <david.daney@cavium.com>
> >>>
> >>>PCI Enhanced Allocation is a new method of allocating MMIO & IO
> >>>resources for PCI devices & bridges. It can be used instead
> >>>of the traditional PCI method of using BARs.
> >>>
> >>>EA entries are hardware-initialized to a fixed address.
> >>>Unlike BARs, regions described by EA are cannot be moved.
> >>>Because of this, only devices which are permanently connected to
> >>>the PCI bus can use EA. A removable PCI card must not use EA.
> >>>
> >>>The Enhanced Allocation ECN is publicly available here:
> >>>https://www.pcisig.com/specifications/conventional/ECN_Enhanced_Allocation_23_Oct_2014_Final.pdf
> >>
> >>Looks like the EA will support more than just fixed address later.
> >>
> >>"Enhanced Allocation is an optional Conventional PCI Capability that
> >>may be implemented by
> >>Functions to indicate fixed (non reprogrammable) I/O and memory ranges
> >>assigned to the
> >>Function, as well as supporting new resource “type” definitions and
> >>future extensibility to also
> >>support reprogrammable allocations."
> >>
> >>so I would prefer to think more to make frame configurable to leave
> >>space for that.
> >>
> >>Bjorn,
> >>
> >>I wonder if we need to revive the add-on resource support patchset
> >>that i suggested couple years ago,
> >>so we can extend it to support EA features.
> >>
> >>URL: https://lkml.org/lkml/2012/3/19/86
> >>
> >>Thanks
> >>
> >>Yinghai
> >
> >This might be useful for fixed resources as well.
> >
> >For some BEI values, EA allows for an arbitrary number of EA entries.
> 
> I think this is true only for BEI = 6 which is for type-1 config
> space only (i.e. for bridges)

It's true for a BEI of 6 or 7.

>From the ECN (specifically section 6.9.1.3):

Rules for use of BEI field:
	...
	It is permitted for an arbitrary number of entries to assign a BEI of 6 or 7.

> I am thinking about splitting out the bridge part of the patch set,
> as my systems work fine without explicitly assigning bridge
> resources via EA.  That would allow us to more forward with the
> patches that are less controversial, and spend more time hashing out
> the proper approach to take with bridges.

I like this idea. Adding support for Endpoints will be much simpler than
bridges.

> 
> >For PF & VF resource ranges, it allows 2 ranges.
> 
> I don't really understand what you are saying here.  My reading of
> the spec. is that BEI[0..5] are PF BARs and each may have any of the
> properties that are allowed for normal BARs (io,
> memory-nonprefetchable, memory-prefetchable).  BEI[9..14] are VF
> BARs, and likewise may have any of the properties that are alloed
> for normal VF bars (memory-nonprefetchable, memory-prefetchable)


>From the ECN (specifically section 6.9.1.3):

Rules for use of BEI field:
...

For Memory or I/O BARs where the Primary or Secondary Property is 00h, 01h or
02h, it is permitted to assign the same BEI in the range of 0-5 once for a range where
Base + MaxOffset is below 4GB, and again for a range where Base + MaxOffset is
greater than 4GB; It is not otherwise permitted to assign the same BEI in the range 0-
5 for more than one entry.

For Virtual Function BARs where the Primary or Secondary Property is 03h or 04h it
is permitted to assign the same BEI in the range of 0-5 once for a range where Base +
MaxOffset is below 4GB, and again for a range where Base + MaxOffset is greater
than 4GB; It is not otherwise permitted to assign the same BEI in the range 0-5 for
more than one VF entry.

(Theres a typo in the VF rule. I think the '0-5' range should be '9-14' for VFs)

> I guess in theory EA allows you to allocate 6 64-bit BARs, where you
> would be limited to only 3 64-bit normal BARs

I guess so, I never thought about that.

> >(one below the 4GB boundry, and one above).
> >I don't think the current pci_dev struct can handle that many resources.
> >
> >-Sean
> >

Let me know if you want any help with the patchset. I'll do what I can.
I'm in today, but I'm out Wed-Fri this week.

-Sean

Re: [PATCH v4 3/5] PCI: Handle IORESOURCE_PCI_FIXED when sizing and assigning resources.

[David Daney]

On 10/05/2015 03:23 PM, Sean O. Stalley wrote:
> On Fri, Oct 02, 2015 at 03:37:54PM -0700, David Daney wrote:
>
> ...
>
>> +/*
>> + * Try to assign any resources marked as IORESOURCE_PCI_FIXED, as they
>> + * are skipped by pbus_assign_resources_sorted().
>> + */
>> +static void pdev_assign_fixed_resources(struct pci_dev *dev)
>> +{
>> +	int i;
>> +
>> +	for (i = 0; i <  PCI_NUM_RESOURCES; i++) {
>> +		struct pci_bus *b;
>> +		struct resource *r = &dev->resource[i];
>> +
>> +		if (r->parent || !(r->flags & IORESOURCE_PCI_FIXED) ||
>> +		    !(r->flags & (IORESOURCE_IO | IORESOURCE_MEM)))
>> +			continue;
>> +
>> +		b = dev->bus;
>> +		while (b && !r->parent) {
>> +			assign_fixed_resource_on_bus(b, r);
>> +			b = b->parent;
>> +		}
>> +		if (!r->parent) {
>> +			/*
>> +			 * If that didn't work, try to fallback to the
>> +			 * ultimate parent.
>> +			 */
>> +			if (r->flags &  IORESOURCE_IO)
>> +				request_resource(&ioport_resource, r);
>> +			else
>> +				request_resource(&iomem_resource, r);
>> +		}
>
> I don't think this check is necessary.
> assign_fixed_resource_on_bus() should find these resources when it reaches the top bus.
> (since since the root bus contains these resources).

You are correct.  It does work fine without that last fallback bit.  I 
will remove that, and re-send the patches.

Thanks,
David Daney



>
>> +	}
>> +}
>
> -Sean
>