[PATCH v2 0/3] arm/arm64: pci: PCI_PROBE_ONLY clean-up

[PATCH v2 0/3] arm/arm64: pci: PCI_PROBE_ONLY clean-up

[Lorenzo Pieralisi]

Current arm/arm64 code prevents enabling resources in the respective
pcibios_enable_device() callbacks if the PCI_PROBE_ONLY flag is set,
in that on those platforms the resources tree was not validated properly
since PCI resources are not currently claimed on arm/arm64 PCI_PROBE_ONLY
systems.

This is a temporary kludge, in that PCI_PROBE_ONLY flag is used to
describe systems with fixed resources, that can nonetheless be enabled
through the standard pci_enable_resources() call present in PCI core code,
(ie via the generic pcibios_enable_device() call).

To remove the PCI_PROBE_ONLY flag in arm/arm64 pcibios_enable_device()
callbacks, the PCI host controllers that can be used with
PCI_PROBE_ONLY configurations must make sure that devices resources are
validated and inserted in the kernel resource tree even on PCI_PROBE_ONLY
systems so that the generic pcibios_enable_device() generic code, while
enabling resources (pci_enable_resources()), does not find dangling
resources pointers (ie missing parent pointers) that are omens of
an incomplete resource tree, causing failures in resources enablement.

PCI core code provides interfaces to assign/reassign/reallocate PCI
bus resources but it is currently lacking an interface to claim
resources for a specific bus. Arches implement resources claiming through
ad-hoc code built on top of pci_claim_resource() API, but that code
cannot be leveraged on architectures like arm/arm64 that rely on the
generic PCI infrastructure to carry out resources claiming/assignment.

Therefore, to clean up the arm/arm64 resources enablement on PCI_PROBE_ONLY
systems this patchset implements three patches:

PATCH 1: Create PCI core code infrastructure to claim bus resources
PATCH 2: Leverage the infrastructure in PATCH 1 to claim resources in
         the PCI generic host controller on PCI_PROBE_ONLY systems
PATCH 3: Remove PCI_PROBE_ONLY kludges from arm/arm64 PCI back-ends to
         complete the clean-up and leverage the PCI generic
         pcibios_enable_device() implementation

Tested on arm/arm64 systems with kvmtool and PCI host generic.

v1 -> v2

- Rewrote patch 1 to recursively claim resources for the whole PCI bus
  hierarchy
- Updated commits logs/tags
- Rebased against v4.5-rc6

v1: https://patchwork.ozlabs.org/patch/545669/

Lorenzo Pieralisi (3):
  drivers: pci: add generic code to claim bus resources
  drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY
    set-ups
  arm/arm64: pci: remove arch specific pcibios_enable_device()

 arch/arm/kernel/bios32.c            | 12 -------
 arch/arm64/kernel/pci.c             | 13 -------
 drivers/pci/host/pci-host-generic.c |  5 ++-
 drivers/pci/setup-bus.c             | 68 +++++++++++++++++++++++++++++++++++++
 include/linux/pci.h                 |  1 +
 5 files changed, 73 insertions(+), 26 deletions(-)

-- 
2.5.1

[PATCH v2 0/3] arm/arm64: pci: PCI_PROBE_ONLY clean-up

[Lorenzo Pieralisi]

Current arm/arm64 code prevents enabling resources in the respective
pcibios_enable_device() callbacks if the PCI_PROBE_ONLY flag is set,
in that on those platforms the resources tree was not validated properly
since PCI resources are not currently claimed on arm/arm64 PCI_PROBE_ONLY
systems.

This is a temporary kludge, in that PCI_PROBE_ONLY flag is used to
describe systems with fixed resources, that can nonetheless be enabled
through the standard pci_enable_resources() call present in PCI core code,
(ie via the generic pcibios_enable_device() call).

To remove the PCI_PROBE_ONLY flag in arm/arm64 pcibios_enable_device()
callbacks, the PCI host controllers that can be used with
PCI_PROBE_ONLY configurations must make sure that devices resources are
validated and inserted in the kernel resource tree even on PCI_PROBE_ONLY
systems so that the generic pcibios_enable_device() generic code, while
enabling resources (pci_enable_resources()), does not find dangling
resources pointers (ie missing parent pointers) that are omens of
an incomplete resource tree, causing failures in resources enablement.

PCI core code provides interfaces to assign/reassign/reallocate PCI
bus resources but it is currently lacking an interface to claim
resources for a specific bus. Arches implement resources claiming through
ad-hoc code built on top of pci_claim_resource() API, but that code
cannot be leveraged on architectures like arm/arm64 that rely on the
generic PCI infrastructure to carry out resources claiming/assignment.

Therefore, to clean up the arm/arm64 resources enablement on PCI_PROBE_ONLY
systems this patchset implements three patches:

PATCH 1: Create PCI core code infrastructure to claim bus resources
PATCH 2: Leverage the infrastructure in PATCH 1 to claim resources in
         the PCI generic host controller on PCI_PROBE_ONLY systems
PATCH 3: Remove PCI_PROBE_ONLY kludges from arm/arm64 PCI back-ends to
         complete the clean-up and leverage the PCI generic
         pcibios_enable_device() implementation

Tested on arm/arm64 systems with kvmtool and PCI host generic.

v1 -> v2

- Rewrote patch 1 to recursively claim resources for the whole PCI bus
  hierarchy
- Updated commits logs/tags
- Rebased against v4.5-rc6

v1: https://patchwork.ozlabs.org/patch/545669/

Lorenzo Pieralisi (3):
  drivers: pci: add generic code to claim bus resources
  drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY
    set-ups
  arm/arm64: pci: remove arch specific pcibios_enable_device()

 arch/arm/kernel/bios32.c            | 12 -------
 arch/arm64/kernel/pci.c             | 13 -------
 drivers/pci/host/pci-host-generic.c |  5 ++-
 drivers/pci/setup-bus.c             | 68 +++++++++++++++++++++++++++++++++++++
 include/linux/pci.h                 |  1 +
 5 files changed, 73 insertions(+), 26 deletions(-)

-- 
2.5.1

[PATCH v2 1/3] drivers: pci: add generic code to claim bus resources

[Lorenzo Pieralisi]

From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>

PCI core code contains a set of functions, eg:

pci_assign_unassigned_bus_resources()

that allow to assign the PCI resources for a given bus after
enumeration.

On systems where the PCI BARs are immutable (ie they must not and can
not be assigned), PCI bus resources should still be inserted in the PCI
resources tree (even if they are not (re)-assigned), but there is
no generic PCI kernel function for that purpose. Currently the PCI bus
resources insertion in the resources tree is implemented in an arch
specific fashion through arch specific callbacks that rely on the PCI
resources claiming API (for bridges and devices) which resulted in arches
implementations that contain duplicated code to claim resources for a
given PCI bus hierarchy.

This patch, based on the x86/ia64 resources claiming arch implementations,
implements a set of functions in core PCI code that provides a PCI core
interface for resources claiming for a given PCI bus hierarchy, paving
the way for further resource claiming consolidation across architectures.

Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/setup-bus.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h     |  1 +
 2 files changed, 69 insertions(+)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 7796d0a..95f0906 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1424,6 +1424,74 @@ void pci_bus_assign_resources(const struct pci_bus *bus)
 }
 EXPORT_SYMBOL(pci_bus_assign_resources);
 
+static void pci_claim_device_resources(struct pci_dev *dev)
+{
+	int i;
+
+	for (i = 0; i < PCI_BRIDGE_RESOURCES; i++) {
+		struct resource *r = &dev->resource[i];
+
+		if (!r->flags || r->parent)
+			continue;
+
+		pci_claim_resource(dev, i);
+	}
+}
+
+static void pci_claim_bridge_resources(struct pci_dev *dev)
+{
+	int i;
+
+	for (i = PCI_BRIDGE_RESOURCES; i < PCI_NUM_RESOURCES; i++) {
+		struct resource *r = &dev->resource[i];
+
+		if (!r->flags || r->parent)
+			continue;
+
+		pci_claim_bridge_resource(dev, i);
+	}
+}
+
+static void pci_bus_allocate_dev_resources(struct pci_bus *b)
+{
+	struct pci_dev *dev;
+	struct pci_bus *child;
+
+	list_for_each_entry(dev, &b->devices, bus_list) {
+		pci_claim_device_resources(dev);
+
+		child = dev->subordinate;
+		if (child)
+			pci_bus_allocate_dev_resources(child);
+	}
+}
+
+static void pci_bus_allocate_resources(struct pci_bus *b)
+{
+	struct pci_bus *child;
+
+	/*
+	 * Carry out a depth-first search on the PCI bus
+	 * tree to allocate bridge apertures. Read the
+	 * programmed bridge bases and recursively claim
+	 * the respective bridge resources.
+	 */
+	if (b->self) {
+		pci_read_bridge_bases(b);
+		pci_claim_bridge_resources(b->self);
+	}
+
+	list_for_each_entry(child, &b->children, node)
+		pci_bus_allocate_resources(child);
+}
+
+void pci_bus_claim_resources(struct pci_bus *b)
+{
+	pci_bus_allocate_resources(b);
+	pci_bus_allocate_dev_resources(b);
+}
+EXPORT_SYMBOL(pci_bus_claim_resources);
+
 static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
 					  struct list_head *add_head,
 					  struct list_head *fail_head)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 2771625..e66eba7 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1102,6 +1102,7 @@ ssize_t pci_write_vpd(struct pci_dev *dev, loff_t pos, size_t count, const void
 /* Helper functions for low-level code (drivers/pci/setup-[bus,res].c) */
 resource_size_t pcibios_retrieve_fw_addr(struct pci_dev *dev, int idx);
 void pci_bus_assign_resources(const struct pci_bus *bus);
+void pci_bus_claim_resources(struct pci_bus *bus);
 void pci_bus_size_bridges(struct pci_bus *bus);
 int pci_claim_resource(struct pci_dev *, int);
 int pci_claim_bridge_resource(struct pci_dev *bridge, int i);
-- 
2.5.1

[PATCH v2 1/3] drivers: pci: add generic code to claim bus resources

[Lorenzo Pieralisi]

From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>

PCI core code contains a set of functions, eg:

pci_assign_unassigned_bus_resources()

that allow to assign the PCI resources for a given bus after
enumeration.

On systems where the PCI BARs are immutable (ie they must not and can
not be assigned), PCI bus resources should still be inserted in the PCI
resources tree (even if they are not (re)-assigned), but there is
no generic PCI kernel function for that purpose. Currently the PCI bus
resources insertion in the resources tree is implemented in an arch
specific fashion through arch specific callbacks that rely on the PCI
resources claiming API (for bridges and devices) which resulted in arches
implementations that contain duplicated code to claim resources for a
given PCI bus hierarchy.

This patch, based on the x86/ia64 resources claiming arch implementations,
implements a set of functions in core PCI code that provides a PCI core
interface for resources claiming for a given PCI bus hierarchy, paving
the way for further resource claiming consolidation across architectures.

Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Yinghai Lu <yinghai@kernel.org>
---
 drivers/pci/setup-bus.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h     |  1 +
 2 files changed, 69 insertions(+)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 7796d0a..95f0906 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1424,6 +1424,74 @@ void pci_bus_assign_resources(const struct pci_bus *bus)
 }
 EXPORT_SYMBOL(pci_bus_assign_resources);
 
+static void pci_claim_device_resources(struct pci_dev *dev)
+{
+	int i;
+
+	for (i = 0; i < PCI_BRIDGE_RESOURCES; i++) {
+		struct resource *r = &dev->resource[i];
+
+		if (!r->flags || r->parent)
+			continue;
+
+		pci_claim_resource(dev, i);
+	}
+}
+
+static void pci_claim_bridge_resources(struct pci_dev *dev)
+{
+	int i;
+
+	for (i = PCI_BRIDGE_RESOURCES; i < PCI_NUM_RESOURCES; i++) {
+		struct resource *r = &dev->resource[i];
+
+		if (!r->flags || r->parent)
+			continue;
+
+		pci_claim_bridge_resource(dev, i);
+	}
+}
+
+static void pci_bus_allocate_dev_resources(struct pci_bus *b)
+{
+	struct pci_dev *dev;
+	struct pci_bus *child;
+
+	list_for_each_entry(dev, &b->devices, bus_list) {
+		pci_claim_device_resources(dev);
+
+		child = dev->subordinate;
+		if (child)
+			pci_bus_allocate_dev_resources(child);
+	}
+}
+
+static void pci_bus_allocate_resources(struct pci_bus *b)
+{
+	struct pci_bus *child;
+
+	/*
+	 * Carry out a depth-first search on the PCI bus
+	 * tree to allocate bridge apertures. Read the
+	 * programmed bridge bases and recursively claim
+	 * the respective bridge resources.
+	 */
+	if (b->self) {
+		pci_read_bridge_bases(b);
+		pci_claim_bridge_resources(b->self);
+	}
+
+	list_for_each_entry(child, &b->children, node)
+		pci_bus_allocate_resources(child);
+}
+
+void pci_bus_claim_resources(struct pci_bus *b)
+{
+	pci_bus_allocate_resources(b);
+	pci_bus_allocate_dev_resources(b);
+}
+EXPORT_SYMBOL(pci_bus_claim_resources);
+
 static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
 					  struct list_head *add_head,
 					  struct list_head *fail_head)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 2771625..e66eba7 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1102,6 +1102,7 @@ ssize_t pci_write_vpd(struct pci_dev *dev, loff_t pos, size_t count, const void
 /* Helper functions for low-level code (drivers/pci/setup-[bus,res].c) */
 resource_size_t pcibios_retrieve_fw_addr(struct pci_dev *dev, int idx);
 void pci_bus_assign_resources(const struct pci_bus *bus);
+void pci_bus_claim_resources(struct pci_bus *bus);
 void pci_bus_size_bridges(struct pci_bus *bus);
 int pci_claim_resource(struct pci_dev *, int);
 int pci_claim_bridge_resource(struct pci_dev *bridge, int i);
-- 
2.5.1

[PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Lorenzo Pieralisi]

The PCI host generic driver does not reassign bus resources on systems
that require the BARs set-up to be immutable (ie PCI_PROBE_ONLY) since
that would trigger system failures. Nonetheless, PCI bus resources
allocated to PCI bridge and devices must be claimed in order to be
validated and inserted in the kernel resource tree, but the current
driver omits the resources claiming and relies on arch specific kludges
to prevent probing failure (ie preventing resources enablement on
PCI_PROBE_ONLY systems).

This patch adds code to the PCI host generic driver that correctly
claims bus resources upon probe on systems that are required to
prevent reassignment after bus enumeration, so that the allocated
resources can be enabled successfully upon PCI device drivers probing,
without resorting to arch back-ends workarounds.

Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: David Daney <david.daney@cavium.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
---
 drivers/pci/host/pci-host-generic.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/host/pci-host-generic.c b/drivers/pci/host/pci-host-generic.c
index 1652bc7..e529825 100644
--- a/drivers/pci/host/pci-host-generic.c
+++ b/drivers/pci/host/pci-host-generic.c
@@ -252,7 +252,10 @@ static int gen_pci_probe(struct platform_device *pdev)
 
 	pci_fixup_irqs(pci_common_swizzle, of_irq_parse_and_map_pci);
 
-	if (!pci_has_flag(PCI_PROBE_ONLY)) {
+
+	if (pci_has_flag(PCI_PROBE_ONLY)) {
+		pci_bus_claim_resources(bus);
+	} else {
 		pci_bus_size_bridges(bus);
 		pci_bus_assign_resources(bus);
 
-- 
2.5.1

[PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Lorenzo Pieralisi]

The PCI host generic driver does not reassign bus resources on systems
that require the BARs set-up to be immutable (ie PCI_PROBE_ONLY) since
that would trigger system failures. Nonetheless, PCI bus resources
allocated to PCI bridge and devices must be claimed in order to be
validated and inserted in the kernel resource tree, but the current
driver omits the resources claiming and relies on arch specific kludges
to prevent probing failure (ie preventing resources enablement on
PCI_PROBE_ONLY systems).

This patch adds code to the PCI host generic driver that correctly
claims bus resources upon probe on systems that are required to
prevent reassignment after bus enumeration, so that the allocated
resources can be enabled successfully upon PCI device drivers probing,
without resorting to arch back-ends workarounds.

Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: David Daney <david.daney@cavium.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
---
 drivers/pci/host/pci-host-generic.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/host/pci-host-generic.c b/drivers/pci/host/pci-host-generic.c
index 1652bc7..e529825 100644
--- a/drivers/pci/host/pci-host-generic.c
+++ b/drivers/pci/host/pci-host-generic.c
@@ -252,7 +252,10 @@ static int gen_pci_probe(struct platform_device *pdev)
 
 	pci_fixup_irqs(pci_common_swizzle, of_irq_parse_and_map_pci);
 
-	if (!pci_has_flag(PCI_PROBE_ONLY)) {
+
+	if (pci_has_flag(PCI_PROBE_ONLY)) {
+		pci_bus_claim_resources(bus);
+	} else {
 		pci_bus_size_bridges(bus);
 		pci_bus_assign_resources(bus);
 
-- 
2.5.1

[PATCH v2 3/3] arm/arm64: pci: remove arch specific pcibios_enable_device()

[Lorenzo Pieralisi]

The arm/arm64 pcibios_enable_device() implementations exist solely
to prevent enabling PCI resources on PROBE_ONLY systems, since
on those systems the PCI resources are currently not claimed (ie
validated and inserted in the PCI resource tree) therefore they can
not be enabled since this would trigger PCI set-ups failures.

By introducing resources claiming in the PCI host controllers set-ups
that have PROBE_ONLY as a probe option, there is no need for arch specific
pcibios_enable_device() implementations anymore in that the kernel can
rely on the generic pcibios_enable_device() implementation without
resorting to arch specific code to work around the missing resources
claiming enumeration step.

This patch removes the pcibios_enable_device() implementations from
the arm/arm64 arches back-end.

Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Russell King <linux@arm.linux.org.uk>
---
 arch/arm/kernel/bios32.c | 12 ------------
 arch/arm64/kernel/pci.c  | 13 -------------
 2 files changed, 25 deletions(-)

diff --git a/arch/arm/kernel/bios32.c b/arch/arm/kernel/bios32.c
index 05e61a2..488545f 100644
--- a/arch/arm/kernel/bios32.c
+++ b/arch/arm/kernel/bios32.c
@@ -590,18 +590,6 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res,
 	return start;
 }
 
-/**
- * pcibios_enable_device - Enable I/O and memory.
- * @dev: PCI device to be enabled
- */
-int pcibios_enable_device(struct pci_dev *dev, int mask)
-{
-	if (pci_has_flag(PCI_PROBE_ONLY))
-		return 0;
-
-	return pci_enable_resources(dev, mask);
-}
-
 int pci_mmap_page_range(struct pci_dev *dev, struct vm_area_struct *vma,
 			enum pci_mmap_state mmap_state, int write_combine)
 {
diff --git a/arch/arm64/kernel/pci.c b/arch/arm64/kernel/pci.c
index b3d098b..4095379 100644
--- a/arch/arm64/kernel/pci.c
+++ b/arch/arm64/kernel/pci.c
@@ -38,19 +38,6 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res,
 	return res->start;
 }
 
-/**
- * pcibios_enable_device - Enable I/O and memory.
- * @dev: PCI device to be enabled
- * @mask: bitmask of BARs to enable
- */
-int pcibios_enable_device(struct pci_dev *dev, int mask)
-{
-	if (pci_has_flag(PCI_PROBE_ONLY))
-		return 0;
-
-	return pci_enable_resources(dev, mask);
-}
-
 /*
  * Try to assign the IRQ number from DT when adding a new device
  */
-- 
2.5.1

[PATCH v2 3/3] arm/arm64: pci: remove arch specific pcibios_enable_device()

[Lorenzo Pieralisi]

The arm/arm64 pcibios_enable_device() implementations exist solely
to prevent enabling PCI resources on PROBE_ONLY systems, since
on those systems the PCI resources are currently not claimed (ie
validated and inserted in the PCI resource tree) therefore they can
not be enabled since this would trigger PCI set-ups failures.

By introducing resources claiming in the PCI host controllers set-ups
that have PROBE_ONLY as a probe option, there is no need for arch specific
pcibios_enable_device() implementations anymore in that the kernel can
rely on the generic pcibios_enable_device() implementation without
resorting to arch specific code to work around the missing resources
claiming enumeration step.

This patch removes the pcibios_enable_device() implementations from
the arm/arm64 arches back-end.

Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Russell King <linux@arm.linux.org.uk>
---
 arch/arm/kernel/bios32.c | 12 ------------
 arch/arm64/kernel/pci.c  | 13 -------------
 2 files changed, 25 deletions(-)

diff --git a/arch/arm/kernel/bios32.c b/arch/arm/kernel/bios32.c
index 05e61a2..488545f 100644
--- a/arch/arm/kernel/bios32.c
+++ b/arch/arm/kernel/bios32.c
@@ -590,18 +590,6 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res,
 	return start;
 }
 
-/**
- * pcibios_enable_device - Enable I/O and memory.
- * @dev: PCI device to be enabled
- */
-int pcibios_enable_device(struct pci_dev *dev, int mask)
-{
-	if (pci_has_flag(PCI_PROBE_ONLY))
-		return 0;
-
-	return pci_enable_resources(dev, mask);
-}
-
 int pci_mmap_page_range(struct pci_dev *dev, struct vm_area_struct *vma,
 			enum pci_mmap_state mmap_state, int write_combine)
 {
diff --git a/arch/arm64/kernel/pci.c b/arch/arm64/kernel/pci.c
index b3d098b..4095379 100644
--- a/arch/arm64/kernel/pci.c
+++ b/arch/arm64/kernel/pci.c
@@ -38,19 +38,6 @@ resource_size_t pcibios_align_resource(void *data, const struct resource *res,
 	return res->start;
 }
 
-/**
- * pcibios_enable_device - Enable I/O and memory.
- * @dev: PCI device to be enabled
- * @mask: bitmask of BARs to enable
- */
-int pcibios_enable_device(struct pci_dev *dev, int mask)
-{
-	if (pci_has_flag(PCI_PROBE_ONLY))
-		return 0;
-
-	return pci_enable_resources(dev, mask);
-}
-
 /*
  * Try to assign the IRQ number from DT when adding a new device
  */
-- 
2.5.1

Re: [PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Bjorn Helgaas]

Hi Lorenzo,

On Tue, Mar 01, 2016 at 02:44:08PM +0000, Lorenzo Pieralisi wrote:
> The PCI host generic driver does not reassign bus resources on systems
> that require the BARs set-up to be immutable (ie PCI_PROBE_ONLY) since
> that would trigger system failures. Nonetheless, PCI bus resources
> allocated to PCI bridge and devices must be claimed in order to be
> validated and inserted in the kernel resource tree, but the current
> driver omits the resources claiming and relies on arch specific kludges
> to prevent probing failure (ie preventing resources enablement on
> PCI_PROBE_ONLY systems).
> 
> This patch adds code to the PCI host generic driver that correctly
> claims bus resources upon probe on systems that are required to
> prevent reassignment after bus enumeration, so that the allocated
> resources can be enabled successfully upon PCI device drivers probing,
> without resorting to arch back-ends workarounds.
> 
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Acked-by: Will Deacon <will.deacon@arm.com>
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: David Daney <david.daney@cavium.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>
> ---
>  drivers/pci/host/pci-host-generic.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/host/pci-host-generic.c b/drivers/pci/host/pci-host-generic.c
> index 1652bc7..e529825 100644
> --- a/drivers/pci/host/pci-host-generic.c
> +++ b/drivers/pci/host/pci-host-generic.c
> @@ -252,7 +252,10 @@ static int gen_pci_probe(struct platform_device *pdev)
>  
>  	pci_fixup_irqs(pci_common_swizzle, of_irq_parse_and_map_pci);
>  
> -	if (!pci_has_flag(PCI_PROBE_ONLY)) {
> +
> +	if (pci_has_flag(PCI_PROBE_ONLY)) {
> +		pci_bus_claim_resources(bus);
> +	} else {
>  		pci_bus_size_bridges(bus);
>  		pci_bus_assign_resources(bus);
>  

The next patch removes the arm and arm64 pcibios_enable_device()
implementations, which implies that arm and arm64 only need the generic
version, which simply calls pci_enable_resources().  That assumes r->parent
is set.

After this patch, we'll call pci_bus_claim_resources() for the
PCI_PROBE_ONLY case, and that sets r->parent for all the resources.

Where does r->parent get set in the non-PCI_PROBE_ONLY case?  Obviously
that path *works*, because you're not changing anything there.  I'd just
like to have a hint that makes this change more obvious.

Bjorn

[PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Bjorn Helgaas]

Hi Lorenzo,

On Tue, Mar 01, 2016 at 02:44:08PM +0000, Lorenzo Pieralisi wrote:
> The PCI host generic driver does not reassign bus resources on systems
> that require the BARs set-up to be immutable (ie PCI_PROBE_ONLY) since
> that would trigger system failures. Nonetheless, PCI bus resources
> allocated to PCI bridge and devices must be claimed in order to be
> validated and inserted in the kernel resource tree, but the current
> driver omits the resources claiming and relies on arch specific kludges
> to prevent probing failure (ie preventing resources enablement on
> PCI_PROBE_ONLY systems).
> 
> This patch adds code to the PCI host generic driver that correctly
> claims bus resources upon probe on systems that are required to
> prevent reassignment after bus enumeration, so that the allocated
> resources can be enabled successfully upon PCI device drivers probing,
> without resorting to arch back-ends workarounds.
> 
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Acked-by: Will Deacon <will.deacon@arm.com>
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: David Daney <david.daney@cavium.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>
> ---
>  drivers/pci/host/pci-host-generic.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/host/pci-host-generic.c b/drivers/pci/host/pci-host-generic.c
> index 1652bc7..e529825 100644
> --- a/drivers/pci/host/pci-host-generic.c
> +++ b/drivers/pci/host/pci-host-generic.c
> @@ -252,7 +252,10 @@ static int gen_pci_probe(struct platform_device *pdev)
>  
>  	pci_fixup_irqs(pci_common_swizzle, of_irq_parse_and_map_pci);
>  
> -	if (!pci_has_flag(PCI_PROBE_ONLY)) {
> +
> +	if (pci_has_flag(PCI_PROBE_ONLY)) {
> +		pci_bus_claim_resources(bus);
> +	} else {
>  		pci_bus_size_bridges(bus);
>  		pci_bus_assign_resources(bus);
>  

The next patch removes the arm and arm64 pcibios_enable_device()
implementations, which implies that arm and arm64 only need the generic
version, which simply calls pci_enable_resources().  That assumes r->parent
is set.

After this patch, we'll call pci_bus_claim_resources() for the
PCI_PROBE_ONLY case, and that sets r->parent for all the resources.

Where does r->parent get set in the non-PCI_PROBE_ONLY case?  Obviously
that path *works*, because you're not changing anything there.  I'd just
like to have a hint that makes this change more obvious.

Bjorn

Re: [PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Lorenzo Pieralisi]

On Mon, Apr 11, 2016 at 11:43:11PM -0500, Bjorn Helgaas wrote:
> Hi Lorenzo,
> 
> On Tue, Mar 01, 2016 at 02:44:08PM +0000, Lorenzo Pieralisi wrote:
> > The PCI host generic driver does not reassign bus resources on systems
> > that require the BARs set-up to be immutable (ie PCI_PROBE_ONLY) since
> > that would trigger system failures. Nonetheless, PCI bus resources
> > allocated to PCI bridge and devices must be claimed in order to be
> > validated and inserted in the kernel resource tree, but the current
> > driver omits the resources claiming and relies on arch specific kludges
> > to prevent probing failure (ie preventing resources enablement on
> > PCI_PROBE_ONLY systems).
> > 
> > This patch adds code to the PCI host generic driver that correctly
> > claims bus resources upon probe on systems that are required to
> > prevent reassignment after bus enumeration, so that the allocated
> > resources can be enabled successfully upon PCI device drivers probing,
> > without resorting to arch back-ends workarounds.
> > 
> > Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> > Acked-by: Will Deacon <will.deacon@arm.com>
> > Cc: Arnd Bergmann <arnd@arndb.de>
> > Cc: David Daney <david.daney@cavium.com>
> > Cc: Will Deacon <will.deacon@arm.com>
> > Cc: Bjorn Helgaas <bhelgaas@google.com>
> > ---
> >  drivers/pci/host/pci-host-generic.c | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/host/pci-host-generic.c b/drivers/pci/host/pci-host-generic.c
> > index 1652bc7..e529825 100644
> > --- a/drivers/pci/host/pci-host-generic.c
> > +++ b/drivers/pci/host/pci-host-generic.c
> > @@ -252,7 +252,10 @@ static int gen_pci_probe(struct platform_device *pdev)
> >  
> >  	pci_fixup_irqs(pci_common_swizzle, of_irq_parse_and_map_pci);
> >  
> > -	if (!pci_has_flag(PCI_PROBE_ONLY)) {
> > +
> > +	if (pci_has_flag(PCI_PROBE_ONLY)) {
> > +		pci_bus_claim_resources(bus);
> > +	} else {
> >  		pci_bus_size_bridges(bus);
> >  		pci_bus_assign_resources(bus);
> >  
> 
> The next patch removes the arm and arm64 pcibios_enable_device()
> implementations, which implies that arm and arm64 only need the generic
> version, which simply calls pci_enable_resources().  That assumes r->parent
> is set.
> 
> After this patch, we'll call pci_bus_claim_resources() for the
> PCI_PROBE_ONLY case, and that sets r->parent for all the resources.
> 
> Where does r->parent get set in the non-PCI_PROBE_ONLY case?  Obviously
> that path *works*, because you're not changing anything there.  I'd just
> like to have a hint that makes this change more obvious.

On all ARM/ARM64 PCI controllers drivers I am aware of (apart from the
kvmtool PCI host controller which does require PCI_PROBE_ONLY, so we need
this patch), resources are always reassigned and the core code reassigning
them takes care of assigning their parent pointers too, to answer your
question.

As for this patch series, given that:

commit (in -next) 903589ca7165 ("ARM: 8554/1: kernel: pci: remove
pci=firmware command line parameter handling") removes the PCI_PROBE_ONLY
handling from the (ARM) command line, the PCI host generic becomes the
last ARM/ARM64 host controller that requires PCI_PROBE_ONLY to function
(depending on DT settings).

The idea behind adding pci_bus_claim_resources (patch 1) to core code
was that it could be reused by other arches too, I do not have evidence
though, I have to prove it, so I'd rather squash patch 1 into this one
and make the code claiming resources local to the PCI host generic,
I can't add a generic PCI core API just for one host controller
(IMHO we should add an API that allows us to claim bus resources and
realloc the ones for which claiming fail - which may mean releasing
bridges resources and realloc/resize them - code is in the kernel already
I have to write that API).

The code claiming resources on x86, IA64 and PowerPC looks extremely
similar but it has to be proven that a generic function has a chance
to work, so patch 1 is not really justified at present.

If you have no objections I will squash patch 1 into this one (moving
the respective code in PCI host generic driver), and I would not merge
this series till the commit above in -next gets in the kernel (which
makes sure that PCI_PROBE_ONLY can't be set on the command line, that's
fundamental to this series, at least on ARM, on ARM64 DT is the only way
PCI_PROBE_ONLY can be set and only on host controllers that check the
chosen node property - ie PCI host generic, that we are patching).

Thanks,
Lorenzo

[PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Lorenzo Pieralisi]

On Mon, Apr 11, 2016 at 11:43:11PM -0500, Bjorn Helgaas wrote:
> Hi Lorenzo,
> 
> On Tue, Mar 01, 2016 at 02:44:08PM +0000, Lorenzo Pieralisi wrote:
> > The PCI host generic driver does not reassign bus resources on systems
> > that require the BARs set-up to be immutable (ie PCI_PROBE_ONLY) since
> > that would trigger system failures. Nonetheless, PCI bus resources
> > allocated to PCI bridge and devices must be claimed in order to be
> > validated and inserted in the kernel resource tree, but the current
> > driver omits the resources claiming and relies on arch specific kludges
> > to prevent probing failure (ie preventing resources enablement on
> > PCI_PROBE_ONLY systems).
> > 
> > This patch adds code to the PCI host generic driver that correctly
> > claims bus resources upon probe on systems that are required to
> > prevent reassignment after bus enumeration, so that the allocated
> > resources can be enabled successfully upon PCI device drivers probing,
> > without resorting to arch back-ends workarounds.
> > 
> > Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> > Acked-by: Will Deacon <will.deacon@arm.com>
> > Cc: Arnd Bergmann <arnd@arndb.de>
> > Cc: David Daney <david.daney@cavium.com>
> > Cc: Will Deacon <will.deacon@arm.com>
> > Cc: Bjorn Helgaas <bhelgaas@google.com>
> > ---
> >  drivers/pci/host/pci-host-generic.c | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/host/pci-host-generic.c b/drivers/pci/host/pci-host-generic.c
> > index 1652bc7..e529825 100644
> > --- a/drivers/pci/host/pci-host-generic.c
> > +++ b/drivers/pci/host/pci-host-generic.c
> > @@ -252,7 +252,10 @@ static int gen_pci_probe(struct platform_device *pdev)
> >  
> >  	pci_fixup_irqs(pci_common_swizzle, of_irq_parse_and_map_pci);
> >  
> > -	if (!pci_has_flag(PCI_PROBE_ONLY)) {
> > +
> > +	if (pci_has_flag(PCI_PROBE_ONLY)) {
> > +		pci_bus_claim_resources(bus);
> > +	} else {
> >  		pci_bus_size_bridges(bus);
> >  		pci_bus_assign_resources(bus);
> >  
> 
> The next patch removes the arm and arm64 pcibios_enable_device()
> implementations, which implies that arm and arm64 only need the generic
> version, which simply calls pci_enable_resources().  That assumes r->parent
> is set.
> 
> After this patch, we'll call pci_bus_claim_resources() for the
> PCI_PROBE_ONLY case, and that sets r->parent for all the resources.
> 
> Where does r->parent get set in the non-PCI_PROBE_ONLY case?  Obviously
> that path *works*, because you're not changing anything there.  I'd just
> like to have a hint that makes this change more obvious.

On all ARM/ARM64 PCI controllers drivers I am aware of (apart from the
kvmtool PCI host controller which does require PCI_PROBE_ONLY, so we need
this patch), resources are always reassigned and the core code reassigning
them takes care of assigning their parent pointers too, to answer your
question.

As for this patch series, given that:

commit (in -next) 903589ca7165 ("ARM: 8554/1: kernel: pci: remove
pci=firmware command line parameter handling") removes the PCI_PROBE_ONLY
handling from the (ARM) command line, the PCI host generic becomes the
last ARM/ARM64 host controller that requires PCI_PROBE_ONLY to function
(depending on DT settings).

The idea behind adding pci_bus_claim_resources (patch 1) to core code
was that it could be reused by other arches too, I do not have evidence
though, I have to prove it, so I'd rather squash patch 1 into this one
and make the code claiming resources local to the PCI host generic,
I can't add a generic PCI core API just for one host controller
(IMHO we should add an API that allows us to claim bus resources and
realloc the ones for which claiming fail - which may mean releasing
bridges resources and realloc/resize them - code is in the kernel already
I have to write that API).

The code claiming resources on x86, IA64 and PowerPC looks extremely
similar but it has to be proven that a generic function has a chance
to work, so patch 1 is not really justified at present.

If you have no objections I will squash patch 1 into this one (moving
the respective code in PCI host generic driver), and I would not merge
this series till the commit above in -next gets in the kernel (which
makes sure that PCI_PROBE_ONLY can't be set on the command line, that's
fundamental to this series, at least on ARM, on ARM64 DT is the only way
PCI_PROBE_ONLY can be set and only on host controllers that check the
chosen node property - ie PCI host generic, that we are patching).

Thanks,
Lorenzo

Re: [PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Bjorn Helgaas]

On Tue, Apr 12, 2016 at 04:48:10PM +0100, Lorenzo Pieralisi wrote:
> On Mon, Apr 11, 2016 at 11:43:11PM -0500, Bjorn Helgaas wrote:
> > Hi Lorenzo,
> > 
> > On Tue, Mar 01, 2016 at 02:44:08PM +0000, Lorenzo Pieralisi wrote:
> > > The PCI host generic driver does not reassign bus resources on systems
> > > that require the BARs set-up to be immutable (ie PCI_PROBE_ONLY) since
> > > that would trigger system failures. Nonetheless, PCI bus resources
> > > allocated to PCI bridge and devices must be claimed in order to be
> > > validated and inserted in the kernel resource tree, but the current
> > > driver omits the resources claiming and relies on arch specific kludges
> > > to prevent probing failure (ie preventing resources enablement on
> > > PCI_PROBE_ONLY systems).
> > > 
> > > This patch adds code to the PCI host generic driver that correctly
> > > claims bus resources upon probe on systems that are required to
> > > prevent reassignment after bus enumeration, so that the allocated
> > > resources can be enabled successfully upon PCI device drivers probing,
> > > without resorting to arch back-ends workarounds.
> > > 
> > > Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> > > Acked-by: Will Deacon <will.deacon@arm.com>
> > > Cc: Arnd Bergmann <arnd@arndb.de>
> > > Cc: David Daney <david.daney@cavium.com>
> > > Cc: Will Deacon <will.deacon@arm.com>
> > > Cc: Bjorn Helgaas <bhelgaas@google.com>
> > > ---
> > >  drivers/pci/host/pci-host-generic.c | 5 ++++-
> > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/pci/host/pci-host-generic.c b/drivers/pci/host/pci-host-generic.c
> > > index 1652bc7..e529825 100644
> > > --- a/drivers/pci/host/pci-host-generic.c
> > > +++ b/drivers/pci/host/pci-host-generic.c
> > > @@ -252,7 +252,10 @@ static int gen_pci_probe(struct platform_device *pdev)
> > >  
> > >  	pci_fixup_irqs(pci_common_swizzle, of_irq_parse_and_map_pci);
> > >  
> > > -	if (!pci_has_flag(PCI_PROBE_ONLY)) {
> > > +
> > > +	if (pci_has_flag(PCI_PROBE_ONLY)) {
> > > +		pci_bus_claim_resources(bus);
> > > +	} else {
> > >  		pci_bus_size_bridges(bus);
> > >  		pci_bus_assign_resources(bus);
> > >  
> > 
> > The next patch removes the arm and arm64 pcibios_enable_device()
> > implementations, which implies that arm and arm64 only need the generic
> > version, which simply calls pci_enable_resources().  That assumes r->parent
> > is set.
> > 
> > After this patch, we'll call pci_bus_claim_resources() for the
> > PCI_PROBE_ONLY case, and that sets r->parent for all the resources.
> > 
> > Where does r->parent get set in the non-PCI_PROBE_ONLY case?  Obviously
> > that path *works*, because you're not changing anything there.  I'd just
> > like to have a hint that makes this change more obvious.
> 
> On all ARM/ARM64 PCI controllers drivers I am aware of (apart from the
> kvmtool PCI host controller which does require PCI_PROBE_ONLY, so we need
> this patch), resources are always reassigned and the core code reassigning
> them takes care of assigning their parent pointers too, to answer your
> question.

Here's what I find confusing.  Consider these three cases:

  1) Firmware programs no BARs and we reassign everything.  We call
  pci_bus_assign_resources(), and the pci_assign_resource() ...
  allocate_resource() path makes sure everything is claimed.  This is
  apparently the normal arm/arm64 path, and it already works.

  2) Firmware programs all BARs and we set PCI_PROBE_ONLY.  After this
  series, we'll claim the resources and remove the PCI_PROBE_ONLY
  special case in pcibios_enable_device().  This is great!

  3) Firmware programs all BARs but we don't set PCI_PROBE_ONLY.  We
  call pci_bus_assign_resources(), but I think it does nothing because
  everything is already assigned.  The resources are not claimed and
  pci_enable_resources() will fail.

This last case 3) is the problem.  I'm guessing this case doesn't
currently occur on arm/arm64, but it's the normal case on x86, and it
seems perverse that things work if firmware does nothing, but they
don't work if firmware does more setup.

So I think we should add some sort of arm/arm64-specific
pci_claim_resource() path similar to the pcibios_allocate_resources()
stuff on x86.

> As for this patch series, given that:
> 
> commit (in -next) 903589ca7165 ("ARM: 8554/1: kernel: pci: remove
> pci=firmware command line parameter handling") removes the PCI_PROBE_ONLY
> handling from the (ARM) command line, the PCI host generic becomes the
> last ARM/ARM64 host controller that requires PCI_PROBE_ONLY to function
> (depending on DT settings).
> 
> The idea behind adding pci_bus_claim_resources (patch 1) to core code
> was that it could be reused by other arches too, I do not have evidence
> though, I have to prove it, so I'd rather squash patch 1 into this one
> and make the code claiming resources local to the PCI host generic,
> I can't add a generic PCI core API just for one host controller
> (IMHO we should add an API that allows us to claim bus resources and
> realloc the ones for which claiming fail - which may mean releasing
> bridges resources and realloc/resize them - code is in the kernel already
> I have to write that API).
> 
> The code claiming resources on x86, IA64 and PowerPC looks extremely
> similar but it has to be proven that a generic function has a chance
> to work, so patch 1 is not really justified at present.

I don't really object to patch 1, but you're right that it's possible
we could do a better job later.  I would certainly like to get that
sort of code (including the pcibios_allocate_resources() stuff I just
mentioned) out of the arches and into the core somehow.

> If you have no objections I will squash patch 1 into this one (moving
> the respective code in PCI host generic driver), and I would not merge
> this series till the commit above in -next gets in the kernel (which
> makes sure that PCI_PROBE_ONLY can't be set on the command line, that's
> fundamental to this series, at least on ARM, on ARM64 DT is the only way
> PCI_PROBE_ONLY can be set and only on host controllers that check the
> chosen node property - ie PCI host generic, that we are patching).

If there's a stable branch containing 903589ca7165 ("ARM: 8554/1:
kernel: pci: remove pci=firmware command line parameter handling"), I
can pull that and merge your series on top of it.

Bjorn

[PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Bjorn Helgaas]

On Tue, Apr 12, 2016 at 04:48:10PM +0100, Lorenzo Pieralisi wrote:
> On Mon, Apr 11, 2016 at 11:43:11PM -0500, Bjorn Helgaas wrote:
> > Hi Lorenzo,
> > 
> > On Tue, Mar 01, 2016 at 02:44:08PM +0000, Lorenzo Pieralisi wrote:
> > > The PCI host generic driver does not reassign bus resources on systems
> > > that require the BARs set-up to be immutable (ie PCI_PROBE_ONLY) since
> > > that would trigger system failures. Nonetheless, PCI bus resources
> > > allocated to PCI bridge and devices must be claimed in order to be
> > > validated and inserted in the kernel resource tree, but the current
> > > driver omits the resources claiming and relies on arch specific kludges
> > > to prevent probing failure (ie preventing resources enablement on
> > > PCI_PROBE_ONLY systems).
> > > 
> > > This patch adds code to the PCI host generic driver that correctly
> > > claims bus resources upon probe on systems that are required to
> > > prevent reassignment after bus enumeration, so that the allocated
> > > resources can be enabled successfully upon PCI device drivers probing,
> > > without resorting to arch back-ends workarounds.
> > > 
> > > Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> > > Acked-by: Will Deacon <will.deacon@arm.com>
> > > Cc: Arnd Bergmann <arnd@arndb.de>
> > > Cc: David Daney <david.daney@cavium.com>
> > > Cc: Will Deacon <will.deacon@arm.com>
> > > Cc: Bjorn Helgaas <bhelgaas@google.com>
> > > ---
> > >  drivers/pci/host/pci-host-generic.c | 5 ++++-
> > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/pci/host/pci-host-generic.c b/drivers/pci/host/pci-host-generic.c
> > > index 1652bc7..e529825 100644
> > > --- a/drivers/pci/host/pci-host-generic.c
> > > +++ b/drivers/pci/host/pci-host-generic.c
> > > @@ -252,7 +252,10 @@ static int gen_pci_probe(struct platform_device *pdev)
> > >  
> > >  	pci_fixup_irqs(pci_common_swizzle, of_irq_parse_and_map_pci);
> > >  
> > > -	if (!pci_has_flag(PCI_PROBE_ONLY)) {
> > > +
> > > +	if (pci_has_flag(PCI_PROBE_ONLY)) {
> > > +		pci_bus_claim_resources(bus);
> > > +	} else {
> > >  		pci_bus_size_bridges(bus);
> > >  		pci_bus_assign_resources(bus);
> > >  
> > 
> > The next patch removes the arm and arm64 pcibios_enable_device()
> > implementations, which implies that arm and arm64 only need the generic
> > version, which simply calls pci_enable_resources().  That assumes r->parent
> > is set.
> > 
> > After this patch, we'll call pci_bus_claim_resources() for the
> > PCI_PROBE_ONLY case, and that sets r->parent for all the resources.
> > 
> > Where does r->parent get set in the non-PCI_PROBE_ONLY case?  Obviously
> > that path *works*, because you're not changing anything there.  I'd just
> > like to have a hint that makes this change more obvious.
> 
> On all ARM/ARM64 PCI controllers drivers I am aware of (apart from the
> kvmtool PCI host controller which does require PCI_PROBE_ONLY, so we need
> this patch), resources are always reassigned and the core code reassigning
> them takes care of assigning their parent pointers too, to answer your
> question.

Here's what I find confusing.  Consider these three cases:

  1) Firmware programs no BARs and we reassign everything.  We call
  pci_bus_assign_resources(), and the pci_assign_resource() ...
  allocate_resource() path makes sure everything is claimed.  This is
  apparently the normal arm/arm64 path, and it already works.

  2) Firmware programs all BARs and we set PCI_PROBE_ONLY.  After this
  series, we'll claim the resources and remove the PCI_PROBE_ONLY
  special case in pcibios_enable_device().  This is great!

  3) Firmware programs all BARs but we don't set PCI_PROBE_ONLY.  We
  call pci_bus_assign_resources(), but I think it does nothing because
  everything is already assigned.  The resources are not claimed and
  pci_enable_resources() will fail.

This last case 3) is the problem.  I'm guessing this case doesn't
currently occur on arm/arm64, but it's the normal case on x86, and it
seems perverse that things work if firmware does nothing, but they
don't work if firmware does more setup.

So I think we should add some sort of arm/arm64-specific
pci_claim_resource() path similar to the pcibios_allocate_resources()
stuff on x86.

> As for this patch series, given that:
> 
> commit (in -next) 903589ca7165 ("ARM: 8554/1: kernel: pci: remove
> pci=firmware command line parameter handling") removes the PCI_PROBE_ONLY
> handling from the (ARM) command line, the PCI host generic becomes the
> last ARM/ARM64 host controller that requires PCI_PROBE_ONLY to function
> (depending on DT settings).
> 
> The idea behind adding pci_bus_claim_resources (patch 1) to core code
> was that it could be reused by other arches too, I do not have evidence
> though, I have to prove it, so I'd rather squash patch 1 into this one
> and make the code claiming resources local to the PCI host generic,
> I can't add a generic PCI core API just for one host controller
> (IMHO we should add an API that allows us to claim bus resources and
> realloc the ones for which claiming fail - which may mean releasing
> bridges resources and realloc/resize them - code is in the kernel already
> I have to write that API).
> 
> The code claiming resources on x86, IA64 and PowerPC looks extremely
> similar but it has to be proven that a generic function has a chance
> to work, so patch 1 is not really justified at present.

I don't really object to patch 1, but you're right that it's possible
we could do a better job later.  I would certainly like to get that
sort of code (including the pcibios_allocate_resources() stuff I just
mentioned) out of the arches and into the core somehow.

> If you have no objections I will squash patch 1 into this one (moving
> the respective code in PCI host generic driver), and I would not merge
> this series till the commit above in -next gets in the kernel (which
> makes sure that PCI_PROBE_ONLY can't be set on the command line, that's
> fundamental to this series, at least on ARM, on ARM64 DT is the only way
> PCI_PROBE_ONLY can be set and only on host controllers that check the
> chosen node property - ie PCI host generic, that we are patching).

If there's a stable branch containing 903589ca7165 ("ARM: 8554/1:
kernel: pci: remove pci=firmware command line parameter handling"), I
can pull that and merge your series on top of it.

Bjorn

Re: [PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Lorenzo Pieralisi]

On Fri, Apr 15, 2016 at 08:08:03AM -0500, Bjorn Helgaas wrote:
> On Tue, Apr 12, 2016 at 04:48:10PM +0100, Lorenzo Pieralisi wrote:

[...]

> > > The next patch removes the arm and arm64 pcibios_enable_device()
> > > implementations, which implies that arm and arm64 only need the generic
> > > version, which simply calls pci_enable_resources().  That assumes r->parent
> > > is set.
> > > 
> > > After this patch, we'll call pci_bus_claim_resources() for the
> > > PCI_PROBE_ONLY case, and that sets r->parent for all the resources.
> > > 
> > > Where does r->parent get set in the non-PCI_PROBE_ONLY case?  Obviously
> > > that path *works*, because you're not changing anything there.  I'd just
> > > like to have a hint that makes this change more obvious.
> > 
> > On all ARM/ARM64 PCI controllers drivers I am aware of (apart from the
> > kvmtool PCI host controller which does require PCI_PROBE_ONLY, so we need
> > this patch), resources are always reassigned and the core code reassigning
> > them takes care of assigning their parent pointers too, to answer your
> > question.
> 
> Here's what I find confusing.  Consider these three cases:
> 
>   1) Firmware programs no BARs and we reassign everything.  We call
>   pci_bus_assign_resources(), and the pci_assign_resource() ...
>   allocate_resource() path makes sure everything is claimed.  This is
>   apparently the normal arm/arm64 path, and it already works.
> 
>   2) Firmware programs all BARs and we set PCI_PROBE_ONLY.  After this
>   series, we'll claim the resources and remove the PCI_PROBE_ONLY
>   special case in pcibios_enable_device().  This is great!
> 
>   3) Firmware programs all BARs but we don't set PCI_PROBE_ONLY.  We
>   call pci_bus_assign_resources(), but I think it does nothing because
>   everything is already assigned.  The resources are not claimed and
>   pci_enable_resources() will fail.

I do not expect (1) and (3) to be different from a kernel resource
allocation perspective.

If the core resource layer is asked to assign resources it will,
regardless of what FW programmed in the BARs (the BAR regions size
matters, that's it), I went through pci_bus_assign_resources() a couple
of times and I have to add a bit of debugging so give me the benefit of
the doubt please, but there is nothing that let me think it won't assign
resources (and therefore assign a parent pointer) if the resources are
already programmed correctly (actually I even think the kernel may
change what FW programmed according to its resource alloc policy).

> This last case 3) is the problem.  I'm guessing this case doesn't
> currently occur on arm/arm64, but it's the normal case on x86, and it
> seems perverse that things work if firmware does nothing, but they
> don't work if firmware does more setup.

IIUC X86 claim resources as programmed by FW so it is not really the
same situation as arm64, that claims nothing. Claimed resources are not
reassigned, they are skipped by resource allocation/sizing code
(because their parent pointer is set).

And as I said above even if FW does some set-up that will still work
on ARM/ARM64, otherwise this means that on ALL ARM/ARM64 systems out there
PCI set-up at kernel handover is non-existent, otherwise we would
have resource enablement failures NOW, right ?

> So I think we should add some sort of arm/arm64-specific
> pci_claim_resource() path similar to the pcibios_allocate_resources()
> stuff on x86.

I agree but it has to be done with care in particular in relation to
bridges apertures, claiming resources on systems that currently reassign
everything may trigger regressions.

In particular, having an API that claims resources is NOT sufficient
for that purpose for two reasons:

- If resource claiming fails for some resources, currently the kernel
  spits too much noise in the logs, claiming should be a best-effort
  approach, if it fails there is no error to report in the logs, if FW
  left resources unset that's *not* an error AFAIK
- Bridge apertures: care must be taken so that resources downstream
  should be sized so that bridges can accomodate them, this may require
  releasing previously claimed bridge apertures and realloc them

The approach should be:

(1) Claim resources
(2) Realloc whatever fails (which may imply releasing bridges resources
    previously claimed)

> > As for this patch series, given that:
> > 
> > commit (in -next) 903589ca7165 ("ARM: 8554/1: kernel: pci: remove
> > pci=firmware command line parameter handling") removes the PCI_PROBE_ONLY
> > handling from the (ARM) command line, the PCI host generic becomes the
> > last ARM/ARM64 host controller that requires PCI_PROBE_ONLY to function
> > (depending on DT settings).
> > 
> > The idea behind adding pci_bus_claim_resources (patch 1) to core code
> > was that it could be reused by other arches too, I do not have evidence
> > though, I have to prove it, so I'd rather squash patch 1 into this one
> > and make the code claiming resources local to the PCI host generic,
> > I can't add a generic PCI core API just for one host controller
> > (IMHO we should add an API that allows us to claim bus resources and
> > realloc the ones for which claiming fail - which may mean releasing
> > bridges resources and realloc/resize them - code is in the kernel already
> > I have to write that API).
> > 
> > The code claiming resources on x86, IA64 and PowerPC looks extremely
> > similar but it has to be proven that a generic function has a chance
> > to work, so patch 1 is not really justified at present.
> 
> I don't really object to patch 1, but you're right that it's possible
> we could do a better job later.  I would certainly like to get that
> sort of code (including the pcibios_allocate_resources() stuff I just
> mentioned) out of the arches and into the core somehow.

Problem with patch 1, is that, if resource claiming fails, it spits
loads of noise in kernel logs (actually it is pci_claim_resource(), that
patch 1 uses, that spits errors on claiming failure), that's a detail but
it will make people unhappy.

I do not think that if claiming for a particular resource fails we can
consider it an error that's why I see patch 1 more PCI host generic
PCI_PROBE_ONLY specific rather than anything else, we can't use
it for !PCI_PROBE_ONLY systems unless I rework it to remove the
kernel error logs on resource claiming failures.

Probably on X86/IA64 resource claiming is spotless because on most
if not all systems FW programmes the PCI bus and the kernel can take
the set-up as-is, it is a question since I have no visibility into that.

> > If you have no objections I will squash patch 1 into this one (moving
> > the respective code in PCI host generic driver), and I would not merge
> > this series till the commit above in -next gets in the kernel (which
> > makes sure that PCI_PROBE_ONLY can't be set on the command line, that's
> > fundamental to this series, at least on ARM, on ARM64 DT is the only way
> > PCI_PROBE_ONLY can be set and only on host controllers that check the
> > chosen node property - ie PCI host generic, that we are patching).
> 
> If there's a stable branch containing 903589ca7165 ("ARM: 8554/1:
> kernel: pci: remove pci=firmware command line parameter handling"), I
> can pull that and merge your series on top of it.

I think I should at least split patch 3 in two (one for ARM and one for
ARM64), if Russell is forced to revert the commit above (because
we discover pci=firmware command line users) I do not want ARM64 to
be affected.

Thanks !
Lorenzo

[PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Lorenzo Pieralisi]

On Fri, Apr 15, 2016 at 08:08:03AM -0500, Bjorn Helgaas wrote:
> On Tue, Apr 12, 2016 at 04:48:10PM +0100, Lorenzo Pieralisi wrote:

[...]

> > > The next patch removes the arm and arm64 pcibios_enable_device()
> > > implementations, which implies that arm and arm64 only need the generic
> > > version, which simply calls pci_enable_resources().  That assumes r->parent
> > > is set.
> > > 
> > > After this patch, we'll call pci_bus_claim_resources() for the
> > > PCI_PROBE_ONLY case, and that sets r->parent for all the resources.
> > > 
> > > Where does r->parent get set in the non-PCI_PROBE_ONLY case?  Obviously
> > > that path *works*, because you're not changing anything there.  I'd just
> > > like to have a hint that makes this change more obvious.
> > 
> > On all ARM/ARM64 PCI controllers drivers I am aware of (apart from the
> > kvmtool PCI host controller which does require PCI_PROBE_ONLY, so we need
> > this patch), resources are always reassigned and the core code reassigning
> > them takes care of assigning their parent pointers too, to answer your
> > question.
> 
> Here's what I find confusing.  Consider these three cases:
> 
>   1) Firmware programs no BARs and we reassign everything.  We call
>   pci_bus_assign_resources(), and the pci_assign_resource() ...
>   allocate_resource() path makes sure everything is claimed.  This is
>   apparently the normal arm/arm64 path, and it already works.
> 
>   2) Firmware programs all BARs and we set PCI_PROBE_ONLY.  After this
>   series, we'll claim the resources and remove the PCI_PROBE_ONLY
>   special case in pcibios_enable_device().  This is great!
> 
>   3) Firmware programs all BARs but we don't set PCI_PROBE_ONLY.  We
>   call pci_bus_assign_resources(), but I think it does nothing because
>   everything is already assigned.  The resources are not claimed and
>   pci_enable_resources() will fail.

I do not expect (1) and (3) to be different from a kernel resource
allocation perspective.

If the core resource layer is asked to assign resources it will,
regardless of what FW programmed in the BARs (the BAR regions size
matters, that's it), I went through pci_bus_assign_resources() a couple
of times and I have to add a bit of debugging so give me the benefit of
the doubt please, but there is nothing that let me think it won't assign
resources (and therefore assign a parent pointer) if the resources are
already programmed correctly (actually I even think the kernel may
change what FW programmed according to its resource alloc policy).

> This last case 3) is the problem.  I'm guessing this case doesn't
> currently occur on arm/arm64, but it's the normal case on x86, and it
> seems perverse that things work if firmware does nothing, but they
> don't work if firmware does more setup.

IIUC X86 claim resources as programmed by FW so it is not really the
same situation as arm64, that claims nothing. Claimed resources are not
reassigned, they are skipped by resource allocation/sizing code
(because their parent pointer is set).

And as I said above even if FW does some set-up that will still work
on ARM/ARM64, otherwise this means that on ALL ARM/ARM64 systems out there
PCI set-up at kernel handover is non-existent, otherwise we would
have resource enablement failures NOW, right ?

> So I think we should add some sort of arm/arm64-specific
> pci_claim_resource() path similar to the pcibios_allocate_resources()
> stuff on x86.

I agree but it has to be done with care in particular in relation to
bridges apertures, claiming resources on systems that currently reassign
everything may trigger regressions.

In particular, having an API that claims resources is NOT sufficient
for that purpose for two reasons:

- If resource claiming fails for some resources, currently the kernel
  spits too much noise in the logs, claiming should be a best-effort
  approach, if it fails there is no error to report in the logs, if FW
  left resources unset that's *not* an error AFAIK
- Bridge apertures: care must be taken so that resources downstream
  should be sized so that bridges can accomodate them, this may require
  releasing previously claimed bridge apertures and realloc them

The approach should be:

(1) Claim resources
(2) Realloc whatever fails (which may imply releasing bridges resources
    previously claimed)

> > As for this patch series, given that:
> > 
> > commit (in -next) 903589ca7165 ("ARM: 8554/1: kernel: pci: remove
> > pci=firmware command line parameter handling") removes the PCI_PROBE_ONLY
> > handling from the (ARM) command line, the PCI host generic becomes the
> > last ARM/ARM64 host controller that requires PCI_PROBE_ONLY to function
> > (depending on DT settings).
> > 
> > The idea behind adding pci_bus_claim_resources (patch 1) to core code
> > was that it could be reused by other arches too, I do not have evidence
> > though, I have to prove it, so I'd rather squash patch 1 into this one
> > and make the code claiming resources local to the PCI host generic,
> > I can't add a generic PCI core API just for one host controller
> > (IMHO we should add an API that allows us to claim bus resources and
> > realloc the ones for which claiming fail - which may mean releasing
> > bridges resources and realloc/resize them - code is in the kernel already
> > I have to write that API).
> > 
> > The code claiming resources on x86, IA64 and PowerPC looks extremely
> > similar but it has to be proven that a generic function has a chance
> > to work, so patch 1 is not really justified at present.
> 
> I don't really object to patch 1, but you're right that it's possible
> we could do a better job later.  I would certainly like to get that
> sort of code (including the pcibios_allocate_resources() stuff I just
> mentioned) out of the arches and into the core somehow.

Problem with patch 1, is that, if resource claiming fails, it spits
loads of noise in kernel logs (actually it is pci_claim_resource(), that
patch 1 uses, that spits errors on claiming failure), that's a detail but
it will make people unhappy.

I do not think that if claiming for a particular resource fails we can
consider it an error that's why I see patch 1 more PCI host generic
PCI_PROBE_ONLY specific rather than anything else, we can't use
it for !PCI_PROBE_ONLY systems unless I rework it to remove the
kernel error logs on resource claiming failures.

Probably on X86/IA64 resource claiming is spotless because on most
if not all systems FW programmes the PCI bus and the kernel can take
the set-up as-is, it is a question since I have no visibility into that.

> > If you have no objections I will squash patch 1 into this one (moving
> > the respective code in PCI host generic driver), and I would not merge
> > this series till the commit above in -next gets in the kernel (which
> > makes sure that PCI_PROBE_ONLY can't be set on the command line, that's
> > fundamental to this series, at least on ARM, on ARM64 DT is the only way
> > PCI_PROBE_ONLY can be set and only on host controllers that check the
> > chosen node property - ie PCI host generic, that we are patching).
> 
> If there's a stable branch containing 903589ca7165 ("ARM: 8554/1:
> kernel: pci: remove pci=firmware command line parameter handling"), I
> can pull that and merge your series on top of it.

I think I should at least split patch 3 in two (one for ARM and one for
ARM64), if Russell is forced to revert the commit above (because
we discover pci=firmware command line users) I do not want ARM64 to
be affected.

Thanks !
Lorenzo

Re: [PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Arnd Bergmann]

On Monday 18 April 2016 11:01:54 Lorenzo Pieralisi wrote:
> On Fri, Apr 15, 2016 at 08:08:03AM -0500, Bjorn Helgaas wrote:
> > On Tue, Apr 12, 2016 at 04:48:10PM +0100, Lorenzo Pieralisi wrote:

> > This last case 3) is the problem.  I'm guessing this case doesn't
> > currently occur on arm/arm64, but it's the normal case on x86, and it
> > seems perverse that things work if firmware does nothing, but they
> > don't work if firmware does more setup.
> 
> IIUC X86 claim resources as programmed by FW so it is not really the
> same situation as arm64, that claims nothing. Claimed resources are not
> reassigned, they are skipped by resource allocation/sizing code
> (because their parent pointer is set).
> 
> And as I said above even if FW does some set-up that will still work
> on ARM/ARM64, otherwise this means that on ALL ARM/ARM64 systems out there
> PCI set-up at kernel handover is non-existent, otherwise we would
> have resource enablement failures NOW, right ?

The embedded systems (in which I would count all arm32 machines) tend
to not do proper bus probing in their bootloaders, so we have to do it
ourselves in the kernel.

For server systems (all UEFI based ones), I'd argue that we should
rely on the firmware to do it just like we do on x86, possibly with
a blacklist of known-broken machines on which we have to do it
manually as well. Once ACPI spreads, we will likely see an increasing
number of machines on which we must not reassign the resources or
bad things happen to stuff that is owned by the BIOS.

	Arnd

[PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Arnd Bergmann]

On Monday 18 April 2016 11:01:54 Lorenzo Pieralisi wrote:
> On Fri, Apr 15, 2016 at 08:08:03AM -0500, Bjorn Helgaas wrote:
> > On Tue, Apr 12, 2016 at 04:48:10PM +0100, Lorenzo Pieralisi wrote:

> > This last case 3) is the problem.  I'm guessing this case doesn't
> > currently occur on arm/arm64, but it's the normal case on x86, and it
> > seems perverse that things work if firmware does nothing, but they
> > don't work if firmware does more setup.
> 
> IIUC X86 claim resources as programmed by FW so it is not really the
> same situation as arm64, that claims nothing. Claimed resources are not
> reassigned, they are skipped by resource allocation/sizing code
> (because their parent pointer is set).
> 
> And as I said above even if FW does some set-up that will still work
> on ARM/ARM64, otherwise this means that on ALL ARM/ARM64 systems out there
> PCI set-up at kernel handover is non-existent, otherwise we would
> have resource enablement failures NOW, right ?

The embedded systems (in which I would count all arm32 machines) tend
to not do proper bus probing in their bootloaders, so we have to do it
ourselves in the kernel.

For server systems (all UEFI based ones), I'd argue that we should
rely on the firmware to do it just like we do on x86, possibly with
a blacklist of known-broken machines on which we have to do it
manually as well. Once ACPI spreads, we will likely see an increasing
number of machines on which we must not reassign the resources or
bad things happen to stuff that is owned by the BIOS.

	Arnd

Re: [PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Lorenzo Pieralisi]

On Mon, Apr 18, 2016 at 04:49:43PM +0200, Arnd Bergmann wrote:
> On Monday 18 April 2016 11:01:54 Lorenzo Pieralisi wrote:
> > On Fri, Apr 15, 2016 at 08:08:03AM -0500, Bjorn Helgaas wrote:
> > > On Tue, Apr 12, 2016 at 04:48:10PM +0100, Lorenzo Pieralisi wrote:
> 
> > > This last case 3) is the problem.  I'm guessing this case doesn't
> > > currently occur on arm/arm64, but it's the normal case on x86, and it
> > > seems perverse that things work if firmware does nothing, but they
> > > don't work if firmware does more setup.
> > 
> > IIUC X86 claim resources as programmed by FW so it is not really the
> > same situation as arm64, that claims nothing. Claimed resources are not
> > reassigned, they are skipped by resource allocation/sizing code
> > (because their parent pointer is set).
> > 
> > And as I said above even if FW does some set-up that will still work
> > on ARM/ARM64, otherwise this means that on ALL ARM/ARM64 systems out there
> > PCI set-up at kernel handover is non-existent, otherwise we would
> > have resource enablement failures NOW, right ?
> 
> The embedded systems (in which I would count all arm32 machines) tend
> to not do proper bus probing in their bootloaders, so we have to do it
> ourselves in the kernel.
> 
> For server systems (all UEFI based ones), I'd argue that we should
> rely on the firmware to do it just like we do on x86, possibly with
> a blacklist of known-broken machines on which we have to do it
> manually as well. Once ACPI spreads, we will likely see an increasing
> number of machines on which we must not reassign the resources or
> bad things happen to stuff that is owned by the BIOS.

The only way I can pull that off, is by writing an ARM64 PCI resource
allocation function that does the following:

- Try to claim the FW set-up
- Realloc on claiming failures, inclusive of bridges resources
  releasing/resizing

When to call it it has to be seen, either I do it on all ARM64 machines
(but this requires significant testing because regressions are more
than likely given that there are platforms on which we reassign everything
already) or on !acpi_disabled (but I think that's wrong because I do not
see why it is *only* dependent on ACPI), the sooner we implement it the
better (and actually that's the reason why I wanted this function to
be in the ACPI host controller code for ARM64 from the beginning - but
if we do it at arch level it can be even more generic - again, when
to call it it must be decided).

Lorenzo

[PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Lorenzo Pieralisi]

On Mon, Apr 18, 2016 at 04:49:43PM +0200, Arnd Bergmann wrote:
> On Monday 18 April 2016 11:01:54 Lorenzo Pieralisi wrote:
> > On Fri, Apr 15, 2016 at 08:08:03AM -0500, Bjorn Helgaas wrote:
> > > On Tue, Apr 12, 2016 at 04:48:10PM +0100, Lorenzo Pieralisi wrote:
> 
> > > This last case 3) is the problem.  I'm guessing this case doesn't
> > > currently occur on arm/arm64, but it's the normal case on x86, and it
> > > seems perverse that things work if firmware does nothing, but they
> > > don't work if firmware does more setup.
> > 
> > IIUC X86 claim resources as programmed by FW so it is not really the
> > same situation as arm64, that claims nothing. Claimed resources are not
> > reassigned, they are skipped by resource allocation/sizing code
> > (because their parent pointer is set).
> > 
> > And as I said above even if FW does some set-up that will still work
> > on ARM/ARM64, otherwise this means that on ALL ARM/ARM64 systems out there
> > PCI set-up at kernel handover is non-existent, otherwise we would
> > have resource enablement failures NOW, right ?
> 
> The embedded systems (in which I would count all arm32 machines) tend
> to not do proper bus probing in their bootloaders, so we have to do it
> ourselves in the kernel.
> 
> For server systems (all UEFI based ones), I'd argue that we should
> rely on the firmware to do it just like we do on x86, possibly with
> a blacklist of known-broken machines on which we have to do it
> manually as well. Once ACPI spreads, we will likely see an increasing
> number of machines on which we must not reassign the resources or
> bad things happen to stuff that is owned by the BIOS.

The only way I can pull that off, is by writing an ARM64 PCI resource
allocation function that does the following:

- Try to claim the FW set-up
- Realloc on claiming failures, inclusive of bridges resources
  releasing/resizing

When to call it it has to be seen, either I do it on all ARM64 machines
(but this requires significant testing because regressions are more
than likely given that there are platforms on which we reassign everything
already) or on !acpi_disabled (but I think that's wrong because I do not
see why it is *only* dependent on ACPI), the sooner we implement it the
better (and actually that's the reason why I wanted this function to
be in the ACPI host controller code for ARM64 from the beginning - but
if we do it at arch level it can be even more generic - again, when
to call it it must be decided).

Lorenzo

Re: [PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Bjorn Helgaas]

On Mon, Apr 18, 2016 at 11:01:54AM +0100, Lorenzo Pieralisi wrote:
> On Fri, Apr 15, 2016 at 08:08:03AM -0500, Bjorn Helgaas wrote:
> > On Tue, Apr 12, 2016 at 04:48:10PM +0100, Lorenzo Pieralisi wrote:
> 
> [...]
> 
> > > > The next patch removes the arm and arm64 pcibios_enable_device()
> > > > implementations, which implies that arm and arm64 only need the generic
> > > > version, which simply calls pci_enable_resources().  That assumes r->parent
> > > > is set.
> > > > 
> > > > After this patch, we'll call pci_bus_claim_resources() for the
> > > > PCI_PROBE_ONLY case, and that sets r->parent for all the resources.
> > > > 
> > > > Where does r->parent get set in the non-PCI_PROBE_ONLY case?  Obviously
> > > > that path *works*, because you're not changing anything there.  I'd just
> > > > like to have a hint that makes this change more obvious.
> > > 
> > > On all ARM/ARM64 PCI controllers drivers I am aware of (apart from the
> > > kvmtool PCI host controller which does require PCI_PROBE_ONLY, so we need
> > > this patch), resources are always reassigned and the core code reassigning
> > > them takes care of assigning their parent pointers too, to answer your
> > > question.
> > 
> > Here's what I find confusing.  Consider these three cases:
> > 
> >   1) Firmware programs no BARs and we reassign everything.  We call
> >   pci_bus_assign_resources(), and the pci_assign_resource() ...
> >   allocate_resource() path makes sure everything is claimed.  This is
> >   apparently the normal arm/arm64 path, and it already works.
> > 
> >   2) Firmware programs all BARs and we set PCI_PROBE_ONLY.  After this
> >   series, we'll claim the resources and remove the PCI_PROBE_ONLY
> >   special case in pcibios_enable_device().  This is great!
> > 
> >   3) Firmware programs all BARs but we don't set PCI_PROBE_ONLY.  We
> >   call pci_bus_assign_resources(), but I think it does nothing because
> >   everything is already assigned.  The resources are not claimed and
> >   pci_enable_resources() will fail.
> 
> I do not expect (1) and (3) to be different from a kernel resource
> allocation perspective.
> 
> If the core resource layer is asked to assign resources it will,
> regardless of what FW programmed in the BARs (the BAR regions size
> matters, that's it), I went through pci_bus_assign_resources() a couple
> of times and I have to add a bit of debugging so give me the benefit of
> the doubt please, but there is nothing that let me think it won't assign
> resources (and therefore assign a parent pointer) if the resources are
> already programmed correctly (actually I even think the kernel may
> change what FW programmed according to its resource alloc policy).

OK.  If you're saying that even if FW programmed the BARs, the core
will assign resources and set r->parent, that's all I'm looking for.
I *would* like a comment where we test PCI_PROBE_ONLY to the effect
that for PCI_PROBE_ONLY we call pci_bus_claim_resources(), and for
!PCI_PROBE_ONLY, we claim the resources in pci_bus_assign_resources().

Bjorn

[PATCH v2 2/3] drivers: pci: host-generic: claim bus resources on PCI_PROBE_ONLY set-ups

[Bjorn Helgaas]

On Mon, Apr 18, 2016 at 11:01:54AM +0100, Lorenzo Pieralisi wrote:
> On Fri, Apr 15, 2016 at 08:08:03AM -0500, Bjorn Helgaas wrote:
> > On Tue, Apr 12, 2016 at 04:48:10PM +0100, Lorenzo Pieralisi wrote:
> 
> [...]
> 
> > > > The next patch removes the arm and arm64 pcibios_enable_device()
> > > > implementations, which implies that arm and arm64 only need the generic
> > > > version, which simply calls pci_enable_resources().  That assumes r->parent
> > > > is set.
> > > > 
> > > > After this patch, we'll call pci_bus_claim_resources() for the
> > > > PCI_PROBE_ONLY case, and that sets r->parent for all the resources.
> > > > 
> > > > Where does r->parent get set in the non-PCI_PROBE_ONLY case?  Obviously
> > > > that path *works*, because you're not changing anything there.  I'd just
> > > > like to have a hint that makes this change more obvious.
> > > 
> > > On all ARM/ARM64 PCI controllers drivers I am aware of (apart from the
> > > kvmtool PCI host controller which does require PCI_PROBE_ONLY, so we need
> > > this patch), resources are always reassigned and the core code reassigning
> > > them takes care of assigning their parent pointers too, to answer your
> > > question.
> > 
> > Here's what I find confusing.  Consider these three cases:
> > 
> >   1) Firmware programs no BARs and we reassign everything.  We call
> >   pci_bus_assign_resources(), and the pci_assign_resource() ...
> >   allocate_resource() path makes sure everything is claimed.  This is
> >   apparently the normal arm/arm64 path, and it already works.
> > 
> >   2) Firmware programs all BARs and we set PCI_PROBE_ONLY.  After this
> >   series, we'll claim the resources and remove the PCI_PROBE_ONLY
> >   special case in pcibios_enable_device().  This is great!
> > 
> >   3) Firmware programs all BARs but we don't set PCI_PROBE_ONLY.  We
> >   call pci_bus_assign_resources(), but I think it does nothing because
> >   everything is already assigned.  The resources are not claimed and
> >   pci_enable_resources() will fail.
> 
> I do not expect (1) and (3) to be different from a kernel resource
> allocation perspective.
> 
> If the core resource layer is asked to assign resources it will,
> regardless of what FW programmed in the BARs (the BAR regions size
> matters, that's it), I went through pci_bus_assign_resources() a couple
> of times and I have to add a bit of debugging so give me the benefit of
> the doubt please, but there is nothing that let me think it won't assign
> resources (and therefore assign a parent pointer) if the resources are
> already programmed correctly (actually I even think the kernel may
> change what FW programmed according to its resource alloc policy).

OK.  If you're saying that even if FW programmed the BARs, the core
will assign resources and set r->parent, that's all I'm looking for.
I *would* like a comment where we test PCI_PROBE_ONLY to the effect
that for PCI_PROBE_ONLY we call pci_bus_claim_resources(), and for
!PCI_PROBE_ONLY, we claim the resources in pci_bus_assign_resources().

Bjorn

Re: [PATCH v2 1/3] drivers: pci: add generic code to claim bus resources

[Lorenzo Pieralisi]

[Replying to self]

On Tue, Mar 01, 2016 at 02:44:07PM +0000, Lorenzo Pieralisi wrote:
> From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
> 
> PCI core code contains a set of functions, eg:
> 
> pci_assign_unassigned_bus_resources()
> 
> that allow to assign the PCI resources for a given bus after
> enumeration.
> 
> On systems where the PCI BARs are immutable (ie they must not and can
> not be assigned), PCI bus resources should still be inserted in the PCI
> resources tree (even if they are not (re)-assigned), but there is
> no generic PCI kernel function for that purpose. Currently the PCI bus
> resources insertion in the resources tree is implemented in an arch
> specific fashion through arch specific callbacks that rely on the PCI
> resources claiming API (for bridges and devices) which resulted in arches
> implementations that contain duplicated code to claim resources for a
> given PCI bus hierarchy.
> 
> This patch, based on the x86/ia64 resources claiming arch implementations,
> implements a set of functions in core PCI code that provides a PCI core
> interface for resources claiming for a given PCI bus hierarchy, paving
> the way for further resource claiming consolidation across architectures.
> 
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: Bjorn Helgaas <bhelgaas@google.com>
> Cc: Yinghai Lu <yinghai@kernel.org>
> ---
>  drivers/pci/setup-bus.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++
>  include/linux/pci.h     |  1 +
>  2 files changed, 69 insertions(+)

I have v3 ready, which does not change this patch, however I have
some questions below before asking to merge it.

> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index 7796d0a..95f0906 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -1424,6 +1424,74 @@ void pci_bus_assign_resources(const struct pci_bus *bus)
>  }
>  EXPORT_SYMBOL(pci_bus_assign_resources);
>  
> +static void pci_claim_device_resources(struct pci_dev *dev)
> +{
> +	int i;
> +
> +	for (i = 0; i < PCI_BRIDGE_RESOURCES; i++) {
> +		struct resource *r = &dev->resource[i];
> +
> +		if (!r->flags || r->parent)
> +			continue;
> +
> +		pci_claim_resource(dev, i);

pci_claim_resource() (and in particular pci_find_parent_resource()) fails
if resource.start == 0x0. Is this intentional ? The resource.start == 0x0
check is used in other bits of the kernel to detect an "invalid"
resource, I am not sure I understand its implications correctly.

> +	}
> +}
> +
> +static void pci_claim_bridge_resources(struct pci_dev *dev)
> +{
> +	int i;
> +
> +	for (i = PCI_BRIDGE_RESOURCES; i < PCI_NUM_RESOURCES; i++) {
> +		struct resource *r = &dev->resource[i];
> +
> +		if (!r->flags || r->parent)
> +			continue;
> +
> +		pci_claim_bridge_resource(dev, i);
> +	}
> +}
> +
> +static void pci_bus_allocate_dev_resources(struct pci_bus *b)
> +{
> +	struct pci_dev *dev;
> +	struct pci_bus *child;
> +
> +	list_for_each_entry(dev, &b->devices, bus_list) {
> +		pci_claim_device_resources(dev);
> +
> +		child = dev->subordinate;
> +		if (child)
> +			pci_bus_allocate_dev_resources(child);
> +	}
> +}
> +
> +static void pci_bus_allocate_resources(struct pci_bus *b)
> +{
> +	struct pci_bus *child;
> +
> +	/*
> +	 * Carry out a depth-first search on the PCI bus
> +	 * tree to allocate bridge apertures. Read the
> +	 * programmed bridge bases and recursively claim
> +	 * the respective bridge resources.
> +	 */
> +	if (b->self) {

Is it *always* correct to check b->self to detect a device backing
the PCI bridge ? All PCI bridges should have their ->self pointer
set-up correctly, I wanted to countercheck anyway.

Comments welcome, thanks.

Lorenzo

> +		pci_read_bridge_bases(b);
> +		pci_claim_bridge_resources(b->self);
> +	}
> +
> +	list_for_each_entry(child, &b->children, node)
> +		pci_bus_allocate_resources(child);
> +}
> +
> +void pci_bus_claim_resources(struct pci_bus *b)
> +{
> +	pci_bus_allocate_resources(b);
> +	pci_bus_allocate_dev_resources(b);
> +}
> +EXPORT_SYMBOL(pci_bus_claim_resources);
> +
>  static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
>  					  struct list_head *add_head,
>  					  struct list_head *fail_head)
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 2771625..e66eba7 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1102,6 +1102,7 @@ ssize_t pci_write_vpd(struct pci_dev *dev, loff_t pos, size_t count, const void
>  /* Helper functions for low-level code (drivers/pci/setup-[bus,res].c) */
>  resource_size_t pcibios_retrieve_fw_addr(struct pci_dev *dev, int idx);
>  void pci_bus_assign_resources(const struct pci_bus *bus);
> +void pci_bus_claim_resources(struct pci_bus *bus);
>  void pci_bus_size_bridges(struct pci_bus *bus);
>  int pci_claim_resource(struct pci_dev *, int);
>  int pci_claim_bridge_resource(struct pci_dev *bridge, int i);
> -- 
> 2.5.1
> 

[PATCH v2 1/3] drivers: pci: add generic code to claim bus resources

[Lorenzo Pieralisi]

[Replying to self]

On Tue, Mar 01, 2016 at 02:44:07PM +0000, Lorenzo Pieralisi wrote:
> From: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
> 
> PCI core code contains a set of functions, eg:
> 
> pci_assign_unassigned_bus_resources()
> 
> that allow to assign the PCI resources for a given bus after
> enumeration.
> 
> On systems where the PCI BARs are immutable (ie they must not and can
> not be assigned), PCI bus resources should still be inserted in the PCI
> resources tree (even if they are not (re)-assigned), but there is
> no generic PCI kernel function for that purpose. Currently the PCI bus
> resources insertion in the resources tree is implemented in an arch
> specific fashion through arch specific callbacks that rely on the PCI
> resources claiming API (for bridges and devices) which resulted in arches
> implementations that contain duplicated code to claim resources for a
> given PCI bus hierarchy.
> 
> This patch, based on the x86/ia64 resources claiming arch implementations,
> implements a set of functions in core PCI code that provides a PCI core
> interface for resources claiming for a given PCI bus hierarchy, paving
> the way for further resource claiming consolidation across architectures.
> 
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: Bjorn Helgaas <bhelgaas@google.com>
> Cc: Yinghai Lu <yinghai@kernel.org>
> ---
>  drivers/pci/setup-bus.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++
>  include/linux/pci.h     |  1 +
>  2 files changed, 69 insertions(+)

I have v3 ready, which does not change this patch, however I have
some questions below before asking to merge it.

> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index 7796d0a..95f0906 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -1424,6 +1424,74 @@ void pci_bus_assign_resources(const struct pci_bus *bus)
>  }
>  EXPORT_SYMBOL(pci_bus_assign_resources);
>  
> +static void pci_claim_device_resources(struct pci_dev *dev)
> +{
> +	int i;
> +
> +	for (i = 0; i < PCI_BRIDGE_RESOURCES; i++) {
> +		struct resource *r = &dev->resource[i];
> +
> +		if (!r->flags || r->parent)
> +			continue;
> +
> +		pci_claim_resource(dev, i);

pci_claim_resource() (and in particular pci_find_parent_resource()) fails
if resource.start == 0x0. Is this intentional ? The resource.start == 0x0
check is used in other bits of the kernel to detect an "invalid"
resource, I am not sure I understand its implications correctly.

> +	}
> +}
> +
> +static void pci_claim_bridge_resources(struct pci_dev *dev)
> +{
> +	int i;
> +
> +	for (i = PCI_BRIDGE_RESOURCES; i < PCI_NUM_RESOURCES; i++) {
> +		struct resource *r = &dev->resource[i];
> +
> +		if (!r->flags || r->parent)
> +			continue;
> +
> +		pci_claim_bridge_resource(dev, i);
> +	}
> +}
> +
> +static void pci_bus_allocate_dev_resources(struct pci_bus *b)
> +{
> +	struct pci_dev *dev;
> +	struct pci_bus *child;
> +
> +	list_for_each_entry(dev, &b->devices, bus_list) {
> +		pci_claim_device_resources(dev);
> +
> +		child = dev->subordinate;
> +		if (child)
> +			pci_bus_allocate_dev_resources(child);
> +	}
> +}
> +
> +static void pci_bus_allocate_resources(struct pci_bus *b)
> +{
> +	struct pci_bus *child;
> +
> +	/*
> +	 * Carry out a depth-first search on the PCI bus
> +	 * tree to allocate bridge apertures. Read the
> +	 * programmed bridge bases and recursively claim
> +	 * the respective bridge resources.
> +	 */
> +	if (b->self) {

Is it *always* correct to check b->self to detect a device backing
the PCI bridge ? All PCI bridges should have their ->self pointer
set-up correctly, I wanted to countercheck anyway.

Comments welcome, thanks.

Lorenzo

> +		pci_read_bridge_bases(b);
> +		pci_claim_bridge_resources(b->self);
> +	}
> +
> +	list_for_each_entry(child, &b->children, node)
> +		pci_bus_allocate_resources(child);
> +}
> +
> +void pci_bus_claim_resources(struct pci_bus *b)
> +{
> +	pci_bus_allocate_resources(b);
> +	pci_bus_allocate_dev_resources(b);
> +}
> +EXPORT_SYMBOL(pci_bus_claim_resources);
> +
>  static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
>  					  struct list_head *add_head,
>  					  struct list_head *fail_head)
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 2771625..e66eba7 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1102,6 +1102,7 @@ ssize_t pci_write_vpd(struct pci_dev *dev, loff_t pos, size_t count, const void
>  /* Helper functions for low-level code (drivers/pci/setup-[bus,res].c) */
>  resource_size_t pcibios_retrieve_fw_addr(struct pci_dev *dev, int idx);
>  void pci_bus_assign_resources(const struct pci_bus *bus);
> +void pci_bus_claim_resources(struct pci_bus *bus);
>  void pci_bus_size_bridges(struct pci_bus *bus);
>  int pci_claim_resource(struct pci_dev *, int);
>  int pci_claim_bridge_resource(struct pci_dev *bridge, int i);
> -- 
> 2.5.1
>