[PATCH V7 00/11] Support for generic ACPI based PCI host controller

[PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Tomasz Nowicki]

>From the functionality point of view this series may be split into the
following logic parts:
1. New ECAM API and update for users of the pci-host-common API
2. Necessary fixes as the preparation for using driver on ARM64.
3. Use new MCFG interface and implement generic ACPI based PCI host controller driver.
4. Enable above driver on ARM64

Patches has been built on top of 4.6-rc7 and can be found here:
git@github.com:semihalf-nowicki-tomasz/linux.git (pci-acpi-v7)

This has been tested on Cavium ThunderX server. Any help in reviewing and
testing is very appreciated.

v6 -> v7
- drop quirks handling
- changes for ACPI companion and domain number assignment approach
- implement arch pcibios_{add|remove}_bus and call acpi_pci_{add|remove}_bus from there
- cleanups around nomenclature
- use resources oriented API for ECAM
- fix for based address calculation before mapping ECAM region
- remove useless lock for MCFG lookup
- move MCFG stuff to separated file pci_mcfg.c
- drop MCFG entries caching
- rebase against 4.6-rc7

v5 -> v6
- drop idea of x86 MMCONFIG code refactoring
- integrate JC's patches which introduce new ECAM API:
  https://lkml.org/lkml/2016/4/11/907
  git: https://github.com/jchandra-brcm/linux/ (arm64-acpi-pci-v3)
- integrate Sinan's fix for releasing IO resources, see patch [06/13]
- added ACPI support for ThunderX ECAM and PEM drivers
- rebase against 4.6-rc2

v4 -> v5
- drop MCFG refactoring group patches 1-6 from series v4 and integrate Jayachandran's patch
  https://patchwork.ozlabs.org/patch/575525/
- rewrite PCI legacy IRQs allocation
- squash two patches 11 and 12 from series v4, fixed bisection issue
- changelog improvements
- rebase against 4.5-rc3

v3 -> v4
- drop Jiang's fix http://lkml.iu.edu/hypermail/linux/kernel/1601.1/04318.html
- add Lorenzo's fix patch 19/24
- ACPI PCI bus domain number assigning cleanup
- change resource management, we now claim and reassign resources
- improvements for applying quirks
- drop Matthew's http://www.spinics.net/lists/linux-pci/msg45950.html dependency
- rebase against 4.5-rc1

v2 -> v3
- fix legacy IRQ assigning and IO ports registration
- remove reference to arch specific companion device for ia64
- move ACPI PCI host controller driver to pci_root.c
- drop generic domain assignment for x86 and ia64 as I am not
  able to run all necessary test variants
- drop patch which cleaned legacy IRQ assignment since it belongs to
  Mathew's series:
  https://patchwork.ozlabs.org/patch/557504/
- extend MCFG quirk code
- rebase against 4.4

v1 -> v2
- move non-arch specific piece of code to dirver/acpi/ directory
- fix IO resource handling
- introduce PCI config accessors quirks matching
- moved ACPI_COMPANION_SET to generic code

v1 - https://lkml.org/lkml/2015/10/27/504
v2 - https://lkml.org/lkml/2015/12/16/246
v3 - http://lkml.iu.edu/hypermail/linux/kernel/1601.1/04308.html
v4 - https://lkml.org/lkml/2016/2/4/646
v5 - https://lkml.org/lkml/2016/2/16/426
v6 - https://lkml.org/lkml/2016/4/15/594

Jayachandran C (2):
  PCI: Provide common functions for ECAM mapping
  PCI: generic, thunder: update to use generic ECAM API

Tomasz Nowicki (9):
  pci, of: Move the PCI I/O space management to PCI core code.
  pci: Add new function to unmap IO resources.
  acpi, pci: Support IO resources when parsing PCI host bridge
    resources.
  pci, acpi: Provide a way to assign bus domain number.
  pci, acpi: Handle ACPI companion assignment.
  pci, acpi: Support for ACPI based generic PCI host controller
  arm64, pci, acpi: ACPI support for legacy IRQs parsing and
    consolidation with DT code.
  arm64, pci, acpi: Provide ACPI-specific prerequisites for PCI bus
    enumeration.
  arm64, pci, acpi: Start using ACPI based PCI host controller driver
    for ARM64.

 arch/arm64/Kconfig                  |   1 +
 arch/arm64/kernel/pci.c             |  34 +++-----
 drivers/acpi/Kconfig                |   8 ++
 drivers/acpi/Makefile               |   1 +
 drivers/acpi/pci_mcfg.c             |  97 ++++++++++++++++++++++
 drivers/acpi/pci_root.c             |  33 ++++++++
 drivers/acpi/pci_root_generic.c     | 149 +++++++++++++++++++++++++++++++++
 drivers/of/address.c                | 116 +-------------------------
 drivers/pci/Kconfig                 |   3 +
 drivers/pci/Makefile                |   2 +
 drivers/pci/ecam.c                  | 161 ++++++++++++++++++++++++++++++++++++
 drivers/pci/ecam.h                  |  72 ++++++++++++++++
 drivers/pci/host/Kconfig            |   1 +
 drivers/pci/host/pci-host-common.c  | 114 +++++++++++--------------
 drivers/pci/host/pci-host-common.h  |  47 -----------
 drivers/pci/host/pci-host-generic.c |  52 +++---------
 drivers/pci/host/pci-thunder-ecam.c |  39 ++-------
 drivers/pci/host/pci-thunder-pem.c  |  92 ++++++++++-----------
 drivers/pci/pci.c                   | 150 ++++++++++++++++++++++++++++++++-
 drivers/pci/probe.c                 |   2 +
 include/linux/of_address.h          |   9 --
 include/linux/pci-acpi.h            |  14 ++++
 include/linux/pci.h                 |  11 ++-
 23 files changed, 823 insertions(+), 385 deletions(-)
 create mode 100644 drivers/acpi/pci_mcfg.c
 create mode 100644 drivers/acpi/pci_root_generic.c
 create mode 100644 drivers/pci/ecam.c
 create mode 100644 drivers/pci/ecam.h
 delete mode 100644 drivers/pci/host/pci-host-common.h

-- 
1.9.1

[PATCH V7 01/11] PCI: Provide common functions for ECAM mapping

[Tomasz Nowicki]

From: Jayachandran C <jchandra@broadcom.com>

Add config option PCI_ECAM and file drivers/pci/ecam.c to provide
generic functions for accessing memory mapped PCI config space.

The API is defined in drivers/pci/ecam.h and is written to replace the
API in drivers/pci/host/pci-host-common.h. The file defines a new
'struct pci_config_window' to hold the information related to a PCI
config area and its mapping. This structure is expected to be used as
sysdata for controllers that have ECAM based mapping.

Helper functions are provided to setup the mapping, free the mapping
and to implement the map_bus method in 'struct pci_ops'

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
---
 drivers/pci/Kconfig  |   3 +
 drivers/pci/Makefile |   2 +
 drivers/pci/ecam.c   | 161 +++++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/ecam.h   |  62 ++++++++++++++++++++
 4 files changed, 228 insertions(+)
 create mode 100644 drivers/pci/ecam.c
 create mode 100644 drivers/pci/ecam.h

diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index 209292e..56389be 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -83,6 +83,9 @@ config HT_IRQ
 config PCI_ATS
 	bool
 
+config PCI_ECAM
+	bool
+
 config PCI_IOV
 	bool "PCI IOV support"
 	depends on PCI
diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
index 2154092..1fa6925 100644
--- a/drivers/pci/Makefile
+++ b/drivers/pci/Makefile
@@ -55,6 +55,8 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o
 
 obj-$(CONFIG_PCI_STUB) += pci-stub.o
 
+obj-$(CONFIG_PCI_ECAM) += ecam.o
+
 obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
 
 obj-$(CONFIG_OF) += of.o
diff --git a/drivers/pci/ecam.c b/drivers/pci/ecam.c
new file mode 100644
index 0000000..3d52005
--- /dev/null
+++ b/drivers/pci/ecam.c
@@ -0,0 +1,161 @@
+/*
+ * Copyright 2016 Broadcom
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation (the "GPL").
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License version 2 (GPLv2) for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 (GPLv2) along with this source code.
+ */
+
+#include <linux/device.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/slab.h>
+
+#include "ecam.h"
+
+/*
+ * On 64 bit systems, we do a single ioremap for the whole config space
+ * since we have enough virtual address range available. On 32 bit, do an
+ * ioremap per bus.
+ */
+static const bool per_bus_mapping = !config_enabled(CONFIG_64BIT);
+
+/*
+ * Create a PCI config space window
+ *  - reserve mem region
+ *  - alloc struct pci_config_window with space for all mappings
+ *  - ioremap the config space
+ */
+struct pci_config_window *pci_ecam_create(struct device *dev,
+		struct resource *cfgres, struct resource *busr,
+		struct pci_ecam_ops *ops)
+{
+	struct pci_config_window *cfg;
+	unsigned int bus_range, bus_range_max, bsz;
+	int i, err;
+
+	if (busr->start > busr->end)
+		return ERR_PTR(-EINVAL);
+
+	cfg = kzalloc(sizeof(*cfg), GFP_KERNEL);
+	if (!cfg)
+		return ERR_PTR(-ENOMEM);
+
+	cfg->ops = ops;
+	cfg->busr.start = busr->start;
+	cfg->busr.end = busr->end;
+	cfg->busr.flags = IORESOURCE_BUS;
+	bus_range = resource_size(&cfg->busr);
+	bus_range_max = resource_size(cfgres) >> ops->bus_shift;
+	if (bus_range > bus_range_max) {
+		dev_warn(dev, "bus max %#x reduced to %#x",
+					bus_range, bus_range_max);
+		bus_range = bus_range_max;
+		cfg->busr.end = busr->start + bus_range - 1;
+	}
+	bsz = 1 << ops->bus_shift;
+
+	cfg->res.start = cfgres->start;
+	cfg->res.end = cfgres->end;
+	cfg->res.flags = IORESOURCE_MEM | IORESOURCE_BUSY;
+	cfg->res.name = "PCI ECAM";
+
+	err = request_resource(&iomem_resource, &cfg->res);
+	if (err) {
+		dev_err(dev, "request ECAM res %pR failed\n", &cfg->res);
+		goto err_exit;
+	}
+
+	if (per_bus_mapping) {
+		cfg->winp = kcalloc(bus_range, sizeof(*cfg->winp), GFP_KERNEL);
+		if (!cfg->winp)
+			goto err_exit_malloc;
+		for (i = 0; i < bus_range; i++) {
+			cfg->winp[i] = ioremap(cfgres->start + i * bsz, bsz);
+			if (!cfg->winp[i])
+				goto err_exit_iomap;
+		}
+	} else {
+		cfg->win = ioremap(cfgres->start, bus_range * bsz);
+		if (!cfg->win)
+			goto err_exit_iomap;
+	}
+
+	if (ops->init) {
+		err = ops->init(dev, cfg);
+		if (err)
+			goto err_exit;
+	}
+	dev_info(dev, "ECAM at %pR for %pR\n", &cfg->res, &cfg->busr);
+	return cfg;
+
+err_exit_iomap:
+	dev_err(dev, "ECAM ioremap failed\n");
+err_exit_malloc:
+	err = -ENOMEM;
+err_exit:
+	pci_ecam_free(cfg);
+	return ERR_PTR(err);
+}
+
+void pci_ecam_free(struct pci_config_window *cfg)
+{
+	int i;
+
+	if (per_bus_mapping) {
+		if (cfg->winp) {
+			for (i = 0; i < resource_size(&cfg->busr); i++)
+				if (cfg->winp[i])
+					iounmap(cfg->winp[i]);
+			kfree(cfg->winp);
+		}
+	} else {
+		if (cfg->win)
+			iounmap(cfg->win);
+	}
+	if (cfg->res.parent)
+		release_resource(&cfg->res);
+	kfree(cfg);
+}
+
+/*
+ * Function to implement the pci_ops ->map_bus method
+ */
+void __iomem *pci_ecam_map_bus(struct pci_bus *bus, unsigned int devfn,
+			       int where)
+{
+	struct pci_config_window *cfg = bus->sysdata;
+	unsigned int devfn_shift = cfg->ops->bus_shift - 8;
+	unsigned int busn = bus->number;
+	void __iomem *base;
+
+	if (busn < cfg->busr.start || busn > cfg->busr.end)
+		return NULL;
+
+	busn -= cfg->busr.start;
+	if (per_bus_mapping)
+		base = cfg->winp[busn];
+	else
+		base = cfg->win + (busn << cfg->ops->bus_shift);
+	return base + (devfn << devfn_shift) + where;
+}
+
+/* ECAM ops */
+struct pci_ecam_ops pci_generic_ecam_ops = {
+	.bus_shift	= 20,
+	.pci_ops	= {
+		.map_bus	= pci_ecam_map_bus,
+		.read		= pci_generic_config_read,
+		.write		= pci_generic_config_write,
+	}
+};
diff --git a/drivers/pci/ecam.h b/drivers/pci/ecam.h
new file mode 100644
index 0000000..57de00d
--- /dev/null
+++ b/drivers/pci/ecam.h
@@ -0,0 +1,62 @@
+/*
+ * Copyright 2016 Broadcom
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation (the "GPL").
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License version 2 (GPLv2) for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 (GPLv2) along with this source code.
+ */
+#ifndef DRIVERS_PCI_ECAM_H
+#define DRIVERS_PCI_ECAM_H
+
+#include <linux/kernel.h>
+#include <linux/platform_device.h>
+
+/*
+ * struct to hold pci ops and bus shift of the config window
+ * for a PCI controller.
+ */
+struct pci_config_window;
+struct pci_ecam_ops {
+	unsigned int			bus_shift;
+	struct pci_ops			pci_ops;
+	int				(*init)(struct device *,
+						struct pci_config_window *);
+};
+
+/*
+ * struct to hold the mappings of a config space window. This
+ * is expected to be used as sysdata for PCI controlllers which
+ * use ECAM.
+ */
+struct pci_config_window {
+	struct resource			res;
+	struct resource			busr;
+	void				*priv;
+	struct pci_ecam_ops		*ops;
+	union {
+		void __iomem		*win;	/* 64-bit single mapping */
+		void __iomem		**winp; /* 32-bit per bus mapping */
+	};
+};
+
+/* create and free for pci_config_window */
+struct pci_config_window *pci_ecam_create(struct device *dev,
+		struct resource *cfgres, struct resource *busr,
+		struct pci_ecam_ops *ops);
+void pci_ecam_free(struct pci_config_window *cfg);
+
+/* map_bus when ->sysdata is an instance of pci_config_window */
+void __iomem *pci_ecam_map_bus(struct pci_bus *bus, unsigned int devfn,
+			       int where);
+/* default ECAM ops, bus shift 20, generic read and write */
+extern struct pci_ecam_ops pci_generic_ecam_ops;
+
+#endif
-- 
1.9.1

[PATCH V7 02/11] PCI: generic, thunder: update to use generic ECAM API

[Tomasz Nowicki]

From: Jayachandran C <jchandra@broadcom.com>

Use functions provided by drivers/pci/ecam.h for mapping the config
space in drivers/pci/host/pci-host-common.c, and update its users to
use 'struct pci_config_window' and 'struct pci_ecam_ops'

The changes are mostly to use 'struct pci_config_window' in place of
'struct gen_pci'. Some of the fields of gen_pci were only used
temporarily and can be eliminated by using local variables or function
arguments, these are not carried over to struct pci_config_window.

pci-thunder-ecam.c and pci-thunder-pem.c are the only users of the
pci_host_common_probe function and the gen_pci structure, these have
been updated to use the new API as well.

The patch does not introduce any functional changes other than a very
minor one: with the new code, on 64-bit platforms, we do just a single
ioremap for the whole config space.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
---
 drivers/pci/ecam.h                  |   5 ++
 drivers/pci/host/Kconfig            |   1 +
 drivers/pci/host/pci-host-common.c  | 114 +++++++++++++++---------------------
 drivers/pci/host/pci-host-common.h  |  47 ---------------
 drivers/pci/host/pci-host-generic.c |  52 ++++------------
 drivers/pci/host/pci-thunder-ecam.c |  39 +++---------
 drivers/pci/host/pci-thunder-pem.c  |  92 ++++++++++++++---------------
 7 files changed, 113 insertions(+), 237 deletions(-)
 delete mode 100644 drivers/pci/host/pci-host-common.h

diff --git a/drivers/pci/ecam.h b/drivers/pci/ecam.h
index 57de00d..1ad2176 100644
--- a/drivers/pci/ecam.h
+++ b/drivers/pci/ecam.h
@@ -59,4 +59,9 @@ void __iomem *pci_ecam_map_bus(struct pci_bus *bus, unsigned int devfn,
 /* default ECAM ops, bus shift 20, generic read and write */
 extern struct pci_ecam_ops pci_generic_ecam_ops;
 
+#ifdef CONFIG_PCI_HOST_GENERIC
+/* for DT based pci controllers that support ECAM */
+int pci_host_common_probe(struct platform_device *pdev,
+			  struct pci_ecam_ops *ops);
+#endif
 #endif
diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
index 7a0780d..963300d 100644
--- a/drivers/pci/host/Kconfig
+++ b/drivers/pci/host/Kconfig
@@ -77,6 +77,7 @@ config PCI_RCAR_GEN2_PCIE
 
 config PCI_HOST_COMMON
 	bool
+	select PCI_ECAM
 
 config PCI_HOST_GENERIC
 	bool "Generic PCI host controller"
diff --git a/drivers/pci/host/pci-host-common.c b/drivers/pci/host/pci-host-common.c
index e9f850f..8cba7ab 100644
--- a/drivers/pci/host/pci-host-common.c
+++ b/drivers/pci/host/pci-host-common.c
@@ -22,27 +22,21 @@
 #include <linux/of_pci.h>
 #include <linux/platform_device.h>
 
-#include "pci-host-common.h"
+#include "../ecam.h"
 
-static void gen_pci_release_of_pci_ranges(struct gen_pci *pci)
-{
-	pci_free_resource_list(&pci->resources);
-}
-
-static int gen_pci_parse_request_of_pci_ranges(struct gen_pci *pci)
+static int gen_pci_parse_request_of_pci_ranges(struct device *dev,
+		       struct list_head *resources, struct resource **bus_range)
 {
 	int err, res_valid = 0;
-	struct device *dev = pci->host.dev.parent;
 	struct device_node *np = dev->of_node;
 	resource_size_t iobase;
 	struct resource_entry *win;
 
-	err = of_pci_get_host_bridge_resources(np, 0, 0xff, &pci->resources,
-					       &iobase);
+	err = of_pci_get_host_bridge_resources(np, 0, 0xff, resources, &iobase);
 	if (err)
 		return err;
 
-	resource_list_for_each_entry(win, &pci->resources) {
+	resource_list_for_each_entry(win, resources) {
 		struct resource *parent, *res = win->res;
 
 		switch (resource_type(res)) {
@@ -60,7 +54,7 @@ static int gen_pci_parse_request_of_pci_ranges(struct gen_pci *pci)
 			res_valid |= !(res->flags & IORESOURCE_PREFETCH);
 			break;
 		case IORESOURCE_BUS:
-			pci->cfg.bus_range = res;
+			*bus_range = res;
 		default:
 			continue;
 		}
@@ -79,65 +73,60 @@ static int gen_pci_parse_request_of_pci_ranges(struct gen_pci *pci)
 	return 0;
 
 out_release_res:
-	gen_pci_release_of_pci_ranges(pci);
 	return err;
 }
 
-static int gen_pci_parse_map_cfg_windows(struct gen_pci *pci)
+static void gen_pci_unmap_cfg(void *ptr)
+{
+	pci_ecam_free((struct pci_config_window *)ptr);
+}
+
+static struct pci_config_window *gen_pci_init(struct device *dev,
+		struct list_head *resources, struct pci_ecam_ops *ops)
 {
 	int err;
-	u8 bus_max;
-	resource_size_t busn;
-	struct resource *bus_range;
-	struct device *dev = pci->host.dev.parent;
-	struct device_node *np = dev->of_node;
-	u32 sz = 1 << pci->cfg.ops->bus_shift;
+	struct resource cfgres;
+	struct resource *bus_range = NULL;
+	struct pci_config_window *cfg;
 
-	err = of_address_to_resource(np, 0, &pci->cfg.res);
+	/* Parse our PCI ranges and request their resources */
+	err = gen_pci_parse_request_of_pci_ranges(dev, resources, &bus_range);
+	if (err)
+		goto err_out;
+
+	err = of_address_to_resource(dev->of_node, 0, &cfgres);
 	if (err) {
 		dev_err(dev, "missing \"reg\" property\n");
-		return err;
+		goto err_out;
 	}
 
-	/* Limit the bus-range to fit within reg */
-	bus_max = pci->cfg.bus_range->start +
-		  (resource_size(&pci->cfg.res) >> pci->cfg.ops->bus_shift) - 1;
-	pci->cfg.bus_range->end = min_t(resource_size_t,
-					pci->cfg.bus_range->end, bus_max);
-
-	pci->cfg.win = devm_kcalloc(dev, resource_size(pci->cfg.bus_range),
-				    sizeof(*pci->cfg.win), GFP_KERNEL);
-	if (!pci->cfg.win)
-		return -ENOMEM;
-
-	/* Map our Configuration Space windows */
-	if (!devm_request_mem_region(dev, pci->cfg.res.start,
-				     resource_size(&pci->cfg.res),
-				     "Configuration Space"))
-		return -ENOMEM;
-
-	bus_range = pci->cfg.bus_range;
-	for (busn = bus_range->start; busn <= bus_range->end; ++busn) {
-		u32 idx = busn - bus_range->start;
-
-		pci->cfg.win[idx] = devm_ioremap(dev,
-						 pci->cfg.res.start + idx * sz,
-						 sz);
-		if (!pci->cfg.win[idx])
-			return -ENOMEM;
+	cfg = pci_ecam_create(dev, &cfgres, bus_range, ops);
+	if (IS_ERR(cfg)) {
+		err = PTR_ERR(cfg);
+		goto err_out;
 	}
 
-	return 0;
+	err = devm_add_action(dev, gen_pci_unmap_cfg, cfg);
+	if (err) {
+		gen_pci_unmap_cfg(cfg);
+		goto err_out;
+	}
+	return cfg;
+
+err_out:
+	pci_free_resource_list(resources);
+	return ERR_PTR(err);
 }
 
 int pci_host_common_probe(struct platform_device *pdev,
-			  struct gen_pci *pci)
+			  struct pci_ecam_ops *ops)
 {
-	int err;
 	const char *type;
 	struct device *dev = &pdev->dev;
 	struct device_node *np = dev->of_node;
 	struct pci_bus *bus, *child;
+	struct pci_config_window *cfg;
+	struct list_head resources;
 
 	type = of_get_property(np, "device_type", NULL);
 	if (!type || strcmp(type, "pci")) {
@@ -147,29 +136,18 @@ int pci_host_common_probe(struct platform_device *pdev,
 
 	of_pci_check_probe_only();
 
-	pci->host.dev.parent = dev;
-	INIT_LIST_HEAD(&pci->host.windows);
-	INIT_LIST_HEAD(&pci->resources);
-
-	/* Parse our PCI ranges and request their resources */
-	err = gen_pci_parse_request_of_pci_ranges(pci);
-	if (err)
-		return err;
-
 	/* Parse and map our Configuration Space windows */
-	err = gen_pci_parse_map_cfg_windows(pci);
-	if (err) {
-		gen_pci_release_of_pci_ranges(pci);
-		return err;
-	}
+	INIT_LIST_HEAD(&resources);
+	cfg = gen_pci_init(dev, &resources, ops);
+	if (IS_ERR(cfg))
+		return PTR_ERR(cfg);
 
 	/* Do not reassign resources if probe only */
 	if (!pci_has_flag(PCI_PROBE_ONLY))
 		pci_add_flags(PCI_REASSIGN_ALL_RSRC | PCI_REASSIGN_ALL_BUS);
 
-
-	bus = pci_scan_root_bus(dev, pci->cfg.bus_range->start,
-				&pci->cfg.ops->ops, pci, &pci->resources);
+	bus = pci_scan_root_bus(dev, cfg->busr.start, &ops->pci_ops, cfg,
+				&resources);
 	if (!bus) {
 		dev_err(dev, "Scanning rootbus failed");
 		return -ENODEV;
diff --git a/drivers/pci/host/pci-host-common.h b/drivers/pci/host/pci-host-common.h
deleted file mode 100644
index 09f3fa0..0000000
--- a/drivers/pci/host/pci-host-common.h
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.  If not, see <http://www.gnu.org/licenses/>.
- *
- * Copyright (C) 2014 ARM Limited
- *
- * Author: Will Deacon <will.deacon@arm.com>
- */
-
-#ifndef _PCI_HOST_COMMON_H
-#define _PCI_HOST_COMMON_H
-
-#include <linux/kernel.h>
-#include <linux/platform_device.h>
-
-struct gen_pci_cfg_bus_ops {
-	u32 bus_shift;
-	struct pci_ops ops;
-};
-
-struct gen_pci_cfg_windows {
-	struct resource				res;
-	struct resource				*bus_range;
-	void __iomem				**win;
-
-	struct gen_pci_cfg_bus_ops		*ops;
-};
-
-struct gen_pci {
-	struct pci_host_bridge			host;
-	struct gen_pci_cfg_windows		cfg;
-	struct list_head			resources;
-};
-
-int pci_host_common_probe(struct platform_device *pdev,
-			  struct gen_pci *pci);
-
-#endif /* _PCI_HOST_COMMON_H */
diff --git a/drivers/pci/host/pci-host-generic.c b/drivers/pci/host/pci-host-generic.c
index e8aa78f..6eaceab 100644
--- a/drivers/pci/host/pci-host-generic.c
+++ b/drivers/pci/host/pci-host-generic.c
@@ -25,41 +25,12 @@
 #include <linux/of_pci.h>
 #include <linux/platform_device.h>
 
-#include "pci-host-common.h"
+#include "../ecam.h"
 
-static void __iomem *gen_pci_map_cfg_bus_cam(struct pci_bus *bus,
-					     unsigned int devfn,
-					     int where)
-{
-	struct gen_pci *pci = bus->sysdata;
-	resource_size_t idx = bus->number - pci->cfg.bus_range->start;
-
-	return pci->cfg.win[idx] + ((devfn << 8) | where);
-}
-
-static struct gen_pci_cfg_bus_ops gen_pci_cfg_cam_bus_ops = {
+static struct pci_ecam_ops gen_pci_cfg_cam_bus_ops = {
 	.bus_shift	= 16,
-	.ops		= {
-		.map_bus	= gen_pci_map_cfg_bus_cam,
-		.read		= pci_generic_config_read,
-		.write		= pci_generic_config_write,
-	}
-};
-
-static void __iomem *gen_pci_map_cfg_bus_ecam(struct pci_bus *bus,
-					      unsigned int devfn,
-					      int where)
-{
-	struct gen_pci *pci = bus->sysdata;
-	resource_size_t idx = bus->number - pci->cfg.bus_range->start;
-
-	return pci->cfg.win[idx] + ((devfn << 12) | where);
-}
-
-static struct gen_pci_cfg_bus_ops gen_pci_cfg_ecam_bus_ops = {
-	.bus_shift	= 20,
-	.ops		= {
-		.map_bus	= gen_pci_map_cfg_bus_ecam,
+	.pci_ops	= {
+		.map_bus	= pci_ecam_map_bus,
 		.read		= pci_generic_config_read,
 		.write		= pci_generic_config_write,
 	}
@@ -70,25 +41,22 @@ static const struct of_device_id gen_pci_of_match[] = {
 	  .data = &gen_pci_cfg_cam_bus_ops },
 
 	{ .compatible = "pci-host-ecam-generic",
-	  .data = &gen_pci_cfg_ecam_bus_ops },
+	  .data = &pci_generic_ecam_ops },
 
 	{ },
 };
+
 MODULE_DEVICE_TABLE(of, gen_pci_of_match);
 
 static int gen_pci_probe(struct platform_device *pdev)
 {
-	struct device *dev = &pdev->dev;
 	const struct of_device_id *of_id;
-	struct gen_pci *pci = devm_kzalloc(dev, sizeof(*pci), GFP_KERNEL);
-
-	if (!pci)
-		return -ENOMEM;
+	struct pci_ecam_ops *ops;
 
-	of_id = of_match_node(gen_pci_of_match, dev->of_node);
-	pci->cfg.ops = (struct gen_pci_cfg_bus_ops *)of_id->data;
+	of_id = of_match_node(gen_pci_of_match, pdev->dev.of_node);
+	ops = (struct pci_ecam_ops *)of_id->data;
 
-	return pci_host_common_probe(pdev, pci);
+	return pci_host_common_probe(pdev, ops);
 }
 
 static struct platform_driver gen_pci_driver = {
diff --git a/drivers/pci/host/pci-thunder-ecam.c b/drivers/pci/host/pci-thunder-ecam.c
index d71935cb..540d030 100644
--- a/drivers/pci/host/pci-thunder-ecam.c
+++ b/drivers/pci/host/pci-thunder-ecam.c
@@ -13,18 +13,7 @@
 #include <linux/of.h>
 #include <linux/platform_device.h>
 
-#include "pci-host-common.h"
-
-/* Mapping is standard ECAM */
-static void __iomem *thunder_ecam_map_bus(struct pci_bus *bus,
-					  unsigned int devfn,
-					  int where)
-{
-	struct gen_pci *pci = bus->sysdata;
-	resource_size_t idx = bus->number - pci->cfg.bus_range->start;
-
-	return pci->cfg.win[idx] + ((devfn << 12) | where);
-}
+#include "../ecam.h"
 
 static void set_val(u32 v, int where, int size, u32 *val)
 {
@@ -99,7 +88,7 @@ static int handle_ea_bar(u32 e0, int bar, struct pci_bus *bus,
 static int thunder_ecam_p2_config_read(struct pci_bus *bus, unsigned int devfn,
 				       int where, int size, u32 *val)
 {
-	struct gen_pci *pci = bus->sysdata;
+	struct pci_config_window *cfg = bus->sysdata;
 	int where_a = where & ~3;
 	void __iomem *addr;
 	u32 node_bits;
@@ -129,7 +118,7 @@ static int thunder_ecam_p2_config_read(struct pci_bus *bus, unsigned int devfn,
 	 * the config space access window.  Since we are working with
 	 * the high-order 32 bits, shift everything down by 32 bits.
 	 */
-	node_bits = (pci->cfg.res.start >> 32) & (1 << 12);
+	node_bits = (cfg->res.start >> 32) & (1 << 12);
 
 	v |= node_bits;
 	set_val(v, where, size, val);
@@ -358,36 +347,24 @@ static int thunder_ecam_config_write(struct pci_bus *bus, unsigned int devfn,
 	return pci_generic_config_write(bus, devfn, where, size, val);
 }
 
-static struct gen_pci_cfg_bus_ops thunder_ecam_bus_ops = {
+static struct pci_ecam_ops pci_thunder_ecam_ops = {
 	.bus_shift	= 20,
-	.ops		= {
-		.map_bus        = thunder_ecam_map_bus,
+	.pci_ops	= {
+		.map_bus        = pci_ecam_map_bus,
 		.read           = thunder_ecam_config_read,
 		.write          = thunder_ecam_config_write,
 	}
 };
 
 static const struct of_device_id thunder_ecam_of_match[] = {
-	{ .compatible = "cavium,pci-host-thunder-ecam",
-	  .data = &thunder_ecam_bus_ops },
-
+	{ .compatible = "cavium,pci-host-thunder-ecam" },
 	{ },
 };
 MODULE_DEVICE_TABLE(of, thunder_ecam_of_match);
 
 static int thunder_ecam_probe(struct platform_device *pdev)
 {
-	struct device *dev = &pdev->dev;
-	const struct of_device_id *of_id;
-	struct gen_pci *pci = devm_kzalloc(dev, sizeof(*pci), GFP_KERNEL);
-
-	if (!pci)
-		return -ENOMEM;
-
-	of_id = of_match_node(thunder_ecam_of_match, dev->of_node);
-	pci->cfg.ops = (struct gen_pci_cfg_bus_ops *)of_id->data;
-
-	return pci_host_common_probe(pdev, pci);
+	return pci_host_common_probe(pdev, &pci_thunder_ecam_ops);
 }
 
 static struct platform_driver thunder_ecam_driver = {
diff --git a/drivers/pci/host/pci-thunder-pem.c b/drivers/pci/host/pci-thunder-pem.c
index cabb92a..a7b61ce 100644
--- a/drivers/pci/host/pci-thunder-pem.c
+++ b/drivers/pci/host/pci-thunder-pem.c
@@ -20,34 +20,22 @@
 #include <linux/of_pci.h>
 #include <linux/platform_device.h>
 
-#include "pci-host-common.h"
+#include "../ecam.h"
 
 #define PEM_CFG_WR 0x28
 #define PEM_CFG_RD 0x30
 
 struct thunder_pem_pci {
-	struct gen_pci	gen_pci;
 	u32		ea_entry[3];
 	void __iomem	*pem_reg_base;
 };
 
-static void __iomem *thunder_pem_map_bus(struct pci_bus *bus,
-					 unsigned int devfn, int where)
-{
-	struct gen_pci *pci = bus->sysdata;
-	resource_size_t idx = bus->number - pci->cfg.bus_range->start;
-
-	return pci->cfg.win[idx] + ((devfn << 16) | where);
-}
-
 static int thunder_pem_bridge_read(struct pci_bus *bus, unsigned int devfn,
 				   int where, int size, u32 *val)
 {
 	u64 read_val;
-	struct thunder_pem_pci *pem_pci;
-	struct gen_pci *pci = bus->sysdata;
-
-	pem_pci = container_of(pci, struct thunder_pem_pci, gen_pci);
+	struct pci_config_window *cfg = bus->sysdata;
+	struct thunder_pem_pci *pem_pci = (struct thunder_pem_pci *)cfg->priv;
 
 	if (devfn != 0 || where >= 2048) {
 		*val = ~0;
@@ -132,17 +120,17 @@ static int thunder_pem_bridge_read(struct pci_bus *bus, unsigned int devfn,
 static int thunder_pem_config_read(struct pci_bus *bus, unsigned int devfn,
 				   int where, int size, u32 *val)
 {
-	struct gen_pci *pci = bus->sysdata;
+	struct pci_config_window *cfg = bus->sysdata;
 
-	if (bus->number < pci->cfg.bus_range->start ||
-	    bus->number > pci->cfg.bus_range->end)
+	if (bus->number < cfg->busr.start ||
+	    bus->number > cfg->busr.end)
 		return PCIBIOS_DEVICE_NOT_FOUND;
 
 	/*
 	 * The first device on the bus is the PEM PCIe bridge.
 	 * Special case its config access.
 	 */
-	if (bus->number == pci->cfg.bus_range->start)
+	if (bus->number == cfg->busr.start)
 		return thunder_pem_bridge_read(bus, devfn, where, size, val);
 
 	return pci_generic_config_read(bus, devfn, where, size, val);
@@ -187,12 +175,11 @@ static u32 thunder_pem_bridge_w1c_bits(int where)
 static int thunder_pem_bridge_write(struct pci_bus *bus, unsigned int devfn,
 				    int where, int size, u32 val)
 {
-	struct gen_pci *pci = bus->sysdata;
-	struct thunder_pem_pci *pem_pci;
+	struct pci_config_window *cfg = bus->sysdata;
+	struct thunder_pem_pci *pem_pci = (struct thunder_pem_pci *)cfg->priv;
 	u64 write_val, read_val;
 	u32 mask = 0;
 
-	pem_pci = container_of(pci, struct thunder_pem_pci, gen_pci);
 
 	if (devfn != 0 || where >= 2048)
 		return PCIBIOS_DEVICE_NOT_FOUND;
@@ -256,53 +243,38 @@ static int thunder_pem_bridge_write(struct pci_bus *bus, unsigned int devfn,
 static int thunder_pem_config_write(struct pci_bus *bus, unsigned int devfn,
 				    int where, int size, u32 val)
 {
-	struct gen_pci *pci = bus->sysdata;
+	struct pci_config_window *cfg = bus->sysdata;
 
-	if (bus->number < pci->cfg.bus_range->start ||
-	    bus->number > pci->cfg.bus_range->end)
+	if (bus->number < cfg->busr.start ||
+	    bus->number > cfg->busr.end)
 		return PCIBIOS_DEVICE_NOT_FOUND;
 	/*
 	 * The first device on the bus is the PEM PCIe bridge.
 	 * Special case its config access.
 	 */
-	if (bus->number == pci->cfg.bus_range->start)
+	if (bus->number == cfg->busr.start)
 		return thunder_pem_bridge_write(bus, devfn, where, size, val);
 
 
 	return pci_generic_config_write(bus, devfn, where, size, val);
 }
 
-static struct gen_pci_cfg_bus_ops thunder_pem_bus_ops = {
-	.bus_shift	= 24,
-	.ops		= {
-		.map_bus	= thunder_pem_map_bus,
-		.read		= thunder_pem_config_read,
-		.write		= thunder_pem_config_write,
-	}
-};
-
-static const struct of_device_id thunder_pem_of_match[] = {
-	{ .compatible = "cavium,pci-host-thunder-pem",
-	  .data = &thunder_pem_bus_ops },
-
-	{ },
-};
-MODULE_DEVICE_TABLE(of, thunder_pem_of_match);
-
-static int thunder_pem_probe(struct platform_device *pdev)
+static int thunder_pem_init(struct device *dev, struct pci_config_window *cfg)
 {
-	struct device *dev = &pdev->dev;
-	const struct of_device_id *of_id;
 	resource_size_t bar4_start;
 	struct resource *res_pem;
 	struct thunder_pem_pci *pem_pci;
+	struct platform_device *pdev;
+
+	/* Only OF support for now */
+	if (!dev->of_node)
+		return -EINVAL;
 
 	pem_pci = devm_kzalloc(dev, sizeof(*pem_pci), GFP_KERNEL);
 	if (!pem_pci)
 		return -ENOMEM;
 
-	of_id = of_match_node(thunder_pem_of_match, dev->of_node);
-	pem_pci->gen_pci.cfg.ops = (struct gen_pci_cfg_bus_ops *)of_id->data;
+	pdev = to_platform_device(dev);
 
 	/*
 	 * The second register range is the PEM bridge to the PCIe
@@ -330,7 +302,29 @@ static int thunder_pem_probe(struct platform_device *pdev)
 	pem_pci->ea_entry[1] = (u32)(res_pem->end - bar4_start) & ~3u;
 	pem_pci->ea_entry[2] = (u32)(bar4_start >> 32);
 
-	return pci_host_common_probe(pdev, &pem_pci->gen_pci);
+	cfg->priv = pem_pci;
+	return 0;
+}
+
+static struct pci_ecam_ops pci_thunder_pem_ops = {
+	.bus_shift	= 24,
+	.init		= thunder_pem_init,
+	.pci_ops	= {
+		.map_bus	= pci_ecam_map_bus,
+		.read		= thunder_pem_config_read,
+		.write		= thunder_pem_config_write,
+	}
+};
+
+static const struct of_device_id thunder_pem_of_match[] = {
+	{ .compatible = "cavium,pci-host-thunder-pem" },
+	{ },
+};
+MODULE_DEVICE_TABLE(of, thunder_pem_of_match);
+
+static int thunder_pem_probe(struct platform_device *pdev)
+{
+	return pci_host_common_probe(pdev, &pci_thunder_pem_ops);
 }
 
 static struct platform_driver thunder_pem_driver = {
-- 
1.9.1

[PATCH V7 03/11] pci, of: Move the PCI I/O space management to PCI core code.

[Tomasz Nowicki]

No functional changes in this patch.

PCI I/O space mapping code does not depend on OF, therefore it can be
moved to PCI core code. This way we will be able to use it
e.g. in ACPI PCI code.

Suggested-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
CC: Arnd Bergmann <arnd@arndb.de>
CC: Liviu Dudau <Liviu.Dudau@arm.com>
---
 drivers/of/address.c       | 116 +--------------------------------------------
 drivers/pci/pci.c          | 115 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/of_address.h |   9 ----
 include/linux/pci.h        |   5 ++
 4 files changed, 121 insertions(+), 124 deletions(-)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 91a469d..0a553c0 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -4,6 +4,7 @@
 #include <linux/ioport.h>
 #include <linux/module.h>
 #include <linux/of_address.h>
+#include <linux/pci.h>
 #include <linux/pci_regs.h>
 #include <linux/sizes.h>
 #include <linux/slab.h>
@@ -673,121 +674,6 @@ const __be32 *of_get_address(struct device_node *dev, int index, u64 *size,
 }
 EXPORT_SYMBOL(of_get_address);
 
-#ifdef PCI_IOBASE
-struct io_range {
-	struct list_head list;
-	phys_addr_t start;
-	resource_size_t size;
-};
-
-static LIST_HEAD(io_range_list);
-static DEFINE_SPINLOCK(io_range_lock);
-#endif
-
-/*
- * Record the PCI IO range (expressed as CPU physical address + size).
- * Return a negative value if an error has occured, zero otherwise
- */
-int __weak pci_register_io_range(phys_addr_t addr, resource_size_t size)
-{
-	int err = 0;
-
-#ifdef PCI_IOBASE
-	struct io_range *range;
-	resource_size_t allocated_size = 0;
-
-	/* check if the range hasn't been previously recorded */
-	spin_lock(&io_range_lock);
-	list_for_each_entry(range, &io_range_list, list) {
-		if (addr >= range->start && addr + size <= range->start + size) {
-			/* range already registered, bail out */
-			goto end_register;
-		}
-		allocated_size += range->size;
-	}
-
-	/* range not registed yet, check for available space */
-	if (allocated_size + size - 1 > IO_SPACE_LIMIT) {
-		/* if it's too big check if 64K space can be reserved */
-		if (allocated_size + SZ_64K - 1 > IO_SPACE_LIMIT) {
-			err = -E2BIG;
-			goto end_register;
-		}
-
-		size = SZ_64K;
-		pr_warn("Requested IO range too big, new size set to 64K\n");
-	}
-
-	/* add the range to the list */
-	range = kzalloc(sizeof(*range), GFP_ATOMIC);
-	if (!range) {
-		err = -ENOMEM;
-		goto end_register;
-	}
-
-	range->start = addr;
-	range->size = size;
-
-	list_add_tail(&range->list, &io_range_list);
-
-end_register:
-	spin_unlock(&io_range_lock);
-#endif
-
-	return err;
-}
-
-phys_addr_t pci_pio_to_address(unsigned long pio)
-{
-	phys_addr_t address = (phys_addr_t)OF_BAD_ADDR;
-
-#ifdef PCI_IOBASE
-	struct io_range *range;
-	resource_size_t allocated_size = 0;
-
-	if (pio > IO_SPACE_LIMIT)
-		return address;
-
-	spin_lock(&io_range_lock);
-	list_for_each_entry(range, &io_range_list, list) {
-		if (pio >= allocated_size && pio < allocated_size + range->size) {
-			address = range->start + pio - allocated_size;
-			break;
-		}
-		allocated_size += range->size;
-	}
-	spin_unlock(&io_range_lock);
-#endif
-
-	return address;
-}
-
-unsigned long __weak pci_address_to_pio(phys_addr_t address)
-{
-#ifdef PCI_IOBASE
-	struct io_range *res;
-	resource_size_t offset = 0;
-	unsigned long addr = -1;
-
-	spin_lock(&io_range_lock);
-	list_for_each_entry(res, &io_range_list, list) {
-		if (address >= res->start && address < res->start + res->size) {
-			addr = address - res->start + offset;
-			break;
-		}
-		offset += res->size;
-	}
-	spin_unlock(&io_range_lock);
-
-	return addr;
-#else
-	if (address > IO_SPACE_LIMIT)
-		return (unsigned long)-1;
-
-	return (unsigned long) address;
-#endif
-}
-
 static int __of_address_to_resource(struct device_node *dev,
 		const __be32 *addrp, u64 size, unsigned int flags,
 		const char *name, struct resource *r)
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 25e0327..bc0c914 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3021,6 +3021,121 @@ int pci_request_regions_exclusive(struct pci_dev *pdev, const char *res_name)
 }
 EXPORT_SYMBOL(pci_request_regions_exclusive);
 
+#ifdef PCI_IOBASE
+struct io_range {
+	struct list_head list;
+	phys_addr_t start;
+	resource_size_t size;
+};
+
+static LIST_HEAD(io_range_list);
+static DEFINE_SPINLOCK(io_range_lock);
+#endif
+
+/*
+ * Record the PCI IO range (expressed as CPU physical address + size).
+ * Return a negative value if an error has occured, zero otherwise
+ */
+int __weak pci_register_io_range(phys_addr_t addr, resource_size_t size)
+{
+	int err = 0;
+
+#ifdef PCI_IOBASE
+	struct io_range *range;
+	resource_size_t allocated_size = 0;
+
+	/* check if the range hasn't been previously recorded */
+	spin_lock(&io_range_lock);
+	list_for_each_entry(range, &io_range_list, list) {
+		if (addr >= range->start && addr + size <= range->start + size) {
+			/* range already registered, bail out */
+			goto end_register;
+		}
+		allocated_size += range->size;
+	}
+
+	/* range not registed yet, check for available space */
+	if (allocated_size + size - 1 > IO_SPACE_LIMIT) {
+		/* if it's too big check if 64K space can be reserved */
+		if (allocated_size + SZ_64K - 1 > IO_SPACE_LIMIT) {
+			err = -E2BIG;
+			goto end_register;
+		}
+
+		size = SZ_64K;
+		pr_warn("Requested IO range too big, new size set to 64K\n");
+	}
+
+	/* add the range to the list */
+	range = kzalloc(sizeof(*range), GFP_ATOMIC);
+	if (!range) {
+		err = -ENOMEM;
+		goto end_register;
+	}
+
+	range->start = addr;
+	range->size = size;
+
+	list_add_tail(&range->list, &io_range_list);
+
+end_register:
+	spin_unlock(&io_range_lock);
+#endif
+
+	return err;
+}
+
+phys_addr_t pci_pio_to_address(unsigned long pio)
+{
+	phys_addr_t address = (phys_addr_t)OF_BAD_ADDR;
+
+#ifdef PCI_IOBASE
+	struct io_range *range;
+	resource_size_t allocated_size = 0;
+
+	if (pio > IO_SPACE_LIMIT)
+		return address;
+
+	spin_lock(&io_range_lock);
+	list_for_each_entry(range, &io_range_list, list) {
+		if (pio >= allocated_size && pio < allocated_size + range->size) {
+			address = range->start + pio - allocated_size;
+			break;
+		}
+		allocated_size += range->size;
+	}
+	spin_unlock(&io_range_lock);
+#endif
+
+	return address;
+}
+
+unsigned long __weak pci_address_to_pio(phys_addr_t address)
+{
+#ifdef PCI_IOBASE
+	struct io_range *res;
+	resource_size_t offset = 0;
+	unsigned long addr = -1;
+
+	spin_lock(&io_range_lock);
+	list_for_each_entry(res, &io_range_list, list) {
+		if (address >= res->start && address < res->start + res->size) {
+			addr = address - res->start + offset;
+			break;
+		}
+		offset += res->size;
+	}
+	spin_unlock(&io_range_lock);
+
+	return addr;
+#else
+	if (address > IO_SPACE_LIMIT)
+		return (unsigned long)-1;
+
+	return (unsigned long) address;
+#endif
+}
+
 /**
  *	pci_remap_iospace - Remap the memory mapped I/O space
  *	@res: Resource describing the I/O space
diff --git a/include/linux/of_address.h b/include/linux/of_address.h
index 01c0a55..3786473 100644
--- a/include/linux/of_address.h
+++ b/include/linux/of_address.h
@@ -47,10 +47,6 @@ void __iomem *of_io_request_and_map(struct device_node *device,
 extern const __be32 *of_get_address(struct device_node *dev, int index,
 			   u64 *size, unsigned int *flags);
 
-extern int pci_register_io_range(phys_addr_t addr, resource_size_t size);
-extern unsigned long pci_address_to_pio(phys_addr_t addr);
-extern phys_addr_t pci_pio_to_address(unsigned long pio);
-
 extern int of_pci_range_parser_init(struct of_pci_range_parser *parser,
 			struct device_node *node);
 extern struct of_pci_range *of_pci_range_parser_one(
@@ -86,11 +82,6 @@ static inline const __be32 *of_get_address(struct device_node *dev, int index,
 	return NULL;
 }
 
-static inline phys_addr_t pci_pio_to_address(unsigned long pio)
-{
-	return 0;
-}
-
 static inline int of_pci_range_parser_init(struct of_pci_range_parser *parser,
 			struct device_node *node)
 {
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 932ec74..d2a57f3 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1165,6 +1165,9 @@ int __must_check pci_bus_alloc_resource(struct pci_bus *bus,
 			void *alignf_data);
 
 
+int pci_register_io_range(phys_addr_t addr, resource_size_t size);
+unsigned long pci_address_to_pio(phys_addr_t addr);
+phys_addr_t pci_pio_to_address(unsigned long pio);
 int pci_remap_iospace(const struct resource *res, phys_addr_t phys_addr);
 
 static inline pci_bus_addr_t pci_bus_address(struct pci_dev *pdev, int bar)
@@ -1481,6 +1484,8 @@ static inline int pci_request_regions(struct pci_dev *dev, const char *res_name)
 { return -EIO; }
 static inline void pci_release_regions(struct pci_dev *dev) { }
 
+static inline unsigned long pci_address_to_pio(phys_addr_t addr) { return -1; }
+
 static inline void pci_block_cfg_access(struct pci_dev *dev) { }
 static inline int pci_block_cfg_access_in_atomic(struct pci_dev *dev)
 { return 0; }
-- 
1.9.1

[PATCH V7 04/11] pci: Add new function to unmap IO resources.

[Tomasz Nowicki]

It is very useful to release I/O resources so that the same I/O resources
can be allocated again (pci_remap_iospace), like in PCI hotplug removal
scenario. Therefore this patch implements new pci_unmap_iospace call which
unmaps I/O space as the symmetry to pci_remap_iospace.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
---
 drivers/pci/pci.c   | 24 ++++++++++++++++++++++++
 include/linux/pci.h |  1 +
 2 files changed, 25 insertions(+)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index bc0c914..ff97a0b 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -25,6 +25,7 @@
 #include <linux/device.h>
 #include <linux/pm_runtime.h>
 #include <linux/pci_hotplug.h>
+#include <linux/vmalloc.h>
 #include <asm/setup.h>
 #include <linux/aer.h>
 #include "pci.h"
@@ -3167,6 +3168,29 @@ int __weak pci_remap_iospace(const struct resource *res, phys_addr_t phys_addr)
 #endif
 }
 
+/**
+ *	pci_unmap_iospace - Unmap the memory mapped I/O space
+ *	@res: resource to be unmapped
+ *
+ *	Unmap the CPU virtual address @res from virtual address space.
+ *	Only architectures that have memory mapped IO functions defined
+ *	(and the PCI_IOBASE value defined) should call this function.
+ */
+void  pci_unmap_iospace(struct resource *res)
+{
+#if defined(PCI_IOBASE) && defined(CONFIG_MMU)
+	unsigned long vaddr = (unsigned long)PCI_IOBASE + res->start;
+
+	unmap_kernel_range(vaddr, resource_size(res));
+#else
+	/*
+	 * This architecture does not have memory mapped I/O space,
+	 * so this function should never be called.
+	 */
+	WARN_ONCE(1, "This architecture does not support memory mapped I/O\n");
+#endif
+}
+
 static void __pci_set_master(struct pci_dev *dev, bool enable)
 {
 	u16 old_cmd, cmd;
diff --git a/include/linux/pci.h b/include/linux/pci.h
index d2a57f3..d6ea6ce 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1169,6 +1169,7 @@ int pci_register_io_range(phys_addr_t addr, resource_size_t size);
 unsigned long pci_address_to_pio(phys_addr_t addr);
 phys_addr_t pci_pio_to_address(unsigned long pio);
 int pci_remap_iospace(const struct resource *res, phys_addr_t phys_addr);
+void pci_unmap_iospace(struct resource *res);
 
 static inline pci_bus_addr_t pci_bus_address(struct pci_dev *pdev, int bar)
 {
-- 
1.9.1

[PATCH V7 05/11] acpi, pci: Support IO resources when parsing PCI host bridge resources.

[Tomasz Nowicki]

Platforms that have memory mapped IO port (such as ARM64) need special
handling for PCI I/O resources. For host bridge's resource probing case
these resources need to be fixed up with pci_register_io_range/pci_remap_iospace etc.

The same I/O resources need to be released after hotplug
removal so that it can be re-added back by the pci_remap_iospace
function during insertion. As a consequence we unmap I/O resources
with pci_unmap_iospace when we release host bridge resources.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
---
 drivers/acpi/pci_root.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
index ae3fe4e..cb3071d 100644
--- a/drivers/acpi/pci_root.c
+++ b/drivers/acpi/pci_root.c
@@ -719,6 +719,34 @@ next:
 			resource_list_add_tail(entry, resources);
 	}
 }
+static void acpi_pci_root_remap_iospace(struct resource_entry *entry)
+{
+#ifdef PCI_IOBASE
+	struct resource *res = entry->res;
+	resource_size_t cpu_addr = res->start;
+	resource_size_t pci_addr = cpu_addr - entry->offset;
+	resource_size_t length = resource_size(res);
+	unsigned long port;
+
+	if (pci_register_io_range(cpu_addr, length))
+		goto err;
+
+	port = pci_address_to_pio(cpu_addr);
+	if (port == (unsigned long)-1)
+		goto err;
+
+	res->start = port;
+	res->end = port + length - 1;
+	entry->offset = port - pci_addr;
+
+	if (pci_remap_iospace(res, cpu_addr) < 0)
+		goto err;
+	pr_info("Remapped I/O %pa to %pR\n", &cpu_addr, res);
+	return;
+err:
+	res->flags |= IORESOURCE_DISABLED;
+#endif
+}
 
 int acpi_pci_probe_root_resources(struct acpi_pci_root_info *info)
 {
@@ -740,6 +768,9 @@ int acpi_pci_probe_root_resources(struct acpi_pci_root_info *info)
 			"no IO and memory resources present in _CRS\n");
 	else {
 		resource_list_for_each_entry_safe(entry, tmp, list) {
+			if (entry->res->flags & IORESOURCE_IO)
+				acpi_pci_root_remap_iospace(entry);
+
 			if (entry->res->flags & IORESOURCE_DISABLED)
 				resource_list_destroy_entry(entry);
 			else
@@ -811,6 +842,8 @@ static void acpi_pci_root_release_info(struct pci_host_bridge *bridge)
 
 	resource_list_for_each_entry(entry, &bridge->windows) {
 		res = entry->res;
+		if (res->flags & IORESOURCE_IO)
+			pci_unmap_iospace(res);
 		if (res->parent &&
 		    (res->flags & (IORESOURCE_MEM | IORESOURCE_IO)))
 			release_resource(res);
-- 
1.9.1

[PATCH V7 06/11] pci, acpi: Provide a way to assign bus domain number.

[Tomasz Nowicki]

This patch provides a way to set the ACPI domain number in PCI code.
pci_create_root_bus is called with NULL as parent in ACPI. This
ends up calling pci_bus_assign_domain_nr with a NULL parent.
So we define acpi_pci_bus_domain_nr() which is meant to retrieve
the PCI domain number based on 'struct pci_bus' in the ACPI way.
pci_bus_assign_domain_nr() is updated to call acpi_pci_bus_domain_nr()
and assign domain number on the root bus, in case of ACPI.

acpi_pci_bus_domain_nr function is stub for now.

While at it, for the sake of code clarity we put ACPI and DT domain
assign methods into the corresponding helpers.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
---
 drivers/pci/pci.c        | 11 +++++++++--
 include/linux/pci-acpi.h |  5 +++++
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index ff97a0b..a1d7bcf 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -19,6 +19,7 @@
 #include <linux/spinlock.h>
 #include <linux/string.h>
 #include <linux/log2.h>
+#include <linux/pci-acpi.h>
 #include <linux/pci-aspm.h>
 #include <linux/pm_wakeup.h>
 #include <linux/interrupt.h>
@@ -4918,7 +4919,7 @@ int pci_get_new_domain_nr(void)
 }
 
 #ifdef CONFIG_PCI_DOMAINS_GENERIC
-void pci_bus_assign_domain_nr(struct pci_bus *bus, struct device *parent)
+static int of_pci_bus_domain_nr(struct device *parent)
 {
 	static int use_dt_domains = -1;
 	int domain = -1;
@@ -4962,7 +4963,13 @@ void pci_bus_assign_domain_nr(struct pci_bus *bus, struct device *parent)
 		domain = -1;
 	}
 
-	bus->domain_nr = domain;
+	return domain;
+}
+
+void pci_bus_assign_domain_nr(struct pci_bus *bus, struct device *parent)
+{
+	bus->domain_nr = acpi_disabled ? of_pci_bus_domain_nr(parent) :
+					 acpi_pci_bus_domain_nr(bus);
 }
 #endif
 #endif
diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
index 89ab057..09f9f02 100644
--- a/include/linux/pci-acpi.h
+++ b/include/linux/pci-acpi.h
@@ -111,6 +111,11 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
 static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
 #endif	/* CONFIG_ACPI */
 
+static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
+{
+	return 0;
+}
+
 #ifdef CONFIG_ACPI_APEI
 extern bool aer_acpi_firmware_first(void);
 #else
-- 
1.9.1

[PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Tomasz Nowicki]

This patch provides a way to set the ACPI companion in PCI code.
We define acpi_pci_set_companion() to set the ACPI companion pointer and
call it from PCI core code. The function is stub for now.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
---
 drivers/pci/probe.c      | 2 ++
 include/linux/pci-acpi.h | 4 ++++
 2 files changed, 6 insertions(+)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 8004f67..fb0b752 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -12,6 +12,7 @@
 #include <linux/slab.h>
 #include <linux/module.h>
 #include <linux/cpumask.h>
+#include <linux/pci-acpi.h>
 #include <linux/pci-aspm.h>
 #include <linux/aer.h>
 #include <linux/acpi.h>
@@ -2141,6 +2142,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
 	bridge->dev.parent = parent;
 	bridge->dev.release = pci_release_host_bridge_dev;
 	dev_set_name(&bridge->dev, "pci%04x:%02x", pci_domain_nr(b), bus);
+	acpi_pci_set_companion(bridge);
 	error = pcibios_root_bridge_prepare(bridge);
 	if (error) {
 		kfree(bridge);
diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
index 09f9f02..1baa515 100644
--- a/include/linux/pci-acpi.h
+++ b/include/linux/pci-acpi.h
@@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
 static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
 #endif	/* CONFIG_ACPI */
 
+static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
+{
+}
+
 static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
 {
 	return 0;
-- 
1.9.1

[PATCH V7 08/11] pci, acpi: Support for ACPI based generic PCI host controller

[Tomasz Nowicki]

This patch is going to implement generic PCI host controller for
ACPI world, similar to what pci-host-generic.c driver does for DT world.

All such drivers, which we have seen so far, were implemented within
arch/ directory since they had some arch assumptions (x86 and ia64).
However, they all are doing similar thing, so it makes sense to find
some common code and abstract it into the generic driver.

In order to handle PCI config space regions properly, we define new
MCFG interface which does sanity checks on MCFG table and keeps its
root pointer. User is able to lookup MCFG regions based on that root
pointer and specified domain:bus_start:bus_end touple. We are using
pci_mmcfg_late_init old prototype to avoid another function name.

The implementation of pci_acpi_scan_root() looks up the MCFG entries
and sets up a new mapping (regions are not mapped until host controller ask
for it). Generic PCI functions are used for accessing config space.
Driver selects PCI_ECAM and uses functions from drivers/pci/ecam.h
to create and access ECAM mappings.

As mentioned in Kconfig help section, ACPI_PCI_HOST_GENERIC choice
should be made on a per-architecture basis.

Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
Signed-off-by: Jayachandran C <jchandra@broadcom.com>
---
 drivers/acpi/Kconfig            |   8 +++
 drivers/acpi/Makefile           |   1 +
 drivers/acpi/pci_mcfg.c         |  97 ++++++++++++++++++++++++++
 drivers/acpi/pci_root_generic.c | 149 ++++++++++++++++++++++++++++++++++++++++
 drivers/pci/ecam.h              |   5 ++
 include/linux/pci-acpi.h        |   5 ++
 include/linux/pci.h             |   5 +-
 7 files changed, 269 insertions(+), 1 deletion(-)
 create mode 100644 drivers/acpi/pci_mcfg.c
 create mode 100644 drivers/acpi/pci_root_generic.c

diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
index 183ffa3..44afc76 100644
--- a/drivers/acpi/Kconfig
+++ b/drivers/acpi/Kconfig
@@ -346,6 +346,14 @@ config ACPI_PCI_SLOT
 	  i.e., segment/bus/device/function tuples, with physical slots in
 	  the system.  If you are unsure, say N.
 
+config ACPI_PCI_HOST_GENERIC
+	bool
+	select PCI_ECAM
+	help
+	  Select this config option from the architecture Kconfig,
+	  if it is preferred to enable ACPI PCI host controller driver which
+	  has no arch-specific assumptions.
+
 config X86_PM_TIMER
 	bool "Power Management Timer Support" if EXPERT
 	depends on X86
diff --git a/drivers/acpi/Makefile b/drivers/acpi/Makefile
index 81e5cbc..627a2b7 100644
--- a/drivers/acpi/Makefile
+++ b/drivers/acpi/Makefile
@@ -40,6 +40,7 @@ acpi-$(CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC) += processor_pdc.o
 acpi-y				+= ec.o
 acpi-$(CONFIG_ACPI_DOCK)	+= dock.o
 acpi-y				+= pci_root.o pci_link.o pci_irq.o
+obj-$(CONFIG_ACPI_PCI_HOST_GENERIC)	+= pci_root_generic.o pci_mcfg.o
 acpi-y				+= acpi_lpss.o acpi_apd.o
 acpi-y				+= acpi_platform.o
 acpi-y				+= acpi_pnp.o
diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c
new file mode 100644
index 0000000..373d079
--- /dev/null
+++ b/drivers/acpi/pci_mcfg.c
@@ -0,0 +1,97 @@
+/*
+ * Copyright (C) 2016 Broadcom
+ *	Author: Jayachandran C <jchandra@broadcom.com>
+ * Copyright (C) 2016 Semihalf
+ * 	Author: Tomasz Nowicki <tn@semihalf.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation (the "GPL").
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License version 2 (GPLv2) for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 (GPLv2) along with this source code.
+ */
+#include <linux/kernel.h>
+#include <linux/pci.h>
+#include <linux/pci-acpi.h>
+
+#define PREFIX	"ACPI: "
+
+/* Root pointer to the mapped MCFG table */
+static struct acpi_table_mcfg *mcfg_table;
+
+#define MCFG_ENTRIES(mcfg_ptr)	(((mcfg_ptr)->header.length -		\
+				sizeof(struct acpi_table_mcfg)) /	\
+				sizeof(struct acpi_mcfg_allocation))
+
+static phys_addr_t pci_mcfg_lookup_static(u16 seg, u8 bus_start, u8 bus_end)
+{
+	struct acpi_mcfg_allocation *mptr;
+	int i;
+
+	if (!mcfg_table) {
+		pr_err(PREFIX "MCFG table not available, lookup failed\n");
+		return -ENXIO;
+	}
+
+	mptr = (struct acpi_mcfg_allocation *) &mcfg_table[1];
+
+	/*
+	 * We expect exact match, unless MCFG entry end bus covers more than
+	 * specified by caller.
+	 */
+	for (i = 0; i < MCFG_ENTRIES(mcfg_table); i++, mptr++) {
+		if (mptr->pci_segment == seg &&
+		    mptr->start_bus_number == bus_start &&
+		    mptr->end_bus_number >= bus_end) {
+			return mptr->address;
+		}
+	}
+
+	return -ENXIO;
+}
+
+phys_addr_t pci_mcfg_lookup(struct acpi_device *device, u16 seg,
+			    struct resource *bus_res)
+{
+	phys_addr_t addr;
+
+	addr = acpi_pci_root_get_mcfg_addr(device->handle);
+	if (addr)
+		return addr;
+
+	return pci_mcfg_lookup_static(seg, bus_res->start, bus_res->end);
+}
+
+static __init int pci_mcfg_parse(struct acpi_table_header *header)
+{
+	struct acpi_table_mcfg *mcfg;
+	int n;
+
+	if (!header)
+		return -EINVAL;
+
+	mcfg = (struct acpi_table_mcfg *)header;
+	n = MCFG_ENTRIES(mcfg);
+	if (n <= 0 || n > 255) {
+		pr_err(PREFIX "MCFG has incorrect entries (%d).\n", n);
+		return -EINVAL;
+	}
+
+	mcfg_table = mcfg;
+	pr_info(PREFIX "MCFG table loaded, %d entries detected\n", n);
+	return 0;
+}
+
+/* Interface called by ACPI - parse and save MCFG table */
+void __init pci_mmcfg_late_init(void)
+{
+	int err = acpi_table_parse(ACPI_SIG_MCFG, pci_mcfg_parse);
+	if (err)
+		pr_err(PREFIX "Failed to parse MCFG (%d)\n", err);
+}
diff --git a/drivers/acpi/pci_root_generic.c b/drivers/acpi/pci_root_generic.c
new file mode 100644
index 0000000..6f4940a
--- /dev/null
+++ b/drivers/acpi/pci_root_generic.c
@@ -0,0 +1,149 @@
+/*
+ * Copyright (C) 2016 Broadcom
+ *	Author: Jayachandran C <jchandra@broadcom.com>
+ * Copyright (C) 2016 Semihalf
+ * 	Author: Tomasz Nowicki <tn@semihalf.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation (the "GPL").
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License version 2 (GPLv2) for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * version 2 (GPLv2) along with this source code.
+ */
+#include <linux/kernel.h>
+#include <linux/pci.h>
+#include <linux/pci-acpi.h>
+#include <linux/slab.h>
+
+#include "../pci/ecam.h"
+
+#define PREFIX	"ACPI PCI: "
+
+/* ACPI info for generic ACPI PCI controller */
+struct acpi_pci_generic_root_info {
+	struct acpi_pci_root_info	common;
+	struct pci_config_window	*cfg;	/* config space mapping */
+};
+
+void acpi_pci_set_companion(struct pci_host_bridge *bridge)
+{
+	struct pci_config_window *cfg = bridge->bus->sysdata;
+
+	ACPI_COMPANION_SET(&bridge->dev, cfg->companion);
+}
+
+int acpi_pci_bus_domain_nr(struct pci_bus *bus)
+{
+	struct pci_config_window *cfg = bus->sysdata;
+
+	return cfg->domain;
+}
+
+/*
+ * Lookup the bus range for the domain in MCFG, and set up config space
+ * mapping.
+ */
+static int pci_acpi_setup_ecam_mapping(struct acpi_pci_root *root,
+				       struct acpi_pci_generic_root_info *ri)
+{
+	struct resource *bus_res = &root->secondary;
+	u16 seg = root->segment;
+	struct pci_config_window *cfg;
+	struct resource cfgres;
+	unsigned int bsz;
+	phys_addr_t addr;
+
+	addr = pci_mcfg_lookup(root->device, seg, bus_res);
+	if (IS_ERR_VALUE(addr)) {
+		pr_err(PREFIX"%04x:%pR MCFG region not found\n", seg, bus_res);
+		return addr;
+	}
+
+	bsz = 1 << pci_generic_ecam_ops.bus_shift;
+	cfgres.start = addr + bus_res->start * bsz;
+	cfgres.end = addr + (bus_res->end + 1) * bsz - 1;
+	cfgres.flags = IORESOURCE_MEM;
+	cfg = pci_ecam_create(&root->device->dev, &cfgres, bus_res,
+						  &pci_generic_ecam_ops);
+	if (IS_ERR(cfg)) {
+		pr_err("%04x:%pR error %ld mapping CAM\n", seg, bus_res,
+		       PTR_ERR(cfg));
+		return PTR_ERR(cfg);
+	}
+
+	cfg->domain = seg;
+	cfg->companion = root->device;
+	ri->cfg = cfg;
+	return 0;
+}
+
+/* release_info: free resrouces allocated by init_info */
+static void pci_acpi_generic_release_info(struct acpi_pci_root_info *ci)
+{
+	struct acpi_pci_generic_root_info *ri;
+
+	ri = container_of(ci, struct acpi_pci_generic_root_info, common);
+	pci_ecam_free(ri->cfg);
+	kfree(ri);
+}
+
+static struct acpi_pci_root_ops acpi_pci_root_ops = {
+	.release_info = pci_acpi_generic_release_info,
+};
+
+/* Interface called from ACPI code to setup PCI host controller */
+struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root)
+{
+	int node = acpi_get_node(root->device->handle);
+	struct acpi_pci_generic_root_info *ri;
+	struct pci_bus *bus, *child;
+	int err;
+
+	ri = kzalloc_node(sizeof(*ri), GFP_KERNEL, node);
+	if (!ri)
+		return NULL;
+
+	err = pci_acpi_setup_ecam_mapping(root, ri);
+	if (err)
+		return NULL;
+
+	acpi_pci_root_ops.pci_ops = &ri->cfg->ops->pci_ops;
+	bus = acpi_pci_root_create(root, &acpi_pci_root_ops, &ri->common,
+				   ri->cfg);
+	if (!bus)
+		return NULL;
+
+	pci_bus_size_bridges(bus);
+	pci_bus_assign_resources(bus);
+
+	list_for_each_entry(child, &bus->children, node)
+		pcie_bus_configure_settings(child);
+
+	return bus;
+}
+
+int raw_pci_read(unsigned int domain, unsigned int busn, unsigned int devfn,
+		 int reg, int len, u32 *val)
+{
+	struct pci_bus *bus = pci_find_bus(domain, busn);
+
+	if (!bus)
+		return PCIBIOS_DEVICE_NOT_FOUND;
+	return bus->ops->read(bus, devfn, reg, len, val);
+}
+
+int raw_pci_write(unsigned int domain, unsigned int busn, unsigned int devfn,
+		  int reg, int len, u32 val)
+{
+	struct pci_bus *bus = pci_find_bus(domain, busn);
+
+	if (!bus)
+		return PCIBIOS_DEVICE_NOT_FOUND;
+	return bus->ops->write(bus, devfn, reg, len, val);
+}
diff --git a/drivers/pci/ecam.h b/drivers/pci/ecam.h
index 1ad2176..1cccf57 100644
--- a/drivers/pci/ecam.h
+++ b/drivers/pci/ecam.h
@@ -45,6 +45,11 @@ struct pci_config_window {
 		void __iomem		*win;	/* 64-bit single mapping */
 		void __iomem		**winp; /* 32-bit per bus mapping */
 	};
+#ifdef CONFIG_ACPI_PCI_HOST_GENERIC
+	struct acpi_device		*companion; /* ACPI companion device */
+#endif
+	int				domain;
+
 };
 
 /* create and free for pci_config_window */
diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
index 1baa515..42ff844 100644
--- a/include/linux/pci-acpi.h
+++ b/include/linux/pci-acpi.h
@@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
 static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
 #endif	/* CONFIG_ACPI */
 
+#ifdef CONFIG_ACPI_PCI_HOST_GENERIC
+void acpi_pci_set_companion(struct pci_host_bridge *bridge);
+int acpi_pci_bus_domain_nr(struct pci_bus *bus);
+#else
 static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
 {
 }
@@ -119,6 +123,7 @@ static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
 {
 	return 0;
 }
+#endif
 
 #ifdef CONFIG_ACPI_APEI
 extern bool aer_acpi_firmware_first(void);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index d6ea6ce..b2e8886 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1722,7 +1722,10 @@ void pcibios_free_irq(struct pci_dev *dev);
 extern struct dev_pm_ops pcibios_pm_ops;
 #endif
 
-#ifdef CONFIG_PCI_MMCONFIG
+#if defined(CONFIG_PCI_MMCONFIG) || defined(CONFIG_ACPI_PCI_HOST_GENERIC)
+struct acpi_device;
+phys_addr_t pci_mcfg_lookup(struct acpi_device *device, u16 seg,
+			    struct resource *bus_res);
 void __init pci_mmcfg_early_init(void);
 void __init pci_mmcfg_late_init(void);
 #else
-- 
1.9.1

[PATCH V7 09/11] arm64, pci, acpi: ACPI support for legacy IRQs parsing and consolidation with DT code.

[Tomasz Nowicki]

To enable PCI legacy IRQs on platforms booting with ACPI, arch code
should include ACPI specific callbacks that parse and set-up the
device IRQ number, equivalent to the DT boot path. Owing to the current
ACPI core scan handlers implementation, ACPI PCI legacy IRQs bindings
cannot be parsed at device add time, since that would trigger ACPI scan
handlers ordering issues depending on how the ACPI tables are defined.

To solve this problem and consolidate FW PCI legacy IRQs parsing in
one single pcibios callback (pending final removal), this patch moves
DT PCI IRQ parsing to the pcibios_alloc_irq() callback (called by
PCI core code at device probe time) and adds ACPI PCI legacy IRQs
parsing to the same callback too, so that FW PCI legacy IRQs parsing
is confined in one single arch callback that can be easily removed
when code parsing PCI legacy IRQs is consolidated and moved to core
PCI code.

Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
Suggested-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
---
 arch/arm64/kernel/pci.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/kernel/pci.c b/arch/arm64/kernel/pci.c
index c72de66..15109c11 100644
--- a/arch/arm64/kernel/pci.c
+++ b/arch/arm64/kernel/pci.c
@@ -50,11 +50,16 @@ int pcibios_enable_device(struct pci_dev *dev, int mask)
 }
 
 /*
- * Try to assign the IRQ number from DT when adding a new device
+ * Try to assign the IRQ number when probing a new device
  */
-int pcibios_add_device(struct pci_dev *dev)
+int pcibios_alloc_irq(struct pci_dev *dev)
 {
-	dev->irq = of_irq_parse_and_map_pci(dev, 0, 0);
+	if (acpi_disabled)
+		dev->irq = of_irq_parse_and_map_pci(dev, 0, 0);
+#ifdef CONFIG_ACPI
+	else
+		return acpi_pci_irq_enable(dev);
+#endif
 
 	return 0;
 }
-- 
1.9.1

[PATCH V7 10/11] arm64, pci, acpi: Provide ACPI-specific prerequisites for PCI bus enumeration.

[Tomasz Nowicki]

ACPI requires to run acpi_pci_{add|remove}_bus while new PCI bus is created.
This allows to do some ACPI-specific additional configuration, like
PCI hotplug slot enumeration. In order to fulfill these requirements,
we implement arch-specific pcibios_{add|remove}_bus calls
and call acpi_pci_{add|remove}_bus from there.

Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
To: Catalin Marinas <catalin.marinas@arm.com>
To: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
To: Will Deacon <will.deacon@arm.com>
To: Arnd Bergmann <arnd@arndb.de>
---
 arch/arm64/kernel/pci.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/arm64/kernel/pci.c b/arch/arm64/kernel/pci.c
index 15109c11..eeec5f6 100644
--- a/arch/arm64/kernel/pci.c
+++ b/arch/arm64/kernel/pci.c
@@ -17,6 +17,7 @@
 #include <linux/mm.h>
 #include <linux/of_pci.h>
 #include <linux/of_platform.h>
+#include <linux/pci-acpi.h>
 #include <linux/slab.h>
 
 /*
@@ -64,6 +65,16 @@ int pcibios_alloc_irq(struct pci_dev *dev)
 	return 0;
 }
 
+void pcibios_add_bus(struct pci_bus *bus)
+{
+	acpi_pci_add_bus(bus);
+}
+
+void pcibios_remove_bus(struct pci_bus *bus)
+{
+	acpi_pci_remove_bus(bus);
+}
+
 /*
  * raw_pci_read/write - Platform-specific PCI config space access.
  */
-- 
1.9.1

[PATCH V7 11/11] arm64, pci, acpi: Start using ACPI based PCI host controller driver for ARM64.

[Tomasz Nowicki]

It is perfectly fine to use ACPI_PCI_HOST_GENERIC for ARM64,
so lets get rid of PCI init and RAW ACPI accessor empty stubs
and go with full-blown PCI host controller driver.

Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
To: Catalin Marinas <catalin.marinas@arm.com>
To: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
To: Will Deacon <will.deacon@arm.com>
To: Arnd Bergmann <arnd@arndb.de>
---
 arch/arm64/Kconfig      |  1 +
 arch/arm64/kernel/pci.c | 24 ------------------------
 2 files changed, 1 insertion(+), 24 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 4f43622..1bded87 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2,6 +2,7 @@ config ARM64
 	def_bool y
 	select ACPI_CCA_REQUIRED if ACPI
 	select ACPI_GENERIC_GSI if ACPI
+	select ACPI_PCI_HOST_GENERIC if ACPI
 	select ACPI_REDUCED_HARDWARE_ONLY if ACPI
 	select ARCH_HAS_DEVMEM_IS_ALLOWED
 	select ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE
diff --git a/arch/arm64/kernel/pci.c b/arch/arm64/kernel/pci.c
index eeec5f6..e484c91 100644
--- a/arch/arm64/kernel/pci.c
+++ b/arch/arm64/kernel/pci.c
@@ -74,27 +74,3 @@ void pcibios_remove_bus(struct pci_bus *bus)
 {
 	acpi_pci_remove_bus(bus);
 }
-
-/*
- * raw_pci_read/write - Platform-specific PCI config space access.
- */
-int raw_pci_read(unsigned int domain, unsigned int bus,
-		  unsigned int devfn, int reg, int len, u32 *val)
-{
-	return -ENXIO;
-}
-
-int raw_pci_write(unsigned int domain, unsigned int bus,
-		unsigned int devfn, int reg, int len, u32 val)
-{
-	return -ENXIO;
-}
-
-#ifdef CONFIG_ACPI
-/* Root bridge scanning */
-struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root)
-{
-	/* TODO: Should be revisited when implementing PCI on ACPI */
-	return NULL;
-}
-#endif
-- 
1.9.1

Re: [PATCH V7 08/11] pci, acpi: Support for ACPI based generic PCI host controller

[Rafael J. Wysocki]

On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> This patch is going to implement generic PCI host controller for
> ACPI world, similar to what pci-host-generic.c driver does for DT world.
>
> All such drivers, which we have seen so far, were implemented within
> arch/ directory since they had some arch assumptions (x86 and ia64).
> However, they all are doing similar thing, so it makes sense to find
> some common code and abstract it into the generic driver.
>
> In order to handle PCI config space regions properly, we define new
> MCFG interface which does sanity checks on MCFG table and keeps its
> root pointer. User is able to lookup MCFG regions based on that root
> pointer and specified domain:bus_start:bus_end touple. We are using
> pci_mmcfg_late_init old prototype to avoid another function name.
>
> The implementation of pci_acpi_scan_root() looks up the MCFG entries
> and sets up a new mapping (regions are not mapped until host controller ask
> for it). Generic PCI functions are used for accessing config space.
> Driver selects PCI_ECAM and uses functions from drivers/pci/ecam.h
> to create and access ECAM mappings.
>
> As mentioned in Kconfig help section, ACPI_PCI_HOST_GENERIC choice
> should be made on a per-architecture basis.
>
> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> ---
>  drivers/acpi/Kconfig            |   8 +++
>  drivers/acpi/Makefile           |   1 +
>  drivers/acpi/pci_mcfg.c         |  97 ++++++++++++++++++++++++++
>  drivers/acpi/pci_root_generic.c | 149 ++++++++++++++++++++++++++++++++++++++++

Why do we need a new file?  Would there be any problem with adding
that code to pci_root.c?

Re: [PATCH V7 03/11] pci, of: Move the PCI I/O space management to PCI core code.

[Rafael J. Wysocki]

On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> No functional changes in this patch.
>
> PCI I/O space mapping code does not depend on OF, therefore it can be
> moved to PCI core code. This way we will be able to use it
> e.g. in ACPI PCI code.
>
> Suggested-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> CC: Arnd Bergmann <arnd@arndb.de>
> CC: Liviu Dudau <Liviu.Dudau@arm.com>
> ---
>  drivers/of/address.c       | 116 +--------------------------------------------
>  drivers/pci/pci.c          | 115 ++++++++++++++++++++++++++++++++++++++++++++
>  include/linux/of_address.h |   9 ----
>  include/linux/pci.h        |   5 ++
>  4 files changed, 121 insertions(+), 124 deletions(-)
>
> diff --git a/drivers/of/address.c b/drivers/of/address.c
> index 91a469d..0a553c0 100644
> --- a/drivers/of/address.c
> +++ b/drivers/of/address.c
> @@ -4,6 +4,7 @@
>  #include <linux/ioport.h>
>  #include <linux/module.h>
>  #include <linux/of_address.h>
> +#include <linux/pci.h>
>  #include <linux/pci_regs.h>
>  #include <linux/sizes.h>
>  #include <linux/slab.h>
> @@ -673,121 +674,6 @@ const __be32 *of_get_address(struct device_node *dev, int index, u64 *size,
>  }
>  EXPORT_SYMBOL(of_get_address);
>
> -#ifdef PCI_IOBASE
> -struct io_range {
> -       struct list_head list;
> -       phys_addr_t start;
> -       resource_size_t size;
> -};
> -
> -static LIST_HEAD(io_range_list);
> -static DEFINE_SPINLOCK(io_range_lock);
> -#endif
> -
> -/*
> - * Record the PCI IO range (expressed as CPU physical address + size).
> - * Return a negative value if an error has occured, zero otherwise
> - */
> -int __weak pci_register_io_range(phys_addr_t addr, resource_size_t size)
> -{
> -       int err = 0;
> -
> -#ifdef PCI_IOBASE
> -       struct io_range *range;
> -       resource_size_t allocated_size = 0;
> -
> -       /* check if the range hasn't been previously recorded */
> -       spin_lock(&io_range_lock);
> -       list_for_each_entry(range, &io_range_list, list) {
> -               if (addr >= range->start && addr + size <= range->start + size) {
> -                       /* range already registered, bail out */
> -                       goto end_register;
> -               }
> -               allocated_size += range->size;
> -       }
> -
> -       /* range not registed yet, check for available space */
> -       if (allocated_size + size - 1 > IO_SPACE_LIMIT) {
> -               /* if it's too big check if 64K space can be reserved */
> -               if (allocated_size + SZ_64K - 1 > IO_SPACE_LIMIT) {
> -                       err = -E2BIG;
> -                       goto end_register;
> -               }
> -
> -               size = SZ_64K;
> -               pr_warn("Requested IO range too big, new size set to 64K\n");
> -       }
> -
> -       /* add the range to the list */
> -       range = kzalloc(sizeof(*range), GFP_ATOMIC);
> -       if (!range) {
> -               err = -ENOMEM;
> -               goto end_register;
> -       }
> -
> -       range->start = addr;
> -       range->size = size;
> -
> -       list_add_tail(&range->list, &io_range_list);
> -
> -end_register:
> -       spin_unlock(&io_range_lock);
> -#endif
> -
> -       return err;
> -}
> -
> -phys_addr_t pci_pio_to_address(unsigned long pio)
> -{
> -       phys_addr_t address = (phys_addr_t)OF_BAD_ADDR;
> -
> -#ifdef PCI_IOBASE
> -       struct io_range *range;
> -       resource_size_t allocated_size = 0;
> -
> -       if (pio > IO_SPACE_LIMIT)
> -               return address;
> -
> -       spin_lock(&io_range_lock);
> -       list_for_each_entry(range, &io_range_list, list) {
> -               if (pio >= allocated_size && pio < allocated_size + range->size) {
> -                       address = range->start + pio - allocated_size;
> -                       break;
> -               }
> -               allocated_size += range->size;
> -       }
> -       spin_unlock(&io_range_lock);
> -#endif
> -
> -       return address;
> -}
> -
> -unsigned long __weak pci_address_to_pio(phys_addr_t address)
> -{
> -#ifdef PCI_IOBASE
> -       struct io_range *res;
> -       resource_size_t offset = 0;
> -       unsigned long addr = -1;
> -
> -       spin_lock(&io_range_lock);
> -       list_for_each_entry(res, &io_range_list, list) {
> -               if (address >= res->start && address < res->start + res->size) {
> -                       addr = address - res->start + offset;
> -                       break;
> -               }
> -               offset += res->size;
> -       }
> -       spin_unlock(&io_range_lock);
> -
> -       return addr;
> -#else
> -       if (address > IO_SPACE_LIMIT)
> -               return (unsigned long)-1;
> -
> -       return (unsigned long) address;
> -#endif
> -}
> -
>  static int __of_address_to_resource(struct device_node *dev,
>                 const __be32 *addrp, u64 size, unsigned int flags,
>                 const char *name, struct resource *r)
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index 25e0327..bc0c914 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -3021,6 +3021,121 @@ int pci_request_regions_exclusive(struct pci_dev *pdev, const char *res_name)
>  }
>  EXPORT_SYMBOL(pci_request_regions_exclusive);
>
> +#ifdef PCI_IOBASE
> +struct io_range {
> +       struct list_head list;
> +       phys_addr_t start;
> +       resource_size_t size;
> +};
> +
> +static LIST_HEAD(io_range_list);
> +static DEFINE_SPINLOCK(io_range_lock);
> +#endif
> +
> +/*
> + * Record the PCI IO range (expressed as CPU physical address + size).
> + * Return a negative value if an error has occured, zero otherwise
> + */
> +int __weak pci_register_io_range(phys_addr_t addr, resource_size_t size)
> +{
> +       int err = 0;
> +
> +#ifdef PCI_IOBASE

I understand that this moves code around, but those in-function
#ifdefs aren't nice.  Any chance to get rid of them but putting whole
functions under the #ifdef?

Re: [PATCH V7 08/11] pci, acpi: Support for ACPI based generic PCI host controller

[Rafael J. Wysocki]

On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> This patch is going to implement generic PCI host controller for
> ACPI world, similar to what pci-host-generic.c driver does for DT world.
>
> All such drivers, which we have seen so far, were implemented within
> arch/ directory since they had some arch assumptions (x86 and ia64).
> However, they all are doing similar thing, so it makes sense to find
> some common code and abstract it into the generic driver.

Does it mean x86 and ia64 will now be able to use this code too?

> In order to handle PCI config space regions properly, we define new
> MCFG interface which does sanity checks on MCFG table and keeps its
> root pointer. User is able to lookup MCFG regions based on that root
> pointer and specified domain:bus_start:bus_end touple. We are using
> pci_mmcfg_late_init old prototype to avoid another function name.
>
> The implementation of pci_acpi_scan_root() looks up the MCFG entries
> and sets up a new mapping (regions are not mapped until host controller ask
> for it). Generic PCI functions are used for accessing config space.
> Driver selects PCI_ECAM and uses functions from drivers/pci/ecam.h
> to create and access ECAM mappings.
>
> As mentioned in Kconfig help section, ACPI_PCI_HOST_GENERIC choice
> should be made on a per-architecture basis.

If that code really is generic and there will be more than one
architecture using it ever, I think it'll be better for the
architectures that don't use it to set something like
ARCH_ACPI_PCI_HOST and whoever doesn't set that will use the generic
thing.  That'd be more logical at least IMO.

> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> ---
>  drivers/acpi/Kconfig            |   8 +++
>  drivers/acpi/Makefile           |   1 +
>  drivers/acpi/pci_mcfg.c         |  97 ++++++++++++++++++++++++++
>  drivers/acpi/pci_root_generic.c | 149 ++++++++++++++++++++++++++++++++++++++++
>  drivers/pci/ecam.h              |   5 ++
>  include/linux/pci-acpi.h        |   5 ++
>  include/linux/pci.h             |   5 +-
>  7 files changed, 269 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/acpi/pci_mcfg.c
>  create mode 100644 drivers/acpi/pci_root_generic.c
>
> diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
> index 183ffa3..44afc76 100644
> --- a/drivers/acpi/Kconfig
> +++ b/drivers/acpi/Kconfig
> @@ -346,6 +346,14 @@ config ACPI_PCI_SLOT
>           i.e., segment/bus/device/function tuples, with physical slots in
>           the system.  If you are unsure, say N.
>
> +config ACPI_PCI_HOST_GENERIC
> +       bool
> +       select PCI_ECAM
> +       help
> +         Select this config option from the architecture Kconfig,
> +         if it is preferred to enable ACPI PCI host controller driver which
> +         has no arch-specific assumptions.
> +
>  config X86_PM_TIMER
>         bool "Power Management Timer Support" if EXPERT
>         depends on X86
> diff --git a/drivers/acpi/Makefile b/drivers/acpi/Makefile
> index 81e5cbc..627a2b7 100644
> --- a/drivers/acpi/Makefile
> +++ b/drivers/acpi/Makefile
> @@ -40,6 +40,7 @@ acpi-$(CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC) += processor_pdc.o
>  acpi-y                         += ec.o
>  acpi-$(CONFIG_ACPI_DOCK)       += dock.o
>  acpi-y                         += pci_root.o pci_link.o pci_irq.o
> +obj-$(CONFIG_ACPI_PCI_HOST_GENERIC)    += pci_root_generic.o pci_mcfg.o
>  acpi-y                         += acpi_lpss.o acpi_apd.o
>  acpi-y                         += acpi_platform.o
>  acpi-y                         += acpi_pnp.o
> diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c
> new file mode 100644
> index 0000000..373d079
> --- /dev/null
> +++ b/drivers/acpi/pci_mcfg.c
> @@ -0,0 +1,97 @@
> +/*
> + * Copyright (C) 2016 Broadcom
> + *     Author: Jayachandran C <jchandra@broadcom.com>
> + * Copyright (C) 2016 Semihalf
> + *     Author: Tomasz Nowicki <tn@semihalf.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License, version 2, as
> + * published by the Free Software Foundation (the "GPL").
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * General Public License version 2 (GPLv2) for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * version 2 (GPLv2) along with this source code.
> + */
> +#include <linux/kernel.h>
> +#include <linux/pci.h>
> +#include <linux/pci-acpi.h>
> +
> +#define PREFIX "ACPI: "

If that is a new file (and I'm totally unconvinced about the need for
it), can we simply define a pr_fmt() here as all messages in it seem
to be printed by the pr_* functions?

Re: [PATCH V7 05/11] acpi, pci: Support IO resources when parsing PCI host bridge resources.

[Rafael J. Wysocki]

On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> Platforms that have memory mapped IO port (such as ARM64) need special
> handling for PCI I/O resources. For host bridge's resource probing case
> these resources need to be fixed up with pci_register_io_range/pci_remap_iospace etc.
>
> The same I/O resources need to be released after hotplug
> removal so that it can be re-added back by the pci_remap_iospace
> function during insertion. As a consequence we unmap I/O resources
> with pci_unmap_iospace when we release host bridge resources.
>
> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> ---
>  drivers/acpi/pci_root.c | 33 +++++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
>
> diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
> index ae3fe4e..cb3071d 100644
> --- a/drivers/acpi/pci_root.c
> +++ b/drivers/acpi/pci_root.c
> @@ -719,6 +719,34 @@ next:
>                         resource_list_add_tail(entry, resources);
>         }
>  }
> +static void acpi_pci_root_remap_iospace(struct resource_entry *entry)
> +{
> +#ifdef PCI_IOBASE

Same comment about the #ifdefs as in the other patch.

> +       struct resource *res = entry->res;
> +       resource_size_t cpu_addr = res->start;
> +       resource_size_t pci_addr = cpu_addr - entry->offset;
> +       resource_size_t length = resource_size(res);
> +       unsigned long port;
> +
> +       if (pci_register_io_range(cpu_addr, length))
> +               goto err;
> +
> +       port = pci_address_to_pio(cpu_addr);
> +       if (port == (unsigned long)-1)
> +               goto err;
> +
> +       res->start = port;
> +       res->end = port + length - 1;
> +       entry->offset = port - pci_addr;
> +
> +       if (pci_remap_iospace(res, cpu_addr) < 0)
> +               goto err;

An empty line here?

> +       pr_info("Remapped I/O %pa to %pR\n", &cpu_addr, res);
> +       return;
> +err:
> +       res->flags |= IORESOURCE_DISABLED;
> +#endif
> +}
>
>  int acpi_pci_probe_root_resources(struct acpi_pci_root_info *info)
>  {
> @@ -740,6 +768,9 @@ int acpi_pci_probe_root_resources(struct acpi_pci_root_info *info)
>                         "no IO and memory resources present in _CRS\n");
>         else {
>                 resource_list_for_each_entry_safe(entry, tmp, list) {
> +                       if (entry->res->flags & IORESOURCE_IO)
> +                               acpi_pci_root_remap_iospace(entry);
> +
>                         if (entry->res->flags & IORESOURCE_DISABLED)
>                                 resource_list_destroy_entry(entry);
>                         else
> @@ -811,6 +842,8 @@ static void acpi_pci_root_release_info(struct pci_host_bridge *bridge)
>
>         resource_list_for_each_entry(entry, &bridge->windows) {
>                 res = entry->res;
> +               if (res->flags & IORESOURCE_IO)
> +                       pci_unmap_iospace(res);
>                 if (res->parent &&
>                     (res->flags & (IORESOURCE_MEM | IORESOURCE_IO)))
>                         release_resource(res);
> --

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Rafael J. Wysocki]

On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> This patch provides a way to set the ACPI companion in PCI code.
> We define acpi_pci_set_companion() to set the ACPI companion pointer and
> call it from PCI core code. The function is stub for now.
>
> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> ---
>  drivers/pci/probe.c      | 2 ++
>  include/linux/pci-acpi.h | 4 ++++
>  2 files changed, 6 insertions(+)
>
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index 8004f67..fb0b752 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -12,6 +12,7 @@
>  #include <linux/slab.h>
>  #include <linux/module.h>
>  #include <linux/cpumask.h>
> +#include <linux/pci-acpi.h>
>  #include <linux/pci-aspm.h>
>  #include <linux/aer.h>
>  #include <linux/acpi.h>
> @@ -2141,6 +2142,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
>         bridge->dev.parent = parent;
>         bridge->dev.release = pci_release_host_bridge_dev;
>         dev_set_name(&bridge->dev, "pci%04x:%02x", pci_domain_nr(b), bus);
> +       acpi_pci_set_companion(bridge);

Yes, we'll probably add something similar here.

Do I think now is the right time to do that?  No.

>         error = pcibios_root_bridge_prepare(bridge);
>         if (error) {
>                 kfree(bridge);
> diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
> index 09f9f02..1baa515 100644
> --- a/include/linux/pci-acpi.h
> +++ b/include/linux/pci-acpi.h
> @@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
>  static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
>  #endif /* CONFIG_ACPI */
>
> +static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
> +{
> +}
> +
>  static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
>  {
>         return 0;
> --

Honestly, to me it looks like this series is trying very hard to avoid
doing any PCI host bridge configuration stuff from arch/arm64/
although (a) that might be simpler and (b) it would allow us to
identify the code that's common between *all* architectures using ACPI
support for host bridge configuration and to move *that* to a common
place later.  As done here it seems to be following the "ARM64 is
generic and the rest of the world is special" line which isn't really
helpful.

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Rafael J. Wysocki]

On Tue, May 10, 2016 at 8:37 PM, Rafael J. Wysocki <rafael@kernel.org> wrote:
> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
>> This patch provides a way to set the ACPI companion in PCI code.
>> We define acpi_pci_set_companion() to set the ACPI companion pointer and
>> call it from PCI core code. The function is stub for now.
>>
>> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
>> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
>> ---
>>  drivers/pci/probe.c      | 2 ++
>>  include/linux/pci-acpi.h | 4 ++++
>>  2 files changed, 6 insertions(+)
>>
>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> index 8004f67..fb0b752 100644
>> --- a/drivers/pci/probe.c
>> +++ b/drivers/pci/probe.c
>> @@ -12,6 +12,7 @@
>>  #include <linux/slab.h>
>>  #include <linux/module.h>
>>  #include <linux/cpumask.h>
>> +#include <linux/pci-acpi.h>
>>  #include <linux/pci-aspm.h>
>>  #include <linux/aer.h>
>>  #include <linux/acpi.h>
>> @@ -2141,6 +2142,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
>>         bridge->dev.parent = parent;
>>         bridge->dev.release = pci_release_host_bridge_dev;
>>         dev_set_name(&bridge->dev, "pci%04x:%02x", pci_domain_nr(b), bus);
>> +       acpi_pci_set_companion(bridge);
>
> Yes, we'll probably add something similar here.
>
> Do I think now is the right time to do that?  No.
>
>>         error = pcibios_root_bridge_prepare(bridge);
>>         if (error) {
>>                 kfree(bridge);
>> diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
>> index 09f9f02..1baa515 100644
>> --- a/include/linux/pci-acpi.h
>> +++ b/include/linux/pci-acpi.h
>> @@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
>>  static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
>>  #endif /* CONFIG_ACPI */
>>
>> +static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
>> +{
>> +}
>> +
>>  static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
>>  {
>>         return 0;
>> --
>
> Honestly, to me it looks like this series is trying very hard to avoid
> doing any PCI host bridge configuration stuff from arch/arm64/
> although (a) that might be simpler and (b) it would allow us to
> identify the code that's common between *all* architectures using ACPI
> support for host bridge configuration and to move *that* to a common
> place later.  As done here it seems to be following the "ARM64 is
> generic and the rest of the world is special" line which isn't really
> helpful.

Speaking of which, at least one of the reasons why the ACPI PCI host
bridge thing on x86 and ia64 went to the arch code was to avoid
explicit references to ACPI-specific data types and related #ifdeffery
in the generic PCI code and data structures.  If you are going to add
those references now anyway, that reason is not relevant any more and
all of that can just be reworked to refer to ACPI explicitly.

Re: [PATCH V7 03/11] pci, of: Move the PCI I/O space management to PCI core code.

[Tomasz Nowicki]

On 10.05.2016 19:59, Rafael J. Wysocki wrote:
> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
>> No functional changes in this patch.
>>
>> PCI I/O space mapping code does not depend on OF, therefore it can be
>> moved to PCI core code. This way we will be able to use it
>> e.g. in ACPI PCI code.
>>
>> Suggested-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
>> CC: Arnd Bergmann <arnd@arndb.de>
>> CC: Liviu Dudau <Liviu.Dudau@arm.com>
>> ---
>>   drivers/of/address.c       | 116 +--------------------------------------------
>>   drivers/pci/pci.c          | 115 ++++++++++++++++++++++++++++++++++++++++++++
>>   include/linux/of_address.h |   9 ----
>>   include/linux/pci.h        |   5 ++
>>   4 files changed, 121 insertions(+), 124 deletions(-)
>>
>> diff --git a/drivers/of/address.c b/drivers/of/address.c
>> index 91a469d..0a553c0 100644
>> --- a/drivers/of/address.c
>> +++ b/drivers/of/address.c
>> @@ -4,6 +4,7 @@
>>   #include <linux/ioport.h>
>>   #include <linux/module.h>
>>   #include <linux/of_address.h>
>> +#include <linux/pci.h>
>>   #include <linux/pci_regs.h>
>>   #include <linux/sizes.h>
>>   #include <linux/slab.h>
>> @@ -673,121 +674,6 @@ const __be32 *of_get_address(struct device_node *dev, int index, u64 *size,
>>   }
>>   EXPORT_SYMBOL(of_get_address);
>>
>> -#ifdef PCI_IOBASE
>> -struct io_range {
>> -       struct list_head list;
>> -       phys_addr_t start;
>> -       resource_size_t size;
>> -};
>> -
>> -static LIST_HEAD(io_range_list);
>> -static DEFINE_SPINLOCK(io_range_lock);
>> -#endif
>> -
>> -/*
>> - * Record the PCI IO range (expressed as CPU physical address + size).
>> - * Return a negative value if an error has occured, zero otherwise
>> - */
>> -int __weak pci_register_io_range(phys_addr_t addr, resource_size_t size)
>> -{
>> -       int err = 0;
>> -
>> -#ifdef PCI_IOBASE
>> -       struct io_range *range;
>> -       resource_size_t allocated_size = 0;
>> -
>> -       /* check if the range hasn't been previously recorded */
>> -       spin_lock(&io_range_lock);
>> -       list_for_each_entry(range, &io_range_list, list) {
>> -               if (addr >= range->start && addr + size <= range->start + size) {
>> -                       /* range already registered, bail out */
>> -                       goto end_register;
>> -               }
>> -               allocated_size += range->size;
>> -       }
>> -
>> -       /* range not registed yet, check for available space */
>> -       if (allocated_size + size - 1 > IO_SPACE_LIMIT) {
>> -               /* if it's too big check if 64K space can be reserved */
>> -               if (allocated_size + SZ_64K - 1 > IO_SPACE_LIMIT) {
>> -                       err = -E2BIG;
>> -                       goto end_register;
>> -               }
>> -
>> -               size = SZ_64K;
>> -               pr_warn("Requested IO range too big, new size set to 64K\n");
>> -       }
>> -
>> -       /* add the range to the list */
>> -       range = kzalloc(sizeof(*range), GFP_ATOMIC);
>> -       if (!range) {
>> -               err = -ENOMEM;
>> -               goto end_register;
>> -       }
>> -
>> -       range->start = addr;
>> -       range->size = size;
>> -
>> -       list_add_tail(&range->list, &io_range_list);
>> -
>> -end_register:
>> -       spin_unlock(&io_range_lock);
>> -#endif
>> -
>> -       return err;
>> -}
>> -
>> -phys_addr_t pci_pio_to_address(unsigned long pio)
>> -{
>> -       phys_addr_t address = (phys_addr_t)OF_BAD_ADDR;
>> -
>> -#ifdef PCI_IOBASE
>> -       struct io_range *range;
>> -       resource_size_t allocated_size = 0;
>> -
>> -       if (pio > IO_SPACE_LIMIT)
>> -               return address;
>> -
>> -       spin_lock(&io_range_lock);
>> -       list_for_each_entry(range, &io_range_list, list) {
>> -               if (pio >= allocated_size && pio < allocated_size + range->size) {
>> -                       address = range->start + pio - allocated_size;
>> -                       break;
>> -               }
>> -               allocated_size += range->size;
>> -       }
>> -       spin_unlock(&io_range_lock);
>> -#endif
>> -
>> -       return address;
>> -}
>> -
>> -unsigned long __weak pci_address_to_pio(phys_addr_t address)
>> -{
>> -#ifdef PCI_IOBASE
>> -       struct io_range *res;
>> -       resource_size_t offset = 0;
>> -       unsigned long addr = -1;
>> -
>> -       spin_lock(&io_range_lock);
>> -       list_for_each_entry(res, &io_range_list, list) {
>> -               if (address >= res->start && address < res->start + res->size) {
>> -                       addr = address - res->start + offset;
>> -                       break;
>> -               }
>> -               offset += res->size;
>> -       }
>> -       spin_unlock(&io_range_lock);
>> -
>> -       return addr;
>> -#else
>> -       if (address > IO_SPACE_LIMIT)
>> -               return (unsigned long)-1;
>> -
>> -       return (unsigned long) address;
>> -#endif
>> -}
>> -
>>   static int __of_address_to_resource(struct device_node *dev,
>>                  const __be32 *addrp, u64 size, unsigned int flags,
>>                  const char *name, struct resource *r)
>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>> index 25e0327..bc0c914 100644
>> --- a/drivers/pci/pci.c
>> +++ b/drivers/pci/pci.c
>> @@ -3021,6 +3021,121 @@ int pci_request_regions_exclusive(struct pci_dev *pdev, const char *res_name)
>>   }
>>   EXPORT_SYMBOL(pci_request_regions_exclusive);
>>
>> +#ifdef PCI_IOBASE
>> +struct io_range {
>> +       struct list_head list;
>> +       phys_addr_t start;
>> +       resource_size_t size;
>> +};
>> +
>> +static LIST_HEAD(io_range_list);
>> +static DEFINE_SPINLOCK(io_range_lock);
>> +#endif
>> +
>> +/*
>> + * Record the PCI IO range (expressed as CPU physical address + size).
>> + * Return a negative value if an error has occured, zero otherwise
>> + */
>> +int __weak pci_register_io_range(phys_addr_t addr, resource_size_t size)
>> +{
>> +       int err = 0;
>> +
>> +#ifdef PCI_IOBASE
>
> I understand that this moves code around, but those in-function
> #ifdefs aren't nice.  Any chance to get rid of them but putting whole
> functions under the #ifdef?
>

This is a __weak implementation, so assuming I would move #ifdef out of 
function I need to provide another empty __weak stub. I do not know 
which solution is more ugly. In any case we can do that cleanup separately.

Tomasz

Re: [PATCH V7 05/11] acpi, pci: Support IO resources when parsing PCI host bridge resources.

[Tomasz Nowicki]

On 10.05.2016 20:20, Rafael J. Wysocki wrote:
> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
>> Platforms that have memory mapped IO port (such as ARM64) need special
>> handling for PCI I/O resources. For host bridge's resource probing case
>> these resources need to be fixed up with pci_register_io_range/pci_remap_iospace etc.
>>
>> The same I/O resources need to be released after hotplug
>> removal so that it can be re-added back by the pci_remap_iospace
>> function during insertion. As a consequence we unmap I/O resources
>> with pci_unmap_iospace when we release host bridge resources.
>>
>> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
>> Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
>> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
>> ---
>>   drivers/acpi/pci_root.c | 33 +++++++++++++++++++++++++++++++++
>>   1 file changed, 33 insertions(+)
>>
>> diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
>> index ae3fe4e..cb3071d 100644
>> --- a/drivers/acpi/pci_root.c
>> +++ b/drivers/acpi/pci_root.c
>> @@ -719,6 +719,34 @@ next:
>>                          resource_list_add_tail(entry, resources);
>>          }
>>   }
>> +static void acpi_pci_root_remap_iospace(struct resource_entry *entry)
>> +{
>> +#ifdef PCI_IOBASE
>
> Same comment about the #ifdefs as in the other patch.

OK

>
>> +       struct resource *res = entry->res;
>> +       resource_size_t cpu_addr = res->start;
>> +       resource_size_t pci_addr = cpu_addr - entry->offset;
>> +       resource_size_t length = resource_size(res);
>> +       unsigned long port;
>> +
>> +       if (pci_register_io_range(cpu_addr, length))
>> +               goto err;
>> +
>> +       port = pci_address_to_pio(cpu_addr);
>> +       if (port == (unsigned long)-1)
>> +               goto err;
>> +
>> +       res->start = port;
>> +       res->end = port + length - 1;
>> +       entry->offset = port - pci_addr;
>> +
>> +       if (pci_remap_iospace(res, cpu_addr) < 0)
>> +               goto err;
>
> An empty line here?

yes, empty line would be nice here.

Tomasz

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Lorenzo Pieralisi]

On Tue, May 10, 2016 at 08:37:00PM +0200, Rafael J. Wysocki wrote:
> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> > This patch provides a way to set the ACPI companion in PCI code.
> > We define acpi_pci_set_companion() to set the ACPI companion pointer and
> > call it from PCI core code. The function is stub for now.
> >
> > Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> > Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> > ---
> >  drivers/pci/probe.c      | 2 ++
> >  include/linux/pci-acpi.h | 4 ++++
> >  2 files changed, 6 insertions(+)
> >
> > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > index 8004f67..fb0b752 100644
> > --- a/drivers/pci/probe.c
> > +++ b/drivers/pci/probe.c
> > @@ -12,6 +12,7 @@
> >  #include <linux/slab.h>
> >  #include <linux/module.h>
> >  #include <linux/cpumask.h>
> > +#include <linux/pci-acpi.h>
> >  #include <linux/pci-aspm.h>
> >  #include <linux/aer.h>
> >  #include <linux/acpi.h>
> > @@ -2141,6 +2142,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
> >         bridge->dev.parent = parent;
> >         bridge->dev.release = pci_release_host_bridge_dev;
> >         dev_set_name(&bridge->dev, "pci%04x:%02x", pci_domain_nr(b), bus);
> > +       acpi_pci_set_companion(bridge);
> 
> Yes, we'll probably add something similar here.
> 
> Do I think now is the right time to do that?  No.
> 
> >         error = pcibios_root_bridge_prepare(bridge);
> >         if (error) {
> >                 kfree(bridge);
> > diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
> > index 09f9f02..1baa515 100644
> > --- a/include/linux/pci-acpi.h
> > +++ b/include/linux/pci-acpi.h
> > @@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
> >  static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
> >  #endif /* CONFIG_ACPI */
> >
> > +static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
> > +{
> > +}
> > +
> >  static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
> >  {
> >         return 0;
> > --
> 
> Honestly, to me it looks like this series is trying very hard to avoid
> doing any PCI host bridge configuration stuff from arch/arm64/
> although (a) that might be simpler and (b) it would allow us to
> identify the code that's common between *all* architectures using ACPI
> support for host bridge configuration and to move *that* to a common
> place later.  As done here it seems to be following the "ARM64 is
> generic and the rest of the world is special" line which isn't really
> helpful.

I think patch [1-2] should be merged regardless (they may require minor
tweaks if we decide to move pci_acpi_scan_root() to arch/arm64 though,
for include files location). I guess you are referring to patch 8 in
your comments above, which boils down to deciding whether:

- pci_acpi_scan_root() (and unfortunately all the MCFG/ECAM handling that
  goes with it) should live in arch/arm64 or drivers/acpi

acpi_pci_bus_domain_nr() is a bit more problematic since it is meant
to be called from PCI core code (ARM64 selects PCI_DOMAINS_GENERIC for
DT and same kernel has to work with OF and ACPI selected) and it is
arch specific (because what we have in bus->sysdata is arch specific,
waiting for the domain number to be embedded in struct pci_host_bridge).

Your point is fair, I am not sure that moving the pci_acpi_scan_root()
to arch/arm64 would make things much simpler though, it is just a matter
of deciding where that code has to live.

How do you want us to proceed ?

Thanks,
Lorenzo

RE: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Gabriele Paoloni]

Hi Tomasz

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> owner@vger.kernel.org] On Behalf Of Tomasz Nowicki
> Sent: 10 May 2016 16:20
> To: helgaas@kernel.org; arnd@arndb.de; will.deacon@arm.com;
> catalin.marinas@arm.com; rafael@kernel.org; hanjun.guo@linaro.org;
> Lorenzo.Pieralisi@arm.com; okaya@codeaurora.org; jchandra@broadcom.com
> Cc: robert.richter@caviumnetworks.com; mw@semihalf.com;
> Liviu.Dudau@arm.com; ddaney@caviumnetworks.com; Wangyijing;
> Suravee.Suthikulpanit@amd.com; msalter@redhat.com; linux-
> pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
> acpi@vger.kernel.org; linux-kernel@vger.kernel.org; linaro-
> acpi@lists.linaro.org; jcm@redhat.com; andrea.gallo@linaro.org;
> dhdang@apm.com; jeremy.linton@arm.com; liudongdong (C);
> cov@codeaurora.org; Tomasz Nowicki
> Subject: [PATCH V7 00/11] Support for generic ACPI based PCI host
> controller
> 
> From the functionality point of view this series may be split into the
> following logic parts:
> 1. New ECAM API and update for users of the pci-host-common API
> 2. Necessary fixes as the preparation for using driver on ARM64.
> 3. Use new MCFG interface and implement generic ACPI based PCI host
> controller driver.
> 4. Enable above driver on ARM64
> 
> Patches has been built on top of 4.6-rc7 and can be found here:
> git@github.com:semihalf-nowicki-tomasz/linux.git (pci-acpi-v7)
> 
> This has been tested on Cavium ThunderX server. Any help in reviewing
> and
> testing is very appreciated.
> 
> v6 -> v7
> - drop quirks handling

Maybe I missed something in the v6 discussion thread; when was it
decided to drop quirk handling?

I think it is important to have this in place to accommodate different
vendors. If the intention is to keep this patchset "clean" maybe
we can add it as a separate patch on top later on...

What’s your view?

Thanks

Gab


> - changes for ACPI companion and domain number assignment approach
> - implement arch pcibios_{add|remove}_bus and call
> acpi_pci_{add|remove}_bus from there
> - cleanups around nomenclature
> - use resources oriented API for ECAM
> - fix for based address calculation before mapping ECAM region
> - remove useless lock for MCFG lookup
> - move MCFG stuff to separated file pci_mcfg.c
> - drop MCFG entries caching
> - rebase against 4.6-rc7
> 
> v5 -> v6
> - drop idea of x86 MMCONFIG code refactoring
> - integrate JC's patches which introduce new ECAM API:
>   https://lkml.org/lkml/2016/4/11/907
>   git: https://github.com/jchandra-brcm/linux/ (arm64-acpi-pci-v3)
> - integrate Sinan's fix for releasing IO resources, see patch [06/13]
> - added ACPI support for ThunderX ECAM and PEM drivers
> - rebase against 4.6-rc2
> 
> v4 -> v5
> - drop MCFG refactoring group patches 1-6 from series v4 and integrate
> Jayachandran's patch
>   https://patchwork.ozlabs.org/patch/575525/
> - rewrite PCI legacy IRQs allocation
> - squash two patches 11 and 12 from series v4, fixed bisection issue
> - changelog improvements
> - rebase against 4.5-rc3
> 
> v3 -> v4
> - drop Jiang's fix
> http://lkml.iu.edu/hypermail/linux/kernel/1601.1/04318.html
> - add Lorenzo's fix patch 19/24
> - ACPI PCI bus domain number assigning cleanup
> - change resource management, we now claim and reassign resources
> - improvements for applying quirks
> - drop Matthew's http://www.spinics.net/lists/linux-pci/msg45950.html
> dependency
> - rebase against 4.5-rc1
> 
> v2 -> v3
> - fix legacy IRQ assigning and IO ports registration
> - remove reference to arch specific companion device for ia64
> - move ACPI PCI host controller driver to pci_root.c
> - drop generic domain assignment for x86 and ia64 as I am not
>   able to run all necessary test variants
> - drop patch which cleaned legacy IRQ assignment since it belongs to
>   Mathew's series:
>   https://patchwork.ozlabs.org/patch/557504/
> - extend MCFG quirk code
> - rebase against 4.4
> 
> v1 -> v2
> - move non-arch specific piece of code to dirver/acpi/ directory
> - fix IO resource handling
> - introduce PCI config accessors quirks matching
> - moved ACPI_COMPANION_SET to generic code
> 
> v1 - https://lkml.org/lkml/2015/10/27/504
> v2 - https://lkml.org/lkml/2015/12/16/246
> v3 - http://lkml.iu.edu/hypermail/linux/kernel/1601.1/04308.html
> v4 - https://lkml.org/lkml/2016/2/4/646
> v5 - https://lkml.org/lkml/2016/2/16/426
> v6 - https://lkml.org/lkml/2016/4/15/594
> 
> Jayachandran C (2):
>   PCI: Provide common functions for ECAM mapping
>   PCI: generic, thunder: update to use generic ECAM API
> 
> Tomasz Nowicki (9):
>   pci, of: Move the PCI I/O space management to PCI core code.
>   pci: Add new function to unmap IO resources.
>   acpi, pci: Support IO resources when parsing PCI host bridge
>     resources.
>   pci, acpi: Provide a way to assign bus domain number.
>   pci, acpi: Handle ACPI companion assignment.
>   pci, acpi: Support for ACPI based generic PCI host controller
>   arm64, pci, acpi: ACPI support for legacy IRQs parsing and
>     consolidation with DT code.
>   arm64, pci, acpi: Provide ACPI-specific prerequisites for PCI bus
>     enumeration.
>   arm64, pci, acpi: Start using ACPI based PCI host controller driver
>     for ARM64.
> 
>  arch/arm64/Kconfig                  |   1 +
>  arch/arm64/kernel/pci.c             |  34 +++-----
>  drivers/acpi/Kconfig                |   8 ++
>  drivers/acpi/Makefile               |   1 +
>  drivers/acpi/pci_mcfg.c             |  97 ++++++++++++++++++++++
>  drivers/acpi/pci_root.c             |  33 ++++++++
>  drivers/acpi/pci_root_generic.c     | 149
> +++++++++++++++++++++++++++++++++
>  drivers/of/address.c                | 116 +-------------------------
>  drivers/pci/Kconfig                 |   3 +
>  drivers/pci/Makefile                |   2 +
>  drivers/pci/ecam.c                  | 161
> ++++++++++++++++++++++++++++++++++++
>  drivers/pci/ecam.h                  |  72 ++++++++++++++++
>  drivers/pci/host/Kconfig            |   1 +
>  drivers/pci/host/pci-host-common.c  | 114 +++++++++++--------------
>  drivers/pci/host/pci-host-common.h  |  47 -----------
>  drivers/pci/host/pci-host-generic.c |  52 +++---------
>  drivers/pci/host/pci-thunder-ecam.c |  39 ++-------
>  drivers/pci/host/pci-thunder-pem.c  |  92 ++++++++++-----------
>  drivers/pci/pci.c                   | 150
> ++++++++++++++++++++++++++++++++-
>  drivers/pci/probe.c                 |   2 +
>  include/linux/of_address.h          |   9 --
>  include/linux/pci-acpi.h            |  14 ++++
>  include/linux/pci.h                 |  11 ++-
>  23 files changed, 823 insertions(+), 385 deletions(-)
>  create mode 100644 drivers/acpi/pci_mcfg.c
>  create mode 100644 drivers/acpi/pci_root_generic.c
>  create mode 100644 drivers/pci/ecam.c
>  create mode 100644 drivers/pci/ecam.h
>  delete mode 100644 drivers/pci/host/pci-host-common.h
> 
> --
> 1.9.1

Re: [PATCH V7 03/11] pci, of: Move the PCI I/O space management to PCI core code.

[Arnd Bergmann]

On Wednesday 11 May 2016 09:36:58 Tomasz Nowicki wrote:
> >
> > I understand that this moves code around, but those in-function
> > #ifdefs aren't nice.  Any chance to get rid of them but putting whole
> > functions under the #ifdef?
> >
> 
> This is a __weak implementation, so assuming I would move #ifdef out of 
> function I need to provide another empty __weak stub. I do not know 
> which solution is more ugly. In any case we can do that cleanup separately.
> 

I'd vote for just dropping the __weak here, given that there is no
non-weak implementation. If we end up needing to override this for
some architecture or host bridge in the future, we can think about how
to best do that then.

I agree that should be a separate patch, this one should only move
code from one file to another.

	Arnd

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Tomasz Nowicki]

Hi Gabriele,

On 11.05.2016 12:41, Gabriele Paoloni wrote:
> Hi Tomasz
>
>> -----Original Message-----
>> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
>> owner@vger.kernel.org] On Behalf Of Tomasz Nowicki
>> Sent: 10 May 2016 16:20
>> To: helgaas@kernel.org; arnd@arndb.de; will.deacon@arm.com;
>> catalin.marinas@arm.com; rafael@kernel.org; hanjun.guo@linaro.org;
>> Lorenzo.Pieralisi@arm.com; okaya@codeaurora.org; jchandra@broadcom.com
>> Cc: robert.richter@caviumnetworks.com; mw@semihalf.com;
>> Liviu.Dudau@arm.com; ddaney@caviumnetworks.com; Wangyijing;
>> Suravee.Suthikulpanit@amd.com; msalter@redhat.com; linux-
>> pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
>> acpi@vger.kernel.org; linux-kernel@vger.kernel.org; linaro-
>> acpi@lists.linaro.org; jcm@redhat.com; andrea.gallo@linaro.org;
>> dhdang@apm.com; jeremy.linton@arm.com; liudongdong (C);
>> cov@codeaurora.org; Tomasz Nowicki
>> Subject: [PATCH V7 00/11] Support for generic ACPI based PCI host
>> controller
>>
>>  From the functionality point of view this series may be split into the
>> following logic parts:
>> 1. New ECAM API and update for users of the pci-host-common API
>> 2. Necessary fixes as the preparation for using driver on ARM64.
>> 3. Use new MCFG interface and implement generic ACPI based PCI host
>> controller driver.
>> 4. Enable above driver on ARM64
>>
>> Patches has been built on top of 4.6-rc7 and can be found here:
>> git@github.com:semihalf-nowicki-tomasz/linux.git (pci-acpi-v7)
>>
>> This has been tested on Cavium ThunderX server. Any help in reviewing
>> and
>> testing is very appreciated.
>>
>> v6 -> v7
>> - drop quirks handling
>
> Maybe I missed something in the v6 discussion thread; when was it
> decided to drop quirk handling?

I had such requests in previous series.

>
> I think it is important to have this in place to accommodate different
> vendors. If the intention is to keep this patchset "clean" maybe
> we can add it as a separate patch on top later on...
>
> What’s your view?

Yes, keeping these things separated should help in review. Obviously I 
agree that we need quirk handling but currently there is no 
implementation which we all agree upon. For the test, you can use quirk 
handling approach from the previous series until we sort out final solution.

Thanks,
Tomasz

RE: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Gabriele Paoloni]

Hi Tomasz

> -----Original Message-----
> From: Tomasz Nowicki [mailto:tn@semihalf.com]
> Sent: 11 May 2016 12:08
> To: Gabriele Paoloni; helgaas@kernel.org; arnd@arndb.de;
> will.deacon@arm.com; catalin.marinas@arm.com; rafael@kernel.org;
> hanjun.guo@linaro.org; Lorenzo.Pieralisi@arm.com; okaya@codeaurora.org;
> jchandra@broadcom.com
> Cc: robert.richter@caviumnetworks.com; mw@semihalf.com;
> Liviu.Dudau@arm.com; ddaney@caviumnetworks.com; Wangyijing;
> Suravee.Suthikulpanit@amd.com; msalter@redhat.com; linux-
> pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
> acpi@vger.kernel.org; linux-kernel@vger.kernel.org; linaro-
> acpi@lists.linaro.org; jcm@redhat.com; andrea.gallo@linaro.org;
> dhdang@apm.com; jeremy.linton@arm.com; liudongdong (C);
> cov@codeaurora.org
> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
> controller
> 
> Hi Gabriele,
> 
> On 11.05.2016 12:41, Gabriele Paoloni wrote:
> > Hi Tomasz
> >
> >> -----Original Message-----
> >> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> >> owner@vger.kernel.org] On Behalf Of Tomasz Nowicki
> >> Sent: 10 May 2016 16:20
> >> To: helgaas@kernel.org; arnd@arndb.de; will.deacon@arm.com;
> >> catalin.marinas@arm.com; rafael@kernel.org; hanjun.guo@linaro.org;
> >> Lorenzo.Pieralisi@arm.com; okaya@codeaurora.org;
> jchandra@broadcom.com
> >> Cc: robert.richter@caviumnetworks.com; mw@semihalf.com;
> >> Liviu.Dudau@arm.com; ddaney@caviumnetworks.com; Wangyijing;
> >> Suravee.Suthikulpanit@amd.com; msalter@redhat.com; linux-
> >> pci@vger.kernel.org; linux-arm-kernel@lists.infradead.org; linux-
> >> acpi@vger.kernel.org; linux-kernel@vger.kernel.org; linaro-
> >> acpi@lists.linaro.org; jcm@redhat.com; andrea.gallo@linaro.org;
> >> dhdang@apm.com; jeremy.linton@arm.com; liudongdong (C);
> >> cov@codeaurora.org; Tomasz Nowicki
> >> Subject: [PATCH V7 00/11] Support for generic ACPI based PCI host
> >> controller
> >>
> >>  From the functionality point of view this series may be split into
> the
> >> following logic parts:
> >> 1. New ECAM API and update for users of the pci-host-common API
> >> 2. Necessary fixes as the preparation for using driver on ARM64.
> >> 3. Use new MCFG interface and implement generic ACPI based PCI host
> >> controller driver.
> >> 4. Enable above driver on ARM64
> >>
> >> Patches has been built on top of 4.6-rc7 and can be found here:
> >> git@github.com:semihalf-nowicki-tomasz/linux.git (pci-acpi-v7)
> >>
> >> This has been tested on Cavium ThunderX server. Any help in
> reviewing
> >> and
> >> testing is very appreciated.
> >>
> >> v6 -> v7
> >> - drop quirks handling
> >
> > Maybe I missed something in the v6 discussion thread; when was it
> > decided to drop quirk handling?
> 
> I had such requests in previous series.
> 
> >
> > I think it is important to have this in place to accommodate
> different
> > vendors. If the intention is to keep this patchset "clean" maybe
> > we can add it as a separate patch on top later on...
> >
> > What's your view?
> 
> Yes, keeping these things separated should help in review. Obviously I
> agree that we need quirk handling but currently there is no
> implementation which we all agree upon. For the test, you can use quirk
> handling approach from the previous series until we sort out final
> solution.

Great

This explains perfectly

We will apply previous series quirks and try to test the patchset
from Huawei side

Many Thanks

Gab

> 
> Thanks,
> Tomasz

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Rafael J. Wysocki]

On Wed, May 11, 2016 at 12:11 PM, Lorenzo Pieralisi
<lorenzo.pieralisi@arm.com> wrote:
> On Tue, May 10, 2016 at 08:37:00PM +0200, Rafael J. Wysocki wrote:
>> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
>> > This patch provides a way to set the ACPI companion in PCI code.
>> > We define acpi_pci_set_companion() to set the ACPI companion pointer and
>> > call it from PCI core code. The function is stub for now.
>> >
>> > Signed-off-by: Jayachandran C <jchandra@broadcom.com>
>> > Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
>> > ---
>> >  drivers/pci/probe.c      | 2 ++
>> >  include/linux/pci-acpi.h | 4 ++++
>> >  2 files changed, 6 insertions(+)
>> >
>> > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> > index 8004f67..fb0b752 100644
>> > --- a/drivers/pci/probe.c
>> > +++ b/drivers/pci/probe.c
>> > @@ -12,6 +12,7 @@
>> >  #include <linux/slab.h>
>> >  #include <linux/module.h>
>> >  #include <linux/cpumask.h>
>> > +#include <linux/pci-acpi.h>
>> >  #include <linux/pci-aspm.h>
>> >  #include <linux/aer.h>
>> >  #include <linux/acpi.h>
>> > @@ -2141,6 +2142,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
>> >         bridge->dev.parent = parent;
>> >         bridge->dev.release = pci_release_host_bridge_dev;
>> >         dev_set_name(&bridge->dev, "pci%04x:%02x", pci_domain_nr(b), bus);
>> > +       acpi_pci_set_companion(bridge);
>>
>> Yes, we'll probably add something similar here.
>>
>> Do I think now is the right time to do that?  No.
>>
>> >         error = pcibios_root_bridge_prepare(bridge);
>> >         if (error) {
>> >                 kfree(bridge);
>> > diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
>> > index 09f9f02..1baa515 100644
>> > --- a/include/linux/pci-acpi.h
>> > +++ b/include/linux/pci-acpi.h
>> > @@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
>> >  static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
>> >  #endif /* CONFIG_ACPI */
>> >
>> > +static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
>> > +{
>> > +}
>> > +
>> >  static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
>> >  {
>> >         return 0;
>> > --
>>
>> Honestly, to me it looks like this series is trying very hard to avoid
>> doing any PCI host bridge configuration stuff from arch/arm64/
>> although (a) that might be simpler and (b) it would allow us to
>> identify the code that's common between *all* architectures using ACPI
>> support for host bridge configuration and to move *that* to a common
>> place later.  As done here it seems to be following the "ARM64 is
>> generic and the rest of the world is special" line which isn't really
>> helpful.
>
> I think patch [1-2] should be merged regardless (they may require minor
> tweaks if we decide to move pci_acpi_scan_root() to arch/arm64 though,
> for include files location). I guess you are referring to patch 8 in
> your comments above, which boils down to deciding whether:
>
> - pci_acpi_scan_root() (and unfortunately all the MCFG/ECAM handling that
>   goes with it) should live in arch/arm64 or drivers/acpi

To be precise, everything under #ifdef CONFIG_ACPI_PCI_HOST_GENERIC or
equivalent is de facto ARM64-specific, because (as it stands in the
patch series) ARM64 is the only architecture that will select that
option.  Unless you are aware of any more architectures planning to
use ACPI (and I'm not aware of any), it will stay the only
architecture selecting it in the foreseeable future.

Therefore you could replace CONFIG_ACPI_PCI_HOST_GENERIC with
CONFIG_ARM64 everywhere in that code which is why in my opinion the
code should live somewhere under arch/arm64/.

Going forward, it should be possible to identify common parts of the
PCI host bridge configuration code in arch/ and move it to
drivers/acpi/ or drivers/pci/, but I bet that won't be the entire code
this series puts under CONFIG_ACPI_PCI_HOST_GENERIC.

The above leads to a quite straightforward conclusion about the order
in which to do things: I'd add ACPI support for PCI host bridge on
ARM64 following what's been done on ia64 (as x86 is more quirky and
kludgy overall) as far as reasonably possible first and then think
about moving common stuff to a common place.

> acpi_pci_bus_domain_nr() is a bit more problematic since it is meant
> to be called from PCI core code (ARM64 selects PCI_DOMAINS_GENERIC for
> DT and same kernel has to work with OF and ACPI selected) and it is
> arch specific (because what we have in bus->sysdata is arch specific,
> waiting for the domain number to be embedded in struct pci_host_bridge).
>
> Your point is fair, I am not sure that moving the pci_acpi_scan_root()
> to arch/arm64 would make things much simpler though, it is just a matter
> of deciding where that code has to live.
>
> How do you want us to proceed ?

Pretty much as stated above. :-)

Thanks,
Rafael

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Bjorn Helgaas]

On Wed, May 11, 2016 at 10:30:51PM +0200, Rafael J. Wysocki wrote:
> On Wed, May 11, 2016 at 12:11 PM, Lorenzo Pieralisi
> <lorenzo.pieralisi@arm.com> wrote:
> > On Tue, May 10, 2016 at 08:37:00PM +0200, Rafael J. Wysocki wrote:
> >> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> >> > This patch provides a way to set the ACPI companion in PCI code.
> >> > We define acpi_pci_set_companion() to set the ACPI companion pointer and
> >> > call it from PCI core code. The function is stub for now.
> >> >
> >> > Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> >> > Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> >> > ---
> >> >  drivers/pci/probe.c      | 2 ++
> >> >  include/linux/pci-acpi.h | 4 ++++
> >> >  2 files changed, 6 insertions(+)
> >> >
> >> > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> >> > index 8004f67..fb0b752 100644
> >> > --- a/drivers/pci/probe.c
> >> > +++ b/drivers/pci/probe.c
> >> > @@ -12,6 +12,7 @@
> >> >  #include <linux/slab.h>
> >> >  #include <linux/module.h>
> >> >  #include <linux/cpumask.h>
> >> > +#include <linux/pci-acpi.h>
> >> >  #include <linux/pci-aspm.h>
> >> >  #include <linux/aer.h>
> >> >  #include <linux/acpi.h>
> >> > @@ -2141,6 +2142,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
> >> >         bridge->dev.parent = parent;
> >> >         bridge->dev.release = pci_release_host_bridge_dev;
> >> >         dev_set_name(&bridge->dev, "pci%04x:%02x", pci_domain_nr(b), bus);
> >> > +       acpi_pci_set_companion(bridge);
> >>
> >> Yes, we'll probably add something similar here.
> >>
> >> Do I think now is the right time to do that?  No.
> >>
> >> >         error = pcibios_root_bridge_prepare(bridge);
> >> >         if (error) {
> >> >                 kfree(bridge);
> >> > diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
> >> > index 09f9f02..1baa515 100644
> >> > --- a/include/linux/pci-acpi.h
> >> > +++ b/include/linux/pci-acpi.h
> >> > @@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
> >> >  static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
> >> >  #endif /* CONFIG_ACPI */
> >> >
> >> > +static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
> >> > +{
> >> > +}
> >> > +
> >> >  static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
> >> >  {
> >> >         return 0;
> >> > --
> >>
> >> Honestly, to me it looks like this series is trying very hard to avoid
> >> doing any PCI host bridge configuration stuff from arch/arm64/
> >> although (a) that might be simpler and (b) it would allow us to
> >> identify the code that's common between *all* architectures using ACPI
> >> support for host bridge configuration and to move *that* to a common
> >> place later.  As done here it seems to be following the "ARM64 is
> >> generic and the rest of the world is special" line which isn't really
> >> helpful.
> >
> > I think patch [1-2] should be merged regardless (they may require minor
> > tweaks if we decide to move pci_acpi_scan_root() to arch/arm64 though,
> > for include files location). I guess you are referring to patch 8 in
> > your comments above, which boils down to deciding whether:
> >
> > - pci_acpi_scan_root() (and unfortunately all the MCFG/ECAM handling that
> >   goes with it) should live in arch/arm64 or drivers/acpi
> 
> To be precise, everything under #ifdef CONFIG_ACPI_PCI_HOST_GENERIC or
> equivalent is de facto ARM64-specific, because (as it stands in the
> patch series) ARM64 is the only architecture that will select that
> option.  Unless you are aware of any more architectures planning to
> use ACPI (and I'm not aware of any), it will stay the only
> architecture selecting it in the foreseeable future.
> 
> Therefore you could replace CONFIG_ACPI_PCI_HOST_GENERIC with
> CONFIG_ARM64 everywhere in that code which is why in my opinion the
> code should live somewhere under arch/arm64/.
> 
> Going forward, it should be possible to identify common parts of the
> PCI host bridge configuration code in arch/ and move it to
> drivers/acpi/ or drivers/pci/, but I bet that won't be the entire code
> this series puts under CONFIG_ACPI_PCI_HOST_GENERIC.
> 
> The above leads to a quite straightforward conclusion about the order
> in which to do things: I'd add ACPI support for PCI host bridge on
> ARM64 following what's been done on ia64 (as x86 is more quirky and
> kludgy overall) as far as reasonably possible first and then think
> about moving common stuff to a common place.

That does seem like a reasonable approach.  I had hoped to get more of
this in for v4.7, but we don't have much time left.  Maybe some of
Rafael's comments can be addressed by moving and slight restructuring
and we can still squeeze it in.

The first three patches:

  PCI: Provide common functions for ECAM mapping
  PCI: generic, thunder: Use generic ECAM API
  PCI, of: Move PCI I/O space management to PCI core code

seem relatively straightforward, and I applied them to pci/arm64 with
the intent of merging them unless there are objections.  I made the
following tweaks, mainly to try to improve some error messages:

diff --git a/drivers/pci/ecam.c b/drivers/pci/ecam.c
index 3d52005..e1add01 100644
--- a/drivers/pci/ecam.c
+++ b/drivers/pci/ecam.c
@@ -24,9 +24,9 @@
 #include "ecam.h"
 
 /*
- * On 64 bit systems, we do a single ioremap for the whole config space
- * since we have enough virtual address range available. On 32 bit, do an
- * ioremap per bus.
+ * On 64-bit systems, we do a single ioremap for the whole config space
+ * since we have enough virtual address range available.  On 32-bit, we
+ * ioremap the config space for each bus individually.
  */
 static const bool per_bus_mapping = !config_enabled(CONFIG_64BIT);
 
@@ -42,6 +42,7 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
 {
 	struct pci_config_window *cfg;
 	unsigned int bus_range, bus_range_max, bsz;
+	struct resource *conflict;
 	int i, err;
 
 	if (busr->start > busr->end)
@@ -58,10 +59,10 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
 	bus_range = resource_size(&cfg->busr);
 	bus_range_max = resource_size(cfgres) >> ops->bus_shift;
 	if (bus_range > bus_range_max) {
-		dev_warn(dev, "bus max %#x reduced to %#x",
-					bus_range, bus_range_max);
 		bus_range = bus_range_max;
 		cfg->busr.end = busr->start + bus_range - 1;
+		dev_warn(dev, "ECAM area %pR can only accommodate %pR (reduced from %pR desired)\n",
+			 cfgres, &cfg->busr, busr);
 	}
 	bsz = 1 << ops->bus_shift;
 
@@ -70,9 +71,11 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
 	cfg->res.flags = IORESOURCE_MEM | IORESOURCE_BUSY;
 	cfg->res.name = "PCI ECAM";
 
-	err = request_resource(&iomem_resource, &cfg->res);
-	if (err) {
-		dev_err(dev, "request ECAM res %pR failed\n", &cfg->res);
+	conflict = request_resource(&iomem_resource, &cfg->res);
+	if (conflict) {
+		err = -EBUSY;
+		dev_err(dev, "can't claim ECAM area %pR: address conflict with %s %pR\n",
+			&cfg->res, conflict->name, conflict);
 		goto err_exit;
 	}
 
diff --git a/drivers/pci/ecam.h b/drivers/pci/ecam.h
index 1ad2176..9878beb 100644
--- a/drivers/pci/ecam.h
+++ b/drivers/pci/ecam.h
@@ -33,7 +33,7 @@ struct pci_ecam_ops {
 
 /*
  * struct to hold the mappings of a config space window. This
- * is expected to be used as sysdata for PCI controlllers which
+ * is expected to be used as sysdata for PCI controllers that
  * use ECAM.
  */
 struct pci_config_window {
@@ -43,11 +43,11 @@ struct pci_config_window {
 	struct pci_ecam_ops		*ops;
 	union {
 		void __iomem		*win;	/* 64-bit single mapping */
-		void __iomem		**winp; /* 32-bit per bus mapping */
+		void __iomem		**winp; /* 32-bit per-bus mapping */
 	};
 };
 
-/* create and free for pci_config_window */
+/* create and free pci_config_window */
 struct pci_config_window *pci_ecam_create(struct device *dev,
 		struct resource *cfgres, struct resource *busr,
 		struct pci_ecam_ops *ops);
@@ -56,11 +56,11 @@ void pci_ecam_free(struct pci_config_window *cfg);
 /* map_bus when ->sysdata is an instance of pci_config_window */
 void __iomem *pci_ecam_map_bus(struct pci_bus *bus, unsigned int devfn,
 			       int where);
-/* default ECAM ops, bus shift 20, generic read and write */
+/* default ECAM ops */
 extern struct pci_ecam_ops pci_generic_ecam_ops;
 
 #ifdef CONFIG_PCI_HOST_GENERIC
-/* for DT based pci controllers that support ECAM */
+/* for DT-based PCI controllers that support ECAM */
 int pci_host_common_probe(struct platform_device *pdev,
 			  struct pci_ecam_ops *ops);
 #endif

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Lorenzo Pieralisi]

On Wed, May 11, 2016 at 05:43:14PM -0500, Bjorn Helgaas wrote:
> On Wed, May 11, 2016 at 10:30:51PM +0200, Rafael J. Wysocki wrote:
> > On Wed, May 11, 2016 at 12:11 PM, Lorenzo Pieralisi
> > <lorenzo.pieralisi@arm.com> wrote:
> > > On Tue, May 10, 2016 at 08:37:00PM +0200, Rafael J. Wysocki wrote:
> > >> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> > >> > This patch provides a way to set the ACPI companion in PCI code.
> > >> > We define acpi_pci_set_companion() to set the ACPI companion pointer and
> > >> > call it from PCI core code. The function is stub for now.
> > >> >
> > >> > Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> > >> > Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> > >> > ---
> > >> >  drivers/pci/probe.c      | 2 ++
> > >> >  include/linux/pci-acpi.h | 4 ++++
> > >> >  2 files changed, 6 insertions(+)
> > >> >
> > >> > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > >> > index 8004f67..fb0b752 100644
> > >> > --- a/drivers/pci/probe.c
> > >> > +++ b/drivers/pci/probe.c
> > >> > @@ -12,6 +12,7 @@
> > >> >  #include <linux/slab.h>
> > >> >  #include <linux/module.h>
> > >> >  #include <linux/cpumask.h>
> > >> > +#include <linux/pci-acpi.h>
> > >> >  #include <linux/pci-aspm.h>
> > >> >  #include <linux/aer.h>
> > >> >  #include <linux/acpi.h>
> > >> > @@ -2141,6 +2142,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
> > >> >         bridge->dev.parent = parent;
> > >> >         bridge->dev.release = pci_release_host_bridge_dev;
> > >> >         dev_set_name(&bridge->dev, "pci%04x:%02x", pci_domain_nr(b), bus);
> > >> > +       acpi_pci_set_companion(bridge);
> > >>
> > >> Yes, we'll probably add something similar here.
> > >>
> > >> Do I think now is the right time to do that?  No.
> > >>
> > >> >         error = pcibios_root_bridge_prepare(bridge);
> > >> >         if (error) {
> > >> >                 kfree(bridge);
> > >> > diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
> > >> > index 09f9f02..1baa515 100644
> > >> > --- a/include/linux/pci-acpi.h
> > >> > +++ b/include/linux/pci-acpi.h
> > >> > @@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
> > >> >  static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
> > >> >  #endif /* CONFIG_ACPI */
> > >> >
> > >> > +static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
> > >> > +{
> > >> > +}
> > >> > +
> > >> >  static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
> > >> >  {
> > >> >         return 0;
> > >> > --
> > >>
> > >> Honestly, to me it looks like this series is trying very hard to avoid
> > >> doing any PCI host bridge configuration stuff from arch/arm64/
> > >> although (a) that might be simpler and (b) it would allow us to
> > >> identify the code that's common between *all* architectures using ACPI
> > >> support for host bridge configuration and to move *that* to a common
> > >> place later.  As done here it seems to be following the "ARM64 is
> > >> generic and the rest of the world is special" line which isn't really
> > >> helpful.
> > >
> > > I think patch [1-2] should be merged regardless (they may require minor
> > > tweaks if we decide to move pci_acpi_scan_root() to arch/arm64 though,
> > > for include files location). I guess you are referring to patch 8 in
> > > your comments above, which boils down to deciding whether:
> > >
> > > - pci_acpi_scan_root() (and unfortunately all the MCFG/ECAM handling that
> > >   goes with it) should live in arch/arm64 or drivers/acpi
> > 
> > To be precise, everything under #ifdef CONFIG_ACPI_PCI_HOST_GENERIC or
> > equivalent is de facto ARM64-specific, because (as it stands in the
> > patch series) ARM64 is the only architecture that will select that
> > option.  Unless you are aware of any more architectures planning to
> > use ACPI (and I'm not aware of any), it will stay the only
> > architecture selecting it in the foreseeable future.
> > 
> > Therefore you could replace CONFIG_ACPI_PCI_HOST_GENERIC with
> > CONFIG_ARM64 everywhere in that code which is why in my opinion the
> > code should live somewhere under arch/arm64/.
> > 
> > Going forward, it should be possible to identify common parts of the
> > PCI host bridge configuration code in arch/ and move it to
> > drivers/acpi/ or drivers/pci/, but I bet that won't be the entire code
> > this series puts under CONFIG_ACPI_PCI_HOST_GENERIC.
> > 
> > The above leads to a quite straightforward conclusion about the order
> > in which to do things: I'd add ACPI support for PCI host bridge on
> > ARM64 following what's been done on ia64 (as x86 is more quirky and
> > kludgy overall) as far as reasonably possible first and then think
> > about moving common stuff to a common place.
> 
> That does seem like a reasonable approach.  I had hoped to get more of
> this in for v4.7, but we don't have much time left.  Maybe some of
> Rafael's comments can be addressed by moving and slight restructuring
> and we can still squeeze it in.

Yes, it seems like a reasonable approach, as long as we accept that
part of this series has to live in arch/arm64 otherwise we are going
round in circles (because that's the gist of this discussion, to
decide where this code has to live, I do not think there is any objection
to the code per-se anymore).

I suggest we post a v8 (with code move to arch/arm64) end of merge
window (or you prefer seeing patches now to prevent any additional
changes later ?), my aim is to get this into -next (whether via arm64 or
pci tree it has to be decided) as early as possible for next cycle (-rc1)
so that it can get exposure and testing, I do not think that missing the
merge window is a big issue if we agree that the code is ready to go.

> The first three patches:
> 
>   PCI: Provide common functions for ECAM mapping
>   PCI: generic, thunder: Use generic ECAM API
>   PCI, of: Move PCI I/O space management to PCI core code
> 
> seem relatively straightforward, and I applied them to pci/arm64 with
> the intent of merging them unless there are objections.  I made the
> following tweaks, mainly to try to improve some error messages:

Ok, thanks a lot !

Lorenzo

> diff --git a/drivers/pci/ecam.c b/drivers/pci/ecam.c
> index 3d52005..e1add01 100644
> --- a/drivers/pci/ecam.c
> +++ b/drivers/pci/ecam.c
> @@ -24,9 +24,9 @@
>  #include "ecam.h"
>  
>  /*
> - * On 64 bit systems, we do a single ioremap for the whole config space
> - * since we have enough virtual address range available. On 32 bit, do an
> - * ioremap per bus.
> + * On 64-bit systems, we do a single ioremap for the whole config space
> + * since we have enough virtual address range available.  On 32-bit, we
> + * ioremap the config space for each bus individually.
>   */
>  static const bool per_bus_mapping = !config_enabled(CONFIG_64BIT);
>  
> @@ -42,6 +42,7 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
>  {
>  	struct pci_config_window *cfg;
>  	unsigned int bus_range, bus_range_max, bsz;
> +	struct resource *conflict;
>  	int i, err;
>  
>  	if (busr->start > busr->end)
> @@ -58,10 +59,10 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
>  	bus_range = resource_size(&cfg->busr);
>  	bus_range_max = resource_size(cfgres) >> ops->bus_shift;
>  	if (bus_range > bus_range_max) {
> -		dev_warn(dev, "bus max %#x reduced to %#x",
> -					bus_range, bus_range_max);
>  		bus_range = bus_range_max;
>  		cfg->busr.end = busr->start + bus_range - 1;
> +		dev_warn(dev, "ECAM area %pR can only accommodate %pR (reduced from %pR desired)\n",
> +			 cfgres, &cfg->busr, busr);
>  	}
>  	bsz = 1 << ops->bus_shift;
>  
> @@ -70,9 +71,11 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
>  	cfg->res.flags = IORESOURCE_MEM | IORESOURCE_BUSY;
>  	cfg->res.name = "PCI ECAM";
>  
> -	err = request_resource(&iomem_resource, &cfg->res);
> -	if (err) {
> -		dev_err(dev, "request ECAM res %pR failed\n", &cfg->res);
> +	conflict = request_resource(&iomem_resource, &cfg->res);
> +	if (conflict) {
> +		err = -EBUSY;
> +		dev_err(dev, "can't claim ECAM area %pR: address conflict with %s %pR\n",
> +			&cfg->res, conflict->name, conflict);
>  		goto err_exit;
>  	}
>  
> diff --git a/drivers/pci/ecam.h b/drivers/pci/ecam.h
> index 1ad2176..9878beb 100644
> --- a/drivers/pci/ecam.h
> +++ b/drivers/pci/ecam.h
> @@ -33,7 +33,7 @@ struct pci_ecam_ops {
>  
>  /*
>   * struct to hold the mappings of a config space window. This
> - * is expected to be used as sysdata for PCI controlllers which
> + * is expected to be used as sysdata for PCI controllers that
>   * use ECAM.
>   */
>  struct pci_config_window {
> @@ -43,11 +43,11 @@ struct pci_config_window {
>  	struct pci_ecam_ops		*ops;
>  	union {
>  		void __iomem		*win;	/* 64-bit single mapping */
> -		void __iomem		**winp; /* 32-bit per bus mapping */
> +		void __iomem		**winp; /* 32-bit per-bus mapping */
>  	};
>  };
>  
> -/* create and free for pci_config_window */
> +/* create and free pci_config_window */
>  struct pci_config_window *pci_ecam_create(struct device *dev,
>  		struct resource *cfgres, struct resource *busr,
>  		struct pci_ecam_ops *ops);
> @@ -56,11 +56,11 @@ void pci_ecam_free(struct pci_config_window *cfg);
>  /* map_bus when ->sysdata is an instance of pci_config_window */
>  void __iomem *pci_ecam_map_bus(struct pci_bus *bus, unsigned int devfn,
>  			       int where);
> -/* default ECAM ops, bus shift 20, generic read and write */
> +/* default ECAM ops */
>  extern struct pci_ecam_ops pci_generic_ecam_ops;
>  
>  #ifdef CONFIG_PCI_HOST_GENERIC
> -/* for DT based pci controllers that support ECAM */
> +/* for DT-based PCI controllers that support ECAM */
>  int pci_host_common_probe(struct platform_device *pdev,
>  			  struct pci_ecam_ops *ops);
>  #endif
> 

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Jayachandran C]

On Thu, May 12, 2016 at 4:13 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Wed, May 11, 2016 at 10:30:51PM +0200, Rafael J. Wysocki wrote:
>> On Wed, May 11, 2016 at 12:11 PM, Lorenzo Pieralisi
>> <lorenzo.pieralisi@arm.com> wrote:
>> > On Tue, May 10, 2016 at 08:37:00PM +0200, Rafael J. Wysocki wrote:
>> >> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
>> >> > This patch provides a way to set the ACPI companion in PCI code.
>> >> > We define acpi_pci_set_companion() to set the ACPI companion pointer and
>> >> > call it from PCI core code. The function is stub for now.
>> >> >
>> >> > Signed-off-by: Jayachandran C <jchandra@broadcom.com>
>> >> > Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
>> >> > ---
>> >> >  drivers/pci/probe.c      | 2 ++
>> >> >  include/linux/pci-acpi.h | 4 ++++
>> >> >  2 files changed, 6 insertions(+)
>> >> >
>> >> > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> >> > index 8004f67..fb0b752 100644
>> >> > --- a/drivers/pci/probe.c
>> >> > +++ b/drivers/pci/probe.c
>> >> > @@ -12,6 +12,7 @@
>> >> >  #include <linux/slab.h>
>> >> >  #include <linux/module.h>
>> >> >  #include <linux/cpumask.h>
>> >> > +#include <linux/pci-acpi.h>
>> >> >  #include <linux/pci-aspm.h>
>> >> >  #include <linux/aer.h>
>> >> >  #include <linux/acpi.h>
>> >> > @@ -2141,6 +2142,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
>> >> >         bridge->dev.parent = parent;
>> >> >         bridge->dev.release = pci_release_host_bridge_dev;
>> >> >         dev_set_name(&bridge->dev, "pci%04x:%02x", pci_domain_nr(b), bus);
>> >> > +       acpi_pci_set_companion(bridge);
>> >>
>> >> Yes, we'll probably add something similar here.
>> >>
>> >> Do I think now is the right time to do that?  No.
>> >>
>> >> >         error = pcibios_root_bridge_prepare(bridge);
>> >> >         if (error) {
>> >> >                 kfree(bridge);
>> >> > diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
>> >> > index 09f9f02..1baa515 100644
>> >> > --- a/include/linux/pci-acpi.h
>> >> > +++ b/include/linux/pci-acpi.h
>> >> > @@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
>> >> >  static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
>> >> >  #endif /* CONFIG_ACPI */
>> >> >
>> >> > +static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
>> >> > +{
>> >> > +}
>> >> > +
>> >> >  static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
>> >> >  {
>> >> >         return 0;
>> >> > --
>> >>
>> >> Honestly, to me it looks like this series is trying very hard to avoid
>> >> doing any PCI host bridge configuration stuff from arch/arm64/
>> >> although (a) that might be simpler and (b) it would allow us to
>> >> identify the code that's common between *all* architectures using ACPI
>> >> support for host bridge configuration and to move *that* to a common
>> >> place later.  As done here it seems to be following the "ARM64 is
>> >> generic and the rest of the world is special" line which isn't really
>> >> helpful.
>> >
>> > I think patch [1-2] should be merged regardless (they may require minor
>> > tweaks if we decide to move pci_acpi_scan_root() to arch/arm64 though,
>> > for include files location). I guess you are referring to patch 8 in
>> > your comments above, which boils down to deciding whether:
>> >
>> > - pci_acpi_scan_root() (and unfortunately all the MCFG/ECAM handling that
>> >   goes with it) should live in arch/arm64 or drivers/acpi
>>
>> To be precise, everything under #ifdef CONFIG_ACPI_PCI_HOST_GENERIC or
>> equivalent is de facto ARM64-specific, because (as it stands in the
>> patch series) ARM64 is the only architecture that will select that
>> option.  Unless you are aware of any more architectures planning to
>> use ACPI (and I'm not aware of any), it will stay the only
>> architecture selecting it in the foreseeable future.
>>
>> Therefore you could replace CONFIG_ACPI_PCI_HOST_GENERIC with
>> CONFIG_ARM64 everywhere in that code which is why in my opinion the
>> code should live somewhere under arch/arm64/.
>>
>> Going forward, it should be possible to identify common parts of the
>> PCI host bridge configuration code in arch/ and move it to
>> drivers/acpi/ or drivers/pci/, but I bet that won't be the entire code
>> this series puts under CONFIG_ACPI_PCI_HOST_GENERIC.
>>
>> The above leads to a quite straightforward conclusion about the order
>> in which to do things: I'd add ACPI support for PCI host bridge on
>> ARM64 following what's been done on ia64 (as x86 is more quirky and
>> kludgy overall) as far as reasonably possible first and then think
>> about moving common stuff to a common place.
>
> That does seem like a reasonable approach.  I had hoped to get more of
> this in for v4.7, but we don't have much time left.  Maybe some of
> Rafael's comments can be addressed by moving and slight restructuring
> and we can still squeeze it in.
>
> The first three patches:
>
>   PCI: Provide common functions for ECAM mapping
>   PCI: generic, thunder: Use generic ECAM API
>   PCI, of: Move PCI I/O space management to PCI core code
>
> seem relatively straightforward, and I applied them to pci/arm64 with
> the intent of merging them unless there are objections.  I made the
> following tweaks, mainly to try to improve some error messages:
>
> diff --git a/drivers/pci/ecam.c b/drivers/pci/ecam.c
> index 3d52005..e1add01 100644
> --- a/drivers/pci/ecam.c
> +++ b/drivers/pci/ecam.c
> @@ -24,9 +24,9 @@
>  #include "ecam.h"
>
>  /*
> - * On 64 bit systems, we do a single ioremap for the whole config space
> - * since we have enough virtual address range available. On 32 bit, do an
> - * ioremap per bus.
> + * On 64-bit systems, we do a single ioremap for the whole config space
> + * since we have enough virtual address range available.  On 32-bit, we
> + * ioremap the config space for each bus individually.
>   */
>  static const bool per_bus_mapping = !config_enabled(CONFIG_64BIT);
>
> @@ -42,6 +42,7 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
>  {
>         struct pci_config_window *cfg;
>         unsigned int bus_range, bus_range_max, bsz;
> +       struct resource *conflict;
>         int i, err;
>
>         if (busr->start > busr->end)
> @@ -58,10 +59,10 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
>         bus_range = resource_size(&cfg->busr);
>         bus_range_max = resource_size(cfgres) >> ops->bus_shift;
>         if (bus_range > bus_range_max) {
> -               dev_warn(dev, "bus max %#x reduced to %#x",
> -                                       bus_range, bus_range_max);
>                 bus_range = bus_range_max;
>                 cfg->busr.end = busr->start + bus_range - 1;
> +               dev_warn(dev, "ECAM area %pR can only accommodate %pR (reduced from %pR desired)\n",
> +                        cfgres, &cfg->busr, busr);
>         }
>         bsz = 1 << ops->bus_shift;
>
> @@ -70,9 +71,11 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
>         cfg->res.flags = IORESOURCE_MEM | IORESOURCE_BUSY;
>         cfg->res.name = "PCI ECAM";
>
> -       err = request_resource(&iomem_resource, &cfg->res);
> -       if (err) {
> -               dev_err(dev, "request ECAM res %pR failed\n", &cfg->res);
> +       conflict = request_resource(&iomem_resource, &cfg->res);
> +       if (conflict) {
> +               err = -EBUSY;
> +               dev_err(dev, "can't claim ECAM area %pR: address conflict with %s %pR\n",
> +                       &cfg->res, conflict->name, conflict);
>                 goto err_exit;
>         }
>
> diff --git a/drivers/pci/ecam.h b/drivers/pci/ecam.h
> index 1ad2176..9878beb 100644
> --- a/drivers/pci/ecam.h
> +++ b/drivers/pci/ecam.h
> @@ -33,7 +33,7 @@ struct pci_ecam_ops {
>
>  /*
>   * struct to hold the mappings of a config space window. This
> - * is expected to be used as sysdata for PCI controlllers which
> + * is expected to be used as sysdata for PCI controllers that
>   * use ECAM.
>   */
>  struct pci_config_window {
> @@ -43,11 +43,11 @@ struct pci_config_window {
>         struct pci_ecam_ops             *ops;
>         union {
>                 void __iomem            *win;   /* 64-bit single mapping */
> -               void __iomem            **winp; /* 32-bit per bus mapping */
> +               void __iomem            **winp; /* 32-bit per-bus mapping */
>         };
>  };
>
> -/* create and free for pci_config_window */
> +/* create and free pci_config_window */
>  struct pci_config_window *pci_ecam_create(struct device *dev,
>                 struct resource *cfgres, struct resource *busr,
>                 struct pci_ecam_ops *ops);
> @@ -56,11 +56,11 @@ void pci_ecam_free(struct pci_config_window *cfg);
>  /* map_bus when ->sysdata is an instance of pci_config_window */
>  void __iomem *pci_ecam_map_bus(struct pci_bus *bus, unsigned int devfn,
>                                int where);
> -/* default ECAM ops, bus shift 20, generic read and write */
> +/* default ECAM ops */
>  extern struct pci_ecam_ops pci_generic_ecam_ops;
>
>  #ifdef CONFIG_PCI_HOST_GENERIC
> -/* for DT based pci controllers that support ECAM */
> +/* for DT-based PCI controllers that support ECAM */
>  int pci_host_common_probe(struct platform_device *pdev,
>                           struct pci_ecam_ops *ops);
>  #endif

If we are moving the ACPI/PCI code from drivers/acpi to
arch/arm64/ , there is an issue in having the header file
ecam.h in drivers/pci

The current include of "../pci/ecam.h" is slightly ugly (Arnd
and David had already noted this), but including the driver
header from arch code would be even worse.

I can either merge ecam.h into include/linux/pci.h
or move it to a new file include/linux/pci-ecam.h, any
suggestion on which is preferable?

JC.

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Tomasz Nowicki]

On 12.05.2016 00:43, Bjorn Helgaas wrote:
> On Wed, May 11, 2016 at 10:30:51PM +0200, Rafael J. Wysocki wrote:
>> On Wed, May 11, 2016 at 12:11 PM, Lorenzo Pieralisi
>> <lorenzo.pieralisi@arm.com> wrote:
>>> On Tue, May 10, 2016 at 08:37:00PM +0200, Rafael J. Wysocki wrote:
>>>> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
>>>>> This patch provides a way to set the ACPI companion in PCI code.
>>>>> We define acpi_pci_set_companion() to set the ACPI companion pointer and
>>>>> call it from PCI core code. The function is stub for now.
>>>>>
>>>>> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
>>>>> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
>>>>> ---
>>>>>   drivers/pci/probe.c      | 2 ++
>>>>>   include/linux/pci-acpi.h | 4 ++++
>>>>>   2 files changed, 6 insertions(+)
>>>>>
>>>>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>>>>> index 8004f67..fb0b752 100644
>>>>> --- a/drivers/pci/probe.c
>>>>> +++ b/drivers/pci/probe.c
>>>>> @@ -12,6 +12,7 @@
>>>>>   #include <linux/slab.h>
>>>>>   #include <linux/module.h>
>>>>>   #include <linux/cpumask.h>
>>>>> +#include <linux/pci-acpi.h>
>>>>>   #include <linux/pci-aspm.h>
>>>>>   #include <linux/aer.h>
>>>>>   #include <linux/acpi.h>
>>>>> @@ -2141,6 +2142,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
>>>>>          bridge->dev.parent = parent;
>>>>>          bridge->dev.release = pci_release_host_bridge_dev;
>>>>>          dev_set_name(&bridge->dev, "pci%04x:%02x", pci_domain_nr(b), bus);
>>>>> +       acpi_pci_set_companion(bridge);
>>>>
>>>> Yes, we'll probably add something similar here.
>>>>
>>>> Do I think now is the right time to do that?  No.
>>>>
>>>>>          error = pcibios_root_bridge_prepare(bridge);
>>>>>          if (error) {
>>>>>                  kfree(bridge);
>>>>> diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
>>>>> index 09f9f02..1baa515 100644
>>>>> --- a/include/linux/pci-acpi.h
>>>>> +++ b/include/linux/pci-acpi.h
>>>>> @@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
>>>>>   static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
>>>>>   #endif /* CONFIG_ACPI */
>>>>>
>>>>> +static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
>>>>> +{
>>>>> +}
>>>>> +
>>>>>   static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
>>>>>   {
>>>>>          return 0;
>>>>> --
>>>>
>>>> Honestly, to me it looks like this series is trying very hard to avoid
>>>> doing any PCI host bridge configuration stuff from arch/arm64/
>>>> although (a) that might be simpler and (b) it would allow us to
>>>> identify the code that's common between *all* architectures using ACPI
>>>> support for host bridge configuration and to move *that* to a common
>>>> place later.  As done here it seems to be following the "ARM64 is
>>>> generic and the rest of the world is special" line which isn't really
>>>> helpful.
>>>
>>> I think patch [1-2] should be merged regardless (they may require minor
>>> tweaks if we decide to move pci_acpi_scan_root() to arch/arm64 though,
>>> for include files location). I guess you are referring to patch 8 in
>>> your comments above, which boils down to deciding whether:
>>>
>>> - pci_acpi_scan_root() (and unfortunately all the MCFG/ECAM handling that
>>>    goes with it) should live in arch/arm64 or drivers/acpi
>>
>> To be precise, everything under #ifdef CONFIG_ACPI_PCI_HOST_GENERIC or
>> equivalent is de facto ARM64-specific, because (as it stands in the
>> patch series) ARM64 is the only architecture that will select that
>> option.  Unless you are aware of any more architectures planning to
>> use ACPI (and I'm not aware of any), it will stay the only
>> architecture selecting it in the foreseeable future.
>>
>> Therefore you could replace CONFIG_ACPI_PCI_HOST_GENERIC with
>> CONFIG_ARM64 everywhere in that code which is why in my opinion the
>> code should live somewhere under arch/arm64/.
>>
>> Going forward, it should be possible to identify common parts of the
>> PCI host bridge configuration code in arch/ and move it to
>> drivers/acpi/ or drivers/pci/, but I bet that won't be the entire code
>> this series puts under CONFIG_ACPI_PCI_HOST_GENERIC.
>>
>> The above leads to a quite straightforward conclusion about the order
>> in which to do things: I'd add ACPI support for PCI host bridge on
>> ARM64 following what's been done on ia64 (as x86 is more quirky and
>> kludgy overall) as far as reasonably possible first and then think
>> about moving common stuff to a common place.
>
> That does seem like a reasonable approach.  I had hoped to get more of
> this in for v4.7, but we don't have much time left.  Maybe some of
> Rafael's comments can be addressed by moving and slight restructuring
> and we can still squeeze it in.
>
> The first three patches:
>
>    PCI: Provide common functions for ECAM mapping
>    PCI: generic, thunder: Use generic ECAM API
>    PCI, of: Move PCI I/O space management to PCI core code
>
> seem relatively straightforward, and I applied them to pci/arm64 with
> the intent of merging them unless there are objections.  I made the
> following tweaks, mainly to try to improve some error messages:
>
> diff --git a/drivers/pci/ecam.c b/drivers/pci/ecam.c
> index 3d52005..e1add01 100644
> --- a/drivers/pci/ecam.c
> +++ b/drivers/pci/ecam.c
> @@ -24,9 +24,9 @@
>   #include "ecam.h"
>
>   /*
> - * On 64 bit systems, we do a single ioremap for the whole config space
> - * since we have enough virtual address range available. On 32 bit, do an
> - * ioremap per bus.
> + * On 64-bit systems, we do a single ioremap for the whole config space
> + * since we have enough virtual address range available.  On 32-bit, we
> + * ioremap the config space for each bus individually.
>    */
>   static const bool per_bus_mapping = !config_enabled(CONFIG_64BIT);
>
> @@ -42,6 +42,7 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
>   {
>   	struct pci_config_window *cfg;
>   	unsigned int bus_range, bus_range_max, bsz;
> +	struct resource *conflict;
>   	int i, err;
>
>   	if (busr->start > busr->end)
> @@ -58,10 +59,10 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
>   	bus_range = resource_size(&cfg->busr);
>   	bus_range_max = resource_size(cfgres) >> ops->bus_shift;
>   	if (bus_range > bus_range_max) {
> -		dev_warn(dev, "bus max %#x reduced to %#x",
> -					bus_range, bus_range_max);
>   		bus_range = bus_range_max;
>   		cfg->busr.end = busr->start + bus_range - 1;
> +		dev_warn(dev, "ECAM area %pR can only accommodate %pR (reduced from %pR desired)\n",
> +			 cfgres, &cfg->busr, busr);
>   	}
>   	bsz = 1 << ops->bus_shift;
>
> @@ -70,9 +71,11 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
>   	cfg->res.flags = IORESOURCE_MEM | IORESOURCE_BUSY;
>   	cfg->res.name = "PCI ECAM";
>
> -	err = request_resource(&iomem_resource, &cfg->res);
> -	if (err) {
> -		dev_err(dev, "request ECAM res %pR failed\n", &cfg->res);
> +	conflict = request_resource(&iomem_resource, &cfg->res);

We need request_resource_conflict here then:
-	conflict = request_resource(&iomem_resource, &cfg->res);
+	conflict = request_resource_conflict(&iomem_resource, &cfg->res);

Thanks,
Tomasz

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Rafael J. Wysocki]

On Thu, May 12, 2016 at 12:43 PM, Jayachandran C <jchandra@broadcom.com> wrote:
> On Thu, May 12, 2016 at 4:13 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:
>> On Wed, May 11, 2016 at 10:30:51PM +0200, Rafael J. Wysocki wrote:
>>> On Wed, May 11, 2016 at 12:11 PM, Lorenzo Pieralisi
>>> <lorenzo.pieralisi@arm.com> wrote:
>>> > On Tue, May 10, 2016 at 08:37:00PM +0200, Rafael J. Wysocki wrote:
>>> >> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:

[cut]

>
> If we are moving the ACPI/PCI code from drivers/acpi to
> arch/arm64/ , there is an issue in having the header file
> ecam.h in drivers/pci
>
> The current include of "../pci/ecam.h" is slightly ugly (Arnd
> and David had already noted this), but including the driver
> header from arch code would be even worse.
>
> I can either merge ecam.h into include/linux/pci.h
> or move it to a new file include/linux/pci-ecam.h, any
> suggestion on which is preferable?

My preference would be pci-ecam.h as we did a similar thing for
pci-dma.h, for example, but basically this is up to Bjorn.

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Bjorn Helgaas]

On Thu, May 12, 2016 at 12:50:07PM +0200, Tomasz Nowicki wrote:
> On 12.05.2016 00:43, Bjorn Helgaas wrote:

> >@@ -70,9 +71,11 @@ struct pci_config_window *pci_ecam_create(struct device *dev,
> >  	cfg->res.flags = IORESOURCE_MEM | IORESOURCE_BUSY;
> >  	cfg->res.name = "PCI ECAM";
> >
> >-	err = request_resource(&iomem_resource, &cfg->res);
> >-	if (err) {
> >-		dev_err(dev, "request ECAM res %pR failed\n", &cfg->res);
> >+	conflict = request_resource(&iomem_resource, &cfg->res);
> 
> We need request_resource_conflict here then:
> -	conflict = request_resource(&iomem_resource, &cfg->res);
> +	conflict = request_resource_conflict(&iomem_resource, &cfg->res);

Whoops, fixed, thanks!

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Duc Dang]

On Tue, May 10, 2016 at 8:19 AM, Tomasz Nowicki <tn@semihalf.com> wrote:
> From the functionality point of view this series may be split into the
> following logic parts:
> 1. New ECAM API and update for users of the pci-host-common API
> 2. Necessary fixes as the preparation for using driver on ARM64.
> 3. Use new MCFG interface and implement generic ACPI based PCI host controller driver.
> 4. Enable above driver on ARM64
>
> Patches has been built on top of 4.6-rc7 and can be found here:
> git@github.com:semihalf-nowicki-tomasz/linux.git (pci-acpi-v7)
>
> This has been tested on Cavium ThunderX server. Any help in reviewing and
> testing is very appreciated.

I tried this series on APM X-Gene platforms (with ECAM fix-up quirk
added) and PCIe works fine.

Regards,
Duc Dang.
>
> v6 -> v7
> - drop quirks handling
> - changes for ACPI companion and domain number assignment approach
> - implement arch pcibios_{add|remove}_bus and call acpi_pci_{add|remove}_bus from there
> - cleanups around nomenclature
> - use resources oriented API for ECAM
> - fix for based address calculation before mapping ECAM region
> - remove useless lock for MCFG lookup
> - move MCFG stuff to separated file pci_mcfg.c
> - drop MCFG entries caching
> - rebase against 4.6-rc7
>
> v5 -> v6
> - drop idea of x86 MMCONFIG code refactoring
> - integrate JC's patches which introduce new ECAM API:
>   https://lkml.org/lkml/2016/4/11/907
>   git: https://github.com/jchandra-brcm/linux/ (arm64-acpi-pci-v3)
> - integrate Sinan's fix for releasing IO resources, see patch [06/13]
> - added ACPI support for ThunderX ECAM and PEM drivers
> - rebase against 4.6-rc2
>
> v4 -> v5
> - drop MCFG refactoring group patches 1-6 from series v4 and integrate Jayachandran's patch
>   https://patchwork.ozlabs.org/patch/575525/
> - rewrite PCI legacy IRQs allocation
> - squash two patches 11 and 12 from series v4, fixed bisection issue
> - changelog improvements
> - rebase against 4.5-rc3
>
> v3 -> v4
> - drop Jiang's fix http://lkml.iu.edu/hypermail/linux/kernel/1601.1/04318.html
> - add Lorenzo's fix patch 19/24
> - ACPI PCI bus domain number assigning cleanup
> - change resource management, we now claim and reassign resources
> - improvements for applying quirks
> - drop Matthew's http://www.spinics.net/lists/linux-pci/msg45950.html dependency
> - rebase against 4.5-rc1
>
> v2 -> v3
> - fix legacy IRQ assigning and IO ports registration
> - remove reference to arch specific companion device for ia64
> - move ACPI PCI host controller driver to pci_root.c
> - drop generic domain assignment for x86 and ia64 as I am not
>   able to run all necessary test variants
> - drop patch which cleaned legacy IRQ assignment since it belongs to
>   Mathew's series:
>   https://patchwork.ozlabs.org/patch/557504/
> - extend MCFG quirk code
> - rebase against 4.4
>
> v1 -> v2
> - move non-arch specific piece of code to dirver/acpi/ directory
> - fix IO resource handling
> - introduce PCI config accessors quirks matching
> - moved ACPI_COMPANION_SET to generic code
>
> v1 - https://lkml.org/lkml/2015/10/27/504
> v2 - https://lkml.org/lkml/2015/12/16/246
> v3 - http://lkml.iu.edu/hypermail/linux/kernel/1601.1/04308.html
> v4 - https://lkml.org/lkml/2016/2/4/646
> v5 - https://lkml.org/lkml/2016/2/16/426
> v6 - https://lkml.org/lkml/2016/4/15/594
>
> Jayachandran C (2):
>   PCI: Provide common functions for ECAM mapping
>   PCI: generic, thunder: update to use generic ECAM API
>
> Tomasz Nowicki (9):
>   pci, of: Move the PCI I/O space management to PCI core code.
>   pci: Add new function to unmap IO resources.
>   acpi, pci: Support IO resources when parsing PCI host bridge
>     resources.
>   pci, acpi: Provide a way to assign bus domain number.
>   pci, acpi: Handle ACPI companion assignment.
>   pci, acpi: Support for ACPI based generic PCI host controller
>   arm64, pci, acpi: ACPI support for legacy IRQs parsing and
>     consolidation with DT code.
>   arm64, pci, acpi: Provide ACPI-specific prerequisites for PCI bus
>     enumeration.
>   arm64, pci, acpi: Start using ACPI based PCI host controller driver
>     for ARM64.
>
>  arch/arm64/Kconfig                  |   1 +
>  arch/arm64/kernel/pci.c             |  34 +++-----
>  drivers/acpi/Kconfig                |   8 ++
>  drivers/acpi/Makefile               |   1 +
>  drivers/acpi/pci_mcfg.c             |  97 ++++++++++++++++++++++
>  drivers/acpi/pci_root.c             |  33 ++++++++
>  drivers/acpi/pci_root_generic.c     | 149 +++++++++++++++++++++++++++++++++
>  drivers/of/address.c                | 116 +-------------------------
>  drivers/pci/Kconfig                 |   3 +
>  drivers/pci/Makefile                |   2 +
>  drivers/pci/ecam.c                  | 161 ++++++++++++++++++++++++++++++++++++
>  drivers/pci/ecam.h                  |  72 ++++++++++++++++
>  drivers/pci/host/Kconfig            |   1 +
>  drivers/pci/host/pci-host-common.c  | 114 +++++++++++--------------
>  drivers/pci/host/pci-host-common.h  |  47 -----------
>  drivers/pci/host/pci-host-generic.c |  52 +++---------
>  drivers/pci/host/pci-thunder-ecam.c |  39 ++-------
>  drivers/pci/host/pci-thunder-pem.c  |  92 ++++++++++-----------
>  drivers/pci/pci.c                   | 150 ++++++++++++++++++++++++++++++++-
>  drivers/pci/probe.c                 |   2 +
>  include/linux/of_address.h          |   9 --
>  include/linux/pci-acpi.h            |  14 ++++
>  include/linux/pci.h                 |  11 ++-
>  23 files changed, 823 insertions(+), 385 deletions(-)
>  create mode 100644 drivers/acpi/pci_mcfg.c
>  create mode 100644 drivers/acpi/pci_root_generic.c
>  create mode 100644 drivers/pci/ecam.c
>  create mode 100644 drivers/pci/ecam.h
>  delete mode 100644 drivers/pci/host/pci-host-common.h
>
> --
> 1.9.1
>

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Lorenzo Pieralisi]

On Thu, May 12, 2016 at 01:27:23PM +0200, Rafael J. Wysocki wrote:
> On Thu, May 12, 2016 at 12:43 PM, Jayachandran C <jchandra@broadcom.com> wrote:
> > On Thu, May 12, 2016 at 4:13 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> >> On Wed, May 11, 2016 at 10:30:51PM +0200, Rafael J. Wysocki wrote:
> >>> On Wed, May 11, 2016 at 12:11 PM, Lorenzo Pieralisi
> >>> <lorenzo.pieralisi@arm.com> wrote:
> >>> > On Tue, May 10, 2016 at 08:37:00PM +0200, Rafael J. Wysocki wrote:
> >>> >> On Tue, May 10, 2016 at 5:19 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> 
> [cut]
> 
> >
> > If we are moving the ACPI/PCI code from drivers/acpi to
> > arch/arm64/ , there is an issue in having the header file
> > ecam.h in drivers/pci
> >
> > The current include of "../pci/ecam.h" is slightly ugly (Arnd
> > and David had already noted this), but including the driver
> > header from arch code would be even worse.
> >
> > I can either merge ecam.h into include/linux/pci.h
> > or move it to a new file include/linux/pci-ecam.h, any
> > suggestion on which is preferable?
> 
> My preference would be pci-ecam.h as we did a similar thing for
> pci-dma.h, for example, but basically this is up to Bjorn.

A word of caution for all interested parties, what we may move
to arch/arm64 (if Catalin and Will are ok with that) here is content
of drivers/acpi/pci_root_generic.c, not drivers/acpi/pci_mcfg.c (and
definitely not the MCFG quirks handling that is coming up next on top
of this series).

I just wanted to make sure we understand that MCFG quirks handling
like eg:

https://lkml.org/lkml/2016/4/28/790

that is coming up following this series has no chance whatsoever to
be handled within arch/arm64, it is just not going to happen.

Maybe I am jumping the gun, I just want to make sure that everyone is
aware that moving part of this series to arch/arm64 has implications,
(and that's why I said that moving part of this code to arch/arm64 is
not as simple as it looks) it may be ok to have an ACPI PCI
implementation that is arch/arm64 specific (mostly for IO space and PCI
resources assignment handling that unfortunately is not uniform across
X86, IA64 and ARM64), but MCFG quirks and related platform code stay out
of arch/arm64 I guess we are all aware of that, just wanted to make
sure :)

Lorenzo

Re: [PATCH V7 08/11] pci, acpi: Support for ACPI based generic PCI host controller

[Jayachandran C]

On Tue, May 10, 2016 at 8:49 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> This patch is going to implement generic PCI host controller for
> ACPI world, similar to what pci-host-generic.c driver does for DT world.
>
> All such drivers, which we have seen so far, were implemented within
> arch/ directory since they had some arch assumptions (x86 and ia64).
> However, they all are doing similar thing, so it makes sense to find
> some common code and abstract it into the generic driver.
>
> In order to handle PCI config space regions properly, we define new
> MCFG interface which does sanity checks on MCFG table and keeps its
> root pointer. User is able to lookup MCFG regions based on that root
> pointer and specified domain:bus_start:bus_end touple. We are using
> pci_mmcfg_late_init old prototype to avoid another function name.
>
> The implementation of pci_acpi_scan_root() looks up the MCFG entries
> and sets up a new mapping (regions are not mapped until host controller ask
> for it). Generic PCI functions are used for accessing config space.
> Driver selects PCI_ECAM and uses functions from drivers/pci/ecam.h
> to create and access ECAM mappings.
>
> As mentioned in Kconfig help section, ACPI_PCI_HOST_GENERIC choice
> should be made on a per-architecture basis.
>
> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> ---
[....]

> diff --git a/drivers/pci/ecam.h b/drivers/pci/ecam.h
> index 1ad2176..1cccf57 100644
> --- a/drivers/pci/ecam.h
> +++ b/drivers/pci/ecam.h
> @@ -45,6 +45,11 @@ struct pci_config_window {
>                 void __iomem            *win;   /* 64-bit single mapping */
>                 void __iomem            **winp; /* 32-bit per bus mapping */
>         };
> +#ifdef CONFIG_ACPI_PCI_HOST_GENERIC
> +       struct acpi_device              *companion; /* ACPI companion device */
> +#endif
> +       int                             domain;
> +
>  };

Using struct pci_config_window to pass along domain and
companion looks bad.  I think there are two possible options
to do this better:

1. add a 'struct fwnode_handle *' or 'struct device *parent_dev'
   instead of the companion and domain fields above. In case of
   ACPI either of them can be used to get the acpi_device and
   both  domain and companion can be set from that.

2. make pci_config_window fully embeddable by moving allocation
   out of pci_ecam_create to its callers. Then it can be embedded
   into acpi_pci_generic_root_info, and container_of can be used
   to get acpi info from ->sysdata.

The first option should be easier to implement but the second may
be better on long run. I would leave it to the Bjorn or Rafael to
suggest which is preferred.

JC.

Re: [PATCH V7 08/11] pci, acpi: Support for ACPI based generic PCI host controller

[Rafael J. Wysocki]

On Fri, May 13, 2016 at 1:25 PM, Jayachandran C <jchandra@broadcom.com> wrote:
> On Tue, May 10, 2016 at 8:49 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
>> This patch is going to implement generic PCI host controller for
>> ACPI world, similar to what pci-host-generic.c driver does for DT world.
>>
>> All such drivers, which we have seen so far, were implemented within
>> arch/ directory since they had some arch assumptions (x86 and ia64).
>> However, they all are doing similar thing, so it makes sense to find
>> some common code and abstract it into the generic driver.
>>
>> In order to handle PCI config space regions properly, we define new
>> MCFG interface which does sanity checks on MCFG table and keeps its
>> root pointer. User is able to lookup MCFG regions based on that root
>> pointer and specified domain:bus_start:bus_end touple. We are using
>> pci_mmcfg_late_init old prototype to avoid another function name.
>>
>> The implementation of pci_acpi_scan_root() looks up the MCFG entries
>> and sets up a new mapping (regions are not mapped until host controller ask
>> for it). Generic PCI functions are used for accessing config space.
>> Driver selects PCI_ECAM and uses functions from drivers/pci/ecam.h
>> to create and access ECAM mappings.
>>
>> As mentioned in Kconfig help section, ACPI_PCI_HOST_GENERIC choice
>> should be made on a per-architecture basis.
>>
>> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
>> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
>> ---
> [....]
>
>> diff --git a/drivers/pci/ecam.h b/drivers/pci/ecam.h
>> index 1ad2176..1cccf57 100644
>> --- a/drivers/pci/ecam.h
>> +++ b/drivers/pci/ecam.h
>> @@ -45,6 +45,11 @@ struct pci_config_window {
>>                 void __iomem            *win;   /* 64-bit single mapping */
>>                 void __iomem            **winp; /* 32-bit per bus mapping */
>>         };
>> +#ifdef CONFIG_ACPI_PCI_HOST_GENERIC
>> +       struct acpi_device              *companion; /* ACPI companion device */
>> +#endif
>> +       int                             domain;
>> +
>>  };
>
> Using struct pci_config_window to pass along domain and
> companion looks bad.  I think there are two possible options
> to do this better:
>
> 1. add a 'struct fwnode_handle *' or 'struct device *parent_dev'
>    instead of the companion and domain fields above. In case of
>    ACPI either of them can be used to get the acpi_device and
>    both  domain and companion can be set from that.
>
> 2. make pci_config_window fully embeddable by moving allocation
>    out of pci_ecam_create to its callers. Then it can be embedded
>    into acpi_pci_generic_root_info, and container_of can be used
>    to get acpi info from ->sysdata.
>
> The first option should be easier to implement but the second may
> be better on long run. I would leave it to the Bjorn or Rafael to
> suggest which is preferred.

Personally, I'd probably try to use fwnode_handle, but the second
option makes sense too in principle.

Re: [PATCH V7 08/11] pci, acpi: Support for ACPI based generic PCI host controller

[Tomasz Nowicki]

On 13.05.2016 13:31, Rafael J. Wysocki wrote:
> On Fri, May 13, 2016 at 1:25 PM, Jayachandran C <jchandra@broadcom.com> wrote:
>> On Tue, May 10, 2016 at 8:49 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
>>> This patch is going to implement generic PCI host controller for
>>> ACPI world, similar to what pci-host-generic.c driver does for DT world.
>>>
>>> All such drivers, which we have seen so far, were implemented within
>>> arch/ directory since they had some arch assumptions (x86 and ia64).
>>> However, they all are doing similar thing, so it makes sense to find
>>> some common code and abstract it into the generic driver.
>>>
>>> In order to handle PCI config space regions properly, we define new
>>> MCFG interface which does sanity checks on MCFG table and keeps its
>>> root pointer. User is able to lookup MCFG regions based on that root
>>> pointer and specified domain:bus_start:bus_end touple. We are using
>>> pci_mmcfg_late_init old prototype to avoid another function name.
>>>
>>> The implementation of pci_acpi_scan_root() looks up the MCFG entries
>>> and sets up a new mapping (regions are not mapped until host controller ask
>>> for it). Generic PCI functions are used for accessing config space.
>>> Driver selects PCI_ECAM and uses functions from drivers/pci/ecam.h
>>> to create and access ECAM mappings.
>>>
>>> As mentioned in Kconfig help section, ACPI_PCI_HOST_GENERIC choice
>>> should be made on a per-architecture basis.
>>>
>>> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
>>> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
>>> ---
>> [....]
>>
>>> diff --git a/drivers/pci/ecam.h b/drivers/pci/ecam.h
>>> index 1ad2176..1cccf57 100644
>>> --- a/drivers/pci/ecam.h
>>> +++ b/drivers/pci/ecam.h
>>> @@ -45,6 +45,11 @@ struct pci_config_window {
>>>                  void __iomem            *win;   /* 64-bit single mapping */
>>>                  void __iomem            **winp; /* 32-bit per bus mapping */
>>>          };
>>> +#ifdef CONFIG_ACPI_PCI_HOST_GENERIC
>>> +       struct acpi_device              *companion; /* ACPI companion device */
>>> +#endif
>>> +       int                             domain;
>>> +
>>>   };
>>
>> Using struct pci_config_window to pass along domain and
>> companion looks bad.  I think there are two possible options
>> to do this better:
>>
>> 1. add a 'struct fwnode_handle *' or 'struct device *parent_dev'
>>     instead of the companion and domain fields above. In case of
>>     ACPI either of them can be used to get the acpi_device and
>>     both  domain and companion can be set from that.
>>
>> 2. make pci_config_window fully embeddable by moving allocation
>>     out of pci_ecam_create to its callers. Then it can be embedded
>>     into acpi_pci_generic_root_info, and container_of can be used
>>     to get acpi info from ->sysdata.
>>
>> The first option should be easier to implement but the second may
>> be better on long run. I would leave it to the Bjorn or Rafael to
>> suggest which is preferred.
>
> Personally, I'd probably try to use fwnode_handle, but the second
> option makes sense too in principle.
>

Thanks for suggestions. I will try to use fwnode_handle

Tomasz

Re: [PATCH V7 08/11] pci, acpi: Support for ACPI based generic PCI host controller

[Jayachandran C]

On Tue, May 10, 2016 at 8:49 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> This patch is going to implement generic PCI host controller for
> ACPI world, similar to what pci-host-generic.c driver does for DT world.
>
> All such drivers, which we have seen so far, were implemented within
> arch/ directory since they had some arch assumptions (x86 and ia64).
> However, they all are doing similar thing, so it makes sense to find
> some common code and abstract it into the generic driver.
>
> In order to handle PCI config space regions properly, we define new
> MCFG interface which does sanity checks on MCFG table and keeps its
> root pointer. User is able to lookup MCFG regions based on that root
> pointer and specified domain:bus_start:bus_end touple. We are using
> pci_mmcfg_late_init old prototype to avoid another function name.
>
> The implementation of pci_acpi_scan_root() looks up the MCFG entries
> and sets up a new mapping (regions are not mapped until host controller ask
> for it). Generic PCI functions are used for accessing config space.
> Driver selects PCI_ECAM and uses functions from drivers/pci/ecam.h
> to create and access ECAM mappings.
>
> As mentioned in Kconfig help section, ACPI_PCI_HOST_GENERIC choice
> should be made on a per-architecture basis.

Looking thru the new code, I see a few issues, please see below

> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> ---
>  drivers/acpi/Kconfig            |   8 +++
>  drivers/acpi/Makefile           |   1 +
>  drivers/acpi/pci_mcfg.c         |  97 ++++++++++++++++++++++++++
>  drivers/acpi/pci_root_generic.c | 149 ++++++++++++++++++++++++++++++++++++++++
>  drivers/pci/ecam.h              |   5 ++
>  include/linux/pci-acpi.h        |   5 ++
>  include/linux/pci.h             |   5 +-
>  7 files changed, 269 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/acpi/pci_mcfg.c
>  create mode 100644 drivers/acpi/pci_root_generic.c
>
> diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
> index 183ffa3..44afc76 100644
> --- a/drivers/acpi/Kconfig
> +++ b/drivers/acpi/Kconfig
> @@ -346,6 +346,14 @@ config ACPI_PCI_SLOT
>           i.e., segment/bus/device/function tuples, with physical slots in
>           the system.  If you are unsure, say N.
>
> +config ACPI_PCI_HOST_GENERIC
> +       bool
> +       select PCI_ECAM
> +       help
> +         Select this config option from the architecture Kconfig,
> +         if it is preferred to enable ACPI PCI host controller driver which
> +         has no arch-specific assumptions.
> +
>  config X86_PM_TIMER
>         bool "Power Management Timer Support" if EXPERT
>         depends on X86
> diff --git a/drivers/acpi/Makefile b/drivers/acpi/Makefile
> index 81e5cbc..627a2b7 100644
> --- a/drivers/acpi/Makefile
> +++ b/drivers/acpi/Makefile
> @@ -40,6 +40,7 @@ acpi-$(CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC) += processor_pdc.o
>  acpi-y                         += ec.o
>  acpi-$(CONFIG_ACPI_DOCK)       += dock.o
>  acpi-y                         += pci_root.o pci_link.o pci_irq.o
> +obj-$(CONFIG_ACPI_PCI_HOST_GENERIC)    += pci_root_generic.o pci_mcfg.o
>  acpi-y                         += acpi_lpss.o acpi_apd.o
>  acpi-y                         += acpi_platform.o
>  acpi-y                         += acpi_pnp.o
> diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c
> new file mode 100644
> index 0000000..373d079
> --- /dev/null
> +++ b/drivers/acpi/pci_mcfg.c
> @@ -0,0 +1,97 @@
> +/*
> + * Copyright (C) 2016 Broadcom
> + *     Author: Jayachandran C <jchandra@broadcom.com>
> + * Copyright (C) 2016 Semihalf
> + *     Author: Tomasz Nowicki <tn@semihalf.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License, version 2, as
> + * published by the Free Software Foundation (the "GPL").
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * General Public License version 2 (GPLv2) for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * version 2 (GPLv2) along with this source code.
> + */
> +#include <linux/kernel.h>
> +#include <linux/pci.h>
> +#include <linux/pci-acpi.h>
> +
> +#define PREFIX "ACPI: "
> +
> +/* Root pointer to the mapped MCFG table */
> +static struct acpi_table_mcfg *mcfg_table;
> +
> +#define MCFG_ENTRIES(mcfg_ptr) (((mcfg_ptr)->header.length -           \
> +                               sizeof(struct acpi_table_mcfg)) /       \
> +                               sizeof(struct acpi_mcfg_allocation))

It would be better if you used static inline function here.

> +static phys_addr_t pci_mcfg_lookup_static(u16 seg, u8 bus_start, u8 bus_end)
> +{
> +       struct acpi_mcfg_allocation *mptr;
> +       int i;
> +
> +       if (!mcfg_table) {
> +               pr_err(PREFIX "MCFG table not available, lookup failed\n");
> +               return -ENXIO;
> +       }
> +
> +       mptr = (struct acpi_mcfg_allocation *) &mcfg_table[1];
> +
> +       /*
> +        * We expect exact match, unless MCFG entry end bus covers more than
> +        * specified by caller.
> +        */
> +       for (i = 0; i < MCFG_ENTRIES(mcfg_table); i++, mptr++) {
> +               if (mptr->pci_segment == seg &&
> +                   mptr->start_bus_number == bus_start &&
> +                   mptr->end_bus_number >= bus_end) {
> +                       return mptr->address;
> +               }
> +       }

There is an issue here, the bus range is obtained if different
ways. If the _CRS has it, this should be fine. But if it is _BBN-0xff
or the default 0-0xff, then I would think taking the MCFG entry end
would be better. This would require updating the bus resource end.

Also (trivial), the braces are not needed.

> +       return -ENXIO;
> +}
> +
> +phys_addr_t pci_mcfg_lookup(struct acpi_device *device, u16 seg,
> +                           struct resource *bus_res)
> +{
> +       phys_addr_t addr;
> +
> +       addr = acpi_pci_root_get_mcfg_addr(device->handle);
> +       if (addr)
> +               return addr;

When you have address from _CBA, you are assuming that the
bus range is also set correctly (from _CRS or as _BBN-0xff). Is
this assumption correct?

(this was discussed earlier)  you are doing the _CBA call again,
I think cleaning up the _CBA, _BBN, _CRS and MCFG based
ECAM area resources and defaults should be a different patch,
which would need changes to pci_root.c. We should use
root->mcfg_addr in this patchset.

> +
> +       return pci_mcfg_lookup_static(seg, bus_res->start, bus_res->end);

There is no need to have a separate function for this, based on by
above comment, you might need to change bus_res, so having it here
would be better.

> +}
> +
> +static __init int pci_mcfg_parse(struct acpi_table_header *header)
> +{
> +       struct acpi_table_mcfg *mcfg;
> +       int n;
> +
> +       if (!header)
> +               return -EINVAL;

This is not needed, the handler is not called if header is NULL

> +
> +       mcfg = (struct acpi_table_mcfg *)header;
> +       n = MCFG_ENTRIES(mcfg);
> +       if (n <= 0 || n > 255) {
> +               pr_err(PREFIX "MCFG has incorrect entries (%d).\n", n);
> +               return -EINVAL;
> +       }
>
> +       mcfg_table = mcfg;

Saving a reference of ACPI mapping seems dangerous, acpi_parse_table
calles early_acpi_os_unmap_memory() after calling handler, which does
not do anything since acpi_gbl_permanent_mmap is set.

I would suggest copying the entries.

> +       pr_info(PREFIX "MCFG table loaded, %d entries detected\n", n);
> +       return 0;
> +}
> +
> +/* Interface called by ACPI - parse and save MCFG table */
> +void __init pci_mmcfg_late_init(void)
> +{
> +       int err = acpi_table_parse(ACPI_SIG_MCFG, pci_mcfg_parse);
> +       if (err)
> +               pr_err(PREFIX "Failed to parse MCFG (%d)\n", err);
> +}


JC.

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Dongdong Liu]

Hi Tomasz

I used the patchset and added "PATCH V6 11/13 specic quirks", tested on HiSilicon D02 board but met the below problem.

[    2.614115] [<ffffff80083b13bc>] hisi_pcie_init+0x6c/0x1ec
[    2.619571] [<ffffff80083ab060>] pci_ecam_create+0x130/0x1ec
[    2.625209] [<ffffff80083f3764>] pci_acpi_scan_root+0x160/0x218
[    2.631096] [<ffffff80083d1f6c>] acpi_pci_root_add+0x36c/0x42c
[    2.636897] [<ffffff80083ce36c>] acpi_bus_attach+0xe4/0x1a8
[    2.642438] [<ffffff80083ce3d8>] acpi_bus_attach+0x150/0x1a8
[    2.648066] [<ffffff80083ce3d8>] acpi_bus_attach+0x150/0x1a8
[    2.653693] [<ffffff80083ce55c>] acpi_bus_scan+0x64/0x74
[    2.658975] [<ffffff8008ae665c>] acpi_scan_init+0x5c/0x19c
[    2.664429] [<ffffff8008ae6408>] acpi_init+0x280/0x2a4
[    2.669538] [<ffffff80080829e8>] do_one_initcall+0x8c/0x19c
[    2.675080] [<ffffff8008ac3af8>] kernel_init_freeable+0x14c/0x1ec
[    2.681139] [<ffffff80087a8438>] kernel_init+0x10/0xfc
[    2.686248] [<ffffff8008085e10>] ret_from_fork+0x10/0x40

In hisi_pcie_init, I used "struct acpi_device *device = ACPI_COMPANION(dev);".
I found the reason is V7 lack the below code. I added the below code, it worked ok.

[PATCH V6 01/13] pci, acpi, x86, ia64: Move ACPI host bridge device companion assignment to core code.
--- a/drivers/acpi/pci_root.c
+++ b/drivers/acpi/pci_root.c
@@ -564,6 +564,11 @@ static int acpi_pci_root_add(struct acpi_device *device,
  		}
  	}

+	/*
+	 * pci_create_root_bus() needs to detect the parent device type,
+	 * so initialize its companion data accordingly.
+	 */
+	ACPI_COMPANION_SET(&device->dev, device);

This code will be upstreamed with the "PATCH V6 11/13 specic quirks" in next time after the patchset is accepted.
Right ?

在 2016/5/10 23:19, Tomasz Nowicki 写道:
> This patch provides a way to set the ACPI companion in PCI code.
> We define acpi_pci_set_companion() to set the ACPI companion pointer and
> call it from PCI core code. The function is stub for now.
>
> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> ---
>   drivers/pci/probe.c      | 2 ++
>   include/linux/pci-acpi.h | 4 ++++
>   2 files changed, 6 insertions(+)
>
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index 8004f67..fb0b752 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -12,6 +12,7 @@
>   #include <linux/slab.h>
>   #include <linux/module.h>
>   #include <linux/cpumask.h>
> +#include <linux/pci-acpi.h>
>   #include <linux/pci-aspm.h>
>   #include <linux/aer.h>
>   #include <linux/acpi.h>
> @@ -2141,6 +2142,7 @@ struct pci_bus *pci_create_root_bus(struct device *parent, int bus,
>   	bridge->dev.parent = parent;
>   	bridge->dev.release = pci_release_host_bridge_dev;
>   	dev_set_name(&bridge->dev, "pci%04x:%02x", pci_domain_nr(b), bus);
> +	acpi_pci_set_companion(bridge);
>   	error = pcibios_root_bridge_prepare(bridge);
>   	if (error) {
>   		kfree(bridge);
> diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h
> index 09f9f02..1baa515 100644
> --- a/include/linux/pci-acpi.h
> +++ b/include/linux/pci-acpi.h
> @@ -111,6 +111,10 @@ static inline void acpi_pci_add_bus(struct pci_bus *bus) { }
>   static inline void acpi_pci_remove_bus(struct pci_bus *bus) { }
>   #endif	/* CONFIG_ACPI */
>
> +static inline void acpi_pci_set_companion(struct pci_host_bridge *bridge)
> +{
> +}
> +
>   static inline int acpi_pci_bus_domain_nr(struct pci_bus *bus)
>   {
>   	return 0;
>

Re: [PATCH V7 07/11] pci, acpi: Handle ACPI companion assignment.

[Tomasz Nowicki]

On 17.05.2016 05:11, Dongdong Liu wrote:
> Hi Tomasz
>
> I used the patchset and added "PATCH V6 11/13 specic quirks", tested on
> HiSilicon D02 board but met the below problem.
>
> [    2.614115] [<ffffff80083b13bc>] hisi_pcie_init+0x6c/0x1ec
> [    2.619571] [<ffffff80083ab060>] pci_ecam_create+0x130/0x1ec
> [    2.625209] [<ffffff80083f3764>] pci_acpi_scan_root+0x160/0x218
> [    2.631096] [<ffffff80083d1f6c>] acpi_pci_root_add+0x36c/0x42c
> [    2.636897] [<ffffff80083ce36c>] acpi_bus_attach+0xe4/0x1a8
> [    2.642438] [<ffffff80083ce3d8>] acpi_bus_attach+0x150/0x1a8
> [    2.648066] [<ffffff80083ce3d8>] acpi_bus_attach+0x150/0x1a8
> [    2.653693] [<ffffff80083ce55c>] acpi_bus_scan+0x64/0x74
> [    2.658975] [<ffffff8008ae665c>] acpi_scan_init+0x5c/0x19c
> [    2.664429] [<ffffff8008ae6408>] acpi_init+0x280/0x2a4
> [    2.669538] [<ffffff80080829e8>] do_one_initcall+0x8c/0x19c
> [    2.675080] [<ffffff8008ac3af8>] kernel_init_freeable+0x14c/0x1ec
> [    2.681139] [<ffffff80087a8438>] kernel_init+0x10/0xfc
> [    2.686248] [<ffffff8008085e10>] ret_from_fork+0x10/0x40
>
> In hisi_pcie_init, I used "struct acpi_device *device =
> ACPI_COMPANION(dev);".
> I found the reason is V7 lack the below code. I added the below code, it
> worked ok.
>
> [PATCH V6 01/13] pci, acpi, x86, ia64: Move ACPI host bridge device
> companion assignment to core code.
> --- a/drivers/acpi/pci_root.c
> +++ b/drivers/acpi/pci_root.c
> @@ -564,6 +564,11 @@ static int acpi_pci_root_add(struct acpi_device
> *device,
>           }
>       }
>
> +    /*
> +     * pci_create_root_bus() needs to detect the parent device type,
> +     * so initialize its companion data accordingly.
> +     */
> +    ACPI_COMPANION_SET(&device->dev, device);
>
> This code will be upstreamed with the "PATCH V6 11/13 specic quirks" in
> next time after the patchset is accepted.
> Right ?

We had that patch in previous series to retrieve PCI domain nicely. But 
that has bad implication to userspace. See:
https://lkml.org/lkml/2016/5/9/918

I understand that:
[PATCH V6 01/13] pci, acpi, x86, ia64: Move ACPI host bridge device 
companion assignment to core code.
helps to get firmware specific info in hisi_pcie_init but we need to 
figure out something better for quirk handling too.

Tomasz

Re: [PATCH V7 08/11] pci, acpi: Support for ACPI based generic PCI host controller

[Matthias Brugger]

On 10/05/16 17:19, Tomasz Nowicki wrote:
> This patch is going to implement generic PCI host controller for
> ACPI world, similar to what pci-host-generic.c driver does for DT world.
>
> All such drivers, which we have seen so far, were implemented within
> arch/ directory since they had some arch assumptions (x86 and ia64).
> However, they all are doing similar thing, so it makes sense to find
> some common code and abstract it into the generic driver.
>
> In order to handle PCI config space regions properly, we define new
> MCFG interface which does sanity checks on MCFG table and keeps its
> root pointer. User is able to lookup MCFG regions based on that root
> pointer and specified domain:bus_start:bus_end touple. We are using
> pci_mmcfg_late_init old prototype to avoid another function name.
>
> The implementation of pci_acpi_scan_root() looks up the MCFG entries
> and sets up a new mapping (regions are not mapped until host controller ask
> for it). Generic PCI functions are used for accessing config space.
> Driver selects PCI_ECAM and uses functions from drivers/pci/ecam.h
> to create and access ECAM mappings.
>
> As mentioned in Kconfig help section, ACPI_PCI_HOST_GENERIC choice
> should be made on a per-architecture basis.
>
> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
> ---
>   drivers/acpi/Kconfig            |   8 +++
>   drivers/acpi/Makefile           |   1 +
>   drivers/acpi/pci_mcfg.c         |  97 ++++++++++++++++++++++++++
>   drivers/acpi/pci_root_generic.c | 149 ++++++++++++++++++++++++++++++++++++++++
>   drivers/pci/ecam.h              |   5 ++
>   include/linux/pci-acpi.h        |   5 ++
>   include/linux/pci.h             |   5 +-
>   7 files changed, 269 insertions(+), 1 deletion(-)
>   create mode 100644 drivers/acpi/pci_mcfg.c
>   create mode 100644 drivers/acpi/pci_root_generic.c
>
> diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
> index 183ffa3..44afc76 100644
> --- a/drivers/acpi/Kconfig
> +++ b/drivers/acpi/Kconfig
> @@ -346,6 +346,14 @@ config ACPI_PCI_SLOT
>   	  i.e., segment/bus/device/function tuples, with physical slots in
>   	  the system.  If you are unsure, say N.
>
> +config ACPI_PCI_HOST_GENERIC
> +	bool
> +	select PCI_ECAM
> +	help
> +	  Select this config option from the architecture Kconfig,
> +	  if it is preferred to enable ACPI PCI host controller driver which
> +	  has no arch-specific assumptions.
> +
>   config X86_PM_TIMER
>   	bool "Power Management Timer Support" if EXPERT
>   	depends on X86
> diff --git a/drivers/acpi/Makefile b/drivers/acpi/Makefile
> index 81e5cbc..627a2b7 100644
> --- a/drivers/acpi/Makefile
> +++ b/drivers/acpi/Makefile
> @@ -40,6 +40,7 @@ acpi-$(CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC) += processor_pdc.o
>   acpi-y				+= ec.o
>   acpi-$(CONFIG_ACPI_DOCK)	+= dock.o
>   acpi-y				+= pci_root.o pci_link.o pci_irq.o
> +obj-$(CONFIG_ACPI_PCI_HOST_GENERIC)	+= pci_root_generic.o pci_mcfg.o
>   acpi-y				+= acpi_lpss.o acpi_apd.o
>   acpi-y				+= acpi_platform.o
>   acpi-y				+= acpi_pnp.o
> diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c
> new file mode 100644
> index 0000000..373d079
> --- /dev/null
> +++ b/drivers/acpi/pci_mcfg.c
> @@ -0,0 +1,97 @@
> +/*
> + * Copyright (C) 2016 Broadcom
> + *	Author: Jayachandran C <jchandra@broadcom.com>
> + * Copyright (C) 2016 Semihalf
> + * 	Author: Tomasz Nowicki <tn@semihalf.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License, version 2, as
> + * published by the Free Software Foundation (the "GPL").
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * General Public License version 2 (GPLv2) for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * version 2 (GPLv2) along with this source code.
> + */
> +#include <linux/kernel.h>
> +#include <linux/pci.h>
> +#include <linux/pci-acpi.h>
> +
> +#define PREFIX	"ACPI: "
> +
> +/* Root pointer to the mapped MCFG table */
> +static struct acpi_table_mcfg *mcfg_table;
> +
> +#define MCFG_ENTRIES(mcfg_ptr)	(((mcfg_ptr)->header.length -		\
> +				sizeof(struct acpi_table_mcfg)) /	\
> +				sizeof(struct acpi_mcfg_allocation))
> +
> +static phys_addr_t pci_mcfg_lookup_static(u16 seg, u8 bus_start, u8 bus_end)
> +{
> +	struct acpi_mcfg_allocation *mptr;
> +	int i;
> +
> +	if (!mcfg_table) {
> +		pr_err(PREFIX "MCFG table not available, lookup failed\n");
> +		return -ENXIO;
> +	}
> +
> +	mptr = (struct acpi_mcfg_allocation *) &mcfg_table[1];
> +
> +	/*
> +	 * We expect exact match, unless MCFG entry end bus covers more than
> +	 * specified by caller.
> +	 */
> +	for (i = 0; i < MCFG_ENTRIES(mcfg_table); i++, mptr++) {
> +		if (mptr->pci_segment == seg &&
> +		    mptr->start_bus_number == bus_start &&
> +		    mptr->end_bus_number >= bus_end) {
> +			return mptr->address;
> +		}
> +	}
> +
> +	return -ENXIO;
> +}
> +
> +phys_addr_t pci_mcfg_lookup(struct acpi_device *device, u16 seg,
> +			    struct resource *bus_res)
> +{
> +	phys_addr_t addr;
> +
> +	addr = acpi_pci_root_get_mcfg_addr(device->handle);
> +	if (addr)
> +		return addr;
> +
> +	return pci_mcfg_lookup_static(seg, bus_res->start, bus_res->end);
> +}
> +
> +static __init int pci_mcfg_parse(struct acpi_table_header *header)
> +{
> +	struct acpi_table_mcfg *mcfg;
> +	int n;
> +
> +	if (!header)
> +		return -EINVAL;

Maybe that's bike shedding, but acpi_table_parse already checks that the 
table passed to the handler exists, so this is redundant.

Regards,
Matthias

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Jeremy Linton]

On 05/10/2016 10:19 AM, Tomasz Nowicki wrote:
>  From the functionality point of view this series may be split into the
> following logic parts:
> 1. New ECAM API and update for users of the pci-host-common API
> 2. Necessary fixes as the preparation for using driver on ARM64.
> 3. Use new MCFG interface and implement generic ACPI based PCI host controller driver.
> 4. Enable above driver on ARM64

Thomasz,

Unless i'm doing something wrong, this patch causes a build break if 
NUMA is enabled in a 4.6+ kernel.

Thanks,



drivers/built-in.o: In function `pci_device_add':
/home/jlinton/linux-main/drivers/pci/probe.c:1744: undefined reference 
to `pcibus_to_node'
drivers/built-in.o: In function `pci_create_root_bus':
/home/jlinton/linux-main/drivers/pci/probe.c:2163: undefined reference 
to `pcibus_to_node'
drivers/built-in.o: In function `cpulistaffinity_show':
/home/jlinton/linux-main/drivers/pci/pci-sysfs.c:123: undefined 
reference to `pcibus_to_node'
/home/jlinton/linux-main/drivers/pci/pci-sysfs.c:123: undefined 
reference to `pcibus_to_node'
drivers/built-in.o: In function `cpuaffinity_show':
/home/jlinton/linux-main/drivers/pci/pci-sysfs.c:114: undefined 
reference to `pcibus_to_node'
drivers/built-in.o:/home/jlinton/linux-main/drivers/pci/pci-sysfs.c:114: 
more undefined references to `pcibus_to_node' follow
Makefile:937: recipe for target 'vmlinux' failed

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Jon Masters]

Hi Tomasz, all,

On 05/11/2016 07:08 AM, Tomasz Nowicki wrote:

> On 11.05.2016 12:41, Gabriele Paoloni wrote:

>>> v6 -> v7
>>> - drop quirks handling
>>
>> Maybe I missed something in the v6 discussion thread; when was it
>> decided to drop quirk handling?
> 
> I had such requests in previous series.

A quick note on quirk handling. This, I believe, applies post-merge of
the base infrastructure, which I realize will likely not have quirks.

We've some "gen1" ARMv8 server platforms where we end up doing quirks
(for things like forcing 32-bit config space accessors and the like) due
to people repurposing existing embedded PCIe IP blocks or using them for
the first time (especially in servers), and those being involved in the
design not necessarily seeing this problem ahead of time, or not
realizing that it would be an issue for servers. In the early days of
ARM server designs 3-4 years ago, many of us had never really played
with ECAM or realized how modern topologies are built.

Anyway. We missed this one in our SBSA requirements. They say (words to
the effect of) "thou shalt do PCIe the way it is done on servers" but
they aren't prescriptive, and they don't tell people how that actually
is in reality. That is being fixed. A lot of things are happening behind
the scenes - especially with third party IP block providers (all of whom
myself and others are speaking with directly about this) - to ensure
that the next wave of designs won't repeat these mistakes. We don't have
a time machine, but we can contain this from becoming an ongoing mess
for upstream, and we will do so. It won't be a zoo.

Various proposals have arisen for how to handle quirks in the longer
term, including elaborate frameworks and tables to describe them
generically. I would like to caution against such approaches, especially
in the case that they deviate from practice on x86, or prior to being
standardized fully with other Operating System vendors. I don't expect
there to be too many more than the existing initial set of quirks we
have seen posted. A number of "future" server SoCs have already been
fixed prior to silicon, and new design starts are being warned not to
make this a problem for us to have to clean up later.

So, I would like to suggest that the eventual framework mirror the
existing approach on x86 systems (matching DMI, etc.) and not be made
into some kind of generic, utopia. This is a case where we want there to
be pain involved (and upstream patches required) when people screw up,
so that they have a level of pain in response to ever making this
mistake in the future. If we try to create too grand a generic scheme
and make it too easy to handle this kind of situation beyond the small
number of existing offenders, we undermine efforts to force vendors to
ensure that their IP blocks are compliant going forward.

Side note: if you're a third party IP vendor and we didn't already speak
about this one, drop me a line, and let's collaborate also on your test
frameworks to make sure you're covered as well.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Ard Biesheuvel]

On 20 May 2016 at 06:41, Jon Masters <jcm@redhat.com> wrote:
> Hi Tomasz, all,
>
> On 05/11/2016 07:08 AM, Tomasz Nowicki wrote:
>
>> On 11.05.2016 12:41, Gabriele Paoloni wrote:
>
>>>> v6 -> v7
>>>> - drop quirks handling
>>>
>>> Maybe I missed something in the v6 discussion thread; when was it
>>> decided to drop quirk handling?
>>
>> I had such requests in previous series.
>
> A quick note on quirk handling. This, I believe, applies post-merge of
> the base infrastructure, which I realize will likely not have quirks.
>
> We've some "gen1" ARMv8 server platforms where we end up doing quirks
> (for things like forcing 32-bit config space accessors and the like) due
> to people repurposing existing embedded PCIe IP blocks or using them for
> the first time (especially in servers), and those being involved in the
> design not necessarily seeing this problem ahead of time, or not
> realizing that it would be an issue for servers. In the early days of
> ARM server designs 3-4 years ago, many of us had never really played
> with ECAM or realized how modern topologies are built.
>
> Anyway. We missed this one in our SBSA requirements. They say (words to
> the effect of) "thou shalt do PCIe the way it is done on servers" but
> they aren't prescriptive, and they don't tell people how that actually
> is in reality. That is being fixed. A lot of things are happening behind
> the scenes - especially with third party IP block providers (all of whom
> myself and others are speaking with directly about this) - to ensure
> that the next wave of designs won't repeat these mistakes. We don't have
> a time machine, but we can contain this from becoming an ongoing mess
> for upstream, and we will do so. It won't be a zoo.
>
> Various proposals have arisen for how to handle quirks in the longer
> term, including elaborate frameworks and tables to describe them
> generically. I would like to caution against such approaches, especially
> in the case that they deviate from practice on x86, or prior to being
> standardized fully with other Operating System vendors. I don't expect
> there to be too many more than the existing initial set of quirks we
> have seen posted. A number of "future" server SoCs have already been
> fixed prior to silicon, and new design starts are being warned not to
> make this a problem for us to have to clean up later.
>
> So, I would like to suggest that the eventual framework mirror the
> existing approach on x86 systems (matching DMI, etc.) and not be made
> into some kind of generic, utopia. This is a case where we want there to
> be pain involved (and upstream patches required) when people screw up,
> so that they have a level of pain in response to ever making this
> mistake in the future. If we try to create too grand a generic scheme
> and make it too easy to handle this kind of situation beyond the small
> number of existing offenders, we undermine efforts to force vendors to
> ensure that their IP blocks are compliant going forward.
>

I understand that there is a desire from the RedHat side to mimic x86
as closely as possible, but I never saw any technical justification
for that. DMI contains strings that are visible to userland, and you
effectively lock those down to certain values just so that the kernel
can distinguish a broken PCIe root complex from a working one. Linux
on x86 had no choice, since the overwhelming majority of existing
hardware misrepresented itself as generic, and DMI was the only thing
available to actually distinguish these broken implementations from
one another. This does not mean we should allow and/or encourage this
first gen hardware to misrepresent non-compliant hardware as compliant
as well.

Since you are talking to all the people involved, how about you
convince them to put something in the ACPI tables that allows the
kernel to distinguish those non-standard PCIe implementations from
hardware that is really generic? This way, we can sidestep the quirks
debate entirely, since it will simply be a different device as far as
the kernel is concerned. This is no worse than a quirk from a
practical point of view, since an older OS will be equally unable to
run on newer hardware, but it is arguably more true to the standards
compliance you tend to preach about, especially since this small pool
of third party IP could potentially be identified directly rather than
based on some divination of the SoC we may or may not be running on. I
am also convinced that adding support for an additional HID() to the
ACPI ECAM driver with some special config space handling wired in is
an easier sell upstream than making the same ugly mess x86 has had to
make because they did not have any choice to begin with.

If we do need a quirks handling mechanism, I still don't see how the
x86 situation extrapolates to ARM. ACPI offers plenty of ways for a
SoC vendor to identify the make and particular revision, and quirks
could be keyed off of that.

-- 
Ard.

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 209 bytes --]

On 05/10/2016 11:19 AM, Tomasz Nowicki wrote:

> This has been tested on Cavium ThunderX server. Any help in reviewing and
> testing is very appreciated.

Boot log from Intel Itanium 2 system attached.

Jon.


[-- Attachment #2: hamartia-4.6.0-rc7.jcm_tomasz_pciv7_test+_boot.txt --]
[-- Type: text/plain, Size: 21539 bytes --]

[    0.000000] Linux version 4.6.0-rc7.jcm_tomasz_pciv7_test+ (jcm@hamartia) (gcc version 4.4.5 (Debian 4.4.5-8) ) #2 SMP Fri May 20 03:24:58 EDT 2016
[    0.000000] EFI v1.10 by HP:
[    0.000000] efi:  SALsystab=0x3fefa000  ACPI 2.0=0x3fd5e000  SMBIOS=0x3fefc000  HCDP=0x3fd5c000 
[    0.000000] booting generic kernel on platform hpzx1
[    0.000000] PCDP: v0 at 0x3fd5c000
[    0.000000] earlycon: uart8250 at MMIO 0x00000000ff5e0000 (options '9600n8')
[    0.000000] bootconsole [uart8250] enabled
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x000000003FD5E000 000028 (v02 HP    )
[    0.000000] ACPI: XSDT 0x000000003FD5E02C 00008C (v01 HP     rx2620   00000000 HP   00000000)
[    0.000000] ACPI: FACP 0x000000003FD672A0 0000F4 (v03 HP     rx2620   00000000 HP   00000000)
[    0.000000] ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 32/16 (20160108/tbfadt-623)
[    0.000000] ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe1Block: 32/16 (20160108/tbfadt-623)
[    0.000000] ACPI: DSDT 0x000000003FD5E100 005F3C (v01 HP     rx2620   00000007 INTL 02012044)
[    0.000000] ACPI: FACS 0x000000003FD67398 000040
[    0.000000] ACPI: SPCR 0x000000003FD673D8 000050 (v01 HP     rx2620   00000000 HP   00000000)
[    0.000000] ACPI: DBGP 0x000000003FD67428 000034 (v01 HP     rx2620   00000000 HP   00000000)
[    0.000000] ACPI: APIC 0x000000003FD67520 0000A4 (v01 HP     rx2620   00000000 HP   00000000)
[    0.000000] ACPI: SPMI 0x000000003FD67460 000050 (v04 HP     rx2620   00000000 HP   00000000)
[    0.000000] ACPI: CPEP 0x000000003FD674B0 000034 (v01 HP     rx2620   00000000 HP   00000000)
[    0.000000] ACPI: SSDT 0x000000003FD64040 0001D6 (v01 HP     rx2620   00000006 INTL 02012044)
[    0.000000] ACPI: SSDT 0x000000003FD64220 000702 (v01 HP     rx2620   00000006 INTL 02012044)
[    0.000000] ACPI: SSDT 0x000000003FD64930 000A16 (v01 HP     rx2620   00000006 INTL 02012044)
[    0.000000] ACPI: SSDT 0x000000003FD65350 000A16 (v01 HP     rx2620   00000006 INTL 02012044)
[    0.000000] ACPI: SSDT 0x000000003FD65D70 000A16 (v01 HP     rx2620   00000006 INTL 02012044)
[    0.000000] ACPI: SSDT 0x000000003FD66790 000A16 (v01 HP     rx2620   00000006 INTL 02012044)
[    0.000000] ACPI: SSDT 0x000000003FD671B0 0000EB (v01 HP     rx2620   00000006 INTL 02012044)
[    0.000000] ACPI: Local APIC address c0000000fee00000
[    0.000000] 1 CPUs available, 1 CPUs total
[    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] warning: skipping physical page 0
[    0.000000] warning: skipping physical page 0
[    0.000000] warning: skipping physical page 0
[    0.000000] warning: skipping physical page 0
[    0.000000] Initial ramdisk at: 0xe00000003d4ac000 (17110715 bytes)
[    0.000000] SAL 3.1: HP version 3.15
[    0.000000] SAL Platform features: None
[    0.000000] SAL: AP wakeup using external interrupt vector 0xff
[    0.000000] MCA related initialization done
[    0.000000] warning: skipping physical page 0
[    0.000000] Virtual mem_map starts at 0xa0007fffffc80000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000004000-0x00000000ffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000004000-0x000000003f4b3fff]
[    0.000000]   node   0: [mem 0x000000003fc00000-0x000000003fc7bfff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000004000-0x000000003fc7bfff]
[    0.000000] Built 1 zonelists in Node order, mobility grouping off.  Total pages: 64619
[    0.000000] Policy zone: DMA
[    0.000000] Kernel command line: BOOT_IMAGE=scsi0:/EFI/debian/vmtest root=/dev/sda2  ro
[    0.000000] PID hash table entries: 4096 (order: 1, 32768 bytes)
[    0.000000] Sorting __ex_table...
[    0.000000] Memory: 989904K/1037488K available (12139K kernel code, 1118K rwdata, 1432K rodata, 880K init, 807K bss, 47584K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=256
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1
[    0.000000] NR_IRQS:1024
[    0.000000] ACPI: Local APIC address c0000000fee00000
[    0.000000] GSI 36 (level, low) -> CPU 0 (0x0000) vector 49
[    0.000000] clocksource: itc: mask: 0xffffffffffffffff max_cycles: 0x17101df767f, max_idle_ns: 440795286379 ns
[    0.000000] Console: colour dummy device 80x25
[    0.192000] Calibrating delay loop... 2390.01 BogoMIPS (lpj=4780032)
[    0.288000] pid_max: default: 32768 minimum: 301
[    0.348000] ACPI: Core revision 20160108
[    0.404000] ACPI: 8 ACPI AML tables successfully acquired and loaded
[    0.488000] 
[    0.504000] Security Framework initialized
[    0.560000] SELinux:  Disabled at boot.
[    0.608000] Dentry cache hash table entries: 131072 (order: 6, 1048576 bytes)
[    0.704000] Inode-cache hash table entries: 65536 (order: 5, 524288 bytes)
[    0.796000] Mount-cache hash table entries: 2048 (order: 0, 16384 bytes)
[    0.884000] Mountpoint-cache hash table entries: 2048 (order: 0, 16384 bytes)
[    0.980000] Boot processor id 0x0/0x0
[    1.028000] Brought up 1 CPUs
[    1.068000] Total of 1 processors activated (2390.01 BogoMIPS).
[    1.144000] devtmpfs: initialized
[    1.188000] SMBIOS 2.3 present.
[    1.232000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    1.360000] NET: Registered protocol family 16
[    1.420000] ACPI: bus type PCI registered
[    1.472000] HugeTLB registered 256 MB page size, pre-allocated 0 pages
[    1.560000] ACPI: Added _OSI(Module Device)
[    1.616000] ACPI: Added _OSI(Processor Device)
[    1.672000] ACPI: Added _OSI(3.0 _SCP Extensions)
[    1.736000] ACPI: Added _OSI(Processor Aggregator Device)
[    1.828000] ACPI: Interpreter enabled
[    1.876000] ACPI: (supports S0 S5)
[    1.920000] ACPI: Using IOSAPIC for interrupt routing
[    2.128000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-1f])
[    2.212000] acpi HWP0002:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    2.320000] acpi HWP0002:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    2.416000] PCI host bridge to bus 0000:00
[    2.472000] pci_bus 0000:00: root bus resource [io  0x0000-0x1fff window]
[    2.560000] pci_bus 0000:00: root bus resource [mem 0x80000000-0x8fffffff window]
[    2.660000] pci_bus 0000:00: root bus resource [mem 0x80004000000-0x80103fffffe window]
[    2.764000] pci_bus 0000:00: root bus resource [bus 00-1f]
[    2.876000] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 20-3f])
[    2.960000] acpi HWP0002:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    3.068000] acpi HWP0002:01: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    3.168000] PCI host bridge to bus 0000:20
[    3.224000] pci_bus 0000:20: root bus resource [io  0x2000-0x2fff window]
[    3.312000] pci_bus 0000:20: root bus resource [mem 0x90000000-0x97ffffff window]
[    3.412000] pci_bus 0000:20: root bus resource [mem 0x90004000000-0x90103fffffe window]
[    3.516000] pci_bus 0000:20: root bus resource [bus 20-3f]
[    3.632000] ACPI: PCI Root Bridge [PCI2] (domain 0000 [bus 40-5f])
[    3.712000] acpi HWP0002:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    3.820000] acpi HWP0002:02: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    3.920000] PCI host bridge to bus 0000:40
[    3.972000] pci_bus 0000:40: root bus resource [io  0x3000-0x5fff window]
[    4.064001] pci_bus 0000:40: root bus resource [mem 0x98000000-0xafffffff window]
[    4.160001] pci_bus 0000:40: root bus resource [mem 0xa0004000000-0xa0103fffffe window]
[    4.268001] pci_bus 0000:40: root bus resource [bus 40-5f]
[    4.364001] ACPI: PCI Root Bridge [PCI3] (domain 0000 [bus 60-7f])
[    4.444001] acpi HWP0002:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    4.552001] acpi HWP0002:03: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    4.652001] PCI host bridge to bus 0000:60
[    4.704001] pci_bus 0000:60: root bus resource [io  0x6000-0x7fff window]
[    4.796001] pci_bus 0000:60: root bus resource [mem 0xb0000000-0xc7ffffff window]
[    4.892001] pci_bus 0000:60: root bus resource [mem 0xb0004000000-0xb0103fffffe window]
[    5.000001] pci_bus 0000:60: root bus resource [bus 60-7f]
[    5.096001] ACPI: PCI Root Bridge [PCI4] (domain 0000 [bus 80-bf])
[    5.176001] acpi HWP0002:04: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    5.284001] acpi HWP0002:04: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    5.384001] PCI host bridge to bus 0000:80
[    5.436001] pci_bus 0000:80: root bus resource [io  0x8000-0xbfff window]
[    5.524001] pci_bus 0000:80: root bus resource [mem 0xc8000000-0xdfffffff window]
[    5.624001] pci_bus 0000:80: root bus resource [mem 0xc0004000000-0xc0103fffffe window]
[    5.728001] pci_bus 0000:80: root bus resource [bus 80-bf]
[    5.828001] ACPI: PCI Root Bridge [PCI6] (domain 0000 [bus c0-df])
[    5.908001] acpi HWP0002:05: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[    6.016001] acpi HWP0002:05: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    6.112001] PCI host bridge to bus 0000:c0
[    6.168001] pci_bus 0000:c0: root bus resource [io  0xc000-0xdfff window]
[    6.256001] pci_bus 0000:c0: root bus resource [mem 0xe0000000-0xefffffff window]
[    6.356001] pci_bus 0000:c0: root bus resource [mem 0xe0004000000-0xe0103fffffe window]
[    6.460001] pci_bus 0000:c0: root bus resource [bus c0-df]
[    6.548001] ACPI: Enabled 1 GPEs in block 10 to 1F
[    6.608001] vgaarb: loaded
[    6.644001] SCSI subsystem initialized
[    6.696001] ACPI: bus type USB registered
[    6.748001] usbcore: registered new interface driver usbfs
[    6.820001] usbcore: registered new interface driver hub
[    6.888001] usbcore: registered new device driver usb
[    6.960001] IOC: zx1 2.3 HPA 0xfed01000 IOVA space 1024Mb at 0x40000000
[    7.096001] clocksource: Switched to clocksource itc
[    7.165391] VFS: Disk quotas dquot_6.6.0
[    7.217160] VFS: Dquot-cache hash table entries: 2048 (order 0, 16384 bytes)
[    7.310088] pnp: PnP ACPI init
[    7.350492] GSI 34 (level, low) -> CPU 0 (0x0000) vector 50
[    7.427065] GSI 35 (level, low) -> CPU 0 (0x0000) vector 51
[    7.542336] pnp: PnP ACPI: found 2 devices
[    7.599353] NET: Registered protocol family 2
[    7.659252] TCP established hash table entries: 8192 (order: 2, 65536 bytes)
[    7.752389] TCP bind hash table entries: 8192 (order: 3, 131072 bytes)
[    7.838429] TCP: Hash tables configured (established 8192 bind 8192)
[    7.922282] UDP hash table entries: 512 (order: 0, 16384 bytes)
[    8.000304] UDP-Lite hash table entries: 512 (order: 0, 16384 bytes)
[    8.084056] NET: Registered protocol family 1
[    8.141446] GSI 16 (level, low) -> CPU 0 (0x0000) vector 52
[    8.216452] GSI 16 (level, low) -> CPU 0 (0x0000) vector 52 unregistered
[    8.306355] GSI 17 (level, low) -> CPU 0 (0x0000) vector 52
[    8.381350] GSI 17 (level, low) -> CPU 0 (0x0000) vector 52 unregistered
[    8.471253] GSI 18 (level, low) -> CPU 0 (0x0000) vector 52
[    8.546247] GSI 18 (level, low) -> CPU 0 (0x0000) vector 52 unregistered
[    8.636151] Unpacking initramfs...
[    9.390337] Freeing initrd memory: 16704kB freed
[    9.452092] perfmon: version 2.0 IRQ 238
[    9.504173] perfmon: Itanium 2 PMU detected, 16 PMCs, 18 PMDs, 4 counters (47 bits)
[    9.605023] perfmon: added sampling format default_format
[    9.676576] perfmon_default_smpl: default_format v2.0 registered
[    9.755645] futex hash table entries: 256 (order: 1, 32768 bytes)
[    9.836164] audit: initializing netlink subsys (disabled)
[    9.907310] audit: type=2000 audit(1463729949.900:1): initialized
[    9.987625] workingset: timestamp_bits=52 max_order=16 bucket_order=0
[   10.072633] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[   10.174301] io scheduler noop registered
[   10.225953] io scheduler deadline registered
[   10.282101] io scheduler cfq registered (default)
[   10.344189] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[   10.441907] ACPI: Power Button [PWRF]
[   10.490124] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1
[   10.587641] ACPI: Sleep Button [SLPF]
[   10.635751] thermal LNXTHERM:00: registered as thermal_zone0
[   10.710659] ACPI: Thermal Zone [THM0] (27 C)
[   10.766799] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[   10.869739] 00:00: ttyS0 at MMIO 0xff5e0000 (irq = 50, base_baud = 115200) is a 16550A
[   10.974038] console [ttyS0] enabled
[   10.974038] console [ttyS0] enabled
[   11.061742] bootconsole [uart8250] disabled
[   11.061742] bootconsole [uart8250] disabled
[   11.187308] 00:01: ttyS1 at MMIO 0xff5e2000 (irq = 51, base_baud = 115200) is a 16550A
[   11.282265] Linux agpgart interface v0.103
[   11.350337] loop: module loaded
[   11.388606] Uniform Multi-Platform E-IDE driver
[   11.442986] ide-cd driver 5.00
[   11.479646] GSI 21 (level, low) -> CPU 0 (0x0000) vector 52
[   11.548304] pata_cmd64x 0000:00:02.0: Secondary port is disabled
[   11.620496] scsi host0: pata_cmd64x
[   11.662883] scsi host1: pata_cmd64x
[   11.704862] ata1: PATA max UDMA/100 cmd 0xd18 ctl 0xd24 bmdma 0xd00 irq 55
[   11.787375] ata2: DUMMY
[   11.816629] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[   11.901233] e1000: Copyright (c) 1999-2006 Intel Corporation.
[   11.970186] GSI 29 (level, low) -> CPU 0 (0x0000) vector 53
[   12.138336] ata1.00: ATAPI: DV-28E-N, E.6A, max UDMA/33
[   12.278336] ata1.00: configured for UDMA/33
[   12.386336] scsi 0:0:0:0: CD-ROM            TEAC     DV-28E-N         E.6A PQ: 0 ANSI: 5
[   12.542338] sr 0:0:0:0: [sr0] scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
[   12.633847] cdrom: Uniform CD-ROM driver Revision: 3.20
[   12.706336] e1000 0000:20:02.0 eth0: (PCI-X:66MHz:64-bit) 00:17:08:7c:18:20
[   12.789892] e1000 0000:20:02.0 eth0: Intel(R) PRO/1000 Network Connection
[   12.871351] sr 0:0:0:0: Attached scsi generic sg0 type 5
[   12.935306] GSI 30 (level, low) -> CPU 0 (0x0000) vector 54
[   13.314340] e1000 0000:20:02.1 eth1: (PCI-X:66MHz:64-bit) 00:17:08:7c:18:21
[   13.399691] e1000 0000:20:02.1 eth1: Intel(R) PRO/1000 Network Connection
[   13.481044] Fusion MPT base driver 3.04.20
[   13.530204] Copyright (c) 1999-2008 LSI Corporation
[   13.588646] Fusion MPT SPI Host driver 3.04.20
[   13.641872] GSI 27 (level, low) -> CPU 0 (0x0000) vector 55
[   13.710213] mptbase: ioc0: Initiating bringup
[   14.242338] ioc0: LSI53C1030 C0: Capabilities={Initiator,Target}
[   14.986338] scsi host2: ioc0: LSI53C1030 C0, FwRev=01032341h, Ports=1, MaxQ=255, IRQ=58
[   15.718338] GSI 28 (level, low) -> CPU 0 (0x0000) vector 56
[   15.787849] mptbase: ioc1: Initiating bringup
[   16.318338] ioc1: LSI53C1030 C0: Capabilities={Initiator,Target}
[   17.062338] scsi host3: ioc1: LSI53C1030 C0, FwRev=01032341h, Ports=1, MaxQ=255, IRQ=59
[   17.160901] scsi 2:0:0:0: Direct-Access     HP 73.4G ST373454LC       HPC2 PQ: 0 ANSI: 3
[   17.259548] scsi target2:0:0: Beginning Domain Validation
[   17.942338] Fusion MPT FC Host driver 3.04.20
[   17.995796] Fusion MPT SAS Host driver 3.04.20
[   18.049128] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   18.127474] ehci-pci: EHCI PCI platform driver
[   18.180798] GSI 18 (level, low) -> CPU 0 (0x0000) vector 57
[   18.249240] ehci-pci 0000:00:01.2: EHCI Host Controller
[   18.312052] ehci-pci 0000:00:01.2: new USB bus registered, assigned bus number 1
[   18.400810] ehci-pci 0000:00:01.2: irq 54, io mem 0x80000000
[   18.474340] scsi target2:0:0: Ending Domain Validation
[   18.537887] ehci-pci 0000:00:01.2: USB 2.0 started, EHCI 0.95
[   18.606851] scsi target2:0:0: FAST-160 WIDE SCSI 320.0 MB/s DT IU QAS RTI WRFLOW PCOMP (6.25 ns, offset 63)
[   18.723742] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[   18.805192] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   18.891860] usb usb1: Product: EHCI Host Controller
[   18.950394] usb usb1: Manufacturer: Linux 4.6.0-rc7.jcm_tomasz_pciv7_test+ ehci_hcd
[   19.042175] usb usb1: SerialNumber: 0000:00:01.2
[   19.097584] hub 1-0:1.0: USB hub found
[   19.142790] hub 1-0:1.0: 5 ports detected
[   19.190915] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[   19.265300] mousedev: PS/2 mouse device common for all mice
[   19.334336] rtc-efi rtc-efi: rtc core: registered rtc-efi as rtc0
[   19.407799] EFI Variables Facility v0.08 2004-May-17
[   19.467590] NET: Registered protocol family 10
[   19.521650] NET: Registered protocol family 17
[   19.575405] registered taskstats version 1
[   19.624776] rtc-efi rtc-efi: setting system clock to 2016-05-20 07:39:20 UTC (1463729960)
[   22.826338] sd 2:0:0:0: Attached scsi generic sg1 type 0
[   22.892021] sd 2:0:0:0: [sda] 143374738 512-byte logical blocks: (73.4 GB/68.4 GiB)
[   22.984321] sd 2:0:0:0: [sda] Write Protect is off
[   23.043271] sd 2:0:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   23.174340]  sda: sda1 sda2 sda3
[   23.218340] sd 2:0:0:0: [sda] Attached SCSI disk
[   23.273689] Freeing unused kernel memory: 880K (e000000004e64000 - e000000004f40000)
[   23.366620] This architecture does not have kernel memory protection.
Loading, please wait...
[   23.462341] udev[112]: starting version 164
FATAL: Could not[   23.806340] EXT4-fs (sda2): mounting ext3 file system using the ext4 subsystem
 load /lib/modules/4.6.0-rc7.jcm[   23.914337] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
_tomasz_pciv7_test+/modules.dep: No such file or directory
Begin: Loading essential drivers ... FATAL: Could not load /lib/modules/4.6.0-rc7.jcm_tomasz_pciv7_test+/modules.dep: No such file or directory
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
FATAL: Could not load /lib/modules/4.6.0-rc7.jcm_tomasz_pciv7_test+/modules.dep: No such file or directory
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
modprobe: FATAL: Could not load /lib/modules/4.6.0-rc7.jcm_tomasz_pciv7_test+/modules.dep: No such file or directory

\rINIT: version 2.88 booting
Using makefile-s[   24.982338] random: nonblocking pool is initialized
tyle concurrent boot in runlevel S.
Starting the hotplug events dispatcher: udevd[   25.330336] udev[267]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...done.
Activating swap...[   25.998339] Adding 2000784k swap on /dev/sda3.  Priority:-1 extents:1 across:2000784k 
done.
[   26.098340] EXT4-fs (sda2): re-mounted. Opts: (null)
Checking root file system...fsck from util-linux-ng 2.17.2
/dev/sda2: clean[   26.262343] EXT4-fs (sda2): re-mounted. Opts: errors=remount-ro
, 93785/4349952 files, 1074005/17397216 blocks
done.
Loading kernel modules...done.
Cleaning up ifupdown....
Setting up networking....
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Configuring network interfaces...done.
Starting portmap daemon....
Starting NFS common utilities: statd.
Cleaning up temporary files....
[   27.090342] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   27.162611] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[   27.249590] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Setting console screen modes and fonts.
cannot (un)set powersave mode
^[[9;30]^[[14;30]Setting kernel variables ...done.
\rINIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting portmap daemon...Already running..
Starting NFS common utilities: statd.
Starting enhanced syslogd: rsyslogd.
Starting ACPI services....
Starting deferred execution scheduler: atd.
Starting mpt-status monitor: mpt-statusd.
Starting periodic command scheduler: cron.
Starting MTA: exim4.
Starting O[   28.750341] sshd (1082): /proc/1082/oom_adj is deprecated, please use /proc/1082/oom_score_adj instead.
penBSD Secure Shell server: sshd.

Debian GNU/Linux 6.0 hamartia ttyS0

hamartia login: root
Password: 
Last login: Fri May 20 03:35:54 EDT 2016 on ttyS0
Linux hamartia 4.6.0-rc7.jcm_tomasz_pciv7_test+ #2 SMP Fri May 20 03:24:58 EDT 2016 ia64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@hamartia:~# uname -a
Linux hamartia 4.6.0-rc7.jcm_tomasz_pciv7_test+ #2 SMP Fri May 20 03:24:58 EDT 2016 ia64 GNU/Linux
root@hamartia:~# 

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Jon Masters]

Hi Ard,

On 05/20/2016 03:37 AM, Ard Biesheuvel wrote:
> On 20 May 2016 at 06:41, Jon Masters <jcm@redhat.com> wrote:
>> Hi Tomasz, all,
>>
>> On 05/11/2016 07:08 AM, Tomasz Nowicki wrote:
>>
>>> On 11.05.2016 12:41, Gabriele Paoloni wrote:
>>
>>>>> v6 -> v7
>>>>> - drop quirks handling
>>>>
>>>> Maybe I missed something in the v6 discussion thread; when was it
>>>> decided to drop quirk handling?
>>>
>>> I had such requests in previous series.
>>
>> A quick note on quirk handling. This, I believe, applies post-merge of
>> the base infrastructure, which I realize will likely not have quirks.
>>
>> We've some "gen1" ARMv8 server platforms where we end up doing quirks
>> (for things like forcing 32-bit config space accessors and the like) due
>> to people repurposing existing embedded PCIe IP blocks or using them for
>> the first time (especially in servers), and those being involved in the
>> design not necessarily seeing this problem ahead of time, or not
>> realizing that it would be an issue for servers. In the early days of
>> ARM server designs 3-4 years ago, many of us had never really played
>> with ECAM or realized how modern topologies are built.
>>
>> Anyway. We missed this one in our SBSA requirements. They say (words to
>> the effect of) "thou shalt do PCIe the way it is done on servers" but
>> they aren't prescriptive, and they don't tell people how that actually
>> is in reality. That is being fixed. A lot of things are happening behind
>> the scenes - especially with third party IP block providers (all of whom
>> myself and others are speaking with directly about this) - to ensure
>> that the next wave of designs won't repeat these mistakes. We don't have
>> a time machine, but we can contain this from becoming an ongoing mess
>> for upstream, and we will do so. It won't be a zoo.
>>
>> Various proposals have arisen for how to handle quirks in the longer
>> term, including elaborate frameworks and tables to describe them
>> generically. I would like to caution against such approaches, especially
>> in the case that they deviate from practice on x86, or prior to being
>> standardized fully with other Operating System vendors. I don't expect
>> there to be too many more than the existing initial set of quirks we
>> have seen posted. A number of "future" server SoCs have already been
>> fixed prior to silicon, and new design starts are being warned not to
>> make this a problem for us to have to clean up later.
>>
>> So, I would like to suggest that the eventual framework mirror the
>> existing approach on x86 systems (matching DMI, etc.) and not be made
>> into some kind of generic, utopia. This is a case where we want there to
>> be pain involved (and upstream patches required) when people screw up,
>> so that they have a level of pain in response to ever making this
>> mistake in the future. If we try to create too grand a generic scheme
>> and make it too easy to handle this kind of situation beyond the small
>> number of existing offenders, we undermine efforts to force vendors to
>> ensure that their IP blocks are compliant going forward.

> I understand that there is a desire from the RedHat side to mimic x86
> as closely as possible, but I never saw any technical justification
> for that.

Understood. My own motivation is always to make the experience as
familiar as possible, both for end users, as well as for ODMs and the
entire ecosystem. There are very many ODMs currently working on v8
server designs and they're already expecting this to be "just like x88".
Intentionally. But as to the specifics of using DMI...

> DMI contains strings that are visible to userland, and you
> effectively lock those down to certain values just so that the kernel
> can distinguish a broken PCIe root complex from a working one. Linux
> on x86 had no choice, since the overwhelming majority of existing
> hardware misrepresented itself as generic, and DMI was the only thing
> available to actually distinguish these broken implementations from
> one another. This does not mean we should allow and/or encourage this
> first gen hardware to misrepresent non-compliant hardware as compliant
> as well.

That's a very reasonable argument. I don't disagree that it would be
nice to have nicer ways to distinguish the non-compliant IP than
treating the whole platform with an ASCII matching sledgehammer.

> Since you are talking to all the people involved, how about you
> convince them to put something in the ACPI tables that allows the
> kernel to distinguish those non-standard PCIe implementations from
> hardware that is really generic?

I'm open to this *BUT* it has to be something that will be adopted
beyond Linux. I have reached out to some non-Linux folks about this. If
there's buy-in, and if there's agreement to go standardize it through
the ASWG, then we should do so. What we should not do is treat ARM as
special in a way that the others aren't involved with. I'll admit DMI
ended up part of the SBBR in part because I wrote that piece in with the
assumption that exactly the same matches as on x86 would happen.

> This way, we can sidestep the quirks
> debate entirely, since it will simply be a different device as far as
> the kernel is concerned. This is no worse than a quirk from a
> practical point of view, since an older OS will be equally unable to
> run on newer hardware, but it is arguably more true to the standards
> compliance you tend to preach about, especially since this small pool
> of third party IP could potentially be identified directly rather than
> based on some divination of the SoC we may or may not be running on. I
> am also convinced that adding support for an additional HID() to the
> ACPI ECAM driver with some special config space handling wired in is
> an easier sell upstream than making the same ugly mess x86 has had to
> make because they did not have any choice to begin with.

Again, open to it. I just don't want to do something that's Linux
specific. So it'll take time. It would be awesome if an interim quirk
solution existed that got platforms that are shipping in production
(e.g. HP Moonshot) actually booting upstream kernels this year. We
/really/ want F25 to be able to run on these without needing to carry an
out-of-tree quirk patch or just not support them. That's much worse.

> If we do need a quirks handling mechanism, I still don't see how the
> x86 situation extrapolates to ARM. ACPI offers plenty of ways for a
> SoC vendor to identify the make and particular revision, and quirks
> could be keyed off of that.

Fair enough. Is there any traction for an interim solution for these
initial platforms do you think? If we wait to add a new table it's
probably going to be the end of the year before we get this done.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

RE: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Gabriele Paoloni]

Hi Ard, Jon

> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> Sent: 20 May 2016 08:38
> To: Jon Masters
> Cc: Tomasz Nowicki; Gabriele Paoloni; helgaas@kernel.org; arnd@arndb.de;
> will.deacon@arm.com; catalin.marinas@arm.com; rafael@kernel.org;
> hanjun.guo@linaro.org; Lorenzo.Pieralisi@arm.com; okaya@codeaurora.org;
> jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
> pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
> ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
> kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
> msalter@redhat.com; Wangyijing; mw@semihalf.com;
> andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
> controller
> 
> On 20 May 2016 at 06:41, Jon Masters <jcm@redhat.com> wrote:
> > Hi Tomasz, all,
> >
> > On 05/11/2016 07:08 AM, Tomasz Nowicki wrote:
> >
> >> On 11.05.2016 12:41, Gabriele Paoloni wrote:
> >
> >>>> v6 -> v7
> >>>> - drop quirks handling
> >>>
> >>> Maybe I missed something in the v6 discussion thread; when was it
> >>> decided to drop quirk handling?
> >>
> >> I had such requests in previous series.
> >
> > A quick note on quirk handling. This, I believe, applies post-merge
> of
> > the base infrastructure, which I realize will likely not have quirks.
> >
> > We've some "gen1" ARMv8 server platforms where we end up doing quirks
> > (for things like forcing 32-bit config space accessors and the like)
> due
> > to people repurposing existing embedded PCIe IP blocks or using them
> for
> > the first time (especially in servers), and those being involved in
> the
> > design not necessarily seeing this problem ahead of time, or not
> > realizing that it would be an issue for servers. In the early days of
> > ARM server designs 3-4 years ago, many of us had never really played
> > with ECAM or realized how modern topologies are built.
> >
> > Anyway. We missed this one in our SBSA requirements. They say (words
> to
> > the effect of) "thou shalt do PCIe the way it is done on servers" but
> > they aren't prescriptive, and they don't tell people how that
> actually
> > is in reality. That is being fixed. A lot of things are happening
> behind
> > the scenes - especially with third party IP block providers (all of
> whom
> > myself and others are speaking with directly about this) - to ensure
> > that the next wave of designs won't repeat these mistakes. We don't
> have
> > a time machine, but we can contain this from becoming an ongoing mess
> > for upstream, and we will do so. It won't be a zoo.
> >
> > Various proposals have arisen for how to handle quirks in the longer
> > term, including elaborate frameworks and tables to describe them
> > generically. I would like to caution against such approaches,
> especially
> > in the case that they deviate from practice on x86, or prior to being
> > standardized fully with other Operating System vendors. I don't
> expect
> > there to be too many more than the existing initial set of quirks we
> > have seen posted. A number of "future" server SoCs have already been
> > fixed prior to silicon, and new design starts are being warned not to
> > make this a problem for us to have to clean up later.
> >
> > So, I would like to suggest that the eventual framework mirror the
> > existing approach on x86 systems (matching DMI, etc.) and not be made
> > into some kind of generic, utopia. This is a case where we want there
> to
> > be pain involved (and upstream patches required) when people screw up,
> > so that they have a level of pain in response to ever making this
> > mistake in the future. If we try to create too grand a generic scheme
> > and make it too easy to handle this kind of situation beyond the
> small
> > number of existing offenders, we undermine efforts to force vendors
> to
> > ensure that their IP blocks are compliant going forward.
> >
> 
> I understand that there is a desire from the RedHat side to mimic x86
> as closely as possible, but I never saw any technical justification
> for that. DMI contains strings that are visible to userland, and you
> effectively lock those down to certain values just so that the kernel
> can distinguish a broken PCIe root complex from a working one. Linux
> on x86 had no choice, since the overwhelming majority of existing
> hardware misrepresented itself as generic, and DMI was the only thing
> available to actually distinguish these broken implementations from
> one another. This does not mean we should allow and/or encourage this
> first gen hardware to misrepresent non-compliant hardware as compliant
> as well.
> 
> Since you are talking to all the people involved, how about you
> convince them to put something in the ACPI tables that allows the
> kernel to distinguish those non-standard PCIe implementations from
> hardware that is really generic? This way, we can sidestep the quirks
> debate entirely, since it will simply be a different device as far as
> the kernel is concerned. This is no worse than a quirk from a
> practical point of view, since an older OS will be equally unable to
> run on newer hardware, but it is arguably more true to the standards
> compliance you tend to preach about, especially since this small pool
> of third party IP could potentially be identified directly rather than
> based on some divination of the SoC we may or may not be running on. I
> am also convinced that adding support for an additional HID() to the
> ACPI ECAM driver with some special config space handling wired in is
> an easier sell upstream than making the same ugly mess x86 has had to
> make because they did not have any choice to begin with.

In our case (HiSilicon Hip05/Hip06) we are using the Designware IP
that unfortunately is non-ECAM for the RC config space.

A possible ACPI table solution was discussed already in this thread

https://lkml.org/lkml/2016/3/14/722

where <<Name (_CID, "PNP0C02")  // Motherboard reserved resource>>
is used to specify an Host Controller specific resource.
It looks to me that this can be an approach that can accommodate
different vendors/scenarios and Bjorn seemed to be quite ok with
it.

It comes without saying that for future HW releases we all should
make an effort to deliver fully ECAM compliant controllers.

What's your view about this approach?

Thanks

Gab

> 
> If we do need a quirks handling mechanism, I still don't see how the
> x86 situation extrapolates to ARM. ACPI offers plenty of ways for a
> SoC vendor to identify the make and particular revision, and quirks
> could be keyed off of that.
> 
> --
> Ard.

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Jon Masters]

On 05/20/2016 04:11 AM, Gabriele Paoloni wrote:
> Hi Ard, Jon

Hi Gabriele :)

> In our case (HiSilicon Hip05/Hip06) we are using the Designware IP
> that unfortunately is non-ECAM for the RC config space.

Yea, I know, and I've pinged them already.

> A possible ACPI table solution was discussed already in this thread
> 
> https://lkml.org/lkml/2016/3/14/722
> 
> where <<Name (_CID, "PNP0C02")  // Motherboard reserved resource>>
> is used to specify an Host Controller specific resource.
> It looks to me that this can be an approach that can accommodate
> different vendors/scenarios and Bjorn seemed to be quite ok with
> it.

Yeah, pondering that. We'll chat with a few others about it.

> It comes without saying that for future HW releases we all should
> make an effort to deliver fully ECAM compliant controllers.

Right. Like I said, a number of designs have been fixed already.

> What's your view about this approach?

Will followup over the weekend.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Ard Biesheuvel]

On 20 May 2016 at 10:01, Jon Masters <jcm@redhat.com> wrote:
> Hi Ard,
>
> On 05/20/2016 03:37 AM, Ard Biesheuvel wrote:
>> On 20 May 2016 at 06:41, Jon Masters <jcm@redhat.com> wrote:
>>> Hi Tomasz, all,
>>>
>>> On 05/11/2016 07:08 AM, Tomasz Nowicki wrote:
>>>
>>>> On 11.05.2016 12:41, Gabriele Paoloni wrote:
>>>
>>>>>> v6 -> v7
>>>>>> - drop quirks handling
>>>>>
>>>>> Maybe I missed something in the v6 discussion thread; when was it
>>>>> decided to drop quirk handling?
>>>>
>>>> I had such requests in previous series.
>>>
>>> A quick note on quirk handling. This, I believe, applies post-merge of
>>> the base infrastructure, which I realize will likely not have quirks.
>>>
>>> We've some "gen1" ARMv8 server platforms where we end up doing quirks
>>> (for things like forcing 32-bit config space accessors and the like) due
>>> to people repurposing existing embedded PCIe IP blocks or using them for
>>> the first time (especially in servers), and those being involved in the
>>> design not necessarily seeing this problem ahead of time, or not
>>> realizing that it would be an issue for servers. In the early days of
>>> ARM server designs 3-4 years ago, many of us had never really played
>>> with ECAM or realized how modern topologies are built.
>>>
>>> Anyway. We missed this one in our SBSA requirements. They say (words to
>>> the effect of) "thou shalt do PCIe the way it is done on servers" but
>>> they aren't prescriptive, and they don't tell people how that actually
>>> is in reality. That is being fixed. A lot of things are happening behind
>>> the scenes - especially with third party IP block providers (all of whom
>>> myself and others are speaking with directly about this) - to ensure
>>> that the next wave of designs won't repeat these mistakes. We don't have
>>> a time machine, but we can contain this from becoming an ongoing mess
>>> for upstream, and we will do so. It won't be a zoo.
>>>
>>> Various proposals have arisen for how to handle quirks in the longer
>>> term, including elaborate frameworks and tables to describe them
>>> generically. I would like to caution against such approaches, especially
>>> in the case that they deviate from practice on x86, or prior to being
>>> standardized fully with other Operating System vendors. I don't expect
>>> there to be too many more than the existing initial set of quirks we
>>> have seen posted. A number of "future" server SoCs have already been
>>> fixed prior to silicon, and new design starts are being warned not to
>>> make this a problem for us to have to clean up later.
>>>
>>> So, I would like to suggest that the eventual framework mirror the
>>> existing approach on x86 systems (matching DMI, etc.) and not be made
>>> into some kind of generic, utopia. This is a case where we want there to
>>> be pain involved (and upstream patches required) when people screw up,
>>> so that they have a level of pain in response to ever making this
>>> mistake in the future. If we try to create too grand a generic scheme
>>> and make it too easy to handle this kind of situation beyond the small
>>> number of existing offenders, we undermine efforts to force vendors to
>>> ensure that their IP blocks are compliant going forward.
>
>> I understand that there is a desire from the RedHat side to mimic x86
>> as closely as possible, but I never saw any technical justification
>> for that.
>
> Understood. My own motivation is always to make the experience as
> familiar as possible, both for end users, as well as for ODMs and the
> entire ecosystem. There are very many ODMs currently working on v8
> server designs and they're already expecting this to be "just like x88".
> Intentionally. But as to the specifics of using DMI...
>
>> DMI contains strings that are visible to userland, and you
>> effectively lock those down to certain values just so that the kernel
>> can distinguish a broken PCIe root complex from a working one. Linux
>> on x86 had no choice, since the overwhelming majority of existing
>> hardware misrepresented itself as generic, and DMI was the only thing
>> available to actually distinguish these broken implementations from
>> one another. This does not mean we should allow and/or encourage this
>> first gen hardware to misrepresent non-compliant hardware as compliant
>> as well.
>
> That's a very reasonable argument. I don't disagree that it would be
> nice to have nicer ways to distinguish the non-compliant IP than
> treating the whole platform with an ASCII matching sledgehammer.
>
>> Since you are talking to all the people involved, how about you
>> convince them to put something in the ACPI tables that allows the
>> kernel to distinguish those non-standard PCIe implementations from
>> hardware that is really generic?
>
> I'm open to this *BUT* it has to be something that will be adopted
> beyond Linux. I have reached out to some non-Linux folks about this. If
> there's buy-in, and if there's agreement to go standardize it through
> the ASWG, then we should do so. What we should not do is treat ARM as
> special in a way that the others aren't involved with. I'll admit DMI
> ended up part of the SBBR in part because I wrote that piece in with the
> assumption that exactly the same matches as on x86 would happen.
>

Is the PCIe root complex so special that you cannot simply describe an
implementation that is not PNP0408 compatible as something else, under
its own unique HID? If everybody is onboard with using ACPI, how is
this any different from describing other parts of the platform
topology? Even if the SBSA mandates generic PCI, they already deviated
from that when they built the hardware, so pretending that it is a
PNP0408 with quirks really does not buy us anything.

>> This way, we can sidestep the quirks
>> debate entirely, since it will simply be a different device as far as
>> the kernel is concerned. This is no worse than a quirk from a
>> practical point of view, since an older OS will be equally unable to
>> run on newer hardware, but it is arguably more true to the standards
>> compliance you tend to preach about, especially since this small pool
>> of third party IP could potentially be identified directly rather than
>> based on some divination of the SoC we may or may not be running on. I
>> am also convinced that adding support for an additional HID() to the
>> ACPI ECAM driver with some special config space handling wired in is
>> an easier sell upstream than making the same ugly mess x86 has had to
>> make because they did not have any choice to begin with.
>
> Again, open to it. I just don't want to do something that's Linux
> specific. So it'll take time. It would be awesome if an interim quirk
> solution existed that got platforms that are shipping in production
> (e.g. HP Moonshot) actually booting upstream kernels this year. We
> /really/ want F25 to be able to run on these without needing to carry an
> out-of-tree quirk patch or just not support them. That's much worse.
>

My whole point is that we don't need quirks in the first place if
non-compliant devices are not being misrepresented as compliant ones.
This is fine for other platform devices, i.e., SATA, network, so
again, why is PCIe so special that we *must* use a generic ID + quirks
rather than a specific ID?

>> If we do need a quirks handling mechanism, I still don't see how the
>> x86 situation extrapolates to ARM. ACPI offers plenty of ways for a
>> SoC vendor to identify the make and particular revision, and quirks
>> could be keyed off of that.
>
> Fair enough. Is there any traction for an interim solution for these
> initial platforms do you think? If we wait to add a new table it's
> probably going to be the end of the year before we get this done.
>

The 'interim solution' is to come to terms with the fact that these
initial platforms are not SBSA compliant, contain a PCIe root complex
that is not PNP0408 but can be identified by its own HID, and we make
the software work with that. This means our message from the beginning
is that, yes, you can have non-compliant hardware and the burden is on
you to get it supported upstream, and no, you don't get to hang out
with the cool SBSA kids if you decide to go that route

RE: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Gabriele Paoloni]

Hi Ard

> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> Sent: 20 May 2016 09:29
> To: Jon Masters
> Cc: Tomasz Nowicki; Gabriele Paoloni; helgaas@kernel.org;
> arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
> rafael@kernel.org; hanjun.guo@linaro.org; Lorenzo.Pieralisi@arm.com;
> okaya@codeaurora.org; jchandra@broadcom.com; linaro-
> acpi@lists.linaro.org; linux-pci@vger.kernel.org; dhdang@apm.com;
> Liviu.Dudau@arm.com; ddaney@caviumnetworks.com; jeremy.linton@arm.com;
> linux-kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
> msalter@redhat.com; Wangyijing; mw@semihalf.com;
> andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
> controller
> 
> On 20 May 2016 at 10:01, Jon Masters <jcm@redhat.com> wrote:
> > Hi Ard,
> >
> > On 05/20/2016 03:37 AM, Ard Biesheuvel wrote:
> >> On 20 May 2016 at 06:41, Jon Masters <jcm@redhat.com> wrote:
> >>> Hi Tomasz, all,
> >>>
> >>> On 05/11/2016 07:08 AM, Tomasz Nowicki wrote:
> >>>
> >>>> On 11.05.2016 12:41, Gabriele Paoloni wrote:
> >>>
> >>>>>> v6 -> v7
> >>>>>> - drop quirks handling
> >>>>>
> >>>>> Maybe I missed something in the v6 discussion thread; when was it
> >>>>> decided to drop quirk handling?
> >>>>
> >>>> I had such requests in previous series.
> >>>
> >>> A quick note on quirk handling. This, I believe, applies post-merge
> of
> >>> the base infrastructure, which I realize will likely not have
> quirks.
> >>>
> >>> We've some "gen1" ARMv8 server platforms where we end up doing
> quirks
> >>> (for things like forcing 32-bit config space accessors and the
> like) due
> >>> to people repurposing existing embedded PCIe IP blocks or using
> them for
> >>> the first time (especially in servers), and those being involved in
> the
> >>> design not necessarily seeing this problem ahead of time, or not
> >>> realizing that it would be an issue for servers. In the early days
> of
> >>> ARM server designs 3-4 years ago, many of us had never really
> played
> >>> with ECAM or realized how modern topologies are built.
> >>>
> >>> Anyway. We missed this one in our SBSA requirements. They say
> (words to
> >>> the effect of) "thou shalt do PCIe the way it is done on servers"
> but
> >>> they aren't prescriptive, and they don't tell people how that
> actually
> >>> is in reality. That is being fixed. A lot of things are happening
> behind
> >>> the scenes - especially with third party IP block providers (all of
> whom
> >>> myself and others are speaking with directly about this) - to
> ensure
> >>> that the next wave of designs won't repeat these mistakes. We don't
> have
> >>> a time machine, but we can contain this from becoming an ongoing
> mess
> >>> for upstream, and we will do so. It won't be a zoo.
> >>>
> >>> Various proposals have arisen for how to handle quirks in the
> longer
> >>> term, including elaborate frameworks and tables to describe them
> >>> generically. I would like to caution against such approaches,
> especially
> >>> in the case that they deviate from practice on x86, or prior to
> being
> >>> standardized fully with other Operating System vendors. I don't
> expect
> >>> there to be too many more than the existing initial set of quirks
> we
> >>> have seen posted. A number of "future" server SoCs have already
> been
> >>> fixed prior to silicon, and new design starts are being warned not
> to
> >>> make this a problem for us to have to clean up later.
> >>>
> >>> So, I would like to suggest that the eventual framework mirror the
> >>> existing approach on x86 systems (matching DMI, etc.) and not be
> made
> >>> into some kind of generic, utopia. This is a case where we want
> there to
> >>> be pain involved (and upstream patches required) when people screw
> up,
> >>> so that they have a level of pain in response to ever making this
> >>> mistake in the future. If we try to create too grand a generic
> scheme
> >>> and make it too easy to handle this kind of situation beyond the
> small
> >>> number of existing offenders, we undermine efforts to force vendors
> to
> >>> ensure that their IP blocks are compliant going forward.
> >
> >> I understand that there is a desire from the RedHat side to mimic
> x86
> >> as closely as possible, but I never saw any technical justification
> >> for that.
> >
> > Understood. My own motivation is always to make the experience as
> > familiar as possible, both for end users, as well as for ODMs and the
> > entire ecosystem. There are very many ODMs currently working on v8
> > server designs and they're already expecting this to be "just like
> x88".
> > Intentionally. But as to the specifics of using DMI...
> >
> >> DMI contains strings that are visible to userland, and you
> >> effectively lock those down to certain values just so that the
> kernel
> >> can distinguish a broken PCIe root complex from a working one. Linux
> >> on x86 had no choice, since the overwhelming majority of existing
> >> hardware misrepresented itself as generic, and DMI was the only
> thing
> >> available to actually distinguish these broken implementations from
> >> one another. This does not mean we should allow and/or encourage
> this
> >> first gen hardware to misrepresent non-compliant hardware as
> compliant
> >> as well.
> >
> > That's a very reasonable argument. I don't disagree that it would be
> > nice to have nicer ways to distinguish the non-compliant IP than
> > treating the whole platform with an ASCII matching sledgehammer.
> >
> >> Since you are talking to all the people involved, how about you
> >> convince them to put something in the ACPI tables that allows the
> >> kernel to distinguish those non-standard PCIe implementations from
> >> hardware that is really generic?
> >
> > I'm open to this *BUT* it has to be something that will be adopted
> > beyond Linux. I have reached out to some non-Linux folks about this.
> If
> > there's buy-in, and if there's agreement to go standardize it through
> > the ASWG, then we should do so. What we should not do is treat ARM as
> > special in a way that the others aren't involved with. I'll admit DMI
> > ended up part of the SBBR in part because I wrote that piece in with
> the
> > assumption that exactly the same matches as on x86 would happen.
> >
> 
> Is the PCIe root complex so special that you cannot simply describe an
> implementation that is not PNP0408 compatible as something else, under
> its own unique HID? If everybody is onboard with using ACPI, how is
> this any different from describing other parts of the platform
> topology? Even if the SBSA mandates generic PCI, they already deviated
> from that when they built the hardware, so pretending that it is a
> PNP0408 with quirks really does not buy us anything.

>From my understanding we want to avoid this as this would allow each
vendor to come up with his own code and it would be much more effort
for the PCI maintainer to rework the PCI framework to accommodate X86
and "all" ARM64 Host Controllers...

I guess this approach is too risky and we want to avoid this. Through
standardization we can more easily maintain the code and scale it to
multiple SoCs...

So this is my understanding; maybe Jon, Tomasz or Lorenzo can give
a bit more explanation...

Thanks

Gab


> 
> >> This way, we can sidestep the quirks
> >> debate entirely, since it will simply be a different device as far
> as
> >> the kernel is concerned. This is no worse than a quirk from a
> >> practical point of view, since an older OS will be equally unable to
> >> run on newer hardware, but it is arguably more true to the standards
> >> compliance you tend to preach about, especially since this small
> pool
> >> of third party IP could potentially be identified directly rather
> than
> >> based on some divination of the SoC we may or may not be running on.
> I
> >> am also convinced that adding support for an additional HID() to the
> >> ACPI ECAM driver with some special config space handling wired in is
> >> an easier sell upstream than making the same ugly mess x86 has had
> to
> >> make because they did not have any choice to begin with.
> >
> > Again, open to it. I just don't want to do something that's Linux
> > specific. So it'll take time. It would be awesome if an interim quirk
> > solution existed that got platforms that are shipping in production
> > (e.g. HP Moonshot) actually booting upstream kernels this year. We
> > /really/ want F25 to be able to run on these without needing to carry
> an
> > out-of-tree quirk patch or just not support them. That's much worse.
> >
> 
> My whole point is that we don't need quirks in the first place if
> non-compliant devices are not being misrepresented as compliant ones.
> This is fine for other platform devices, i.e., SATA, network, so
> again, why is PCIe so special that we *must* use a generic ID + quirks
> rather than a specific ID?
> 
> >> If we do need a quirks handling mechanism, I still don't see how the
> >> x86 situation extrapolates to ARM. ACPI offers plenty of ways for a
> >> SoC vendor to identify the make and particular revision, and quirks
> >> could be keyed off of that.
> >
> > Fair enough. Is there any traction for an interim solution for these
> > initial platforms do you think? If we wait to add a new table it's
> > probably going to be the end of the year before we get this done.
> >
> 
> The 'interim solution' is to come to terms with the fact that these
> initial platforms are not SBSA compliant, contain a PCIe root complex
> that is not PNP0408 but can be identified by its own HID, and we make
> the software work with that. This means our message from the beginning
> is that, yes, you can have non-compliant hardware and the burden is on
> you to get it supported upstream, and no, you don't get to hang out
> with the cool SBSA kids if you decide to go that route

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Ard Biesheuvel]

On 20 May 2016 at 10:40, Gabriele Paoloni <gabriele.paoloni@huawei.com> wrote:
> Hi Ard
>
>> -----Original Message-----
>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
[...]
>>
>> Is the PCIe root complex so special that you cannot simply describe an
>> implementation that is not PNP0408 compatible as something else, under
>> its own unique HID? If everybody is onboard with using ACPI, how is
>> this any different from describing other parts of the platform
>> topology? Even if the SBSA mandates generic PCI, they already deviated
>> from that when they built the hardware, so pretending that it is a
>> PNP0408 with quirks really does not buy us anything.
>
> From my understanding we want to avoid this as this would allow each
> vendor to come up with his own code and it would be much more effort
> for the PCI maintainer to rework the PCI framework to accommodate X86
> and "all" ARM64 Host Controllers...
>
> I guess this approach is too risky and we want to avoid this. Through
> standardization we can more easily maintain the code and scale it to
> multiple SoCs...
>
> So this is my understanding; maybe Jon, Tomasz or Lorenzo can give
> a bit more explanation...
>

OK, so that boils down to recommending to vendors to represent known
non-compliant hardware as compliant, just so that we don't have to
change the code to support additional flavors of ECAM ? It's fine to
be pragmatic, but that sucks.

We keep confusing the x86 case with the ARM case here: for x86, they
needed to deal with broken hardware *after* the fact, and all they
could do is find /some/ distinguishing feature in order to guess which
exact hardware they might be running on. For arm64, it is the opposite
case. We are currently in a position where we can demand vendors to
comply with the standards they endorsed themselves, and (ab)using ACPI
+ DMI as a de facto platform description rather than plain ACPI makes
me think the DT crowd were actually right from the beginning. It
*directly* violates the standardization principle, since it requires a
priori knowledge inside the OS that a certain 'generic' device must be
driven in a special way.

So can anyone comment on the feasibility of adding support for devices
with vendor specific HIDs (and no generic CIDs) to the current ACPI
ECAM driver in Linux?

Re: [PATCH V7 04/11] pci: Add new function to unmap IO resources.

[Jayachandran C]

On Tue, May 10, 2016 at 8:49 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
> It is very useful to release I/O resources so that the same I/O resources
> can be allocated again (pci_remap_iospace), like in PCI hotplug removal
> scenario. Therefore this patch implements new pci_unmap_iospace call which
> unmaps I/O space as the symmetry to pci_remap_iospace.
>
> Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
> ---
>  drivers/pci/pci.c   | 24 ++++++++++++++++++++++++
>  include/linux/pci.h |  1 +
>  2 files changed, 25 insertions(+)
>
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index bc0c914..ff97a0b 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -25,6 +25,7 @@
>  #include <linux/device.h>
>  #include <linux/pm_runtime.h>
>  #include <linux/pci_hotplug.h>
> +#include <linux/vmalloc.h>
>  #include <asm/setup.h>
>  #include <linux/aer.h>
>  #include "pci.h"
> @@ -3167,6 +3168,29 @@ int __weak pci_remap_iospace(const struct resource *res, phys_addr_t phys_addr)
>  #endif
>  }
>
> +/**
> + *     pci_unmap_iospace - Unmap the memory mapped I/O space
> + *     @res: resource to be unmapped
> + *
> + *     Unmap the CPU virtual address @res from virtual address space.
> + *     Only architectures that have memory mapped IO functions defined
> + *     (and the PCI_IOBASE value defined) should call this function.
> + */
> +void  pci_unmap_iospace(struct resource *res)
> +{
> +#if defined(PCI_IOBASE) && defined(CONFIG_MMU)
> +       unsigned long vaddr = (unsigned long)PCI_IOBASE + res->start;
> +
> +       unmap_kernel_range(vaddr, resource_size(res));
> +#else
> +       /*
> +        * This architecture does not have memory mapped I/O space,
> +        * so this function should never be called.
> +        */
> +       WARN_ONCE(1, "This architecture does not support memory mapped I/O\n");
> +#endif
> +}

WARN is not needed here, since we would have already done it in
pci_remap_iospace.

Ideally, we should undo the pci_register_io_range as well, but
re-registering the same range seems to be fine.

JC.

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Lorenzo Pieralisi]

On Fri, May 20, 2016 at 11:14:03AM +0200, Ard Biesheuvel wrote:
> On 20 May 2016 at 10:40, Gabriele Paoloni <gabriele.paoloni@huawei.com> wrote:
> > Hi Ard
> >
> >> -----Original Message-----
> >> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> [...]
> >>
> >> Is the PCIe root complex so special that you cannot simply describe an
> >> implementation that is not PNP0408 compatible as something else, under
> >> its own unique HID? If everybody is onboard with using ACPI, how is
> >> this any different from describing other parts of the platform
> >> topology? Even if the SBSA mandates generic PCI, they already deviated
> >> from that when they built the hardware, so pretending that it is a
> >> PNP0408 with quirks really does not buy us anything.
> >
> > From my understanding we want to avoid this as this would allow each
> > vendor to come up with his own code and it would be much more effort
> > for the PCI maintainer to rework the PCI framework to accommodate X86
> > and "all" ARM64 Host Controllers...
> >
> > I guess this approach is too risky and we want to avoid this. Through
> > standardization we can more easily maintain the code and scale it to
> > multiple SoCs...
> >
> > So this is my understanding; maybe Jon, Tomasz or Lorenzo can give
> > a bit more explanation...
> >
> 
> OK, so that boils down to recommending to vendors to represent known
> non-compliant hardware as compliant, just so that we don't have to
> change the code to support additional flavors of ECAM ? It's fine to
> be pragmatic, but that sucks.
> 
> We keep confusing the x86 case with the ARM case here: for x86, they
> needed to deal with broken hardware *after* the fact, and all they
> could do is find /some/ distinguishing feature in order to guess which
> exact hardware they might be running on. For arm64, it is the opposite
> case. We are currently in a position where we can demand vendors to
> comply with the standards they endorsed themselves, and (ab)using ACPI
> + DMI as a de facto platform description rather than plain ACPI makes
> me think the DT crowd were actually right from the beginning. It
> *directly* violates the standardization principle, since it requires a
> priori knowledge inside the OS that a certain 'generic' device must be
> driven in a special way.
> 
> So can anyone comment on the feasibility of adding support for devices
> with vendor specific HIDs (and no generic CIDs) to the current ACPI
> ECAM driver in Linux?

Host bridges in ACPI are handled through PNP0A08/PNP0A03 ids, and
most of the arch specific code is handled in the respective arch
directories (X86 and IA64, even though IA64 does not rely on ECAM/MCFG for
PCI ops), it is not a driver per-se, PNP0A08/PNP0A03 are detected through
ACPI scan handlers and the respective arch code (ie pci_acpi_scan_root)
sets-up resources AND config space on an arch specific basis.

X86 deals with that with code in arch/x86 that sets-up the pci_raw_ops
on a platform specific basis (and it is not nice, but it works because
as you all know the number of platforms in X86 world is contained).

Will this happen for ARM64 in arch/arm64 based on vendor specific
HIDs ?

No.

So given the current state of play (we were requested to move the
arch/arm64 specific ACPI PCI bits to arch/arm64), we would end up
with arch/arm64 code requiring code in /drivers to set-up pci_ops
in a platform specific way, it is horrible, if feasible at all.

The only way this can be implemented is by pretending that the
ACPI/PCI arch/arm64 implementation is generic code (that's what this
series does), move it to /drivers (where it is in this series), and
implement _DSD vendor specific bindings (per HID) to set-up the pci
operations; whether this solution should go upstream, given that it
is just a short-term solution for early platforms bugs, it is another
story and my personal answer is no.

Lorenzo

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Dongdong Liu]

在 2016/5/10 23:19, Tomasz Nowicki 写道:
>>From the functionality point of view this series may be split into the
> following logic parts:
> 1. New ECAM API and update for users of the pci-host-common API
> 2. Necessary fixes as the preparation for using driver on ARM64.
> 3. Use new MCFG interface and implement generic ACPI based PCI host controller driver.
> 4. Enable above driver on ARM64
>
> Patches has been built on top of 4.6-rc7 and can be found here:
> git@github.com:semihalf-nowicki-tomasz/linux.git (pci-acpi-v7)
>
> This has been tested on Cavium ThunderX server. Any help in reviewing and
> testing is very appreciated.
>

Based on the patchset (with ECAM fix-up quirk added) and added the patch(Add ACPI support for HiSilicon PCIe Host Controllers).
Tested on the HiSilicon ARM64 D02 board.
It can work ok with Intel 82599 networking card.
This is the bootup log which contains PCIe host and Intel 82599 networking card part.

Tested-by: Dongdong Liu <liudongdong3@huawei.com>

EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
GMAC ExitBootServicesEvent
SMMU ExitBootServicesEvent
[    0.000000] Booting Linux on physical CPU 0x20000
[    0.000000] Linux version 4.6.0-rc1+ (l00290354@linux-ioko) (gcc version 4.9.3 20150211 (prerelease) (20150316) ) #43 SMP PREEMPT Thu May 19 17:15:30 CST 2016
[    0.000000] Boot CPU: AArch64 Processor [411fd071]
[    0.000000] earlycon: uart8250 at MMIO32 0x0000000080300000 (options '')
[    0.000000] bootconsole [uart8250] enabled
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] EFI v2.50 by EDK II
[    0.000000] efi:  SMBIOS=0x7a650000  SMBIOS 3.0=0x7a630000  ACPI=0x7aba0000  ACPI 2.0=0x7aba0014
[    0.000000] cma: Reserved 16 MiB at 0x000000007e800000
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x000000007ABA0014 000024 (v02 HISI  )
[    0.000000] ACPI: XSDT 0x000000007A7000E8 000064 (v01 HISI   HISI-D02 20140727      01000013)
[    0.000000] ACPI: FACP 0x000000007A5F0000 00010C (v05 HISI   HISI-D02 20140727 HISI 00000099)
[    0.000000] ACPI: DSDT 0x000000007A5A0000 001656 (v01 HISI   HISI-D02 20140727 INTL 20150619)
[    0.000000] ACPI: DBG2 0x000000007A610000 00005A (v00 HISI   HISI-D02 20140727 HISI 00000099)
[    0.000000] ACPI: GTDT 0x000000007A5E0000 000060 (v02 HISI   HISI-D02 20140727 HISI 00000099)
[    0.000000] ACPI: APIC 0x000000007A5D0000 000554 (v01 HISI   HISI-D02 20140727 HISI 00000099)
[    0.000000] ACPI: MCFG 0x000000007A5C0000 00004C (v01 HISI   HISI-D02 20140727 HISI 00000099)
[    0.000000] ACPI: SPCR 0x000000007A5B0000 000050 (v02 HISI   HISI-D02 20140727 HISI 00000099)
[    0.000000] ACPI: IORT 0x000000007A590000 0001FC (v00 INTEL  TEMPLATE 00000000 INTL 20150619)
[    0.000000] ACPI: SSDT 0x000000007A580000 00046E (v01 HISI   SAS0     20140727 INTL 20150619)
[    0.000000] ACPI: SPCR: console: uart,mmio,0x80300000,115200
[    0.000000] psci: probing for conduit method from ACPI.
NOTICE:  [psci_smc_handler]:[347L] PSCI_VERSION CALL
NOTICE:  [psci_version]:[99L] PSCI_MAJOR_VER: 10000: PSCI_MINOR_VER: 0

0808?844
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs

0808?844
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.

0808?844

0808?844
[    0.000000] percpu: Embedded 20 pages/cpu @ffffffd1ffe7e000 s43008 r8192 d30720 u81920
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for ARM erratum 832075
[    0.000000] CPU features: enabling workaround for ARM erratum 834220
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 2063376
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon=uart8250,mmio32,0x80300000 initrd=filesystem.cpio.gz acpi=force pcie_aspm=off
[    0.000000] PCIe ASPM is disabled
[    0.000000] log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] log_buf_len total cpu_extra contributions: 61440 bytes
[    0.000000] log_buf_len min size: 16384 bytes
[    0.000000] log_buf_len: 131072 bytes
[    0.000000] early log buf free: 12988(79%)
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.000000] software IO TLB [mem 0x764e0000-0x7a4e0000] (64MB) mapped at [ffffffc0764e0000-ffffffc07a4dffff]
[    0.000000] Memory: 8110296K/8384512K available (7240K kernel code, 632K rwdata, 3028K rodata, 840K init, 247K bss, 257832K reserved, 16384K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffffff8000000000 - 0xffffff8008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffffff8008000000 - 0xffffffbdbfff0000   (   246 GB)
[    0.000000]       .text : 0xffffff8008080000 - 0xffffff8008790000   (  7232 KB)
[    0.000000]     .rodata : 0xffffff8008790000 - 0xffffff8008a89000   (  3044 KB)
[    0.000000]       .init : 0xffffff8008a89000 - 0xffffff8008b5b000   (   840 KB)
[    0.000000]       .data : 0xffffff8008b5b000 - 0xffffff8008bf9200   (   633 KB)
[    0.000000]     vmemmap : 0xffffffbdc0000000 - 0xffffffbfc0000000   (     8 GB maximum)
[    0.000000]               0xffffffbdc0000000 - 0xffffffbe08000000   (  1152 MB actual)
[    0.000000]     fixed   : 0xffffffbffe7fd000 - 0xffffffbffec00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffffffbffee00000 - 0xffffffbfffe00000   (    16 MB)
[    0.000000]     memory  : 0xffffffc000000000 - 0xffffffd200000000   ( 73728 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=16, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=16.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=16
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] ITS@0x8c000000
[    0.000000] ITS: allocated 65536 Devices @11f6c80000 (psz 4K, shr 1)
[    0.000000] ITS: allocated 512 Virtual CPUs @11f6c0f000 (psz 4K, shr 1)
[    0.000000] ITS: allocated 512 Interrupt Collections @11f6c20000 (psz 4K, shr 1)
[    0.000000] ITS@0xc6000000
[    0.000000] ITS: allocated 65536 Devices @11f6d00000 (psz 4K, shr 1)
[    0.000000] ITS: allocated 512 Virtual CPUs @11f6c21000 (psz 4K, shr 1)
[    0.000000] ITS: allocated 512 Interrupt Collections @11f6c22000 (psz 4K, shr 1)
[    0.000000] ITS@0xa3000000
[    0.000000] ITS: allocated 65536 Devices @11f6d80000 (psz 4K, shr 1)
[    0.000000] ITS: allocated 512 Virtual CPUs @11f6c24000 (psz 4K, shr 1)
[    0.000000] ITS: allocated 512 Interrupt Collections @11f6c25000 (psz 4K, shr 1)
[    0.000000] ITS@0xb7000000
[    0.000000] ITS: allocated 65536 Devices @11f6e00000 (psz 4K, shr 1)
[    0.000000] ITS: allocated 512 Virtual CPUs @11f6c26000 (psz 4K, shr 1)
[    0.000000] ITS: allocated 512 Interrupt Collections @11f6c27000 (psz 4K, shr 1)
[    0.000000] GIC: using LPI property table @0x00000011f6c60000
[    0.000000] ITS: Allocated 1792 chunks for LPIs
[    0.000000] CPU0: found redistributor 20000 region 0:0x000000008d100000
[    0.000000] CPU0: using LPI pending table @0x00000011f6c70000
[    0.000000] Unable to get hardware information used for virtualization
[    0.000000] GTDT: No Platform Timer structures.
[    0.000000] arch_timer: Can't find GT Block.
[    0.000000] Architected cp15 and mmio timer(s) running at 50.00MHz (phys/phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
[    0.000001] sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
[    0.008025] Console: colour dummy device 80x25
[    0.012459] Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000)
[    0.022850] pid_max: default: 32768 minimum: 301
[    0.027450] ACPI: Core revision 20160108
[    0.032284] ACPI: 2 ACPI AML tables successfully acquired and loaded
[    0.038617]
[    0.040126] Security Framework initialized
[    0.044213] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes)
[    0.051053] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes)
[    0.058747] ASID allocator initialised with 65536 entries
[    0.064393] PCI/MSI: ITS@0x8c000000 domain created
[    0.069162] PCI/MSI: ITS@0xc6000000 domain created
[    0.073927] PCI/MSI: ITS@0xa3000000 domain created
[    0.078693] PCI/MSI: ITS@0xb7000000 domain created
[    0.083463] Platform MSI: irqchip@000000008c000000 domain created
[    0.089525] Platform MSI: irqchip@00000000c6000000 domain created
[    0.095586] Platform MSI: irqchip@00000000a3000000 domain created
[    0.101648] Platform MSI: irqchip@00000000b7000000 domain created
[    0.107753] Remapping and enabling EFI services.
[    0.112370]   EFI remap 0x000000007a4e0000 => 0000000020000000
[    0.118185]   EFI remap 0x000000007a530000 => 0000000020050000
[    0.124008]   EFI remap 0x000000007a620000 => 00000000200a0000
[    0.129822]   EFI remap 0x000000007a6b0000 => 0000000020130000
[    0.135636]   EFI remap 0x000000007a710000 => 0000000020180000
[    0.141454]   EFI remap 0x000000007a760000 => 00000000201d0000
[    0.147269]   EFI remap 0x000000007a7b0000 => 0000000020220000
[    0.153083]   EFI remap 0x000000007a800000 => 0000000020270000
[    0.158897]   EFI remap 0x000000007a850000 => 00000000202c0000
[    0.164711]   EFI remap 0x000000007a8a0000 => 0000000020310000
[    0.170525]   EFI remap 0x000000007a8f0000 => 0000000020360000
[    0.176339]   EFI remap 0x000000007a940000 => 00000000203b0000
[    0.182160]   EFI remap 0x000000007a990000 => 0000000020400000
[    0.187974]   EFI remap 0x000000007aa00000 => 0000000020470000
[    0.193788]   EFI remap 0x000000007aa50000 => 00000000204c0000
[    0.199602]   EFI remap 0x000000007aaa0000 => 0000000020510000
[    0.205417]   EFI remap 0x000000007aaf0000 => 0000000020560000
[    0.211230]   EFI remap 0x000000007ab40000 => 00000000205b0000
[    0.217043]   EFI remap 0x000000007fbb0000 => 0000000020600000
[    0.222849]   EFI remap 0x0000000080300000 => 0000000020630000
[    0.228654]   EFI remap 0x00000000a00f0000 => 0000000020640000
[    0.234457]   EFI remap 0x00000000a4000000 => 0000000020800000
[    0.240266]   EFI remap 0x00000000a6000000 => 0000000021800000
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20001 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x1

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x1


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c080
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3d190
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20002 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x1

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x1


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c100
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3d3a0
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20003 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x1

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x1


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c180
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3d5b0
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20100 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x1

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x3


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c200
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3d7c0
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20101 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x3

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x3


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c280
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3d9d0
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20102 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x3

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x3


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c300
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3dbe0
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20103 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x3

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x3


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c380
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3ddf0
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20200 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x3

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x7


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c400
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3e000
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20201 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x7

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x7


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c480
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3e210
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20202 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x7

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x7


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c500
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3e420
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20203 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x7

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0x7


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c580
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3e630
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20300 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0x7

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0xf


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c600
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3e840
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20301 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0xf

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0xf


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c680
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3ea50
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20302 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0xf

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0xf


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c700
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3ec60
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
NOTICE:  [psci_smc_handler]:[408L] PSCI_CPU_ON_AARCH64 CALL
NOTICE:  [psci_smc_handler]:[409L] x1=0x20303 x2=0x82870 x3=0x0
NOTICE:  [scpi_set_css_power_state]:[85L] domain_cluster=0xf

NOTICE:  [scpi_set_css_power_state]:[93L] domain_cluster=0xf


0808?8AB44
NOTICE:  [psci_afflvl_power_on_finish]:[504L]
NOTICE:  [cm_prepare_el3_exit]:[262L] read_tpidr_el3 = 7fc3c780
NOTICE:  [cm_prepare_el3_exit]:[319L] ctx add = 7fc3ee70
NOTICE:  [psci_afflvl_power_on_finish]:[562L]

00082870
[    0.288031] Detected PIPT I-cache on CPU1
[    0.288043] CPU1: found redistributor 20001 region 1:0x000000008d130000
[    0.288064] CPU1: using LPI pending table @0x00000011f6410000
[    0.288122] CPU1: Booted secondary processor [411fd071]
[    0.331143] Detected PIPT I-cache on CPU2
[    0.331150] CPU2: found redistributor 20002 region 2:0x000000008d160000
[    0.331170] CPU2: using LPI pending table @0x00000011f6440000
[    0.331218] CPU2: Booted secondary processor [411fd071]
[    0.374257] Detected PIPT I-cache on CPU3
[    0.374263] CPU3: found redistributor 20003 region 3:0x000000008d190000
[    0.374283] CPU3: using LPI pending table @0x00000011f6480000
[    0.374328] CPU3: Booted secondary processor [411fd071]
[    0.417372] Detected PIPT I-cache on CPU4
[    0.417380] CPU4: found redistributor 20100 region 4:0x000000008d1c0000
[    0.417400] CPU4: using LPI pending table @0x00000011f64c0000
[    0.417447] CPU4: Booted secondary processor [411fd071]
[    0.460484] Detected PIPT I-cache on CPU5
[    0.460491] CPU5: found redistributor 20101 region 5:0x000000008d1f0000
[    0.460511] CPU5: using LPI pending table @0x00000011f64f0000
[    0.460558] CPU5: Booted secondary processor [411fd071]
[    0.503598] Detected PIPT I-cache on CPU6
[    0.503605] CPU6: found redistributor 20102 region 6:0x000000008d220000
[    0.503625] CPU6: using LPI pending table @0x00000011f6530000
[    0.503669] CPU6: Booted secondary processor [411fd071]
[    0.546711] Detected PIPT I-cache on CPU7
[    0.546718] CPU7: found redistributor 20103 region 7:0x000000008d250000
[    0.546738] CPU7: using LPI pending table @0x00000011f6560000
[    0.546783] CPU7: Booted secondary processor [411fd071]
[    0.589826] Detected PIPT I-cache on CPU8
[    0.589835] CPU8: found redistributor 20200 region 8:0x000000008d280000
[    0.589857] CPU8: using LPI pending table @0x00000011f65a0000
[    0.589910] CPU8: Booted secondary processor [411fd071]
[    0.632939] Detected PIPT I-cache on CPU9
[    0.632946] CPU9: found redistributor 20201 region 9:0x000000008d2b0000
[    0.632967] CPU9: using LPI pending table @0x00000011f65e0000
[    0.633014] CPU9: Booted secondary processor [411fd071]
[    0.676052] Detected PIPT I-cache on CPU10
[    0.676060] CPU10: found redistributor 20202 region 10:0x000000008d2e0000
[    0.676081] CPU10: using LPI pending table @0x00000011f6610000
[    0.676126] CPU10: Booted secondary processor [411fd071]
[    0.719166] Detected PIPT I-cache on CPU11
[    0.719173] CPU11: found redistributor 20203 region 11:0x000000008d310000
[    0.719195] CPU11: using LPI pending table @0x00000011f6650000
[    0.719240] CPU11: Booted secondary processor [411fd071]
[    0.762280] Detected PIPT I-cache on CPU12
[    0.762289] CPU12: found redistributor 20300 region 12:0x000000008d340000
[    0.762311] CPU12: using LPI pending table @0x00000011f6680000
[    0.762360] CPU12: Booted secondary processor [411fd071]
[    0.805392] Detected PIPT I-cache on CPU13
[    0.805400] CPU13: found redistributor 20301 region 13:0x000000008d370000
[    0.805421] CPU13: using LPI pending table @0x00000011f66c0000
[    0.805466] CPU13: Booted secondary processor [411fd071]
[    0.848506] Detected PIPT I-cache on CPU14
[    0.848513] CPU14: found redistributor 20302 region 14:0x000000008d3a0000
[    0.848534] CPU14: using LPI pending table @0x00000011f6700000
[    0.848579] CPU14: Booted secondary processor [411fd071]
[    0.891620] Detected PIPT I-cache on CPU15
[    0.891628] CPU15: found redistributor 20303 region 15:0x000000008d3d0000
[    0.891648] CPU15: using LPI pending table @0x00000011f6750000
[    0.891693] CPU15: Booted secondary processor [411fd071]
[    0.891723] Brought up 16 CPUs
[    1.220754] SMP: Total of 16 processors activated.
[    1.225520] CPU features: detected feature: GIC system register CPU interface
[    1.232620] CPU: All CPU(s) started at EL2
[    1.236718] alternatives: patching kernel code
[    1.243905] devtmpfs: initialized
[    1.247446] SMBIOS 3.0.0 present.
[    1.250851] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    1.260790] pinctrl core: initialized pinctrl subsystem
[    1.266365] NET: Registered protocol family 16
[    1.282800] cpuidle: using governor menu
[    1.286747] vdso: 2 pages (1 code @ ffffff8008796000, 1 data @ ffffff8008b60000)
[    1.294122] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    1.301211] DMA: preallocated 256 KiB pool for atomic allocations
[    1.307340] ACPI: bus type PCI registered
[    1.311388] Serial: AMBA PL011 UART driver
[    1.331638] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.338373] ACPI: Added _OSI(Module Device)
[    1.342562] ACPI: Added _OSI(Processor Device)
[    1.346984] ACPI: Added _OSI(3.0 _SCP Extensions)
[    1.351664] ACPI: Added _OSI(Processor Aggregator Device)
[    1.358469] ACPI: Interpreter enabled
[    1.362116] ACPI: Using GIC for interrupt routing
[    1.366815] ACPI: MCFG table loaded, 2 entries detected
[    1.375180] Hisilicon MBIGEN-V1 HISI0151:00: Allocated 256 MSIs
[    1.381158] Hisilicon MBIGEN-V1 HISI0151:01: Allocated 640 MSIs
[    1.387123] Hisilicon MBIGEN-V1 HISI0151:02: Allocated 256 MSIs
[    1.393091] Hisilicon MBIGEN-V1 HISI0151:03: Allocated 640 MSIs
[    1.399213] ACPI: IORT: can't find node related to (null) device
[    1.405292] ACPI: IORT: can't find node related to (null) device
[    1.411324] ACPI: IORT: can't find node related to (null) device
[    1.417348] ACPI: IORT: can't find node related to (null) device
[    1.423370] ACPI: IORT: can't find node related to (null) device
[    1.429391] ACPI: IORT: can't find node related to (null) device
[    1.436634] ACPI: IORT: can't find node related to (null) device
[    1.442681] ACPI: IORT: can't find node related to (null) device
[    1.448698] ACPI: IORT: can't find node related to (null) device
[    1.454712] ACPI: IORT: can't find node related to (null) device
[    1.460724] ACPI: IORT: can't find node related to (null) device
[    1.466764] ACPI: PCI Root Bridge [PCI1] (domain 0001 [bus 40-7f])
[    1.472918] acpi HISI0080:00: _OSC: OS supports [ExtendedConfig Segments MSI]
[    1.480025] acpi HISI0080:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    1.486771] acpi HISI0080:00: ECAM at [mem 0x22004000000-0x22007ffffff] for [bus 40-7f]
[    1.494762] Remapped I/O 0x000002200fff0000 to [io  0x0000-0xffff window]
[    1.501592] PCI host bridge to bus 0001:40
[    1.505671] pci_bus 0001:40: root bus resource [mem 0x22008000000-0x2200ffeffff window] (bus address [0xb0000000-0xb7feffff])
[    1.516919] pci_bus 0001:40: root bus resource [io  0x0000-0xffff window]
[    1.523675] pci_bus 0001:40: root bus resource [bus 40-7f]
[    1.534090] pci 0001:41:00.0: VF(n) BAR0 space: [mem 0x22008e08000-0x22008f07fff 64bit pref] (contains BAR0 for 64 VFs)
[    1.545323] pci 0001:41:00.0: VF(n) BAR3 space: [mem 0x22008f08000-0x22009007fff 64bit pref] (contains BAR3 for 64 VFs)
[    1.568223] pci 0001:41:00.1: VF(n) BAR0 space: [mem 0x22008c04000-0x22008d03fff 64bit pref] (contains BAR0 for 64 VFs)
[    1.579460] pci 0001:41:00.1: VF(n) BAR3 space: [mem 0x22008d04000-0x22008e03fff 64bit pref] (contains BAR3 for 64 VFs)
[    1.597823] pci 0001:40:00.0: BAR 15: assigned [mem 0x22008000000-0x220095fffff pref]
[    1.605616] pci 0001:40:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    1.611771] pci 0001:41:00.0: BAR 0: assigned [mem 0x22008000000-0x220083fffff 64bit pref]
[    1.620245] pci 0001:41:00.0: BAR 6: assigned [mem 0x22008400000-0x220087fffff pref]
[    1.627952] pci 0001:41:00.1: BAR 0: assigned [mem 0x22008800000-0x22008bfffff 64bit pref]
[    1.636430] pci 0001:41:00.1: BAR 6: assigned [mem 0x22008c00000-0x22008ffffff pref]
[    1.644138] pci 0001:41:00.0: BAR 4: assigned [mem 0x22009000000-0x22009003fff 64bit pref]
[    1.652610] pci 0001:41:00.0: BAR 7: assigned [mem 0x22009004000-0x22009103fff 64bit pref]
[    1.661083] pci 0001:41:00.0: BAR 10: assigned [mem 0x22009104000-0x22009203fff 64bit pref]
[    1.669640] pci 0001:41:00.1: BAR 4: assigned [mem 0x22009204000-0x22009207fff 64bit pref]
[    1.678115] pci 0001:41:00.1: BAR 7: assigned [mem 0x22009208000-0x22009307fff 64bit pref]
[    1.686591] pci 0001:41:00.1: BAR 10: assigned [mem 0x22009308000-0x22009407fff 64bit pref]
[    1.695148] pci 0001:41:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    1.701285] pci 0001:41:00.1: BAR 2: assigned [io  0x1020-0x103f]
[    1.707421] pci 0001:40:00.0: PCI bridge to [bus 41-42]
[    1.712621] pci 0001:40:00.0:   bridge window [io  0x1000-0x1fff]
[    1.718685] pci 0001:40:00.0:   bridge window [mem 0x22008000000-0x220095fffff pref]
[    1.726434] ACPI: PCI Root Bridge [PCI2] (domain 0002 [bus 80-bf])
[    1.732594] acpi HISI0080:01: _OSC: OS supports [ExtendedConfig Segments MSI]
[    1.739696] acpi HISI0080:01: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    1.746425] acpi HISI0080:01: link status is down
[    1.751106] acpi HISI0080:01: ECAM at [mem 0x24008000000-0x2400bffffff] for [bus 80-bf]
[    1.759090] Remapped I/O 0x000002400fff0000 to [io  0x10000-0x1ffff window]
[    1.766086] PCI host bridge to bus 0002:80
[    1.770172] pci_bus 0002:80: root bus resource [mem 0x2400c000000-0x2400ffeffff window] (bus address [0xc0000000-0xc3feffff])
[    1.781422] pci_bus 0002:80: root bus resource [io  0x10000-0x1ffff window] (bus address [0x0000-0xffff])
[    1.790940] pci_bus 0002:80: root bus resource [bus 80-bf]
[    1.796414] pci 0002:80:00.0: ignoring class 0x000000 (doesn't match header type 01)
[    1.804311] pci 0002:80:00.0: not setting up bridge for bus 0002:81
[    1.810956] ACPI: IORT: can't find node related to (null) device
[    1.817228] ACPI: IORT: can't find node related to (null) device
[    1.823453] vgaarb: loaded
[    1.826246] SCSI subsystem initialized
[    1.830103] ACPI: bus type USB registered
[    1.834130] usbcore: registered new interface driver usbfs
[    1.839606] usbcore: registered new interface driver hub
[    1.844933] usbcore: registered new device driver usb
[    1.850014] pps_core: LinuxPPS API ver. 1 registered
[    1.854953] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.864048] PTP clock support registered
[    1.868069] Advanced Linux Sound Architecture Driver Initialized.
[    1.874376] clocksource: Switched to clocksource arch_sys_counter
[    1.880489] VFS: Disk quotas dquot_6.6.0
[    1.884414] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.891385] pnp: PnP ACPI init
[    1.894570] system 00:00: [mem 0xb0080000-0xb008ffff] has been reserved
[    1.901209] system 00:01: [mem 0xb0090000-0xb009ffff] has been reserved
[    1.907813] pnp: PnP ACPI: found 2 devices
[    1.914467] NET: Registered protocol family 2
[    1.919030] TCP established hash table entries: 65536 (order: 7, 524288 bytes)
[    1.926352] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    1.933333] TCP: Hash tables configured (established 65536 bind 65536)
[    1.939876] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[    1.945968] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[    1.952576] NET: Registered protocol family 1
[    1.957013] RPC: Registered named UNIX socket transport module.
[    1.962905] RPC: Registered udp transport module.
[    1.967586] RPC: Registered tcp transport module.
[    1.972266] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.978872] Unpacking initramfs...
[    2.340835] Freeing initrd memory: 27492K (ffffffc01e520000 - ffffffc01fff9000)
[    2.348645] kvm [1]: 8-bit VMID
[    2.351776] kvm [1]: Hyp mode initialized successfully
[    2.356892] kvm [1]: error: KVM vGIC probing failed
[    2.361790] kvm [1]: virtual timer IRQ3
[    2.366650] ACPI: IORT: can't find node related to (null) device
[    2.373057] futex hash table entries: 4096 (order: 7, 524288 bytes)
[    2.379403] audit: initializing netlink subsys (disabled)
[    2.384813] audit: type=2000 audit(1.864:1): initialized
[    2.390302] workingset: timestamp_bits=44 max_order=21 bucket_order=0
[    2.400222] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    2.406342] NFS: Registering the id_resolver key type
[    2.411393] Key type id_resolver registered
[    2.415557] Key type id_legacy registered
[    2.419609] fuse init (API version 7.24)
[    2.423687] 9p: Installing v9fs 9p2000 file system support
[    2.429982] io scheduler noop registered
[    2.433946] io scheduler cfq registered (default)
[    2.439027] pcieport 0001:40:00.0: can't derive routing for PCI INT A
[    2.445438] pcieport 0001:40:00.0: PCI INT A: no GSI
[    2.451126] xenfs: not registering filesystem on non-xen platform
[    2.458262] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    2.465049] console [ttyS0] disabled
[    2.468627] APMC0D08:00: ttyS0 at MMIO 0x80300000 (irq = 5, base_baud = 12500000) is a 16550A
[    2.477133] console [ttyS0] enabled
[    2.477133] console [ttyS0] enabled
[    2.484137] bootconsole [uart8250] disabled
[    2.484137] bootconsole [uart8250] disabled
[    2.492728] SuperH (H)SCI(F) driver initialized
[    2.497322] msm_serial: driver initialized
[    2.501649] Failed to find cpu0 device node
[    2.505852] Unable to detect cache hierarchy from DT for CPU 0
[    2.514093] loop: module loaded
[    2.517694] tun: Universal TUN/TAP device driver, 1.6
[    2.522771] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    2.529079] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    2.534939] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    2.540908] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.3.0-k
[    2.547900] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.553518] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.0.2-k
[    2.561382] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    2.567350] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.2.1-k
[    2.575039] ixgbe: Copyright (c) 1999-2015 Intel Corporation.
[    2.580854] pcieport 0001:40:00.0: can't derive routing for PCI INT A
[    2.587326] ixgbe 0001:41:00.0: PCI INT A: no GSI
[    2.592178] ixgbe 0001:41:00.0: enabling device (0000 -> 0002)
[    2.753716] ixgbe 0001:41:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16
[    2.762393] ixgbe 0001:41:00.0: PCI Express bandwidth of 32GT/s available
[    2.769213] ixgbe 0001:41:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    2.776457] ixgbe 0001:41:00.0: MAC: 2, PHY: 17, SFP+: 5, PBA No: FFFFFF-0FF
[    2.783537] ixgbe 0001:41:00.0: 68:a8:28:2e:c9:10
[    2.792732] ixgbe 0001:41:00.0: Intel(R) 10 Gigabit Network Connection
[    2.799312] pcieport 0001:40:00.0: can't derive routing for PCI INT B
[    2.805783] ixgbe 0001:41:00.1: PCI INT B: no GSI
[    2.810598] ixgbe 0001:41:00.1: enabling device (0000 -> 0002)
[    3.949697] ixgbe 0001:41:00.1: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16
[    3.958365] ixgbe 0001:41:00.1: PCI Express bandwidth of 32GT/s available
[    3.965185] ixgbe 0001:41:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    3.972427] ixgbe 0001:41:00.1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    3.978634] ixgbe 0001:41:00.1: 68:a8:28:2e:c9:11
[    3.987790] ixgbe 0001:41:00.1: Intel(R) 10 Gigabit Network Connection
[    3.994388] sky2: driver version 1.30
[    3.998201] VFIO - User Level meta-driver version: 0.3
[    4.003889] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    4.010459] ehci-pci: EHCI PCI platform driver
[    4.014944] ehci-platform: EHCI generic platform driver
[    4.020226] ehci-platform PNP0D20:00: EHCI Host Controller
[    4.025745] ehci-platform PNP0D20:00: new USB bus registered, assigned bus number 1
[    4.033572] ehci-platform PNP0D20:00: irq 6, io mem 0xa1000000
[    4.050387] ehci-platform PNP0D20:00: USB 2.0 started, EHCI 1.00
[    4.056652] hub 1-0:1.0: USB hub found
[    4.060428] hub 1-0:1.0: 1 port detected
[    4.064517] ehci-msm: Qualcomm On-Chip EHCI Host Controller
[    4.070137] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    4.076351] ohci-pci: OHCI PCI platform driver
[    4.080834] ohci-platform: OHCI generic platform driver
[    4.086166] usbcore: registered new interface driver usb-storage
[    4.092427] mousedev: PS/2 mouse device common for all mice
[    4.161533] rtc-efi rtc-efi: rtc core: registered rtc-efi as rtc0
[    4.168513] i2c /dev entries driver
[    4.172251] sdhci: Secure Digital Host Controller Interface driver
[    4.178458] sdhci: Copyright(c) Pierre Ossman
[    4.182850] Synopsys Designware Multimedia Card Interface Driver
[    4.188933] sdhci-pltfm: SDHCI platform and OF driver helper
[    4.194701] ledtrig-cpu: registered to indicate activity on CPUs
[    4.200950] usbcore: registered new interface driver usbhid
[    4.206549] usbhid: USB HID core driver
[    4.210569] ACPI: IORT: can't find node related to (null) device
[    4.216837] NET: Registered protocol family 17
[    4.221334] 9pnet: Installing 9P2000 support
[    4.225652] Key type dns_resolver registered
[    4.230166] registered taskstats version 1
[    4.297850] rtc-efi rtc-efi: hctosys: unable to read the hardware clock
[    4.304576] ALSA device list:
[    4.307561]   No soundcards found.
[    4.311063] ttyS0 - failed to request DMA
[    4.315348] Freeing unused kernel memory: 840K (ffffff8008a89000 - ffffff8008b5b000)
root@(none)$ ifconfig eth0 192.168.20.188
[   15.679957] ixgbe 0001:41:00.0: registered PHC device on eth0
root@(none)$ [   15.851142] ixgbe 0001:41:00.0 eth0: detected SFP+: 5
[   15.990419] ixgbe 0001:41:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

root@(none)$ ping 192.168.20.188
PING 192.168.20.4 (192.168.20.4): 56 data bytes
64 bytes from 192.168.20.4: seq=14 ttl=128 time=1.465 ms
64 bytes from 192.168.20.4: seq=15 ttl=128 time=0.616 ms
64 bytes from 192.168.20.4: seq=16 ttl=128 time=0.391 ms
64 bytes from 192.168.20.4: seq=17 ttl=128 time=0.698 ms
64 bytes from 192.168.20.4: seq=18 ttl=128 time=0.676 ms
64 bytes from 192.168.20.4: seq=19 ttl=128 time=0.524 ms
64 bytes from 192.168.20.4: seq=20 ttl=128 time=0.301 ms
64 bytes from 192.168.20.4: seq=21 ttl=128 time=0.248 ms
64 bytes from 192.168.20.4: seq=22 ttl=128 time=0.403 ms

Re: [PATCH V7 08/11] pci, acpi: Support for ACPI based generic PCI host controller

[Tomasz Nowicki]

On 14.05.2016 11:07, Jayachandran C wrote:
> On Tue, May 10, 2016 at 8:49 PM, Tomasz Nowicki <tn@semihalf.com> wrote:
>> This patch is going to implement generic PCI host controller for
>> ACPI world, similar to what pci-host-generic.c driver does for DT world.
>>
>> All such drivers, which we have seen so far, were implemented within
>> arch/ directory since they had some arch assumptions (x86 and ia64).
>> However, they all are doing similar thing, so it makes sense to find
>> some common code and abstract it into the generic driver.
>>
>> In order to handle PCI config space regions properly, we define new
>> MCFG interface which does sanity checks on MCFG table and keeps its
>> root pointer. User is able to lookup MCFG regions based on that root
>> pointer and specified domain:bus_start:bus_end touple. We are using
>> pci_mmcfg_late_init old prototype to avoid another function name.
>>
>> The implementation of pci_acpi_scan_root() looks up the MCFG entries
>> and sets up a new mapping (regions are not mapped until host controller ask
>> for it). Generic PCI functions are used for accessing config space.
>> Driver selects PCI_ECAM and uses functions from drivers/pci/ecam.h
>> to create and access ECAM mappings.
>>
>> As mentioned in Kconfig help section, ACPI_PCI_HOST_GENERIC choice
>> should be made on a per-architecture basis.
>
> Looking thru the new code, I see a few issues, please see below
>
>> Signed-off-by: Tomasz Nowicki <tn@semihalf.com>
>> Signed-off-by: Jayachandran C <jchandra@broadcom.com>
>> ---
>>   drivers/acpi/Kconfig            |   8 +++
>>   drivers/acpi/Makefile           |   1 +
>>   drivers/acpi/pci_mcfg.c         |  97 ++++++++++++++++++++++++++
>>   drivers/acpi/pci_root_generic.c | 149 ++++++++++++++++++++++++++++++++++++++++
>>   drivers/pci/ecam.h              |   5 ++
>>   include/linux/pci-acpi.h        |   5 ++
>>   include/linux/pci.h             |   5 +-
>>   7 files changed, 269 insertions(+), 1 deletion(-)
>>   create mode 100644 drivers/acpi/pci_mcfg.c
>>   create mode 100644 drivers/acpi/pci_root_generic.c
>>
>> diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
>> index 183ffa3..44afc76 100644
>> --- a/drivers/acpi/Kconfig
>> +++ b/drivers/acpi/Kconfig
>> @@ -346,6 +346,14 @@ config ACPI_PCI_SLOT
>>            i.e., segment/bus/device/function tuples, with physical slots in
>>            the system.  If you are unsure, say N.
>>
>> +config ACPI_PCI_HOST_GENERIC
>> +       bool
>> +       select PCI_ECAM
>> +       help
>> +         Select this config option from the architecture Kconfig,
>> +         if it is preferred to enable ACPI PCI host controller driver which
>> +         has no arch-specific assumptions.
>> +
>>   config X86_PM_TIMER
>>          bool "Power Management Timer Support" if EXPERT
>>          depends on X86
>> diff --git a/drivers/acpi/Makefile b/drivers/acpi/Makefile
>> index 81e5cbc..627a2b7 100644
>> --- a/drivers/acpi/Makefile
>> +++ b/drivers/acpi/Makefile
>> @@ -40,6 +40,7 @@ acpi-$(CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC) += processor_pdc.o
>>   acpi-y                         += ec.o
>>   acpi-$(CONFIG_ACPI_DOCK)       += dock.o
>>   acpi-y                         += pci_root.o pci_link.o pci_irq.o
>> +obj-$(CONFIG_ACPI_PCI_HOST_GENERIC)    += pci_root_generic.o pci_mcfg.o
>>   acpi-y                         += acpi_lpss.o acpi_apd.o
>>   acpi-y                         += acpi_platform.o
>>   acpi-y                         += acpi_pnp.o
>> diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c
>> new file mode 100644
>> index 0000000..373d079
>> --- /dev/null
>> +++ b/drivers/acpi/pci_mcfg.c
>> @@ -0,0 +1,97 @@
>> +/*
>> + * Copyright (C) 2016 Broadcom
>> + *     Author: Jayachandran C <jchandra@broadcom.com>
>> + * Copyright (C) 2016 Semihalf
>> + *     Author: Tomasz Nowicki <tn@semihalf.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License, version 2, as
>> + * published by the Free Software Foundation (the "GPL").
>> + *
>> + * This program is distributed in the hope that it will be useful, but
>> + * WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> + * General Public License version 2 (GPLv2) for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * version 2 (GPLv2) along with this source code.
>> + */
>> +#include <linux/kernel.h>
>> +#include <linux/pci.h>
>> +#include <linux/pci-acpi.h>
>> +
>> +#define PREFIX "ACPI: "
>> +
>> +/* Root pointer to the mapped MCFG table */
>> +static struct acpi_table_mcfg *mcfg_table;
>> +
>> +#define MCFG_ENTRIES(mcfg_ptr) (((mcfg_ptr)->header.length -           \
>> +                               sizeof(struct acpi_table_mcfg)) /       \
>> +                               sizeof(struct acpi_mcfg_allocation))
>
> It would be better if you used static inline function here.

OK

>
>> +static phys_addr_t pci_mcfg_lookup_static(u16 seg, u8 bus_start, u8 bus_end)
>> +{
>> +       struct acpi_mcfg_allocation *mptr;
>> +       int i;
>> +
>> +       if (!mcfg_table) {
>> +               pr_err(PREFIX "MCFG table not available, lookup failed\n");
>> +               return -ENXIO;
>> +       }
>> +
>> +       mptr = (struct acpi_mcfg_allocation *) &mcfg_table[1];
>> +
>> +       /*
>> +        * We expect exact match, unless MCFG entry end bus covers more than
>> +        * specified by caller.
>> +        */
>> +       for (i = 0; i < MCFG_ENTRIES(mcfg_table); i++, mptr++) {
>> +               if (mptr->pci_segment == seg &&
>> +                   mptr->start_bus_number == bus_start &&
>> +                   mptr->end_bus_number >= bus_end) {
>> +                       return mptr->address;
>> +               }
>> +       }
>
> There is an issue here, the bus range is obtained if different
> ways. If the _CRS has it, this should be fine. But if it is _BBN-0xff
> or the default 0-0xff, then I would think taking the MCFG entry end
> would be better. This would require updating the bus resource end.

Yeah we can implement algorithm like that but IMO it would try to guess 
what FW designer meant putting various combination of BUS resources into 
the tables. We should rather be strict about this so I agree with Bjorn:
https://lkml.org/lkml/2016/4/28/807
Either we get exact match or MCFG covers more than the host bridge range.

Host bridge bus range definition -> MCFG
_BBN-0xff -> we expect exact _BBN-0xff match
0-0xff -> ditto
_CRS (the only way to trim host bridge bus rage) -> exact match or MCFG 
covers more

>
> Also (trivial), the braces are not needed.

OK

>
>> +       return -ENXIO;
>> +}
>> +
>> +phys_addr_t pci_mcfg_lookup(struct acpi_device *device, u16 seg,
>> +                           struct resource *bus_res)
>> +{
>> +       phys_addr_t addr;
>> +
>> +       addr = acpi_pci_root_get_mcfg_addr(device->handle);
>> +       if (addr)
>> +               return addr;
>
> When you have address from _CBA, you are assuming that the
> bus range is also set correctly (from _CRS or as _BBN-0xff). Is
> this assumption correct?

I think so, see above. Maybe Bjorn may advise here.

>
> (this was discussed earlier)  you are doing the _CBA call again,
> I think cleaning up the _CBA, _BBN, _CRS and MCFG based
> ECAM area resources and defaults should be a different patch,
> which would need changes to pci_root.c. We should use
> root->mcfg_addr in this patchset.

OK

>
>> +
>> +       return pci_mcfg_lookup_static(seg, bus_res->start, bus_res->end);
>
> There is no need to have a separate function for this, based on by
> above comment, you might need to change bus_res, so having it here
> would be better.
>
>> +}
>> +
>> +static __init int pci_mcfg_parse(struct acpi_table_header *header)
>> +{
>> +       struct acpi_table_mcfg *mcfg;
>> +       int n;
>> +
>> +       if (!header)
>> +               return -EINVAL;
>
> This is not needed, the handler is not called if header is NULL

OK

>
>> +
>> +       mcfg = (struct acpi_table_mcfg *)header;
>> +       n = MCFG_ENTRIES(mcfg);
>> +       if (n <= 0 || n > 255) {
>> +               pr_err(PREFIX "MCFG has incorrect entries (%d).\n", n);
>> +               return -EINVAL;
>> +       }
>>
>> +       mcfg_table = mcfg;
>
> Saving a reference of ACPI mapping seems dangerous, acpi_parse_table
> calles early_acpi_os_unmap_memory() after calling handler, which does
> not do anything since acpi_gbl_permanent_mmap is set.
>
> I would suggest copying the entries.

You got the point about early_acpi_os_unmap_memory but that would be the 
case if we call pci_mmcfg_late_init before acpi_early_init. But 
pci_mmcfg_late_init is called much later. Also the code is much simpler 
as is now.

Thanks,
Tomasz

RE: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Gabriele Paoloni]

Hi Lorenzo

> -----Original Message-----
> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> Sent: 23 May 2016 11:57
> To: Ard Biesheuvel
> Cc: Gabriele Paoloni; Jon Masters; Tomasz Nowicki; helgaas@kernel.org;
> arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
> rafael@kernel.org; hanjun.guo@linaro.org; okaya@codeaurora.org;
> jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
> pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
> ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
> kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
> msalter@redhat.com; Wangyijing; mw@semihalf.com;
> andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
> controller
> 
> On Fri, May 20, 2016 at 11:14:03AM +0200, Ard Biesheuvel wrote:
> > On 20 May 2016 at 10:40, Gabriele Paoloni
> <gabriele.paoloni@huawei.com> wrote:
> > > Hi Ard
> > >
> > >> -----Original Message-----
> > >> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> > [...]
> > >>
> > >> Is the PCIe root complex so special that you cannot simply
> describe an
> > >> implementation that is not PNP0408 compatible as something else,
> under
> > >> its own unique HID? If everybody is onboard with using ACPI, how
> is
> > >> this any different from describing other parts of the platform
> > >> topology? Even if the SBSA mandates generic PCI, they already
> deviated
> > >> from that when they built the hardware, so pretending that it is a
> > >> PNP0408 with quirks really does not buy us anything.
> > >
> > > From my understanding we want to avoid this as this would allow
> each
> > > vendor to come up with his own code and it would be much more
> effort
> > > for the PCI maintainer to rework the PCI framework to accommodate
> X86
> > > and "all" ARM64 Host Controllers...
> > >
> > > I guess this approach is too risky and we want to avoid this.
> Through
> > > standardization we can more easily maintain the code and scale it
> to
> > > multiple SoCs...
> > >
> > > So this is my understanding; maybe Jon, Tomasz or Lorenzo can give
> > > a bit more explanation...
> > >
> >
> > OK, so that boils down to recommending to vendors to represent known
> > non-compliant hardware as compliant, just so that we don't have to
> > change the code to support additional flavors of ECAM ? It's fine to
> > be pragmatic, but that sucks.
> >
> > We keep confusing the x86 case with the ARM case here: for x86, they
> > needed to deal with broken hardware *after* the fact, and all they
> > could do is find /some/ distinguishing feature in order to guess
> which
> > exact hardware they might be running on. For arm64, it is the
> opposite
> > case. We are currently in a position where we can demand vendors to
> > comply with the standards they endorsed themselves, and (ab)using
> ACPI
> > + DMI as a de facto platform description rather than plain ACPI makes
> > me think the DT crowd were actually right from the beginning. It
> > *directly* violates the standardization principle, since it requires
> a
> > priori knowledge inside the OS that a certain 'generic' device must
> be
> > driven in a special way.
> >
> > So can anyone comment on the feasibility of adding support for
> devices
> > with vendor specific HIDs (and no generic CIDs) to the current ACPI
> > ECAM driver in Linux?
> 
> Host bridges in ACPI are handled through PNP0A08/PNP0A03 ids, and
> most of the arch specific code is handled in the respective arch
> directories (X86 and IA64, even though IA64 does not rely on ECAM/MCFG
> for
> PCI ops), it is not a driver per-se, PNP0A08/PNP0A03 are detected
> through
> ACPI scan handlers and the respective arch code (ie pci_acpi_scan_root)
> sets-up resources AND config space on an arch specific basis.
> 
> X86 deals with that with code in arch/x86 that sets-up the pci_raw_ops
> on a platform specific basis (and it is not nice, but it works because
> as you all know the number of platforms in X86 world is contained).
> 
> Will this happen for ARM64 in arch/arm64 based on vendor specific
> HIDs ?
> 
> No.
> 
> So given the current state of play (we were requested to move the
> arch/arm64 specific ACPI PCI bits to arch/arm64), we would end up
> with arch/arm64 code requiring code in /drivers to set-up pci_ops
> in a platform specific way, it is horrible, if feasible at all.
> 
> The only way this can be implemented is by pretending that the
> ACPI/PCI arch/arm64 implementation is generic code (that's what this
> series does), move it to /drivers (where it is in this series), and
> implement _DSD vendor specific bindings (per HID) to set-up the pci
> operations; whether this solution should go upstream, given that it
> is just a short-term solution for early platforms bugs, it is another
> story and my personal answer is no.

I think it shouldn't be too bad to move quirk handling mechanism to
arch/arm64. Effectively we would not move platform specific code into
arch/arm64 but just the mechanism checking if there is any quirk that
is defined.

i.e.:

extern struct pci_cfg_fixup __start_acpi_mcfg_fixups[];
extern struct pci_cfg_fixup __end_acpi_mcfg_fixups[];

static struct pci_ecam_ops *pci_acpi_get_ops(struct acpi_pci_root *root)
{
        int bus_num = root->secondary.start;
        int domain = root->segment;
        struct pci_cfg_fixup *f;

        /*
         * Match against platform specific quirks and return corresponding
         * CAM ops.
         *
         * First match against PCI topology <domain:bus> then use DMI or
         * custom match handler.
         */
        for (f = __start_acpi_mcfg_fixups; f < __end_acpi_mcfg_fixups; f++) {
                if ((f->domain == domain || f->domain == PCI_MCFG_DOMAIN_ANY) &&
                    (f->bus_num == bus_num || f->bus_num == PCI_MCFG_BUS_ANY) &&
                    (f->system ? dmi_check_system(f->system) : 1) &&
                    (f->match ? f->match(f, root) : 1))
                        return f->ops;
        }
        /* No quirks, use ECAM */
        return &pci_generic_ecam_ops;
}

Such quirks will be defined anyway in drivers/pci/host/ in the vendor
specific quirk implementations.

e.g. in HiSilicon case we would have

DECLARE_ACPI_MCFG_FIXUP(NULL, hisi_pcie_match, &hisi_pcie_ecam_ops,
			PCI_MCFG_DOMAIN_ANY, PCI_MCFG_BUS_ANY);

in "drivers/pci/host/pcie-hisi-acpi.c "

Thanks

Gab

> 
> Lorenzo

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Sinan Kaya]

On 5/10/2016 11:19 AM, Tomasz Nowicki wrote:
> From the functionality point of view this series may be split into the
> following logic parts:
> 1. New ECAM API and update for users of the pci-host-common API
> 2. Necessary fixes as the preparation for using driver on ARM64.
> 3. Use new MCFG interface and implement generic ACPI based PCI host controller driver.
> 4. Enable above driver on ARM64
> 
> Patches has been built on top of 4.6-rc7 and can be found here:
> git@github.com:semihalf-nowicki-tomasz/linux.git (pci-acpi-v7)
> 
> This has been tested on Cavium ThunderX server. Any help in reviewing and
> testing is very appreciated.
> 

Tested on Qualcomm QDF2XXX server using Mellanox CX3 and Intel e1000e adapters.

Tested-by: Sinan Kaya <okaya@codeaurora.org>



-- 
Sinan Kaya
Qualcomm Technologies, Inc. on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Bjorn Helgaas]

On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:
> Hi Lorenzo
> 
> > -----Original Message-----
> > From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> > Sent: 23 May 2016 11:57
> > To: Ard Biesheuvel
> > Cc: Gabriele Paoloni; Jon Masters; Tomasz Nowicki; helgaas@kernel.org;
> > arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
> > rafael@kernel.org; hanjun.guo@linaro.org; okaya@codeaurora.org;
> > jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
> > pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
> > ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
> > kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> > robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
> > msalter@redhat.com; Wangyijing; mw@semihalf.com;
> > andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
> > Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
> > controller
> > 
> > On Fri, May 20, 2016 at 11:14:03AM +0200, Ard Biesheuvel wrote:
> > > On 20 May 2016 at 10:40, Gabriele Paoloni
> > <gabriele.paoloni@huawei.com> wrote:
> > > > Hi Ard
> > > >
> > > >> -----Original Message-----
> > > >> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> > > [...]
> > > >>
> > > >> Is the PCIe root complex so special that you cannot simply
> > describe an
> > > >> implementation that is not PNP0408 compatible as something else,
> > under
> > > >> its own unique HID? If everybody is onboard with using ACPI, how
> > is
> > > >> this any different from describing other parts of the platform
> > > >> topology? Even if the SBSA mandates generic PCI, they already
> > deviated
> > > >> from that when they built the hardware, so pretending that it is a
> > > >> PNP0408 with quirks really does not buy us anything.
> > > >
> > > > From my understanding we want to avoid this as this would allow
> > each
> > > > vendor to come up with his own code and it would be much more
> > effort
> > > > for the PCI maintainer to rework the PCI framework to accommodate
> > X86
> > > > and "all" ARM64 Host Controllers...
> > > >
> > > > I guess this approach is too risky and we want to avoid this.
> > Through
> > > > standardization we can more easily maintain the code and scale it
> > to
> > > > multiple SoCs...
> > > >
> > > > So this is my understanding; maybe Jon, Tomasz or Lorenzo can give
> > > > a bit more explanation...
> > > >
> > >
> > > OK, so that boils down to recommending to vendors to represent known
> > > non-compliant hardware as compliant, just so that we don't have to
> > > change the code to support additional flavors of ECAM ? It's fine to
> > > be pragmatic, but that sucks.
> > >
> > > We keep confusing the x86 case with the ARM case here: for x86, they
> > > needed to deal with broken hardware *after* the fact, and all they
> > > could do is find /some/ distinguishing feature in order to guess
> > which
> > > exact hardware they might be running on. For arm64, it is the
> > opposite
> > > case. We are currently in a position where we can demand vendors to
> > > comply with the standards they endorsed themselves, and (ab)using
> > ACPI
> > > + DMI as a de facto platform description rather than plain ACPI makes
> > > me think the DT crowd were actually right from the beginning. It
> > > *directly* violates the standardization principle, since it requires
> > a
> > > priori knowledge inside the OS that a certain 'generic' device must
> > be
> > > driven in a special way.
> > >
> > > So can anyone comment on the feasibility of adding support for
> > devices
> > > with vendor specific HIDs (and no generic CIDs) to the current ACPI
> > > ECAM driver in Linux?

I don't think of ECAM support itself as a "driver".  It's just a
service available to drivers, similar to OF resource parsing.

Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
supports extended config space.  It doesn't specify how we access that
config space, so I think hardware with non-standard ECAM should still
have PNP0A03 and PNP0A08 in _CID or _HID.

"ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
r3.2, sec 4.1) means:

  (a) a memory-mapped model for config space access, and
  (b) a specific mapping of address bits to bus/device/function/
      register

MCFG and _CBA assume both (a) and (b), so I think a device with
non-standard ECAM mappings should not be described in MCFG or _CBA.

If a bridge has ECAM with non-standard mappings, I think either a
vendor-specific _HID or a device-specific method, e.g., _DSM, could
communicate that.

Jon, I agree that we should avoid describing non-standardized hardware
in Linux-specific ways.  Is there a mechanism in use already?  How
does Windows handle this?  DMI is a poor long-term solution because it
requires ongoing maintenance for new platforms, but I think it's OK
for getting started with platforms already shipping.

A _DSM has the advantage that once it is defined and supported, OEMs
can ship new platforms without requiring a new quirk or a new _HID to
be added to a driver.

There would still be the problem of config access before the namespace
is available, i.e., the MCFG use case.  I don't know how important
that is.  Defining an MCFG extension seems like the most obvious
solution.

If we only expect a few non-standard devices, maybe it's enough to
have DMI quirks to statically set up ECAM and just live with the
inconvenience of requiring a kernel change for every new non-standard
device.

> > Host bridges in ACPI are handled through PNP0A08/PNP0A03 ids, and
> > most of the arch specific code is handled in the respective arch
> > directories (X86 and IA64, even though IA64 does not rely on ECAM/MCFG
> > for
> > PCI ops), it is not a driver per-se, PNP0A08/PNP0A03 are detected
> > through
> > ACPI scan handlers and the respective arch code (ie pci_acpi_scan_root)
> > sets-up resources AND config space on an arch specific basis.
> > 
> > X86 deals with that with code in arch/x86 that sets-up the pci_raw_ops
> > on a platform specific basis (and it is not nice, but it works because
> > as you all know the number of platforms in X86 world is contained).
> > 
> > Will this happen for ARM64 in arch/arm64 based on vendor specific
> > HIDs ?
> > 
> > No.
> > 
> > So given the current state of play (we were requested to move the
> > arch/arm64 specific ACPI PCI bits to arch/arm64), we would end up
> > with arch/arm64 code requiring code in /drivers to set-up pci_ops
> > in a platform specific way, it is horrible, if feasible at all.
> > 
> > The only way this can be implemented is by pretending that the
> > ACPI/PCI arch/arm64 implementation is generic code (that's what this
> > series does), move it to /drivers (where it is in this series), and
> > implement _DSD vendor specific bindings (per HID) to set-up the pci
> > operations; whether this solution should go upstream, given that it
> > is just a short-term solution for early platforms bugs, it is another
> > story and my personal answer is no.

It seems like there should be a way to look for a _DSM before we call
acpi_pci_root_get_mcfg_addr() to look for _CBA.

Currently we call acpi_pci_root_get_mcfg_addr() (to read _CBA) from
the generic acpi_pci_root_add(), but the result (root->mcfg_addr) is
only used in x86-specific code.  I think it would be nicer if the
lookup and the use were together.  Then it would be easier to override
it because the mapping assumptions would all be in one place.

> I think it shouldn't be too bad to move quirk handling mechanism to
> arch/arm64. Effectively we would not move platform specific code into
> arch/arm64 but just the mechanism checking if there is any quirk that
> is defined.
> 
> i.e.:
> 
> extern struct pci_cfg_fixup __start_acpi_mcfg_fixups[];
> extern struct pci_cfg_fixup __end_acpi_mcfg_fixups[];
> 
> static struct pci_ecam_ops *pci_acpi_get_ops(struct acpi_pci_root *root)
> {
>         int bus_num = root->secondary.start;
>         int domain = root->segment;
>         struct pci_cfg_fixup *f;
> 
>         /*
>          * Match against platform specific quirks and return corresponding
>          * CAM ops.
>          *
>          * First match against PCI topology <domain:bus> then use DMI or
>          * custom match handler.
>          */
>         for (f = __start_acpi_mcfg_fixups; f < __end_acpi_mcfg_fixups; f++) {
>                 if ((f->domain == domain || f->domain == PCI_MCFG_DOMAIN_ANY) &&
>                     (f->bus_num == bus_num || f->bus_num == PCI_MCFG_BUS_ANY) &&
>                     (f->system ? dmi_check_system(f->system) : 1) &&
>                     (f->match ? f->match(f, root) : 1))
>                         return f->ops;
>         }
>         /* No quirks, use ECAM */
>         return &pci_generic_ecam_ops;
> }
> 
> Such quirks will be defined anyway in drivers/pci/host/ in the vendor
> specific quirk implementations.
> 
> e.g. in HiSilicon case we would have
> 
> DECLARE_ACPI_MCFG_FIXUP(NULL, hisi_pcie_match, &hisi_pcie_ecam_ops,
> 			PCI_MCFG_DOMAIN_ANY, PCI_MCFG_BUS_ANY);
> 
> in "drivers/pci/host/pcie-hisi-acpi.c "
> 
> Thanks
> 
> Gab

Sorry Gab, I guess I was really responding to earlier messages :)

I don't really have much to say here, except that it doesn't seem
right to have an MCFG that describes a non-standard ECAM mapping.
I suppose there's already firmware in the field that does that,
though?

Bjorn

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Jon Masters]

Bjorn,

Out walking so sorry about top posting. Quick reply though:

1. I checked with the Windows team. They usually avoid quirks entirely but when it has happened, it has been done via the MCFG/FADT not DSDT.

2. They would be ok if we were to key off the OEM  name and revision for the IP in the MCFG table.

3. I have already verified existing shipping ARMv8 systems provide enough unique data in that entry, and have asked that vendors guarantee to rev it in future IP (which I will verify on models pre tapeout and certainly in early firmware builds). One vendor has a platform that isn't public yet that uses a non-public name in the MCFG but I spoke with them on Friday and they will shortly update their firmware so that a quirk could be posted.

4. I have requested (and Linaro are investigating) that Linaro (with ARM's assistance) begin to drive a separate thread around upstreaming (independent of this core effort) quirks that use the OEM fields in the MCFG as a more scalable approach than one per platform via DMI.

5. I will drive a clarification to the SBBR that does not encourage or endorse quirks but does merely reinforce that data must be unique in such tables. I am driving a separate series of conversations with vendors to ensure that this is the case on all future platforms - though just generally, there is no more high end top shelf "Xeon class" silicon needing common quirks in the pipeline.

More later.

Jon.

-- 
Computer Architect | Sent from my 64-bit #ARM Powered phone

> On May 23, 2016, at 19:39, Bjorn Helgaas <helgaas@kernel.org> wrote:
> 
>> On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:
>> Hi Lorenzo
>> 
>>> -----Original Message-----
>>> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
>>> Sent: 23 May 2016 11:57
>>> To: Ard Biesheuvel
>>> Cc: Gabriele Paoloni; Jon Masters; Tomasz Nowicki; helgaas@kernel.org;
>>> arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
>>> rafael@kernel.org; hanjun.guo@linaro.org; okaya@codeaurora.org;
>>> jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
>>> pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
>>> ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
>>> kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
>>> robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
>>> msalter@redhat.com; Wangyijing; mw@semihalf.com;
>>> andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
>>> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
>>> controller
>>> 
>>>> On Fri, May 20, 2016 at 11:14:03AM +0200, Ard Biesheuvel wrote:
>>>> On 20 May 2016 at 10:40, Gabriele Paoloni
>>> <gabriele.paoloni@huawei.com> wrote:
>>>>> Hi Ard
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
>>>> [...]
>>>>>> 
>>>>>> Is the PCIe root complex so special that you cannot simply
>>> describe an
>>>>>> implementation that is not PNP0408 compatible as something else,
>>> under
>>>>>> its own unique HID? If everybody is onboard with using ACPI, how
>>> is
>>>>>> this any different from describing other parts of the platform
>>>>>> topology? Even if the SBSA mandates generic PCI, they already
>>> deviated
>>>>>> from that when they built the hardware, so pretending that it is a
>>>>>> PNP0408 with quirks really does not buy us anything.
>>>>> 
>>>>> From my understanding we want to avoid this as this would allow
>>> each
>>>>> vendor to come up with his own code and it would be much more
>>> effort
>>>>> for the PCI maintainer to rework the PCI framework to accommodate
>>> X86
>>>>> and "all" ARM64 Host Controllers...
>>>>> 
>>>>> I guess this approach is too risky and we want to avoid this.
>>> Through
>>>>> standardization we can more easily maintain the code and scale it
>>> to
>>>>> multiple SoCs...
>>>>> 
>>>>> So this is my understanding; maybe Jon, Tomasz or Lorenzo can give
>>>>> a bit more explanation...
>>>> 
>>>> OK, so that boils down to recommending to vendors to represent known
>>>> non-compliant hardware as compliant, just so that we don't have to
>>>> change the code to support additional flavors of ECAM ? It's fine to
>>>> be pragmatic, but that sucks.
>>>> 
>>>> We keep confusing the x86 case with the ARM case here: for x86, they
>>>> needed to deal with broken hardware *after* the fact, and all they
>>>> could do is find /some/ distinguishing feature in order to guess
>>> which
>>>> exact hardware they might be running on. For arm64, it is the
>>> opposite
>>>> case. We are currently in a position where we can demand vendors to
>>>> comply with the standards they endorsed themselves, and (ab)using
>>> ACPI
>>>> + DMI as a de facto platform description rather than plain ACPI makes
>>>> me think the DT crowd were actually right from the beginning. It
>>>> *directly* violates the standardization principle, since it requires
>>> a
>>>> priori knowledge inside the OS that a certain 'generic' device must
>>> be
>>>> driven in a special way.
>>>> 
>>>> So can anyone comment on the feasibility of adding support for
>>> devices
>>>> with vendor specific HIDs (and no generic CIDs) to the current ACPI
>>>> ECAM driver in Linux?
> 
> I don't think of ECAM support itself as a "driver".  It's just a
> service available to drivers, similar to OF resource parsing.
> 
> Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
> host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
> supports extended config space.  It doesn't specify how we access that
> config space, so I think hardware with non-standard ECAM should still
> have PNP0A03 and PNP0A08 in _CID or _HID.
> 
> "ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
> r3.2, sec 4.1) means:
> 
>  (a) a memory-mapped model for config space access, and
>  (b) a specific mapping of address bits to bus/device/function/
>      register
> 
> MCFG and _CBA assume both (a) and (b), so I think a device with
> non-standard ECAM mappings should not be described in MCFG or _CBA.
> 
> If a bridge has ECAM with non-standard mappings, I think either a
> vendor-specific _HID or a device-specific method, e.g., _DSM, could
> communicate that.
> 
> Jon, I agree that we should avoid describing non-standardized hardware
> in Linux-specific ways.  Is there a mechanism in use already?  How
> does Windows handle this?  DMI is a poor long-term solution because it
> requires ongoing maintenance for new platforms, but I think it's OK
> for getting started with platforms already shipping.
> 
> A _DSM has the advantage that once it is defined and supported, OEMs
> can ship new platforms without requiring a new quirk or a new _HID to
> be added to a driver.
> 
> There would still be the problem of config access before the namespace
> is available, i.e., the MCFG use case.  I don't know how important
> that is.  Defining an MCFG extension seems like the most obvious
> solution.
> 
> If we only expect a few non-standard devices, maybe it's enough to
> have DMI quirks to statically set up ECAM and just live with the
> inconvenience of requiring a kernel change for every new non-standard
> device.
> 
>>> Host bridges in ACPI are handled through PNP0A08/PNP0A03 ids, and
>>> most of the arch specific code is handled in the respective arch
>>> directories (X86 and IA64, even though IA64 does not rely on ECAM/MCFG
>>> for
>>> PCI ops), it is not a driver per-se, PNP0A08/PNP0A03 are detected
>>> through
>>> ACPI scan handlers and the respective arch code (ie pci_acpi_scan_root)
>>> sets-up resources AND config space on an arch specific basis.
>>> 
>>> X86 deals with that with code in arch/x86 that sets-up the pci_raw_ops
>>> on a platform specific basis (and it is not nice, but it works because
>>> as you all know the number of platforms in X86 world is contained).
>>> 
>>> Will this happen for ARM64 in arch/arm64 based on vendor specific
>>> HIDs ?
>>> 
>>> No.
>>> 
>>> So given the current state of play (we were requested to move the
>>> arch/arm64 specific ACPI PCI bits to arch/arm64), we would end up
>>> with arch/arm64 code requiring code in /drivers to set-up pci_ops
>>> in a platform specific way, it is horrible, if feasible at all.
>>> 
>>> The only way this can be implemented is by pretending that the
>>> ACPI/PCI arch/arm64 implementation is generic code (that's what this
>>> series does), move it to /drivers (where it is in this series), and
>>> implement _DSD vendor specific bindings (per HID) to set-up the pci
>>> operations; whether this solution should go upstream, given that it
>>> is just a short-term solution for early platforms bugs, it is another
>>> story and my personal answer is no.
> 
> It seems like there should be a way to look for a _DSM before we call
> acpi_pci_root_get_mcfg_addr() to look for _CBA.
> 
> Currently we call acpi_pci_root_get_mcfg_addr() (to read _CBA) from
> the generic acpi_pci_root_add(), but the result (root->mcfg_addr) is
> only used in x86-specific code.  I think it would be nicer if the
> lookup and the use were together.  Then it would be easier to override
> it because the mapping assumptions would all be in one place.
> 
>> I think it shouldn't be too bad to move quirk handling mechanism to
>> arch/arm64. Effectively we would not move platform specific code into
>> arch/arm64 but just the mechanism checking if there is any quirk that
>> is defined.
>> 
>> i.e.:
>> 
>> extern struct pci_cfg_fixup __start_acpi_mcfg_fixups[];
>> extern struct pci_cfg_fixup __end_acpi_mcfg_fixups[];
>> 
>> static struct pci_ecam_ops *pci_acpi_get_ops(struct acpi_pci_root *root)
>> {
>>        int bus_num = root->secondary.start;
>>        int domain = root->segment;
>>        struct pci_cfg_fixup *f;
>> 
>>        /*
>>         * Match against platform specific quirks and return corresponding
>>         * CAM ops.
>>         *
>>         * First match against PCI topology <domain:bus> then use DMI or
>>         * custom match handler.
>>         */
>>        for (f = __start_acpi_mcfg_fixups; f < __end_acpi_mcfg_fixups; f++) {
>>                if ((f->domain == domain || f->domain == PCI_MCFG_DOMAIN_ANY) &&
>>                    (f->bus_num == bus_num || f->bus_num == PCI_MCFG_BUS_ANY) &&
>>                    (f->system ? dmi_check_system(f->system) : 1) &&
>>                    (f->match ? f->match(f, root) : 1))
>>                        return f->ops;
>>        }
>>        /* No quirks, use ECAM */
>>        return &pci_generic_ecam_ops;
>> }
>> 
>> Such quirks will be defined anyway in drivers/pci/host/ in the vendor
>> specific quirk implementations.
>> 
>> e.g. in HiSilicon case we would have
>> 
>> DECLARE_ACPI_MCFG_FIXUP(NULL, hisi_pcie_match, &hisi_pcie_ecam_ops,
>>            PCI_MCFG_DOMAIN_ANY, PCI_MCFG_BUS_ANY);
>> 
>> in "drivers/pci/host/pcie-hisi-acpi.c "
>> 
>> Thanks
>> 
>> Gab
> 
> Sorry Gab, I guess I was really responding to earlier messages :)
> 
> I don't really have much to say here, except that it doesn't seem
> right to have an MCFG that describes a non-standard ECAM mapping.
> I suppose there's already firmware in the field that does that,
> though?
> 
> Bjorn

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Jon Masters]

Additional: I would like to thank Ard for suggesting this approach. It turns out (apparently) that Mark Salter's initial X-Gene quirks internal to RH did it this way as well. You great minds think alike. If this works for folks then I hope it leads to upstream kernel support in F25 (we have a bunch of Moonshot hardware we would like to deeply in Fedora but can't without the PCIe network...). I rant only because I care :)

Jon.

-- 
Computer Architect | Sent from my 64-bit #ARM Powered phone

> On May 23, 2016, at 21:11, Jon Masters <jcm@redhat.com> wrote:
> 
> Bjorn,
> 
> Out walking so sorry about top posting. Quick reply though:
> 
> 1. I checked with the Windows team. They usually avoid quirks entirely but when it has happened, it has been done via the MCFG/FADT not DSDT.
> 
> 2. They would be ok if we were to key off the OEM  name and revision for the IP in the MCFG table.
> 
> 3. I have already verified existing shipping ARMv8 systems provide enough unique data in that entry, and have asked that vendors guarantee to rev it in future IP (which I will verify on models pre tapeout and certainly in early firmware builds). One vendor has a platform that isn't public yet that uses a non-public name in the MCFG but I spoke with them on Friday and they will shortly update their firmware so that a quirk could be posted.
> 
> 4. I have requested (and Linaro are investigating) that Linaro (with ARM's assistance) begin to drive a separate thread around upstreaming (independent of this core effort) quirks that use the OEM fields in the MCFG as a more scalable approach than one per platform via DMI.
> 
> 5. I will drive a clarification to the SBBR that does not encourage or endorse quirks but does merely reinforce that data must be unique in such tables. I am driving a separate series of conversations with vendors to ensure that this is the case on all future platforms - though just generally, there is no more high end top shelf "Xeon class" silicon needing common quirks in the pipeline.
> 
> More later.
> 
> Jon.
> 
> -- 
> Computer Architect | Sent from my 64-bit #ARM Powered phone
> 
>>> On May 23, 2016, at 19:39, Bjorn Helgaas <helgaas@kernel.org> wrote:
>>> 
>>> On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:
>>> Hi Lorenzo
>>> 
>>>> -----Original Message-----
>>>> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
>>>> Sent: 23 May 2016 11:57
>>>> To: Ard Biesheuvel
>>>> Cc: Gabriele Paoloni; Jon Masters; Tomasz Nowicki; helgaas@kernel.org;
>>>> arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
>>>> rafael@kernel.org; hanjun.guo@linaro.org; okaya@codeaurora.org;
>>>> jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
>>>> pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
>>>> ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
>>>> kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
>>>> robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
>>>> msalter@redhat.com; Wangyijing; mw@semihalf.com;
>>>> andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
>>>> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
>>>> controller
>>>> 
>>>>> On Fri, May 20, 2016 at 11:14:03AM +0200, Ard Biesheuvel wrote:
>>>>> On 20 May 2016 at 10:40, Gabriele Paoloni
>>>> <gabriele.paoloni@huawei.com> wrote:
>>>>>> Hi Ard
>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
>>>>> [...]
>>>>>>> 
>>>>>>> Is the PCIe root complex so special that you cannot simply
>>>> describe an
>>>>>>> implementation that is not PNP0408 compatible as something else,
>>>> under
>>>>>>> its own unique HID? If everybody is onboard with using ACPI, how
>>>> is
>>>>>>> this any different from describing other parts of the platform
>>>>>>> topology? Even if the SBSA mandates generic PCI, they already
>>>> deviated
>>>>>>> from that when they built the hardware, so pretending that it is a
>>>>>>> PNP0408 with quirks really does not buy us anything.
>>>>>> 
>>>>>> From my understanding we want to avoid this as this would allow
>>>> each
>>>>>> vendor to come up with his own code and it would be much more
>>>> effort
>>>>>> for the PCI maintainer to rework the PCI framework to accommodate
>>>> X86
>>>>>> and "all" ARM64 Host Controllers...
>>>>>> 
>>>>>> I guess this approach is too risky and we want to avoid this.
>>>> Through
>>>>>> standardization we can more easily maintain the code and scale it
>>>> to
>>>>>> multiple SoCs...
>>>>>> 
>>>>>> So this is my understanding; maybe Jon, Tomasz or Lorenzo can give
>>>>>> a bit more explanation...
>>>>> 
>>>>> OK, so that boils down to recommending to vendors to represent known
>>>>> non-compliant hardware as compliant, just so that we don't have to
>>>>> change the code to support additional flavors of ECAM ? It's fine to
>>>>> be pragmatic, but that sucks.
>>>>> 
>>>>> We keep confusing the x86 case with the ARM case here: for x86, they
>>>>> needed to deal with broken hardware *after* the fact, and all they
>>>>> could do is find /some/ distinguishing feature in order to guess
>>>> which
>>>>> exact hardware they might be running on. For arm64, it is the
>>>> opposite
>>>>> case. We are currently in a position where we can demand vendors to
>>>>> comply with the standards they endorsed themselves, and (ab)using
>>>> ACPI
>>>>> + DMI as a de facto platform description rather than plain ACPI makes
>>>>> me think the DT crowd were actually right from the beginning. It
>>>>> *directly* violates the standardization principle, since it requires
>>>> a
>>>>> priori knowledge inside the OS that a certain 'generic' device must
>>>> be
>>>>> driven in a special way.
>>>>> 
>>>>> So can anyone comment on the feasibility of adding support for
>>>> devices
>>>>> with vendor specific HIDs (and no generic CIDs) to the current ACPI
>>>>> ECAM driver in Linux?
>> 
>> I don't think of ECAM support itself as a "driver".  It's just a
>> service available to drivers, similar to OF resource parsing.
>> 
>> Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
>> host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
>> supports extended config space.  It doesn't specify how we access that
>> config space, so I think hardware with non-standard ECAM should still
>> have PNP0A03 and PNP0A08 in _CID or _HID.
>> 
>> "ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
>> r3.2, sec 4.1) means:
>> 
>> (a) a memory-mapped model for config space access, and
>> (b) a specific mapping of address bits to bus/device/function/
>>     register
>> 
>> MCFG and _CBA assume both (a) and (b), so I think a device with
>> non-standard ECAM mappings should not be described in MCFG or _CBA.
>> 
>> If a bridge has ECAM with non-standard mappings, I think either a
>> vendor-specific _HID or a device-specific method, e.g., _DSM, could
>> communicate that.
>> 
>> Jon, I agree that we should avoid describing non-standardized hardware
>> in Linux-specific ways.  Is there a mechanism in use already?  How
>> does Windows handle this?  DMI is a poor long-term solution because it
>> requires ongoing maintenance for new platforms, but I think it's OK
>> for getting started with platforms already shipping.
>> 
>> A _DSM has the advantage that once it is defined and supported, OEMs
>> can ship new platforms without requiring a new quirk or a new _HID to
>> be added to a driver.
>> 
>> There would still be the problem of config access before the namespace
>> is available, i.e., the MCFG use case.  I don't know how important
>> that is.  Defining an MCFG extension seems like the most obvious
>> solution.
>> 
>> If we only expect a few non-standard devices, maybe it's enough to
>> have DMI quirks to statically set up ECAM and just live with the
>> inconvenience of requiring a kernel change for every new non-standard
>> device.
>> 
>>>> Host bridges in ACPI are handled through PNP0A08/PNP0A03 ids, and
>>>> most of the arch specific code is handled in the respective arch
>>>> directories (X86 and IA64, even though IA64 does not rely on ECAM/MCFG
>>>> for
>>>> PCI ops), it is not a driver per-se, PNP0A08/PNP0A03 are detected
>>>> through
>>>> ACPI scan handlers and the respective arch code (ie pci_acpi_scan_root)
>>>> sets-up resources AND config space on an arch specific basis.
>>>> 
>>>> X86 deals with that with code in arch/x86 that sets-up the pci_raw_ops
>>>> on a platform specific basis (and it is not nice, but it works because
>>>> as you all know the number of platforms in X86 world is contained).
>>>> 
>>>> Will this happen for ARM64 in arch/arm64 based on vendor specific
>>>> HIDs ?
>>>> 
>>>> No.
>>>> 
>>>> So given the current state of play (we were requested to move the
>>>> arch/arm64 specific ACPI PCI bits to arch/arm64), we would end up
>>>> with arch/arm64 code requiring code in /drivers to set-up pci_ops
>>>> in a platform specific way, it is horrible, if feasible at all.
>>>> 
>>>> The only way this can be implemented is by pretending that the
>>>> ACPI/PCI arch/arm64 implementation is generic code (that's what this
>>>> series does), move it to /drivers (where it is in this series), and
>>>> implement _DSD vendor specific bindings (per HID) to set-up the pci
>>>> operations; whether this solution should go upstream, given that it
>>>> is just a short-term solution for early platforms bugs, it is another
>>>> story and my personal answer is no.
>> 
>> It seems like there should be a way to look for a _DSM before we call
>> acpi_pci_root_get_mcfg_addr() to look for _CBA.
>> 
>> Currently we call acpi_pci_root_get_mcfg_addr() (to read _CBA) from
>> the generic acpi_pci_root_add(), but the result (root->mcfg_addr) is
>> only used in x86-specific code.  I think it would be nicer if the
>> lookup and the use were together.  Then it would be easier to override
>> it because the mapping assumptions would all be in one place.
>> 
>>> I think it shouldn't be too bad to move quirk handling mechanism to
>>> arch/arm64. Effectively we would not move platform specific code into
>>> arch/arm64 but just the mechanism checking if there is any quirk that
>>> is defined.
>>> 
>>> i.e.:
>>> 
>>> extern struct pci_cfg_fixup __start_acpi_mcfg_fixups[];
>>> extern struct pci_cfg_fixup __end_acpi_mcfg_fixups[];
>>> 
>>> static struct pci_ecam_ops *pci_acpi_get_ops(struct acpi_pci_root *root)
>>> {
>>>       int bus_num = root->secondary.start;
>>>       int domain = root->segment;
>>>       struct pci_cfg_fixup *f;
>>> 
>>>       /*
>>>        * Match against platform specific quirks and return corresponding
>>>        * CAM ops.
>>>        *
>>>        * First match against PCI topology <domain:bus> then use DMI or
>>>        * custom match handler.
>>>        */
>>>       for (f = __start_acpi_mcfg_fixups; f < __end_acpi_mcfg_fixups; f++) {
>>>               if ((f->domain == domain || f->domain == PCI_MCFG_DOMAIN_ANY) &&
>>>                   (f->bus_num == bus_num || f->bus_num == PCI_MCFG_BUS_ANY) &&
>>>                   (f->system ? dmi_check_system(f->system) : 1) &&
>>>                   (f->match ? f->match(f, root) : 1))
>>>                       return f->ops;
>>>       }
>>>       /* No quirks, use ECAM */
>>>       return &pci_generic_ecam_ops;
>>> }
>>> 
>>> Such quirks will be defined anyway in drivers/pci/host/ in the vendor
>>> specific quirk implementations.
>>> 
>>> e.g. in HiSilicon case we would have
>>> 
>>> DECLARE_ACPI_MCFG_FIXUP(NULL, hisi_pcie_match, &hisi_pcie_ecam_ops,
>>>           PCI_MCFG_DOMAIN_ANY, PCI_MCFG_BUS_ANY);
>>> 
>>> in "drivers/pci/host/pcie-hisi-acpi.c "
>>> 
>>> Thanks
>>> 
>>> Gab
>> 
>> Sorry Gab, I guess I was really responding to earlier messages :)
>> 
>> I don't really have much to say here, except that it doesn't seem
>> right to have an MCFG that describes a non-standard ECAM mapping.
>> I suppose there's already firmware in the field that does that,
>> though?
>> 
>> Bjorn

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Jon Masters]

On 05/23/2016 06:56 AM, Lorenzo Pieralisi wrote:

> The only way this can be implemented is by pretending that the
> ACPI/PCI arch/arm64 implementation is generic code (that's what this
> series does), move it to /drivers (where it is in this series), and
> implement _DSD vendor specific bindings (per HID) to set-up the pci
> operations; whether this solution should go upstream, given that it
> is just a short-term solution for early platforms bugs, it is another
> story and my personal answer is no.

Just for completeness, let me also followup to this one.

We have real, shipping, systems in the field based on ARMv8. For
example, HPE Moonshot ProLiant m400. Not everyone loves the first
generation of anything (Applied get a lot of stick, but the reality is
that someone had to go first, and whoever that was was going to learn
all of the lessons that others don't need to) but it is out there and we
need an upstream kernel solution that includes support for that.

In the server world, we (speaking as a major distro vendor here) are not
going to entertain a situation in which non-upstream patches are needed
to even boot a platform. That simply won't do. We need to separate out
the issue of getting the core in place from the quirks, but then we need
quirks that include support for all early ARMv8 platforms that are out
there today. If we can't get to a point where a Moonshot[0] cartridge
boots out of the box with an upstream kernel, let's just give up and do
something else instead :) (joke)

Jon.

[0] HPE have been *amazingly* patient with this stuff. They've reworked
the firmware when someone (cough) pointed out that the early stuff
they'd been fed was not built according to the standards (U-Boot). They
have *really good* UEFI and ACPI enabled firmware that is running
RHEL(SA) great. But that's not good enough. We don't ship a distro with
hacks. We ship a distro derived from upstream code (although we might
have to backport a lot of stuff later). There's wiggle room, but there
is not wiggle room for core platforms. On ARM, users and developers
*will* be able to take an upstream kernel and boot it on their RHEL
install. And they *will* be able to reproduce problems against upstream,
and help develop against upstream, and further the upstream first
mentality in the ARM ecosystem. There will not be "oh well, it runs RHEL
so that's good enough for the early generation...".

-- 
Computer Architect | Sent from my Fedora powered laptop

RE: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Gabriele Paoloni]

Hi Bjorn

> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: 24 May 2016 00:39
> To: Gabriele Paoloni
> Cc: Lorenzo Pieralisi; Ard Biesheuvel; Jon Masters; Tomasz Nowicki;
> arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
> rafael@kernel.org; hanjun.guo@linaro.org; okaya@codeaurora.org;
> jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
> pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
> ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
> kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
> msalter@redhat.com; Wangyijing; mw@semihalf.com;
> andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
> controller
> 
> On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:
> > Hi Lorenzo
> >
> > > -----Original Message-----
> > > From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> > > Sent: 23 May 2016 11:57
> > > To: Ard Biesheuvel
> > > Cc: Gabriele Paoloni; Jon Masters; Tomasz Nowicki;
> helgaas@kernel.org;
> > > arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
> > > rafael@kernel.org; hanjun.guo@linaro.org; okaya@codeaurora.org;
> > > jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
> > > pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
> > > ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
> > > kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> > > robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
> > > msalter@redhat.com; Wangyijing; mw@semihalf.com;
> > > andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
> > > Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI
> host
> > > controller
> > >
> > > On Fri, May 20, 2016 at 11:14:03AM +0200, Ard Biesheuvel wrote:
> > > > On 20 May 2016 at 10:40, Gabriele Paoloni
> > > <gabriele.paoloni@huawei.com> wrote:
> > > > > Hi Ard
> > > > >
> > > > >> -----Original Message-----
> > > > >> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> > > > [...]
> > > > >>
> > > > >> Is the PCIe root complex so special that you cannot simply
> > > describe an
> > > > >> implementation that is not PNP0408 compatible as something
> else,
> > > under
> > > > >> its own unique HID? If everybody is onboard with using ACPI,
> how
> > > is
> > > > >> this any different from describing other parts of the platform
> > > > >> topology? Even if the SBSA mandates generic PCI, they already
> > > deviated
> > > > >> from that when they built the hardware, so pretending that it
> is a
> > > > >> PNP0408 with quirks really does not buy us anything.
> > > > >
> > > > > From my understanding we want to avoid this as this would allow
> > > each
> > > > > vendor to come up with his own code and it would be much more
> > > effort
> > > > > for the PCI maintainer to rework the PCI framework to
> accommodate
> > > X86
> > > > > and "all" ARM64 Host Controllers...
> > > > >
> > > > > I guess this approach is too risky and we want to avoid this.
> > > Through
> > > > > standardization we can more easily maintain the code and scale
> it
> > > to
> > > > > multiple SoCs...
> > > > >
> > > > > So this is my understanding; maybe Jon, Tomasz or Lorenzo can
> give
> > > > > a bit more explanation...
> > > > >
> > > >
> > > > OK, so that boils down to recommending to vendors to represent
> known
> > > > non-compliant hardware as compliant, just so that we don't have
> to
> > > > change the code to support additional flavors of ECAM ? It's fine
> to
> > > > be pragmatic, but that sucks.
> > > >
> > > > We keep confusing the x86 case with the ARM case here: for x86,
> they
> > > > needed to deal with broken hardware *after* the fact, and all
> they
> > > > could do is find /some/ distinguishing feature in order to guess
> > > which
> > > > exact hardware they might be running on. For arm64, it is the
> > > opposite
> > > > case. We are currently in a position where we can demand vendors
> to
> > > > comply with the standards they endorsed themselves, and (ab)using
> > > ACPI
> > > > + DMI as a de facto platform description rather than plain ACPI
> makes
> > > > me think the DT crowd were actually right from the beginning. It
> > > > *directly* violates the standardization principle, since it
> requires
> > > a
> > > > priori knowledge inside the OS that a certain 'generic' device
> must
> > > be
> > > > driven in a special way.
> > > >
> > > > So can anyone comment on the feasibility of adding support for
> > > devices
> > > > with vendor specific HIDs (and no generic CIDs) to the current
> ACPI
> > > > ECAM driver in Linux?
> 
> I don't think of ECAM support itself as a "driver".  It's just a
> service available to drivers, similar to OF resource parsing.
> 
> Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
> host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
> supports extended config space.  It doesn't specify how we access that
> config space, so I think hardware with non-standard ECAM should still
> have PNP0A03 and PNP0A08 in _CID or _HID.
> 
> "ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
> r3.2, sec 4.1) means:
> 
>   (a) a memory-mapped model for config space access, and
>   (b) a specific mapping of address bits to bus/device/function/
>       register
> 
> MCFG and _CBA assume both (a) and (b), so I think a device with
> non-standard ECAM mappings should not be described in MCFG or _CBA.
> 
> If a bridge has ECAM with non-standard mappings, I think either a
> vendor-specific _HID or a device-specific method, e.g., _DSM, could
> communicate that.
> 
> Jon, I agree that we should avoid describing non-standardized hardware
> in Linux-specific ways.  Is there a mechanism in use already?  How
> does Windows handle this?  DMI is a poor long-term solution because it
> requires ongoing maintenance for new platforms, but I think it's OK
> for getting started with platforms already shipping.
> 
> A _DSM has the advantage that once it is defined and supported, OEMs
> can ship new platforms without requiring a new quirk or a new _HID to
> be added to a driver.
> 
> There would still be the problem of config access before the namespace
> is available, i.e., the MCFG use case.  I don't know how important
> that is.  Defining an MCFG extension seems like the most obvious
> solution.
> 
> If we only expect a few non-standard devices, maybe it's enough to
> have DMI quirks to statically set up ECAM and just live with the
> inconvenience of requiring a kernel change for every new non-standard
> device.
> 
> > > Host bridges in ACPI are handled through PNP0A08/PNP0A03 ids, and
> > > most of the arch specific code is handled in the respective arch
> > > directories (X86 and IA64, even though IA64 does not rely on
> ECAM/MCFG
> > > for
> > > PCI ops), it is not a driver per-se, PNP0A08/PNP0A03 are detected
> > > through
> > > ACPI scan handlers and the respective arch code (ie
> pci_acpi_scan_root)
> > > sets-up resources AND config space on an arch specific basis.
> > >
> > > X86 deals with that with code in arch/x86 that sets-up the
> pci_raw_ops
> > > on a platform specific basis (and it is not nice, but it works
> because
> > > as you all know the number of platforms in X86 world is contained).
> > >
> > > Will this happen for ARM64 in arch/arm64 based on vendor specific
> > > HIDs ?
> > >
> > > No.
> > >
> > > So given the current state of play (we were requested to move the
> > > arch/arm64 specific ACPI PCI bits to arch/arm64), we would end up
> > > with arch/arm64 code requiring code in /drivers to set-up pci_ops
> > > in a platform specific way, it is horrible, if feasible at all.
> > >
> > > The only way this can be implemented is by pretending that the
> > > ACPI/PCI arch/arm64 implementation is generic code (that's what
> this
> > > series does), move it to /drivers (where it is in this series), and
> > > implement _DSD vendor specific bindings (per HID) to set-up the pci
> > > operations; whether this solution should go upstream, given that it
> > > is just a short-term solution for early platforms bugs, it is
> another
> > > story and my personal answer is no.
> 
> It seems like there should be a way to look for a _DSM before we call
> acpi_pci_root_get_mcfg_addr() to look for _CBA.
> 
> Currently we call acpi_pci_root_get_mcfg_addr() (to read _CBA) from
> the generic acpi_pci_root_add(), but the result (root->mcfg_addr) is
> only used in x86-specific code.  I think it would be nicer if the
> lookup and the use were together.  Then it would be easier to override
> it because the mapping assumptions would all be in one place.
> 
> > I think it shouldn't be too bad to move quirk handling mechanism to
> > arch/arm64. Effectively we would not move platform specific code into
> > arch/arm64 but just the mechanism checking if there is any quirk that
> > is defined.
> >
> > i.e.:
> >
> > extern struct pci_cfg_fixup __start_acpi_mcfg_fixups[];
> > extern struct pci_cfg_fixup __end_acpi_mcfg_fixups[];
> >
> > static struct pci_ecam_ops *pci_acpi_get_ops(struct acpi_pci_root
> *root)
> > {
> >         int bus_num = root->secondary.start;
> >         int domain = root->segment;
> >         struct pci_cfg_fixup *f;
> >
> >         /*
> >          * Match against platform specific quirks and return
> corresponding
> >          * CAM ops.
> >          *
> >          * First match against PCI topology <domain:bus> then use DMI
> or
> >          * custom match handler.
> >          */
> >         for (f = __start_acpi_mcfg_fixups; f <
> __end_acpi_mcfg_fixups; f++) {
> >                 if ((f->domain == domain || f->domain ==
> PCI_MCFG_DOMAIN_ANY) &&
> >                     (f->bus_num == bus_num || f->bus_num ==
> PCI_MCFG_BUS_ANY) &&
> >                     (f->system ? dmi_check_system(f->system) : 1) &&
> >                     (f->match ? f->match(f, root) : 1))
> >                         return f->ops;
> >         }
> >         /* No quirks, use ECAM */
> >         return &pci_generic_ecam_ops;
> > }
> >
> > Such quirks will be defined anyway in drivers/pci/host/ in the vendor
> > specific quirk implementations.
> >
> > e.g. in HiSilicon case we would have
> >
> > DECLARE_ACPI_MCFG_FIXUP(NULL, hisi_pcie_match, &hisi_pcie_ecam_ops,
> > 			PCI_MCFG_DOMAIN_ANY, PCI_MCFG_BUS_ANY);
> >
> > in "drivers/pci/host/pcie-hisi-acpi.c "
> >
> > Thanks
> >
> > Gab
> 
> Sorry Gab, I guess I was really responding to earlier messages :)
> 
> I don't really have much to say here, except that it doesn't seem
> right to have an MCFG that describes a non-standard ECAM mapping.

The ACPI table that this mechanism relies upon is the one discussed
in:
https://lkml.org/lkml/2016/3/9/91

As you can see MCFG describes ECAM mappings, but we have a motherboard
reserved resource outside the MCFG:
Device (RES0)
{
	Name (_HID, "HISI0081") // HiSi PCIe RC config base address
	Name (_CID, "PNP0C02") // Motherboard reserved resource
	Name (_CRS, ResourceTemplate (){
	Memory32Fixed (ReadWrite, 0xb0080000 , 0x10000)
	})
}

This allows us to retrieve the address we need for accessing
the config space on the RC (that is non-ECAM).

I was thinking that such mechanism could fit different vendors
and allow them define their own quirks without spoiling the 
official and standard MCFG; also from the thread discussion
you seemed quite ok with such solution...?

> I suppose there's already firmware in the field that does that,
> though?

We have "experimental" firmware that is based on the ACPI table
described above, however it is not widely distributed and, obviously,
it is not supported by Linux mainline (so we have room to rework
if we decide that another solution is more appropriate)

Thanks and Regards

Gab

> 
> Bjorn

RE: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Gabriele Paoloni]

Hi Jon

> -----Original Message-----
> From: Jon Masters [mailto:jcm@redhat.com]
> Sent: 24 May 2016 02:11
> To: Bjorn Helgaas
> Cc: Gabriele Paoloni; Lorenzo Pieralisi; Ard Biesheuvel; Tomasz
> Nowicki; arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
> rafael@kernel.org; hanjun.guo@linaro.org; okaya@codeaurora.org;
> jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
> pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
> ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
> kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
> msalter@redhat.com; Wangyijing; mw@semihalf.com;
> andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
> controller
> 
> Bjorn,
> 
> Out walking so sorry about top posting. Quick reply though:
> 
> 1. I checked with the Windows team. They usually avoid quirks entirely
> but when it has happened, it has been done via the MCFG/FADT not DSDT.
> 
> 2. They would be ok if we were to key off the OEM  name and revision
> for the IP in the MCFG table.

I see some problems with this approach:

1) We would need to modify the ACPI specs to accommodate quirks in the MCFG,
   correct?

2) Just adding OEM info would not fit some other people (like us for Designware
   based solutions). In our case for example the addresses defined in the MCFG
   are not compatible with the ones used by the Designware IP, therefore we would
   also need specific quirk data

I think that we can use an approach where we use MCFG entries for the ECAM
address spaces and motherboard reserved resources for those BUSes that are
outside the MCFG table and therefore are non ECAM.

I think it is a more generic approach that would suit anybody and there
is no need to redefine the ACPI specs for MCFG... ?

Thanks

Gab

> 
> 3. I have already verified existing shipping ARMv8 systems provide
> enough unique data in that entry, and have asked that vendors guarantee
> to rev it in future IP (which I will verify on models pre tapeout and
> certainly in early firmware builds). One vendor has a platform that
> isn't public yet that uses a non-public name in the MCFG but I spoke
> with them on Friday and they will shortly update their firmware so that
> a quirk could be posted.
> 
> 4. I have requested (and Linaro are investigating) that Linaro (with
> ARM's assistance) begin to drive a separate thread around upstreaming
> (independent of this core effort) quirks that use the OEM fields in the
> MCFG as a more scalable approach than one per platform via DMI.
> 
> 5. I will drive a clarification to the SBBR that does not encourage or
> endorse quirks but does merely reinforce that data must be unique in
> such tables. I am driving a separate series of conversations with
> vendors to ensure that this is the case on all future platforms -
> though just generally, there is no more high end top shelf "Xeon class"
> silicon needing common quirks in the pipeline.
> 
> More later.
> 
> Jon.
> 
> --
> Computer Architect | Sent from my 64-bit #ARM Powered phone
> 
> > On May 23, 2016, at 19:39, Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> >> On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:
> >> Hi Lorenzo
> >>
> >>> -----Original Message-----
> >>> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> >>> Sent: 23 May 2016 11:57
> >>> To: Ard Biesheuvel
> >>> Cc: Gabriele Paoloni; Jon Masters; Tomasz Nowicki;
> helgaas@kernel.org;
> >>> arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
> >>> rafael@kernel.org; hanjun.guo@linaro.org; okaya@codeaurora.org;
> >>> jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
> >>> pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
> >>> ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
> >>> kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> >>> robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
> >>> msalter@redhat.com; Wangyijing; mw@semihalf.com;
> >>> andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
> >>> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI
> host
> >>> controller
> >>>
> >>>> On Fri, May 20, 2016 at 11:14:03AM +0200, Ard Biesheuvel wrote:
> >>>> On 20 May 2016 at 10:40, Gabriele Paoloni
> >>> <gabriele.paoloni@huawei.com> wrote:
> >>>>> Hi Ard
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Ard Biesheuvel [mailto:ard.biesheuvel@linaro.org]
> >>>> [...]
> >>>>>>
> >>>>>> Is the PCIe root complex so special that you cannot simply
> >>> describe an
> >>>>>> implementation that is not PNP0408 compatible as something else,
> >>> under
> >>>>>> its own unique HID? If everybody is onboard with using ACPI, how
> >>> is
> >>>>>> this any different from describing other parts of the platform
> >>>>>> topology? Even if the SBSA mandates generic PCI, they already
> >>> deviated
> >>>>>> from that when they built the hardware, so pretending that it is
> a
> >>>>>> PNP0408 with quirks really does not buy us anything.
> >>>>>
> >>>>> From my understanding we want to avoid this as this would allow
> >>> each
> >>>>> vendor to come up with his own code and it would be much more
> >>> effort
> >>>>> for the PCI maintainer to rework the PCI framework to accommodate
> >>> X86
> >>>>> and "all" ARM64 Host Controllers...
> >>>>>
> >>>>> I guess this approach is too risky and we want to avoid this.
> >>> Through
> >>>>> standardization we can more easily maintain the code and scale it
> >>> to
> >>>>> multiple SoCs...
> >>>>>
> >>>>> So this is my understanding; maybe Jon, Tomasz or Lorenzo can
> give
> >>>>> a bit more explanation...
> >>>>
> >>>> OK, so that boils down to recommending to vendors to represent
> known
> >>>> non-compliant hardware as compliant, just so that we don't have to
> >>>> change the code to support additional flavors of ECAM ? It's fine
> to
> >>>> be pragmatic, but that sucks.
> >>>>
> >>>> We keep confusing the x86 case with the ARM case here: for x86,
> they
> >>>> needed to deal with broken hardware *after* the fact, and all they
> >>>> could do is find /some/ distinguishing feature in order to guess
> >>> which
> >>>> exact hardware they might be running on. For arm64, it is the
> >>> opposite
> >>>> case. We are currently in a position where we can demand vendors
> to
> >>>> comply with the standards they endorsed themselves, and (ab)using
> >>> ACPI
> >>>> + DMI as a de facto platform description rather than plain ACPI
> makes
> >>>> me think the DT crowd were actually right from the beginning. It
> >>>> *directly* violates the standardization principle, since it
> requires
> >>> a
> >>>> priori knowledge inside the OS that a certain 'generic' device
> must
> >>> be
> >>>> driven in a special way.
> >>>>
> >>>> So can anyone comment on the feasibility of adding support for
> >>> devices
> >>>> with vendor specific HIDs (and no generic CIDs) to the current
> ACPI
> >>>> ECAM driver in Linux?
> >
> > I don't think of ECAM support itself as a "driver".  It's just a
> > service available to drivers, similar to OF resource parsing.
> >
> > Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
> > host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
> > supports extended config space.  It doesn't specify how we access
> that
> > config space, so I think hardware with non-standard ECAM should still
> > have PNP0A03 and PNP0A08 in _CID or _HID.
> >
> > "ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
> > r3.2, sec 4.1) means:
> >
> >  (a) a memory-mapped model for config space access, and
> >  (b) a specific mapping of address bits to bus/device/function/
> >      register
> >
> > MCFG and _CBA assume both (a) and (b), so I think a device with
> > non-standard ECAM mappings should not be described in MCFG or _CBA.
> >
> > If a bridge has ECAM with non-standard mappings, I think either a
> > vendor-specific _HID or a device-specific method, e.g., _DSM, could
> > communicate that.
> >
> > Jon, I agree that we should avoid describing non-standardized
> hardware
> > in Linux-specific ways.  Is there a mechanism in use already?  How
> > does Windows handle this?  DMI is a poor long-term solution because
> it
> > requires ongoing maintenance for new platforms, but I think it's OK
> > for getting started with platforms already shipping.
> >
> > A _DSM has the advantage that once it is defined and supported, OEMs
> > can ship new platforms without requiring a new quirk or a new _HID to
> > be added to a driver.
> >
> > There would still be the problem of config access before the
> namespace
> > is available, i.e., the MCFG use case.  I don't know how important
> > that is.  Defining an MCFG extension seems like the most obvious
> > solution.
> >
> > If we only expect a few non-standard devices, maybe it's enough to
> > have DMI quirks to statically set up ECAM and just live with the
> > inconvenience of requiring a kernel change for every new non-standard
> > device.
> >
> >>> Host bridges in ACPI are handled through PNP0A08/PNP0A03 ids, and
> >>> most of the arch specific code is handled in the respective arch
> >>> directories (X86 and IA64, even though IA64 does not rely on
> ECAM/MCFG
> >>> for
> >>> PCI ops), it is not a driver per-se, PNP0A08/PNP0A03 are detected
> >>> through
> >>> ACPI scan handlers and the respective arch code (ie
> pci_acpi_scan_root)
> >>> sets-up resources AND config space on an arch specific basis.
> >>>
> >>> X86 deals with that with code in arch/x86 that sets-up the
> pci_raw_ops
> >>> on a platform specific basis (and it is not nice, but it works
> because
> >>> as you all know the number of platforms in X86 world is contained).
> >>>
> >>> Will this happen for ARM64 in arch/arm64 based on vendor specific
> >>> HIDs ?
> >>>
> >>> No.
> >>>
> >>> So given the current state of play (we were requested to move the
> >>> arch/arm64 specific ACPI PCI bits to arch/arm64), we would end up
> >>> with arch/arm64 code requiring code in /drivers to set-up pci_ops
> >>> in a platform specific way, it is horrible, if feasible at all.
> >>>
> >>> The only way this can be implemented is by pretending that the
> >>> ACPI/PCI arch/arm64 implementation is generic code (that's what
> this
> >>> series does), move it to /drivers (where it is in this series), and
> >>> implement _DSD vendor specific bindings (per HID) to set-up the pci
> >>> operations; whether this solution should go upstream, given that it
> >>> is just a short-term solution for early platforms bugs, it is
> another
> >>> story and my personal answer is no.
> >
> > It seems like there should be a way to look for a _DSM before we call
> > acpi_pci_root_get_mcfg_addr() to look for _CBA.
> >
> > Currently we call acpi_pci_root_get_mcfg_addr() (to read _CBA) from
> > the generic acpi_pci_root_add(), but the result (root->mcfg_addr) is
> > only used in x86-specific code.  I think it would be nicer if the
> > lookup and the use were together.  Then it would be easier to
> override
> > it because the mapping assumptions would all be in one place.
> >
> >> I think it shouldn't be too bad to move quirk handling mechanism to
> >> arch/arm64. Effectively we would not move platform specific code
> into
> >> arch/arm64 but just the mechanism checking if there is any quirk
> that
> >> is defined.
> >>
> >> i.e.:
> >>
> >> extern struct pci_cfg_fixup __start_acpi_mcfg_fixups[];
> >> extern struct pci_cfg_fixup __end_acpi_mcfg_fixups[];
> >>
> >> static struct pci_ecam_ops *pci_acpi_get_ops(struct acpi_pci_root
> *root)
> >> {
> >>        int bus_num = root->secondary.start;
> >>        int domain = root->segment;
> >>        struct pci_cfg_fixup *f;
> >>
> >>        /*
> >>         * Match against platform specific quirks and return
> corresponding
> >>         * CAM ops.
> >>         *
> >>         * First match against PCI topology <domain:bus> then use DMI
> or
> >>         * custom match handler.
> >>         */
> >>        for (f = __start_acpi_mcfg_fixups; f <
> __end_acpi_mcfg_fixups; f++) {
> >>                if ((f->domain == domain || f->domain ==
> PCI_MCFG_DOMAIN_ANY) &&
> >>                    (f->bus_num == bus_num || f->bus_num ==
> PCI_MCFG_BUS_ANY) &&
> >>                    (f->system ? dmi_check_system(f->system) : 1) &&
> >>                    (f->match ? f->match(f, root) : 1))
> >>                        return f->ops;
> >>        }
> >>        /* No quirks, use ECAM */
> >>        return &pci_generic_ecam_ops;
> >> }
> >>
> >> Such quirks will be defined anyway in drivers/pci/host/ in the
> vendor
> >> specific quirk implementations.
> >>
> >> e.g. in HiSilicon case we would have
> >>
> >> DECLARE_ACPI_MCFG_FIXUP(NULL, hisi_pcie_match, &hisi_pcie_ecam_ops,
> >>            PCI_MCFG_DOMAIN_ANY, PCI_MCFG_BUS_ANY);
> >>
> >> in "drivers/pci/host/pcie-hisi-acpi.c "
> >>
> >> Thanks
> >>
> >> Gab
> >
> > Sorry Gab, I guess I was really responding to earlier messages :)
> >
> > I don't really have much to say here, except that it doesn't seem
> > right to have an MCFG that describes a non-standard ECAM mapping.
> > I suppose there's already firmware in the field that does that,
> > though?
> >
> > Bjorn

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Jon Masters]

Hi Gabriele, all,

On 05/24/2016 03:23 AM, Gabriele Paoloni wrote:

>> Sorry Gab, I guess I was really responding to earlier messages :)
>>
>> I don't really have much to say here, except that it doesn't seem
>> right to have an MCFG that describes a non-standard ECAM mapping.
> 
> The ACPI table that this mechanism relies upon is the one discussed
> in:
> https://lkml.org/lkml/2016/3/9/91
> 
> As you can see MCFG describes ECAM mappings, but we have a motherboard
> reserved resource outside the MCFG:
> Device (RES0)
> {
> 	Name (_HID, "HISI0081") // HiSi PCIe RC config base address
> 	Name (_CID, "PNP0C02") // Motherboard reserved resource
> 	Name (_CRS, ResourceTemplate (){
> 	Memory32Fixed (ReadWrite, 0xb0080000 , 0x10000)
> 	})
> }
> 
> This allows us to retrieve the address we need for accessing
> the config space on the RC (that is non-ECAM).
> 
> I was thinking that such mechanism could fit different vendors
> and allow them define their own quirks without spoiling the 
> official and standard MCFG; also from the thread discussion
> you seemed quite ok with such solution...?

This could have been useful 2-3 years ago (when myself and others first
pulled the fire alarm concerning the lack of upstreaming of the ACPI
enablement for PCIe - which should have been fully upstream before the
first platforms ever even shipped) but at this time we have shipping
platforms that don't have tables built in this way. While we can go back
around to vendors and try to get them to rebuild firmware, it would be
by far preferable to adopt a solution that works with what is already
being deployed in the field today. Such as OEM match in MCFG.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Lorenzo Pieralisi]

Hi Bjorn,

On Mon, May 23, 2016 at 06:39:18PM -0500, Bjorn Helgaas wrote:

[...]

> On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:
> I don't think of ECAM support itself as a "driver".  It's just a
> service available to drivers, similar to OF resource parsing.
> 
> Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
> host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
> supports extended config space.  It doesn't specify how we access that
> config space, so I think hardware with non-standard ECAM should still
> have PNP0A03 and PNP0A08 in _CID or _HID.
> 
> "ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
> r3.2, sec 4.1) means:
> 
>   (a) a memory-mapped model for config space access, and
>   (b) a specific mapping of address bits to bus/device/function/
>       register
> 
> MCFG and _CBA assume both (a) and (b), so I think a device with
> non-standard ECAM mappings should not be described in MCFG or _CBA.
> 
> If a bridge has ECAM with non-standard mappings, I think either a
> vendor-specific _HID or a device-specific method, e.g., _DSM, could
> communicate that.
> 
> Jon, I agree that we should avoid describing non-standardized hardware
> in Linux-specific ways.  Is there a mechanism in use already?  How
> does Windows handle this?  DMI is a poor long-term solution because it
> requires ongoing maintenance for new platforms, but I think it's OK
> for getting started with platforms already shipping.
> 
> A _DSM has the advantage that once it is defined and supported, OEMs
> can ship new platforms without requiring a new quirk or a new _HID to
> be added to a driver.
> 
> There would still be the problem of config access before the namespace
> is available, i.e., the MCFG use case.  I don't know how important
> that is.  Defining an MCFG extension seems like the most obvious
> solution.

Your summary above is a perfect representation of the situation.

We had an opportunity to sync-up on the current status of ACPI PCI
for ARM64 (and talked about a way forward for this series, which
includes quirks handling), let me summarize it here for everyone
involved so that we can agree on a way forward.

1) ACPI PCI support for PNP0A03/PNP0A08 host bridges on top of MCFG
   ECAM for config space is basically ready (Tomasz and JC addressed
   Rafael's concerns in relation to ARM64 specific code, and managed
   to find a way to allocate domain numbers in preparation for Arnd
   pci_create_root_bus() clean-up, v8 to be posted shortly and should
   be final). This provides support for de-facto ACPI/PCI ECAM base
   standard for ARM64 (with a clean-split between generic code and ARM64
   bits, where ARM64, like X86 and IA64, manages in arch code IO space and
   PCI resources, to be further consolidated in the near future).
   I do not think anyone can complain about the generality of what we
   achieved, for systems that are PCI standard (yes, PCI STANDARD) this
   would just be sufficient.
2) In a real world (1) is not enough. Some ARM64 platforms, not entirely
   ECAM compliant, already shipped with the corresponding firmware that
   we can't update. HW has ECAM quirks and to work around it in the kernel
   we put forward many solutions to the problem, it is time we found a
   solution (when, of course, (1) is completed and upstream).
   Using the MCFG table OEMID matching floated around in this thread
   would work fine for most of the platforms (and cross-OS) that have
   shipped with HW ECAM quirks, so I think that's the starting point for
   our solution and that's how we can sort this out, _today_.

   The solution is a trivial look-up table:
   MCFG OEMID <-> PCI config space ops

3) (2) does not just work on some platforms (and we can't predict the
   future either - actually I can, it is three letters, ECAM), simply
   because MCFG OEMID matching does not provide a way to attach further
   data to the MCFG (eg if config space for, say, bus 0 domain 0, is not
   ECAM compliant, the config region can't be handled and must not be
   handled through a corresponding MCFG region.
   That's the problem Gabriele is facing and wants to solve through
   something like:

   https://lkml.org/lkml/2016/3/9/91

   in the respective ACPI tables-bindings. It may be an idea worth
   pursuing, it does not solve (2) simply because that FW has shipped,
   we can't patch it any longer.

Hence to finally support ACPI PCI on ARM64 I suggest we carry out the
following steps, in order:

- Let's complete/merge (1), that's fundamental to this whole thread
- On top of (1) we apply a quirking mechanism based on (2) that allows
  us to boot mainline with boxes shipping today with no FW update required.
- We devise a way to handle quirks that is more generic than (2) so that
  can we can accomodate further platforms that can't rely on (2) but
  have more leeway in terms of FW updates.

I hope that's a reasonable plan, Tomasz's v8 series coming to kick it off.

Thank you,
Lorenzo

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Jon Masters]

A big +1 to the below :) :) :)

-- 
Computer Architect | Sent from my 64-bit #ARM Powered phone

> On May 24, 2016, at 13:24, Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> wrote:
> 
> Hi Bjorn,
> 
> On Mon, May 23, 2016 at 06:39:18PM -0500, Bjorn Helgaas wrote:
> 
> [...]
> 
>> On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:
>> I don't think of ECAM support itself as a "driver".  It's just a
>> service available to drivers, similar to OF resource parsing.
>> 
>> Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
>> host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
>> supports extended config space.  It doesn't specify how we access that
>> config space, so I think hardware with non-standard ECAM should still
>> have PNP0A03 and PNP0A08 in _CID or _HID.
>> 
>> "ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
>> r3.2, sec 4.1) means:
>> 
>>  (a) a memory-mapped model for config space access, and
>>  (b) a specific mapping of address bits to bus/device/function/
>>      register
>> 
>> MCFG and _CBA assume both (a) and (b), so I think a device with
>> non-standard ECAM mappings should not be described in MCFG or _CBA.
>> 
>> If a bridge has ECAM with non-standard mappings, I think either a
>> vendor-specific _HID or a device-specific method, e.g., _DSM, could
>> communicate that.
>> 
>> Jon, I agree that we should avoid describing non-standardized hardware
>> in Linux-specific ways.  Is there a mechanism in use already?  How
>> does Windows handle this?  DMI is a poor long-term solution because it
>> requires ongoing maintenance for new platforms, but I think it's OK
>> for getting started with platforms already shipping.
>> 
>> A _DSM has the advantage that once it is defined and supported, OEMs
>> can ship new platforms without requiring a new quirk or a new _HID to
>> be added to a driver.
>> 
>> There would still be the problem of config access before the namespace
>> is available, i.e., the MCFG use case.  I don't know how important
>> that is.  Defining an MCFG extension seems like the most obvious
>> solution.
> 
> Your summary above is a perfect representation of the situation.
> 
> We had an opportunity to sync-up on the current status of ACPI PCI
> for ARM64 (and talked about a way forward for this series, which
> includes quirks handling), let me summarize it here for everyone
> involved so that we can agree on a way forward.
> 
> 1) ACPI PCI support for PNP0A03/PNP0A08 host bridges on top of MCFG
>   ECAM for config space is basically ready (Tomasz and JC addressed
>   Rafael's concerns in relation to ARM64 specific code, and managed
>   to find a way to allocate domain numbers in preparation for Arnd
>   pci_create_root_bus() clean-up, v8 to be posted shortly and should
>   be final). This provides support for de-facto ACPI/PCI ECAM base
>   standard for ARM64 (with a clean-split between generic code and ARM64
>   bits, where ARM64, like X86 and IA64, manages in arch code IO space and
>   PCI resources, to be further consolidated in the near future).
>   I do not think anyone can complain about the generality of what we
>   achieved, for systems that are PCI standard (yes, PCI STANDARD) this
>   would just be sufficient.
> 2) In a real world (1) is not enough. Some ARM64 platforms, not entirely
>   ECAM compliant, already shipped with the corresponding firmware that
>   we can't update. HW has ECAM quirks and to work around it in the kernel
>   we put forward many solutions to the problem, it is time we found a
>   solution (when, of course, (1) is completed and upstream).
>   Using the MCFG table OEMID matching floated around in this thread
>   would work fine for most of the platforms (and cross-OS) that have
>   shipped with HW ECAM quirks, so I think that's the starting point for
>   our solution and that's how we can sort this out, _today_.
> 
>   The solution is a trivial look-up table:
>   MCFG OEMID <-> PCI config space ops
> 
> 3) (2) does not just work on some platforms (and we can't predict the
>   future either - actually I can, it is three letters, ECAM), simply
>   because MCFG OEMID matching does not provide a way to attach further
>   data to the MCFG (eg if config space for, say, bus 0 domain 0, is not
>   ECAM compliant, the config region can't be handled and must not be
>   handled through a corresponding MCFG region.
>   That's the problem Gabriele is facing and wants to solve through
>   something like:
> 
>   https://lkml.org/lkml/2016/3/9/91
> 
>   in the respective ACPI tables-bindings. It may be an idea worth
>   pursuing, it does not solve (2) simply because that FW has shipped,
>   we can't patch it any longer.
> 
> Hence to finally support ACPI PCI on ARM64 I suggest we carry out the
> following steps, in order:
> 
> - Let's complete/merge (1), that's fundamental to this whole thread
> - On top of (1) we apply a quirking mechanism based on (2) that allows
>  us to boot mainline with boxes shipping today with no FW update required.
> - We devise a way to handle quirks that is more generic than (2) so that
>  can we can accomodate further platforms that can't rely on (2) but
>  have more leeway in terms of FW updates.
> 
> I hope that's a reasonable plan, Tomasz's v8 series coming to kick it off.
> 
> Thank you,
> Lorenzo

Re: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Bjorn Helgaas]

On Tue, May 24, 2016 at 06:24:23PM +0100, Lorenzo Pieralisi wrote:
> Hi Bjorn,
> 
> On Mon, May 23, 2016 at 06:39:18PM -0500, Bjorn Helgaas wrote:
> 
> [...]
> 
> > On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:
> > I don't think of ECAM support itself as a "driver".  It's just a
> > service available to drivers, similar to OF resource parsing.
> > 
> > Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
> > host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
> > supports extended config space.  It doesn't specify how we access that
> > config space, so I think hardware with non-standard ECAM should still
> > have PNP0A03 and PNP0A08 in _CID or _HID.
> > 
> > "ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
> > r3.2, sec 4.1) means:
> > 
> >   (a) a memory-mapped model for config space access, and
> >   (b) a specific mapping of address bits to bus/device/function/
> >       register
> > 
> > MCFG and _CBA assume both (a) and (b), so I think a device with
> > non-standard ECAM mappings should not be described in MCFG or _CBA.
> > 
> > If a bridge has ECAM with non-standard mappings, I think either a
> > vendor-specific _HID or a device-specific method, e.g., _DSM, could
> > communicate that.
> > 
> > Jon, I agree that we should avoid describing non-standardized hardware
> > in Linux-specific ways.  Is there a mechanism in use already?  How
> > does Windows handle this?  DMI is a poor long-term solution because it
> > requires ongoing maintenance for new platforms, but I think it's OK
> > for getting started with platforms already shipping.
> > 
> > A _DSM has the advantage that once it is defined and supported, OEMs
> > can ship new platforms without requiring a new quirk or a new _HID to
> > be added to a driver.
> > 
> > There would still be the problem of config access before the namespace
> > is available, i.e., the MCFG use case.  I don't know how important
> > that is.  Defining an MCFG extension seems like the most obvious
> > solution.
> 
> Your summary above is a perfect representation of the situation.
> 
> We had an opportunity to sync-up on the current status of ACPI PCI
> for ARM64 (and talked about a way forward for this series, which
> includes quirks handling), let me summarize it here for everyone
> involved so that we can agree on a way forward.
> 
> 1) ACPI PCI support for PNP0A03/PNP0A08 host bridges on top of MCFG
>    ECAM for config space is basically ready (Tomasz and JC addressed
>    Rafael's concerns in relation to ARM64 specific code, and managed
>    to find a way to allocate domain numbers in preparation for Arnd
>    pci_create_root_bus() clean-up, v8 to be posted shortly and should
>    be final). This provides support for de-facto ACPI/PCI ECAM base
>    standard for ARM64 (with a clean-split between generic code and ARM64
>    bits, where ARM64, like X86 and IA64, manages in arch code IO space and
>    PCI resources, to be further consolidated in the near future).
>    I do not think anyone can complain about the generality of what we
>    achieved, for systems that are PCI standard (yes, PCI STANDARD) this
>    would just be sufficient.

Sounds good to me.

> 2) In a real world (1) is not enough. Some ARM64 platforms, not entirely
>    ECAM compliant, already shipped with the corresponding firmware that
>    we can't update. HW has ECAM quirks and to work around it in the kernel
>    we put forward many solutions to the problem, it is time we found a
>    solution (when, of course, (1) is completed and upstream).
>    Using the MCFG table OEMID matching floated around in this thread
>    would work fine for most of the platforms (and cross-OS) that have
>    shipped with HW ECAM quirks, so I think that's the starting point for
>    our solution and that's how we can sort this out, _today_.
> 
>    The solution is a trivial look-up table:
>    MCFG OEMID <-> PCI config space ops

Sounds reasonable to me.

> 3) (2) does not just work on some platforms (and we can't predict the
>    future either - actually I can, it is three letters, ECAM), simply
>    because MCFG OEMID matching does not provide a way to attach further
>    data to the MCFG (eg if config space for, say, bus 0 domain 0, is not
>    ECAM compliant, the config region can't be handled and must not be
>    handled through a corresponding MCFG region.

Couldn't this be handled by custom pci_ops that do something special
for bus 0 domain 0, and default to some different pci_ops for the
rest?

>    That's the problem Gabriele is facing and wants to solve through
>    something like:
> 
>    https://lkml.org/lkml/2016/3/9/91
> 
>    in the respective ACPI tables-bindings. It may be an idea worth
>    pursuing, it does not solve (2) simply because that FW has shipped,
>    we can't patch it any longer.

(2) is for quirks to deal with MCFG.  Gabriele's post is a proposal
for ACPI namespace.  We can't use anything in the namespace to
implement MCFG quirks because MCFG is needed before the namespace is
available.

I think Gabriele's post is a good proposal for the namespace, but I
would propose the following modifications:

  - Add PNP0A08 to the PCI1 _CID since this is a PCIe host bridge
  - Add a PCI1 _DSM describing the ECAM space
  - Remove the HISI0081 _HID

The result would be:

  Device (PCI1) {
    Name(_HID, "HISI0080")
    Name(_CID, "PNP0A03,PNP0A08")
    Method(_CRS) { ... }
    Method(_DSM) { <describe ECAM base and mapping function> }
  }
  Device (RES0) {
    Name(_HID, "PNP0C02")
    Name(_CRS) { <describe ECAM base and size> }
  }

RES0 could also be contained within PCI1, as Gabriele suggested.  I
don't really care whether it's contained or not, and making it
contained might make it easier for firmware, because addition/removal
of PCI1 and RES0 should always happen together.

I think the _DSM is important because it is really ugly if the
HISI0080 driver has to look for a separate HISI0081 device to learn
about the ECAM space.  There are several PCI drivers that do something
similar, using for_each_pci_dev(), and I cringe every time I see them,
because this totally screws up the driver model.  A driver should
claim a device via a .probe() method called by the core, and it
shouldn't look at devices it hasn't claimed.  This is required to make
hotplug work correctly.

> Hence to finally support ACPI PCI on ARM64 I suggest we carry out the
> following steps, in order:
> 
> - Let's complete/merge (1), that's fundamental to this whole thread
> - On top of (1) we apply a quirking mechanism based on (2) that allows
>   us to boot mainline with boxes shipping today with no FW update required.
> - We devise a way to handle quirks that is more generic than (2) so that
>   can we can accomodate further platforms that can't rely on (2) but
>   have more leeway in terms of FW updates.
> 
> I hope that's a reasonable plan, Tomasz's v8 series coming to kick it off.

Sounds very good to me; I'm looking forward to v8.

Bjorn

RE: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Gabriele Paoloni]

Hi Lorenzo

> -----Original Message-----
> From: Lorenzo Pieralisi [mailto:lorenzo.pieralisi@arm.com]
> Sent: 24 May 2016 18:24
> To: Bjorn Helgaas
> Cc: Gabriele Paoloni; Ard Biesheuvel; Jon Masters; Tomasz Nowicki;
> arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
> rafael@kernel.org; hanjun.guo@linaro.org; okaya@codeaurora.org;
> jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
> pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
> ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
> kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
> msalter@redhat.com; Wangyijing; mw@semihalf.com;
> andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
> controller
> 
> Hi Bjorn,
> 
> On Mon, May 23, 2016 at 06:39:18PM -0500, Bjorn Helgaas wrote:
> 
> [...]
> 
> > On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:
> > I don't think of ECAM support itself as a "driver".  It's just a
> > service available to drivers, similar to OF resource parsing.
> >
> > Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
> > host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
> > supports extended config space.  It doesn't specify how we access
> that
> > config space, so I think hardware with non-standard ECAM should still
> > have PNP0A03 and PNP0A08 in _CID or _HID.
> >
> > "ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
> > r3.2, sec 4.1) means:
> >
> >   (a) a memory-mapped model for config space access, and
> >   (b) a specific mapping of address bits to bus/device/function/
> >       register
> >
> > MCFG and _CBA assume both (a) and (b), so I think a device with
> > non-standard ECAM mappings should not be described in MCFG or _CBA.
> >
> > If a bridge has ECAM with non-standard mappings, I think either a
> > vendor-specific _HID or a device-specific method, e.g., _DSM, could
> > communicate that.
> >
> > Jon, I agree that we should avoid describing non-standardized
> hardware
> > in Linux-specific ways.  Is there a mechanism in use already?  How
> > does Windows handle this?  DMI is a poor long-term solution because
> it
> > requires ongoing maintenance for new platforms, but I think it's OK
> > for getting started with platforms already shipping.
> >
> > A _DSM has the advantage that once it is defined and supported, OEMs
> > can ship new platforms without requiring a new quirk or a new _HID to
> > be added to a driver.
> >
> > There would still be the problem of config access before the
> namespace
> > is available, i.e., the MCFG use case.  I don't know how important
> > that is.  Defining an MCFG extension seems like the most obvious
> > solution.
> 
> Your summary above is a perfect representation of the situation.
> 
> We had an opportunity to sync-up on the current status of ACPI PCI
> for ARM64 (and talked about a way forward for this series, which
> includes quirks handling), let me summarize it here for everyone
> involved so that we can agree on a way forward.
> 
> 1) ACPI PCI support for PNP0A03/PNP0A08 host bridges on top of MCFG
>    ECAM for config space is basically ready (Tomasz and JC addressed
>    Rafael's concerns in relation to ARM64 specific code, and managed
>    to find a way to allocate domain numbers in preparation for Arnd
>    pci_create_root_bus() clean-up, v8 to be posted shortly and should
>    be final). This provides support for de-facto ACPI/PCI ECAM base
>    standard for ARM64 (with a clean-split between generic code and
> ARM64
>    bits, where ARM64, like X86 and IA64, manages in arch code IO space
> and
>    PCI resources, to be further consolidated in the near future).
>    I do not think anyone can complain about the generality of what we
>    achieved, for systems that are PCI standard (yes, PCI STANDARD) this
>    would just be sufficient.
> 2) In a real world (1) is not enough. Some ARM64 platforms, not
> entirely
>    ECAM compliant, already shipped with the corresponding firmware that
>    we can't update. HW has ECAM quirks and to work around it in the
> kernel
>    we put forward many solutions to the problem, it is time we found a
>    solution (when, of course, (1) is completed and upstream).
>    Using the MCFG table OEMID matching floated around in this thread
>    would work fine for most of the platforms (and cross-OS) that have
>    shipped with HW ECAM quirks, so I think that's the starting point
> for
>    our solution and that's how we can sort this out, _today_.
> 
>    The solution is a trivial look-up table:
>    MCFG OEMID <-> PCI config space ops
> 
> 3) (2) does not just work on some platforms (and we can't predict the
>    future either - actually I can, it is three letters, ECAM), simply
>    because MCFG OEMID matching does not provide a way to attach further
>    data to the MCFG (eg if config space for, say, bus 0 domain 0, is
> not
>    ECAM compliant, the config region can't be handled and must not be
>    handled through a corresponding MCFG region.
>    That's the problem Gabriele is facing and wants to solve through
>    something like:
> 
>    https://lkml.org/lkml/2016/3/9/91
> 
>    in the respective ACPI tables-bindings. It may be an idea worth
>    pursuing, it does not solve (2) simply because that FW has shipped,
>    we can't patch it any longer.
> 
> Hence to finally support ACPI PCI on ARM64 I suggest we carry out the
> following steps, in order:
> 
> - Let's complete/merge (1), that's fundamental to this whole thread
> - On top of (1) we apply a quirking mechanism based on (2) that allows
>   us to boot mainline with boxes shipping today with no FW update
> required.
> - We devise a way to handle quirks that is more generic than (2) so
> that
>   can we can accomodate further platforms that can't rely on (2) but
>   have more leeway in terms of FW updates.
> 
> I hope that's a reasonable plan, Tomasz's v8 series coming to kick it
> off.

Thanks for summarizing.

100% agree on the summary and next steps.

Cheers

Gab


> 
> Thank you,
> Lorenzo

RE: [PATCH V7 00/11] Support for generic ACPI based PCI host controller

[Gabriele Paoloni]

Hi Bjorn many thanks for your suggestions

> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: 24 May 2016 20:01
> To: Lorenzo Pieralisi
> Cc: Gabriele Paoloni; Ard Biesheuvel; Jon Masters; Tomasz Nowicki;
> arnd@arndb.de; will.deacon@arm.com; catalin.marinas@arm.com;
> rafael@kernel.org; hanjun.guo@linaro.org; okaya@codeaurora.org;
> jchandra@broadcom.com; linaro-acpi@lists.linaro.org; linux-
> pci@vger.kernel.org; dhdang@apm.com; Liviu.Dudau@arm.com;
> ddaney@caviumnetworks.com; jeremy.linton@arm.com; linux-
> kernel@vger.kernel.org; linux-acpi@vger.kernel.org;
> robert.richter@caviumnetworks.com; Suravee.Suthikulpanit@amd.com;
> msalter@redhat.com; Wangyijing; mw@semihalf.com;
> andrea.gallo@linaro.org; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH V7 00/11] Support for generic ACPI based PCI host
> controller
> 
> On Tue, May 24, 2016 at 06:24:23PM +0100, Lorenzo Pieralisi wrote:
> > Hi Bjorn,
> >
> > On Mon, May 23, 2016 at 06:39:18PM -0500, Bjorn Helgaas wrote:
> >
> > [...]
> >
> > > On Mon, May 23, 2016 at 03:16:01PM +0000, Gabriele Paoloni wrote:
> > > I don't think of ECAM support itself as a "driver".  It's just a
> > > service available to drivers, similar to OF resource parsing.
> > >
> > > Per PCI Firmware r3.2, sec 4.1.5, "PNP0A03" means a PCI/PCI-X/PCIe
> > > host bridge.  "PNP0A08" means a PCI-X Mode 2 or PCIe bridge that
> > > supports extended config space.  It doesn't specify how we access
> that
> > > config space, so I think hardware with non-standard ECAM should
> still
> > > have PNP0A03 and PNP0A08 in _CID or _HID.
> > >
> > > "ECAM" as used in the specs (PCIe r3.0, sec 7.2.2, and PCI Firmware
> > > r3.2, sec 4.1) means:
> > >
> > >   (a) a memory-mapped model for config space access, and
> > >   (b) a specific mapping of address bits to bus/device/function/
> > >       register
> > >
> > > MCFG and _CBA assume both (a) and (b), so I think a device with
> > > non-standard ECAM mappings should not be described in MCFG or _CBA.
> > >
> > > If a bridge has ECAM with non-standard mappings, I think either a
> > > vendor-specific _HID or a device-specific method, e.g., _DSM, could
> > > communicate that.
> > >
> > > Jon, I agree that we should avoid describing non-standardized
> hardware
> > > in Linux-specific ways.  Is there a mechanism in use already?  How
> > > does Windows handle this?  DMI is a poor long-term solution because
> it
> > > requires ongoing maintenance for new platforms, but I think it's OK
> > > for getting started with platforms already shipping.
> > >
> > > A _DSM has the advantage that once it is defined and supported,
> OEMs
> > > can ship new platforms without requiring a new quirk or a new _HID
> to
> > > be added to a driver.
> > >
> > > There would still be the problem of config access before the
> namespace
> > > is available, i.e., the MCFG use case.  I don't know how important
> > > that is.  Defining an MCFG extension seems like the most obvious
> > > solution.
> >
> > Your summary above is a perfect representation of the situation.
> >
> > We had an opportunity to sync-up on the current status of ACPI PCI
> > for ARM64 (and talked about a way forward for this series, which
> > includes quirks handling), let me summarize it here for everyone
> > involved so that we can agree on a way forward.
> >
> > 1) ACPI PCI support for PNP0A03/PNP0A08 host bridges on top of MCFG
> >    ECAM for config space is basically ready (Tomasz and JC addressed
> >    Rafael's concerns in relation to ARM64 specific code, and managed
> >    to find a way to allocate domain numbers in preparation for Arnd
> >    pci_create_root_bus() clean-up, v8 to be posted shortly and should
> >    be final). This provides support for de-facto ACPI/PCI ECAM base
> >    standard for ARM64 (with a clean-split between generic code and
> ARM64
> >    bits, where ARM64, like X86 and IA64, manages in arch code IO
> space and
> >    PCI resources, to be further consolidated in the near future).
> >    I do not think anyone can complain about the generality of what we
> >    achieved, for systems that are PCI standard (yes, PCI STANDARD)
> this
> >    would just be sufficient.
> 
> Sounds good to me.
> 
> > 2) In a real world (1) is not enough. Some ARM64 platforms, not
> entirely
> >    ECAM compliant, already shipped with the corresponding firmware
> that
> >    we can't update. HW has ECAM quirks and to work around it in the
> kernel
> >    we put forward many solutions to the problem, it is time we found
> a
> >    solution (when, of course, (1) is completed and upstream).
> >    Using the MCFG table OEMID matching floated around in this thread
> >    would work fine for most of the platforms (and cross-OS) that have
> >    shipped with HW ECAM quirks, so I think that's the starting point
> for
> >    our solution and that's how we can sort this out, _today_.
> >
> >    The solution is a trivial look-up table:
> >    MCFG OEMID <-> PCI config space ops
> 
> Sounds reasonable to me.
> 
> > 3) (2) does not just work on some platforms (and we can't predict the
> >    future either - actually I can, it is three letters, ECAM), simply
> >    because MCFG OEMID matching does not provide a way to attach
> further
> >    data to the MCFG (eg if config space for, say, bus 0 domain 0, is
> not
> >    ECAM compliant, the config region can't be handled and must not be
> >    handled through a corresponding MCFG region.
> 
> Couldn't this be handled by custom pci_ops that do something special
> for bus 0 domain 0, and default to some different pci_ops for the
> rest?

My idea was to remove bus 0 from domain 0 MCFG (and also the other
buses corresponding to the root complexes ports)

So the driver would use the ECAM access with addresses retrieved from
MCFG for any devices except the RCs.

For the RC we could retrieve special addresses from the "PNP0C02"
reserved resource... 


> 
> >    That's the problem Gabriele is facing and wants to solve through
> >    something like:
> >
> >    https://lkml.org/lkml/2016/3/9/91
> >
> >    in the respective ACPI tables-bindings. It may be an idea worth
> >    pursuing, it does not solve (2) simply because that FW has
> shipped,
> >    we can't patch it any longer.
> 
> (2) is for quirks to deal with MCFG.  Gabriele's post is a proposal
> for ACPI namespace.  We can't use anything in the namespace to
> implement MCFG quirks because MCFG is needed before the namespace is
> available.

Honestly I have to dig better into this (and I will once V8 is out),
However from my current understanding so far we have look-up where
MCFG OEMID is going to tell which specific pci-ops are going to be used.

Now from my quirk idea I need to retrieve the "special address" from the
ACPI namespace before cfg rd/wr take place...if this is not doable then
I need to find a different solution...than can be the one you proposed
below (many thanks for this).

I will look into details later on once v8 is out.

Thanks

Gab

> 
> I think Gabriele's post is a good proposal for the namespace, but I
> would propose the following modifications:
> 
>   - Add PNP0A08 to the PCI1 _CID since this is a PCIe host bridge
>   - Add a PCI1 _DSM describing the ECAM space
>   - Remove the HISI0081 _HID
> 
> The result would be:
> 
>   Device (PCI1) {
>     Name(_HID, "HISI0080")
>     Name(_CID, "PNP0A03,PNP0A08")
>     Method(_CRS) { ... }
>     Method(_DSM) { <describe ECAM base and mapping function> }
>   }
>   Device (RES0) {
>     Name(_HID, "PNP0C02")
>     Name(_CRS) { <describe ECAM base and size> }
>   }
> 
> RES0 could also be contained within PCI1, as Gabriele suggested.  I
> don't really care whether it's contained or not, and making it
> contained might make it easier for firmware, because addition/removal
> of PCI1 and RES0 should always happen together.
> 
> I think the _DSM is important because it is really ugly if the
> HISI0080 driver has to look for a separate HISI0081 device to learn
> about the ECAM space.  There are several PCI drivers that do something
> similar, using for_each_pci_dev(), and I cringe every time I see them,
> because this totally screws up the driver model.  A driver should
> claim a device via a .probe() method called by the core, and it
> shouldn't look at devices it hasn't claimed.  This is required to make
> hotplug work correctly.
> 
> > Hence to finally support ACPI PCI on ARM64 I suggest we carry out the
> > following steps, in order:
> >
> > - Let's complete/merge (1), that's fundamental to this whole thread
> > - On top of (1) we apply a quirking mechanism based on (2) that
> allows
> >   us to boot mainline with boxes shipping today with no FW update
> required.
> > - We devise a way to handle quirks that is more generic than (2) so
> that
> >   can we can accomodate further platforms that can't rely on (2) but
> >   have more leeway in terms of FW updates.
> >
> > I hope that's a reasonable plan, Tomasz's v8 series coming to kick it
> off.
> 
> Sounds very good to me; I'm looking forward to v8.
> 
> Bjorn