[PATCH 00/14] Rewrite Tegra PCIe driver

[PATCH 00/14] Rewrite Tegra PCIe driver

[Thierry Reding]

This patch series contains an almost complete rewrite of the Tegra PCIe
driver. The code is moved to the drivers/pci/host directory and turned
into a proper platform driver, adding MSI and DT support while at it.
Other PCI host controller drivers can be added to that directory in an
attempt to make it easier to factor out common code.

Patches 1 to 4 add generic OF helpers to parse a PCI node's ranges
property as well as obtain the bus, device and function numbers from a
node's reg property.

Patch 5 provides an implementation of a cache for I/O mappings. This can
be used in situations where a large physical memory region needs to be
ioremap()'ed. On Tegra, the PCIe standard and extended configuration
spaces can be accessed through a 256 MiB window. Mapping that in one
chunk is wasteful and may not always work because the vmalloc area may
not be large enough. The implementation in this patch uses an LRU based
approach to map a limited amount of pages on an as-needed basis.

Patches 6 and 7 prepare the ARM PCI code to enable PCI host controller
drivers to load after the init stage, which can happen due to deferred
probing, and to allow private data to be passed for each controller.

Patches 8 and 9 move some of the Tegra specific code around to allow it
to be used from outside the arch/arm/mach-tegra directory.

Patch 10 moves the Tegra PCIe controller driver to the drivers/pci/host
directory and turns it into a proper platform driver. It also adds MSI
(based on patches by NVIDIA) and DT support.

Patches 11 to 14 add device tree based probing for the TEC, Harmony and
TrimSlice boards.

Thierry

Andrew Murray (1):
  of/pci: Provide support for parsing PCI DT ranges property

Thierry Reding (13):
  of/pci: Add of_pci_get_devfn() function
  of/pci: Add of_pci_get_bus() function
  of/pci: Add of_pci_parse_bus_range() function
  lib: Add I/O map cache implementation
  ARM: pci: Keep pci_common_init() around after init
  ARM: pci: Allow passing per-controller private data
  ARM: tegra: Move tegra_pcie_xclk_clamp() to PMC
  ARM: tegra: Move pmc.h to include/mach
  PCI: tegra: Move PCIe driver to drivers/pci/host
  ARM: tegra: tamonten: Add PCIe support
  ARM: tegra: tec: Add PCIe support
  ARM: tegra: harmony: Initialize PCIe from DT
  ARM: tegra: trimslice: Initialize PCIe from DT

 .../bindings/pci/nvidia,tegra20-pcie.txt           |  115 ++
 arch/arm/Kconfig                                   |    2 +
 arch/arm/boot/dts/tegra20-harmony.dts              |   20 +-
 arch/arm/boot/dts/tegra20-tamonten.dtsi            |    2 +-
 arch/arm/boot/dts/tegra20-tec.dts                  |  198 +++
 arch/arm/boot/dts/tegra20-trimslice.dts            |   12 +
 arch/arm/boot/dts/tegra20.dtsi                     |   45 +
 arch/arm/include/asm/mach/pci.h                    |    1 +
 arch/arm/kernel/bios32.c                           |    9 +-
 arch/arm/mach-tegra/Kconfig                        |    5 -
 arch/arm/mach-tegra/Makefile                       |    3 -
 arch/arm/mach-tegra/board-dt-tegra20.c             |   24 -
 arch/arm/mach-tegra/board-harmony-pcie.c           |   84 --
 arch/arm/mach-tegra/board.h                        |    4 +-
 arch/arm/mach-tegra/common.c                       |    2 +-
 arch/arm/mach-tegra/include/mach/pmc.h             |   24 +
 arch/arm/mach-tegra/iomap.h                        |    3 -
 arch/arm/mach-tegra/pcie.c                         |  887 -------------
 arch/arm/mach-tegra/pmc.c                          |   16 +
 arch/arm/mach-tegra/pmc.h                          |   23 -
 drivers/of/address.c                               |   63 +
 drivers/of/of_pci.c                                |   78 +-
 drivers/pci/Kconfig                                |    2 +
 drivers/pci/Makefile                               |    3 +
 drivers/pci/host/Kconfig                           |    8 +
 drivers/pci/host/Makefile                          |    1 +
 drivers/pci/host/pci-tegra.c                       | 1322 ++++++++++++++++++++
 include/linux/io.h                                 |   12 +
 include/linux/of_address.h                         |    9 +
 include/linux/of_pci.h                             |    3 +
 lib/ioremap.c                                      |  266 ++++
 31 files changed, 2203 insertions(+), 1043 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/pci/nvidia,tegra20-pcie.txt
 delete mode 100644 arch/arm/mach-tegra/board-harmony-pcie.c
 create mode 100644 arch/arm/mach-tegra/include/mach/pmc.h
 delete mode 100644 arch/arm/mach-tegra/pcie.c
 delete mode 100644 arch/arm/mach-tegra/pmc.h
 create mode 100644 drivers/pci/host/Kconfig
 create mode 100644 drivers/pci/host/Makefile
 create mode 100644 drivers/pci/host/pci-tegra.c

-- 
1.8.1

[PATCH 01/14] of/pci: Provide support for parsing PCI DT ranges property

[Thierry Reding]

From: Andrew Murray <andrew.murray@arm.com>

DT bindings for PCI host bridges often use the ranges property to describe
memory and IO ranges - this binding tends to be the same across architectures
yet several parsing implementations exist, e.g. arch/mips/pci/pci.c,
arch/powerpc/kernel/pci-common.c, arch/sparc/kernel/pci.c and
arch/microblaze/pci/pci-common.c (clone of PPC). Some of these duplicate
functionality provided by drivers/of/address.c.

This patch provides a common iterator-based parser for the ranges property, it
is hoped this will reduce DT representation differences between architectures
and that architectures will migrate in part to this new parser.

It is also hoped (and the motativation for the patch) that this patch will
reduce duplication of code when writing host bridge drivers that are supported
by multiple architectures.

This patch provides struct resources from a device tree node, e.g.:

	u32 *last = NULL;
	struct resource res;
	while ((last = of_pci_process_ranges(np, res, last))) {
		//do something with res
	}

Platforms with quirks can then do what they like with the resource or migrate
common quirk handling to the parser. In an ideal world drivers can just request
the obtained resources and pass them on (e.g. pci_add_resource_offset).

Signed-off-by: Andrew Murray <Andrew.Murray@arm.com>
Signed-off-by: Liviu Dudau <Liviu.Dudau@arm.com>
Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 drivers/of/address.c       | 63 ++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/of_address.h |  9 +++++++
 2 files changed, 72 insertions(+)

diff --git a/drivers/of/address.c b/drivers/of/address.c
index 0125524..d659527 100644
--- a/drivers/of/address.c
+++ b/drivers/of/address.c
@@ -13,6 +13,7 @@
 #define OF_CHECK_COUNTS(na, ns)	(OF_CHECK_ADDR_COUNT(na) && (ns) > 0)
 
 static struct of_bus *of_match_bus(struct device_node *np);
+static struct of_bus *of_find_bus(const char *name);
 static int __of_address_to_resource(struct device_node *dev,
 		const __be32 *addrp, u64 size, unsigned int flags,
 		const char *name, struct resource *r);
@@ -227,6 +228,57 @@ int of_pci_address_to_resource(struct device_node *dev, int bar,
 	return __of_address_to_resource(dev, addrp, size, flags, NULL, r);
 }
 EXPORT_SYMBOL_GPL(of_pci_address_to_resource);
+
+const __be32 *of_pci_process_ranges(struct device_node *node,
+				    struct resource *res, const __be32 *from)
+{
+	const __be32 *start, *end;
+	int na, ns, np, pna;
+	int rlen;
+	struct of_bus *bus;
+
+	WARN_ON(!res);
+
+	bus = of_find_bus("pci");
+	bus->count_cells(node, &na, &ns);
+	if (!OF_CHECK_COUNTS(na, ns)) {
+		pr_err("Bad cell count for %s\n", node->full_name);
+		return NULL;
+	}
+
+	pna = of_n_addr_cells(node);
+	np = pna + na + ns;
+
+	start = of_get_property(node, "ranges", &rlen);
+	if (start == NULL)
+		return NULL;
+
+	end = start + rlen / sizeof(__be32);
+
+	if (!from)
+		from = start;
+
+	while (from + np <= end) {
+		u64 cpu_addr, size;
+
+		cpu_addr = of_translate_address(node, from + na);
+		size = of_read_number(from + na + pna, ns);
+		res->flags = bus->get_flags(from);
+		from += np;
+
+		if (cpu_addr == OF_BAD_ADDR || size == 0)
+			continue;
+
+		res->name = node->full_name;
+		res->start = cpu_addr;
+		res->end = res->start + size - 1;
+		res->parent = res->child = res->sibling = NULL;
+		return from;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(of_pci_process_ranges);
 #endif /* CONFIG_PCI */
 
 /*
@@ -337,6 +389,17 @@ static struct of_bus *of_match_bus(struct device_node *np)
 	return NULL;
 }
 
+static struct of_bus *of_find_bus(const char *name)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(of_busses); i++)
+		if (strcmp(name, of_busses[i].name) == 0)
+			return &of_busses[i];
+
+	return NULL;
+}
+
 static int of_translate_one(struct device_node *parent, struct of_bus *bus,
 			    struct of_bus *pbus, __be32 *addr,
 			    int na, int ns, int pna, const char *rprop)
diff --git a/include/linux/of_address.h b/include/linux/of_address.h
index 0506eb5..751e889 100644
--- a/include/linux/of_address.h
+++ b/include/linux/of_address.h
@@ -27,6 +27,8 @@ static inline unsigned long pci_address_to_pio(phys_addr_t addr) { return -1; }
 #define pci_address_to_pio pci_address_to_pio
 #endif
 
+const __be32 *of_pci_process_ranges(struct device_node *node,
+				    struct resource *res, const __be32 *from);
 #else /* CONFIG_OF_ADDRESS */
 #ifndef of_address_to_resource
 static inline int of_address_to_resource(struct device_node *dev, int index,
@@ -53,6 +55,13 @@ static inline const __be32 *of_get_address(struct device_node *dev, int index,
 {
 	return NULL;
 }
+
+static inline const __be32 *of_pci_process_ranges(struct device_node *node,
+						  struct resource *res,
+						  const __be32 *from)
+{
+	return NULL;
+}
 #endif /* CONFIG_OF_ADDRESS */
 
 
-- 
1.8.1

[PATCH 02/14] of/pci: Add of_pci_get_devfn() function

[Thierry Reding]

This function can be used to parse the device and function number from a
standard 5-cell PCI resource. PCI_SLOT() and PCI_FUNC() can be used on
the returned value obtain the device and function numbers respectively.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 drivers/of/of_pci.c    | 32 ++++++++++++++++++++++++++++----
 include/linux/of_pci.h |  1 +
 2 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/drivers/of/of_pci.c b/drivers/of/of_pci.c
index 13e37e2..0dd52df 100644
--- a/drivers/of/of_pci.c
+++ b/drivers/of/of_pci.c
@@ -7,12 +7,13 @@
 static inline int __of_pci_pci_compare(struct device_node *node,
 				       unsigned int devfn)
 {
-	unsigned int size;
-	const __be32 *reg = of_get_property(node, "reg", &size);
+	int err;
 
-	if (!reg || size < 5 * sizeof(__be32))
+	err = of_pci_get_devfn(node);
+	if (err < 0)
 		return 0;
-	return ((be32_to_cpup(&reg[0]) >> 8) & 0xff) == devfn;
+
+	return devfn == err;
 }
 
 struct device_node *of_pci_find_child_device(struct device_node *parent,
@@ -40,3 +41,26 @@ struct device_node *of_pci_find_child_device(struct device_node *parent,
 	return NULL;
 }
 EXPORT_SYMBOL_GPL(of_pci_find_child_device);
+
+/**
+ * of_pci_get_devfn() - Get device and function numbers for a device node
+ * @np: device node
+ *
+ * Parses a standard 5-cell PCI resource and returns an 8-bit value that can
+ * be passed to the PCI_SLOT() and PCI_FUNC() macros to extract the device
+ * and function numbers respectively. On error a negative error code is
+ * returned.
+ */
+int of_pci_get_devfn(struct device_node *np)
+{
+	unsigned int size;
+	const __be32 *reg;
+
+	reg = of_get_property(np, "reg", &size);
+
+	if (!reg || size < 5 * sizeof(__be32))
+		return -EINVAL;
+
+	return (be32_to_cpup(reg) >> 8) & 0xff;
+}
+EXPORT_SYMBOL_GPL(of_pci_get_devfn);
diff --git a/include/linux/of_pci.h b/include/linux/of_pci.h
index bb115de..91ec484 100644
--- a/include/linux/of_pci.h
+++ b/include/linux/of_pci.h
@@ -10,5 +10,6 @@ int of_irq_map_pci(const struct pci_dev *pdev, struct of_irq *out_irq);
 struct device_node;
 struct device_node *of_pci_find_child_device(struct device_node *parent,
 					     unsigned int devfn);
+int of_pci_get_devfn(struct device_node *np);
 
 #endif
-- 
1.8.1

[PATCH 03/14] of/pci: Add of_pci_get_bus() function

[Thierry Reding]

This function can be used to parse the number of a device's parent PCI
bus from a standard 5-cell PCI resource.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 drivers/of/of_pci.c    | 21 +++++++++++++++++++++
 include/linux/of_pci.h |  1 +
 2 files changed, 22 insertions(+)

diff --git a/drivers/of/of_pci.c b/drivers/of/of_pci.c
index 0dd52df..7d5d2c3 100644
--- a/drivers/of/of_pci.c
+++ b/drivers/of/of_pci.c
@@ -43,6 +43,27 @@ struct device_node *of_pci_find_child_device(struct device_node *parent,
 EXPORT_SYMBOL_GPL(of_pci_find_child_device);
 
 /**
+ * of_pci_get_bus() - Get bus number for a device node
+ * @np: device node
+ *
+ * Parses a standard 5-cell PCI resource and returns the 8-bit bus number of
+ * the device's parent PCI bus. On error a negative error code is returned.
+ */
+int of_pci_get_bus(struct device_node *np)
+{
+	unsigned int size;
+	const __be32 *reg;
+
+	reg = of_get_property(np, "reg", &size);
+
+	if (!reg || size < 5 * sizeof(__be32))
+		return -EINVAL;
+
+	return (be32_to_cpup(reg) >> 16) & 0xff;
+}
+EXPORT_SYMBOL_GPL(of_pci_get_bus);
+
+/**
  * of_pci_get_devfn() - Get device and function numbers for a device node
  * @np: device node
  *
diff --git a/include/linux/of_pci.h b/include/linux/of_pci.h
index 91ec484..9118321 100644
--- a/include/linux/of_pci.h
+++ b/include/linux/of_pci.h
@@ -10,6 +10,7 @@ int of_irq_map_pci(const struct pci_dev *pdev, struct of_irq *out_irq);
 struct device_node;
 struct device_node *of_pci_find_child_device(struct device_node *parent,
 					     unsigned int devfn);
+int of_pci_get_bus(struct device_node *np);
 int of_pci_get_devfn(struct device_node *np);
 
 #endif
-- 
1.8.1

[PATCH 04/14] of/pci: Add of_pci_parse_bus_range() function

[Thierry Reding]

This function can be used to parse a bus-range property as specified by
device nodes representing PCI bridges.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 drivers/of/of_pci.c    | 25 +++++++++++++++++++++++++
 include/linux/of_pci.h |  1 +
 2 files changed, 26 insertions(+)

diff --git a/drivers/of/of_pci.c b/drivers/of/of_pci.c
index 7d5d2c3..c311166 100644
--- a/drivers/of/of_pci.c
+++ b/drivers/of/of_pci.c
@@ -85,3 +85,28 @@ int of_pci_get_devfn(struct device_node *np)
 	return (be32_to_cpup(reg) >> 8) & 0xff;
 }
 EXPORT_SYMBOL_GPL(of_pci_get_devfn);
+
+/**
+ * of_pci_parse_bus_range() - parse the bus-range property of a PCI device
+ * @node: device node
+ * @res: address to a struct resource to return the bus-range
+ *
+ * Returns 0 on success or a negative error-code on failure.
+ */
+int of_pci_parse_bus_range(struct device_node *node, struct resource *res)
+{
+	const __be32 *values;
+	int len;
+
+	values = of_get_property(node, "bus-range", &len);
+	if (!values || len < sizeof(*values) * 2)
+		return -EINVAL;
+
+	res->name = node->name;
+	res->start = be32_to_cpup(values++);
+	res->end = be32_to_cpup(values);
+	res->flags = IORESOURCE_BUS;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(of_pci_parse_bus_range);
diff --git a/include/linux/of_pci.h b/include/linux/of_pci.h
index 9118321..fb6e95e 100644
--- a/include/linux/of_pci.h
+++ b/include/linux/of_pci.h
@@ -12,5 +12,6 @@ struct device_node *of_pci_find_child_device(struct device_node *parent,
 					     unsigned int devfn);
 int of_pci_get_bus(struct device_node *np);
 int of_pci_get_devfn(struct device_node *np);
+int of_pci_parse_bus_range(struct device_node *node, struct resource *res);
 
 #endif
-- 
1.8.1

[PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

The I/O map cache is used to map large regions of physical memory in
smaller chunks to avoid running out of vmalloc()/ioremap() space.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 include/linux/io.h |  12 +++
 lib/ioremap.c      | 266 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 278 insertions(+)

diff --git a/include/linux/io.h b/include/linux/io.h
index 069e407..c5d296c 100644
--- a/include/linux/io.h
+++ b/include/linux/io.h
@@ -76,4 +76,16 @@ void devm_ioremap_release(struct device *dev, void *res);
 #define arch_has_dev_port()     (1)
 #endif
 
+struct iomap_cache;
+struct resource;
+
+struct iomap_cache *iomap_cache_create(const struct resource *region);
+void iomap_cache_free(struct iomap_cache *cache);
+void __iomem *iomap_cache_map(struct iomap_cache *cache, unsigned long offset);
+void iomap_cache_unmap(struct iomap_cache *cache, void __iomem *addr);
+
+struct iomap_cache *devm_iomap_cache_create(struct device *dev,
+					    const struct resource *region);
+void devm_iomap_cache_free(struct device *dev, struct iomap_cache *cache);
+
 #endif /* _LINUX_IO_H */
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 0c9216c..8a13d97 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -5,11 +5,16 @@
  *
  * (C) Copyright 1995 1996 Linus Torvalds
  */
+
+#include <linux/device.h>
+#include <linux/err.h>
 #include <linux/vmalloc.h>
 #include <linux/mm.h>
 #include <linux/sched.h>
 #include <linux/io.h>
+#include <linux/ioport.h>
 #include <linux/export.h>
+#include <linux/slab.h>
 #include <asm/cacheflush.h>
 #include <asm/pgtable.h>
 
@@ -92,3 +97,264 @@ int ioremap_page_range(unsigned long addr,
 	return err;
 }
 EXPORT_SYMBOL_GPL(ioremap_page_range);
+
+/**
+ * struct iomap_cache_page - page in an I/O map cache
+ * @region: subregion mapped by the page
+ * @list: chain in cache list
+ * @virt: virtual address of mapped region
+ */
+struct iomap_cache_page {
+	struct resource *region;
+	struct list_head list;
+	void __iomem *virt;
+};
+
+static struct iomap_cache_page *iomap_cache_page_create(void)
+{
+	struct iomap_cache_page *page;
+
+	page = kzalloc(sizeof(*page), GFP_KERNEL);
+	if (!page)
+		return NULL;
+
+	INIT_LIST_HEAD(&page->list);
+
+	return page;
+}
+
+static void iomap_cache_page_unmap(struct iomap_cache_page *page)
+{
+	release_resource(page->region);
+	page->region = NULL;
+
+	iounmap(page->virt);
+	page->virt = NULL;
+}
+
+static void iomap_cache_page_free(struct iomap_cache_page *page)
+{
+	iomap_cache_page_unmap(page);
+	list_del(&page->list);
+	kfree(page);
+}
+
+/**
+ * struct iomap_cache - cache of I/O mapped pages
+ * @region: region mapped by the cache
+ * @pages: list of pages in the cache
+ * @num_pages: number of pages in the cache
+ * @max_pages: maximum number of pages that the cache can map simultaneously
+ */
+struct iomap_cache {
+	struct resource region;
+	struct list_head pages;
+	unsigned int num_pages;
+	unsigned int max_pages;
+};
+
+/**
+ * iomap_cache_create() - create an I/O map cache
+ * @region: memory region to map
+ *
+ * Returns a new I/O map cache that can be used to map the given region on a
+ * page by page basis. On failure, a negative error code is returned.
+ */
+struct iomap_cache *iomap_cache_create(const struct resource *region)
+{
+	struct iomap_cache *cache;
+
+	cache = kzalloc(sizeof(*cache), GFP_KERNEL);
+	if (!cache)
+		return ERR_PTR(-ENOMEM);
+
+	memcpy(&cache->region, region, sizeof(*region));
+	INIT_LIST_HEAD(&cache->pages);
+	cache->num_pages = 0;
+	cache->max_pages = 1;
+
+	return cache;
+}
+
+/**
+ * iomap_cache_free() - free an I/O map cache
+ * @cache: I/O map cache
+ */
+void iomap_cache_free(struct iomap_cache *cache)
+{
+	struct iomap_cache_page *page, *tmp;
+
+	if (!cache)
+		return;
+
+	list_for_each_entry_safe(page, tmp, &cache->pages, list)
+		iomap_cache_page_free(page);
+
+	kfree(cache);
+}
+
+/**
+ * iomap_cache_map() - map a given offset in the cache's region
+ * @cache: I/O map cache
+ * @offset: offset into the cache's region of the address to map
+ *
+ * Returns the virtual address of mapped offset into the cache's region or
+ * NULL if the offset is outside of the region or if not enough memory is
+ * available to map the page.
+ */
+void __iomem *iomap_cache_map(struct iomap_cache *cache, unsigned long offset)
+{
+	struct iomap_cache_page *page;
+	struct resource *region;
+	unsigned long phys;
+
+	if (!cache || offset >= resource_size(&cache->region))
+		return NULL;
+
+	phys = cache->region.start + (offset & PAGE_MASK);
+
+	list_for_each_entry(page, &cache->pages, list) {
+		resource_size_t start, end;
+
+		if (!page->region || !page->virt)
+			continue;
+
+		start = page->region->start - cache->region.start;
+		end = page->region->end - cache->region.start;
+
+		/* address is within an already mapped page */
+		if (offset >= start && offset <= end) {
+			/* move page to end of the LRU list */
+			list_del_init(&page->list);
+			list_add_tail(&page->list, &cache->pages);
+			goto out;
+		}
+	}
+
+	/* find an unmapped page */
+	list_for_each_entry(page, &cache->pages, list) {
+		if (!page->region || !page->virt) {
+			list_del_init(&page->list);
+			break;
+		}
+	}
+
+	/* no unmapped page found */
+	if (&page->list == &cache->pages) {
+		/* add a new page if more space is available */
+		if (cache->num_pages < cache->max_pages) {
+			page = iomap_cache_page_create();
+			if (!page)
+				return NULL;
+
+			cache->num_pages++;
+		} else {
+			/*
+			 * If all pages are in use and there's no space left
+			 * for a new one, evict the first page in the list.
+			 */
+			page = list_first_entry(&cache->pages,
+						struct iomap_cache_page,
+						list);
+			iomap_cache_page_unmap(page);
+			list_del_init(&page->list);
+		}
+	}
+
+	/* insert page at the end of the LRU list */
+	list_add_tail(&page->list, &cache->pages);
+
+	region = __request_region(&cache->region, phys, PAGE_SIZE, NULL,
+				  cache->region.flags);
+	if (!region)
+		return NULL;
+
+	page->virt = ioremap(region->start, resource_size(region));
+	if (!page->virt) {
+		release_resource(region);
+		return NULL;
+	}
+
+	page->region = region;
+
+out:
+	return page->virt + (offset & ~PAGE_MASK);
+}
+
+/**
+ * iomap_cache_unmap() - remove a mapping from the cache
+ * @cache: I/O map cache
+ * @addr: virtual address of the mapping to remove
+ */
+void iomap_cache_unmap(struct iomap_cache *cache, void __iomem *addr)
+{
+	struct iomap_cache_page *page;
+
+	if (!cache)
+		return;
+
+	list_for_each_entry(page, &cache->pages, list) {
+		if (page->virt == addr) {
+			iomap_cache_page_unmap(page);
+			break;
+		}
+	}
+}
+
+static void devm_iomap_cache_release(struct device *dev, void *res)
+{
+	iomap_cache_free(*(struct iomap_cache **)res);
+}
+
+static int devm_iomap_cache_match(struct device *dev, void *res, void *data)
+{
+	struct iomap_cache **p = res;
+
+	if (WARN_ON(!p || !*p))
+		return 0;
+
+	return *p == data;
+}
+
+/**
+ * devm_iomap_cache_create() - create an I/O map cache
+ * @dev: device to attach this I/O map cache to
+ * @region: memory region to map
+ *
+ * Returns a new I/O map cache that can be used to map the given region on a
+ * page by page basis. On failure, a negative error code is returned.
+ *
+ * This function is a device-managed version of iomap_cache_create() which
+ * will automatically be freed when the device disappears.
+ */
+struct iomap_cache *devm_iomap_cache_create(struct device *dev,
+					    const struct resource *region)
+{
+	struct iomap_cache **ptr, *cache;
+
+	ptr = devres_alloc(devm_iomap_cache_release, sizeof(**ptr), GFP_KERNEL);
+	if (!ptr)
+		return ERR_PTR(-ENOMEM);
+
+	cache = iomap_cache_create(region);
+	if (IS_ERR(cache)) {
+		devres_free(ptr);
+		return cache;
+	}
+
+	*ptr = cache;
+	devres_add(dev, ptr);
+
+	return cache;
+}
+
+/**
+ * devm_iomap_cache_free() - free an I/O map cache
+ * @dev: device that this I/O map cached was attached to
+ * @cache: I/O map cache
+ */
+void devm_iomap_cache_free(struct device *dev, struct iomap_cache *cache)
+{
+	WARN_ON(devres_release(dev, devm_iomap_cache_release,
+			       devm_iomap_cache_match, cache));
+}
-- 
1.8.1

[PATCH 06/14] ARM: pci: Keep pci_common_init() around after init

[Thierry Reding]

When using deferred driver probing, PCI host controller drivers may
actually require this function after the init stage.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 arch/arm/kernel/bios32.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm/kernel/bios32.c b/arch/arm/kernel/bios32.c
index 379cf32..da7b0c9 100644
--- a/arch/arm/kernel/bios32.c
+++ b/arch/arm/kernel/bios32.c
@@ -413,7 +413,7 @@ static int pcibios_map_irq(const struct pci_dev *dev, u8 slot, u8 pin)
 	return irq;
 }
 
-static int __init pcibios_init_resources(int busnr, struct pci_sys_data *sys)
+static int pcibios_init_resources(int busnr, struct pci_sys_data *sys)
 {
 	int ret;
 	struct pci_host_bridge_window *window;
@@ -445,7 +445,7 @@ static int __init pcibios_init_resources(int busnr, struct pci_sys_data *sys)
 	return 0;
 }
 
-static void __init pcibios_init_hw(struct hw_pci *hw, struct list_head *head)
+static void pcibios_init_hw(struct hw_pci *hw, struct list_head *head)
 {
 	struct pci_sys_data *sys = NULL;
 	int ret;
@@ -493,7 +493,7 @@ static void __init pcibios_init_hw(struct hw_pci *hw, struct list_head *head)
 	}
 }
 
-void __init pci_common_init(struct hw_pci *hw)
+void pci_common_init(struct hw_pci *hw)
 {
 	struct pci_sys_data *sys;
 	LIST_HEAD(head);
-- 
1.8.1

[PATCH 07/14] ARM: pci: Allow passing per-controller private data

[Thierry Reding]

In order to allow drivers to specify private data for each controller,
this commit adds a private_data field to the struct hw_pci. This field
is an array of nr_controllers pointers that will be used to initialize
the private_data field of the corresponding controller's pci_sys_data
structure.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 arch/arm/include/asm/mach/pci.h | 1 +
 arch/arm/kernel/bios32.c        | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/arch/arm/include/asm/mach/pci.h b/arch/arm/include/asm/mach/pci.h
index db9fedb..5cf2e97 100644
--- a/arch/arm/include/asm/mach/pci.h
+++ b/arch/arm/include/asm/mach/pci.h
@@ -23,6 +23,7 @@ struct hw_pci {
 #endif
 	struct pci_ops	*ops;
 	int		nr_controllers;
+	void		**private_data;
 	int		(*setup)(int nr, struct pci_sys_data *);
 	struct pci_bus *(*scan)(int nr, struct pci_sys_data *);
 	void		(*preinit)(void);
diff --git a/arch/arm/kernel/bios32.c b/arch/arm/kernel/bios32.c
index da7b0c9..a1f73b5 100644
--- a/arch/arm/kernel/bios32.c
+++ b/arch/arm/kernel/bios32.c
@@ -464,6 +464,9 @@ static void pcibios_init_hw(struct hw_pci *hw, struct list_head *head)
 		sys->map_irq = hw->map_irq;
 		INIT_LIST_HEAD(&sys->resources);
 
+		if (hw->private_data)
+			sys->private_data = hw->private_data[nr];
+
 		ret = hw->setup(nr, sys);
 
 		if (ret > 0) {
-- 
1.8.1

[PATCH 08/14] ARM: tegra: Move tegra_pcie_xclk_clamp() to PMC

[Thierry Reding]

The PMC code already accesses to PMC registers so it makes sense to
move this function there as well. While at it, rename the function to
tegra_pmc_pcie_xclk_clamp() for consistency.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
Acked-by: Stephen Warren <swarren@wwwdotorg.org>
---
Changes in v3:
- none

Changes in v2:
- none
---
 arch/arm/mach-tegra/pcie.c | 30 ++++--------------------------
 arch/arm/mach-tegra/pmc.c  | 16 ++++++++++++++++
 arch/arm/mach-tegra/pmc.h  |  1 +
 3 files changed, 21 insertions(+), 26 deletions(-)

diff --git a/arch/arm/mach-tegra/pcie.c b/arch/arm/mach-tegra/pcie.c
index bffcd64..c2d55f8 100644
--- a/arch/arm/mach-tegra/pcie.c
+++ b/arch/arm/mach-tegra/pcie.c
@@ -42,6 +42,7 @@
 
 #include "board.h"
 #include "iomap.h"
+#include "pmc.h"
 
 /* Hack - need to parse this from DT */
 #define INT_PCIE_INTR 130
@@ -148,17 +149,6 @@
 #define  PADS_PLL_CTL_TXCLKREF_DIV10		(0 << 20)
 #define  PADS_PLL_CTL_TXCLKREF_DIV5		(1 << 20)
 
-/* PMC access is required for PCIE xclk (un)clamping */
-#define PMC_SCRATCH42		0x144
-#define PMC_SCRATCH42_PCX_CLAMP	(1 << 0)
-
-static void __iomem *reg_pmc_base = IO_ADDRESS(TEGRA_PMC_BASE);
-
-#define pmc_writel(value, reg) \
-	__raw_writel(value, reg_pmc_base + (reg))
-#define pmc_readl(reg) \
-	__raw_readl(reg_pmc_base + (reg))
-
 /*
  * Tegra2 defines 1GB in the AXI address map for PCIe.
  *
@@ -640,18 +630,6 @@ static int tegra_pcie_enable_controller(void)
 	return 0;
 }
 
-static void tegra_pcie_xclk_clamp(bool clamp)
-{
-	u32 reg;
-
-	reg = pmc_readl(PMC_SCRATCH42) & ~PMC_SCRATCH42_PCX_CLAMP;
-
-	if (clamp)
-		reg |= PMC_SCRATCH42_PCX_CLAMP;
-
-	pmc_writel(reg, PMC_SCRATCH42);
-}
-
 static void tegra_pcie_power_off(void)
 {
 	tegra_periph_reset_assert(tegra_pcie.pcie_xclk);
@@ -659,7 +637,7 @@ static void tegra_pcie_power_off(void)
 	tegra_periph_reset_assert(tegra_pcie.pex_clk);
 
 	tegra_powergate_power_off(TEGRA_POWERGATE_PCIE);
-	tegra_pcie_xclk_clamp(true);
+	tegra_pmc_pcie_xclk_clamp(true);
 }
 
 static int tegra_pcie_power_regate(void)
@@ -668,7 +646,7 @@ static int tegra_pcie_power_regate(void)
 
 	tegra_pcie_power_off();
 
-	tegra_pcie_xclk_clamp(true);
+	tegra_pmc_pcie_xclk_clamp(true);
 
 	tegra_periph_reset_assert(tegra_pcie.pcie_xclk);
 	tegra_periph_reset_assert(tegra_pcie.afi_clk);
@@ -682,7 +660,7 @@ static int tegra_pcie_power_regate(void)
 
 	tegra_periph_reset_deassert(tegra_pcie.afi_clk);
 
-	tegra_pcie_xclk_clamp(false);
+	tegra_pmc_pcie_xclk_clamp(false);
 
 	clk_prepare_enable(tegra_pcie.afi_clk);
 	clk_prepare_enable(tegra_pcie.pex_clk);
diff --git a/arch/arm/mach-tegra/pmc.c b/arch/arm/mach-tegra/pmc.c
index d4fdb5f..a335a93 100644
--- a/arch/arm/mach-tegra/pmc.c
+++ b/arch/arm/mach-tegra/pmc.c
@@ -24,6 +24,10 @@
 #define PMC_CTRL		0x0
 #define PMC_CTRL_INTR_LOW	(1 << 17)
 
+/* PMC access is required for PCIE xclk (un)clamping */
+#define PMC_SCRATCH42		0x144
+#define PMC_SCRATCH42_PCX_CLAMP	(1 << 0)
+
 static inline u32 tegra_pmc_readl(u32 reg)
 {
 	return readl(IO_ADDRESS(TEGRA_PMC_BASE + reg));
@@ -74,3 +78,15 @@ void __init tegra_pmc_init(void)
 		val &= ~PMC_CTRL_INTR_LOW;
 	tegra_pmc_writel(val, PMC_CTRL);
 }
+
+void tegra_pmc_pcie_xclk_clamp(bool clamp)
+{
+	u32 reg;
+
+	reg = tegra_pmc_readl(PMC_SCRATCH42) & ~PMC_SCRATCH42_PCX_CLAMP;
+
+	if (clamp)
+		reg |= PMC_SCRATCH42_PCX_CLAMP;
+
+	tegra_pmc_writel(reg, PMC_SCRATCH42);
+}
diff --git a/arch/arm/mach-tegra/pmc.h b/arch/arm/mach-tegra/pmc.h
index 8995ee4..2631c9a 100644
--- a/arch/arm/mach-tegra/pmc.h
+++ b/arch/arm/mach-tegra/pmc.h
@@ -19,5 +19,6 @@
 #define __MACH_TEGRA_PMC_H
 
 void tegra_pmc_init(void);
+void tegra_pmc_pcie_xclk_clamp(bool clamp);
 
 #endif
-- 
1.8.1

[PATCH 09/14] ARM: tegra: Move pmc.h to include/mach

[Thierry Reding]

In preparation for moving the PCIe driver into the drivers/pci/host
directory, this header, which contains prototypes that are required by
the PCIe driver, needs to be moved to a globally visible location.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
Note that eventually the code in pmc.c and powergate.c should probably
be split out into a separate driver. The PMC registers are also directly
accessed from tegra20_clocks.c and tegra30_clocks.c, so that it might be
required to provide that functionality through the new driver as well.
---
 arch/arm/mach-tegra/common.c           |  2 +-
 arch/arm/mach-tegra/include/mach/pmc.h | 24 ++++++++++++++++++++++++
 arch/arm/mach-tegra/pmc.h              | 24 ------------------------
 3 files changed, 25 insertions(+), 25 deletions(-)
 create mode 100644 arch/arm/mach-tegra/include/mach/pmc.h
 delete mode 100644 arch/arm/mach-tegra/pmc.h

diff --git a/arch/arm/mach-tegra/common.c b/arch/arm/mach-tegra/common.c
index 3efe80b..2498f74 100644
--- a/arch/arm/mach-tegra/common.c
+++ b/arch/arm/mach-tegra/common.c
@@ -26,6 +26,7 @@
 #include <asm/hardware/cache-l2x0.h>
 #include <asm/hardware/gic.h>
 
+#include <mach/pmc.h>
 #include <mach/powergate.h>
 
 #include "board.h"
@@ -33,7 +34,6 @@
 #include "common.h"
 #include "fuse.h"
 #include "iomap.h"
-#include "pmc.h"
 #include "apbio.h"
 #include "sleep.h"
 #include "pm.h"
diff --git a/arch/arm/mach-tegra/include/mach/pmc.h b/arch/arm/mach-tegra/include/mach/pmc.h
new file mode 100644
index 0000000..2631c9a
--- /dev/null
+++ b/arch/arm/mach-tegra/include/mach/pmc.h
@@ -0,0 +1,24 @@
+/*
+ * Copyright (C) 2012 NVIDIA CORPORATION. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#ifndef __MACH_TEGRA_PMC_H
+#define __MACH_TEGRA_PMC_H
+
+void tegra_pmc_init(void);
+void tegra_pmc_pcie_xclk_clamp(bool clamp);
+
+#endif
diff --git a/arch/arm/mach-tegra/pmc.h b/arch/arm/mach-tegra/pmc.h
deleted file mode 100644
index 2631c9a..0000000
--- a/arch/arm/mach-tegra/pmc.h
+++ /dev/null
@@ -1,24 +0,0 @@
-/*
- * Copyright (C) 2012 NVIDIA CORPORATION. All rights reserved.
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms and conditions of the GNU General Public License,
- * version 2, as published by the Free Software Foundation.
- *
- * This program is distributed in the hope it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
- * more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.  If not, see <http://www.gnu.org/licenses/>.
- *
- */
-
-#ifndef __MACH_TEGRA_PMC_H
-#define __MACH_TEGRA_PMC_H
-
-void tegra_pmc_init(void);
-void tegra_pmc_pcie_xclk_clamp(bool clamp);
-
-#endif
-- 
1.8.1

[PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

Move the PCIe driver from arch/arm/mach-tegra into the drivers/pci/host
directory. The motivation is to collect various host controller drivers
in the same location in order to facilitate refactoring.

The Tegra PCIe driver has been largely rewritten, both in order to turn
it into a proper platform driver and to add MSI (based on code by
Krishna Kishore <kthota@nvidia.com>) as well as device tree support.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 .../bindings/pci/nvidia,tegra20-pcie.txt           |  115 ++
 arch/arm/Kconfig                                   |    2 +
 arch/arm/boot/dts/tegra20.dtsi                     |   45 +
 arch/arm/mach-tegra/Kconfig                        |    5 -
 arch/arm/mach-tegra/Makefile                       |    1 -
 arch/arm/mach-tegra/board-dt-tegra20.c             |    6 +-
 arch/arm/mach-tegra/board-harmony-pcie.c           |   26 +-
 arch/arm/mach-tegra/board.h                        |    4 +-
 arch/arm/mach-tegra/iomap.h                        |    3 -
 arch/arm/mach-tegra/pcie.c                         |  865 -------------
 drivers/pci/Kconfig                                |    2 +
 drivers/pci/Makefile                               |    3 +
 drivers/pci/host/Kconfig                           |    8 +
 drivers/pci/host/Makefile                          |    1 +
 drivers/pci/host/pci-tegra.c                       | 1322 ++++++++++++++++++++
 15 files changed, 1517 insertions(+), 891 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/pci/nvidia,tegra20-pcie.txt
 delete mode 100644 arch/arm/mach-tegra/pcie.c
 create mode 100644 drivers/pci/host/Kconfig
 create mode 100644 drivers/pci/host/Makefile
 create mode 100644 drivers/pci/host/pci-tegra.c

diff --git a/Documentation/devicetree/bindings/pci/nvidia,tegra20-pcie.txt b/Documentation/devicetree/bindings/pci/nvidia,tegra20-pcie.txt
new file mode 100644
index 0000000..1ebc526
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/nvidia,tegra20-pcie.txt
@@ -0,0 +1,115 @@
+NVIDIA Tegra PCIe controller
+
+Required properties:
+- compatible: "nvidia,tegra20-pcie"
+- reg: physical base address and length of the controller's registers
+- interrupts: the interrupt outputs of the controller
+- pex-clk-supply: supply voltage for internal reference clock
+- vdd-supply: power supply for controller (1.05V)
+- ranges: describes the translation of addresses for root ports
+- bus-range: range of bus numbers associated with this controller
+- #address-cells: address representation for root ports (must be 3)
+  - cell 0 specifies the bus and device numbers of the root port:
+    [23:16]: bus number
+    [15:11]: device number
+  - cell 1 denotes the upper 32 address bits and should be 0
+  - cell 2 contains the lower 32 address bits and is used to translate to the
+    CPU address space
+- #size-cells: size representation for root ports (must be 2)
+
+Root ports are defined as subnodes of the PCIe controller node.
+
+Required properties:
+- device_type: must be "pciex"
+- reg: address and size of the port configuration registers
+- #address-cells: must be 3
+- #size-cells: must be 2
+- ranges: sub-ranges distributed from the PCIe controller node
+- nvidia,num-lanes: number of lanes to use for this port
+
+Example:
+
+SoC DTSI:
+
+	pcie-controller {
+		compatible = "nvidia,tegra20-pcie";
+		reg = <0x80003000 0x00000800   /* PADS registers */
+		       0x80003800 0x00000200   /* AFI registers */
+		       0x81000000 0x01000000   /* configuration space */
+		       0x90000000 0x10000000>; /* extended configuration space */
+		interrupts = <0 98 0x04   /* controller interrupt */
+		              0 99 0x04>; /* MSI interrupt */
+		status = "disabled";
+
+		ranges = <0x000800 0 0x80000000 0x80000000 0 0x00001000   /* port 0 configuration space */
+			  0x000800 0 0x82000000 0x82000000 0 0x00010000   /* port 0 downstream I/O */
+			  0x000800 0 0xa0000000 0xa0000000 0 0x08000000   /* port 0 non-prefetchable memory */
+			  0x000800 0 0xb0000000 0xb0000000 0 0x08000000   /* port 0 prefetchable memory */
+
+			  0x001000 0 0x80001000 0x80001000 0 0x00001000   /* port 1 configuration space */
+			  0x001000 0 0x82010000 0x82010000 0 0x00010000   /* port 1 downstream I/O */
+			  0x001000 0 0xa8000000 0xa8000000 0 0x08000000   /* port 1 non-prefetchable memory */
+			  0x001000 0 0xb8000000 0xb8000000 0 0x08000000>; /* port 1 prefetchable memory */
+
+		bus-range = <0x00 0xff>;
+		#address-cells = <3>;
+		#size-cells = <2>;
+
+		pci@1,0 {
+			device_type = "pciex";
+			reg = <0x000800 0 0x80000000 0 0x1000>;
+			status = "disabled";
+
+			#address-cells = <3>;
+			#size-cells = <2>;
+
+			ranges = <0x81000000 0 0 0x000800 0 0x82000000 0 0x00010000   /* I/O */
+				  0x82000000 0 0 0x000800 0 0xa0000000 0 0x08000000   /* non-prefetchable memory */
+				  0xc2000000 0 0 0x000800 0 0xb0000000 0 0x08000000>; /* prefetchable memory */
+
+			nvidia,num-lanes = <2>;
+		};
+
+		pci@2,0 {
+			device_type = "pciex";
+			reg = <0x0010000 0 0x80001000 0 0x1000>;
+			status = "disabled";
+
+			#address-cells = <3>;
+			#size-cells = <2>;
+
+			ranges = <0x81000000 0 0 0x001000 0 0x82010000 0 0x00010000   /* I/O */
+				  0x82000000 0 0 0x001000 0 0xa8000000 0 0x08000000   /* non-prefetchable memory */
+				  0xc2000000 0 0 0x001000 0 0xb8000000 0 0x08000000>; /* prefetchable memory */
+
+			nvidia,num-lanes = <2>;
+		};
+	};
+
+Board DTS:
+
+	pcie-controller {
+		vdd-supply = <&pci_vdd_reg>;
+		pex-clk-supply = <&pci_clk_reg>;
+		status = "okay";
+
+		/* root port 00:01.0 */
+		pci@1,0 {
+			status = "okay";
+
+			/* bridge 01:00.0 */
+			pci@0,0 {
+				reg = <0x010000 0 0 0 0>;
+
+				#address-cells = <3>;
+				#size-cells = <2>;
+
+				device_type = "pci";
+
+				/* endpoint 02:00.0 */
+				pci@0,0 {
+					reg = <0x020000 0 0 0 0>;
+				};
+			};
+		};
+	};
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 9862590..8853fef 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -653,6 +653,8 @@ config ARCH_TEGRA
 	select MIGHT_HAVE_CACHE_L2X0
 	select SPARSE_IRQ
 	select USE_OF
+	select MIGHT_HAVE_PCI
+	select ARCH_SUPPORTS_MSI
 	help
 	  This enables support for NVIDIA Tegra based systems (Tegra APX,
 	  Tegra 6xx and Tegra 2 series).
diff --git a/arch/arm/boot/dts/tegra20.dtsi b/arch/arm/boot/dts/tegra20.dtsi
index 37973e6..1d30bf3 100644
--- a/arch/arm/boot/dts/tegra20.dtsi
+++ b/arch/arm/boot/dts/tegra20.dtsi
@@ -386,6 +386,51 @@
 		#size-cells = <0>;
 	};
 
+	pcie-controller {
+		compatible = "nvidia,tegra20-pcie";
+		reg = <0x80003000 0x00000800   /* PADS registers */
+		       0x80003800 0x00000200   /* AFI registers */
+		       0x90000000 0x10000000>; /* configuration space */
+		reg-names = "pads", "afi", "cs";
+		interrupts = <0 98 0x04   /* controller interrupt */
+		              0 99 0x04>; /* MSI interrupt */
+		interrupt-names = "intr", "msi";
+
+		bus-range = <0x00 0xff>;
+		#address-cells = <3>;
+		#size-cells = <2>;
+
+		ranges = <0x00000800 0 0x80000000 0x80000000 0 0x00001000   /* port 0 registers */
+			  0x00001000 0 0x80001000 0x80001000 0 0x00001000   /* port 1 registers */
+			  0x81000000 0 0          0x82000000 0 0x00010000   /* downstream I/O */
+			  0x82000000 0 0          0xa0000000 0 0x10000000   /* non-prefetchable memory */
+			  0xc2000000 0 0          0xb0000000 0 0x10000000>; /* prefetchable memory */
+
+		status = "disabled";
+
+		pci@1,0 {
+			device_type = "pciex";
+			reg = <0x000800 0 0x80000000 0 0x1000>;
+			status = "disabled";
+
+			#address-cells = <3>;
+			#size-cells = <2>;
+
+			nvidia,num-lanes = <2>;
+		};
+
+		pci@2,0 {
+			device_type = "pciex";
+			reg = <0x001000 0 0x80001000 0 0x1000>;
+			status = "disabled";
+
+			#address-cells = <3>;
+			#size-cells = <2>;
+
+			nvidia,num-lanes = <2>;
+		};
+	};
+
 	usb@c5000000 {
 		compatible = "nvidia,tegra20-ehci", "usb-ehci";
 		reg = <0xc5000000 0x4000>;
diff --git a/arch/arm/mach-tegra/Kconfig b/arch/arm/mach-tegra/Kconfig
index 1ec7f80..79aa3bf 100644
--- a/arch/arm/mach-tegra/Kconfig
+++ b/arch/arm/mach-tegra/Kconfig
@@ -44,11 +44,6 @@ config ARCH_TEGRA_3x_SOC
 	  Support for NVIDIA Tegra T30 processor family, based on the
 	  ARM CortexA9MP CPU and the ARM PL310 L2 cache controller
 
-config TEGRA_PCI
-	bool "PCI Express support"
-	depends on ARCH_TEGRA_2x_SOC
-	select PCI
-
 config TEGRA_AHB
 	bool "Enable AHB driver for NVIDIA Tegra SoCs"
 	default y
diff --git a/arch/arm/mach-tegra/Makefile b/arch/arm/mach-tegra/Makefile
index 660ee03..463a77e 100644
--- a/arch/arm/mach-tegra/Makefile
+++ b/arch/arm/mach-tegra/Makefile
@@ -29,7 +29,6 @@ endif
 obj-$(CONFIG_SMP)			+= platsmp.o headsmp.o
 obj-$(CONFIG_HOTPLUG_CPU)               += hotplug.o
 obj-$(CONFIG_CPU_FREQ)                  += cpu-tegra.o
-obj-$(CONFIG_TEGRA_PCI)			+= pcie.o
 
 obj-$(CONFIG_ARCH_TEGRA_2x_SOC)		+= board-dt-tegra20.o
 obj-$(CONFIG_ARCH_TEGRA_3x_SOC)		+= board-dt-tegra30.o
diff --git a/arch/arm/mach-tegra/board-dt-tegra20.c b/arch/arm/mach-tegra/board-dt-tegra20.c
index 5d28db3..5644dfb 100644
--- a/arch/arm/mach-tegra/board-dt-tegra20.c
+++ b/arch/arm/mach-tegra/board-dt-tegra20.c
@@ -148,11 +148,7 @@ static void __init tegra_dt_init(void)
 static void __init trimslice_init(void)
 {
 #ifdef CONFIG_TEGRA_PCI
-	int ret;
-
-	ret = tegra_pcie_init(true, true);
-	if (ret)
-		pr_err("tegra_pci_init() failed: %d\n", ret);
+	platform_device_register(&tegra_pcie_device);
 #endif
 }
 
diff --git a/arch/arm/mach-tegra/board-harmony-pcie.c b/arch/arm/mach-tegra/board-harmony-pcie.c
index 3cdc1bb..94c0529 100644
--- a/arch/arm/mach-tegra/board-harmony-pcie.c
+++ b/arch/arm/mach-tegra/board-harmony-pcie.c
@@ -23,11 +23,12 @@
 
 #include <asm/mach-types.h>
 
+#include <mach/pci-tegra.h>
+
 #include "board.h"
 
 #ifdef CONFIG_TEGRA_PCI
-
-int __init harmony_pcie_init(void)
+static int harmony_pcie_board_init(struct platform_device *pdev)
 {
 	struct device_node *np;
 	int en_vdd_1v05;
@@ -64,21 +65,24 @@ int __init harmony_pcie_init(void)
 
 	regulator_enable(regulator);
 
-	err = tegra_pcie_init(true, true);
-	if (err) {
-		pr_err("%s: tegra_pcie_init failed: %d\n", __func__, err);
-		goto err_pcie;
-	}
-
 	return 0;
 
-err_pcie:
-	regulator_disable(regulator);
-	regulator_put(regulator);
 err_reg:
 	gpio_free(en_vdd_1v05);
 
 	return err;
 }
 
+int __init harmony_pcie_init(void)
+{
+	tegra_pcie_pdata.init = harmony_pcie_board_init;
+	platform_device_register(&tegra_pcie_device);
+
+	return 0;
+}
+#else
+int __init harmony_pcie_init(void)
+{
+	return 0;
+}
 #endif
diff --git a/arch/arm/mach-tegra/board.h b/arch/arm/mach-tegra/board.h
index da8f5a3..8294ed3 100644
--- a/arch/arm/mach-tegra/board.h
+++ b/arch/arm/mach-tegra/board.h
@@ -30,7 +30,6 @@ void __init tegra30_init_early(void);
 void __init tegra_map_common_io(void);
 void __init tegra_init_irq(void);
 void __init tegra_dt_init_irq(void);
-int __init tegra_pcie_init(bool init_port0, bool init_port1);
 
 void tegra_init_late(void);
 
@@ -49,6 +48,9 @@ static inline int tegra_powergate_debugfs_init(void) { return 0; }
 int __init harmony_regulator_init(void);
 #ifdef CONFIG_TEGRA_PCI
 int __init harmony_pcie_init(void);
+
+extern struct tegra_pcie_pdata tegra_pcie_pdata;
+extern struct platform_device tegra_pcie_device;
 #else
 static inline int harmony_pcie_init(void) { return 0; }
 #endif
diff --git a/arch/arm/mach-tegra/iomap.h b/arch/arm/mach-tegra/iomap.h
index db8be51..216046d 100644
--- a/arch/arm/mach-tegra/iomap.h
+++ b/arch/arm/mach-tegra/iomap.h
@@ -287,9 +287,6 @@
 #define IO_APB_VIRT	IOMEM(0xFE300000)
 #define IO_APB_SIZE	SZ_1M
 
-#define TEGRA_PCIE_BASE		0x80000000
-#define TEGRA_PCIE_IO_BASE	(TEGRA_PCIE_BASE + SZ_4M)
-
 #define IO_TO_VIRT_BETWEEN(p, st, sz)	((p) >= (st) && (p) < ((st) + (sz)))
 #define IO_TO_VIRT_XLATE(p, pst, vst)	(((p) - (pst) + (vst)))
 
diff --git a/arch/arm/mach-tegra/pcie.c b/arch/arm/mach-tegra/pcie.c
deleted file mode 100644
index c2d55f8..0000000
--- a/arch/arm/mach-tegra/pcie.c
+++ /dev/null
@@ -1,865 +0,0 @@
-/*
- * arch/arm/mach-tegra/pci.c
- *
- * PCIe host controller driver for TEGRA(2) SOCs
- *
- * Copyright (c) 2010, CompuLab, Ltd.
- * Author: Mike Rapoport <mike@compulab.co.il>
- *
- * Based on NVIDIA PCIe driver
- * Copyright (c) 2008-2009, NVIDIA Corporation.
- *
- * Bits taken from arch/arm/mach-dove/pcie.c
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
- * more details.
- *
- * You should have received a copy of the GNU General Public License along
- * with this program; if not, write to the Free Software Foundation, Inc.,
- * 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
- */
-
-#include <linux/kernel.h>
-#include <linux/pci.h>
-#include <linux/interrupt.h>
-#include <linux/irq.h>
-#include <linux/clk.h>
-#include <linux/delay.h>
-#include <linux/export.h>
-
-#include <asm/sizes.h>
-#include <asm/mach/pci.h>
-
-#include <mach/clk.h>
-#include <mach/powergate.h>
-
-#include "board.h"
-#include "iomap.h"
-#include "pmc.h"
-
-/* Hack - need to parse this from DT */
-#define INT_PCIE_INTR 130
-
-/* register definitions */
-#define AFI_OFFSET	0x3800
-#define PADS_OFFSET	0x3000
-#define RP0_OFFSET	0x0000
-#define RP1_OFFSET	0x1000
-
-#define AFI_AXI_BAR0_SZ	0x00
-#define AFI_AXI_BAR1_SZ	0x04
-#define AFI_AXI_BAR2_SZ	0x08
-#define AFI_AXI_BAR3_SZ	0x0c
-#define AFI_AXI_BAR4_SZ	0x10
-#define AFI_AXI_BAR5_SZ	0x14
-
-#define AFI_AXI_BAR0_START	0x18
-#define AFI_AXI_BAR1_START	0x1c
-#define AFI_AXI_BAR2_START	0x20
-#define AFI_AXI_BAR3_START	0x24
-#define AFI_AXI_BAR4_START	0x28
-#define AFI_AXI_BAR5_START	0x2c
-
-#define AFI_FPCI_BAR0	0x30
-#define AFI_FPCI_BAR1	0x34
-#define AFI_FPCI_BAR2	0x38
-#define AFI_FPCI_BAR3	0x3c
-#define AFI_FPCI_BAR4	0x40
-#define AFI_FPCI_BAR5	0x44
-
-#define AFI_CACHE_BAR0_SZ	0x48
-#define AFI_CACHE_BAR0_ST	0x4c
-#define AFI_CACHE_BAR1_SZ	0x50
-#define AFI_CACHE_BAR1_ST	0x54
-
-#define AFI_MSI_BAR_SZ		0x60
-#define AFI_MSI_FPCI_BAR_ST	0x64
-#define AFI_MSI_AXI_BAR_ST	0x68
-
-#define AFI_CONFIGURATION		0xac
-#define  AFI_CONFIGURATION_EN_FPCI	(1 << 0)
-
-#define AFI_FPCI_ERROR_MASKS	0xb0
-
-#define AFI_INTR_MASK		0xb4
-#define  AFI_INTR_MASK_INT_MASK	(1 << 0)
-#define  AFI_INTR_MASK_MSI_MASK	(1 << 8)
-
-#define AFI_INTR_CODE		0xb8
-#define  AFI_INTR_CODE_MASK	0xf
-#define  AFI_INTR_MASTER_ABORT	4
-#define  AFI_INTR_LEGACY	6
-
-#define AFI_INTR_SIGNATURE	0xbc
-#define AFI_SM_INTR_ENABLE	0xc4
-
-#define AFI_AFI_INTR_ENABLE		0xc8
-#define  AFI_INTR_EN_INI_SLVERR		(1 << 0)
-#define  AFI_INTR_EN_INI_DECERR		(1 << 1)
-#define  AFI_INTR_EN_TGT_SLVERR		(1 << 2)
-#define  AFI_INTR_EN_TGT_DECERR		(1 << 3)
-#define  AFI_INTR_EN_TGT_WRERR		(1 << 4)
-#define  AFI_INTR_EN_DFPCI_DECERR	(1 << 5)
-#define  AFI_INTR_EN_AXI_DECERR		(1 << 6)
-#define  AFI_INTR_EN_FPCI_TIMEOUT	(1 << 7)
-
-#define AFI_PCIE_CONFIG					0x0f8
-#define  AFI_PCIE_CONFIG_PCIEC0_DISABLE_DEVICE		(1 << 1)
-#define  AFI_PCIE_CONFIG_PCIEC1_DISABLE_DEVICE		(1 << 2)
-#define  AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_MASK	(0xf << 20)
-#define  AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_SINGLE	(0x0 << 20)
-#define  AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_DUAL	(0x1 << 20)
-
-#define AFI_FUSE			0x104
-#define  AFI_FUSE_PCIE_T0_GEN2_DIS	(1 << 2)
-
-#define AFI_PEX0_CTRL			0x110
-#define AFI_PEX1_CTRL			0x118
-#define  AFI_PEX_CTRL_RST		(1 << 0)
-#define  AFI_PEX_CTRL_REFCLK_EN		(1 << 3)
-
-#define RP_VEND_XP	0x00000F00
-#define  RP_VEND_XP_DL_UP	(1 << 30)
-
-#define RP_LINK_CONTROL_STATUS			0x00000090
-#define  RP_LINK_CONTROL_STATUS_LINKSTAT_MASK	0x3fff0000
-
-#define PADS_CTL_SEL		0x0000009C
-
-#define PADS_CTL		0x000000A0
-#define  PADS_CTL_IDDQ_1L	(1 << 0)
-#define  PADS_CTL_TX_DATA_EN_1L	(1 << 6)
-#define  PADS_CTL_RX_DATA_EN_1L	(1 << 10)
-
-#define PADS_PLL_CTL				0x000000B8
-#define  PADS_PLL_CTL_RST_B4SM			(1 << 1)
-#define  PADS_PLL_CTL_LOCKDET			(1 << 8)
-#define  PADS_PLL_CTL_REFCLK_MASK		(0x3 << 16)
-#define  PADS_PLL_CTL_REFCLK_INTERNAL_CML	(0 << 16)
-#define  PADS_PLL_CTL_REFCLK_INTERNAL_CMOS	(1 << 16)
-#define  PADS_PLL_CTL_REFCLK_EXTERNAL		(2 << 16)
-#define  PADS_PLL_CTL_TXCLKREF_MASK		(0x1 << 20)
-#define  PADS_PLL_CTL_TXCLKREF_DIV10		(0 << 20)
-#define  PADS_PLL_CTL_TXCLKREF_DIV5		(1 << 20)
-
-/*
- * Tegra2 defines 1GB in the AXI address map for PCIe.
- *
- * That address space is split into different regions, with sizes and
- * offsets as follows:
- *
- * 0x80000000 - 0x80003fff - PCI controller registers
- * 0x80004000 - 0x80103fff - PCI configuration space
- * 0x80104000 - 0x80203fff - PCI extended configuration space
- * 0x80203fff - 0x803fffff - unused
- * 0x80400000 - 0x8040ffff - downstream IO
- * 0x80410000 - 0x8fffffff - unused
- * 0x90000000 - 0x9fffffff - non-prefetchable memory
- * 0xa0000000 - 0xbfffffff - prefetchable memory
- */
-#define PCIE_REGS_SZ		SZ_16K
-#define PCIE_CFG_OFF		PCIE_REGS_SZ
-#define PCIE_CFG_SZ		SZ_1M
-#define PCIE_EXT_CFG_OFF	(PCIE_CFG_SZ + PCIE_CFG_OFF)
-#define PCIE_EXT_CFG_SZ		SZ_1M
-#define PCIE_IOMAP_SZ		(PCIE_REGS_SZ + PCIE_CFG_SZ + PCIE_EXT_CFG_SZ)
-
-#define MEM_BASE_0		(TEGRA_PCIE_BASE + SZ_256M)
-#define MEM_SIZE_0		SZ_128M
-#define MEM_BASE_1		(MEM_BASE_0 + MEM_SIZE_0)
-#define MEM_SIZE_1		SZ_128M
-#define PREFETCH_MEM_BASE_0	(MEM_BASE_1 + MEM_SIZE_1)
-#define PREFETCH_MEM_SIZE_0	SZ_128M
-#define PREFETCH_MEM_BASE_1	(PREFETCH_MEM_BASE_0 + PREFETCH_MEM_SIZE_0)
-#define PREFETCH_MEM_SIZE_1	SZ_128M
-
-#define  PCIE_CONF_BUS(b)	((b) << 16)
-#define  PCIE_CONF_DEV(d)	((d) << 11)
-#define  PCIE_CONF_FUNC(f)	((f) << 8)
-#define  PCIE_CONF_REG(r)	\
-	(((r) & ~0x3) | (((r) < 256) ? PCIE_CFG_OFF : PCIE_EXT_CFG_OFF))
-
-struct tegra_pcie_port {
-	int			index;
-	u8			root_bus_nr;
-	void __iomem		*base;
-
-	bool			link_up;
-
-	char			mem_space_name[16];
-	char			prefetch_space_name[20];
-	struct resource		res[2];
-};
-
-struct tegra_pcie_info {
-	struct tegra_pcie_port	port[2];
-	int			num_ports;
-
-	void __iomem		*regs;
-	struct resource		res_mmio;
-
-	struct clk		*pex_clk;
-	struct clk		*afi_clk;
-	struct clk		*pcie_xclk;
-	struct clk		*pll_e;
-};
-
-static struct tegra_pcie_info tegra_pcie;
-
-static inline void afi_writel(u32 value, unsigned long offset)
-{
-	writel(value, offset + AFI_OFFSET + tegra_pcie.regs);
-}
-
-static inline u32 afi_readl(unsigned long offset)
-{
-	return readl(offset + AFI_OFFSET + tegra_pcie.regs);
-}
-
-static inline void pads_writel(u32 value, unsigned long offset)
-{
-	writel(value, offset + PADS_OFFSET + tegra_pcie.regs);
-}
-
-static inline u32 pads_readl(unsigned long offset)
-{
-	return readl(offset + PADS_OFFSET + tegra_pcie.regs);
-}
-
-static struct tegra_pcie_port *bus_to_port(int bus)
-{
-	int i;
-
-	for (i = tegra_pcie.num_ports - 1; i >= 0; i--) {
-		int rbus = tegra_pcie.port[i].root_bus_nr;
-		if (rbus != -1 && rbus == bus)
-			break;
-	}
-
-	return i >= 0 ? tegra_pcie.port + i : NULL;
-}
-
-static int tegra_pcie_read_conf(struct pci_bus *bus, unsigned int devfn,
-				int where, int size, u32 *val)
-{
-	struct tegra_pcie_port *pp = bus_to_port(bus->number);
-	void __iomem *addr;
-
-	if (pp) {
-		if (devfn != 0) {
-			*val = 0xffffffff;
-			return PCIBIOS_DEVICE_NOT_FOUND;
-		}
-
-		addr = pp->base + (where & ~0x3);
-	} else {
-		addr = tegra_pcie.regs + (PCIE_CONF_BUS(bus->number) +
-					  PCIE_CONF_DEV(PCI_SLOT(devfn)) +
-					  PCIE_CONF_FUNC(PCI_FUNC(devfn)) +
-					  PCIE_CONF_REG(where));
-	}
-
-	*val = readl(addr);
-
-	if (size == 1)
-		*val = (*val >> (8 * (where & 3))) & 0xff;
-	else if (size == 2)
-		*val = (*val >> (8 * (where & 3))) & 0xffff;
-
-	return PCIBIOS_SUCCESSFUL;
-}
-
-static int tegra_pcie_write_conf(struct pci_bus *bus, unsigned int devfn,
-				 int where, int size, u32 val)
-{
-	struct tegra_pcie_port *pp = bus_to_port(bus->number);
-	void __iomem *addr;
-
-	u32 mask;
-	u32 tmp;
-
-	if (pp) {
-		if (devfn != 0)
-			return PCIBIOS_DEVICE_NOT_FOUND;
-
-		addr = pp->base + (where & ~0x3);
-	} else {
-		addr = tegra_pcie.regs + (PCIE_CONF_BUS(bus->number) +
-					  PCIE_CONF_DEV(PCI_SLOT(devfn)) +
-					  PCIE_CONF_FUNC(PCI_FUNC(devfn)) +
-					  PCIE_CONF_REG(where));
-	}
-
-	if (size == 4) {
-		writel(val, addr);
-		return PCIBIOS_SUCCESSFUL;
-	}
-
-	if (size == 2)
-		mask = ~(0xffff << ((where & 0x3) * 8));
-	else if (size == 1)
-		mask = ~(0xff << ((where & 0x3) * 8));
-	else
-		return PCIBIOS_BAD_REGISTER_NUMBER;
-
-	tmp = readl(addr) & mask;
-	tmp |= val << ((where & 0x3) * 8);
-	writel(tmp, addr);
-
-	return PCIBIOS_SUCCESSFUL;
-}
-
-static struct pci_ops tegra_pcie_ops = {
-	.read	= tegra_pcie_read_conf,
-	.write	= tegra_pcie_write_conf,
-};
-
-static void tegra_pcie_fixup_bridge(struct pci_dev *dev)
-{
-	u16 reg;
-
-	if ((dev->class >> 16) == PCI_BASE_CLASS_BRIDGE) {
-		pci_read_config_word(dev, PCI_COMMAND, &reg);
-		reg |= (PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
-			PCI_COMMAND_MASTER | PCI_COMMAND_SERR);
-		pci_write_config_word(dev, PCI_COMMAND, reg);
-	}
-}
-DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, tegra_pcie_fixup_bridge);
-
-/* Tegra PCIE root complex wrongly reports device class */
-static void tegra_pcie_fixup_class(struct pci_dev *dev)
-{
-	dev->class = PCI_CLASS_BRIDGE_PCI << 8;
-}
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0bf0, tegra_pcie_fixup_class);
-DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0bf1, tegra_pcie_fixup_class);
-
-/* Tegra PCIE requires relaxed ordering */
-static void tegra_pcie_relax_enable(struct pci_dev *dev)
-{
-	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN);
-}
-DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, tegra_pcie_relax_enable);
-
-static int tegra_pcie_setup(int nr, struct pci_sys_data *sys)
-{
-	struct tegra_pcie_port *pp;
-
-	if (nr >= tegra_pcie.num_ports)
-		return 0;
-
-	pp = tegra_pcie.port + nr;
-	pp->root_bus_nr = sys->busnr;
-
-	pci_ioremap_io(nr * SZ_64K, TEGRA_PCIE_IO_BASE);
-
-	/*
-	 * IORESOURCE_MEM
-	 */
-	snprintf(pp->mem_space_name, sizeof(pp->mem_space_name),
-		 "PCIe %d MEM", pp->index);
-	pp->mem_space_name[sizeof(pp->mem_space_name) - 1] = 0;
-	pp->res[0].name = pp->mem_space_name;
-	if (pp->index == 0) {
-		pp->res[0].start = MEM_BASE_0;
-		pp->res[0].end = pp->res[0].start + MEM_SIZE_0 - 1;
-	} else {
-		pp->res[0].start = MEM_BASE_1;
-		pp->res[0].end = pp->res[0].start + MEM_SIZE_1 - 1;
-	}
-	pp->res[0].flags = IORESOURCE_MEM;
-	if (request_resource(&iomem_resource, &pp->res[0]))
-		panic("Request PCIe Memory resource failed\n");
-	pci_add_resource_offset(&sys->resources, &pp->res[0], sys->mem_offset);
-
-	/*
-	 * IORESOURCE_MEM | IORESOURCE_PREFETCH
-	 */
-	snprintf(pp->prefetch_space_name, sizeof(pp->prefetch_space_name),
-		 "PCIe %d PREFETCH MEM", pp->index);
-	pp->prefetch_space_name[sizeof(pp->prefetch_space_name) - 1] = 0;
-	pp->res[1].name = pp->prefetch_space_name;
-	if (pp->index == 0) {
-		pp->res[1].start = PREFETCH_MEM_BASE_0;
-		pp->res[1].end = pp->res[1].start + PREFETCH_MEM_SIZE_0 - 1;
-	} else {
-		pp->res[1].start = PREFETCH_MEM_BASE_1;
-		pp->res[1].end = pp->res[1].start + PREFETCH_MEM_SIZE_1 - 1;
-	}
-	pp->res[1].flags = IORESOURCE_MEM | IORESOURCE_PREFETCH;
-	if (request_resource(&iomem_resource, &pp->res[1]))
-		panic("Request PCIe Prefetch Memory resource failed\n");
-	pci_add_resource_offset(&sys->resources, &pp->res[1], sys->mem_offset);
-
-	return 1;
-}
-
-static int tegra_pcie_map_irq(const struct pci_dev *dev, u8 slot, u8 pin)
-{
-	return INT_PCIE_INTR;
-}
-
-static struct pci_bus __init *tegra_pcie_scan_bus(int nr,
-						  struct pci_sys_data *sys)
-{
-	struct tegra_pcie_port *pp;
-
-	if (nr >= tegra_pcie.num_ports)
-		return NULL;
-
-	pp = tegra_pcie.port + nr;
-	pp->root_bus_nr = sys->busnr;
-
-	return pci_scan_root_bus(NULL, sys->busnr, &tegra_pcie_ops, sys,
-				 &sys->resources);
-}
-
-static struct hw_pci tegra_pcie_hw __initdata = {
-	.nr_controllers	= 2,
-	.setup		= tegra_pcie_setup,
-	.scan		= tegra_pcie_scan_bus,
-	.map_irq	= tegra_pcie_map_irq,
-};
-
-
-static irqreturn_t tegra_pcie_isr(int irq, void *arg)
-{
-	const char *err_msg[] = {
-		"Unknown",
-		"AXI slave error",
-		"AXI decode error",
-		"Target abort",
-		"Master abort",
-		"Invalid write",
-		"Response decoding error",
-		"AXI response decoding error",
-		"Transcation timeout",
-	};
-
-	u32 code, signature;
-
-	code = afi_readl(AFI_INTR_CODE) & AFI_INTR_CODE_MASK;
-	signature = afi_readl(AFI_INTR_SIGNATURE);
-	afi_writel(0, AFI_INTR_CODE);
-
-	if (code == AFI_INTR_LEGACY)
-		return IRQ_NONE;
-
-	if (code >= ARRAY_SIZE(err_msg))
-		code = 0;
-
-	/*
-	 * do not pollute kernel log with master abort reports since they
-	 * happen a lot during enumeration
-	 */
-	if (code == AFI_INTR_MASTER_ABORT)
-		pr_debug("PCIE: %s, signature: %08x\n", err_msg[code], signature);
-	else
-		pr_err("PCIE: %s, signature: %08x\n", err_msg[code], signature);
-
-	return IRQ_HANDLED;
-}
-
-static void tegra_pcie_setup_translations(void)
-{
-	u32 fpci_bar;
-	u32 size;
-	u32 axi_address;
-
-	/* Bar 0: config Bar */
-	fpci_bar = ((u32)0xfdff << 16);
-	size = PCIE_CFG_SZ;
-	axi_address = TEGRA_PCIE_BASE + PCIE_CFG_OFF;
-	afi_writel(axi_address, AFI_AXI_BAR0_START);
-	afi_writel(size >> 12, AFI_AXI_BAR0_SZ);
-	afi_writel(fpci_bar, AFI_FPCI_BAR0);
-
-	/* Bar 1: extended config Bar */
-	fpci_bar = ((u32)0xfe1 << 20);
-	size = PCIE_EXT_CFG_SZ;
-	axi_address = TEGRA_PCIE_BASE + PCIE_EXT_CFG_OFF;
-	afi_writel(axi_address, AFI_AXI_BAR1_START);
-	afi_writel(size >> 12, AFI_AXI_BAR1_SZ);
-	afi_writel(fpci_bar, AFI_FPCI_BAR1);
-
-	/* Bar 2: downstream IO bar */
-	fpci_bar = ((__u32)0xfdfc << 16);
-	size = SZ_128K;
-	axi_address = TEGRA_PCIE_IO_BASE;
-	afi_writel(axi_address, AFI_AXI_BAR2_START);
-	afi_writel(size >> 12, AFI_AXI_BAR2_SZ);
-	afi_writel(fpci_bar, AFI_FPCI_BAR2);
-
-	/* Bar 3: prefetchable memory BAR */
-	fpci_bar = (((PREFETCH_MEM_BASE_0 >> 12) & 0x0fffffff) << 4) | 0x1;
-	size =  PREFETCH_MEM_SIZE_0 +  PREFETCH_MEM_SIZE_1;
-	axi_address = PREFETCH_MEM_BASE_0;
-	afi_writel(axi_address, AFI_AXI_BAR3_START);
-	afi_writel(size >> 12, AFI_AXI_BAR3_SZ);
-	afi_writel(fpci_bar, AFI_FPCI_BAR3);
-
-	/* Bar 4: non prefetchable memory BAR */
-	fpci_bar = (((MEM_BASE_0 >> 12)	& 0x0FFFFFFF) << 4) | 0x1;
-	size = MEM_SIZE_0 + MEM_SIZE_1;
-	axi_address = MEM_BASE_0;
-	afi_writel(axi_address, AFI_AXI_BAR4_START);
-	afi_writel(size >> 12, AFI_AXI_BAR4_SZ);
-	afi_writel(fpci_bar, AFI_FPCI_BAR4);
-
-	/* Bar 5: NULL out the remaining BAR as it is not used */
-	fpci_bar = 0;
-	size = 0;
-	axi_address = 0;
-	afi_writel(axi_address, AFI_AXI_BAR5_START);
-	afi_writel(size >> 12, AFI_AXI_BAR5_SZ);
-	afi_writel(fpci_bar, AFI_FPCI_BAR5);
-
-	/* map all upstream transactions as uncached */
-	afi_writel(PHYS_OFFSET, AFI_CACHE_BAR0_ST);
-	afi_writel(0, AFI_CACHE_BAR0_SZ);
-	afi_writel(0, AFI_CACHE_BAR1_ST);
-	afi_writel(0, AFI_CACHE_BAR1_SZ);
-
-	/* No MSI */
-	afi_writel(0, AFI_MSI_FPCI_BAR_ST);
-	afi_writel(0, AFI_MSI_BAR_SZ);
-	afi_writel(0, AFI_MSI_AXI_BAR_ST);
-	afi_writel(0, AFI_MSI_BAR_SZ);
-}
-
-static int tegra_pcie_enable_controller(void)
-{
-	u32 val, reg;
-	int i, timeout;
-
-	/* Enable slot clock and pulse the reset signals */
-	for (i = 0, reg = AFI_PEX0_CTRL; i < 2; i++, reg += 0x8) {
-		val = afi_readl(reg) |  AFI_PEX_CTRL_REFCLK_EN;
-		afi_writel(val, reg);
-		val &= ~AFI_PEX_CTRL_RST;
-		afi_writel(val, reg);
-
-		val = afi_readl(reg) | AFI_PEX_CTRL_RST;
-		afi_writel(val, reg);
-	}
-
-	/* Enable dual controller and both ports */
-	val = afi_readl(AFI_PCIE_CONFIG);
-	val &= ~(AFI_PCIE_CONFIG_PCIEC0_DISABLE_DEVICE |
-		 AFI_PCIE_CONFIG_PCIEC1_DISABLE_DEVICE |
-		 AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_MASK);
-	val |= AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_DUAL;
-	afi_writel(val, AFI_PCIE_CONFIG);
-
-	val = afi_readl(AFI_FUSE) & ~AFI_FUSE_PCIE_T0_GEN2_DIS;
-	afi_writel(val, AFI_FUSE);
-
-	/* Initialze internal PHY, enable up to 16 PCIE lanes */
-	pads_writel(0x0, PADS_CTL_SEL);
-
-	/* override IDDQ to 1 on all 4 lanes */
-	val = pads_readl(PADS_CTL) | PADS_CTL_IDDQ_1L;
-	pads_writel(val, PADS_CTL);
-
-	/*
-	 * set up PHY PLL inputs select PLLE output as refclock,
-	 * set TX ref sel to div10 (not div5)
-	 */
-	val = pads_readl(PADS_PLL_CTL);
-	val &= ~(PADS_PLL_CTL_REFCLK_MASK | PADS_PLL_CTL_TXCLKREF_MASK);
-	val |= (PADS_PLL_CTL_REFCLK_INTERNAL_CML | PADS_PLL_CTL_TXCLKREF_DIV10);
-	pads_writel(val, PADS_PLL_CTL);
-
-	/* take PLL out of reset  */
-	val = pads_readl(PADS_PLL_CTL) | PADS_PLL_CTL_RST_B4SM;
-	pads_writel(val, PADS_PLL_CTL);
-
-	/*
-	 * Hack, set the clock voltage to the DEFAULT provided by hw folks.
-	 * This doesn't exist in the documentation
-	 */
-	pads_writel(0xfa5cfa5c, 0xc8);
-
-	/* Wait for the PLL to lock */
-	timeout = 300;
-	do {
-		val = pads_readl(PADS_PLL_CTL);
-		usleep_range(1000, 1000);
-		if (--timeout == 0) {
-			pr_err("Tegra PCIe error: timeout waiting for PLL\n");
-			return -EBUSY;
-		}
-	} while (!(val & PADS_PLL_CTL_LOCKDET));
-
-	/* turn off IDDQ override */
-	val = pads_readl(PADS_CTL) & ~PADS_CTL_IDDQ_1L;
-	pads_writel(val, PADS_CTL);
-
-	/* enable TX/RX data */
-	val = pads_readl(PADS_CTL);
-	val |= (PADS_CTL_TX_DATA_EN_1L | PADS_CTL_RX_DATA_EN_1L);
-	pads_writel(val, PADS_CTL);
-
-	/* Take the PCIe interface module out of reset */
-	tegra_periph_reset_deassert(tegra_pcie.pcie_xclk);
-
-	/* Finally enable PCIe */
-	val = afi_readl(AFI_CONFIGURATION) | AFI_CONFIGURATION_EN_FPCI;
-	afi_writel(val, AFI_CONFIGURATION);
-
-	val = (AFI_INTR_EN_INI_SLVERR | AFI_INTR_EN_INI_DECERR |
-	       AFI_INTR_EN_TGT_SLVERR | AFI_INTR_EN_TGT_DECERR |
-	       AFI_INTR_EN_TGT_WRERR | AFI_INTR_EN_DFPCI_DECERR);
-	afi_writel(val, AFI_AFI_INTR_ENABLE);
-	afi_writel(0xffffffff, AFI_SM_INTR_ENABLE);
-
-	/* FIXME: No MSI for now, only INT */
-	afi_writel(AFI_INTR_MASK_INT_MASK, AFI_INTR_MASK);
-
-	/* Disable all execptions */
-	afi_writel(0, AFI_FPCI_ERROR_MASKS);
-
-	return 0;
-}
-
-static void tegra_pcie_power_off(void)
-{
-	tegra_periph_reset_assert(tegra_pcie.pcie_xclk);
-	tegra_periph_reset_assert(tegra_pcie.afi_clk);
-	tegra_periph_reset_assert(tegra_pcie.pex_clk);
-
-	tegra_powergate_power_off(TEGRA_POWERGATE_PCIE);
-	tegra_pmc_pcie_xclk_clamp(true);
-}
-
-static int tegra_pcie_power_regate(void)
-{
-	int err;
-
-	tegra_pcie_power_off();
-
-	tegra_pmc_pcie_xclk_clamp(true);
-
-	tegra_periph_reset_assert(tegra_pcie.pcie_xclk);
-	tegra_periph_reset_assert(tegra_pcie.afi_clk);
-
-	err = tegra_powergate_sequence_power_up(TEGRA_POWERGATE_PCIE,
-						tegra_pcie.pex_clk);
-	if (err) {
-		pr_err("PCIE: powerup sequence failed: %d\n", err);
-		return err;
-	}
-
-	tegra_periph_reset_deassert(tegra_pcie.afi_clk);
-
-	tegra_pmc_pcie_xclk_clamp(false);
-
-	clk_prepare_enable(tegra_pcie.afi_clk);
-	clk_prepare_enable(tegra_pcie.pex_clk);
-	return clk_prepare_enable(tegra_pcie.pll_e);
-}
-
-static int tegra_pcie_clocks_get(void)
-{
-	int err;
-
-	tegra_pcie.pex_clk = clk_get(NULL, "pex");
-	if (IS_ERR(tegra_pcie.pex_clk))
-		return PTR_ERR(tegra_pcie.pex_clk);
-
-	tegra_pcie.afi_clk = clk_get(NULL, "afi");
-	if (IS_ERR(tegra_pcie.afi_clk)) {
-		err = PTR_ERR(tegra_pcie.afi_clk);
-		goto err_afi_clk;
-	}
-
-	tegra_pcie.pcie_xclk = clk_get(NULL, "pcie_xclk");
-	if (IS_ERR(tegra_pcie.pcie_xclk)) {
-		err =  PTR_ERR(tegra_pcie.pcie_xclk);
-		goto err_pcie_xclk;
-	}
-
-	tegra_pcie.pll_e = clk_get_sys(NULL, "pll_e");
-	if (IS_ERR(tegra_pcie.pll_e)) {
-		err = PTR_ERR(tegra_pcie.pll_e);
-		goto err_pll_e;
-	}
-
-	return 0;
-
-err_pll_e:
-	clk_put(tegra_pcie.pcie_xclk);
-err_pcie_xclk:
-	clk_put(tegra_pcie.afi_clk);
-err_afi_clk:
-	clk_put(tegra_pcie.pex_clk);
-
-	return err;
-}
-
-static void tegra_pcie_clocks_put(void)
-{
-	clk_put(tegra_pcie.pll_e);
-	clk_put(tegra_pcie.pcie_xclk);
-	clk_put(tegra_pcie.afi_clk);
-	clk_put(tegra_pcie.pex_clk);
-}
-
-static int __init tegra_pcie_get_resources(void)
-{
-	int err;
-
-	err = tegra_pcie_clocks_get();
-	if (err) {
-		pr_err("PCIE: failed to get clocks: %d\n", err);
-		return err;
-	}
-
-	err = tegra_pcie_power_regate();
-	if (err) {
-		pr_err("PCIE: failed to power up: %d\n", err);
-		goto err_pwr_on;
-	}
-
-	tegra_pcie.regs = ioremap_nocache(TEGRA_PCIE_BASE, PCIE_IOMAP_SZ);
-	if (tegra_pcie.regs == NULL) {
-		pr_err("PCIE: Failed to map PCI/AFI registers\n");
-		err = -ENOMEM;
-		goto err_map_reg;
-	}
-
-	err = request_irq(INT_PCIE_INTR, tegra_pcie_isr,
-			  IRQF_SHARED, "PCIE", &tegra_pcie);
-	if (err) {
-		pr_err("PCIE: Failed to register IRQ: %d\n", err);
-		goto err_req_io;
-	}
-	set_irq_flags(INT_PCIE_INTR, IRQF_VALID);
-
-	return 0;
-
-err_req_io:
-	iounmap(tegra_pcie.regs);
-err_map_reg:
-	tegra_pcie_power_off();
-err_pwr_on:
-	tegra_pcie_clocks_put();
-
-	return err;
-}
-
-/*
- * FIXME: If there are no PCIe cards attached, then calling this function
- * can result in the increase of the bootup time as there are big timeout
- * loops.
- */
-#define TEGRA_PCIE_LINKUP_TIMEOUT	200	/* up to 1.2 seconds */
-static bool tegra_pcie_check_link(struct tegra_pcie_port *pp, int idx,
-				  u32 reset_reg)
-{
-	u32 reg;
-	int retries = 3;
-	int timeout;
-
-	do {
-		timeout = TEGRA_PCIE_LINKUP_TIMEOUT;
-		while (timeout) {
-			reg = readl(pp->base + RP_VEND_XP);
-
-			if (reg & RP_VEND_XP_DL_UP)
-				break;
-
-			mdelay(1);
-			timeout--;
-		}
-
-		if (!timeout)  {
-			pr_err("PCIE: port %d: link down, retrying\n", idx);
-			goto retry;
-		}
-
-		timeout = TEGRA_PCIE_LINKUP_TIMEOUT;
-		while (timeout) {
-			reg = readl(pp->base + RP_LINK_CONTROL_STATUS);
-
-			if (reg & 0x20000000)
-				return true;
-
-			mdelay(1);
-			timeout--;
-		}
-
-retry:
-		/* Pulse the PEX reset */
-		reg = afi_readl(reset_reg) | AFI_PEX_CTRL_RST;
-		afi_writel(reg, reset_reg);
-		mdelay(1);
-		reg = afi_readl(reset_reg) & ~AFI_PEX_CTRL_RST;
-		afi_writel(reg, reset_reg);
-
-		retries--;
-	} while (retries);
-
-	return false;
-}
-
-static void __init tegra_pcie_add_port(int index, u32 offset, u32 reset_reg)
-{
-	struct tegra_pcie_port *pp;
-
-	pp = tegra_pcie.port + tegra_pcie.num_ports;
-
-	pp->index = -1;
-	pp->base = tegra_pcie.regs + offset;
-	pp->link_up = tegra_pcie_check_link(pp, index, reset_reg);
-
-	if (!pp->link_up) {
-		pp->base = NULL;
-		printk(KERN_INFO "PCIE: port %d: link down, ignoring\n", index);
-		return;
-	}
-
-	tegra_pcie.num_ports++;
-	pp->index = index;
-	pp->root_bus_nr = -1;
-	memset(pp->res, 0, sizeof(pp->res));
-}
-
-int __init tegra_pcie_init(bool init_port0, bool init_port1)
-{
-	int err;
-
-	if (!(init_port0 || init_port1))
-		return -ENODEV;
-
-	pcibios_min_mem = 0;
-
-	err = tegra_pcie_get_resources();
-	if (err)
-		return err;
-
-	err = tegra_pcie_enable_controller();
-	if (err)
-		return err;
-
-	/* setup the AFI address translations */
-	tegra_pcie_setup_translations();
-
-	if (init_port0)
-		tegra_pcie_add_port(0, RP0_OFFSET, AFI_PEX0_CTRL);
-
-	if (init_port1)
-		tegra_pcie_add_port(1, RP1_OFFSET, AFI_PEX1_CTRL);
-
-	pci_common_init(&tegra_pcie_hw);
-
-	return 0;
-}
diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index 6d51aa6..ac45398 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -119,3 +119,5 @@ config PCI_IOAPIC
 config PCI_LABEL
 	def_bool y if (DMI || ACPI)
 	select NLS
+
+source "drivers/pci/host/Kconfig"
diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
index 0c3efcf..6ebf5bf 100644
--- a/drivers/pci/Makefile
+++ b/drivers/pci/Makefile
@@ -67,3 +67,6 @@ obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
 obj-$(CONFIG_OF) += of.o
 
 ccflags-$(CONFIG_PCI_DEBUG) := -DDEBUG
+
+# PCI host controller drivers
+obj-y += host/
diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
new file mode 100644
index 0000000..683e105
--- /dev/null
+++ b/drivers/pci/host/Kconfig
@@ -0,0 +1,8 @@
+menu "PCI host controller drivers"
+	depends on PCI
+
+config PCI_TEGRA
+	bool "NVIDIA Tegra PCIe controller"
+	depends on ARCH_TEGRA_2x_SOC
+
+endmenu
diff --git a/drivers/pci/host/Makefile b/drivers/pci/host/Makefile
new file mode 100644
index 0000000..bf3adff
--- /dev/null
+++ b/drivers/pci/host/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_PCI_TEGRA) += pci-tegra.o
diff --git a/drivers/pci/host/pci-tegra.c b/drivers/pci/host/pci-tegra.c
new file mode 100644
index 0000000..e1199fb
--- /dev/null
+++ b/drivers/pci/host/pci-tegra.c
@@ -0,0 +1,1322 @@
+/*
+ * PCIe host controller driver for TEGRA(2) SOCs
+ *
+ * Copyright (c) 2010, CompuLab, Ltd.
+ * Author: Mike Rapoport <mike@compulab.co.il>
+ *
+ * Based on NVIDIA PCIe driver
+ * Copyright (c) 2008-2009, NVIDIA Corporation.
+ *
+ * Bits taken from arch/arm/mach-dove/pcie.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+ */
+
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+#include <linux/interrupt.h>
+#include <linux/irq.h>
+#include <linux/irqdomain.h>
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/export.h>
+#include <linux/msi.h>
+#include <linux/of_address.h>
+#include <linux/of_pci.h>
+#include <linux/of_platform.h>
+#include <linux/regulator/consumer.h>
+
+#include <asm/sizes.h>
+#include <asm/mach/irq.h>
+#include <asm/mach/pci.h>
+
+#include <mach/clk.h>
+#include <mach/pmc.h>
+#include <mach/powergate.h>
+
+#define INT_PCI_MSI_NR (8 * 32)
+
+/* register definitions */
+
+#define AFI_AXI_BAR0_SZ	0x00
+#define AFI_AXI_BAR1_SZ	0x04
+#define AFI_AXI_BAR2_SZ	0x08
+#define AFI_AXI_BAR3_SZ	0x0c
+#define AFI_AXI_BAR4_SZ	0x10
+#define AFI_AXI_BAR5_SZ	0x14
+
+#define AFI_AXI_BAR0_START	0x18
+#define AFI_AXI_BAR1_START	0x1c
+#define AFI_AXI_BAR2_START	0x20
+#define AFI_AXI_BAR3_START	0x24
+#define AFI_AXI_BAR4_START	0x28
+#define AFI_AXI_BAR5_START	0x2c
+
+#define AFI_FPCI_BAR0	0x30
+#define AFI_FPCI_BAR1	0x34
+#define AFI_FPCI_BAR2	0x38
+#define AFI_FPCI_BAR3	0x3c
+#define AFI_FPCI_BAR4	0x40
+#define AFI_FPCI_BAR5	0x44
+
+#define AFI_CACHE_BAR0_SZ	0x48
+#define AFI_CACHE_BAR0_ST	0x4c
+#define AFI_CACHE_BAR1_SZ	0x50
+#define AFI_CACHE_BAR1_ST	0x54
+
+#define AFI_MSI_BAR_SZ		0x60
+#define AFI_MSI_FPCI_BAR_ST	0x64
+#define AFI_MSI_AXI_BAR_ST	0x68
+
+#define AFI_MSI_VEC0		0x6c
+#define AFI_MSI_VEC1		0x70
+#define AFI_MSI_VEC2		0x74
+#define AFI_MSI_VEC3		0x78
+#define AFI_MSI_VEC4		0x7c
+#define AFI_MSI_VEC5		0x80
+#define AFI_MSI_VEC6		0x84
+#define AFI_MSI_VEC7		0x88
+
+#define AFI_MSI_EN_VEC0		0x8c
+#define AFI_MSI_EN_VEC1		0x90
+#define AFI_MSI_EN_VEC2		0x94
+#define AFI_MSI_EN_VEC3		0x98
+#define AFI_MSI_EN_VEC4		0x9c
+#define AFI_MSI_EN_VEC5		0xa0
+#define AFI_MSI_EN_VEC6		0xa4
+#define AFI_MSI_EN_VEC7		0xa8
+
+#define AFI_CONFIGURATION		0xac
+#define  AFI_CONFIGURATION_EN_FPCI	(1 << 0)
+
+#define AFI_FPCI_ERROR_MASKS	0xb0
+
+#define AFI_INTR_MASK		0xb4
+#define  AFI_INTR_MASK_INT_MASK	(1 << 0)
+#define  AFI_INTR_MASK_MSI_MASK	(1 << 8)
+
+#define AFI_INTR_CODE		0xb8
+#define  AFI_INTR_CODE_MASK	0xf
+#define  AFI_INTR_MASTER_ABORT	4
+#define  AFI_INTR_LEGACY	6
+
+#define AFI_INTR_SIGNATURE	0xbc
+#define AFI_UPPER_FPCI_ADDRESS	0xc0
+#define AFI_SM_INTR_ENABLE	0xc4
+
+#define AFI_AFI_INTR_ENABLE		0xc8
+#define  AFI_INTR_EN_INI_SLVERR		(1 << 0)
+#define  AFI_INTR_EN_INI_DECERR		(1 << 1)
+#define  AFI_INTR_EN_TGT_SLVERR		(1 << 2)
+#define  AFI_INTR_EN_TGT_DECERR		(1 << 3)
+#define  AFI_INTR_EN_TGT_WRERR		(1 << 4)
+#define  AFI_INTR_EN_DFPCI_DECERR	(1 << 5)
+#define  AFI_INTR_EN_AXI_DECERR		(1 << 6)
+#define  AFI_INTR_EN_FPCI_TIMEOUT	(1 << 7)
+
+#define AFI_PCIE_CONFIG					0x0f8
+#define  AFI_PCIE_CONFIG_PCIEC0_DISABLE_DEVICE		(1 << 1)
+#define  AFI_PCIE_CONFIG_PCIEC1_DISABLE_DEVICE		(1 << 2)
+#define  AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_MASK	(0xf << 20)
+#define  AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_SINGLE	(0x0 << 20)
+#define  AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_DUAL	(0x1 << 20)
+
+#define AFI_FUSE			0x104
+#define  AFI_FUSE_PCIE_T0_GEN2_DIS	(1 << 2)
+
+#define AFI_PEX0_CTRL			0x110
+#define AFI_PEX1_CTRL			0x118
+#define  AFI_PEX_CTRL_RST		(1 << 0)
+#define  AFI_PEX_CTRL_REFCLK_EN		(1 << 3)
+
+#define RP_VEND_XP	0x00000F00
+#define  RP_VEND_XP_DL_UP	(1 << 30)
+
+#define RP_LINK_CONTROL_STATUS			0x00000090
+#define  RP_LINK_CONTROL_STATUS_LINKSTAT_MASK	0x3fff0000
+
+#define PADS_CTL_SEL		0x0000009C
+
+#define PADS_CTL		0x000000A0
+#define  PADS_CTL_IDDQ_1L	(1 << 0)
+#define  PADS_CTL_TX_DATA_EN_1L	(1 << 6)
+#define  PADS_CTL_RX_DATA_EN_1L	(1 << 10)
+
+#define PADS_PLL_CTL				0x000000B8
+#define  PADS_PLL_CTL_RST_B4SM			(1 << 1)
+#define  PADS_PLL_CTL_LOCKDET			(1 << 8)
+#define  PADS_PLL_CTL_REFCLK_MASK		(0x3 << 16)
+#define  PADS_PLL_CTL_REFCLK_INTERNAL_CML	(0 << 16)
+#define  PADS_PLL_CTL_REFCLK_INTERNAL_CMOS	(1 << 16)
+#define  PADS_PLL_CTL_REFCLK_EXTERNAL		(2 << 16)
+#define  PADS_PLL_CTL_TXCLKREF_MASK		(0x1 << 20)
+#define  PADS_PLL_CTL_TXCLKREF_DIV10		(0 << 20)
+#define  PADS_PLL_CTL_TXCLKREF_DIV5		(1 << 20)
+
+/*
+ * The configuration space mapping on Tegra is somewhat similar to the ECAM
+ * defined by PCIe. However it deviates a bit in how the 4 bits for extended
+ * register accesses are mapped:
+ *
+ *    [27:24] extended register number
+ *    [23:16] bus number
+ *    [15:11] device number
+ *    [10: 8] function number
+ *    [ 7: 0] register number
+ */
+#define PCIE_CONF_BUS(b)	((b) << 16)
+#define PCIE_CONF_DEV(d)	((d) << 11)
+#define PCIE_CONF_FUNC(f)	((f) << 8)
+#define PCIE_CONF_REG(r)	((((r) & 0xf00) << 16) | (((r) & 0xff) & ~3))
+
+struct tegra_pcie_msi {
+	DECLARE_BITMAP(used, INT_PCI_MSI_NR);
+	struct irq_domain *domain;
+	unsigned long pages;
+	struct mutex lock;
+	int irq;
+};
+
+struct tegra_pcie {
+	struct device *dev;
+
+	void __iomem *pads;
+	void __iomem *afi;
+	int irq;
+
+	struct iomap_cache *csc;
+	struct resource *cs;
+
+	struct resource io;
+	struct resource mem;
+	struct resource prefetch;
+	struct resource busn;
+
+	struct clk *pex_clk;
+	struct clk *afi_clk;
+	struct clk *pcie_xclk;
+	struct clk *pll_e;
+
+	struct tegra_pcie_msi *msi;
+
+	struct list_head ports;
+	unsigned int num_ports;
+
+	struct regulator *pex_clk_supply;
+	struct regulator *vdd_supply;
+};
+
+struct tegra_pcie_port {
+	struct tegra_pcie *pcie;
+	struct list_head list;
+	void __iomem *base;
+	unsigned int index;
+};
+
+static inline struct tegra_pcie *sys_to_pcie(struct pci_sys_data *sys)
+{
+	return sys->private_data;
+}
+
+static inline void afi_writel(struct tegra_pcie *pcie, u32 value,
+			      unsigned long offset)
+{
+	writel(value, pcie->afi + offset);
+}
+
+static inline u32 afi_readl(struct tegra_pcie *pcie, unsigned long offset)
+{
+	return readl(pcie->afi + offset);
+}
+
+static inline void pads_writel(struct tegra_pcie *pcie, u32 value,
+			       unsigned long offset)
+{
+	writel(value, pcie->pads + offset);
+}
+
+static inline u32 pads_readl(struct tegra_pcie *pcie, unsigned long offset)
+{
+	return readl(pcie->pads + offset);
+}
+
+static int tegra_pcie_read_conf(struct pci_bus *bus, unsigned int devfn,
+				int where, int size, u32 *value)
+{
+	struct tegra_pcie *pcie = sys_to_pcie(bus->sysdata);
+	void __iomem *addr = NULL;
+	unsigned long offset;
+
+	offset = PCIE_CONF_BUS(bus->number) + PCIE_CONF_DEV(PCI_SLOT(devfn)) +
+		 PCIE_CONF_FUNC(PCI_FUNC(devfn)) + PCIE_CONF_REG(where);
+
+	if (bus->number == 0) {
+		unsigned int slot = PCI_SLOT(devfn);
+		struct tegra_pcie_port *port;
+
+		list_for_each_entry(port, &pcie->ports, list) {
+			if (port->index + 1 == slot) {
+				addr = port->base + (where & ~3);
+				break;
+			}
+		}
+	} else
+		addr = iomap_cache_map(pcie->csc, offset);
+
+	if (!addr) {
+		*value = 0xffffffff;
+		return PCIBIOS_DEVICE_NOT_FOUND;
+	}
+
+	*value = readl(addr);
+
+	if (size == 1)
+		*value = (*value >> (8 * (where & 3))) & 0xff;
+	else if (size == 2)
+		*value = (*value >> (8 * (where & 3))) & 0xffff;
+
+	return PCIBIOS_SUCCESSFUL;
+}
+
+static int tegra_pcie_write_conf(struct pci_bus *bus, unsigned int devfn,
+				 int where, int size, u32 value)
+{
+	struct tegra_pcie *pcie = sys_to_pcie(bus->sysdata);
+	void __iomem *addr = NULL;
+	unsigned long offset;
+	u32 mask, tmp;
+
+	offset = PCIE_CONF_BUS(bus->number) + PCIE_CONF_DEV(PCI_SLOT(devfn)) +
+		 PCIE_CONF_FUNC(PCI_FUNC(devfn)) + PCIE_CONF_REG(where);
+
+	if (bus->number == 0) {
+		unsigned int slot = PCI_SLOT(devfn);
+		struct tegra_pcie_port *port;
+
+		list_for_each_entry(port, &pcie->ports, list) {
+			if (port->index + 1 == slot) {
+				addr = port->base + (where & ~3);
+				break;
+			}
+		}
+	} else
+		addr = iomap_cache_map(pcie->csc, offset);
+
+	if (size == 4) {
+		writel(value, addr);
+		return PCIBIOS_SUCCESSFUL;
+	}
+
+	if (size == 2)
+		mask = ~(0xffff << ((where & 0x3) * 8));
+	else if (size == 1)
+		mask = ~(0xff << ((where & 0x3) * 8));
+	else
+		return PCIBIOS_BAD_REGISTER_NUMBER;
+
+	tmp = readl(addr) & mask;
+	tmp |= value << ((where & 0x3) * 8);
+	writel(tmp, addr);
+
+	return PCIBIOS_SUCCESSFUL;
+}
+
+static struct pci_ops tegra_pcie_ops = {
+	.read	= tegra_pcie_read_conf,
+	.write	= tegra_pcie_write_conf,
+};
+
+static void tegra_pcie_fixup_bridge(struct pci_dev *dev)
+{
+	u16 reg;
+
+	if ((dev->class >> 16) == PCI_BASE_CLASS_BRIDGE) {
+		pci_read_config_word(dev, PCI_COMMAND, &reg);
+		reg |= (PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
+			PCI_COMMAND_MASTER | PCI_COMMAND_SERR);
+		pci_write_config_word(dev, PCI_COMMAND, reg);
+	}
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, tegra_pcie_fixup_bridge);
+
+/* Tegra PCIE root complex wrongly reports device class */
+static void tegra_pcie_fixup_class(struct pci_dev *dev)
+{
+	dev->class = PCI_CLASS_BRIDGE_PCI << 8;
+}
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0bf0, tegra_pcie_fixup_class);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_NVIDIA, 0x0bf1, tegra_pcie_fixup_class);
+
+/* Tegra PCIE requires relaxed ordering */
+static void tegra_pcie_relax_enable(struct pci_dev *dev)
+{
+	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN);
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, tegra_pcie_relax_enable);
+
+static int tegra_pcie_setup(int nr, struct pci_sys_data *sys)
+{
+	struct tegra_pcie *pcie = sys_to_pcie(sys);
+
+	pci_add_resource_offset(&sys->resources, &pcie->io, sys->io_offset);
+	pci_add_resource_offset(&sys->resources, &pcie->mem, sys->mem_offset);
+	pci_add_resource_offset(&sys->resources, &pcie->prefetch, sys->mem_offset);
+	pci_add_resource(&sys->resources, &pcie->busn);
+
+	pci_ioremap_io(nr * SZ_64K, pcie->io.start);
+
+	return 1;
+}
+
+static int tegra_pcie_map_irq(const struct pci_dev *pdev, u8 slot, u8 pin)
+{
+	struct tegra_pcie *pcie = sys_to_pcie(pdev->bus->sysdata);
+
+	return pcie->irq;
+}
+
+static struct pci_bus *tegra_pcie_scan_bus(int nr, struct pci_sys_data *sys)
+{
+	struct tegra_pcie *pcie = sys_to_pcie(sys);
+	struct pci_host_bridge_window *window;
+	struct pci_bus *bus;
+	bool found = false;
+
+	list_for_each_entry(window, &sys->resources, list) {
+		if (window->res->flags & IORESOURCE_BUS) {
+			found = true;
+			break;
+		}
+	}
+
+	if (!found)
+		dev_err(pcie->dev, "no bus resource found!\n");
+
+	bus = pci_create_root_bus(pcie->dev, sys->busnr, &tegra_pcie_ops,
+				  sys, &sys->resources);
+	if (!bus)
+		return NULL;
+
+	pci_scan_child_bus(bus);
+
+	return bus;
+}
+
+static irqreturn_t tegra_pcie_isr(int irq, void *arg)
+{
+	const char *err_msg[] = {
+		"Unknown",
+		"AXI slave error",
+		"AXI decode error",
+		"Target abort",
+		"Master abort",
+		"Invalid write",
+		"Response decoding error",
+		"AXI response decoding error",
+		"Transaction timeout",
+	};
+	struct tegra_pcie *pcie = arg;
+	u32 code, signature;
+
+	code = afi_readl(pcie, AFI_INTR_CODE) & AFI_INTR_CODE_MASK;
+	signature = afi_readl(pcie, AFI_INTR_SIGNATURE);
+	afi_writel(pcie, 0, AFI_INTR_CODE);
+
+	if (code == AFI_INTR_LEGACY)
+		return IRQ_NONE;
+
+	if (code >= ARRAY_SIZE(err_msg))
+		code = 0;
+
+	/*
+	 * do not pollute kernel log with master abort reports since they
+	 * happen a lot during enumeration
+	 */
+	if (code == AFI_INTR_MASTER_ABORT) {
+		dev_dbg(pcie->dev, "%s, signature: %08x\n", err_msg[code],
+			signature);
+	} else
+		dev_err(pcie->dev, "%s, signature: %08x\n", err_msg[code],
+			signature);
+
+	if (code == 3 || code == 4 || code == 7) {
+		u32 fpci = afi_readl(pcie, AFI_UPPER_FPCI_ADDRESS) & 0xff;
+		u64 address = (u64)fpci << 32 | (signature & 0xfffffffc);
+		dev_dbg(pcie->dev, "  FPCI address: %10llx\n", address);
+	}
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * FPCI map is as follows:
+ * - 0xfdfc000000: I/O space
+ * - 0xfdfe000000: type 0 configuration space
+ * - 0xfdff000000: type 1 configuration space
+ * - 0xfe00000000: type 0 extended configuration space
+ * - 0xfe10000000: type 1 extended configuration space
+ */
+static void tegra_pcie_setup_translations(struct tegra_pcie *pcie)
+{
+	u32 fpci_bar, size, axi_address;
+
+	/* Bar 0: type 1 extended configuration space */
+	fpci_bar = 0xfe100000;
+	size = resource_size(pcie->cs);
+	axi_address = pcie->cs->start;
+	afi_writel(pcie, axi_address, AFI_AXI_BAR0_START);
+	afi_writel(pcie, size >> 12, AFI_AXI_BAR0_SZ);
+	afi_writel(pcie, fpci_bar, AFI_FPCI_BAR0);
+
+	/* Bar 1: downstream IO bar */
+	fpci_bar = 0xfdfc0000;
+	size = resource_size(&pcie->io);
+	axi_address = pcie->io.start;
+	afi_writel(pcie, axi_address, AFI_AXI_BAR1_START);
+	afi_writel(pcie, size >> 12, AFI_AXI_BAR1_SZ);
+	afi_writel(pcie, fpci_bar, AFI_FPCI_BAR1);
+
+	/* Bar 2: prefetchable memory BAR */
+	fpci_bar = (((pcie->prefetch.start >> 12) & 0x0fffffff) << 4) | 0x1;
+	size = resource_size(&pcie->prefetch);
+	axi_address = pcie->prefetch.start;
+	afi_writel(pcie, axi_address, AFI_AXI_BAR2_START);
+	afi_writel(pcie, size >> 12, AFI_AXI_BAR2_SZ);
+	afi_writel(pcie, fpci_bar, AFI_FPCI_BAR2);
+
+	/* Bar 3: non prefetchable memory BAR */
+	fpci_bar = (((pcie->mem.start >> 12) & 0x0fffffff) << 4) | 0x1;
+	size = resource_size(&pcie->mem);
+	axi_address = pcie->mem.start;
+	afi_writel(pcie, axi_address, AFI_AXI_BAR3_START);
+	afi_writel(pcie, size >> 12, AFI_AXI_BAR3_SZ);
+	afi_writel(pcie, fpci_bar, AFI_FPCI_BAR3);
+
+	/* NULL out the remaining BARs as they are not used */
+	afi_writel(pcie, 0, AFI_AXI_BAR4_START);
+	afi_writel(pcie, 0, AFI_AXI_BAR4_SZ);
+	afi_writel(pcie, 0, AFI_FPCI_BAR4);
+
+	afi_writel(pcie, 0, AFI_AXI_BAR5_START);
+	afi_writel(pcie, 0, AFI_AXI_BAR5_SZ);
+	afi_writel(pcie, 0, AFI_FPCI_BAR5);
+
+	/* map all upstream transactions as uncached */
+	afi_writel(pcie, PHYS_OFFSET, AFI_CACHE_BAR0_ST);
+	afi_writel(pcie, 0, AFI_CACHE_BAR0_SZ);
+	afi_writel(pcie, 0, AFI_CACHE_BAR1_ST);
+	afi_writel(pcie, 0, AFI_CACHE_BAR1_SZ);
+
+	/* MSI translations are setup only when needed */
+	afi_writel(pcie, 0, AFI_MSI_FPCI_BAR_ST);
+	afi_writel(pcie, 0, AFI_MSI_BAR_SZ);
+	afi_writel(pcie, 0, AFI_MSI_AXI_BAR_ST);
+	afi_writel(pcie, 0, AFI_MSI_BAR_SZ);
+}
+
+static int tegra_pcie_enable_controller(struct tegra_pcie *pcie)
+{
+	unsigned int timeout;
+	unsigned long value;
+
+	/* enable dual controller and both ports */
+	value = afi_readl(pcie, AFI_PCIE_CONFIG);
+	value &= ~(AFI_PCIE_CONFIG_PCIEC0_DISABLE_DEVICE |
+		   AFI_PCIE_CONFIG_PCIEC1_DISABLE_DEVICE |
+		   AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_MASK);
+	value |= AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_DUAL;
+	afi_writel(pcie, value, AFI_PCIE_CONFIG);
+
+	value = afi_readl(pcie, AFI_FUSE);
+	value &= ~AFI_FUSE_PCIE_T0_GEN2_DIS;
+	afi_writel(pcie, value, AFI_FUSE);
+
+	/* initialze internal PHY, enable up to 16 PCIE lanes */
+	pads_writel(pcie, 0x0, PADS_CTL_SEL);
+
+	/* override IDDQ to 1 on all 4 lanes */
+	value = pads_readl(pcie, PADS_CTL);
+	value |= PADS_CTL_IDDQ_1L;
+	pads_writel(pcie, value, PADS_CTL);
+
+	/*
+	 * Set up PHY PLL inputs select PLLE output as refclock,
+	 * set TX ref sel to div10 (not div5).
+	 */
+	value = pads_readl(pcie, PADS_PLL_CTL);
+	value &= ~(PADS_PLL_CTL_REFCLK_MASK | PADS_PLL_CTL_TXCLKREF_MASK);
+	value |= (PADS_PLL_CTL_REFCLK_INTERNAL_CML | PADS_PLL_CTL_TXCLKREF_DIV10);
+	pads_writel(pcie, value, PADS_PLL_CTL);
+
+	/* take PLL out of reset  */
+	value = pads_readl(pcie, PADS_PLL_CTL);
+	value |= PADS_PLL_CTL_RST_B4SM;
+	pads_writel(pcie, value, PADS_PLL_CTL);
+
+	/*
+	 * Hack, set the clock voltage to the DEFAULT provided by hw folks.
+	 * This doesn't exist in the documentation.
+	 */
+	pads_writel(pcie, 0xfa5cfa5c, 0xc8);
+
+	/* wait for the PLL to lock */
+	timeout = 300;
+	do {
+		value = pads_readl(pcie, PADS_PLL_CTL);
+		usleep_range(1000, 1000);
+		if (--timeout == 0) {
+			pr_err("Tegra PCIe error: timeout waiting for PLL\n");
+			return -EBUSY;
+		}
+	} while (!(value & PADS_PLL_CTL_LOCKDET));
+
+	/* turn off IDDQ override */
+	value = pads_readl(pcie, PADS_CTL);
+	value &= ~PADS_CTL_IDDQ_1L;
+	pads_writel(pcie, value, PADS_CTL);
+
+	/* enable TX/RX data */
+	value = pads_readl(pcie, PADS_CTL);
+	value |= PADS_CTL_TX_DATA_EN_1L | PADS_CTL_RX_DATA_EN_1L;
+	pads_writel(pcie, value, PADS_CTL);
+
+	/* take the PCIe interface module out of reset */
+	tegra_periph_reset_deassert(pcie->pcie_xclk);
+
+	/* finally enable PCIe */
+	value = afi_readl(pcie, AFI_CONFIGURATION);
+	value |= AFI_CONFIGURATION_EN_FPCI;
+	afi_writel(pcie, value, AFI_CONFIGURATION);
+
+	value = AFI_INTR_EN_INI_SLVERR | AFI_INTR_EN_INI_DECERR |
+		AFI_INTR_EN_TGT_SLVERR | AFI_INTR_EN_TGT_DECERR |
+		AFI_INTR_EN_TGT_WRERR | AFI_INTR_EN_DFPCI_DECERR;
+	afi_writel(pcie, value, AFI_AFI_INTR_ENABLE);
+	afi_writel(pcie, 0xffffffff, AFI_SM_INTR_ENABLE);
+
+	/* don't enable MSI for now, only when needed */
+	afi_writel(pcie, AFI_INTR_MASK_INT_MASK, AFI_INTR_MASK);
+
+	/* disable all exceptions */
+	afi_writel(pcie, 0, AFI_FPCI_ERROR_MASKS);
+
+	return 0;
+}
+
+static void tegra_pcie_power_off(struct tegra_pcie *pcie)
+{
+	int err;
+
+	/* TODO: disable and unprepare clocks? */
+
+	tegra_periph_reset_assert(pcie->pcie_xclk);
+	tegra_periph_reset_assert(pcie->afi_clk);
+	tegra_periph_reset_assert(pcie->pex_clk);
+
+	tegra_powergate_power_off(TEGRA_POWERGATE_PCIE);
+	tegra_pmc_pcie_xclk_clamp(true);
+
+	if (!IS_ERR_OR_NULL(pcie->pex_clk_supply)) {
+		err = regulator_disable(pcie->pex_clk_supply);
+		if (err < 0) {
+			dev_err(pcie->dev,
+				"failed to disable pex-clk regulator: %d\n",
+				err);
+		}
+	}
+
+	if (!IS_ERR_OR_NULL(pcie->vdd_supply)) {
+		err = regulator_disable(pcie->vdd_supply);
+		if (err < 0) {
+			dev_err(pcie->dev,
+				"failed to disable VDD regulator: %d\n", err);
+		}
+	}
+}
+
+static int tegra_pcie_power_on(struct tegra_pcie *pcie)
+{
+	int err;
+
+	tegra_periph_reset_assert(pcie->pcie_xclk);
+	tegra_periph_reset_assert(pcie->afi_clk);
+	tegra_periph_reset_assert(pcie->pex_clk);
+
+	tegra_powergate_power_off(TEGRA_POWERGATE_PCIE);
+	tegra_pmc_pcie_xclk_clamp(true);
+
+	/* enable regulators */
+	if (!IS_ERR_OR_NULL(pcie->vdd_supply)) {
+		err = regulator_enable(pcie->vdd_supply);
+		if (err < 0) {
+			dev_err(pcie->dev,
+				"failed to enable VDD regulator: %d\n", err);
+			return err;
+		}
+	}
+
+	if (!IS_ERR_OR_NULL(pcie->pex_clk_supply)) {
+		err = regulator_enable(pcie->pex_clk_supply);
+		if (err < 0) {
+			dev_err(pcie->dev,
+				"failed to enable pex-clk regulator: %d\n",
+				err);
+			return err;
+		}
+	}
+
+	err = tegra_powergate_sequence_power_up(TEGRA_POWERGATE_PCIE,
+						pcie->pex_clk);
+	if (err) {
+		dev_err(pcie->dev, "powerup sequence failed: %d\n", err);
+		return err;
+	}
+
+	tegra_periph_reset_deassert(pcie->afi_clk);
+
+	tegra_pmc_pcie_xclk_clamp(false);
+
+	clk_prepare_enable(pcie->afi_clk);
+	return clk_prepare_enable(pcie->pll_e);
+}
+
+static int tegra_pcie_clocks_get(struct tegra_pcie *pcie)
+{
+	pcie->pex_clk = devm_clk_get(pcie->dev, "pex");
+	if (IS_ERR(pcie->pex_clk))
+		return PTR_ERR(pcie->pex_clk);
+
+	pcie->afi_clk = devm_clk_get(pcie->dev, "afi");
+	if (IS_ERR(pcie->afi_clk))
+		return PTR_ERR(pcie->afi_clk);
+
+	pcie->pcie_xclk = devm_clk_get(pcie->dev, "pcie_xclk");
+	if (IS_ERR(pcie->pcie_xclk))
+		return PTR_ERR(pcie->pcie_xclk);
+
+	pcie->pll_e = devm_clk_get(pcie->dev, "pll_e");
+	if (IS_ERR(pcie->pll_e))
+		return PTR_ERR(pcie->pll_e);
+
+	return 0;
+}
+
+static int tegra_pcie_get_resources(struct tegra_pcie *pcie)
+{
+	struct platform_device *pdev = to_platform_device(pcie->dev);
+	struct resource *pads, *afi, *res;
+	int err;
+
+	err = tegra_pcie_clocks_get(pcie);
+	if (err) {
+		dev_err(&pdev->dev, "failed to get clocks: %d\n", err);
+		return err;
+	}
+
+	err = tegra_pcie_power_on(pcie);
+	if (err) {
+		dev_err(&pdev->dev, "failed to power up: %d\n", err);
+		return err;
+	}
+
+	/* request and remap controller registers */
+	pads = platform_get_resource_byname(pdev, IORESOURCE_MEM, "pads");
+	if (!pads) {
+		err = -EADDRNOTAVAIL;
+		goto poweroff;
+	}
+
+	afi = platform_get_resource_byname(pdev, IORESOURCE_MEM, "afi");
+	if (!afi) {
+		err = -EADDRNOTAVAIL;
+		goto poweroff;
+	}
+
+	pcie->pads = devm_request_and_ioremap(&pdev->dev, pads);
+	if (!pcie->pads) {
+		err = -EADDRNOTAVAIL;
+		goto poweroff;
+	}
+
+	pcie->afi = devm_request_and_ioremap(&pdev->dev, afi);
+	if (!pcie->afi) {
+		err = -EADDRNOTAVAIL;
+		goto poweroff;
+	}
+
+	/* request and remap configuration space */
+	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "cs");
+	if (!res) {
+		err = -EADDRNOTAVAIL;
+		goto poweroff;
+	}
+
+	pcie->cs = devm_request_mem_region(pcie->dev, res->start,
+					   resource_size(res), res->name);
+	if (!pcie->cs) {
+		err = -EADDRNOTAVAIL;
+		goto poweroff;
+	}
+
+	pcie->csc = devm_iomap_cache_create(pcie->dev, pcie->cs);
+	if (!pcie->csc) {
+		err = -ENOMEM;
+		goto poweroff;
+	}
+
+	/* request interrupt */
+	err = platform_get_irq_byname(pdev, "intr");
+	if (err < 0) {
+		dev_err(&pdev->dev, "failed to get IRQ: %d\n", err);
+		goto poweroff;
+	}
+
+	pcie->irq = err;
+
+	err = devm_request_irq(&pdev->dev, pcie->irq, tegra_pcie_isr,
+			       IRQF_SHARED, "PCIE", pcie);
+	if (err) {
+		dev_err(&pdev->dev, "failed to register IRQ: %d\n", err);
+		goto poweroff;
+	}
+
+	return 0;
+
+poweroff:
+	tegra_pcie_power_off(pcie);
+	return err;
+}
+
+static int tegra_pcie_put_resources(struct tegra_pcie *pcie)
+{
+	tegra_pcie_power_off(pcie);
+	return 0;
+}
+
+static int tegra_pcie_msi_alloc(struct tegra_pcie *pcie)
+{
+	int msi;
+
+	mutex_lock(&pcie->msi->lock);
+
+	msi = find_first_zero_bit(pcie->msi->used, INT_PCI_MSI_NR);
+	if (msi < INT_PCI_MSI_NR)
+		set_bit(msi, pcie->msi->used);
+	else
+		msi = -ENOSPC;
+
+	mutex_unlock(&pcie->msi->lock);
+
+	return msi;
+}
+
+static void tegra_pcie_msi_free(struct tegra_pcie *pcie, unsigned long irq)
+{
+	mutex_lock(&pcie->msi->lock);
+
+	if (!test_bit(irq, pcie->msi->used))
+		dev_err(pcie->dev, "trying to free unused MSI#%lu\n", irq);
+	else
+		clear_bit(irq, pcie->msi->used);
+
+	mutex_unlock(&pcie->msi->lock);
+}
+
+static irqreturn_t tegra_pcie_msi_irq(int irq, void *data)
+{
+	struct tegra_pcie *pcie = data;
+	unsigned int i;
+
+	for (i = 0; i < 8; i++) {
+		unsigned long reg = afi_readl(pcie, AFI_MSI_VEC0 + i * 4);
+
+		while (reg) {
+			unsigned int offset = find_first_bit(&reg, 32);
+			unsigned int index = i * 32 + offset;
+			unsigned int irq;
+
+			/* clear the interrupt */
+			afi_writel(pcie, 1 << offset, AFI_MSI_VEC0 + i * 4);
+
+			irq = irq_find_mapping(pcie->msi->domain, index);
+			if (irq) {
+				if (test_bit(index, pcie->msi->used))
+					generic_handle_irq(irq);
+				else
+					dev_info(pcie->dev, "unhandled MSI\n");
+			} else {
+				/*
+				 * that's weird who triggered this?
+				 * just clear it
+				 */
+				dev_info(pcie->dev, "unexpected MSI\n");
+			}
+
+			/* see if there's any more pending in this vector */
+			reg = afi_readl(pcie, AFI_MSI_VEC0 + i * 4);
+		}
+	}
+
+	return IRQ_HANDLED;
+}
+
+/* called by arch_setup_msi_irqs in drivers/pci/msi.c */
+int arch_setup_msi_irq(struct pci_dev *pdev, struct msi_desc *desc)
+{
+	struct tegra_pcie *pcie = sys_to_pcie(pdev->bus->sysdata);
+	struct msi_msg msg;
+	unsigned int irq;
+	int hwirq;
+
+	hwirq = tegra_pcie_msi_alloc(pcie);
+	if (hwirq < 0)
+		return hwirq;
+
+	irq = irq_create_mapping(pcie->msi->domain, hwirq);
+	if (!irq)
+		return -EINVAL;
+
+	irq_set_msi_desc(irq, desc);
+
+	msg.address_lo = afi_readl(pcie, AFI_MSI_AXI_BAR_ST);
+	/* 32 bit address only */
+	msg.address_hi = 0;
+	msg.data = hwirq;
+
+	write_msi_msg(irq, &msg);
+
+	return 0;
+}
+
+void arch_teardown_msi_irq(unsigned int irq)
+{
+	struct tegra_pcie *pcie = irq_get_chip_data(irq);
+	struct irq_data *d = irq_get_irq_data(irq);
+
+	tegra_pcie_msi_free(pcie, d->hwirq);
+}
+
+static struct irq_chip tegra_pcie_msi_irq_chip = {
+	.name = "Tegra PCIe MSI",
+	.irq_enable = unmask_msi_irq,
+	.irq_disable = mask_msi_irq,
+	.irq_mask = mask_msi_irq,
+	.irq_unmask = unmask_msi_irq,
+};
+
+static int tegra_pcie_msi_map(struct irq_domain *domain, unsigned int irq,
+			      irq_hw_number_t hwirq)
+{
+	irq_set_chip_and_handler(irq, &tegra_pcie_msi_irq_chip,
+				 handle_simple_irq);
+	irq_set_chip_data(irq, domain->host_data);
+	set_irq_flags(irq, IRQF_VALID);
+
+	return 0;
+}
+
+static const struct irq_domain_ops msi_domain_ops = {
+	.map = tegra_pcie_msi_map,
+};
+
+static int tegra_pcie_enable_msi(struct tegra_pcie *pcie)
+{
+	struct platform_device *pdev = to_platform_device(pcie->dev);
+	unsigned long base;
+	int err;
+	u32 reg;
+
+	pcie->msi = devm_kzalloc(&pdev->dev, sizeof(*pcie->msi), GFP_KERNEL);
+	if (!pcie->msi)
+		return -ENOMEM;
+
+	mutex_init(&pcie->msi->lock);
+
+	pcie->msi->domain = irq_domain_add_linear(pcie->dev->of_node,
+						  INT_PCI_MSI_NR,
+						  &msi_domain_ops, pcie);
+	if (!pcie->msi->domain) {
+		dev_err(&pdev->dev, "failed to create IRQ domain\n");
+		return -ENOMEM;
+	}
+
+	err = platform_get_irq_byname(pdev, "msi");
+	if (err < 0) {
+		dev_err(&pdev->dev, "failed to get IRQ: %d\n", err);
+		goto err;
+	}
+
+	pcie->msi->irq = err;
+
+	err = devm_request_irq(&pdev->dev, pcie->msi->irq, tegra_pcie_msi_irq,
+			       0, tegra_pcie_msi_irq_chip.name, pcie);
+	if (err < 0) {
+		dev_err(&pdev->dev, "failed to request IRQ: %d\n", err);
+		goto err;
+	}
+
+	/* setup AFI/FPCI range */
+	pcie->msi->pages = __get_free_pages(GFP_KERNEL, 3);
+	base = virt_to_phys((void *)pcie->msi->pages);
+
+	afi_writel(pcie, base, AFI_MSI_FPCI_BAR_ST);
+	afi_writel(pcie, base, AFI_MSI_AXI_BAR_ST);
+	/* this register is in 4K increments */
+	afi_writel(pcie, 1, AFI_MSI_BAR_SZ);
+
+	/* enable all MSI vectors */
+	afi_writel(pcie, 0xffffffff, AFI_MSI_EN_VEC0);
+	afi_writel(pcie, 0xffffffff, AFI_MSI_EN_VEC1);
+	afi_writel(pcie, 0xffffffff, AFI_MSI_EN_VEC2);
+	afi_writel(pcie, 0xffffffff, AFI_MSI_EN_VEC3);
+	afi_writel(pcie, 0xffffffff, AFI_MSI_EN_VEC4);
+	afi_writel(pcie, 0xffffffff, AFI_MSI_EN_VEC5);
+	afi_writel(pcie, 0xffffffff, AFI_MSI_EN_VEC6);
+	afi_writel(pcie, 0xffffffff, AFI_MSI_EN_VEC7);
+
+	/* and unmask the MSI interrupt */
+	reg = afi_readl(pcie, AFI_INTR_MASK);
+	reg |= AFI_INTR_MASK_MSI_MASK;
+	afi_writel(pcie, reg, AFI_INTR_MASK);
+
+	return 0;
+
+err:
+	irq_domain_remove(pcie->msi->domain);
+	return err;
+}
+
+static int tegra_pcie_disable_msi(struct tegra_pcie *pcie)
+{
+	unsigned int i, irq;
+	u32 value;
+
+	/* mask the MSI interrupt */
+	value = afi_readl(pcie, AFI_INTR_MASK);
+	value &= ~AFI_INTR_MASK_MSI_MASK;
+	afi_writel(pcie, value, AFI_INTR_MASK);
+
+	/* disable all MSI vectors */
+	afi_writel(pcie, 0, AFI_MSI_EN_VEC0);
+	afi_writel(pcie, 0, AFI_MSI_EN_VEC1);
+	afi_writel(pcie, 0, AFI_MSI_EN_VEC2);
+	afi_writel(pcie, 0, AFI_MSI_EN_VEC3);
+	afi_writel(pcie, 0, AFI_MSI_EN_VEC4);
+	afi_writel(pcie, 0, AFI_MSI_EN_VEC5);
+	afi_writel(pcie, 0, AFI_MSI_EN_VEC6);
+	afi_writel(pcie, 0, AFI_MSI_EN_VEC7);
+
+	free_pages(pcie->msi->pages, 3);
+
+	for (i = 0; i < INT_PCI_MSI_NR; i++) {
+		irq = irq_find_mapping(pcie->msi->domain, i);
+		if (irq > 0)
+			irq_dispose_mapping(irq);
+	}
+
+	irq_domain_remove(pcie->msi->domain);
+
+	return 0;
+}
+
+static int tegra_pcie_parse_dt(struct tegra_pcie *pcie)
+{
+	struct device_node *np = pcie->dev->of_node;
+	const __be32 *range = NULL;
+	struct resource res;
+	int err;
+
+	pcie->vdd_supply = devm_regulator_get(pcie->dev, "vdd");
+	if (IS_ERR(pcie->vdd_supply))
+		return PTR_ERR(pcie->vdd_supply);
+
+	pcie->pex_clk_supply = devm_regulator_get(pcie->dev, "pex-clk");
+	if (IS_ERR(pcie->pex_clk_supply))
+		return PTR_ERR(pcie->pex_clk_supply);
+
+	while ((range = of_pci_process_ranges(np, &res, range)) != NULL) {
+		switch (res.flags & IORESOURCE_TYPE_BITS) {
+		case IORESOURCE_IO:
+			memcpy(&pcie->io, &res, sizeof(res));
+			pcie->io.name = "I/O";
+			break;
+
+		case IORESOURCE_MEM:
+			if (res.flags & IORESOURCE_PREFETCH) {
+				memcpy(&pcie->prefetch, &res, sizeof(res));
+				pcie->prefetch.name = "PREFETCH";
+			} else {
+				memcpy(&pcie->mem, &res, sizeof(res));
+				pcie->mem.name = "MEM";
+			}
+			break;
+		}
+	}
+
+	err = of_pci_parse_bus_range(np, &pcie->busn);
+	if (err < 0) {
+		dev_err(pcie->dev, "failed to parse ranges property: %d\n",
+			err);
+		pcie->busn.name = np->name;
+		pcie->busn.start = 0;
+		pcie->busn.end = 0xff;
+		pcie->busn.flags = IORESOURCE_BUS;
+	}
+
+	return 0;
+}
+
+static unsigned long tegra_pcie_port_get_pex_ctrl(struct tegra_pcie_port *port)
+{
+	unsigned long ret = 0;
+
+	switch (port->index) {
+	case 0:
+		ret = AFI_PEX0_CTRL;
+		break;
+
+	case 1:
+		ret = AFI_PEX1_CTRL;
+		break;
+	}
+
+	return ret;
+}
+
+/*
+ * FIXME: If there are no PCIe cards attached, then calling this function
+ * can result in the increase of the bootup time as there are big timeout
+ * loops.
+ */
+#define TEGRA_PCIE_LINKUP_TIMEOUT	200	/* up to 1.2 seconds */
+static bool tegra_pcie_port_check_link(struct tegra_pcie_port *port)
+{
+	unsigned long value, ctrl = tegra_pcie_port_get_pex_ctrl(port);
+	unsigned int retries = 3;
+
+	/* enable reference clock */
+	value = afi_readl(port->pcie, ctrl);
+	value |= AFI_PEX_CTRL_REFCLK_EN;
+	afi_writel(port->pcie, value, ctrl);
+
+	do {
+		unsigned int timeout = TEGRA_PCIE_LINKUP_TIMEOUT;
+
+		do {
+			value = readl(port->base + RP_VEND_XP);
+
+			if (value & RP_VEND_XP_DL_UP)
+				break;
+
+			usleep_range(1000, 1000);
+		} while (--timeout);
+
+		if (!timeout) {
+			dev_err(port->pcie->dev, "link %u down, retrying\n",
+				port->index);
+			goto retry;
+		}
+
+		timeout = TEGRA_PCIE_LINKUP_TIMEOUT;
+
+		do {
+			value = readl(port->base + RP_LINK_CONTROL_STATUS);
+
+			if (value & 0x20000000)
+				return true;
+
+			usleep_range(1000, 1000);
+		} while (--timeout);
+
+retry:
+		/* Pulse the PEX reset */
+		value = afi_readl(port->pcie, ctrl);
+		value |= AFI_PEX_CTRL_RST;
+		afi_writel(port->pcie, value, ctrl);
+
+		usleep_range(1000, 1000);
+
+		value = afi_readl(port->pcie, ctrl);
+		value &= ~AFI_PEX_CTRL_RST;
+		afi_writel(port->pcie, value, ctrl);
+	} while (--retries);
+
+	return false;
+}
+
+static int tegra_pcie_enable(struct tegra_pcie *pcie)
+{
+	struct hw_pci hw;
+
+	memset(&hw, 0, sizeof(hw));
+
+	hw.nr_controllers = 1;
+	hw.private_data = (void **)&pcie;
+	hw.setup = tegra_pcie_setup;
+	hw.scan = tegra_pcie_scan_bus;
+	hw.map_irq = tegra_pcie_map_irq;
+
+	pci_common_init(&hw);
+
+	return 0;
+}
+
+static int tegra_pcie_add_port(struct tegra_pcie *pcie, struct device_node *np)
+{
+	struct tegra_pcie_port *port;
+	struct resource regs;
+	unsigned int index;
+	int err;
+
+	err = of_address_to_resource(np, 0, &regs);
+	if (err < 0) {
+		dev_dbg(pcie->dev, "of_address_to_resource(): %d\n", err);
+		return err;
+	}
+
+	err = of_pci_get_devfn(np);
+	if (err < 0)
+		return err;
+
+	index = PCI_SLOT(err) - 1;
+
+	port = devm_kzalloc(pcie->dev, sizeof(*port), GFP_KERNEL);
+	if (!port)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&port->list);
+	port->index = index;
+	port->pcie = pcie;
+
+	port->base = devm_request_and_ioremap(pcie->dev, &regs);
+	if (!port->base)
+		return -EADDRNOTAVAIL;
+
+	if (!tegra_pcie_port_check_link(port)) {
+		dev_info(pcie->dev, "link %u down, ignoring\n", port->index);
+		return -ENODEV;
+	}
+
+	list_add_tail(&port->list, &pcie->ports);
+
+	return 0;
+}
+
+static int tegra_pcie_probe(struct platform_device *pdev)
+{
+	struct device_node *port;
+	struct tegra_pcie *pcie;
+	int err;
+
+	pcie = devm_kzalloc(&pdev->dev, sizeof(*pcie), GFP_KERNEL);
+	if (!pcie)
+		return -ENOMEM;
+
+	INIT_LIST_HEAD(&pcie->ports);
+	pcie->dev = &pdev->dev;
+
+	err = tegra_pcie_parse_dt(pcie);
+	if (err < 0)
+		return err;
+
+	pcibios_min_mem = 0;
+
+	err = tegra_pcie_get_resources(pcie);
+	if (err < 0) {
+		dev_err(&pdev->dev, "failed to request resources: %d\n", err);
+		return err;
+	}
+
+	err = tegra_pcie_enable_controller(pcie);
+	if (err)
+		goto put_resources;
+
+	/* probe root ports */
+	for_each_child_of_node(pdev->dev.of_node, port) {
+		if (!of_device_is_available(port))
+			continue;
+
+		err = tegra_pcie_add_port(pcie, port);
+		if (err < 0) {
+			dev_err(&pdev->dev, "failed to add port %s: %d\n",
+				port->name, err);
+		}
+	}
+
+	/* setup the AFI address translations */
+	tegra_pcie_setup_translations(pcie);
+
+	if (IS_ENABLED(CONFIG_PCI_MSI)) {
+		err = tegra_pcie_enable_msi(pcie);
+		if (err < 0) {
+			dev_err(&pdev->dev,
+				"failed to enable MSI support: %d\n",
+				err);
+			goto put_resources;
+		}
+	}
+
+	err = tegra_pcie_enable(pcie);
+	if (err < 0) {
+		dev_err(&pdev->dev, "failed to enable PCIe ports: %d\n", err);
+		goto disable_msi;
+	}
+
+	platform_set_drvdata(pdev, pcie);
+	return 0;
+
+disable_msi:
+	if (IS_ENABLED(CONFIG_PCI_MSI))
+		tegra_pcie_disable_msi(pcie);
+put_resources:
+	tegra_pcie_put_resources(pcie);
+	return err;
+}
+
+static int tegra_pcie_remove(struct platform_device *pdev)
+{
+	struct tegra_pcie *pcie = platform_get_drvdata(pdev);
+	int err;
+
+	if (IS_ENABLED(CONFIG_PCI_MSI)) {
+		err = tegra_pcie_disable_msi(pcie);
+		if (err < 0)
+			return err;
+	}
+
+	err = tegra_pcie_put_resources(pcie);
+	if (err < 0)
+		return err;
+
+	return 0;
+}
+
+static const struct of_device_id tegra_pcie_of_match[] = {
+	{ .compatible = "nvidia,tegra20-pcie", },
+	{ },
+};
+
+static struct platform_driver tegra_pcie_driver = {
+	.driver = {
+		.name = "tegra-pcie",
+		.owner = THIS_MODULE,
+		.of_match_table = tegra_pcie_of_match,
+	},
+	.probe = tegra_pcie_probe,
+	.remove = tegra_pcie_remove,
+};
+module_platform_driver(tegra_pcie_driver);
-- 
1.8.1

[PATCH 11/14] ARM: tegra: tamonten: Add PCIe support

[Thierry Reding]

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 arch/arm/boot/dts/tegra20-tamonten.dtsi | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/tegra20-tamonten.dtsi b/arch/arm/boot/dts/tegra20-tamonten.dtsi
index 7edfb9a..0053c72 100644
--- a/arch/arm/boot/dts/tegra20-tamonten.dtsi
+++ b/arch/arm/boot/dts/tegra20-tamonten.dtsi
@@ -439,7 +439,7 @@
 					regulator-always-on;
 				};
 
-				ldo0 {
+				pci_clk_reg: ldo0 {
 					regulator-name = "vdd_ldo0,vddio_pex_clk";
 					regulator-min-microvolt = <3300000>;
 					regulator-max-microvolt = <3300000>;
-- 
1.8.1

[PATCH 12/14] ARM: tegra: tec: Add PCIe support

[Thierry Reding]

Enable the first PCIe root port which is connected to an FPGA on the
Tamonten Evaluation Carrier and add device nodes for each of the PCI
endpoints available in the standard configuration.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 arch/arm/boot/dts/tegra20-tec.dts | 198 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 198 insertions(+)

diff --git a/arch/arm/boot/dts/tegra20-tec.dts b/arch/arm/boot/dts/tegra20-tec.dts
index 25f70ee..b75f979 100644
--- a/arch/arm/boot/dts/tegra20-tec.dts
+++ b/arch/arm/boot/dts/tegra20-tec.dts
@@ -60,6 +60,204 @@
 		};
 	};
 
+	pcie-controller {
+		vdd-supply = <&pci_vdd_reg>;
+		pex-clk-supply = <&pci_clk_reg>;
+		status = "okay";
+
+		pci@1,0 {
+			bus-range = <0x01 0x0a>;
+			status = "okay";
+
+			pci@0,0 {
+				reg = <0x010000 0 0 0 0>;
+				bus-range = <0x02 0x0a>;
+
+				compatible = "plda,pcie";
+				device_type = "pci";
+
+				#address-cells = <3>;
+				#size-cells = <2>;
+
+				pci@0,0 {
+					reg = <0x020000 0 0 0 0>;
+					bus-range = <0x03 0x03>;
+
+					compatible = "ad,pcie";
+					device_type = "pci";
+
+					#address-cells = <3>;
+					#size-cells = <2>;
+
+					pci-test-ep@0,0 {
+						compatible = "ad,pcietest";
+						reg = <0x030000 0 0 0 0>;
+					};
+				};
+
+				pci@1,0 {
+					reg = <0x020800 0 0 0 0>;
+					bus-range = <0x04 0x04>;
+
+					compatible = "ad,pcie";
+					device_type = "pci";
+
+					#address-cells = <3>;
+					#size-cells = <2>;
+
+					pci@0,0 {
+						compatible = "opencores,spi";
+						reg = <0x040000 0 0 0 0>;
+
+						#address-cells = <1>;
+						#size-cells = <0>;
+
+						clock-frequency = <66000000>;
+
+						m25p80@0 {
+							compatible = "m25p80";
+							reg = <0>;
+
+							spi-max-frequency = <25000000>;
+
+							#address-cells = <1>;
+							#size-cells = <1>;
+
+							partition@0 {
+								reg = <0x00000000 0x00400000>;
+								label = "FPGA";
+							};
+
+							partition@400000 {
+								reg = <0x00400000 0x00400000>;
+								label = "FPGA (gold)";
+								read-only;
+							};
+						};
+					};
+				};
+
+				pci@2,0 {
+					reg = <0x021000 0 0 0 0>;
+					bus-range = <0x05 0x05>;
+
+					compatible = "ad,pcie";
+					device_type = "pci";
+
+					#address-cells = <3>;
+					#size-cells = <2>;
+
+					pci-i2c-ep@0,0 {
+						compatible = "opencores,i2c";
+						reg = <0x050000 0 0 0 0>;
+
+						clock-frequency = <66000000>;
+						reg-io-width = <4>;
+						reg-shift = <3>;
+					};
+				};
+
+				pci@3,0 {
+					reg = <0x021800 0 0 0 0>;
+					bus-range = <0x06 0x06>;
+
+					compatible = "ad,pcie";
+					device_type = "pci";
+
+					#address-cells = <3>;
+					#size-cells = <2>;
+
+					pci@0,0 {
+						compatible = "ad,gpio-ad64p";
+						reg = <0x060000 0 0 0 0>;
+
+						gpio-controller;
+						#gpio-cells = <2>;
+
+						interrupt-controller;
+						#interrupt-cells = <2>;
+					};
+				};
+
+				pci@4,0 {
+					reg = <0x022000 0 0 0 0>;
+					bus-range = <0x07 0x07>;
+
+					compatible = "ad,pcie";
+					device_type = "pci";
+
+					#address-cells = <3>;
+					#size-cells = <2>;
+
+					pci@0,0 {
+						compatible = "opencores,uart";
+						reg = <0x070000 0 0 0 0>;
+					};
+				};
+
+				pci@5,0 {
+					reg = <0x022800 0 0 0 0>;
+					bus-range = <0x08 0x08>;
+
+					compatible = "ad,pcie";
+					device_type = "pci";
+
+					#address-cells = <3>;
+					#size-cells = <2>;
+
+					pci@0,0 {
+						compatible = "opencores,ethmac";
+						reg = <0x080000 0 0 0 0>;
+					};
+				};
+
+				pci@6,0 {
+					reg = <0x023000 0 0 0 0>;
+					bus-range = <0x09 0x09>;
+
+					compatible = "ad,pcie";
+					device_type = "pci";
+
+					#address-cells = <3>;
+					#size-cells = <2>;
+
+					pci@0,0 {
+						compatible = "opencores,can";
+						reg = <0x090000 0 0 0 0>;
+					};
+				};
+
+				pci@7,0 {
+					reg = <0x023800 0 0 0 0>;
+					bus-range = <0x0a 0x0a>;
+
+					compatible = "ad,pcie";
+					device_type = "pci";
+
+					#address-cells = <3>;
+					#size-cells = <2>;
+
+					pci@0,0 {
+						compatible = "opencores,can";
+						reg = <0x0a0000 0 0 0 0>;
+					};
+				};
+			};
+		};
+	};
+
+	regulators {
+		pci_vdd_reg: regulator@1 {
+			compatible = "regulator-fixed";
+			reg = <1>;
+			regulator-name = "vdd_1v05";
+			regulator-min-microvolt = <1050000>;
+			regulator-max-microvolt = <1050000>;
+			gpio = <&pmic 2 0>;
+			enable-active-high;
+		};
+	};
+
 	sound {
 		compatible = "ad,tegra-audio-wm8903-tec",
 			     "nvidia,tegra-audio-wm8903";
-- 
1.8.1

[PATCH 13/14] ARM: tegra: harmony: Initialize PCIe from DT

[Thierry Reding]

With the device tree support in place, probe the PCIe controller from
the device tree and remove the corresponding workaround in the board
file.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
 arch/arm/boot/dts/tegra20-harmony.dts    | 20 ++++++--
 arch/arm/mach-tegra/Makefile             |  2 -
 arch/arm/mach-tegra/board-dt-tegra20.c   | 12 -----
 arch/arm/mach-tegra/board-harmony-pcie.c | 88 --------------------------------
 4 files changed, 16 insertions(+), 106 deletions(-)
 delete mode 100644 arch/arm/mach-tegra/board-harmony-pcie.c

diff --git a/arch/arm/boot/dts/tegra20-harmony.dts b/arch/arm/boot/dts/tegra20-harmony.dts
index b506de9..e5b3aec 100644
--- a/arch/arm/boot/dts/tegra20-harmony.dts
+++ b/arch/arm/boot/dts/tegra20-harmony.dts
@@ -335,7 +335,7 @@
 					regulator-always-on;
 				};
 
-				ldo0 {
+				pci_clk_reg: ldo0 {
 					regulator-name = "vdd_ldo0,vddio_pex_clk";
 					regulator-min-microvolt = <3300000>;
 					regulator-max-microvolt = <3300000>;
@@ -419,6 +419,20 @@
 		nvidia,invert-interrupt;
 	};
 
+	pcie-controller {
+		pex-clk-supply = <&pci_clk_reg>;
+		vdd-supply = <&pci_vdd_reg>;
+		status = "okay";
+
+		pci@1,0 {
+			status = "okay";
+		};
+
+		pci@2,0 {
+			status = "okay";
+		};
+	};
+
 	usb@c5000000 {
 		status = "okay";
 	};
@@ -481,7 +495,7 @@
 			enable-active-high;
 		};
 
-		regulator@3 {
+		pci_vdd_reg: regulator@3 {
 			compatible = "regulator-fixed";
 			reg = <3>;
 			regulator-name = "vdd_1v05";
@@ -489,8 +503,6 @@
 			regulator-max-microvolt = <1050000>;
 			gpio = <&pmic 2 0>;
 			enable-active-high;
-			/* Hack until board-harmony-pcie.c is removed */
-			status = "disabled";
 		};
 
 		regulator@4 {
diff --git a/arch/arm/mach-tegra/Makefile b/arch/arm/mach-tegra/Makefile
index 463a77e..9003d05 100644
--- a/arch/arm/mach-tegra/Makefile
+++ b/arch/arm/mach-tegra/Makefile
@@ -33,6 +33,4 @@ obj-$(CONFIG_CPU_FREQ)                  += cpu-tegra.o
 obj-$(CONFIG_ARCH_TEGRA_2x_SOC)		+= board-dt-tegra20.o
 obj-$(CONFIG_ARCH_TEGRA_3x_SOC)		+= board-dt-tegra30.o
 
-obj-$(CONFIG_ARCH_TEGRA_2x_SOC)		+= board-harmony-pcie.o
-
 obj-$(CONFIG_ARCH_TEGRA_2x_SOC)		+= board-paz00.o
diff --git a/arch/arm/mach-tegra/board-dt-tegra20.c b/arch/arm/mach-tegra/board-dt-tegra20.c
index 5644dfb..9e8256ab 100644
--- a/arch/arm/mach-tegra/board-dt-tegra20.c
+++ b/arch/arm/mach-tegra/board-dt-tegra20.c
@@ -152,17 +152,6 @@ static void __init trimslice_init(void)
 #endif
 }
 
-static void __init harmony_init(void)
-{
-#ifdef CONFIG_TEGRA_PCI
-	int ret;
-
-	ret = harmony_pcie_init();
-	if (ret)
-		pr_err("harmony_pcie_init() failed: %d\n", ret);
-#endif
-}
-
 static void __init paz00_init(void)
 {
 	tegra_paz00_wifikill_init();
@@ -173,7 +162,6 @@ static struct {
 	void (*init)(void);
 } board_init_funcs[] = {
 	{ "compulab,trimslice", trimslice_init },
-	{ "nvidia,harmony", harmony_init },
 	{ "compal,paz00", paz00_init },
 };
 
diff --git a/arch/arm/mach-tegra/board-harmony-pcie.c b/arch/arm/mach-tegra/board-harmony-pcie.c
deleted file mode 100644
index 94c0529..0000000
--- a/arch/arm/mach-tegra/board-harmony-pcie.c
+++ /dev/null
@@ -1,88 +0,0 @@
-/*
- * arch/arm/mach-tegra/board-harmony-pcie.c
- *
- * Copyright (C) 2010 CompuLab, Ltd.
- * Mike Rapoport <mike@compulab.co.il>
- *
- * This software is licensed under the terms of the GNU General Public
- * License version 2, as published by the Free Software Foundation, and
- * may be copied, distributed, and modified under those terms.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- */
-
-#include <linux/kernel.h>
-#include <linux/gpio.h>
-#include <linux/err.h>
-#include <linux/of_gpio.h>
-#include <linux/regulator/consumer.h>
-
-#include <asm/mach-types.h>
-
-#include <mach/pci-tegra.h>
-
-#include "board.h"
-
-#ifdef CONFIG_TEGRA_PCI
-static int harmony_pcie_board_init(struct platform_device *pdev)
-{
-	struct device_node *np;
-	int en_vdd_1v05;
-	struct regulator *regulator = NULL;
-	int err;
-
-	np = of_find_node_by_path("/regulators/regulator@3");
-	if (!np) {
-		pr_err("%s: of_find_node_by_path failed\n", __func__);
-		return -ENODEV;
-	}
-
-	en_vdd_1v05 = of_get_named_gpio(np, "gpio", 0);
-	if (en_vdd_1v05 < 0) {
-		pr_err("%s: of_get_named_gpio failed: %d\n", __func__,
-		       en_vdd_1v05);
-		return en_vdd_1v05;
-	}
-
-	err = gpio_request(en_vdd_1v05, "EN_VDD_1V05");
-	if (err) {
-		pr_err("%s: gpio_request failed: %d\n", __func__, err);
-		return err;
-	}
-
-	gpio_direction_output(en_vdd_1v05, 1);
-
-	regulator = regulator_get(NULL, "vdd_ldo0,vddio_pex_clk");
-	if (IS_ERR_OR_NULL(regulator)) {
-		pr_err("%s: regulator_get failed: %d\n", __func__,
-		       (int)PTR_ERR(regulator));
-		goto err_reg;
-	}
-
-	regulator_enable(regulator);
-
-	return 0;
-
-err_reg:
-	gpio_free(en_vdd_1v05);
-
-	return err;
-}
-
-int __init harmony_pcie_init(void)
-{
-	tegra_pcie_pdata.init = harmony_pcie_board_init;
-	platform_device_register(&tegra_pcie_device);
-
-	return 0;
-}
-#else
-int __init harmony_pcie_init(void)
-{
-	return 0;
-}
-#endif
-- 
1.8.1

[PATCH 14/14] ARM: tegra: trimslice: Initialize PCIe from DT

[Thierry Reding]

With the device tree support in place, probe the PCIe controller from
the device tree and remove the corresponding workaround in the board
file.

Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
---
Changes in v3:
- update for new DT binding

Changes in v2:
- new patch
---
 arch/arm/boot/dts/tegra20-trimslice.dts | 12 ++++++++++++
 arch/arm/mach-tegra/board-dt-tegra20.c  |  8 --------
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/arch/arm/boot/dts/tegra20-trimslice.dts b/arch/arm/boot/dts/tegra20-trimslice.dts
index 4b6c486..ebb4c17 100644
--- a/arch/arm/boot/dts/tegra20-trimslice.dts
+++ b/arch/arm/boot/dts/tegra20-trimslice.dts
@@ -301,6 +301,18 @@
 		};
 	};
 
+	pcie-controller {
+		status = "okay";
+
+		pci@0 {
+			status = "okay";
+		};
+
+		pci@1 {
+			status = "okay";
+		};
+	};
+
 	usb@c5000000 {
 		status = "okay";
 		nvidia,vbus-gpio = <&gpio 170 0>; /* gpio PV2 */
diff --git a/arch/arm/mach-tegra/board-dt-tegra20.c b/arch/arm/mach-tegra/board-dt-tegra20.c
index 9e8256ab..67090fe 100644
--- a/arch/arm/mach-tegra/board-dt-tegra20.c
+++ b/arch/arm/mach-tegra/board-dt-tegra20.c
@@ -145,13 +145,6 @@ static void __init tegra_dt_init(void)
 				tegra20_auxdata_lookup, NULL);
 }
 
-static void __init trimslice_init(void)
-{
-#ifdef CONFIG_TEGRA_PCI
-	platform_device_register(&tegra_pcie_device);
-#endif
-}
-
 static void __init paz00_init(void)
 {
 	tegra_paz00_wifikill_init();
@@ -161,7 +154,6 @@ static struct {
 	char *machine;
 	void (*init)(void);
 } board_init_funcs[] = {
-	{ "compulab,trimslice", trimslice_init },
 	{ "compal,paz00", paz00_init },
 };
 
-- 
1.8.1

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Arnd Bergmann]

On Wednesday 09 January 2013, Thierry Reding wrote:
> The I/O map cache is used to map large regions of physical memory in
> smaller chunks to avoid running out of vmalloc()/ioremap() space.
> 
> Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>

Can you explain why this is necessary and which problem it solves?
The implementation looks reasonable, but it's not clear to me if
we really need this new interface.

In what cases does it actually save ioremap space?

	Arnd

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Arnd Bergmann]

On Wednesday 09 January 2013, Thierry Reding wrote:
> Move the PCIe driver from arch/arm/mach-tegra into the drivers/pci/host
> directory. The motivation is to collect various host controller drivers
> in the same location in order to facilitate refactoring.
> 
> The Tegra PCIe driver has been largely rewritten, both in order to turn
> it into a proper platform driver and to add MSI (based on code by
> Krishna Kishore <kthota@nvidia.com>) as well as device tree support.
> 
> Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>

Can you split this patch into two patches, one that moves the file,
and another one that does all the changes? It's a little hard to
review when I can't easily see which code is actually new here.

	Arnd

Re: [PATCH 11/14] ARM: tegra: tamonten: Add PCIe support

[Arnd Bergmann]

On Wednesday 09 January 2013, Thierry Reding wrote:
> Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> ---
>  arch/arm/boot/dts/tegra20-tamonten.dtsi | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

It's not clear how this one-line patch adds PCIe support to the
platform. Are you missing the actual change here?

	Arnd

Re: [PATCH 00/14] Rewrite Tegra PCIe driver

[Thomas Petazzoni]

Dear Thierry Reding,

On Wed,  9 Jan 2013 21:43:00 +0100, Thierry Reding wrote:
> This patch series contains an almost complete rewrite of the Tegra PCIe
> driver. The code is moved to the drivers/pci/host directory and turned
> into a proper platform driver, adding MSI and DT support while at it.
> Other PCI host controller drivers can be added to that directory in an
> attempt to make it easier to factor out common code.

Thanks!

I have started basing the Marvell PCIe code on some of your earlier
versions. But apparently in this final version, you no longer have the
emulated Host bridge. Why so?

For the Marvell PCIe code, I've used your emulated Host bridge, and
added an emulated PCI-to-PCI bridge implementation, in order to get the
following hierarchy:

 + Host Bridge
   + PCI-to-PCI bridge
     + PCI Device
   + PCI-to-PCI bridge
     + PCI device

So, I instantiate one unique emulated Host Bridge, and then one
emulated PCI-to-PCI Bridge for each PCIe interface that I have.

The nice thing about that is that I can then read the configuration
space of the PCI-to-PCI bridge to find out how much I/O space and
memory space is needed for the device connected to this interface, and
at which address is has been mapped. This greatly helps my "address
decoding" problem, and removes the ad-hoc virtual space allocator I had
written.

Is there a reason for having given up on this idea? Is there still a
hope for a different PCIe implementation to use this idea?

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Russell King - ARM Linux]

On Wed, Jan 09, 2013 at 09:43:05PM +0100, Thierry Reding wrote:
> The I/O map cache is used to map large regions of physical memory in
> smaller chunks to avoid running out of vmalloc()/ioremap() space.
> 
> Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>

We already have a means where we record the mappings which ioremap()
creates.  If you look at /proc/vmallocinfo, you'll notice lines such
as:

0xf7b72000-0xf7b74000    8192 e1000_probe+0x291/0xa68 [e1000e] phys=fc025000 ioremap

which gives you the virtual address range, physical address and type
of the mapping.  Why do we need a duplicated data structure?

Moreover, you seem to suggest that you want to break up a large
ioremap() mapping into several smaller mappings.  Why?  The idea
behind ioremap() is that this relationship holds true:

	ptr = ioremap(cookie + n, size);

For any 'n' in the range 0 .. size, the location shall be accessible
via ptr + n when using the IO space accessors.  If you're going to
break up a mapping into several smaller ones, this no longer holds
true.

If the problem is that you're ioremapping huge address ranges because
you're passing larger-than-required resources to devices, then that's
part of the problem too.

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]

On Wed, Jan 09, 2013 at 09:19:56PM +0000, Arnd Bergmann wrote:
> On Wednesday 09 January 2013, Thierry Reding wrote:
> > The I/O map cache is used to map large regions of physical memory in
> > smaller chunks to avoid running out of vmalloc()/ioremap() space.
> > 
> > Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> 
> Can you explain why this is necessary and which problem it solves?
> The implementation looks reasonable, but it's not clear to me if
> we really need this new interface.
> 
> In what cases does it actually save ioremap space?

I briefly described this in the cover letter of this series and didn't
realize I hadn't put more information in this patch's commit message.

What happens on Tegra is that we need to map 256 MiB of physical memory
to access all the PCIe extended configuration space. However, ioremap()
on such a large region fails if not enough vmalloc() space is available.

This was observed when somebody tested this on CardHu which has a 1 GiB
of RAM and therefore remapping the full 256 MiB fails.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1701 bytes --]

On Wed, Jan 09, 2013 at 09:28:47PM +0000, Russell King - ARM Linux wrote:
> On Wed, Jan 09, 2013 at 09:43:05PM +0100, Thierry Reding wrote:
> > The I/O map cache is used to map large regions of physical memory in
> > smaller chunks to avoid running out of vmalloc()/ioremap() space.
> > 
> > Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> 
> We already have a means where we record the mappings which ioremap()
> creates.  If you look at /proc/vmallocinfo, you'll notice lines such
> as:
> 
> 0xf7b72000-0xf7b74000    8192 e1000_probe+0x291/0xa68 [e1000e] phys=fc025000 ioremap
> 
> which gives you the virtual address range, physical address and type
> of the mapping.  Why do we need a duplicated data structure?
> 
> Moreover, you seem to suggest that you want to break up a large
> ioremap() mapping into several smaller mappings.  Why?  The idea
> behind ioremap() is that this relationship holds true:
> 
> 	ptr = ioremap(cookie + n, size);
> 
> For any 'n' in the range 0 .. size, the location shall be accessible
> via ptr + n when using the IO space accessors.  If you're going to
> break up a mapping into several smaller ones, this no longer holds
> true.
> 
> If the problem is that you're ioremapping huge address ranges because
> you're passing larger-than-required resources to devices, then that's
> part of the problem too.

The resource is not larger than required. It's just that in order to
address the extended configuration space of all PCI devices we require a
full 256 MiB window.

What this patch tries to achieve is allow a driver to take a large
region and ioremap() it page by page on an as needed basis.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

On Wed, Jan 09, 2013 at 09:22:07PM +0000, Arnd Bergmann wrote:
> On Wednesday 09 January 2013, Thierry Reding wrote:
> > Move the PCIe driver from arch/arm/mach-tegra into the drivers/pci/host
> > directory. The motivation is to collect various host controller drivers
> > in the same location in order to facilitate refactoring.
> > 
> > The Tegra PCIe driver has been largely rewritten, both in order to turn
> > it into a proper platform driver and to add MSI (based on code by
> > Krishna Kishore <kthota@nvidia.com>) as well as device tree support.
> > 
> > Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> 
> Can you split this patch into two patches, one that moves the file,
> and another one that does all the changes? It's a little hard to
> review when I can't easily see which code is actually new here.

Stephen suggested I post this as removal/addition because pretty much
everything changed. Reviewing the individual changes would be more
confusing than actually reviewing a new driver.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Arnd Bergmann]

On Wednesday 09 January 2013, Thierry Reding wrote:
> Stephen suggested I post this as removal/addition because pretty much
> everything changed. Reviewing the individual changes would be more
> confusing than actually reviewing a new driver.
> 

Ok, fair enough.

	Arnd

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Arnd Bergmann]

On Wednesday 09 January 2013, Thierry Reding wrote:
> What happens on Tegra is that we need to map 256 MiB of physical memory
> to access all the PCIe extended configuration space. However, ioremap()
> on such a large region fails if not enough vmalloc() space is available.
> 
> This was observed when somebody tested this on CardHu which has a 1 GiB
> of RAM and therefore remapping the full 256 MiB fails.

Hmm, config space accesses are fairly rare and generally not expected
to be fast, and 256 MB is really a huge waste of virtual address space,
so I agree that just ioremapping the entire space is not a good
solution.

However, it's not clear that a cache is necessary. Have you measured
a significant performance benefit of this implementation over just
iorempping and unmapping a single page for every config space access?
Have you checked if the hardware supports an alternative config
space access mechanism that does not depend on a huge address range?
A lot of them provide an index/data register pair somewhere, as the
original PC implementation did.

Even if we actually want a cache, how about a private implementation
that just remembers a single page in LRU? I doubt that there are
more drivers that would benefit from a generalized version that you
provide.

	Arnd

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Stephen Warren]

On 01/09/2013 03:10 PM, Arnd Bergmann wrote:
> On Wednesday 09 January 2013, Thierry Reding wrote:
>> What happens on Tegra is that we need to map 256 MiB of physical memory
>> to access all the PCIe extended configuration space. However, ioremap()
>> on such a large region fails if not enough vmalloc() space is available.
>>
>> This was observed when somebody tested this on CardHu which has a 1 GiB
>> of RAM and therefore remapping the full 256 MiB fails.
...
> Have you checked if the hardware supports an alternative config
> space access mechanism that does not depend on a huge address range?
> A lot of them provide an index/data register pair somewhere, as the
> original PC implementation did.

That would be nice, but I've talked to the HW engineers, and there's no
indication that any alternative mechanism exists.

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Jason Gunthorpe]

On Wed, Jan 09, 2013 at 04:12:31PM -0700, Stephen Warren wrote:
> On 01/09/2013 03:10 PM, Arnd Bergmann wrote:
> > On Wednesday 09 January 2013, Thierry Reding wrote:
> >> What happens on Tegra is that we need to map 256 MiB of physical memory
> >> to access all the PCIe extended configuration space. However, ioremap()
> >> on such a large region fails if not enough vmalloc() space is available.
> >>
> >> This was observed when somebody tested this on CardHu which has a 1 GiB
> >> of RAM and therefore remapping the full 256 MiB fails.
> ...
> > Have you checked if the hardware supports an alternative config
> > space access mechanism that does not depend on a huge address range?
> > A lot of them provide an index/data register pair somewhere, as the
> > original PC implementation did.
> 
> That would be nice, but I've talked to the HW engineers, and there's no
> indication that any alternative mechanism exists.

It seems to be convention that extended config space is often only
accessible through mmio space, that was true on x86 last I checked
too..

You could decrease the size of the mapping to only span the bus
numbers that are configured for use via DT.

Are there any concerns about these config registers being accessed
from a context where a new mapping can't be made? Interrupt? Machine
Check? PCI-E Advanced Error Reporting?

Cheers,
Jason

Re: [PATCH 00/14] Rewrite Tegra PCIe driver

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2598 bytes --]

On Wed, Jan 09, 2013 at 10:25:17PM +0100, Thomas Petazzoni wrote:
> Dear Thierry Reding,
> 
> On Wed,  9 Jan 2013 21:43:00 +0100, Thierry Reding wrote:
> > This patch series contains an almost complete rewrite of the Tegra PCIe
> > driver. The code is moved to the drivers/pci/host directory and turned
> > into a proper platform driver, adding MSI and DT support while at it.
> > Other PCI host controller drivers can be added to that directory in an
> > attempt to make it easier to factor out common code.
> 
> Thanks!
> 
> I have started basing the Marvell PCIe code on some of your earlier
> versions. But apparently in this final version, you no longer have the
> emulated Host bridge. Why so?

The reason is that with the latest bindings the matching of root ports
to device tree nodes works as-is and nothing else indicates that the
emulated host bridge is actually required to make any of this work. So
in order not to introduce unneeded code I've left it out for now. If
somebody decides that we actually need this host bridge (for standards
compliance or whatnot) it could easily be added back.

However, before the emulated bridge implementation can be merged I think
the PCI ID issue needs to be resolved.

> For the Marvell PCIe code, I've used your emulated Host bridge, and
> added an emulated PCI-to-PCI bridge implementation, in order to get the
> following hierarchy:
> 
>  + Host Bridge
>    + PCI-to-PCI bridge
>      + PCI Device
>    + PCI-to-PCI bridge
>      + PCI device
> 
> So, I instantiate one unique emulated Host Bridge, and then one
> emulated PCI-to-PCI Bridge for each PCIe interface that I have.

Oh dear, that's even worse than on Tegra. The Marvell hardware doesn't
even expose the root ports as PCI devices on the bus?

> The nice thing about that is that I can then read the configuration
> space of the PCI-to-PCI bridge to find out how much I/O space and
> memory space is needed for the device connected to this interface, and
> at which address is has been mapped. This greatly helps my "address
> decoding" problem, and removes the ad-hoc virtual space allocator I had
> written.
> 
> Is there a reason for having given up on this idea? Is there still a
> hope for a different PCIe implementation to use this idea?

I suppose that in your case it really makes sense because you already
need the emulated PCI-to-PCI bridges and therefore adding an emulated
host bridge doesn't add much. As I said, for Tegra everything still
works without, so I didn't see a reason to add needless code.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2271 bytes --]

On Wed, Jan 09, 2013 at 10:10:49PM +0000, Arnd Bergmann wrote:
> On Wednesday 09 January 2013, Thierry Reding wrote:
> > What happens on Tegra is that we need to map 256 MiB of physical memory
> > to access all the PCIe extended configuration space. However, ioremap()
> > on such a large region fails if not enough vmalloc() space is available.
> > 
> > This was observed when somebody tested this on CardHu which has a 1 GiB
> > of RAM and therefore remapping the full 256 MiB fails.
> 
> Hmm, config space accesses are fairly rare and generally not expected
> to be fast, and 256 MB is really a huge waste of virtual address space,
> so I agree that just ioremapping the entire space is not a good
> solution.
> 
> However, it's not clear that a cache is necessary. Have you measured
> a significant performance benefit of this implementation over just
> iorempping and unmapping a single page for every config space access?

No, I had not actually implemented it that way because I figured I might
just as well implement something generic with the added benefit that
most remapping operations would be cached automatically since the PCI
enumeration algorithms usually access the configuration space of a
single device at a time, so it actually maps to the best case for an LRU
based cache approach.

> Even if we actually want a cache, how about a private implementation
> that just remembers a single page in LRU? I doubt that there are
> more drivers that would benefit from a generalized version that you
> provide.

I can move the code to the Tegra PCIe driver, but there's quite a bit of
code that isn't actually related to the PCI functionality and I'd really
like to avoid cluttering the driver with this implementation. Keeping it
in a more central location will certainly increase the code's visibility
and make it easier for other potential users to find.

Also I just noticed that I hadn't actually added a parameter to the
iomap_cache_create() function to specify the maximum number of pages, so
currently the code only uses a single page anyway. It should be trivial
to change. I guess performance was good enough with a single page that I
didn't have a reason to increase the maximum number of pages.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1930 bytes --]

On Wed, Jan 09, 2013 at 04:17:58PM -0700, Jason Gunthorpe wrote:
> On Wed, Jan 09, 2013 at 04:12:31PM -0700, Stephen Warren wrote:
> > On 01/09/2013 03:10 PM, Arnd Bergmann wrote:
> > > On Wednesday 09 January 2013, Thierry Reding wrote:
> > >> What happens on Tegra is that we need to map 256 MiB of physical memory
> > >> to access all the PCIe extended configuration space. However, ioremap()
> > >> on such a large region fails if not enough vmalloc() space is available.
> > >>
> > >> This was observed when somebody tested this on CardHu which has a 1 GiB
> > >> of RAM and therefore remapping the full 256 MiB fails.
> > ...
> > > Have you checked if the hardware supports an alternative config
> > > space access mechanism that does not depend on a huge address range?
> > > A lot of them provide an index/data register pair somewhere, as the
> > > original PC implementation did.
> > 
> > That would be nice, but I've talked to the HW engineers, and there's no
> > indication that any alternative mechanism exists.
> 
> It seems to be convention that extended config space is often only
> accessible through mmio space, that was true on x86 last I checked
> too..
> 
> You could decrease the size of the mapping to only span the bus
> numbers that are configured for use via DT.

That won't work, unfortunately. The mapping is such that the bus number
is not encoded in the uppermost bits, the extended register number is.
So the only thing that we could do is decrease the size of the extended
register space for *all* devices.

> Are there any concerns about these config registers being accessed
> from a context where a new mapping can't be made? Interrupt? Machine
> Check? PCI-E Advanced Error Reporting?

I haven't checked but I would expect configuration space accesses to not
happen in interrupt context. Usually they are limited to enumeration and
driver probe.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 00/14] Rewrite Tegra PCIe driver

[Thomas Petazzoni]

Dear Thierry Reding,

On Thu, 10 Jan 2013 07:55:37 +0100, Thierry Reding wrote:

> The reason is that with the latest bindings the matching of root ports
> to device tree nodes works as-is and nothing else indicates that the
> emulated host bridge is actually required to make any of this work. So
> in order not to introduce unneeded code I've left it out for now. If
> somebody decides that we actually need this host bridge (for standards
> compliance or whatnot) it could easily be added back.

Ok.

> However, before the emulated bridge implementation can be merged I think
> the PCI ID issue needs to be resolved.

Indeed. I am not sure yet how to solve that, though.

> > So, I instantiate one unique emulated Host Bridge, and then one
> > emulated PCI-to-PCI Bridge for each PCIe interface that I have.
> 
> Oh dear, that's even worse than on Tegra. The Marvell hardware doesn't
> even expose the root ports as PCI devices on the bus?

My understanding is that no, it doesn't, but I am still figuring out
many things in this PCIe topic.

> I suppose that in your case it really makes sense because you already
> need the emulated PCI-to-PCI bridges and therefore adding an emulated
> host bridge doesn't add much. As I said, for Tegra everything still
> works without, so I didn't see a reason to add needless code.

Ok, thanks!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Arnd Bergmann]

On Thursday 10 January 2013, Thierry Reding wrote:
> On Wed, Jan 09, 2013 at 04:17:58PM -0700, Jason Gunthorpe wrote:
> > On Wed, Jan 09, 2013 at 04:12:31PM -0700, Stephen Warren wrote:
> > You could decrease the size of the mapping to only span the bus
> > numbers that are configured for use via DT.
> 
> That won't work, unfortunately. The mapping is such that the bus number
> is not encoded in the uppermost bits, the extended register number is.
> So the only thing that we could do is decrease the size of the extended
> register space for *all* devices.

But you could still a method to map 16 separate areas per bus, each 65536
bytes long, which results in 1MB per bus. That is probably ok, since
very few systems have more than a handful of buses in practice.

In theory, doing static mappings on a per-page base would let you
do 16 devices at a time, but it's probably worth doing at this fine
granularity.

> > Are there any concerns about these config registers being accessed
> > from a context where a new mapping can't be made? Interrupt? Machine
> > Check? PCI-E Advanced Error Reporting?
> 
> I haven't checked but I would expect configuration space accesses to not
> happen in interrupt context. Usually they are limited to enumeration and
> driver probe.

Actually, AER probably needs this, and I believe some broken devices 
need to mask interrupts using the PCI command word in the config space,
it it can happen.

	Arnd

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2303 bytes --]

On Thu, Jan 10, 2013 at 09:17:19AM +0000, Arnd Bergmann wrote:
> On Thursday 10 January 2013, Thierry Reding wrote:
> > On Wed, Jan 09, 2013 at 04:17:58PM -0700, Jason Gunthorpe wrote:
> > > On Wed, Jan 09, 2013 at 04:12:31PM -0700, Stephen Warren wrote:
> > > You could decrease the size of the mapping to only span the bus
> > > numbers that are configured for use via DT.
> > 
> > That won't work, unfortunately. The mapping is such that the bus number
> > is not encoded in the uppermost bits, the extended register number is.
> > So the only thing that we could do is decrease the size of the extended
> > register space for *all* devices.
> 
> But you could still a method to map 16 separate areas per bus, each 65536
> bytes long, which results in 1MB per bus. That is probably ok, since
> very few systems have more than a handful of buses in practice.
> 
> In theory, doing static mappings on a per-page base would let you
> do 16 devices at a time, but it's probably worth doing at this fine
> granularity.

I don't understand how this would help. The encoding is like this:

	[27:24] extended register number
	[23:16] bus number
	[15:11] device number
	[10: 8] function number
	[ 7: 0] register number

So it doesn't matter whether I use separate areas per bus or not. As
soon as the whole extended configuration space needs to be accessed a
whopping 28 bits (256 MiB) are required.

What you propose would work if only regular configuration space is
supported. I'm not sure if that's an option.

> > > Are there any concerns about these config registers being accessed
> > > from a context where a new mapping can't be made? Interrupt? Machine
> > > Check? PCI-E Advanced Error Reporting?
> > 
> > I haven't checked but I would expect configuration space accesses to not
> > happen in interrupt context. Usually they are limited to enumeration and
> > driver probe.
> 
> Actually, AER probably needs this, and I believe some broken devices 
> need to mask interrupts using the PCI command word in the config space,
> it it can happen.

Ugh... that would kill any such dynamic mapping approach. Perhaps if we
could mark a device as requiring a static mapping we could pin that
cache entry. But that doesn't sound very encouraging.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Jason Gunthorpe]

On Thu, Jan 10, 2013 at 11:25:44AM +0100, Thierry Reding wrote:
> On Thu, Jan 10, 2013 at 09:17:19AM +0000, Arnd Bergmann wrote:
> > On Thursday 10 January 2013, Thierry Reding wrote:
> > > On Wed, Jan 09, 2013 at 04:17:58PM -0700, Jason Gunthorpe wrote:
> > > > On Wed, Jan 09, 2013 at 04:12:31PM -0700, Stephen Warren wrote:
> > > > You could decrease the size of the mapping to only span the bus
> > > > numbers that are configured for use via DT.
> > > 
> > > That won't work, unfortunately. The mapping is such that the bus number
> > > is not encoded in the uppermost bits, the extended register number is.
> > > So the only thing that we could do is decrease the size of the extended
> > > register space for *all* devices.
> > 
> > But you could still a method to map 16 separate areas per bus, each 65536
> > bytes long, which results in 1MB per bus. That is probably ok, since
> > very few systems have more than a handful of buses in practice.
> > 
> > In theory, doing static mappings on a per-page base would let you
> > do 16 devices at a time, but it's probably worth doing at this fine
> > granularity.
> 
> I don't understand how this would help. The encoding is like this:
> 
> 	[27:24] extended register number
> 	[23:16] bus number
> 	[15:11] device number
> 	[10: 8] function number
> 	[ 7: 0] register number
> 
> So it doesn't matter whether I use separate areas per bus or not. As
> soon as the whole extended configuration space needs to be accessed a
> whopping 28 bits (256 MiB) are required.

You'd piece a mapping together, each bus requires 16 64k mappings, a
simple 2d array of busnr*16 of pointers would do the trick. A more
clever solution would be to allocate contiguous virtual memory and
split that up..

> > Actually, AER probably needs this, and I believe some broken devices 
> > need to mask interrupts using the PCI command word in the config space,
> > it it can happen.
> 
> Ugh... that would kill any such dynamic mapping approach. Perhaps if we
> could mark a device as requiring a static mapping we could pin that
> cache entry. But that doesn't sound very encouraging.

AER applies to pretty much every PCI-E device these days.

Jason

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Arnd Bergmann]

On Thursday 10 January 2013, Thierry Reding wrote:
> I don't understand how this would help. The encoding is like this:
> 
>         [27:24] extended register number
>         [23:16] bus number
>         [15:11] device number
>         [10: 8] function number
>         [ 7: 0] register number
> 
> So it doesn't matter whether I use separate areas per bus or not. As
> soon as the whole extended configuration space needs to be accessed a
> whopping 28 bits (256 MiB) are required.
> 
> What you propose would work if only regular configuration space is
> supported. I'm not sure if that's an option.

I mean something like:

struct tegra_bus_private {
	...
	void __iomem *config_space[16];
};


void tegra_scan_bus(struct pci_bus *bus)
{
	int i;
	struct tegra_bus_private *priv = bus->dev->private;

	for (i=0; i<16; i++)
		priv->config_space[i] = ioremap(config_space_phys +
				 65536 * bus->primary + i * SZ_1M, 65536);

	...
}

	Arnd

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2608 bytes --]

On Thu, Jan 10, 2013 at 11:20:07AM -0700, Jason Gunthorpe wrote:
> On Thu, Jan 10, 2013 at 11:25:44AM +0100, Thierry Reding wrote:
> > On Thu, Jan 10, 2013 at 09:17:19AM +0000, Arnd Bergmann wrote:
> > > On Thursday 10 January 2013, Thierry Reding wrote:
> > > > On Wed, Jan 09, 2013 at 04:17:58PM -0700, Jason Gunthorpe wrote:
> > > > > On Wed, Jan 09, 2013 at 04:12:31PM -0700, Stephen Warren wrote:
> > > > > You could decrease the size of the mapping to only span the bus
> > > > > numbers that are configured for use via DT.
> > > > 
> > > > That won't work, unfortunately. The mapping is such that the bus number
> > > > is not encoded in the uppermost bits, the extended register number is.
> > > > So the only thing that we could do is decrease the size of the extended
> > > > register space for *all* devices.
> > > 
> > > But you could still a method to map 16 separate areas per bus, each 65536
> > > bytes long, which results in 1MB per bus. That is probably ok, since
> > > very few systems have more than a handful of buses in practice.
> > > 
> > > In theory, doing static mappings on a per-page base would let you
> > > do 16 devices at a time, but it's probably worth doing at this fine
> > > granularity.
> > 
> > I don't understand how this would help. The encoding is like this:
> > 
> > 	[27:24] extended register number
> > 	[23:16] bus number
> > 	[15:11] device number
> > 	[10: 8] function number
> > 	[ 7: 0] register number
> > 
> > So it doesn't matter whether I use separate areas per bus or not. As
> > soon as the whole extended configuration space needs to be accessed a
> > whopping 28 bits (256 MiB) are required.
> 
> You'd piece a mapping together, each bus requires 16 64k mappings, a
> simple 2d array of busnr*16 of pointers would do the trick. A more
> clever solution would be to allocate contiguous virtual memory and
> split that up..

Oh, I see. I'm not very familiar with the internals of remapping, so
I'll need to do some more reading. Thanks for the hints.

> > > Actually, AER probably needs this, and I believe some broken devices 
> > > need to mask interrupts using the PCI command word in the config space,
> > > it it can happen.
> > 
> > Ugh... that would kill any such dynamic mapping approach. Perhaps if we
> > could mark a device as requiring a static mapping we could pin that
> > cache entry. But that doesn't sound very encouraging.
> 
> AER applies to pretty much every PCI-E device these days.

So given there's no way around a segmented static mapping as you
suggested, right?

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

On Thu, Jan 10, 2013 at 06:26:55PM +0000, Arnd Bergmann wrote:
> On Thursday 10 January 2013, Thierry Reding wrote:
> > I don't understand how this would help. The encoding is like this:
> > 
> >         [27:24] extended register number
> >         [23:16] bus number
> >         [15:11] device number
> >         [10: 8] function number
> >         [ 7: 0] register number
> > 
> > So it doesn't matter whether I use separate areas per bus or not. As
> > soon as the whole extended configuration space needs to be accessed a
> > whopping 28 bits (256 MiB) are required.
> > 
> > What you propose would work if only regular configuration space is
> > supported. I'm not sure if that's an option.
> 
> I mean something like:
> 
> struct tegra_bus_private {
> 	...
> 	void __iomem *config_space[16];
> };
> 
> 
> void tegra_scan_bus(struct pci_bus *bus)
> {
> 	int i;
> 	struct tegra_bus_private *priv = bus->dev->private;
> 
> 	for (i=0; i<16; i++)
> 		priv->config_space[i] = ioremap(config_space_phys +
> 				 65536 * bus->primary + i * SZ_1M, 65536);
> 
> 	...
> }

Okay, I see. It's a bit kludgy, but I guess so was the I/O map cache.
It'll take some more time to work this out and test, but I'll give it
a shot.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2350 bytes --]

On Thu, Jan 10, 2013 at 07:55:05PM +0100, Thierry Reding wrote:
> On Thu, Jan 10, 2013 at 11:20:07AM -0700, Jason Gunthorpe wrote:
> > On Thu, Jan 10, 2013 at 11:25:44AM +0100, Thierry Reding wrote:
> > > On Thu, Jan 10, 2013 at 09:17:19AM +0000, Arnd Bergmann wrote:
> > > > On Thursday 10 January 2013, Thierry Reding wrote:
> > > > > On Wed, Jan 09, 2013 at 04:17:58PM -0700, Jason Gunthorpe wrote:
> > > > > > On Wed, Jan 09, 2013 at 04:12:31PM -0700, Stephen Warren wrote:
> > > > > > You could decrease the size of the mapping to only span the bus
> > > > > > numbers that are configured for use via DT.
> > > > > 
> > > > > That won't work, unfortunately. The mapping is such that the bus number
> > > > > is not encoded in the uppermost bits, the extended register number is.
> > > > > So the only thing that we could do is decrease the size of the extended
> > > > > register space for *all* devices.
> > > > 
> > > > But you could still a method to map 16 separate areas per bus, each 65536
> > > > bytes long, which results in 1MB per bus. That is probably ok, since
> > > > very few systems have more than a handful of buses in practice.
> > > > 
> > > > In theory, doing static mappings on a per-page base would let you
> > > > do 16 devices at a time, but it's probably worth doing at this fine
> > > > granularity.
> > > 
> > > I don't understand how this would help. The encoding is like this:
> > > 
> > > 	[27:24] extended register number
> > > 	[23:16] bus number
> > > 	[15:11] device number
> > > 	[10: 8] function number
> > > 	[ 7: 0] register number
> > > 
> > > So it doesn't matter whether I use separate areas per bus or not. As
> > > soon as the whole extended configuration space needs to be accessed a
> > > whopping 28 bits (256 MiB) are required.
> > 
> > You'd piece a mapping together, each bus requires 16 64k mappings, a
> > simple 2d array of busnr*16 of pointers would do the trick. A more
> > clever solution would be to allocate contiguous virtual memory and
> > split that up..
> 
> Oh, I see. I'm not very familiar with the internals of remapping, so
> I'll need to do some more reading. Thanks for the hints.

I forgot to ask. What's the advantage of having a contiguous virtual
memory area and splitting it up versus remapping each chunk separately?

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Jason Gunthorpe]

On Thu, Jan 10, 2013 at 08:03:27PM +0100, Thierry Reding wrote:

> > > You'd piece a mapping together, each bus requires 16 64k mappings, a
> > > simple 2d array of busnr*16 of pointers would do the trick. A more
> > > clever solution would be to allocate contiguous virtual memory and
> > > split that up..
 
> > Oh, I see. I'm not very familiar with the internals of remapping, so
> > I'll need to do some more reading. Thanks for the hints.
> 
> I forgot to ask. What's the advantage of having a contiguous virtual
> memory area and splitting it up versus remapping each chunk separately?

Not alot, really, but it saves you from the pointer array and
associated overhead. IIRC it is fairly easy to do in the kernel.

Arnd's version is good too, but you would be restricted to aligned
powers of two for the bus number range in the DT, which is probably
not that big a deal either?

Jason

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1771 bytes --]

On Thu, Jan 10, 2013 at 12:24:17PM -0700, Jason Gunthorpe wrote:
> On Thu, Jan 10, 2013 at 08:03:27PM +0100, Thierry Reding wrote:
> 
> > > > You'd piece a mapping together, each bus requires 16 64k mappings, a
> > > > simple 2d array of busnr*16 of pointers would do the trick. A more
> > > > clever solution would be to allocate contiguous virtual memory and
> > > > split that up..
>  
> > > Oh, I see. I'm not very familiar with the internals of remapping, so
> > > I'll need to do some more reading. Thanks for the hints.
> > 
> > I forgot to ask. What's the advantage of having a contiguous virtual
> > memory area and splitting it up versus remapping each chunk separately?
> 
> Not alot, really, but it saves you from the pointer array and
> associated overhead. IIRC it is fairly easy to do in the kernel.

I've been investigating this a bit, and one problem is that it will
prevent the driver from ever building as a module because the necessary
functions aren't exported and I'm not sure exporting them would be
acceptable. Currently PCI host controller drivers with MSI support can't
be built as modules because the MSI infrastructure requires it, but I
briefly discussed this with Bjorn at some point and it should be easy to
remove that requirement.

> Arnd's version is good too, but you would be restricted to aligned
> powers of two for the bus number range in the DT, which is probably
> not that big a deal either?

Stephen suggested on IRC that we could try to keep a bit of dynamicity
in the allocation scheme if we create the bus mapping when the first
device on the bus is probed and discard the mapping if no devices are
found.

Sounds like a good plan to me. Does anybody see any potential pitfalls?

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 11/14] ARM: tegra: tamonten: Add PCIe support

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

On Wed, Jan 09, 2013 at 09:23:37PM +0000, Arnd Bergmann wrote:
> On Wednesday 09 January 2013, Thierry Reding wrote:
> > Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> > ---
> >  arch/arm/boot/dts/tegra20-tamonten.dtsi | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> It's not clear how this one-line patch adds PCIe support to the
> platform. Are you missing the actual change here?

This is really just required to associate a name with the regulator, but
I think I can achieve the same by adding an alias to the TEC DTSI.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Jason Gunthorpe]

On Thu, Jan 10, 2013 at 09:20:07PM +0100, Thierry Reding wrote:

> > Arnd's version is good too, but you would be restricted to aligned
> > powers of two for the bus number range in the DT, which is probably
> > not that big a deal either?
> 
> Stephen suggested on IRC that we could try to keep a bit of dynamicity
> in the allocation scheme if we create the bus mapping when the first
> device on the bus is probed and discard the mapping if no devices are
> found.

You probably don't need to mess around with 'discard on empty' the
kernel should only access bus numbers that are in the range of the
subordinate busses of the bridges. So if you establish a mapping on a
bus-by-bus basis at first access, it should be fine and very close to
minimal..

Jason

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Stephen Warren]

On 01/09/2013 01:43 PM, Thierry Reding wrote:
> Move the PCIe driver from arch/arm/mach-tegra into the drivers/pci/host
> directory. The motivation is to collect various host controller drivers
> in the same location in order to facilitate refactoring.
> 
> The Tegra PCIe driver has been largely rewritten, both in order to turn
> it into a proper platform driver and to add MSI (based on code by
> Krishna Kishore <kthota@nvidia.com>) as well as device tree support.

This driver doesn't compile unless CONFIG_PCI_MSI is also enabled.
Should it select that, or contain a few ifdefs?

drivers/pci/host/pci-tegra.c:900: undefined reference to `write_msi_msg'

Re: [PATCH 14/14] ARM: tegra: trimslice: Initialize PCIe from DT

[Stephen Warren]

On 01/09/2013 01:43 PM, Thierry Reding wrote:
> With the device tree support in place, probe the PCIe controller from
> the device tree and remove the corresponding workaround in the board
> file.

Thierry, there are a couple things missing from this patch; I include a
fixup for you to squash in, but also see beyond that for a problem.

>  arch/arm/boot/dts/tegra20-trimslice.dts |   26 +++++++++++++++++++++-----
>  1 file changed, 21 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/arm/boot/dts/tegra20-trimslice.dts b/arch/arm/boot/dts/tegra20-trimslice.dts
> index ebb4c17..41fc45b 100644
> --- a/arch/arm/boot/dts/tegra20-trimslice.dts
> +++ b/arch/arm/boot/dts/tegra20-trimslice.dts
> @@ -303,12 +303,10 @@
>  
>  	pcie-controller {
>  		status = "okay";
> +		pex-clk-supply = <&pci_clk_reg>;
> +		vdd-supply = <&pci_vdd_reg>;
>  
> -		pci@0 {
> -			status = "okay";
> -		};
> -
> -		pci@1 {
> +		pci@1,0 {
>  			status = "okay";
>  		};
>  	};
> @@ -366,6 +364,24 @@
>  			regulator-max-microvolt = <1800000>;
>  			regulator-always-on;
>  		};
> +
> +		pci_clk_reg: regulator@2 {
> +			compatible = "regulator-fixed";
> +			reg = <2>;
> +			regulator-name = "pci_clk";
> +			regulator-min-microvolt = <3300000>;
> +			regulator-max-microvolt = <3300000>;
> +			regulator-always-on;
> +		};
> +
> +		pci_vdd_reg: regulator@3 {
> +			compatible = "regulator-fixed";
> +			reg = <3>;
> +			regulator-name = "pci_vdd";
> +			regulator-min-microvolt = <1050000>;
> +			regulator-max-microvolt = <1050000>;
> +			regulator-always-on;
> +		};
>  	};
>  
>  	sound {
> -- 
> 1.7.10.4

However, PCIe still doesn't work on TrimSlice; the device enumeration
fails. The log is below:

> [    0.790425] tegra-pcie 80003000.pcie-controller: link 0 down, retrying
> [    0.801351] tegra-pcie 80003000.pcie-controller: PCI host bridge to bus 0000:00
> [    0.808945] pci_bus 0000:00: root bus resource [io  0x82000000-0x8200ffff]
> [    0.816109] pci_bus 0000:00: root bus resource [mem 0xa0000000-0xafffffff]
> [    0.823303] pci_bus 0000:00: root bus resource [mem 0xb0000000-0xbfffffff pref]
> [    0.830893] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    0.836728] pci 0000:00:01.0: [10de:0bf0] type 01 class 0x060000
> [    0.843097] pci 0000:00:01.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.850047] PCI: bus0: Fast back to back transfers disabled
> [    0.855853] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [    0.864308] tegra-pcie 80003000.pcie-controller: AXI response decoding error, signature: 10010001
> [    0.873489] tegra-pcie 80003000.pcie-controller:   FPCI address: fe10010000

^^^ For some reason, the config register access fails.

> [    0.880760] PCI: bus1: Fast back to back transfers enabled
> [    0.886482] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
> [    0.893381] pci 0000:00:01.0: PCI bridge to [bus 01]
> [    0.898577] PCI: enabling device 0000:00:01.0 (0140 -> 0143)
> [    0.904535] pci 0000:00:01.0: nv_msi_ht_cap_quirk didn't locate host bridge

With plain next-20130109, the device enumeration succeeds.

Re: [PATCH 13/14] ARM: tegra: harmony: Initialize PCIe from DT

[Stephen Warren]

On 01/09/2013 01:43 PM, Thierry Reding wrote:
> With the device tree support in place, probe the PCIe controller from
> the device tree and remove the corresponding workaround in the board
> file.

For some reason, the ports on Harmony fail to initialize correctly with
this driver, although they do with plain next-20130109:

> [    2.698922] tegra-pcie 80003000.pcie-controller: link 0 down, retrying
> [    2.925722] tegra-pcie 80003000.pcie-controller: link 0 down, retrying
> [    3.155411] tegra-pcie 80003000.pcie-controller: link 0 down, retrying
> [    3.178428] tegra-pcie 80003000.pcie-controller: link 0 down, ignoring
> [    3.199713] tegra-pcie 80003000.pcie-controller: failed to add port pci: -19

^^^ There's definitely something plugged into this port, and it's
enumerated OK with the old driver.

> [    3.433551] tegra-pcie 80003000.pcie-controller: link 1 down, retrying
> [    3.667624] tegra-pcie 80003000.pcie-controller: link 1 down, retrying
...
> [    3.901493] tegra-pcie 80003000.pcie-controller: link 1 down, retrying
> [    3.902589] tegra-pcie 80003000.pcie-controller: link 1 down, ignoring
> [    3.902615] tegra-pcie 80003000.pcie-controller: failed to add port pci: -19
> [    3.903780] tegra-pcie 80003000.pcie-controller: PCI host bridge to bus 0000:00
> [    3.903815] pci_bus 0000:00: root bus resource [io  0x82000000-0x8200ffff]
> [    3.903844] pci_bus 0000:00: root bus resource [mem 0xa0000000-0xafffffff]
> [    3.903871] pci_bus 0000:00: root bus resource [mem 0xb0000000-0xbfffffff pref]
> [    3.903901] pci_bus 0000:00: root bus resource [bus 00-ff]

Re: [PATCH 01/14] of/pci: Provide support for parsing PCI DT ranges property

[Stephen Warren]

On 01/09/2013 01:43 PM, Thierry Reding wrote:
> From: Andrew Murray <andrew.murray@arm.com>
> 
> DT bindings for PCI host bridges often use the ranges property to describe
> memory and IO ranges - this binding tends to be the same across architectures
> yet several parsing implementations exist, e.g. arch/mips/pci/pci.c,
> arch/powerpc/kernel/pci-common.c, arch/sparc/kernel/pci.c and
> arch/microblaze/pci/pci-common.c (clone of PPC). Some of these duplicate
> functionality provided by drivers/of/address.c.
> 
> This patch provides a common iterator-based parser for the ranges property, it
> is hoped this will reduce DT representation differences between architectures
> and that architectures will migrate in part to this new parser.
...

> diff --git a/drivers/of/address.c b/drivers/of/address.c

> +const __be32 *of_pci_process_ranges(struct device_node *node,

> +	while (from + np <= end) {
> +		u64 cpu_addr, size;
> +
> +		cpu_addr = of_translate_address(node, from + na);
> +		size = of_read_number(from + na + pna, ns);
> +		res->flags = bus->get_flags(from);
> +		from += np;
> +
> +		if (cpu_addr == OF_BAD_ADDR || size == 0)
> +			continue;

Hmmm. That seems to just ignore bad entries in the ranges property. Is
that really the right thing to do? At least printing a diagnostic might
be a good idea, even if the code does just soldier on in the hope
everything still works.

> +		res->name = node->full_name;
> +		res->start = cpu_addr;
> +		res->end = res->start + size - 1;
> +		res->parent = res->child = res->sibling = NULL;
> +		return from;
> +	}
> +
> +	return NULL;
> +}

Re: [PATCH 02/14] of/pci: Add of_pci_get_devfn() function

[Stephen Warren]

On 01/09/2013 01:43 PM, Thierry Reding wrote:
> This function can be used to parse the device and function number from a
> standard 5-cell PCI resource. PCI_SLOT() and PCI_FUNC() can be used on
> the returned value obtain the device and function numbers respectively.

> diff --git a/drivers/of/of_pci.c b/drivers/of/of_pci.c

>  static inline int __of_pci_pci_compare(struct device_node *node,
>  				       unsigned int devfn)
>  {
> +	int err;
>  
> +	err = of_pci_get_devfn(node);
> +	if (err < 0)
>  		return 0;
> +
> +	return devfn == err;

I know this is really picky, but calling that "err" when it's hopefully
not an error but rather a PCI device/function ID seems a little obscure.
Perhaps node_devfn? I assume that fact that devfn is unsigned and err is
signed won't be an issue.

Re: [PATCH 09/14] ARM: tegra: Move pmc.h to include/mach

[Stephen Warren]

On 01/09/2013 01:43 PM, Thierry Reding wrote:
> In preparation for moving the PCIe driver into the drivers/pci/host
> directory, this header, which contains prototypes that are required by
> the PCIe driver, needs to be moved to a globally visible location.
> 
> Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> ---
> Note that eventually the code in pmc.c and powergate.c should probably
> be split out into a separate driver. The PMC registers are also directly
> accessed from tegra20_clocks.c and tegra30_clocks.c, so that it might be
> required to provide that functionality through the new driver as well.
> ---
>  arch/arm/mach-tegra/common.c           |  2 +-
>  arch/arm/mach-tegra/include/mach/pmc.h | 24 ++++++++++++++++++++++++
>  arch/arm/mach-tegra/pmc.h              | 24 ------------------------

On IRC, we'd talked about putting the public functionality in
include/linux/tegra-pmc.h so that we wouldn't add to include/mach, which
would make it harder to make Tegra support ARM multi-platform. Perhaps
that IRC discussion happened after you posted this series?

Re: [PATCH 12/14] ARM: tegra: tec: Add PCIe support

[Stephen Warren]

On 01/09/2013 01:43 PM, Thierry Reding wrote:
> Enable the first PCIe root port which is connected to an FPGA on the
> Tamonten Evaluation Carrier and add device nodes for each of the PCI
> endpoints available in the standard configuration.

> diff --git a/arch/arm/boot/dts/tegra20-tec.dts b/arch/arm/boot/dts/tegra20-tec.dts

> +	pcie-controller {
> +		vdd-supply = <&pci_vdd_reg>;
> +		pex-clk-supply = <&pci_clk_reg>;
> +		status = "okay";

Sorry this is also really picky. I'd prefer properties that exist in
/include/d files and are overidden here to appear first, followed by new
properties. In other words, move the status property to be first. I
believe/hope all the other (Tegra) .dts files follow this convention.

> +		pci@1,0 {
> +			bus-range = <0x01 0x0a>;
> +			status = "okay";
> +
> +			pci@0,0 {
> +				reg = <0x010000 0 0 0 0>;

Hmm. The unit address in that node name doesn't match the address in the
reg property, although I suppose there's nothing we can do about it
since those formats are both defined by the standard PCI binding?

What do the numbers "0,0" represent here; device/function? Is the same
true for the "0,0" in the child nodes?

> +				bus-range = <0x02 0x0a>;
> +
> +				compatible = "plda,pcie";

Are there DT binding documents for all these devices; plda,pcie,
ad,pcie, ad,pcie-test, etc.?

> +				pci@4,0 {
> +					reg = <0x022000 0 0 0 0>;
> +					bus-range = <0x07 0x07>;
> +
> +					compatible = "ad,pcie";
> +					device_type = "pci";
> +
> +					#address-cells = <3>;
> +					#size-cells = <2>;
> +
> +					pci@0,0 {
> +						compatible = "opencores,uart";
> +						reg = <0x070000 0 0 0 0>;
> +					};
> +				};

Do you need to include a node for the UART; I can see you need to for
the SPI/I2C controllers so you can instantiate the appropriate devices
on non-probe-able buses, but I think you can just let regular PCI device
probing find the UART, Ethernet MAC, etc., can't you?

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Stephen Warren]

On 01/09/2013 01:43 PM, Thierry Reding wrote:
> Move the PCIe driver from arch/arm/mach-tegra into the drivers/pci/host
> directory. The motivation is to collect various host controller drivers
> in the same location in order to facilitate refactoring.
> 
> The Tegra PCIe driver has been largely rewritten, both in order to turn
> it into a proper platform driver and to add MSI (based on code by
> Krishna Kishore <kthota@nvidia.com>) as well as device tree support.

> diff --git a/arch/arm/mach-tegra/board-dt-tegra20.c b/arch/arm/mach-tegra/board-dt-tegra20.c

>  static void __init trimslice_init(void)
>  {
>  #ifdef CONFIG_TEGRA_PCI
> -	int ret;
> -
> -	ret = tegra_pcie_init(true, true);
> -	if (ret)
> -		pr_err("tegra_pci_init() failed: %d\n", ret);
> +	platform_device_register(&tegra_pcie_device);

That struct doesn't actually exist anywhere; only an extern definition
is added (and that extern definition isn't removed by patch 14 either).

> diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig

> +config PCI_TEGRA
> +	bool "NVIDIA Tegra PCIe controller"
> +	depends on ARCH_TEGRA_2x_SOC

Perhaps depend on ARCH_TEGRA; that will save churn once this is ported
to Tegra30, and shouldn't cause any problems before then.

> diff --git a/drivers/pci/host/pci-tegra.c b/drivers/pci/host/pci-tegra.c

> +#define AFI_INTR_CODE		0xb8
> +#define  AFI_INTR_CODE_MASK	0xf
> +#define  AFI_INTR_MASTER_ABORT	4
> +#define  AFI_INTR_LEGACY	6

Adding defines for at least some other codes here, would help further
below ...

> +static irqreturn_t tegra_pcie_isr(int irq, void *arg)

> +	if (code == AFI_INTR_MASTER_ABORT) {
> +		dev_dbg(pcie->dev, "%s, signature: %08x\n", err_msg[code],
> +			signature);
> +	} else
> +		dev_err(pcie->dev, "%s, signature: %08x\n", err_msg[code],
> +			signature);
> +
> +	if (code == 3 || code == 4 || code == 7) {

... i.e. here.

> +		u32 fpci = afi_readl(pcie, AFI_UPPER_FPCI_ADDRESS) & 0xff;
> +		u64 address = (u64)fpci << 32 | (signature & 0xfffffffc);
> +		dev_dbg(pcie->dev, "  FPCI address: %10llx\n", address);

I'd suggest making that dev_err(), or at least something higher than
debug, since the message indicating the error happened is dev_err(), so
the complete details may as well be available since they're small.

> +static int tegra_pcie_enable_controller(struct tegra_pcie *pcie)
> +{
> +	unsigned int timeout;
> +	unsigned long value;
> +
> +	/* enable dual controller and both ports */
> +	value = afi_readl(pcie, AFI_PCIE_CONFIG);
> +	value &= ~(AFI_PCIE_CONFIG_PCIEC0_DISABLE_DEVICE |
> +		   AFI_PCIE_CONFIG_PCIEC1_DISABLE_DEVICE |
> +		   AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_MASK);
> +	value |= AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_DUAL;
> +	afi_writel(pcie, value, AFI_PCIE_CONFIG);

Eventually, we should probably derive the port enables from the state of
the root port DT nodes, so that we can disable some and presumably save
a little power. Also, I notice that the nvidia,num-lanes property isn't
implemented yet. Still, we can probably take care of this later.

> +static void tegra_pcie_power_off(struct tegra_pcie *pcie)

> +	if (!IS_ERR_OR_NULL(pcie->pex_clk_supply)) {

Hmm. I think we should make supplies mandatory; it doesn't make sense
for regulator support to be disabled on Tegra, and where a specific
board doesn't actually have a regulator, you're supposed to provide a
dummy fixed regulator so the driver doesn't have to care.

The same comment obviously applies to tegra_pcie_power_on() and wherever
regulator_get() happens.

> +static int tegra_pcie_parse_dt(struct tegra_pcie *pcie)

> +	pcie->vdd_supply = devm_regulator_get(pcie->dev, "vdd");
> +	if (IS_ERR(pcie->vdd_supply))
> +		return PTR_ERR(pcie->vdd_supply);
> +
> +	pcie->pex_clk_supply = devm_regulator_get(pcie->dev, "pex-clk");
> +	if (IS_ERR(pcie->pex_clk_supply))
> +		return PTR_ERR(pcie->pex_clk_supply);

Oh, I guess the regulator_get() calls are already strict.

> +static int tegra_pcie_add_port(struct tegra_pcie *pcie, struct device_node *np)

> +	port = devm_kzalloc(pcie->dev, sizeof(*port), GFP_KERNEL);
> +	if (!port)
> +		return -ENOMEM;
> +
> +	INIT_LIST_HEAD(&port->list);
> +	port->index = index;
> +	port->pcie = pcie;
> +
> +	port->base = devm_request_and_ioremap(pcie->dev, &regs);
> +	if (!port->base)
> +		return -EADDRNOTAVAIL;
> +
> +	if (!tegra_pcie_port_check_link(port)) {
> +		dev_info(pcie->dev, "link %u down, ignoring\n", port->index);

Perhaps devm_kfree(port)? Not a big leak, but equally if you don't, it's
an unreferenced memory block.

> +		return -ENODEV;
> +	}
> +
> +	list_add_tail(&port->list, &pcie->ports);
> +
> +	return 0;
> +}

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1088 bytes --]

On Thu, Jan 10, 2013 at 04:54:30PM -0700, Stephen Warren wrote:
> On 01/09/2013 01:43 PM, Thierry Reding wrote:
> > Move the PCIe driver from arch/arm/mach-tegra into the drivers/pci/host
> > directory. The motivation is to collect various host controller drivers
> > in the same location in order to facilitate refactoring.
> > 
> > The Tegra PCIe driver has been largely rewritten, both in order to turn
> > it into a proper platform driver and to add MSI (based on code by
> > Krishna Kishore <kthota@nvidia.com>) as well as device tree support.
> 
> This driver doesn't compile unless CONFIG_PCI_MSI is also enabled.
> Should it select that, or contain a few ifdefs?
> 
> drivers/pci/host/pci-tegra.c:900: undefined reference to `write_msi_msg'

Right, it'll need #ifdefs around the arch_{setup,teardown}_msi_irq(). Or
select PCI_MSI unconditionally. Once this is merged I was going to post
a patch that enables PCI_MSI in tegra_defconfig anyway. But it might be
better to keep it optional anyway since the remainder of the code copes
with it properly.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 5788 bytes --]

On Thu, Jan 10, 2013 at 05:48:46PM -0700, Stephen Warren wrote:
> On 01/09/2013 01:43 PM, Thierry Reding wrote:
> > Move the PCIe driver from arch/arm/mach-tegra into the drivers/pci/host
> > directory. The motivation is to collect various host controller drivers
> > in the same location in order to facilitate refactoring.
> > 
> > The Tegra PCIe driver has been largely rewritten, both in order to turn
> > it into a proper platform driver and to add MSI (based on code by
> > Krishna Kishore <kthota@nvidia.com>) as well as device tree support.
> 
> > diff --git a/arch/arm/mach-tegra/board-dt-tegra20.c b/arch/arm/mach-tegra/board-dt-tegra20.c
> 
> >  static void __init trimslice_init(void)
> >  {
> >  #ifdef CONFIG_TEGRA_PCI
> > -	int ret;
> > -
> > -	ret = tegra_pcie_init(true, true);
> > -	if (ret)
> > -		pr_err("tegra_pci_init() failed: %d\n", ret);
> > +	platform_device_register(&tegra_pcie_device);
> 
> That struct doesn't actually exist anywhere; only an extern definition
> is added (and that extern definition isn't removed by patch 14 either).

Right, this shouldn't be there. In fact TEGRA_PCI is removed by this
patch, so I should go over the code more carefully again.

> > diff --git a/drivers/pci/host/Kconfig b/drivers/pci/host/Kconfig
> 
> > +config PCI_TEGRA
> > +	bool "NVIDIA Tegra PCIe controller"
> > +	depends on ARCH_TEGRA_2x_SOC
> 
> Perhaps depend on ARCH_TEGRA; that will save churn once this is ported
> to Tegra30, and shouldn't cause any problems before then.

Okay, I can do that.

> > diff --git a/drivers/pci/host/pci-tegra.c b/drivers/pci/host/pci-tegra.c
> 
> > +#define AFI_INTR_CODE		0xb8
> > +#define  AFI_INTR_CODE_MASK	0xf
> > +#define  AFI_INTR_MASTER_ABORT	4
> > +#define  AFI_INTR_LEGACY	6
> 
> Adding defines for at least some other codes here, would help further
> below ...
> 
> > +static irqreturn_t tegra_pcie_isr(int irq, void *arg)
> 
> > +	if (code == AFI_INTR_MASTER_ABORT) {
> > +		dev_dbg(pcie->dev, "%s, signature: %08x\n", err_msg[code],
> > +			signature);
> > +	} else
> > +		dev_err(pcie->dev, "%s, signature: %08x\n", err_msg[code],
> > +			signature);
> > +
> > +	if (code == 3 || code == 4 || code == 7) {
> 
> ... i.e. here.

Will do.

> 
> > +		u32 fpci = afi_readl(pcie, AFI_UPPER_FPCI_ADDRESS) & 0xff;
> > +		u64 address = (u64)fpci << 32 | (signature & 0xfffffffc);
> > +		dev_dbg(pcie->dev, "  FPCI address: %10llx\n", address);
> 
> I'd suggest making that dev_err(), or at least something higher than
> debug, since the message indicating the error happened is dev_err(), so
> the complete details may as well be available since they're small.

I can make it conditional on !AFI_INTR_MASTER_ABORT to match the
previous output. Or rather move it into the branches above.

> > +static int tegra_pcie_enable_controller(struct tegra_pcie *pcie)
> > +{
> > +	unsigned int timeout;
> > +	unsigned long value;
> > +
> > +	/* enable dual controller and both ports */
> > +	value = afi_readl(pcie, AFI_PCIE_CONFIG);
> > +	value &= ~(AFI_PCIE_CONFIG_PCIEC0_DISABLE_DEVICE |
> > +		   AFI_PCIE_CONFIG_PCIEC1_DISABLE_DEVICE |
> > +		   AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_MASK);
> > +	value |= AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_DUAL;
> > +	afi_writel(pcie, value, AFI_PCIE_CONFIG);
> 
> Eventually, we should probably derive the port enables from the state of
> the root port DT nodes, so that we can disable some and presumably save
> a little power. Also, I notice that the nvidia,num-lanes property isn't
> implemented yet. Still, we can probably take care of this later.

Yes, the plan was to eventually derive the disable bits from the port
status and setup the XBAR_CONFIG field based on the combination of
nvidia,num-lanes properties.

I assume we should simply fail if the configuration specified by
nvidia,num-lanes is invalid?

> > +static void tegra_pcie_power_off(struct tegra_pcie *pcie)
> 
> > +	if (!IS_ERR_OR_NULL(pcie->pex_clk_supply)) {
> 
> Hmm. I think we should make supplies mandatory; it doesn't make sense
> for regulator support to be disabled on Tegra, and where a specific
> board doesn't actually have a regulator, you're supposed to provide a
> dummy fixed regulator so the driver doesn't have to care.
> 
> The same comment obviously applies to tegra_pcie_power_on() and wherever
> regulator_get() happens.

Okay, I'll fix that.

> > +static int tegra_pcie_parse_dt(struct tegra_pcie *pcie)
> 
> > +	pcie->vdd_supply = devm_regulator_get(pcie->dev, "vdd");
> > +	if (IS_ERR(pcie->vdd_supply))
> > +		return PTR_ERR(pcie->vdd_supply);
> > +
> > +	pcie->pex_clk_supply = devm_regulator_get(pcie->dev, "pex-clk");
> > +	if (IS_ERR(pcie->pex_clk_supply))
> > +		return PTR_ERR(pcie->pex_clk_supply);
> 
> Oh, I guess the regulator_get() calls are already strict.

Yeah, I think they can't return NULL, right? In that case I can just
drop the extra checks in tegra_pcie_power_{on,off}().

> > +static int tegra_pcie_add_port(struct tegra_pcie *pcie, struct device_node *np)
> 
> > +	port = devm_kzalloc(pcie->dev, sizeof(*port), GFP_KERNEL);
> > +	if (!port)
> > +		return -ENOMEM;
> > +
> > +	INIT_LIST_HEAD(&port->list);
> > +	port->index = index;
> > +	port->pcie = pcie;
> > +
> > +	port->base = devm_request_and_ioremap(pcie->dev, &regs);
> > +	if (!port->base)
> > +		return -EADDRNOTAVAIL;
> > +
> > +	if (!tegra_pcie_port_check_link(port)) {
> > +		dev_info(pcie->dev, "link %u down, ignoring\n", port->index);
> 
> Perhaps devm_kfree(port)? Not a big leak, but equally if you don't, it's
> an unreferenced memory block.

I suppose I should do devm_iounmap() and devm_release_mem_region() as
well.

Thanks for reviewing!
Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 01/14] of/pci: Provide support for parsing PCI DT ranges property

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2309 bytes --]

On Thu, Jan 10, 2013 at 05:06:48PM -0700, Stephen Warren wrote:
> On 01/09/2013 01:43 PM, Thierry Reding wrote:
> > From: Andrew Murray <andrew.murray@arm.com>
> > 
> > DT bindings for PCI host bridges often use the ranges property to describe
> > memory and IO ranges - this binding tends to be the same across architectures
> > yet several parsing implementations exist, e.g. arch/mips/pci/pci.c,
> > arch/powerpc/kernel/pci-common.c, arch/sparc/kernel/pci.c and
> > arch/microblaze/pci/pci-common.c (clone of PPC). Some of these duplicate
> > functionality provided by drivers/of/address.c.
> > 
> > This patch provides a common iterator-based parser for the ranges property, it
> > is hoped this will reduce DT representation differences between architectures
> > and that architectures will migrate in part to this new parser.
> ...
> 
> > diff --git a/drivers/of/address.c b/drivers/of/address.c
> 
> > +const __be32 *of_pci_process_ranges(struct device_node *node,
> 
> > +	while (from + np <= end) {
> > +		u64 cpu_addr, size;
> > +
> > +		cpu_addr = of_translate_address(node, from + na);
> > +		size = of_read_number(from + na + pna, ns);
> > +		res->flags = bus->get_flags(from);
> > +		from += np;
> > +
> > +		if (cpu_addr == OF_BAD_ADDR || size == 0)
> > +			continue;
> 
> Hmmm. That seems to just ignore bad entries in the ranges property. Is
> that really the right thing to do? At least printing a diagnostic might
> be a good idea, even if the code does just soldier on in the hope
> everything still works.

That's true. However, erroring out here wouln't be useful either since a
NULL return value is used to mark the end of the iteration and the
caller would have to assume that no more ranges are present. Maybe
that's better than continuing anyway, even if a message is printed.

Alternatively, the of_pci_process_ranges() could be changed to return an
ERR_PTR() encoded errno. This is one case where it makes a lot of sense.
I have a feeling that Grant won't like that very much, though.

Another possibility would be to change away from an iterator-based
approach and return an integer for the number of ranges and negative
error code on failure while returning an allocated array of resources
through an output parameter.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 02/14] of/pci: Add of_pci_get_devfn() function

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

On Thu, Jan 10, 2013 at 05:09:43PM -0700, Stephen Warren wrote:
> On 01/09/2013 01:43 PM, Thierry Reding wrote:
> > This function can be used to parse the device and function number from a
> > standard 5-cell PCI resource. PCI_SLOT() and PCI_FUNC() can be used on
> > the returned value obtain the device and function numbers respectively.
> 
> > diff --git a/drivers/of/of_pci.c b/drivers/of/of_pci.c
> 
> >  static inline int __of_pci_pci_compare(struct device_node *node,
> >  				       unsigned int devfn)
> >  {
> > +	int err;
> >  
> > +	err = of_pci_get_devfn(node);
> > +	if (err < 0)
> >  		return 0;
> > +
> > +	return devfn == err;
> 
> I know this is really picky, but calling that "err" when it's hopefully
> not an error but rather a PCI device/function ID seems a little obscure.
> Perhaps node_devfn? I assume that fact that devfn is unsigned and err is
> signed won't be an issue.

Maybe renaming the devfn parameter to data and using devfn for the local
variable would be more obvious.

The signedness shouldn't be problematic. devfn is an 8-bit unsigned
integer and sign mismatch is excluded by the error return already.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 09/14] ARM: tegra: Move pmc.h to include/mach

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

On Thu, Jan 10, 2013 at 05:15:07PM -0700, Stephen Warren wrote:
> On 01/09/2013 01:43 PM, Thierry Reding wrote:
> > In preparation for moving the PCIe driver into the drivers/pci/host
> > directory, this header, which contains prototypes that are required by
> > the PCIe driver, needs to be moved to a globally visible location.
> > 
> > Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> > ---
> > Note that eventually the code in pmc.c and powergate.c should probably
> > be split out into a separate driver. The PMC registers are also directly
> > accessed from tegra20_clocks.c and tegra30_clocks.c, so that it might be
> > required to provide that functionality through the new driver as well.
> > ---
> >  arch/arm/mach-tegra/common.c           |  2 +-
> >  arch/arm/mach-tegra/include/mach/pmc.h | 24 ++++++++++++++++++++++++
> >  arch/arm/mach-tegra/pmc.h              | 24 ------------------------
> 
> On IRC, we'd talked about putting the public functionality in
> include/linux/tegra-pmc.h so that we wouldn't add to include/mach, which
> would make it harder to make Tegra support ARM multi-platform. Perhaps
> that IRC discussion happened after you posted this series?

I'm not sure, it's equally possible that I just forgot. Will fix it up.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 12/14] ARM: tegra: tec: Add PCIe support

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 4186 bytes --]

On Thu, Jan 10, 2013 at 05:22:30PM -0700, Stephen Warren wrote:
> On 01/09/2013 01:43 PM, Thierry Reding wrote:
> > Enable the first PCIe root port which is connected to an FPGA on the
> > Tamonten Evaluation Carrier and add device nodes for each of the PCI
> > endpoints available in the standard configuration.
> 
> > diff --git a/arch/arm/boot/dts/tegra20-tec.dts b/arch/arm/boot/dts/tegra20-tec.dts
> 
> > +	pcie-controller {
> > +		vdd-supply = <&pci_vdd_reg>;
> > +		pex-clk-supply = <&pci_clk_reg>;
> > +		status = "okay";
> 
> Sorry this is also really picky. I'd prefer properties that exist in
> /include/d files and are overidden here to appear first, followed by new
> properties. In other words, move the status property to be first. I
> believe/hope all the other (Tegra) .dts files follow this convention.

Okay, I'll fix that.

> > +		pci@1,0 {
> > +			bus-range = <0x01 0x0a>;
> > +			status = "okay";
> > +
> > +			pci@0,0 {
> > +				reg = <0x010000 0 0 0 0>;
> 
> Hmm. The unit address in that node name doesn't match the address in the
> reg property, although I suppose there's nothing we can do about it
> since those formats are both defined by the standard PCI binding?

That's the standard encoding for unit addresses for PCI devices. The
first cell in the reg property encodes bus/device/function (amongst
other things) and the node name is supposed to be pci@<dev>,<fn>.

> What do the numbers "0,0" represent here; device/function? Is the same
> true for the "0,0" in the child nodes?

Yes, exactly.

> > +				bus-range = <0x02 0x0a>;
> > +
> > +				compatible = "plda,pcie";
> 
> Are there DT binding documents for all these devices; plda,pcie,
> ad,pcie, ad,pcie-test, etc.?

No. To be honest I don't quite know how to handle this. For the PLDA
things aren't so bad since it has a proper PCI ID, but the other cores
don't since they are custom IP or taken from opencores.org and made
available via PCIe. We're still in the process of obtaining our own PCI
vendor ID so that these can be properly assigned.

Also, as you have guessed, most of these are not required. I just wanted
to include them here for completeness (and maybe reference in case
somebody else, myself included, needs a working example to base other
work on).

> > +				pci@4,0 {
> > +					reg = <0x022000 0 0 0 0>;
> > +					bus-range = <0x07 0x07>;
> > +
> > +					compatible = "ad,pcie";
> > +					device_type = "pci";
> > +
> > +					#address-cells = <3>;
> > +					#size-cells = <2>;
> > +
> > +					pci@0,0 {
> > +						compatible = "opencores,uart";
> > +						reg = <0x070000 0 0 0 0>;
> > +					};
> > +				};
> 
> Do you need to include a node for the UART; I can see you need to for
> the SPI/I2C controllers so you can instantiate the appropriate devices
> on non-probe-able buses, but I think you can just let regular PCI device
> probing find the UART, Ethernet MAC, etc., can't you?

Yes, that's correct. Only SPI and I2C actually require these nodes. I'm
not sure if the PCI binding requires all nodes to be present or not.
Other examples I've seen (e.g. arch/x86/platform/ce4100/falconfalls.dts)
contain nodes for everything, most of which don't seem to be necessary
for things to work.

One other thing that I've seen is the usage of the more standard pci*
values for the compatible property. None of them are very descriptive
which is why I used a vendor,device type of value instead.

Going over the PCI binding again, however, it looks like there's no
requirement to make the node name pci@dev,fn and pci can be anything.
Making it uart@0,0 and then adjusting the compatible value to be as the
binding requires could be an option. In that case I suppose even the
bindings documentation shouldn't be necessary.

That doesn't cover the nodes where non-standard properties are needed
(I2C and SPI), which do need binding documents. I wouldn't know how to
name them, though. I'm not sure going with the current convention would
be any good since it'll be hard to find the right document if you have
to look it up by matching vendor and device IDs or PCI class.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Arnd Bergmann]

On Friday 11 January 2013, Thierry Reding wrote:
> Right, it'll need #ifdefs around the arch_{setup,teardown}_msi_irq(). Or
> select PCI_MSI unconditionally. Once this is merged I was going to post
> a patch that enables PCI_MSI in tegra_defconfig anyway. But it might be
> better to keep it optional anyway since the remainder of the code copes
> with it properly.
> 
Actually, we need something better than that. You cannot define
arch_setup_msi_irq in a tegra specific pci host driver, because that
will seriously mess up other platforms in multiplatform configurations
by giving a link error when they also define this function, or with a
run-time error when they don't support it.

I think what we should do here is fix it the right way by adding
a pci host specific callback rather than an architecture specific
callback in drivers/pci/msi.c. There is already a default version
of arch_setup_msi_irqs (with s), and we can probably do the
same for arch_setup_msi_irq (without s) to fall back to the
arch version for most architectures.
Most architectures (at least powerpc, sparc, ia64 and x86) already
multiplex the msi handlers internally, but ARM does not because
there is only one implementation (iop33x) at the moment.

We can add a generix multiplex and then move architectures over to
use it.

	Arnd

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1912 bytes --]

On Fri, Jan 11, 2013 at 03:36:14PM +0000, Arnd Bergmann wrote:
> On Friday 11 January 2013, Thierry Reding wrote:
> > Right, it'll need #ifdefs around the arch_{setup,teardown}_msi_irq(). Or
> > select PCI_MSI unconditionally. Once this is merged I was going to post
> > a patch that enables PCI_MSI in tegra_defconfig anyway. But it might be
> > better to keep it optional anyway since the remainder of the code copes
> > with it properly.
> > 
> Actually, we need something better than that. You cannot define
> arch_setup_msi_irq in a tegra specific pci host driver, because that
> will seriously mess up other platforms in multiplatform configurations
> by giving a link error when they also define this function, or with a
> run-time error when they don't support it.
> 
> I think what we should do here is fix it the right way by adding
> a pci host specific callback rather than an architecture specific
> callback in drivers/pci/msi.c. There is already a default version
> of arch_setup_msi_irqs (with s), and we can probably do the
> same for arch_setup_msi_irq (without s) to fall back to the
> arch version for most architectures.
> Most architectures (at least powerpc, sparc, ia64 and x86) already
> multiplex the msi handlers internally, but ARM does not because
> there is only one implementation (iop33x) at the moment.
> 
> We can add a generix multiplex and then move architectures over to
> use it.

I already hinted at that in one of the other subthreads. Having such a
multiplex would also allow the driver to be built as a module. I had
already thought about this when I was working on an earlier version of
these patches. Basically these would be two ops attached to the host
bridge, and the generic arch_setup_msi_irq() could then look that up
given the struct pci_dev that is passed to it and call this new per-
host bridge .setup_msi_irq().

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 14/14] ARM: tegra: trimslice: Initialize PCIe from DT

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 3719 bytes --]

On Thu, Jan 10, 2013 at 04:56:39PM -0700, Stephen Warren wrote:
> On 01/09/2013 01:43 PM, Thierry Reding wrote:
> > With the device tree support in place, probe the PCIe controller from
> > the device tree and remove the corresponding workaround in the board
> > file.
> 
> Thierry, there are a couple things missing from this patch; I include a
> fixup for you to squash in, but also see beyond that for a problem.
> 
> >  arch/arm/boot/dts/tegra20-trimslice.dts |   26 +++++++++++++++++++++-----
> >  1 file changed, 21 insertions(+), 5 deletions(-)
> > 
> > diff --git a/arch/arm/boot/dts/tegra20-trimslice.dts b/arch/arm/boot/dts/tegra20-trimslice.dts
> > index ebb4c17..41fc45b 100644
> > --- a/arch/arm/boot/dts/tegra20-trimslice.dts
> > +++ b/arch/arm/boot/dts/tegra20-trimslice.dts
> > @@ -303,12 +303,10 @@
> >  
> >  	pcie-controller {
> >  		status = "okay";
> > +		pex-clk-supply = <&pci_clk_reg>;
> > +		vdd-supply = <&pci_vdd_reg>;
> >  
> > -		pci@0 {
> > -			status = "okay";
> > -		};
> > -
> > -		pci@1 {
> > +		pci@1,0 {
> >  			status = "okay";
> >  		};
> >  	};
> > @@ -366,6 +364,24 @@
> >  			regulator-max-microvolt = <1800000>;
> >  			regulator-always-on;
> >  		};
> > +
> > +		pci_clk_reg: regulator@2 {
> > +			compatible = "regulator-fixed";
> > +			reg = <2>;
> > +			regulator-name = "pci_clk";
> > +			regulator-min-microvolt = <3300000>;
> > +			regulator-max-microvolt = <3300000>;
> > +			regulator-always-on;
> > +		};
> > +
> > +		pci_vdd_reg: regulator@3 {
> > +			compatible = "regulator-fixed";
> > +			reg = <3>;
> > +			regulator-name = "pci_vdd";
> > +			regulator-min-microvolt = <1050000>;
> > +			regulator-max-microvolt = <1050000>;
> > +			regulator-always-on;
> > +		};
> >  	};
> >  
> >  	sound {
> > -- 
> > 1.7.10.4
> 
> However, PCIe still doesn't work on TrimSlice; the device enumeration
> fails. The log is below:
> 
> > [    0.790425] tegra-pcie 80003000.pcie-controller: link 0 down, retrying
> > [    0.801351] tegra-pcie 80003000.pcie-controller: PCI host bridge to bus 0000:00
> > [    0.808945] pci_bus 0000:00: root bus resource [io  0x82000000-0x8200ffff]
> > [    0.816109] pci_bus 0000:00: root bus resource [mem 0xa0000000-0xafffffff]
> > [    0.823303] pci_bus 0000:00: root bus resource [mem 0xb0000000-0xbfffffff pref]
> > [    0.830893] pci_bus 0000:00: root bus resource [bus 00-ff]
> > [    0.836728] pci 0000:00:01.0: [10de:0bf0] type 01 class 0x060000
> > [    0.843097] pci 0000:00:01.0: PME# supported from D0 D1 D2 D3hot D3cold
> > [    0.850047] PCI: bus0: Fast back to back transfers disabled
> > [    0.855853] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> > [    0.864308] tegra-pcie 80003000.pcie-controller: AXI response decoding error, signature: 10010001
> > [    0.873489] tegra-pcie 80003000.pcie-controller:   FPCI address: fe10010000
> 
> ^^^ For some reason, the config register access fails.
> 
> > [    0.880760] PCI: bus1: Fast back to back transfers enabled
> > [    0.886482] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
> > [    0.893381] pci 0000:00:01.0: PCI bridge to [bus 01]
> > [    0.898577] PCI: enabling device 0000:00:01.0 (0140 -> 0143)
> > [    0.904535] pci 0000:00:01.0: nv_msi_ht_cap_quirk didn't locate host bridge
> 
> With plain next-20130109, the device enumeration succeeds.

That's very strange. I haven't seen that on my setup. Looks like I need
to dig out the TrimSlice again and debug this. It'll take me a while and
there are plenty of other requested changes already, so if I find a fix
I can roll it into the next version.

Thanks for testing,
Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Stephen Warren]

On 01/10/2013 08:52 PM, Thierry Reding wrote:
> On Thu, Jan 10, 2013 at 05:48:46PM -0700, Stephen Warren wrote:
>> On 01/09/2013 01:43 PM, Thierry Reding wrote:
>>> Move the PCIe driver from arch/arm/mach-tegra into the
>>> drivers/pci/host directory. The motivation is to collect
>>> various host controller drivers in the same location in order
>>> to facilitate refactoring.
>>> 
>>> The Tegra PCIe driver has been largely rewritten, both in order
>>> to turn it into a proper platform driver and to add MSI (based
>>> on code by Krishna Kishore <kthota@nvidia.com>) as well as
>>> device tree support.
>> 
>>> diff --git a/arch/arm/mach-tegra/board-dt-tegra20.c
>>> b/arch/arm/mach-tegra/board-dt-tegra20.c

>>> +static int tegra_pcie_enable_controller(struct tegra_pcie
>>> *pcie) +{ +	unsigned int timeout; +	unsigned long value; + +	/*
>>> enable dual controller and both ports */ +	value =
>>> afi_readl(pcie, AFI_PCIE_CONFIG); +	value &=
>>> ~(AFI_PCIE_CONFIG_PCIEC0_DISABLE_DEVICE | +
>>> AFI_PCIE_CONFIG_PCIEC1_DISABLE_DEVICE | +
>>> AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_MASK); +	value |=
>>> AFI_PCIE_CONFIG_SM2TMS0_XBAR_CONFIG_DUAL; +	afi_writel(pcie,
>>> value, AFI_PCIE_CONFIG);
>> 
>> Eventually, we should probably derive the port enables from the
>> state of the root port DT nodes, so that we can disable some and
>> presumably save a little power. Also, I notice that the
>> nvidia,num-lanes property isn't implemented yet. Still, we can
>> probably take care of this later.
> 
> Yes, the plan was to eventually derive the disable bits from the
> port status and setup the XBAR_CONFIG field based on the
> combination of nvidia,num-lanes properties.
> 
> I assume we should simply fail if the configuration specified by 
> nvidia,num-lanes is invalid?

Makes sense to me.

>>> +static int tegra_pcie_parse_dt(struct tegra_pcie *pcie)
>> 
>>> +	pcie->vdd_supply = devm_regulator_get(pcie->dev, "vdd"); +	if
>>> (IS_ERR(pcie->vdd_supply)) +		return
>>> PTR_ERR(pcie->vdd_supply); + +	pcie->pex_clk_supply =
>>> devm_regulator_get(pcie->dev, "pex-clk"); +	if
>>> (IS_ERR(pcie->pex_clk_supply)) +		return
>>> PTR_ERR(pcie->pex_clk_supply);
>> 
>> Oh, I guess the regulator_get() calls are already strict.
> 
> Yeah, I think they can't return NULL, right? In that case I can
> just drop the extra checks in tegra_pcie_power_{on,off}().

It looks like NULL can be returned if !CONFIG_REGULATOR. The comment
in the dummy implementation implies that drivers should treat a NULL
value as valid regulator in most cases.

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2222 bytes --]

On Fri, Jan 11, 2013 at 04:45:16PM +0100, Thierry Reding wrote:
> On Fri, Jan 11, 2013 at 03:36:14PM +0000, Arnd Bergmann wrote:
> > On Friday 11 January 2013, Thierry Reding wrote:
> > > Right, it'll need #ifdefs around the arch_{setup,teardown}_msi_irq(). Or
> > > select PCI_MSI unconditionally. Once this is merged I was going to post
> > > a patch that enables PCI_MSI in tegra_defconfig anyway. But it might be
> > > better to keep it optional anyway since the remainder of the code copes
> > > with it properly.
> > > 
> > Actually, we need something better than that. You cannot define
> > arch_setup_msi_irq in a tegra specific pci host driver, because that
> > will seriously mess up other platforms in multiplatform configurations
> > by giving a link error when they also define this function, or with a
> > run-time error when they don't support it.
> > 
> > I think what we should do here is fix it the right way by adding
> > a pci host specific callback rather than an architecture specific
> > callback in drivers/pci/msi.c. There is already a default version
> > of arch_setup_msi_irqs (with s), and we can probably do the
> > same for arch_setup_msi_irq (without s) to fall back to the
> > arch version for most architectures.
> > Most architectures (at least powerpc, sparc, ia64 and x86) already
> > multiplex the msi handlers internally, but ARM does not because
> > there is only one implementation (iop33x) at the moment.
> > 
> > We can add a generix multiplex and then move architectures over to
> > use it.
> 
> I already hinted at that in one of the other subthreads. Having such a
> multiplex would also allow the driver to be built as a module. I had
> already thought about this when I was working on an earlier version of
> these patches. Basically these would be two ops attached to the host
> bridge, and the generic arch_setup_msi_irq() could then look that up
> given the struct pci_dev that is passed to it and call this new per-
> host bridge .setup_msi_irq().

struct pci_ops looks like a good place to put these. They'll be
available from each struct pci_bus, so should be easy to call from
arch_setup_msi_irq().

Any objections?

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Arnd Bergmann]

On Saturday 12 January 2013, Thierry Reding wrote:
> > I already hinted at that in one of the other subthreads. Having such a
> > multiplex would also allow the driver to be built as a module. I had
> > already thought about this when I was working on an earlier version of
> > these patches. Basically these would be two ops attached to the host
> > bridge, and the generic arch_setup_msi_irq() could then look that up
> > given the struct pci_dev that is passed to it and call this new per-
> > host bridge .setup_msi_irq().
> 
> struct pci_ops looks like a good place to put these. They'll be
> available from each struct pci_bus, so should be easy to call from
> arch_setup_msi_irq().
> 
> Any objections?
> 

struct pci_ops has a long history of being specifically about
config space read/write operations, so on the one hand it does
not feel like the right place to put interrupt specific operations,
but on the other hand, the name sounds appropriate and I cannot
think of any other place to put this, so it's fine with me.

The only alternative I can think of is to introduce a new
structure next to it in struct pci_bus, but that feels a bit
pointless. Maybe Bjorn has a preference one way or the other.

	Arnd

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1557 bytes --]

On Sat, Jan 12, 2013 at 09:12:25PM +0000, Arnd Bergmann wrote:
> On Saturday 12 January 2013, Thierry Reding wrote:
> > > I already hinted at that in one of the other subthreads. Having such a
> > > multiplex would also allow the driver to be built as a module. I had
> > > already thought about this when I was working on an earlier version of
> > > these patches. Basically these would be two ops attached to the host
> > > bridge, and the generic arch_setup_msi_irq() could then look that up
> > > given the struct pci_dev that is passed to it and call this new per-
> > > host bridge .setup_msi_irq().
> > 
> > struct pci_ops looks like a good place to put these. They'll be
> > available from each struct pci_bus, so should be easy to call from
> > arch_setup_msi_irq().
> > 
> > Any objections?
> > 
> 
> struct pci_ops has a long history of being specifically about
> config space read/write operations, so on the one hand it does
> not feel like the right place to put interrupt specific operations,
> but on the other hand, the name sounds appropriate and I cannot
> think of any other place to put this, so it's fine with me.
> 
> The only alternative I can think of is to introduce a new
> structure next to it in struct pci_bus, but that feels a bit
> pointless. Maybe Bjorn has a preference one way or the other.

The name pci_ops is certainly generic enough. Also the comment above the
structure declaration says "Low-level architecture-dependent routines",
which applies to the MSI functions as well.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Andrew Murray]

On Sun, Jan 13, 2013 at 09:58:06AM +0000, Thierry Reding wrote:
> On Sat, Jan 12, 2013 at 09:12:25PM +0000, Arnd Bergmann wrote:
> > On Saturday 12 January 2013, Thierry Reding wrote:
> > > > I already hinted at that in one of the other subthreads. Having such a
> > > > multiplex would also allow the driver to be built as a module. I had
> > > > already thought about this when I was working on an earlier version of
> > > > these patches. Basically these would be two ops attached to the host
> > > > bridge, and the generic arch_setup_msi_irq() could then look that up
> > > > given the struct pci_dev that is passed to it and call this new per-
> > > > host bridge .setup_msi_irq().
> > > 
> > > struct pci_ops looks like a good place to put these. They'll be
> > > available from each struct pci_bus, so should be easy to call from
> > > arch_setup_msi_irq().
> > > 
> > > Any objections?
> > > 
> > 
> > struct pci_ops has a long history of being specifically about
> > config space read/write operations, so on the one hand it does
> > not feel like the right place to put interrupt specific operations,
> > but on the other hand, the name sounds appropriate and I cannot
> > think of any other place to put this, so it's fine with me.
> > 
> > The only alternative I can think of is to introduce a new
> > structure next to it in struct pci_bus, but that feels a bit
> > pointless. Maybe Bjorn has a preference one way or the other.
> 
> The name pci_ops is certainly generic enough. Also the comment above the
> structure declaration says "Low-level architecture-dependent routines",
> which applies to the MSI functions as well.

I've previously looked into this. It seems that architectures handle this
in different ways, some use vector tables, others use a multiplex and others
just let the end user implement the callback directly.

I've made an attempt to find a more common way. Though my implementation, which
I will try to share later today for reference provides a registration function
in drivers/pci/msi.c to provide implementations of the
(setup|teardown)_msi_irq(s) ops. This seems slightly better than the current
approach and doesn't break existing users - but is still ugly.

At present the PCI and MSI frameworks are largely uncoupled from each other and
so I was keen to not pollute PCI structures (e.g. pci_ops) with MSI ops. Just
because most PCI host bridges also provide MSI support I don't think there is a
reason why they should always come as a pair or be provided by the same chip.

Perhaps the solution is to support MSI controller drivers and a means to
associate them with PCI host controller drivers?

Andrew Murray

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 3171 bytes --]

On Mon, Jan 14, 2013 at 09:57:07AM +0000, Andrew Murray wrote:
> On Sun, Jan 13, 2013 at 09:58:06AM +0000, Thierry Reding wrote:
> > On Sat, Jan 12, 2013 at 09:12:25PM +0000, Arnd Bergmann wrote:
> > > On Saturday 12 January 2013, Thierry Reding wrote:
> > > > > I already hinted at that in one of the other subthreads. Having such a
> > > > > multiplex would also allow the driver to be built as a module. I had
> > > > > already thought about this when I was working on an earlier version of
> > > > > these patches. Basically these would be two ops attached to the host
> > > > > bridge, and the generic arch_setup_msi_irq() could then look that up
> > > > > given the struct pci_dev that is passed to it and call this new per-
> > > > > host bridge .setup_msi_irq().
> > > > 
> > > > struct pci_ops looks like a good place to put these. They'll be
> > > > available from each struct pci_bus, so should be easy to call from
> > > > arch_setup_msi_irq().
> > > > 
> > > > Any objections?
> > > > 
> > > 
> > > struct pci_ops has a long history of being specifically about
> > > config space read/write operations, so on the one hand it does
> > > not feel like the right place to put interrupt specific operations,
> > > but on the other hand, the name sounds appropriate and I cannot
> > > think of any other place to put this, so it's fine with me.
> > > 
> > > The only alternative I can think of is to introduce a new
> > > structure next to it in struct pci_bus, but that feels a bit
> > > pointless. Maybe Bjorn has a preference one way or the other.
> > 
> > The name pci_ops is certainly generic enough. Also the comment above the
> > structure declaration says "Low-level architecture-dependent routines",
> > which applies to the MSI functions as well.
> 
> I've previously looked into this. It seems that architectures handle this
> in different ways, some use vector tables, others use a multiplex and others
> just let the end user implement the callback directly.
> 
> I've made an attempt to find a more common way. Though my implementation, which
> I will try to share later today for reference provides a registration function
> in drivers/pci/msi.c to provide implementations of the
> (setup|teardown)_msi_irq(s) ops. This seems slightly better than the current
> approach and doesn't break existing users - but is still ugly.
> 
> At present the PCI and MSI frameworks are largely uncoupled from each other and
> so I was keen to not pollute PCI structures (e.g. pci_ops) with MSI ops. Just
> because most PCI host bridges also provide MSI support I don't think there is a
> reason why they should always come as a pair or be provided by the same chip.
> 
> Perhaps the solution is to support MSI controller drivers and a means to
> associate them with PCI host controller drivers?

I'm not sure I follow you're reasoning here. Is it possible to use MSIs
without PCI? If not then I think there's little sense in keeping the
implementations separate.

Furthermore, if MSI controller and PCI host bridge are separate entities
how do you look up the MSI controller given a PCI device?

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Arnd Bergmann]

On Tuesday 15 January 2013, Thierry Reding wrote:
> I'm not sure I follow you're reasoning here. Is it possible to use MSIs
> without PCI? If not then I think there's little sense in keeping the
> implementations separate.

Conceptually, you can use MSI for any device, but the Linux interfaces
for MSI are tied to PCI. If you use an MSI controller for a non-PCI
device, it would probably just appear as a regular interrupt controller.

> Furthermore, if MSI controller and PCI host bridge are separate entities
> how do you look up the MSI controller given a PCI device?

The host bridge can contain a pointer ot the MSI controller. You can
have multiple host bridges sharing a single MSI controller or you
can have separate ones for each host.

	Arnd

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Andrew Murray]

On Tue, Jan 15, 2013 at 12:44:12PM +0000, Arnd Bergmann wrote:
> On Tuesday 15 January 2013, Thierry Reding wrote:
> > I'm not sure I follow you're reasoning here. Is it possible to use MSIs
> > without PCI? If not then I think there's little sense in keeping the
> > implementations separate.
> 
> Conceptually, you can use MSI for any device, but the Linux interfaces
> for MSI are tied to PCI. If you use an MSI controller for a non-PCI
> device, it would probably just appear as a regular interrupt controller.
> 
> > Furthermore, if MSI controller and PCI host bridge are separate entities
> > how do you look up the MSI controller given a PCI device?
> 
> The host bridge can contain a pointer ot the MSI controller. You can
> have multiple host bridges sharing a single MSI controller or you
> can have separate ones for each host.

Yes and I hoped this relationship would be described by a device tree phandle
as is done for relating devices to their interrupt-parent (where device trees
are used). This would provide (arguably unnecessarily) greater flexibility,
e.g. if you have two PCI/MSI controller pairs, the MSIs only offer limited MSIs
and you only use one PCI fabric - you could service different parts of the
fabric by different MSI controllers (assuming you relate MSI controllers to
part of the fabric and that you'd want to). Perhaps there would be benefits for
virtualisation as well?

Andrew Murray

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1763 bytes --]

On Tue, Jan 15, 2013 at 03:40:38PM +0000, Andrew Murray wrote:
> On Tue, Jan 15, 2013 at 12:44:12PM +0000, Arnd Bergmann wrote:
> > On Tuesday 15 January 2013, Thierry Reding wrote:
> > > I'm not sure I follow you're reasoning here. Is it possible to use MSIs
> > > without PCI? If not then I think there's little sense in keeping the
> > > implementations separate.
> > 
> > Conceptually, you can use MSI for any device, but the Linux interfaces
> > for MSI are tied to PCI. If you use an MSI controller for a non-PCI
> > device, it would probably just appear as a regular interrupt controller.
> > 
> > > Furthermore, if MSI controller and PCI host bridge are separate entities
> > > how do you look up the MSI controller given a PCI device?
> > 
> > The host bridge can contain a pointer ot the MSI controller. You can
> > have multiple host bridges sharing a single MSI controller or you
> > can have separate ones for each host.
> 
> Yes and I hoped this relationship would be described by a device tree phandle
> as is done for relating devices to their interrupt-parent (where device trees
> are used). This would provide (arguably unnecessarily) greater flexibility,
> e.g. if you have two PCI/MSI controller pairs, the MSIs only offer limited MSIs
> and you only use one PCI fabric - you could service different parts of the
> fabric by different MSI controllers (assuming you relate MSI controllers to
> part of the fabric and that you'd want to). Perhaps there would be benefits for
> virtualisation as well?

Is there actually hardware that supports this? I assumed that the MSI
controller would have to be tightly coupled to the PCI host bridge in
order to raise an interrupt when an MSI is received via PCI.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2504 bytes --]

On Thu, Jan 10, 2013 at 12:24:17PM -0700, Jason Gunthorpe wrote:
> On Thu, Jan 10, 2013 at 08:03:27PM +0100, Thierry Reding wrote:
> 
> > > > You'd piece a mapping together, each bus requires 16 64k mappings, a
> > > > simple 2d array of busnr*16 of pointers would do the trick. A more
> > > > clever solution would be to allocate contiguous virtual memory and
> > > > split that up..
>  
> > > Oh, I see. I'm not very familiar with the internals of remapping, so
> > > I'll need to do some more reading. Thanks for the hints.
> > 
> > I forgot to ask. What's the advantage of having a contiguous virtual
> > memory area and splitting it up versus remapping each chunk separately?
> 
> Not alot, really, but it saves you from the pointer array and
> associated overhead. IIRC it is fairly easy to do in the kernel.
> 
> Arnd's version is good too, but you would be restricted to aligned
> powers of two for the bus number range in the DT, which is probably
> not that big a deal either?

I've been trying to make this work, but this implementation always
triggers a BUG_ON() in lib/ioremap.c, line 27:

	27 		BUG_ON(!pte_none(*pte));

which seems to indicate that the page is already mapped, right?

Below is the relevant code:

struct tegra_pcie_bus {
	struct vm_struct *area;
	struct list_head list;
	unsigned int nr;
};

static struct tegra_pcie_bus *tegra_pcie_bus_alloc(struct tegra_pcie *pcie,
						   unsigned int busnr)
{
	unsigned long flags = VM_READ | VM_WRITE | VM_IO | VM_PFNMAP |
			      VM_DONTEXPAND | VM_DONTDUMP;
	phys_addr_t cs = pcie->cs->start;
	struct tegra_pcie_bus *bus;
	struct vm_struct *vm;
	unsigned int i;
	int err;

	bus = devm_kzalloc(pcie->dev, sizeof(*bus), GFP_KERNEL);
	if (!bus)
		return ERR_PTR(-ENOMEM);

	INIT_LIST_HEAD(&bus->list);
	bus->nr = busnr;

	bus->area = get_vm_area(SZ_1M, VM_IOREMAP);
	if (!bus->area) {
		err = -ENOMEM;
		goto free;
	}

	for (i = 0; i < 16; i++) {
		unsigned long virt = (unsigned long)bus->area->addr +
				     i * SZ_64K;
		phys_addr_t phys = cs + busnr * SZ_64K + i * SZ_1M;

		err = ioremap_page_range(virt, virt + SZ_64K - 1, phys,
					 vm_get_page_prot(flags));
		if (err < 0) {
			dev_err(pcie->dev, "ioremap_page_range() failed: %d\n",
				err);
			goto unmap;
		}
	}

	return bus;

unmap:
	vunmap(bus->area->addr);
	free_vm_area(bus->area);
free:
	devm_kfree(pcie->dev, bus);
	return ERR_PTR(err);
}

Anybody see what's wrong with that?

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Russell King - ARM Linux]

On Wed, Jan 16, 2013 at 11:18:22AM +0100, Thierry Reding wrote:
> 		err = ioremap_page_range(virt, virt + SZ_64K - 1, phys,

Why -1 here?

Re: [PATCH 05/14] lib: Add I/O map cache implementation

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

On Wed, Jan 16, 2013 at 11:25:56AM +0000, Russell King - ARM Linux wrote:
> On Wed, Jan 16, 2013 at 11:18:22AM +0100, Thierry Reding wrote:
> > 		err = ioremap_page_range(virt, virt + SZ_64K - 1, phys,
> 
> Why -1 here?

Right, I forgot that end in these functions always means one byte after
the end. Removing the -1 seems to get past the remapping at least.
Reading the configuration space through the mapping doesn't though. I'll
investigate some more.

Thanks for pointing this out Russell.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Arnd Bergmann]

On Tuesday 15 January 2013, Thierry Reding wrote:
> Is there actually hardware that supports this? I assumed that the MSI
> controller would have to be tightly coupled to the PCI host bridge in
> order to raise an interrupt when an MSI is received via PCI.

No, as long as it's guaranteed that the MSI notification won't arrive
at the CPU before any inbound DMA data before it, the MSI controller
can be anywhere. Typically, the MSI controller is actually closer to
the CPU core than to the PCI bridge. On X86, I believe the MSI address
is on normally on the the "local APIC" on each CPU.

	Arnd

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Andrew Murray]

On Wed, Jan 16, 2013 at 02:00:26PM +0000, Arnd Bergmann wrote:
> On Tuesday 15 January 2013, Thierry Reding wrote:
> > Is there actually hardware that supports this? I assumed that the MSI
> > controller would have to be tightly coupled to the PCI host bridge in
> > order to raise an interrupt when an MSI is received via PCI.
> 
> No, as long as it's guaranteed that the MSI notification won't arrive
> at the CPU before any inbound DMA data before it, the MSI controller
> can be anywhere. Typically, the MSI controller is actually closer to
> the CPU core than to the PCI bridge. On X86, I believe the MSI address
> is on normally on the the "local APIC" on each CPU.

MSIs are indistinguishable from other memory-write transactions originating
from the RC other than the address they target. Anything that can capture
that write in the address space (even a page fault) could be an MSI controller
and call interrupt handlers. And so the RC / MSI controllers don't need to
be aware of each other.

Andrew Murray

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]

On Wed, Jan 16, 2013 at 04:17:16PM +0000, Andrew Murray wrote:
> On Wed, Jan 16, 2013 at 02:00:26PM +0000, Arnd Bergmann wrote:
> > On Tuesday 15 January 2013, Thierry Reding wrote:
> > > Is there actually hardware that supports this? I assumed that the MSI
> > > controller would have to be tightly coupled to the PCI host bridge in
> > > order to raise an interrupt when an MSI is received via PCI.
> > 
> > No, as long as it's guaranteed that the MSI notification won't arrive
> > at the CPU before any inbound DMA data before it, the MSI controller
> > can be anywhere. Typically, the MSI controller is actually closer to
> > the CPU core than to the PCI bridge. On X86, I believe the MSI address
> > is on normally on the the "local APIC" on each CPU.
> 
> MSIs are indistinguishable from other memory-write transactions originating
> from the RC other than the address they target. Anything that can capture
> that write in the address space (even a page fault) could be an MSI controller
> and call interrupt handlers. And so the RC / MSI controllers don't need to
> be aware of each other.

Alright, putting the functions into pci_ops doesn't sound like a very
good idea then. Or perhaps it would make sense for hardware where the
root complex and the MSI controller are handled by the same driver.
Basically it could be done as a shortcut and if those are not filled
in, the drivers could still opt to look up an MSI controller from a
phandle specified in DT.

Even another alternative would be to keep the functions within the
struct pci_ops and use generic ones if an external MSI controller is
used. Just tossing around ideas.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Andrew Murray]

On Wed, Jan 16, 2013 at 06:31:01PM +0000, Thierry Reding wrote:
> Alright, putting the functions into pci_ops doesn't sound like a very
> good idea then. Or perhaps it would make sense for hardware where the
> root complex and the MSI controller are handled by the same driver.
> Basically it could be done as a shortcut and if those are not filled
> in, the drivers could still opt to look up an MSI controller from a
> phandle specified in DT.
> 
> Even another alternative would be to keep the functions within the
> struct pci_ops and use generic ones if an external MSI controller is
> used. Just tossing around ideas.

I think an ideal solution would be for additional logic in drivers/msi.c
(e.g. in functions like msi_capability_init) to determine (based on the
passed in pci_dev) which MSI controller ops to use. I'm not sure the best
way to implement an association between an MSI controller and PCI busses
(I believe arch/sparc does something like this - perhaps there will be
inspiration there).

As you've pointed out, most RCs will have their own MSI controllers - so
it should be easy to register and associate both together.

I've submitted my previous work on MSI controller registration, but it
doesn't quite solve this problem - perhaps it can be a starting point?

Andrew Murray

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2012 bytes --]

On Thu, Jan 17, 2013 at 03:42:36PM +0000, Andrew Murray wrote:
> On Wed, Jan 16, 2013 at 06:31:01PM +0000, Thierry Reding wrote:
> > Alright, putting the functions into pci_ops doesn't sound like a very
> > good idea then. Or perhaps it would make sense for hardware where the
> > root complex and the MSI controller are handled by the same driver.
> > Basically it could be done as a shortcut and if those are not filled
> > in, the drivers could still opt to look up an MSI controller from a
> > phandle specified in DT.
> > 
> > Even another alternative would be to keep the functions within the
> > struct pci_ops and use generic ones if an external MSI controller is
> > used. Just tossing around ideas.
> 
> I think an ideal solution would be for additional logic in drivers/msi.c
> (e.g. in functions like msi_capability_init) to determine (based on the
> passed in pci_dev) which MSI controller ops to use. I'm not sure the best
> way to implement an association between an MSI controller and PCI busses
> (I believe arch/sparc does something like this - perhaps there will be
> inspiration there).
> 
> As you've pointed out, most RCs will have their own MSI controllers - so
> it should be easy to register and associate both together.
> 
> I've submitted my previous work on MSI controller registration, but it
> doesn't quite solve this problem - perhaps it can be a starting point?

We basically have two cases:

  - The PCI host bridge contains registers for MSI support. In that case
    it makes little sense to uncouple the MSI implementation from the
    host bridge driver.

  - An MSI controller exists outside of the PCI host bridge. The PCI
    host bridge would in that case have to lookup an MSI controller (via
    DT phandle or some other method).

In either of those cases, does it make sense to use the MSI support
outside the scope of the PCI infrastructure? That is, would devices
other than PCI devices be able to generate an MSI?

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Andrew Murray]

On Thu, Jan 17, 2013 at 04:05:02PM +0000, Thierry Reding wrote:
> On Thu, Jan 17, 2013 at 03:42:36PM +0000, Andrew Murray wrote:
> > On Wed, Jan 16, 2013 at 06:31:01PM +0000, Thierry Reding wrote:
> > > Alright, putting the functions into pci_ops doesn't sound like a very
> > > good idea then. Or perhaps it would make sense for hardware where the
> > > root complex and the MSI controller are handled by the same driver.
> > > Basically it could be done as a shortcut and if those are not filled
> > > in, the drivers could still opt to look up an MSI controller from a
> > > phandle specified in DT.
> > > 
> > > Even another alternative would be to keep the functions within the
> > > struct pci_ops and use generic ones if an external MSI controller is
> > > used. Just tossing around ideas.
> > 
> > I think an ideal solution would be for additional logic in drivers/msi.c
> > (e.g. in functions like msi_capability_init) to determine (based on the
> > passed in pci_dev) which MSI controller ops to use. I'm not sure the best
> > way to implement an association between an MSI controller and PCI busses
> > (I believe arch/sparc does something like this - perhaps there will be
> > inspiration there).
> > 
> > As you've pointed out, most RCs will have their own MSI controllers - so
> > it should be easy to register and associate both together.
> > 
> > I've submitted my previous work on MSI controller registration, but it
> > doesn't quite solve this problem - perhaps it can be a starting point?
> 
> We basically have two cases:
> 
>   - The PCI host bridge contains registers for MSI support. In that case
>     it makes little sense to uncouple the MSI implementation from the
>     host bridge driver.
> 
>   - An MSI controller exists outside of the PCI host bridge. The PCI
>     host bridge would in that case have to lookup an MSI controller (via
>     DT phandle or some other method).
> 
> In either of those cases, does it make sense to use the MSI support
> outside the scope of the PCI infrastructure? That is, would devices
> other than PCI devices be able to generate an MSI?

I've come around to your way of thinking. Your approach sounds good for
registration of MSI ops - let the RC host driver do it (it probably has its
own), or use a helper for following a phandle to get ops that are not part
of the driver. MSIs won't be used outside of PCI devices.

Though existing drivers will use MSI framework functions to request MSIs, that
will result in callbacks to the arch_setup_msi_irqs type functions. These
functions would need to be updated to find these new ops if they exist, i.e. by
traversing the pci_dev structure up to the RC and finding a suitable structure.

Perhaps the msi ops could live alongside pci_ops in the pci_bus structure. This
way when traversing up the buses from the provided pci_dev - the first bus with
msi ops populated would be used?

If no ops are found, the standard arch callbacks can be called - thus preserving
exiting functionality.

Andrew Murray

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 4638 bytes --]

On Thu, Jan 17, 2013 at 04:22:18PM +0000, Andrew Murray wrote:
> On Thu, Jan 17, 2013 at 04:05:02PM +0000, Thierry Reding wrote:
> > On Thu, Jan 17, 2013 at 03:42:36PM +0000, Andrew Murray wrote:
> > > On Wed, Jan 16, 2013 at 06:31:01PM +0000, Thierry Reding wrote:
> > > > Alright, putting the functions into pci_ops doesn't sound like a very
> > > > good idea then. Or perhaps it would make sense for hardware where the
> > > > root complex and the MSI controller are handled by the same driver.
> > > > Basically it could be done as a shortcut and if those are not filled
> > > > in, the drivers could still opt to look up an MSI controller from a
> > > > phandle specified in DT.
> > > > 
> > > > Even another alternative would be to keep the functions within the
> > > > struct pci_ops and use generic ones if an external MSI controller is
> > > > used. Just tossing around ideas.
> > > 
> > > I think an ideal solution would be for additional logic in drivers/msi.c
> > > (e.g. in functions like msi_capability_init) to determine (based on the
> > > passed in pci_dev) which MSI controller ops to use. I'm not sure the best
> > > way to implement an association between an MSI controller and PCI busses
> > > (I believe arch/sparc does something like this - perhaps there will be
> > > inspiration there).
> > > 
> > > As you've pointed out, most RCs will have their own MSI controllers - so
> > > it should be easy to register and associate both together.
> > > 
> > > I've submitted my previous work on MSI controller registration, but it
> > > doesn't quite solve this problem - perhaps it can be a starting point?
> > 
> > We basically have two cases:
> > 
> >   - The PCI host bridge contains registers for MSI support. In that case
> >     it makes little sense to uncouple the MSI implementation from the
> >     host bridge driver.
> > 
> >   - An MSI controller exists outside of the PCI host bridge. The PCI
> >     host bridge would in that case have to lookup an MSI controller (via
> >     DT phandle or some other method).
> > 
> > In either of those cases, does it make sense to use the MSI support
> > outside the scope of the PCI infrastructure? That is, would devices
> > other than PCI devices be able to generate an MSI?
> 
> I've come around to your way of thinking. Your approach sounds good for
> registration of MSI ops - let the RC host driver do it (it probably has its
> own), or use a helper for following a phandle to get ops that are not part
> of the driver. MSIs won't be used outside of PCI devices.
> 
> Though existing drivers will use MSI framework functions to request MSIs, that
> will result in callbacks to the arch_setup_msi_irqs type functions. These
> functions would need to be updated to find these new ops if they exist, i.e. by
> traversing the pci_dev structure up to the RC and finding a suitable structure.
> 
> Perhaps the msi ops could live alongside pci_ops in the pci_bus structure. This
> way when traversing up the buses from the provided pci_dev - the first bus with
> msi ops populated would be used?
> 
> If no ops are found, the standard arch callbacks can be called - thus preserving
> exiting functionality.

Yes, what you describe is exactly what I had in mind. I've been thinking
about a possible implementation and there may be some details that could
prove difficult to resolve. For instance, we likely need to pass context
around for the MSI ops, or else make sure that they can find the context
from the struct pci_dev or by traversing upwards from it.

I think for the case where the MSI hardware is controlled by the same
driver as the PCI host bridge, doing this is easy because the context
could be part of the PCI host bridge context, which in case of Tegra is
stored in struct pci_bus' sysdata field (which is actually an ARM struct
pci_sys_data and in turn stores a pointer to the struct tegra_pcie in
the .private_data field). Other drivers often just use a global variable
assuming that there will only ever be a single instance of the PCI host
bridge.

If the MSI controller is external to the PCI host bridge, things get a
little more complicated. The easiest way would probably be to store the
context along with the PCI host bridge context and use simple wrappers
around the actual implementations to retrieve the PHB context and pass
the attached MSI context.

Maybe this could even be made more generic by adding a struct msi_ops *
along with a struct msi_chip * in struct pci_bus. Perhaps I should try
and code something up to make things more concrete.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Andrew Murray]

On Thu, Jan 17, 2013 at 08:30:10PM +0000, Thierry Reding wrote:
> On Thu, Jan 17, 2013 at 04:22:18PM +0000, Andrew Murray wrote:
> > On Thu, Jan 17, 2013 at 04:05:02PM +0000, Thierry Reding wrote:
> > > On Thu, Jan 17, 2013 at 03:42:36PM +0000, Andrew Murray wrote:
> > > > On Wed, Jan 16, 2013 at 06:31:01PM +0000, Thierry Reding wrote:
> > > > > Alright, putting the functions into pci_ops doesn't sound like a very
> > > > > good idea then. Or perhaps it would make sense for hardware where the
> > > > > root complex and the MSI controller are handled by the same driver.
> > > > > Basically it could be done as a shortcut and if those are not filled
> > > > > in, the drivers could still opt to look up an MSI controller from a
> > > > > phandle specified in DT.
> > > > > 
> > > > > Even another alternative would be to keep the functions within the
> > > > > struct pci_ops and use generic ones if an external MSI controller is
> > > > > used. Just tossing around ideas.
> > > > 
> > > > I think an ideal solution would be for additional logic in drivers/msi.c
> > > > (e.g. in functions like msi_capability_init) to determine (based on the
> > > > passed in pci_dev) which MSI controller ops to use. I'm not sure the best
> > > > way to implement an association between an MSI controller and PCI busses
> > > > (I believe arch/sparc does something like this - perhaps there will be
> > > > inspiration there).
> > > > 
> > > > As you've pointed out, most RCs will have their own MSI controllers - so
> > > > it should be easy to register and associate both together.
> > > > 
> > > > I've submitted my previous work on MSI controller registration, but it
> > > > doesn't quite solve this problem - perhaps it can be a starting point?
> > > 
> > > We basically have two cases:
> > > 
> > >   - The PCI host bridge contains registers for MSI support. In that case
> > >     it makes little sense to uncouple the MSI implementation from the
> > >     host bridge driver.
> > > 
> > >   - An MSI controller exists outside of the PCI host bridge. The PCI
> > >     host bridge would in that case have to lookup an MSI controller (via
> > >     DT phandle or some other method).
> > > 
> > > In either of those cases, does it make sense to use the MSI support
> > > outside the scope of the PCI infrastructure? That is, would devices
> > > other than PCI devices be able to generate an MSI?
> > 
> > I've come around to your way of thinking. Your approach sounds good for
> > registration of MSI ops - let the RC host driver do it (it probably has its
> > own), or use a helper for following a phandle to get ops that are not part
> > of the driver. MSIs won't be used outside of PCI devices.
> > 
> > Though existing drivers will use MSI framework functions to request MSIs, that
> > will result in callbacks to the arch_setup_msi_irqs type functions. These
> > functions would need to be updated to find these new ops if they exist, i.e. by
> > traversing the pci_dev structure up to the RC and finding a suitable structure.
> > 
> > Perhaps the msi ops could live alongside pci_ops in the pci_bus structure. This
> > way when traversing up the buses from the provided pci_dev - the first bus with
> > msi ops populated would be used?
> > 
> > If no ops are found, the standard arch callbacks can be called - thus preserving
> > exiting functionality.
> 
> Yes, what you describe is exactly what I had in mind. I've been thinking
> about a possible implementation and there may be some details that could
> prove difficult to resolve. For instance, we likely need to pass context
> around for the MSI ops, or else make sure that they can find the context
> from the struct pci_dev or by traversing upwards from it.
> 
> I think for the case where the MSI hardware is controlled by the same
> driver as the PCI host bridge, doing this is easy because the context
> could be part of the PCI host bridge context, which in case of Tegra is
> stored in struct pci_bus' sysdata field (which is actually an ARM struct
> pci_sys_data and in turn stores a pointer to the struct tegra_pcie in
> the .private_data field). Other drivers often just use a global variable
> assuming that there will only ever be a single instance of the PCI host
> bridge.

Yes.

> If the MSI controller is external to the PCI host bridge, things get a
> little more complicated. The easiest way would probably be to store the
> context along with the PCI host bridge context and use simple wrappers
> around the actual implementations to retrieve the PHB context and pass
> the attached MSI context.

This would be nice, but may not be necessary. The MSI controller could use
a global (file-scope) variable to hold context gained from its probe and
assume it will be the only instance of that MSI controller.

> 
> Maybe this could even be made more generic by adding a struct msi_ops *
> along with a struct msi_chip * in struct pci_bus. Perhaps I should try
> and code something up to make things more concrete.

This would be the most complete way to handle this.

Andrew Murray

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Andrew Murray]

On Wed, Jan 09, 2013 at 08:43:10PM +0000, Thierry Reding wrote:
> Move the PCIe driver from arch/arm/mach-tegra into the drivers/pci/host
> directory. The motivation is to collect various host controller drivers
> in the same location in order to facilitate refactoring.
> 
> The Tegra PCIe driver has been largely rewritten, both in order to turn
> it into a proper platform driver and to add MSI (based on code by
> Krishna Kishore <kthota@nvidia.com>) as well as device tree support.
> 
> Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>

[snip]

> +static int tegra_pcie_enable(struct tegra_pcie *pcie)
> +{
> +       struct hw_pci hw;
> +
> +       memset(&hw, 0, sizeof(hw));
> +
> +       hw.nr_controllers = 1;
> +       hw.private_data = (void **)&pcie;
> +       hw.setup = tegra_pcie_setup;
> +       hw.scan = tegra_pcie_scan_bus;
> +       hw.map_irq = tegra_pcie_map_irq;
> +
> +       pci_common_init(&hw);
> +
> +       return 0;
> +}

[snip]

> +static int tegra_pcie_probe(struct platform_device *pdev)
> +{
> +       struct device_node *port;
> +       struct tegra_pcie *pcie;
> +       int err;
> +
> +       pcie = devm_kzalloc(&pdev->dev, sizeof(*pcie), GFP_KERNEL);
> +       if (!pcie)
> +               return -ENOMEM;
> +
> +       INIT_LIST_HEAD(&pcie->ports);
> +       pcie->dev = &pdev->dev;
> +
> +       err = tegra_pcie_parse_dt(pcie);
> +       if (err < 0)
> +               return err;
> +
> +       pcibios_min_mem = 0;
> +
> +       err = tegra_pcie_get_resources(pcie);
> +       if (err < 0) {
> +               dev_err(&pdev->dev, "failed to request resources: %d\n", err);
> +               return err;
> +       }
> +
> +       err = tegra_pcie_enable_controller(pcie);
> +       if (err)
> +               goto put_resources;
> +
> +       /* probe root ports */
> +       for_each_child_of_node(pdev->dev.of_node, port) {
> +               if (!of_device_is_available(port))
> +                       continue;
> +
> +               err = tegra_pcie_add_port(pcie, port);
> +               if (err < 0) {
> +                       dev_err(&pdev->dev, "failed to add port %s: %d\n",
> +                               port->name, err);
> +               }
> +       }
> +
> +       /* setup the AFI address translations */
> +       tegra_pcie_setup_translations(pcie);
> +
> +       if (IS_ENABLED(CONFIG_PCI_MSI)) {
> +               err = tegra_pcie_enable_msi(pcie);
> +               if (err < 0) {
> +                       dev_err(&pdev->dev,
> +                               "failed to enable MSI support: %d\n",
> +                               err);
> +                       goto put_resources;
> +               }
> +       }
> +
> +       err = tegra_pcie_enable(pcie);
> +       if (err < 0) {
> +               dev_err(&pdev->dev, "failed to enable PCIe ports: %d\n", err);
> +               goto disable_msi;
> +       }
> +
> +       platform_set_drvdata(pdev, pcie);
> +       return 0;
> +
> +disable_msi:
> +       if (IS_ENABLED(CONFIG_PCI_MSI))
> +               tegra_pcie_disable_msi(pcie);
> +put_resources:
> +       tegra_pcie_put_resources(pcie);
> +       return err;
> +}
> +

[snip]

> +
> +static const struct of_device_id tegra_pcie_of_match[] = {
> +       { .compatible = "nvidia,tegra20-pcie", },
> +       { },
> +};
> +
> +static struct platform_driver tegra_pcie_driver = {
> +       .driver = {
> +               .name = "tegra-pcie",
> +               .owner = THIS_MODULE,
> +               .of_match_table = tegra_pcie_of_match,
> +       },
> +       .probe = tegra_pcie_probe,
> +       .remove = tegra_pcie_remove,
> +};
> +module_platform_driver(tegra_pcie_driver);

If you have multiple 'nvidia,tegra20-pcie's in your DT then you will end up
with multiple calls to tegra_pcie_probe/tegra_pcie_enable/pci_common_init.

However pci_common_init/pcibios_init_hw assumes it will only ever be called
once, and will thus result in trying to create multiple busses with the same
bus number. (The first root bus it creates is always zero provided you haven't
implemented hw->scan).

I have a patch for this if you want to fold it into your series? (I see you've
made changes to bios32 for per-controller data).

Andrew Murray

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 5191 bytes --]

On Fri, Jan 18, 2013 at 09:56:20AM +0000, Andrew Murray wrote:
> On Wed, Jan 09, 2013 at 08:43:10PM +0000, Thierry Reding wrote:
> > Move the PCIe driver from arch/arm/mach-tegra into the drivers/pci/host
> > directory. The motivation is to collect various host controller drivers
> > in the same location in order to facilitate refactoring.
> > 
> > The Tegra PCIe driver has been largely rewritten, both in order to turn
> > it into a proper platform driver and to add MSI (based on code by
> > Krishna Kishore <kthota@nvidia.com>) as well as device tree support.
> > 
> > Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> 
> [snip]
> 
> > +static int tegra_pcie_enable(struct tegra_pcie *pcie)
> > +{
> > +       struct hw_pci hw;
> > +
> > +       memset(&hw, 0, sizeof(hw));
> > +
> > +       hw.nr_controllers = 1;
> > +       hw.private_data = (void **)&pcie;
> > +       hw.setup = tegra_pcie_setup;
> > +       hw.scan = tegra_pcie_scan_bus;
> > +       hw.map_irq = tegra_pcie_map_irq;
> > +
> > +       pci_common_init(&hw);
> > +
> > +       return 0;
> > +}
> 
> [snip]
> 
> > +static int tegra_pcie_probe(struct platform_device *pdev)
> > +{
> > +       struct device_node *port;
> > +       struct tegra_pcie *pcie;
> > +       int err;
> > +
> > +       pcie = devm_kzalloc(&pdev->dev, sizeof(*pcie), GFP_KERNEL);
> > +       if (!pcie)
> > +               return -ENOMEM;
> > +
> > +       INIT_LIST_HEAD(&pcie->ports);
> > +       pcie->dev = &pdev->dev;
> > +
> > +       err = tegra_pcie_parse_dt(pcie);
> > +       if (err < 0)
> > +               return err;
> > +
> > +       pcibios_min_mem = 0;
> > +
> > +       err = tegra_pcie_get_resources(pcie);
> > +       if (err < 0) {
> > +               dev_err(&pdev->dev, "failed to request resources: %d\n", err);
> > +               return err;
> > +       }
> > +
> > +       err = tegra_pcie_enable_controller(pcie);
> > +       if (err)
> > +               goto put_resources;
> > +
> > +       /* probe root ports */
> > +       for_each_child_of_node(pdev->dev.of_node, port) {
> > +               if (!of_device_is_available(port))
> > +                       continue;
> > +
> > +               err = tegra_pcie_add_port(pcie, port);
> > +               if (err < 0) {
> > +                       dev_err(&pdev->dev, "failed to add port %s: %d\n",
> > +                               port->name, err);
> > +               }
> > +       }
> > +
> > +       /* setup the AFI address translations */
> > +       tegra_pcie_setup_translations(pcie);
> > +
> > +       if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > +               err = tegra_pcie_enable_msi(pcie);
> > +               if (err < 0) {
> > +                       dev_err(&pdev->dev,
> > +                               "failed to enable MSI support: %d\n",
> > +                               err);
> > +                       goto put_resources;
> > +               }
> > +       }
> > +
> > +       err = tegra_pcie_enable(pcie);
> > +       if (err < 0) {
> > +               dev_err(&pdev->dev, "failed to enable PCIe ports: %d\n", err);
> > +               goto disable_msi;
> > +       }
> > +
> > +       platform_set_drvdata(pdev, pcie);
> > +       return 0;
> > +
> > +disable_msi:
> > +       if (IS_ENABLED(CONFIG_PCI_MSI))
> > +               tegra_pcie_disable_msi(pcie);
> > +put_resources:
> > +       tegra_pcie_put_resources(pcie);
> > +       return err;
> > +}
> > +
> 
> [snip]
> 
> > +
> > +static const struct of_device_id tegra_pcie_of_match[] = {
> > +       { .compatible = "nvidia,tegra20-pcie", },
> > +       { },
> > +};
> > +
> > +static struct platform_driver tegra_pcie_driver = {
> > +       .driver = {
> > +               .name = "tegra-pcie",
> > +               .owner = THIS_MODULE,
> > +               .of_match_table = tegra_pcie_of_match,
> > +       },
> > +       .probe = tegra_pcie_probe,
> > +       .remove = tegra_pcie_remove,
> > +};
> > +module_platform_driver(tegra_pcie_driver);
> 
> If you have multiple 'nvidia,tegra20-pcie's in your DT then you will end up
> with multiple calls to tegra_pcie_probe/tegra_pcie_enable/pci_common_init.
> 
> However pci_common_init/pcibios_init_hw assumes it will only ever be called
> once, and will thus result in trying to create multiple busses with the same
> bus number. (The first root bus it creates is always zero provided you haven't
> implemented hw->scan).

Right, I hadn't noticed. There's currently no hardware that actually has
two PCIe host bridges but I wanted to keep the driver properly prepared
in case this ever happened.

Actually I've reimplemented hw->scan, but it still forwards the bus
number setup by pcibios_init_hw() (sys->busnr) to pci_create_root_bus()
so it will still break. I wonder, though, if a better approach would be
to take this number from the bus-range property in DT instead.

> I have a patch for this if you want to fold it into your series? (I see you've
> made changes to bios32 for per-controller data).

I would certainly like to take a look at it.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Jason Gunthorpe]

On Thu, Jan 17, 2013 at 04:22:18PM +0000, Andrew Murray wrote:

> > In either of those cases, does it make sense to use the MSI support
> > outside the scope of the PCI infrastructure? That is, would devices
> > other than PCI devices be able to generate an MSI?
> 
> I've come around to your way of thinking. Your approach sounds good for
> registration of MSI ops - let the RC host driver do it (it probably has its
> own), or use a helper for following a phandle to get ops that are not part
> of the driver. MSIs won't be used outside of PCI devices.

Here is a bit of additional info on some MSI stuff..

This can be pretty complex. For instance on hyper transport systems
the PCI to HT bridge has an MSI controller that maps between PCI and
HT MSI formats, that mapping is configurable, so technically each
brige could be considered a MSI controller. Typically the mapping
controllers are all setup the same so there is not much problem with
this. However *native* HT devices can (which are super rare) can use a
different MSI format than PCI devices. From a linux perspective HT is
just a variant of PCI.
 
On x86 the MSI is delivered to the CPU APIC complex which converts it
into a vectored interrupt - part of the value of MSI is that the MSI
data can vector the interrupt to a specific CPU, or group of CPUs or
whatever.

Presumably SMP ARMs will evolve similar MSI based interrupt vectoring
capabilities, and presumably on-chip, non-PCI peripherals will evolve
options to use MSI as well (ie multi-queue ethernet). So it might be
worth giving some thought to how things could migrate in that
direction someday.

I have a bit hacky MSI driver for Kirkwood, this work you have to
generalize the interface could let me actually upstream it :) The MSI
is built using the Host2CPU doorbell registers, so it is entirely
unrelated to the PCI-E RC driver.

However, my use of the MSI driver on kirkwood is to assign MSIs to a
PCI-E device via non-standard registers, more like an on chip
peripheral. This is because the Host2CPU doorbell doesn't fit 100%
perfectly with the standard PCI MSI stuff, and the hardware has funny
needs.. So an 'allocate a MSI interrupt' API would be snazzy too :)

Jason

Re: [PATCH 00/14] Rewrite Tegra PCIe driver

[Bjorn Helgaas]

On Wed, Jan 9, 2013 at 1:43 PM, Thierry Reding
<thierry.reding@avionic-design.de> wrote:
> This patch series contains an almost complete rewrite of the Tegra PCIe
> driver. The code is moved to the drivers/pci/host directory and turned
> into a proper platform driver, adding MSI and DT support while at it.
> Other PCI host controller drivers can be added to that directory in an
> attempt to make it easier to factor out common code.
>
> Patches 1 to 4 add generic OF helpers to parse a PCI node's ranges
> property as well as obtain the bus, device and function numbers from a
> node's reg property.
>
> Patch 5 provides an implementation of a cache for I/O mappings. This can
> be used in situations where a large physical memory region needs to be
> ioremap()'ed. On Tegra, the PCIe standard and extended configuration
> spaces can be accessed through a 256 MiB window. Mapping that in one
> chunk is wasteful and may not always work because the vmalloc area may
> not be large enough. The implementation in this patch uses an LRU based
> approach to map a limited amount of pages on an as-needed basis.
>
> Patches 6 and 7 prepare the ARM PCI code to enable PCI host controller
> drivers to load after the init stage, which can happen due to deferred
> probing, and to allow private data to be passed for each controller.
>
> Patches 8 and 9 move some of the Tegra specific code around to allow it
> to be used from outside the arch/arm/mach-tegra directory.
>
> Patch 10 moves the Tegra PCIe controller driver to the drivers/pci/host
> directory and turns it into a proper platform driver. It also adds MSI
> (based on patches by NVIDIA) and DT support.
>
> Patches 11 to 14 add device tree based probing for the TEC, Harmony and
> TrimSlice boards.
>
> Thierry
>
> Andrew Murray (1):
>   of/pci: Provide support for parsing PCI DT ranges property
>
> Thierry Reding (13):
>   of/pci: Add of_pci_get_devfn() function
>   of/pci: Add of_pci_get_bus() function
>   of/pci: Add of_pci_parse_bus_range() function
>   lib: Add I/O map cache implementation
>   ARM: pci: Keep pci_common_init() around after init
>   ARM: pci: Allow passing per-controller private data
>   ARM: tegra: Move tegra_pcie_xclk_clamp() to PMC
>   ARM: tegra: Move pmc.h to include/mach
>   PCI: tegra: Move PCIe driver to drivers/pci/host
>   ARM: tegra: tamonten: Add PCIe support
>   ARM: tegra: tec: Add PCIe support
>   ARM: tegra: harmony: Initialize PCIe from DT
>   ARM: tegra: trimslice: Initialize PCIe from DT
>
>  .../bindings/pci/nvidia,tegra20-pcie.txt           |  115 ++
>  arch/arm/Kconfig                                   |    2 +
>  arch/arm/boot/dts/tegra20-harmony.dts              |   20 +-
>  arch/arm/boot/dts/tegra20-tamonten.dtsi            |    2 +-
>  arch/arm/boot/dts/tegra20-tec.dts                  |  198 +++
>  arch/arm/boot/dts/tegra20-trimslice.dts            |   12 +
>  arch/arm/boot/dts/tegra20.dtsi                     |   45 +
>  arch/arm/include/asm/mach/pci.h                    |    1 +
>  arch/arm/kernel/bios32.c                           |    9 +-
>  arch/arm/mach-tegra/Kconfig                        |    5 -
>  arch/arm/mach-tegra/Makefile                       |    3 -
>  arch/arm/mach-tegra/board-dt-tegra20.c             |   24 -
>  arch/arm/mach-tegra/board-harmony-pcie.c           |   84 --
>  arch/arm/mach-tegra/board.h                        |    4 +-
>  arch/arm/mach-tegra/common.c                       |    2 +-
>  arch/arm/mach-tegra/include/mach/pmc.h             |   24 +
>  arch/arm/mach-tegra/iomap.h                        |    3 -
>  arch/arm/mach-tegra/pcie.c                         |  887 -------------
>  arch/arm/mach-tegra/pmc.c                          |   16 +
>  arch/arm/mach-tegra/pmc.h                          |   23 -
>  drivers/of/address.c                               |   63 +
>  drivers/of/of_pci.c                                |   78 +-
>  drivers/pci/Kconfig                                |    2 +
>  drivers/pci/Makefile                               |    3 +
>  drivers/pci/host/Kconfig                           |    8 +
>  drivers/pci/host/Makefile                          |    1 +
>  drivers/pci/host/pci-tegra.c                       | 1322 ++++++++++++++++++++
>  include/linux/io.h                                 |   12 +
>  include/linux/of_address.h                         |    9 +
>  include/linux/of_pci.h                             |    3 +
>  lib/ioremap.c                                      |  266 ++++
>  31 files changed, 2203 insertions(+), 1043 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/pci/nvidia,tegra20-pcie.txt
>  delete mode 100644 arch/arm/mach-tegra/board-harmony-pcie.c
>  create mode 100644 arch/arm/mach-tegra/include/mach/pmc.h
>  delete mode 100644 arch/arm/mach-tegra/pcie.c
>  delete mode 100644 arch/arm/mach-tegra/pmc.h
>  create mode 100644 drivers/pci/host/Kconfig
>  create mode 100644 drivers/pci/host/Makefile
>  create mode 100644 drivers/pci/host/pci-tegra.c

Hi Thierry,

Since the bulk of this series affects ARM, I'm assuming this will be
merged via the ARM tree or at least some non-PCI route.  It looks like
[10/14] is the only one that touches drivers/pci, and since it's only
adding new stuff (the rewrite of the Tegra PCIe driver, if I
understand correctly), that's fine with me.

There have been enough comments that I assume you'll be posting an
updated series soon.  I'll watch for it and review and ack the
drivers/pci parts.  Let me know if you want me to do anything else.

Bjorn

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Andrew Murray]

On Tue, Jan 22, 2013 at 07:29:01PM +0000, Jason Gunthorpe wrote:
> On Thu, Jan 17, 2013 at 04:22:18PM +0000, Andrew Murray wrote:
> 
> > > In either of those cases, does it make sense to use the MSI support
> > > outside the scope of the PCI infrastructure? That is, would devices
> > > other than PCI devices be able to generate an MSI?
> > 
> > I've come around to your way of thinking. Your approach sounds good for
> > registration of MSI ops - let the RC host driver do it (it probably has its
> > own), or use a helper for following a phandle to get ops that are not part
> > of the driver. MSIs won't be used outside of PCI devices.
> 
> Here is a bit of additional info on some MSI stuff..
> 
> This can be pretty complex. For instance on hyper transport systems
> the PCI to HT bridge has an MSI controller that maps between PCI and
> HT MSI formats, that mapping is configurable, so technically each
> brige could be considered a MSI controller. Typically the mapping
> controllers are all setup the same so there is not much problem with
> this. However *native* HT devices can (which are super rare) can use a
> different MSI format than PCI devices. From a linux perspective HT is
> just a variant of PCI.
>  
> On x86 the MSI is delivered to the CPU APIC complex which converts it
> into a vectored interrupt - part of the value of MSI is that the MSI
> data can vector the interrupt to a specific CPU, or group of CPUs or
> whatever.
> 
> Presumably SMP ARMs will evolve similar MSI based interrupt vectoring
> capabilities, and presumably on-chip, non-PCI peripherals will evolve
> options to use MSI as well (ie multi-queue ethernet). So it might be
> worth giving some thought to how things could migrate in that
> direction someday.
> 
> I have a bit hacky MSI driver for Kirkwood, this work you have to
> generalize the interface could let me actually upstream it :) The MSI
> is built using the Host2CPU doorbell registers, so it is entirely
> unrelated to the PCI-E RC driver.
> 
> However, my use of the MSI driver on kirkwood is to assign MSIs to a
> PCI-E device via non-standard registers, more like an on chip
> peripheral. This is because the Host2CPU doorbell doesn't fit 100%
> perfectly with the standard PCI MSI stuff, and the hardware has funny
> needs.. So an 'allocate a MSI interrupt' API would be snazzy too :)

Thanks for this. I believe Thierry may be working on improving the MSI
API - so perhaps we can see where that takes us.

Andrew Murray

Re: [PATCH 06/14] ARM: pci: Keep pci_common_init() around after init

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Wed, Jan 09, 2013 at 09:43:06PM +0100, Thierry Reding wrote:
> When using deferred driver probing, PCI host controller drivers may
> actually require this function after the init stage.
> 
> Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> ---
>  arch/arm/kernel/bios32.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

Russell,

Can this patch and patch 7 (ARM: pci: Allow passing per-controller
private data) of this series be applied for 3.9? Thomas uses them in his
Marvell PCIe series as well and it would allow to reduce the complexity
of the dependencies.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 06/14] ARM: pci: Keep pci_common_init() around after init

[Thomas Petazzoni]

Dear Thierry Reding,

On Wed,  9 Jan 2013 21:43:06 +0100, Thierry Reding wrote:
> When using deferred driver probing, PCI host controller drivers may
> actually require this function after the init stage.
> 
> Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>

Tested-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Re: [PATCH 06/14] ARM: pci: Keep pci_common_init() around after init

[Russell King - ARM Linux]

On Tue, Feb 05, 2013 at 09:41:48PM +0100, Thierry Reding wrote:
> On Wed, Jan 09, 2013 at 09:43:06PM +0100, Thierry Reding wrote:
> > When using deferred driver probing, PCI host controller drivers may
> > actually require this function after the init stage.
> > 
> > Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> > ---
> >  arch/arm/kernel/bios32.c | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> Russell,
> 
> Can this patch and patch 7 (ARM: pci: Allow passing per-controller
> private data) of this series be applied for 3.9? Thomas uses them in his
> Marvell PCIe series as well and it would allow to reduce the complexity
> of the dependencies.

It'll need to go into the patch system in that case...

Re: [PATCH 06/14] ARM: pci: Keep pci_common_init() around after init

[Linus Walleij]

On Wed, Jan 9, 2013 at 9:43 PM, Thierry Reding
<thierry.reding@avionic-design.de> wrote:

> When using deferred driver probing, PCI host controller drivers may
> actually require this function after the init stage.
>
> Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>

There seem to be a proliferation of these patches now.

Isn't this just papering over the real problem? The discarding
of __init sections need to happen *after* all deferred probes
are complete, lest we have to remove *all* __init sections from
*all* drivers in the kernel, don't we?

Yours,
Linus Walleij

Re: [PATCH 06/14] ARM: pci: Keep pci_common_init() around after init

[Linus Walleij]

On Thu, Feb 7, 2013 at 1:54 AM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday 06 February 2013 17:38:20 Linus Walleij wrote:
>> On Wed, Jan 9, 2013 at 9:43 PM, Thierry Reding
>> <thierry.reding@avionic-design.de> wrote:
>>
>> > When using deferred driver probing, PCI host controller drivers may
>> > actually require this function after the init stage.
>> >
>> > Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
>>
>> There seem to be a proliferation of these patches now.
>>
>> Isn't this just papering over the real problem? The discarding
>> of __init sections need to happen *after* all deferred probes
>> are complete, lest we have to remove *all* __init sections from
>> *all* drivers in the kernel, don't we?
>
> No, I think it's not quite that bad. I think the rule is still
> just that .probe() functions and anything called from them must
> not be __init. They used to be __devinit, which would cause
> problems with deferred probing on !HOTPLUG systems but that's
> gone in 3.9.
>
> Thierry's patch is just necessary because pci_common_init used
> to be called only from actual __init functions, and not it
> gets called from a .probe() function for the first time.

Aha OK, then it feels much better now.

However it leaves the question of how much __init, __initdata
and __initconst we have littering around. Oh, well, we'll see
I guess.

Thanks,
Linus Walleij

Re: [PATCH 06/14] ARM: pci: Keep pci_common_init() around after init

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 947 bytes --]

On Wed, Feb 06, 2013 at 04:30:41PM +0000, Russell King - ARM Linux wrote:
> On Tue, Feb 05, 2013 at 09:41:48PM +0100, Thierry Reding wrote:
> > On Wed, Jan 09, 2013 at 09:43:06PM +0100, Thierry Reding wrote:
> > > When using deferred driver probing, PCI host controller drivers may
> > > actually require this function after the init stage.
> > > 
> > > Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> > > ---
> > >  arch/arm/kernel/bios32.c | 6 +++---
> > >  1 file changed, 3 insertions(+), 3 deletions(-)
> > 
> > Russell,
> > 
> > Can this patch and patch 7 (ARM: pci: Allow passing per-controller
> > private data) of this series be applied for 3.9? Thomas uses them in his
> > Marvell PCIe series as well and it would allow to reduce the complexity
> > of the dependencies.
> 
> It'll need to go into the patch system in that case...

Alright, I'll submit them to the patch system. Thanks.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 06/14] ARM: pci: Keep pci_common_init() around after init

[Arnd Bergmann]

On Wednesday 06 February 2013 17:38:20 Linus Walleij wrote:
> On Wed, Jan 9, 2013 at 9:43 PM, Thierry Reding
> <thierry.reding@avionic-design.de> wrote:
> 
> > When using deferred driver probing, PCI host controller drivers may
> > actually require this function after the init stage.
> >
> > Signed-off-by: Thierry Reding <thierry.reding@avionic-design.de>
> 
> There seem to be a proliferation of these patches now.
> 
> Isn't this just papering over the real problem? The discarding
> of __init sections need to happen *after* all deferred probes
> are complete, lest we have to remove *all* __init sections from
> *all* drivers in the kernel, don't we?

No, I think it's not quite that bad. I think the rule is still
just that .probe() functions and anything called from them must
not be __init. They used to be __devinit, which would cause
problems with deferred probing on !HOTPLUG systems but that's
gone in 3.9.

Thierry's patch is just necessary because pci_common_init used
to be called only from actual __init functions, and not it
gets called from a .probe() function for the first time.

	Arnd

Re: [PATCH 06/14] ARM: pci: Keep pci_common_init() around after init

[Arnd Bergmann]

On Wednesday 06 February 2013 18:07:53 Linus Walleij wrote:
> However it leaves the question of how much __init, __initdata
> and __initconst we have littering around. Oh, well, we'll see
> I guess.

Actually, kbuild is pretty good at warning around the
bugs there.

	Arnd

Re: [PATCH 10/14] PCI: tegra: Move PCIe driver to drivers/pci/host

[Thomas Petazzoni]

Dear Grant Likely,

On Wed, 13 Feb 2013 22:59:50 +0000, Grant Likely wrote:

> There isn't a whole lot of value in this patch without another user.
> I'll need to see the other patches that make use of this.

See the proposed Marvell PCIe driver and Tegra PCIe driver:
 http://lists.infradead.org/pipermail/linux-arm-kernel/2013-February/149232.html
 http://lists.infradead.org/pipermail/linux-arm-kernel/2013-January/140649.html

Both Thierry (doing the Tegra PCIe driver) and myself (doing the
Marvell PCIe driver) already have fairly long patch series to add
support for the PCIe interfaces in our respective SoC. In order to ease
the process of getting those merged, we'd like to get some of the
base functions we depend on to be merged first, so that our patch
series become a bit smaller and therefore more manageable. My Marvell
PCIe patch series already has 32 patches (including the ones we are
currently discussing), and I will need even more patches for the
upcoming fourth version (due to additional comments made during the
review of the third version of the patch set).

So, it would really be helpful if this base infrastructure, for which
users already exist in the form of submitted patches, that have already
gone through multiple iterations, could be merged.

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com