[PATCH v6 0/3] [RFC] Add support for PCI in AArch64

[PATCH v6 0/3] [RFC] Add support for PCI in AArch64

[Liviu Dudau]

Hi,

This patch adds support for PCI to AArch64. It is based on my v6 patch
that adds support for creating generic host bridge structure from
device tree. With that in place, I was able to boot a platform that
has PCIe host bridge support and use a PCIe network card.

Changes from v5:
  - Removed pcibios_fixup_bridge_ranges() as the week default version is fine.
  - Removed the ALIGN() call in pcibios_align_resource()
  - Stopped exporting pcibios_align_resource()

Changes from v4:
  - Fixed the pci_domain_nr() implementation for arm64. Now we use
    find_pci_host_bride() to find the host bridge before we retrieve
    the domain number.

Changes from v3:
  - Added Acks accumulated so far ;)
  - Still carrying Catalin's patch for moving the PCI_IO_BASE until it
    lands in linux-next or mainline, in order to ease applying the series

Changes from v2:
  - Implement an arch specific version of pci_register_io_range() and
    pci_address_to_pio().
  - Return 1 from pci_proc_domain().

Changes from v1:
  - Added Catalin's patch for moving the PCI_IO_BASE location and extend
    its size to 16MB
  - Integrated Arnd's version of pci_ioremap_io that uses a bitmap for
    keeping track of assigned IO space and returns an io_offset. At the
    moment the code is added in arch/arm64 but it can be moved in drivers/pci.
  - Added a fix for the generic ioport_map() function when !CONFIG_GENERIC_IOMAP
    as suggested by Arnd.

v5 thread here: https://lkml.org/lkml/2014/3/4/307
v4 thread here: https://lkml.org/lkml/2014/3/3/298
v3 thread here: https://lkml.org/lkml/2014/2/28/211
v2 thread here: https://lkml.org/lkml/2014/2/27/255
v1 thread here: https://lkml.org/lkml/2014/2/3/389


The API used is different from the one used by ARM architecture. There is
no pci_common_init_dev() function and no hw_pci structure, as that is no
longer needed. Once the last signature is added to the legal agreement, I
will post the host bridge driver code that I am using. Meanwhile, here
is an example of what the probe function looks like, posted as an example:

static int myhostbridge_probe(struct platform_device *pdev)
{
	int err;
	struct device_node *dev;
	struct pci_host_bridge *bridge;
	struct myhostbridge_port *pp;
	resource_size_t lastbus;

	dev = pdev->dev.of_node;

	if (!of_device_is_available(dev)) {
		pr_warn("%s: disabled\n", dev->full_name);
		return -ENODEV;
	}

	pp = kzalloc(sizeof(struct myhostbridge_port), GFP_KERNEL);
	if (!pp)
		return -ENOMEM;

	bridge = of_create_pci_host_bridge(&pdev->dev, &myhostbridge_ops, pp);
	if (IS_ERR(bridge)) {
		err = PTR_ERR(bridge);
		goto bridge_create_fail;
	}

	err = myhostbridge_setup(bridge->bus);
	if (err)
		goto bridge_setup_fail;

	/* We always enable PCI domains and we keep domain 0 backward
	 * compatible in /proc for video cards
	 */
	pci_add_flags(PCI_ENABLE_PROC_DOMAINS);
	pci_add_flags(PCI_REASSIGN_ALL_BUS | PCI_REASSIGN_ALL_RSRC);

	lastbus = pci_scan_child_bus(bridge->bus);
	pci_bus_update_busn_res_end(bridge->bus, lastbus);

	pci_assign_unassigned_bus_resources(bridge->bus);

	pci_bus_add_devices(bridge->bus);

	return 0;

bridge_setup_fail:
	put_device(&bridge->dev);
	device_unregister(&bridge->dev);
bridge_create_fail:
	kfree(pp);
	return err;
}

Best regards,
Liviu



Catalin Marinas (1):
  arm64: Extend the PCI I/O space to 16MB

Liviu Dudau (2):
  Fix ioport_map() for !CONFIG_GENERIC_IOMAP cases.
  arm64: Add architecture support for PCI

 Documentation/arm64/memory.txt |  16 ++--
 arch/arm64/Kconfig             |  19 ++++-
 arch/arm64/include/asm/Kbuild  |   1 +
 arch/arm64/include/asm/io.h    |   5 +-
 arch/arm64/include/asm/pci.h   |  49 ++++++++++++
 arch/arm64/kernel/Makefile     |   1 +
 arch/arm64/kernel/pci.c        | 173 +++++++++++++++++++++++++++++++++++++++++
 include/asm-generic/io.h       |   2 +-
 8 files changed, 256 insertions(+), 10 deletions(-)
 create mode 100644 arch/arm64/include/asm/pci.h
 create mode 100644 arch/arm64/kernel/pci.c

-- 
1.9.0

[PATCH v6 1/3] Fix ioport_map() for !CONFIG_GENERIC_IOMAP cases.

[Liviu Dudau]

The inline version of ioport_map() that gets used when !CONFIG_GENERIC_IOMAP
is wrong. It returns a mapped (i.e. virtual) address that can start from
zero and completely ignores the PCI_IOBASE and IO_SPACE_LIMIT that most
architectures that use !CONFIG_GENERIC_MAP define.

Signed-off-by: Liviu Dudau <Liviu.Dudau@arm.com>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Tested-by: Tanmay Inamdar <tinamdar@apm.com>
---
 include/asm-generic/io.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/asm-generic/io.h b/include/asm-generic/io.h
index d5afe96..df72051 100644
--- a/include/asm-generic/io.h
+++ b/include/asm-generic/io.h
@@ -331,7 +331,7 @@ static inline void iounmap(void __iomem *addr)
 #ifndef CONFIG_GENERIC_IOMAP
 static inline void __iomem *ioport_map(unsigned long port, unsigned int nr)
 {
-	return (void __iomem *) port;
+	return (void __iomem *)(PCI_IOBASE + (port & IO_SPACE_LIMIT));
 }
 
 static inline void ioport_unmap(void __iomem *p)
-- 
1.9.0

[PATCH v6 2/3] arm64: Extend the PCI I/O space to 16MB

[Liviu Dudau]

From: Catalin Marinas <catalin.marinas@arm.com>

The patch moves the PCI I/O space (currently at 64K) before the
earlyprintk mapping and extends it to 16MB.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
 Documentation/arm64/memory.txt | 16 ++++++++++------
 arch/arm64/include/asm/io.h    |  2 +-
 2 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
index 5e054bf..85e24c4 100644
--- a/Documentation/arm64/memory.txt
+++ b/Documentation/arm64/memory.txt
@@ -35,11 +35,13 @@ ffffffbc00000000	ffffffbdffffffff	   8GB		vmemmap
 
 ffffffbe00000000	ffffffbffbbfffff	  ~8GB		[guard, future vmmemap]
 
-ffffffbffbc00000	ffffffbffbdfffff	   2MB		earlyprintk device
+ffffffbffa000000	ffffffbffaffffff	  16MB		PCI I/O space
+
+ffffffbffb000000	ffffffbffbbfffff	  12MB		[guard]
 
-ffffffbffbe00000	ffffffbffbe0ffff	  64KB		PCI I/O space
+ffffffbffbc00000	ffffffbffbdfffff	   2MB		earlyprintk device
 
-ffffffbffbe10000	ffffffbcffffffff	  ~2MB		[guard]
+ffffffbffbe00000	ffffffbffbffffff	   2MB		[guard]
 
 ffffffbffc000000	ffffffbfffffffff	  64MB		modules
 
@@ -60,11 +62,13 @@ fffffdfc00000000	fffffdfdffffffff	   8GB		vmemmap
 
 fffffdfe00000000	fffffdfffbbfffff	  ~8GB		[guard, future vmmemap]
 
-fffffdfffbc00000	fffffdfffbdfffff	   2MB		earlyprintk device
+fffffdfffa000000	fffffdfffaffffff	  16MB		PCI I/O space
+
+fffffdfffb000000	fffffdfffbbfffff	  12MB		[guard]
 
-fffffdfffbe00000	fffffdfffbe0ffff	  64KB		PCI I/O space
+fffffdfffbc00000	fffffdfffbdfffff	   2MB		earlyprintk device
 
-fffffdfffbe10000	fffffdfffbffffff	  ~2MB		[guard]
+fffffdfffbe00000	fffffdfffbffffff	   2MB		[guard]
 
 fffffdfffc000000	fffffdffffffffff	  64MB		modules
 
diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h
index 4cc813e..7846a6b 100644
--- a/arch/arm64/include/asm/io.h
+++ b/arch/arm64/include/asm/io.h
@@ -121,7 +121,7 @@ static inline u64 __raw_readq(const volatile void __iomem *addr)
  *  I/O port access primitives.
  */
 #define IO_SPACE_LIMIT		0xffff
-#define PCI_IOBASE		((void __iomem *)(MODULES_VADDR - SZ_2M))
+#define PCI_IOBASE		((void __iomem *)(MODULES_VADDR - SZ_32M))
 
 static inline u8 inb(unsigned long addr)
 {
-- 
1.9.0

[PATCH v6 3/3] arm64: Add architecture support for PCI

[Liviu Dudau]

Use the generic host bridge functions to provide support for
PCI Express on arm64. There is no support for ISA memory.

Signed-off-by: Liviu Dudau <Liviu.Dudau@arm.com>
Tested-by: Tanmay Inamdar <tinamdar@apm.com>
---
 arch/arm64/Kconfig            |  19 +++-
 arch/arm64/include/asm/Kbuild |   1 +
 arch/arm64/include/asm/io.h   |   3 +-
 arch/arm64/include/asm/pci.h  |  49 +++++++++
 arch/arm64/kernel/Makefile    |   1 +
 arch/arm64/kernel/pci.c       | 173 ++++++++++++++++++++++++++++++++
 6 files changed, 244 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm64/include/asm/pci.h
 create mode 100644 arch/arm64/kernel/pci.c

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 27bbcfc..d1c8568 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -62,7 +62,7 @@ config MMU
 	def_bool y
 
 config NO_IOPORT
-	def_bool y
+	def_bool y if !PCI
 
 config STACKTRACE_SUPPORT
 	def_bool y
@@ -134,6 +134,23 @@ menu "Bus support"
 config ARM_AMBA
 	bool
 
+config PCI
+	bool "PCI support"
+	help
+	  This feature enables support for PCIe bus system. If you say Y
+	  here, the kernel will include drivers and infrastructure code
+	  to support PCIe bus devices.
+
+config PCI_DOMAINS
+	def_bool PCI
+
+config PCI_SYSCALL
+	def_bool PCI
+
+source "drivers/pci/Kconfig"
+source "drivers/pci/pcie/Kconfig"
+source "drivers/pci/hotplug/Kconfig"
+
 endmenu
 
 menu "Kernel Features"
diff --git a/arch/arm64/include/asm/Kbuild b/arch/arm64/include/asm/Kbuild
index 71c53ec..46924bc 100644
--- a/arch/arm64/include/asm/Kbuild
+++ b/arch/arm64/include/asm/Kbuild
@@ -26,6 +26,7 @@ generic-y += mman.h
 generic-y += msgbuf.h
 generic-y += mutex.h
 generic-y += pci.h
+generic-y += pci-bridge.h
 generic-y += poll.h
 generic-y += posix_types.h
 generic-y += resource.h
diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h
index 7846a6b..67463a5 100644
--- a/arch/arm64/include/asm/io.h
+++ b/arch/arm64/include/asm/io.h
@@ -120,7 +120,8 @@ static inline u64 __raw_readq(const volatile void __iomem *addr)
 /*
  *  I/O port access primitives.
  */
-#define IO_SPACE_LIMIT		0xffff
+#define arch_has_dev_port()	(1)
+#define IO_SPACE_LIMIT		0x1ffffff
 #define PCI_IOBASE		((void __iomem *)(MODULES_VADDR - SZ_32M))
 
 static inline u8 inb(unsigned long addr)
diff --git a/arch/arm64/include/asm/pci.h b/arch/arm64/include/asm/pci.h
new file mode 100644
index 0000000..6f1fb3d
--- /dev/null
+++ b/arch/arm64/include/asm/pci.h
@@ -0,0 +1,49 @@
+#ifndef __ASM_PCI_H
+#define __ASM_PCI_H
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/dma-mapping.h>
+
+#include <asm/io.h>
+#include <asm-generic/pci-bridge.h>
+#include <asm-generic/pci-dma-compat.h>
+
+#define PCIBIOS_MIN_IO		0x1000
+#define PCIBIOS_MIN_MEM		0
+
+struct pci_host_bridge *find_pci_host_bridge(struct pci_bus *bus);
+
+/*
+ * Set to 1 if the kernel should re-assign all PCI bus numbers
+ */
+#define pcibios_assign_all_busses() \
+	(pci_has_flag(PCI_REASSIGN_ALL_BUS))
+
+/*
+ * PCI address space differs from physical memory address space
+ */
+#define PCI_DMA_BUS_IS_PHYS	(0)
+
+extern int isa_dma_bridge_buggy;
+
+static inline int pci_domain_nr(struct pci_bus *bus)
+{
+	struct pci_host_bridge *bridge = find_pci_host_bridge(bus);
+
+	if (bridge)
+		return bridge->domain_nr;
+
+	return 0;
+}
+
+static inline int pci_proc_domain(struct pci_bus *bus)
+{
+	return 1;
+}
+
+extern unsigned long pci_ioremap_io(const struct resource *res, phys_addr_t phys_addr);
+
+#endif  /* __KERNEL__ */
+#endif  /* __ASM_PCI_H */
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 2d4554b..64fc479 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -20,6 +20,7 @@ arm64-obj-$(CONFIG_HAVE_HW_BREAKPOINT)+= hw_breakpoint.o
 arm64-obj-$(CONFIG_EARLY_PRINTK)	+= early_printk.o
 arm64-obj-$(CONFIG_ARM64_CPU_SUSPEND)	+= sleep.o suspend.o
 arm64-obj-$(CONFIG_JUMP_LABEL)		+= jump_label.o
+arm64-obj-$(CONFIG_PCI)			+= pci.o
 
 obj-y					+= $(arm64-obj-y) vdso/
 obj-m					+= $(arm64-obj-m)
diff --git a/arch/arm64/kernel/pci.c b/arch/arm64/kernel/pci.c
new file mode 100644
index 0000000..9f29c9a
--- /dev/null
+++ b/arch/arm64/kernel/pci.c
@@ -0,0 +1,173 @@
+/*
+ * Code borrowed from powerpc/kernel/pci-common.c
+ *
+ * Copyright (C) 2003 Anton Blanchard <anton@au.ibm.com>, IBM
+ * Copyright (C) 2014 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2 as published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/init.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/mm.h>
+#include <linux/of_pci.h>
+#include <linux/of_platform.h>
+#include <linux/slab.h>
+
+#include <asm/pci-bridge.h>
+
+struct ioresource {
+	struct list_head list;
+	phys_addr_t start;
+	resource_size_t size;
+};
+
+static LIST_HEAD(io_list);
+
+int pci_register_io_range(phys_addr_t address, resource_size_t size)
+{
+	struct ioresource *res;
+	resource_size_t allocated_size = 0;
+
+	/* find if the range has not been already allocated */
+	list_for_each_entry(res, &io_list, list) {
+		if (address >= res->start &&
+			address + size <= res->start + size)
+			return 0;
+		allocated_size += res->size;
+	}
+
+	/* range not already registered, check for space */
+	if (allocated_size + size > IO_SPACE_LIMIT)
+		return -E2BIG;
+
+	/* add the range in the list */
+	res = kzalloc(sizeof(*res), GFP_KERNEL);
+	if (!res)
+		return -ENOMEM;
+	res->start = address;
+	res->size = size;
+
+	list_add_tail(&res->list, &io_list);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(pci_register_io_range);
+
+unsigned long pci_address_to_pio(phys_addr_t address)
+{
+	struct ioresource *res;
+
+	list_for_each_entry(res, &io_list, list) {
+		if (address >= res->start &&
+			address < res->start + res->size) {
+			return res->start - address;
+		}
+	}
+
+	return (unsigned long)-1;
+}
+EXPORT_SYMBOL_GPL(pci_address_to_pio);
+
+/*
+ * Called after each bus is probed, but before its children are examined
+ */
+void pcibios_fixup_bus(struct pci_bus *bus)
+{
+	struct pci_dev *dev;
+	struct resource *res;
+	int i;
+
+	if (!pci_is_root_bus(bus)) {
+		pci_read_bridge_bases(bus);
+
+		pci_bus_for_each_resource(bus, res, i) {
+			if (!res || !res->flags || res->parent)
+				continue;
+
+			/*
+			 * If we are going to reassign everything, we can
+			 * shrink the P2P resource to have zero size to
+			 * save space
+			 */
+			if (pci_has_flag(PCI_REASSIGN_ALL_RSRC)) {
+				res->flags |= IORESOURCE_UNSET;
+				res->start = 0;
+				res->end = -1;
+				continue;
+			}
+		}
+	}
+
+	list_for_each_entry(dev, &bus->devices, bus_list) {
+		/* Ignore fully discovered devices */
+		if (dev->is_added)
+			continue;
+
+		set_dev_node(&dev->dev, pcibus_to_node(dev->bus));
+
+		/* Read default IRQs and fixup if necessary */
+		dev->irq = of_irq_parse_and_map_pci(dev, 0, 0);
+	}
+}
+EXPORT_SYMBOL(pcibios_fixup_bus);
+
+/*
+ * We don't have to worry about legacy ISA devices, so nothing to do here
+ */
+resource_size_t pcibios_align_resource(void *data, const struct resource *res,
+				resource_size_t size, resource_size_t align)
+{
+	return res->start;
+}
+
+int pcibios_enable_device(struct pci_dev *dev, int mask)
+{
+	return pci_enable_resources(dev, mask);
+}
+
+#define IO_SPACE_PAGES	((IO_SPACE_LIMIT + 1) / PAGE_SIZE)
+static DECLARE_BITMAP(pci_iospace, IO_SPACE_PAGES);
+
+unsigned long pci_ioremap_io(const struct resource *res, phys_addr_t phys_addr)
+{
+	unsigned long start, len, virt_start;
+	int err;
+
+	if (res->end > IO_SPACE_LIMIT)
+		return -EINVAL;
+
+	/*
+	 * try finding free space for the whole size first,
+	 * fall back to 64K if not available
+	 */
+	len = resource_size(res);
+	start = bitmap_find_next_zero_area(pci_iospace, IO_SPACE_PAGES,
+				res->start / PAGE_SIZE, len / PAGE_SIZE, 0);
+	if (start == IO_SPACE_PAGES && len > SZ_64K) {
+		len = SZ_64K;
+		start = 0;
+		start = bitmap_find_next_zero_area(pci_iospace, IO_SPACE_PAGES,
+					start, len / PAGE_SIZE, 0);
+	}
+
+	/* no 64K area found */
+	if (start == IO_SPACE_PAGES)
+		return -ENOMEM;
+
+	/* ioremap physical aperture to virtual aperture */
+	virt_start = start * PAGE_SIZE + (unsigned long)PCI_IOBASE;
+	err = ioremap_page_range(virt_start, virt_start + len,
+				phys_addr, __pgprot(PROT_DEVICE_nGnRE));
+	if (err)
+		return err;
+
+	bitmap_set(pci_iospace, start, len / PAGE_SIZE);
+
+	/* return io_offset */
+	return start * PAGE_SIZE - res->start;
+}
-- 
1.9.0

Re: [PATCH v6 1/3] Fix ioport_map() for !CONFIG_GENERIC_IOMAP cases.

[Russell King - ARM Linux]

On Wed, Mar 05, 2014 at 11:49:08AM +0000, Liviu Dudau wrote:
> The inline version of ioport_map() that gets used when !CONFIG_GENERIC_IOMAP
> is wrong. It returns a mapped (i.e. virtual) address that can start from
> zero and completely ignores the PCI_IOBASE and IO_SPACE_LIMIT that most
> architectures that use !CONFIG_GENERIC_MAP define.

What value does PCI_IOBASE and IO_SPACE_LIMIT have on other architectures
who make use of asm-generic/io.h ?

$ git grep asm-generic/io.h arch/
arch/arc/include/asm/io.h:#include <asm-generic/io.h>
arch/blackfin/include/asm/io.h:#include <asm-generic/io.h>
arch/metag/include/asm/io.h:#include <asm-generic/io.h>
arch/microblaze/include/asm/io.h:/* from asm-generic/io.h */
arch/openrisc/include/asm/io.h:#include <asm-generic/io.h>
arch/s390/include/asm/io.h:#include <asm-generic/io.h>
arch/score/include/asm/io.h:#include <asm-generic/io.h>
arch/unicore32/include/asm/io.h:#include <asm-generic/io.h>
arch/xtensa/include/asm/io.h:#include <asm-generic/io.h>
$ arch/arc/include/asm/io.h:#define PCI_IOBASE ((void __iomem *)0)
arch/arm64/include/asm/io.h:#define PCI_IOBASE          ((void __iomem *)(MODULES_VADDR - SZ_2M))
arch/arm64/include/asm/io.h:    return readb(addr + PCI_IOBASE);
arch/arm64/include/asm/io.h:    return readw(addr + PCI_IOBASE);
arch/arm64/include/asm/io.h:    return readl(addr + PCI_IOBASE);
arch/arm64/include/asm/io.h:    writeb(b, addr + PCI_IOBASE);
arch/arm64/include/asm/io.h:    writew(b, addr + PCI_IOBASE);
arch/arm64/include/asm/io.h:    writel(b, addr + PCI_IOBASE);
arch/arm64/include/asm/io.h:            *buf++ = __raw_readb(addr + PCI_IOBASE);arch/arm64/include/asm/io.h:            *buf++ = __raw_readw(addr + PCI_IOBASE);arch/arm64/include/asm/io.h:            *buf++ = __raw_readl(addr + PCI_IOBASE);arch/arm64/include/asm/io.h:            __raw_writeb(*buf++, addr + PCI_IOBASE);arch/arm64/include/asm/io.h:            __raw_writew(*buf++, addr + PCI_IOBASE);arch/arm64/include/asm/io.h:            __raw_writel(*buf++, addr + PCI_IOBASE);arch/unicore32/include/asm/io.h:#define PCI_IOBASE      PKUNITY_PCILIO_BASE
arch/unicore32/include/asm/io.h:#define PIO_OFFSET              (unsigned int)(PCI_IOBASE)


-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Re: [PATCH v6 1/3] Fix ioport_map() for !CONFIG_GENERIC_IOMAP cases.

[Liviu Dudau]

On Wed, Mar 05, 2014 at 11:31:35PM +0000, Russell King - ARM Linux wrote:
> On Wed, Mar 05, 2014 at 11:49:08AM +0000, Liviu Dudau wrote:
> > The inline version of ioport_map() that gets used when !CONFIG_GENERIC_IOMAP
> > is wrong. It returns a mapped (i.e. virtual) address that can start from
> > zero and completely ignores the PCI_IOBASE and IO_SPACE_LIMIT that most
> > architectures that use !CONFIG_GENERIC_MAP define.
> 
> What value does PCI_IOBASE and IO_SPACE_LIMIT have on other architectures
> who make use of asm-generic/io.h ?

Hi Russell,

Sorry for being a bit opaque in the commit message, I probably conflated two
issues into one. The first issue is that ioport_map() is supposed to return
a virtual address for the IO port. I struggle to believe that a virtual address
of zero for IO is valid for most architectures other than x86. My guess is that
most of the architectures that you have listed as including asm-generic/io.h
have no support for PCI whatsoever. The other issue is that *if* you specify an
PCI_IOBASE and don't come up with your own version of ioport_map() then you
get the wrong virtual addresses back.

One way to fix all this is to use PCI_IOBASE inside the generic version and to
define it to a non-zero values for architectures that have memory mapped IO.
That should reduce the number of custom versions of ioport_map() that we currently
have.

The other implied message that I'm getting is that you are suggesting that the
commit message is generalising a bit too much? With that I agree, in light of
your analysis. I can change it to something like:

   The inline version of ioport_map() that gets used for !CONFIG_GENERIC_IOMAP is
   wrong when PCI_IOBASE has a non-zero value. The function is supposed to return
   a virtual address for IO ports and for architectures that memory map the IO
   areas that is giving incorrect results as it ignores PCI_IOBASE. Fix this and
   also limit the port range to the IO_SPACE_LIMIT mask.

> 
> $ git grep asm-generic/io.h arch/
> arch/arc/include/asm/io.h:#include <asm-generic/io.h>
> arch/blackfin/include/asm/io.h:#include <asm-generic/io.h>
> arch/metag/include/asm/io.h:#include <asm-generic/io.h>
> arch/microblaze/include/asm/io.h:/* from asm-generic/io.h */
> arch/openrisc/include/asm/io.h:#include <asm-generic/io.h>
> arch/s390/include/asm/io.h:#include <asm-generic/io.h>
> arch/score/include/asm/io.h:#include <asm-generic/io.h>
> arch/unicore32/include/asm/io.h:#include <asm-generic/io.h>

PCI_IOBASE = PKUNITY_PCILIO_BASE = PKUNITY_PCI_BASE + 0x00030000 = 
    io_p2v(0x80000000) + 0x00030000

All other ones bar arm64 are either happy with PCI_IOBASE being zero (proxy for
"no PCI support") or define their own version of ioport_map().

Best regards,
Liviu

> arch/xtensa/include/asm/io.h:#include <asm-generic/io.h>
> $ arch/arc/include/asm/io.h:#define PCI_IOBASE ((void __iomem *)0)
> arch/arm64/include/asm/io.h:#define PCI_IOBASE          ((void __iomem *)(MODULES_VADDR - SZ_2M))
> arch/arm64/include/asm/io.h:    return readb(addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:    return readw(addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:    return readl(addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:    writeb(b, addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:    writew(b, addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:    writel(b, addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:            *buf++ = __raw_readb(addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:            *buf++ = __raw_readw(addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:            *buf++ = __raw_readl(addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:            __raw_writeb(*buf++, addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:            __raw_writew(*buf++, addr + PCI_IOBASE);
> arch/arm64/include/asm/io.h:            __raw_writel(*buf++, addr + PCI_IOBASE);
> arch/unicore32/include/asm/io.h:#define PCI_IOBASE      PKUNITY_PCILIO_BASE
> arch/unicore32/include/asm/io.h:#define PIO_OFFSET              (unsigned int)(PCI_IOBASE)
> 
> 
> -- 
> FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
> improving, and getting towards what was expected from it.
> 

-- 
====================
| I would like to |
| fix the world,  |
| but they're not |
| giving me the   |
 \ source code!  /
  ---------------
    ¯\_(ツ)_/¯

Re: [PATCH v6 1/3] Fix ioport_map() for !CONFIG_GENERIC_IOMAP cases.

[Arnd Bergmann]

On Thursday 06 March 2014, Russell King - ARM Linux wrote:
> On Wed, Mar 05, 2014 at 11:49:08AM +0000, Liviu Dudau wrote:
> > The inline version of ioport_map() that gets used when !CONFIG_GENERIC_IOMAP
> > is wrong. It returns a mapped (i.e. virtual) address that can start from
> > zero and completely ignores the PCI_IOBASE and IO_SPACE_LIMIT that most
> > architectures that use !CONFIG_GENERIC_MAP define.
> 
> What value does PCI_IOBASE and IO_SPACE_LIMIT have on other architectures
> who make use of asm-generic/io.h ?
> 
> $ git grep asm-generic/io.h arch/
> arch/arc/include/asm/io.h:#include <asm-generic/io.h>

PCI support hasn't been upstreamed.

> arch/blackfin/include/asm/io.h:#include <asm-generic/io.h>
> arch/metag/include/asm/io.h:#include <asm-generic/io.h>

No PCI support

> arch/microblaze/include/asm/io.h:/* from asm-generic/io.h */

PCI_IOBASE=0, IO_SPACE_LIMIT=0xffffffff, so no change.

> arch/openrisc/include/asm/io.h:#include <asm-generic/io.h>

No PCI support

> arch/s390/include/asm/io.h:#include <asm-generic/io.h>

s390 supports PCI but no I/O space

> arch/score/include/asm/io.h:#include <asm-generic/io.h>

No PCI support

> arch/unicore32/include/asm/io.h:#include <asm-generic/io.h>

unicore32 is broken currently, the patch fixes it.

> arch/xtensa/include/asm/io.h:#include <asm-generic/io.h>

PCI_IOBASE=0, IO_SPACE_LIMIT=0xffffffff, so no change.

For most of these, I assume we actually want to remove support
for inb/outb as they don't support I/O space accesses. The other
ones look correct to me.

	Arnd

Re: [PATCH v6 1/3] Fix ioport_map() for !CONFIG_GENERIC_IOMAP cases.

[Russell King - ARM Linux]

On Fri, Mar 07, 2014 at 01:37:38AM +0100, Arnd Bergmann wrote:
> On Thursday 06 March 2014, Russell King - ARM Linux wrote:
> > On Wed, Mar 05, 2014 at 11:49:08AM +0000, Liviu Dudau wrote:
> > > The inline version of ioport_map() that gets used when !CONFIG_GENERIC_IOMAP
> > > is wrong. It returns a mapped (i.e. virtual) address that can start from
> > > zero and completely ignores the PCI_IOBASE and IO_SPACE_LIMIT that most
> > > architectures that use !CONFIG_GENERIC_MAP define.
> > 
> > What value does PCI_IOBASE and IO_SPACE_LIMIT have on other architectures
> > who make use of asm-generic/io.h ?
> > 
> > $ git grep asm-generic/io.h arch/
> > arch/arc/include/asm/io.h:#include <asm-generic/io.h>
> 
> PCI support hasn't been upstreamed.
> 
> > arch/blackfin/include/asm/io.h:#include <asm-generic/io.h>
> > arch/metag/include/asm/io.h:#include <asm-generic/io.h>
> 
> No PCI support
> 
> > arch/microblaze/include/asm/io.h:/* from asm-generic/io.h */
> 
> PCI_IOBASE=0, IO_SPACE_LIMIT=0xffffffff, so no change.

Seems to define _IO_BASE not PCI_IOBASE.

> > arch/openrisc/include/asm/io.h:#include <asm-generic/io.h>
> 
> No PCI support
> 
> > arch/s390/include/asm/io.h:#include <asm-generic/io.h>
> 
> s390 supports PCI but no I/O space
> 
> > arch/score/include/asm/io.h:#include <asm-generic/io.h>
> 
> No PCI support
> 
> > arch/unicore32/include/asm/io.h:#include <asm-generic/io.h>
> 
> unicore32 is broken currently, the patch fixes it.
> 
> > arch/xtensa/include/asm/io.h:#include <asm-generic/io.h>
> 
> PCI_IOBASE=0, IO_SPACE_LIMIT=0xffffffff, so no change.

Doesn't appear to define PCI_IOBASE.

Maybe there's other patches required for these?

> For most of these, I assume we actually want to remove support
> for inb/outb as they don't support I/O space accesses. The other
> ones look correct to me.

Right, so:

#ifdef CONFIG_HAS_IOPORT
#ifndef CONFIG_GENERIC_IOMAP
static inline void __iomem *ioport_map(unsigned long port, unsigned int nr)
{
        return (void __iomem *) port;
}

changing that to include PCI_IOBASE in there will result in a build
failure if the C compiler sees that.  In other words, when HAS_IOPORT=y
and GENERIC_IOMAP=n.

HAS_IOPORT is set when HAS_IOMEM is also set and NO_IOPORT unset.

It looks to me like blackfin doesn't set NO_IOPORT nor NO_IOMEM, so
this would have HAS_IOPORT set, and from what I can see doesn't set
GENERIC_IOMAP.  So, this change probably breaks blackfin.

I haven't looked deeply at the others.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

Re: [PATCH v6 1/3] Fix ioport_map() for !CONFIG_GENERIC_IOMAP cases.

[Liviu Dudau]

On Fri, Mar 07, 2014 at 01:09:50AM +0000, Russell King - ARM Linux wrote:
> On Fri, Mar 07, 2014 at 01:37:38AM +0100, Arnd Bergmann wrote:
> > On Thursday 06 March 2014, Russell King - ARM Linux wrote:
> > > On Wed, Mar 05, 2014 at 11:49:08AM +0000, Liviu Dudau wrote:
> > > > The inline version of ioport_map() that gets used when !CONFIG_GENERIC_IOMAP
> > > > is wrong. It returns a mapped (i.e. virtual) address that can start from
> > > > zero and completely ignores the PCI_IOBASE and IO_SPACE_LIMIT that most
> > > > architectures that use !CONFIG_GENERIC_MAP define.
> > > 
> > > What value does PCI_IOBASE and IO_SPACE_LIMIT have on other architectures
> > > who make use of asm-generic/io.h ?
> > > 
> > > $ git grep asm-generic/io.h arch/
> > > arch/arc/include/asm/io.h:#include <asm-generic/io.h>
> > 
> > PCI support hasn't been upstreamed.
> > 
> > > arch/blackfin/include/asm/io.h:#include <asm-generic/io.h>
> > > arch/metag/include/asm/io.h:#include <asm-generic/io.h>
> > 
> > No PCI support
> > 
> > > arch/microblaze/include/asm/io.h:/* from asm-generic/io.h */
> > 
> > PCI_IOBASE=0, IO_SPACE_LIMIT=0xffffffff, so no change.
> 
> Seems to define _IO_BASE not PCI_IOBASE.
> 
> > > arch/openrisc/include/asm/io.h:#include <asm-generic/io.h>
> > 
> > No PCI support
> > 
> > > arch/s390/include/asm/io.h:#include <asm-generic/io.h>
> > 
> > s390 supports PCI but no I/O space
> > 
> > > arch/score/include/asm/io.h:#include <asm-generic/io.h>
> > 
> > No PCI support
> > 
> > > arch/unicore32/include/asm/io.h:#include <asm-generic/io.h>
> > 
> > unicore32 is broken currently, the patch fixes it.
> > 
> > > arch/xtensa/include/asm/io.h:#include <asm-generic/io.h>
> > 
> > PCI_IOBASE=0, IO_SPACE_LIMIT=0xffffffff, so no change.
> 
> Doesn't appear to define PCI_IOBASE.
> 
> Maybe there's other patches required for these?
> 
> > For most of these, I assume we actually want to remove support
> > for inb/outb as they don't support I/O space accesses. The other
> > ones look correct to me.
> 
> Right, so:
> 
> #ifdef CONFIG_HAS_IOPORT
> #ifndef CONFIG_GENERIC_IOMAP
> static inline void __iomem *ioport_map(unsigned long port, unsigned int nr)
> {
>         return (void __iomem *) port;
> }
> 
> changing that to include PCI_IOBASE in there will result in a build
> failure if the C compiler sees that.  In other words, when HAS_IOPORT=y
> and GENERIC_IOMAP=n.
> 
> HAS_IOPORT is set when HAS_IOMEM is also set and NO_IOPORT unset.
> 
> It looks to me like blackfin doesn't set NO_IOPORT nor NO_IOMEM, so
> this would have HAS_IOPORT set, and from what I can see doesn't set
> GENERIC_IOMAP.  So, this change probably breaks blackfin.
> 
> I haven't looked deeply at the others.

Maybe I'm missing something obvious, but line 118 in include/asm-generic/io.h has:

#ifndef PCI_IOBASE
#define PCI_IOBASE ((void __iomem *)0)
#endif

That doesn't seem to be guarded by any other #ifdef (other than the overall
__ASM_GENERIC_IO_H), so PCI_IOBASE is defined regardless of CONFIG_HAS_IOPORT and 
CONFIG_GENERIC_IOMAP values.

Best regards,
Liviu

> 
> -- 
> FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
> improving, and getting towards what was expected from it.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------
   .oooO
   (   )
    \ (  Oooo.
     \_) (   )
          ) /
         (_/

 One small step
   for me ...

Re: [PATCH v6 3/3] arm64: Add architecture support for PCI

[Jingoo Han]

On Wednesday, March 05, 2014 8:49 PM, Liviu Dudau wrote:
> 
> Use the generic host bridge functions to provide support for
> PCI Express on arm64. There is no support for ISA memory.
> 
> Signed-off-by: Liviu Dudau <Liviu.Dudau@arm.com>
> Tested-by: Tanmay Inamdar <tinamdar@apm.com>
> ---
>  arch/arm64/Kconfig            |  19 +++-
>  arch/arm64/include/asm/Kbuild |   1 +
>  arch/arm64/include/asm/io.h   |   3 +-
>  arch/arm64/include/asm/pci.h  |  49 +++++++++
>  arch/arm64/kernel/Makefile    |   1 +
>  arch/arm64/kernel/pci.c       | 173 ++++++++++++++++++++++++++++++++
>  6 files changed, 244 insertions(+), 2 deletions(-)
>  create mode 100644 arch/arm64/include/asm/pci.h
>  create mode 100644 arch/arm64/kernel/pci.c

[.....]

> --- /dev/null
> +++ b/arch/arm64/include/asm/pci.h

[.....]

> +
> +static inline int pci_domain_nr(struct pci_bus *bus)
> +{
> +	struct pci_host_bridge *bridge = find_pci_host_bridge(bus);
> +
> +	if (bridge)
> +		return bridge->domain_nr;
> +
> +	return 0;
> +}

Hi Liviu Dudau,

When CONFIG_PCI=n, the following build errors happen. :-(
Would you confirm this?

In file included from include/linux/pci.h:1393:0,
                 from drivers/scsi/scsi_lib.c:19:
arch/arm64/include/asm/pci.h:31:19: error: redefinition of 'pci_domain_nr'
 static inline int pci_domain_nr(struct pci_bus *bus)
                   ^
In file included from drivers/scsi/scsi_lib.c:19:0:
include/linux/pci.h:1383:19: note: previous definition of 'pci_domain_nr' was here
 static inline int pci_domain_nr(struct pci_bus *bus) { return 0; }
                   ^
             .....

Best regards,
Jingoo Han

Re: [PATCH v6 3/3] arm64: Add architecture support for PCI

[Liviu Dudau]

On Wed, Mar 12, 2014 at 05:41:33PM +0900, Jingoo Han wrote:
> On Wednesday, March 05, 2014 8:49 PM, Liviu Dudau wrote:
> > 
> > Use the generic host bridge functions to provide support for
> > PCI Express on arm64. There is no support for ISA memory.
> > 
> > Signed-off-by: Liviu Dudau <Liviu.Dudau@arm.com>
> > Tested-by: Tanmay Inamdar <tinamdar@apm.com>
> > ---
> >  arch/arm64/Kconfig            |  19 +++-
> >  arch/arm64/include/asm/Kbuild |   1 +
> >  arch/arm64/include/asm/io.h   |   3 +-
> >  arch/arm64/include/asm/pci.h  |  49 +++++++++
> >  arch/arm64/kernel/Makefile    |   1 +
> >  arch/arm64/kernel/pci.c       | 173 ++++++++++++++++++++++++++++++++
> >  6 files changed, 244 insertions(+), 2 deletions(-)
> >  create mode 100644 arch/arm64/include/asm/pci.h
> >  create mode 100644 arch/arm64/kernel/pci.c
> 
> [.....]
> 
> > --- /dev/null
> > +++ b/arch/arm64/include/asm/pci.h
> 
> [.....]
> 
> > +
> > +static inline int pci_domain_nr(struct pci_bus *bus)
> > +{
> > +	struct pci_host_bridge *bridge = find_pci_host_bridge(bus);
> > +
> > +	if (bridge)
> > +		return bridge->domain_nr;
> > +
> > +	return 0;
> > +}
> 
> Hi Liviu Dudau,
> 
> When CONFIG_PCI=n, the following build errors happen. :-(
> Would you confirm this?
> 
> In file included from include/linux/pci.h:1393:0,
>                  from drivers/scsi/scsi_lib.c:19:
> arch/arm64/include/asm/pci.h:31:19: error: redefinition of 'pci_domain_nr'
>  static inline int pci_domain_nr(struct pci_bus *bus)
>                    ^
> In file included from drivers/scsi/scsi_lib.c:19:0:
> include/linux/pci.h:1383:19: note: previous definition of 'pci_domain_nr' was here
>  static inline int pci_domain_nr(struct pci_bus *bus) { return 0; }
>                    ^
>              .....

Hi Jingoo,

I confirm the build error. Sorry for missing out on this test. I've now got out
of some more pressing tasks and I will post v7 tomorrow.

Best regards,
Liviu

> 
> Best regards,
> Jingoo Han
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------
   .oooO
   (   )
    \ (  Oooo.
     \_) (   )
          ) /
         (_/

 One small step
   for me ...

Re: [PATCH v6 1/3] Fix ioport_map() for !CONFIG_GENERIC_IOMAP cases.

[Arnd Bergmann]

On Friday 07 March 2014, Russell King - ARM Linux wrote:
> On Fri, Mar 07, 2014 at 01:37:38AM +0100, Arnd Bergmann wrote:
> > On Thursday 06 March 2014, Russell King - ARM Linux wrote:

> > > arch/microblaze/include/asm/io.h:/* from asm-generic/io.h */
> > 
> > PCI_IOBASE=0, IO_SPACE_LIMIT=0xffffffff, so no change.
> 
> Seems to define _IO_BASE not PCI_IOBASE.

It gets the implicit PCI_IOBASE from asm-generic/io.h at the moment.

> > 
> > PCI_IOBASE=0, IO_SPACE_LIMIT=0xffffffff, so no change.
> 
> Doesn't appear to define PCI_IOBASE.

Same here.

> > For most of these, I assume we actually want to remove support
> > for inb/outb as they don't support I/O space accesses. The other
> > ones look correct to me.
> 
> Right, so:
> 
> #ifdef CONFIG_HAS_IOPORT
> #ifndef CONFIG_GENERIC_IOMAP
> static inline void __iomem *ioport_map(unsigned long port, unsigned int nr)
> {
>         return (void __iomem *) port;
> }
> 
> changing that to include PCI_IOBASE in there will result in a build
> failure if the C compiler sees that.  In other words, when HAS_IOPORT=y
> and GENERIC_IOMAP=n.
> 
> HAS_IOPORT is set when HAS_IOMEM is also set and NO_IOPORT unset.
> 
> It looks to me like blackfin doesn't set NO_IOPORT nor NO_IOMEM, so
> this would have HAS_IOPORT set, and from what I can see doesn't set
> GENERIC_IOMAP.  So, this change probably breaks blackfin.

I also see the same thing that Liviu mentioned, that PCI_IOBASE=0
is always provided by asm-generic/io.h if not set otherwise.

On a related topic, Uwe Kleine-König has submitted a patch to
rename CONFIG_HAS_IOPORT to CONFIG_HAS_IOPORT_MAP to clarify what
it does, and to allow us to add a new CONFIG_HAS_IOPORT option
that will let us remove all the I/O port handling code for
architectures that don't have any I/O access method.

	Arnd