[PATCH 00/12] MIPS: BRIDGE Updates

[PATCH 00/12] MIPS: BRIDGE Updates

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

This series performs a number of clean-ups on the code for the BRIDGE
ASIC found in several SGI platforms.  It must be applied after the
Xtalk clean-up patch series is applied.

Notable changes include:
 - Genericize the IP27 BRIDGE driver so future platforms can use it
 - Prepare the generic BRIDGE driver to become a platform_driver
 - Clean-up/replace IRIX-era structures and macros in bridge.h
 - Fix the recent PCI_PROBE_ONLY change to work for IP27 again.

Joshua Kinard (12):
  MIPS: BRIDGE: Rename pci-ip27.c to pci-bridge.c
  MIPS: IP27: Add pcibr.h header for IP27
  MIPS: PCI: Minor clean-ups to pci-legacy.c
  MIPS: PCI: Add BRIDGE 'pre_enable' hook
  MIPS: BRIDGE: Clean-up bridge.h header file
  MIPS: BRIDGE: Overhaul bridge.h header file
  MIPS: BRIDGE: Add XBRIDGE revs and SWAP bit
  MIPS: BRIDGE: Update ops-bridge.c
  MIPS: BRIDGE: Use !pci_is_root_bus(bus) to check for bus->number > 0
  MIPS: BRIDGE: Overhaul pci-bridge.c driver
  MIPS: IP27: Update IRQ code to work with the new BRIDGE code
  MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

 arch/mips/include/asm/mach-ip27/pcibr.h |   50 +
 arch/mips/include/asm/pci.h             |    3 +
 arch/mips/include/asm/pci/bridge.h      | 1353 ++++++++++++---------
 arch/mips/pci/Makefile                  |    2 +-
 arch/mips/pci/ops-bridge.c              |  129 +-
 arch/mips/pci/pci-bridge.c              |  321 +++++
 arch/mips/pci/pci-ip27.c                |  230 ----
 arch/mips/pci/pci-legacy.c              |   74 +-
 arch/mips/sgi-ip27/ip27-irq-pci.c       |   72 +-
 arch/mips/sgi-ip27/ip27-irq.c           |    5 +
 10 files changed, 1313 insertions(+), 926 deletions(-)
 create mode 100644 arch/mips/include/asm/mach-ip27/pcibr.h
 create mode 100644 arch/mips/pci/pci-bridge.c
 delete mode 100644 arch/mips/pci/pci-ip27.c

-- 
2.11.1

[PATCH 01/12] MIPS: BRIDGE: Rename pci-ip27.c to pci-bridge.c

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Rename arch/mips/pci/pci-ip27.c to arch/mips/pci/pci-bridge.c so that
it can become a generic driver for all BRIDGE/XBRIDGE-based systems
and update the Makefile for this change.

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
---
 arch/mips/pci/Makefile                     | 2 +-
 arch/mips/pci/{pci-ip27.c => pci-bridge.c} | 0
 2 files changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile
index 4b821481dd44..7ff66cce2cf7 100644
--- a/arch/mips/pci/Makefile
+++ b/arch/mips/pci/Makefile
@@ -37,7 +37,7 @@ obj-$(CONFIG_MIPS_MALTA)	+= fixup-malta.o pci-malta.o
 obj-$(CONFIG_PMC_MSP7120_GW)	+= fixup-pmcmsp.o ops-pmcmsp.o
 obj-$(CONFIG_PMC_MSP7120_EVAL)	+= fixup-pmcmsp.o ops-pmcmsp.o
 obj-$(CONFIG_PMC_MSP7120_FPGA)	+= fixup-pmcmsp.o ops-pmcmsp.o
-obj-$(CONFIG_SGI_IP27)		+= ops-bridge.o pci-ip27.o
+obj-$(CONFIG_SGI_IP27)		+= ops-bridge.o pci-bridge.o
 obj-$(CONFIG_SGI_IP32)		+= fixup-ip32.o ops-mace.o pci-ip32.o
 obj-$(CONFIG_SIBYTE_SB1250)	+= fixup-sb1250.o pci-sb1250.o
 obj-$(CONFIG_SIBYTE_BCM112X)	+= fixup-sb1250.o pci-sb1250.o
diff --git a/arch/mips/pci/pci-ip27.c b/arch/mips/pci/pci-bridge.c
similarity index 100%
rename from arch/mips/pci/pci-ip27.c
rename to arch/mips/pci/pci-bridge.c
-- 
2.11.1

[PATCH 02/12] MIPS: IP27: Add pcibr.h header for IP27

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Add a new header 'pcibr.h' to arch/mips/include/asm/mach-ip27 which
contains IP27-specific definitions for the BRIDGE ASIC used to connect
PCI devices to the Crosstalk bus.

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
---
 arch/mips/include/asm/mach-ip27/pcibr.h | 50 +++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/arch/mips/include/asm/mach-ip27/pcibr.h b/arch/mips/include/asm/mach-ip27/pcibr.h
new file mode 100644
index 000000000000..6664843c30f9
--- /dev/null
+++ b/arch/mips/include/asm/mach-ip27/pcibr.h
@@ -0,0 +1,50 @@
+/*
+ * Definitions for PCI bridges in IP27.  Derived from pcibr.h in the
+ * IP30 port.
+ *
+ * Copyright (C) 2004-2007 Stanislaw Skowronek <skylark@unaligned.org>
+ * Copyright (C) 2015-2016 Joshua Kinard <kumba@gentoo.org>
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License.  See the file "COPYING" in the main directory of this archive
+ * for more details.
+ */
+
+#ifndef __ASM_MACH_IP27_PCIBR_H
+#define __ASM_MACH_IP27_PCIBR_H
+
+#include <asm/pci/bridge.h>
+#include <asm/mach-ip27/irq.h>
+
+/* Xtalk */
+#define PCIBR_OFFSET_MEM	0x200000
+#define PCIBR_OFFSET_IO		0xa00000
+#define PCIBR_OFFSET_END	0xc00000
+
+/*
+ * This is how XIO sees HUB's PI_INT_PEND_MOD register.
+ */
+#define PCIBR_XIO_SEES_HUB	0x01800090
+
+/*
+ * Max # of PCI buses we can handle; ie, max #PCI bridges.
+ */
+#define PCIBR_MAX_NUM_PCIBUS	40
+
+/*
+ * Max # of PCI devices (like scsi controllers) we handle on a bus.
+ */
+#define PCIBR_MAX_DEV_PCIBUS	8
+
+/*
+ * Used by ip27-bridge.c and ip27-irq.c.
+ */
+#define PCIBR_MAX_BUS_X_DEV	(PCIBR_MAX_NUM_PCIBUS * PCIBR_MAX_DEV_PCIBUS)
+extern struct bridge_controller *irq_to_bridge[PCIBR_MAX_BUS_X_DEV];
+extern u32 irq_to_slot[PCIBR_MAX_BUS_X_DEV];
+
+/* XXX: Temporary until IP27 "mega update". */
+extern int request_bridge_irq(struct bridge_controller *bc);
+
+#endif /* __ASM_MACH_IP27_PCIBR_H */
+
-- 
2.11.1

[PATCH 03/12] MIPS: PCI: Minor clean-ups to pci-legacy.c

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Apply several minor clean-ups to arch/mips/pci/pci-legacy.c:
 - Collapse one-line comments to use a single /* ... */
 - Move 2nd arg to pci_add_resource_offset up a line
 - Put a pr_info message all on one line (checkpatch)
 - Replace several printk's with pr_info/pr_warn/pr_err
 - Add whitespace around operators in for loop and conditional

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
---
 arch/mips/pci/pci-legacy.c | 48 ++++++++++++++++--------------------
 1 file changed, 21 insertions(+), 27 deletions(-)

diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c
index 014649be158d..98f36ed8f7da 100644
--- a/arch/mips/pci/pci-legacy.c
+++ b/arch/mips/pci/pci-legacy.c
@@ -58,9 +58,7 @@ pcibios_align_resource(void *data, const struct resource *res,
 		if (start < PCIBIOS_MIN_IO + hose->io_resource->start)
 			start = PCIBIOS_MIN_IO + hose->io_resource->start;
 
-		/*
-		 * Put everything into 0x00-0xff region modulo 0x400
-		 */
+		/* Put everything into 0x00-0xff region modulo 0x400 */
 		if (start & 0x300)
 			start = (start + 0x3ff) & ~0x3ff;
 	} else if (res->flags & IORESOURCE_MEM) {
@@ -82,12 +80,12 @@ static void pcibios_scanbus(struct pci_controller *hose)
 	if (hose->get_busno && pci_has_flag(PCI_PROBE_ONLY))
 		next_busno = (*hose->get_busno)();
 
-	pci_add_resource_offset(&resources,
-				hose->mem_resource, hose->mem_offset);
-	pci_add_resource_offset(&resources,
-				hose->io_resource, hose->io_offset);
-	pci_add_resource_offset(&resources,
-				hose->busn_resource, hose->busn_offset);
+	pci_add_resource_offset(&resources, hose->mem_resource,
+				hose->mem_offset);
+	pci_add_resource_offset(&resources, hose->io_resource,
+				hose->io_offset);
+	pci_add_resource_offset(&resources, hose->busn_resource,
+				hose->busn_offset);
 	bus = pci_scan_root_bus(NULL, next_busno, hose->pci_ops, hose,
 				&resources);
 	hose->bus = bus;
@@ -139,18 +137,16 @@ void pci_load_of_ranges(struct pci_controller *hose, struct device_node *node)
 
 		switch (range.flags & IORESOURCE_TYPE_BITS) {
 		case IORESOURCE_IO:
-			pr_info("  IO 0x%016llx..0x%016llx\n",
-				range.cpu_addr,
-				range.cpu_addr + range.size - 1);
+			pr_info("  IO 0x%016llx..0x%016llx\n", range.cpu_addr,
+				(range.cpu_addr + range.size - 1));
 			hose->io_map_base =
 				(unsigned long)ioremap(range.cpu_addr,
 						       range.size);
 			res = hose->io_resource;
 			break;
 		case IORESOURCE_MEM:
-			pr_info(" MEM 0x%016llx..0x%016llx\n",
-				range.cpu_addr,
-				range.cpu_addr + range.size - 1);
+			pr_info(" MEM 0x%016llx..0x%016llx\n", range.cpu_addr,
+				(range.cpu_addr + range.size - 1));
 			res = hose->mem_resource;
 			break;
 		}
@@ -195,10 +191,8 @@ void register_pci_controller(struct pci_controller *hose)
 	/*
 	 * Do not panic here but later - this might happen before console init.
 	 */
-	if (!hose->io_map_base) {
-		printk(KERN_WARNING
-		       "registering PCI controller with io_map_base unset\n");
-	}
+	if (!hose->io_map_base)
+		pr_warn("registering PCI controller with io_map_base unset\n");
 
 	/*
 	 * Scan the bus if it is register after the PCI subsystem
@@ -213,8 +207,7 @@ void register_pci_controller(struct pci_controller *hose)
 	return;
 
 out:
-	printk(KERN_WARNING
-	       "Skipping PCI bus scan due to resource conflict\n");
+	pr_warn("Skipping PCI bus scan due to resource conflict\n");
 }
 
 static int __init pcibios_init(void)
@@ -242,9 +235,9 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask)
 
 	pci_read_config_word(dev, PCI_COMMAND, &cmd);
 	old_cmd = cmd;
-	for (idx=0; idx < PCI_NUM_RESOURCES; idx++) {
+	for (idx = 0; idx < PCI_NUM_RESOURCES; idx++) {
 		/* Only set up the requested stuff */
-		if (!(mask & (1<<idx)))
+		if (!(mask & (1 << idx)))
 			continue;
 
 		r = &dev->resource[idx];
@@ -254,8 +247,7 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask)
 				(!(r->flags & IORESOURCE_ROM_ENABLE)))
 			continue;
 		if (!r->start && r->end) {
-			printk(KERN_ERR "PCI: Device %s not available "
-			       "because of resource collisions\n",
+			pr_err("PCI: Device %s not available because of resource collisions\n",
 			       pci_name(dev));
 			return -EINVAL;
 		}
@@ -265,10 +257,11 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask)
 			cmd |= PCI_COMMAND_MEMORY;
 	}
 	if (cmd != old_cmd) {
-		printk("PCI: Enabling device %s (%04x -> %04x)\n",
-		       pci_name(dev), old_cmd, cmd);
+		pr_info("PCI: Enabling device %s (%04x -> %04x)\n",
+			pci_name(dev), old_cmd, cmd);
 		pci_write_config_word(dev, PCI_COMMAND, cmd);
 	}
+
 	return 0;
 }
 
@@ -298,5 +291,6 @@ char *__init pcibios_setup(char *str)
 {
 	if (pcibios_plat_setup)
 		return pcibios_plat_setup(str);
+
 	return str;
 }
-- 
2.11.1

[PATCH 04/12] MIPS: PCI: Add BRIDGE 'pre_enable' hook

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Add a hook for a 'pre_enable' function in struct pci_controller in
arch/mips/include/asm/pci.h and arch/mips/pci/pci-legacy.c.  This is
used to do any needed housekeeping on the BRIDGE/XBRIDGE ASICs prior
to PCI detection, such setting up DevIO windows or applying any known
BRIDGE WARs.  This change originates from the earliest days of the
IP30 patchset by Stanislaw Skowronek.

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
Cc: Stanislaw Skowronek <skylark@unaligned.org>
---
 arch/mips/include/asm/pci.h |  3 +++
 arch/mips/pci/pci-legacy.c  | 11 +++++++++++
 2 files changed, 14 insertions(+)

diff --git a/arch/mips/include/asm/pci.h b/arch/mips/include/asm/pci.h
index 30d1129d8624..0f1e381a79e3 100644
--- a/arch/mips/include/asm/pci.h
+++ b/arch/mips/include/asm/pci.h
@@ -48,6 +48,9 @@ struct pci_controller {
 	unsigned int need_domain_info;
 #endif
 
+	/* BRIDGE/XBRIDGE may need to config things before bringup */
+	int (*pre_enable)(struct pci_controller *, struct pci_dev *, int);
+
 	/* Optional access methods for reading/writing the bus number
 	   of the PCI controller */
 	int (*get_busno)(void);
diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c
index 98f36ed8f7da..68268bbb15b8 100644
--- a/arch/mips/pci/pci-legacy.c
+++ b/arch/mips/pci/pci-legacy.c
@@ -7,6 +7,7 @@
  * Copyright (C) 2003, 04, 11 Ralf Baechle (ralf@linux-mips.org)
  * Copyright (C) 2011 Wind River Systems,
  *   written by Ralf Baechle (ralf@linux-mips.org)
+ * Copyright (C) 2004-2007 Stanislaw Skowronek <skylark@unaligned.org>
  */
 #include <linux/bug.h>
 #include <linux/kernel.h>
@@ -232,6 +233,7 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask)
 	u16 cmd, old_cmd;
 	int idx;
 	struct resource *r;
+	struct pci_controller *hose = (struct pci_controller *)dev->sysdata;
 
 	pci_read_config_word(dev, PCI_COMMAND, &cmd);
 	old_cmd = cmd;
@@ -240,6 +242,15 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask)
 		if (!(mask & (1 << idx)))
 			continue;
 
+		/*
+		 * Some devices, like BRIDGE/XBRIDGE, may need to do a little
+		 * housekeeping prior to the generic code setting up the PCI
+		 * resources.
+		 */
+		if (hose->pre_enable)
+			if (hose->pre_enable(hose, dev, idx) < 0)
+				return -EINVAL;
+
 		r = &dev->resource[idx];
 		if (!(r->flags & (IORESOURCE_IO | IORESOURCE_MEM)))
 			continue;
-- 
2.11.1

[PATCH 05/12] MIPS: BRIDGE: Clean-up bridge.h header file

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Apply a number of clean-ups to arch/mips/include/asm/pci/bridge.h:
 - Replace (0x1 << x) shifts with BIT(x) or BIT_ULL
 - Replace (mask << x) shifts with GENMASK or GENMASK_ULL
 - Clean-up several multi-line macros for readability
 - Other whitespace and comment fixes as needed

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
---
 arch/mips/include/asm/pci/bridge.h | 520 ++++++++++++++-------------
 1 file changed, 280 insertions(+), 240 deletions(-)

diff --git a/arch/mips/include/asm/pci/bridge.h b/arch/mips/include/asm/pci/bridge.h
index 3206245d1ed6..248a390e89b3 100644
--- a/arch/mips/include/asm/pci/bridge.h
+++ b/arch/mips/include/asm/pci/bridge.h
@@ -388,79 +388,80 @@ typedef struct bridge_err_cmdword_s {
  *    Bridge register bit field definitions
  */
 
-/* Widget part number of bridge */
+/* Widget part numbers. */
 #define BRIDGE_WIDGET_PART_NUM		0xc002
 #define XBRIDGE_WIDGET_PART_NUM		0xd002
 
-/* Manufacturer of bridge */
+/* Widget manufacturer numbers. */
 #define BRIDGE_WIDGET_MFGR_NUM		0x036
 #define XBRIDGE_WIDGET_MFGR_NUM		0x024
 
-/* Revision numbers for known Bridge revisions */
+/* Widget revision numbers. */
 #define BRIDGE_REV_A			0x1
 #define BRIDGE_REV_B			0x2
 #define BRIDGE_REV_C			0x3
 #define BRIDGE_REV_D			0x4
 
 /* Bridge widget status register bits definition */
-
-#define BRIDGE_STAT_LLP_REC_CNT		(0xFFu << 24)
-#define BRIDGE_STAT_LLP_TX_CNT		(0xFF << 16)
-#define BRIDGE_STAT_FLASH_SELECT	(0x1 << 6)
-#define BRIDGE_STAT_PCI_GIO_N		(0x1 << 5)
-#define BRIDGE_STAT_PENDING		(0x1F << 0)
+#define BRIDGE_STAT_LLP_REC_CNT		GENMASK(31, 24)
+#define BRIDGE_STAT_LLP_TX_CNT		GENMASK(23, 16)
+#define BRIDGE_STAT_FLASH_SELECT	BIT(6)
+#define BRIDGE_STAT_PCI_GIO_N		BIT(5)
+#define BRIDGE_STAT_PENDING		GENMASK(4, 0)
 
 /* Bridge widget control register bits definition */
-#define BRIDGE_CTRL_FLASH_WR_EN		(0x1ul << 31)
-#define BRIDGE_CTRL_EN_CLK50		(0x1 << 30)
-#define BRIDGE_CTRL_EN_CLK40		(0x1 << 29)
-#define BRIDGE_CTRL_EN_CLK33		(0x1 << 28)
+#define BRIDGE_CTRL_FLASH_WR_EN		BIT(31)
+#define BRIDGE_CTRL_EN_CLK50		BIT(30)
+#define BRIDGE_CTRL_EN_CLK40		BIT(29)
+#define BRIDGE_CTRL_EN_CLK33		BIT(28)
 #define BRIDGE_CTRL_RST(n)		((n) << 24)
-#define BRIDGE_CTRL_RST_MASK		(BRIDGE_CTRL_RST(0xF))
-#define BRIDGE_CTRL_RST_PIN(x)		(BRIDGE_CTRL_RST(0x1 << (x)))
-#define BRIDGE_CTRL_IO_SWAP		(0x1 << 23)
-#define BRIDGE_CTRL_MEM_SWAP		(0x1 << 22)
-#define BRIDGE_CTRL_PAGE_SIZE		(0x1 << 21)
-#define BRIDGE_CTRL_SS_PAR_BAD		(0x1 << 20)
-#define BRIDGE_CTRL_SS_PAR_EN		(0x1 << 19)
+#define BRIDGE_CTRL_RST_MASK		GENMASK(27, 24)
+#define BRIDGE_CTRL_RST_PIN(x)		(BRIDGE_CTRL_RST(1UL << (x)))
+#define BRIDGE_CTRL_IO_SWAP		BIT(23)
+#define BRIDGE_CTRL_MEM_SWAP		BIT(22)
+#define BRIDGE_CTRL_PAGE_SIZE		BIT(21)
+#define BRIDGE_CTRL_SS_PAR_BAD		BIT(20)
+#define BRIDGE_CTRL_SS_PAR_EN		BIT(19)
 #define BRIDGE_CTRL_SSRAM_SIZE(n)	((n) << 17)
 #define BRIDGE_CTRL_SSRAM_SIZE_MASK	(BRIDGE_CTRL_SSRAM_SIZE(0x3))
 #define BRIDGE_CTRL_SSRAM_512K		(BRIDGE_CTRL_SSRAM_SIZE(0x3))
 #define BRIDGE_CTRL_SSRAM_128K		(BRIDGE_CTRL_SSRAM_SIZE(0x2))
 #define BRIDGE_CTRL_SSRAM_64K		(BRIDGE_CTRL_SSRAM_SIZE(0x1))
 #define BRIDGE_CTRL_SSRAM_1K		(BRIDGE_CTRL_SSRAM_SIZE(0x0))
-#define BRIDGE_CTRL_F_BAD_PKT		(0x1 << 16)
+#define BRIDGE_CTRL_F_BAD_PKT		BIT(16)
 #define BRIDGE_CTRL_LLP_XBAR_CRD(n)	((n) << 12)
 #define BRIDGE_CTRL_LLP_XBAR_CRD_MASK	(BRIDGE_CTRL_LLP_XBAR_CRD(0xf))
-#define BRIDGE_CTRL_CLR_RLLP_CNT	(0x1 << 11)
-#define BRIDGE_CTRL_CLR_TLLP_CNT	(0x1 << 10)
-#define BRIDGE_CTRL_SYS_END		(0x1 << 9)
+#define BRIDGE_CTRL_CLR_RLLP_CNT	BIT(11)
+#define BRIDGE_CTRL_CLR_TLLP_CNT	BIT(10)
+#define BRIDGE_CTRL_SYS_END		BIT(9)
 #define BRIDGE_CTRL_MAX_TRANS(n)	((n) << 4)
 #define BRIDGE_CTRL_MAX_TRANS_MASK	(BRIDGE_CTRL_MAX_TRANS(0x1f))
 #define BRIDGE_CTRL_WIDGET_ID(n)	((n) << 0)
 #define BRIDGE_CTRL_WIDGET_ID_MASK	(BRIDGE_CTRL_WIDGET_ID(0xf))
 
 /* Bridge Response buffer Error Upper Register bit fields definition */
-#define BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT (20)
-#define BRIDGE_RESP_ERRUPPR_DEVNUM_MASK (0x7 << BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT)
-#define BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT (16)
-#define BRIDGE_RESP_ERRUPPR_BUFNUM_MASK (0xF << BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT)
-#define BRIDGE_RESP_ERRRUPPR_BUFMASK	(0xFFFF)
-
-#define BRIDGE_RESP_ERRUPPR_BUFNUM(x)	\
-			(((x) & BRIDGE_RESP_ERRUPPR_BUFNUM_MASK) >> \
-				BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT)
-
-#define BRIDGE_RESP_ERRUPPR_DEVICE(x)	\
-			(((x) &	 BRIDGE_RESP_ERRUPPR_DEVNUM_MASK) >> \
-				 BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT)
+#define BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT	(20)
+#define BRIDGE_RESP_ERRUPPR_DEVNUM_MASK					\
+	(0x7 << BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT)
+#define BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT	(16)
+#define BRIDGE_RESP_ERRUPPR_BUFNUM_MASK					\
+	(0xf << BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT)
+#define BRIDGE_RESP_ERRRUPPR_BUFMASK	(0xffff)
+
+#define BRIDGE_RESP_ERRUPPR_BUFNUM(x)					\
+	(((x) & BRIDGE_RESP_ERRUPPR_BUFNUM_MASK) >>			\
+		BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT)
+
+#define BRIDGE_RESP_ERRUPPR_DEVICE(x)					\
+	(((x) &	 BRIDGE_RESP_ERRUPPR_DEVNUM_MASK) >>			\
+		 BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT)
 
 /* Bridge direct mapping register bits definition */
 #define BRIDGE_DIRMAP_W_ID_SHFT		20
-#define BRIDGE_DIRMAP_W_ID		(0xf << BRIDGE_DIRMAP_W_ID_SHFT)
-#define BRIDGE_DIRMAP_RMF_64		(0x1 << 18)
-#define BRIDGE_DIRMAP_ADD512		(0x1 << 17)
-#define BRIDGE_DIRMAP_OFF		(0x1ffff << 0)
+#define BRIDGE_DIRMAP_W_ID		GENMASK(23, BRIDGE_DIRMAP_W_ID_SHFT)
+#define BRIDGE_DIRMAP_RMF_64		BIT(18)
+#define BRIDGE_DIRMAP_ADD512		BIT(17)
+#define BRIDGE_DIRMAP_OFF		GENMASK(16, 0)
 #define BRIDGE_DIRMAP_OFF_ADDRSHFT	(31)	/* lsbit of DIRMAP_OFF is xtalk address bit 31 */
 
 /* Bridge Arbitration register bits definition */
@@ -468,85 +469,98 @@ typedef struct bridge_err_cmdword_s {
 #define BRIDGE_ARB_REQ_WAIT_TICK_MASK	BRIDGE_ARB_REQ_WAIT_TICK(0x3)
 #define BRIDGE_ARB_REQ_WAIT_EN(x)	((x) << 8)
 #define BRIDGE_ARB_REQ_WAIT_EN_MASK	BRIDGE_ARB_REQ_WAIT_EN(0xff)
-#define BRIDGE_ARB_FREEZE_GNT		(1 << 6)
-#define BRIDGE_ARB_HPRI_RING_B2		(1 << 5)
-#define BRIDGE_ARB_HPRI_RING_B1		(1 << 4)
-#define BRIDGE_ARB_HPRI_RING_B0		(1 << 3)
-#define BRIDGE_ARB_LPRI_RING_B2		(1 << 2)
-#define BRIDGE_ARB_LPRI_RING_B1		(1 << 1)
-#define BRIDGE_ARB_LPRI_RING_B0		(1 << 0)
+#define BRIDGE_ARB_FREEZE_GNT		BIT(6)
+#define BRIDGE_ARB_HPRI_RING_B2		BIT(5)
+#define BRIDGE_ARB_HPRI_RING_B1		BIT(4)
+#define BRIDGE_ARB_HPRI_RING_B0		BIT(3)
+#define BRIDGE_ARB_LPRI_RING_B2		BIT(2)
+#define BRIDGE_ARB_LPRI_RING_B1		BIT(1)
+#define BRIDGE_ARB_LPRI_RING_B0		BIT(0)
 
 /* Bridge Bus time-out register bits definition */
 #define BRIDGE_BUS_PCI_RETRY_HLD(x)	((x) << 16)
 #define BRIDGE_BUS_PCI_RETRY_HLD_MASK	BRIDGE_BUS_PCI_RETRY_HLD(0x1f)
-#define BRIDGE_BUS_GIO_TIMEOUT		(1 << 12)
+#define BRIDGE_BUS_GIO_TIMEOUT		BIT(12)
 #define BRIDGE_BUS_PCI_RETRY_CNT(x)	((x) << 0)
 #define BRIDGE_BUS_PCI_RETRY_MASK	BRIDGE_BUS_PCI_RETRY_CNT(0x3ff)
 
 /* Bridge interrupt status register bits definition */
-#define BRIDGE_ISR_MULTI_ERR		(0x1u << 31)
-#define BRIDGE_ISR_PMU_ESIZE_FAULT	(0x1 << 30)
-#define BRIDGE_ISR_UNEXP_RESP		(0x1 << 29)
-#define BRIDGE_ISR_BAD_XRESP_PKT	(0x1 << 28)
-#define BRIDGE_ISR_BAD_XREQ_PKT		(0x1 << 27)
-#define BRIDGE_ISR_RESP_XTLK_ERR	(0x1 << 26)
-#define BRIDGE_ISR_REQ_XTLK_ERR		(0x1 << 25)
-#define BRIDGE_ISR_INVLD_ADDR		(0x1 << 24)
-#define BRIDGE_ISR_UNSUPPORTED_XOP	(0x1 << 23)
-#define BRIDGE_ISR_XREQ_FIFO_OFLOW	(0x1 << 22)
-#define BRIDGE_ISR_LLP_REC_SNERR	(0x1 << 21)
-#define BRIDGE_ISR_LLP_REC_CBERR	(0x1 << 20)
-#define BRIDGE_ISR_LLP_RCTY		(0x1 << 19)
-#define BRIDGE_ISR_LLP_TX_RETRY		(0x1 << 18)
-#define BRIDGE_ISR_LLP_TCTY		(0x1 << 17)
-#define BRIDGE_ISR_SSRAM_PERR		(0x1 << 16)
-#define BRIDGE_ISR_PCI_ABORT		(0x1 << 15)
-#define BRIDGE_ISR_PCI_PARITY		(0x1 << 14)
-#define BRIDGE_ISR_PCI_SERR		(0x1 << 13)
-#define BRIDGE_ISR_PCI_PERR		(0x1 << 12)
-#define BRIDGE_ISR_PCI_MST_TIMEOUT	(0x1 << 11)
+#define BRIDGE_ISR_MULTI_ERR		BIT(31)
+#define BRIDGE_ISR_PMU_ESIZE_FAULT	BIT(30)
+#define BRIDGE_ISR_UNEXP_RESP		BIT(29)
+#define BRIDGE_ISR_BAD_XRESP_PKT	BIT(28)
+#define BRIDGE_ISR_BAD_XREQ_PKT		BIT(27)
+#define BRIDGE_ISR_RESP_XTLK_ERR	BIT(26)
+#define BRIDGE_ISR_REQ_XTLK_ERR		BIT(25)
+#define BRIDGE_ISR_INVLD_ADDR		BIT(24)
+#define BRIDGE_ISR_UNSUPPORTED_XOP	BIT(23)
+#define BRIDGE_ISR_XREQ_FIFO_OFLOW	BIT(22)
+#define BRIDGE_ISR_LLP_REC_SNERR	BIT(21)
+#define BRIDGE_ISR_LLP_REC_CBERR	BIT(20)
+#define BRIDGE_ISR_LLP_RCTY		BIT(19)
+#define BRIDGE_ISR_LLP_TX_RETRY		BIT(18)
+#define BRIDGE_ISR_LLP_TCTY		BIT(17)
+#define BRIDGE_ISR_SSRAM_PERR		BIT(16)
+#define BRIDGE_ISR_PCI_ABORT		BIT(15)
+#define BRIDGE_ISR_PCI_PARITY		BIT(14)
+#define BRIDGE_ISR_PCI_SERR		BIT(13)
+#define BRIDGE_ISR_PCI_PERR		BIT(12)
+#define BRIDGE_ISR_PCI_MST_TIMEOUT	BIT(11)
 #define BRIDGE_ISR_GIO_MST_TIMEOUT	BRIDGE_ISR_PCI_MST_TIMEOUT
-#define BRIDGE_ISR_PCI_RETRY_CNT	(0x1 << 10)
-#define BRIDGE_ISR_XREAD_REQ_TIMEOUT	(0x1 << 9)
-#define BRIDGE_ISR_GIO_B_ENBL_ERR	(0x1 << 8)
-#define BRIDGE_ISR_INT_MSK		(0xff << 0)
-#define BRIDGE_ISR_INT(x)		(0x1 << (x))
-
-#define BRIDGE_ISR_LINK_ERROR		\
-		(BRIDGE_ISR_LLP_REC_SNERR|BRIDGE_ISR_LLP_REC_CBERR|	\
-		 BRIDGE_ISR_LLP_RCTY|BRIDGE_ISR_LLP_TX_RETRY|		\
+#define BRIDGE_ISR_PCI_RETRY_CNT	BIT(10)
+#define BRIDGE_ISR_XREAD_REQ_TIMEOUT	BIT(9)
+#define BRIDGE_ISR_GIO_B_ENBL_ERR	BIT(8)
+#define BRIDGE_ISR_INT_MSK		GENMASK(7, 0)
+#define BRIDGE_ISR_INT(x)		BIT((x))
+
+#define BRIDGE_ISR_LINK_ERROR						\
+		(BRIDGE_ISR_LLP_REC_SNERR |				\
+		 BRIDGE_ISR_LLP_REC_CBERR |				\
+		 BRIDGE_ISR_LLP_RCTY |					\
+		 BRIDGE_ISR_LLP_TX_RETRY |				\
 		 BRIDGE_ISR_LLP_TCTY)
 
-#define BRIDGE_ISR_PCIBUS_PIOERR	\
-		(BRIDGE_ISR_PCI_MST_TIMEOUT|BRIDGE_ISR_PCI_ABORT)
+#define BRIDGE_ISR_PCIBUS_PIOERR					\
+		(BRIDGE_ISR_PCI_MST_TIMEOUT |				\
+		 BRIDGE_ISR_PCI_ABORT)
 
-#define BRIDGE_ISR_PCIBUS_ERROR		\
-		(BRIDGE_ISR_PCIBUS_PIOERR|BRIDGE_ISR_PCI_PERR|		\
-		 BRIDGE_ISR_PCI_SERR|BRIDGE_ISR_PCI_RETRY_CNT|		\
+#define BRIDGE_ISR_PCIBUS_ERROR						\
+		(BRIDGE_ISR_PCIBUS_PIOERR |				\
+		 BRIDGE_ISR_PCI_PERR |					\
+		 BRIDGE_ISR_PCI_SERR |					\
+		 BRIDGE_ISR_PCI_RETRY_CNT |				\
 		 BRIDGE_ISR_PCI_PARITY)
 
-#define BRIDGE_ISR_XTALK_ERROR		\
-		(BRIDGE_ISR_XREAD_REQ_TIMEOUT|BRIDGE_ISR_XREQ_FIFO_OFLOW|\
-		 BRIDGE_ISR_UNSUPPORTED_XOP|BRIDGE_ISR_INVLD_ADDR|	\
-		 BRIDGE_ISR_REQ_XTLK_ERR|BRIDGE_ISR_RESP_XTLK_ERR|	\
-		 BRIDGE_ISR_BAD_XREQ_PKT|BRIDGE_ISR_BAD_XRESP_PKT|	\
+#define BRIDGE_ISR_XTALK_ERROR						\
+		(BRIDGE_ISR_XREAD_REQ_TIMEOUT |				\
+		 BRIDGE_ISR_XREQ_FIFO_OFLOW |				\
+		 BRIDGE_ISR_UNSUPPORTED_XOP |				\
+		 BRIDGE_ISR_INVLD_ADDR |				\
+		 BRIDGE_ISR_REQ_XTLK_ERR |				\
+		 BRIDGE_ISR_RESP_XTLK_ERR |				\
+		 BRIDGE_ISR_BAD_XREQ_PKT |				\
+		 BRIDGE_ISR_BAD_XRESP_PKT |				\
 		 BRIDGE_ISR_UNEXP_RESP)
 
-#define BRIDGE_ISR_ERRORS		\
-		(BRIDGE_ISR_LINK_ERROR|BRIDGE_ISR_PCIBUS_ERROR|		\
-		 BRIDGE_ISR_XTALK_ERROR|BRIDGE_ISR_SSRAM_PERR|		\
+#define BRIDGE_ISR_ERRORS						\
+		(BRIDGE_ISR_LINK_ERROR |				\
+		 BRIDGE_ISR_PCIBUS_ERROR |				\
+		 BRIDGE_ISR_XTALK_ERROR |				\
+		 BRIDGE_ISR_SSRAM_PERR |				\
 		 BRIDGE_ISR_PMU_ESIZE_FAULT)
 
-/*
- * List of Errors which are fatal and kill the system
- */
-#define BRIDGE_ISR_ERROR_FATAL		\
-		((BRIDGE_ISR_XTALK_ERROR & ~BRIDGE_ISR_XREAD_REQ_TIMEOUT)|\
-		 BRIDGE_ISR_PCI_SERR|BRIDGE_ISR_PCI_PARITY )
+/* List of Errors which are fatal and kill the system */
+#define BRIDGE_ISR_ERROR_FATAL						\
+		((BRIDGE_ISR_XTALK_ERROR &				\
+		  ~BRIDGE_ISR_XREAD_REQ_TIMEOUT) |			\
+		 BRIDGE_ISR_PCI_SERR |					\
+		 BRIDGE_ISR_PCI_PARITY)
 
-#define BRIDGE_ISR_ERROR_DUMP		\
-		(BRIDGE_ISR_PCIBUS_ERROR|BRIDGE_ISR_PMU_ESIZE_FAULT|	\
-		 BRIDGE_ISR_XTALK_ERROR|BRIDGE_ISR_SSRAM_PERR)
+#define BRIDGE_ISR_ERROR_DUMP						\
+		(BRIDGE_ISR_PCIBUS_ERROR |				\
+		 BRIDGE_ISR_PMU_ESIZE_FAULT |				\
+		 BRIDGE_ISR_XTALK_ERROR |				\
+		 BRIDGE_ISR_SSRAM_PERR)
 
 /* Bridge interrupt enable register bits definition */
 #define BRIDGE_IMR_UNEXP_RESP		BRIDGE_ISR_UNEXP_RESP
@@ -577,60 +591,73 @@ typedef struct bridge_err_cmdword_s {
 #define BRIDGE_IMR_INT(x)		BRIDGE_ISR_INT(x)
 
 /* Bridge interrupt reset register bits definition */
-#define BRIDGE_IRR_MULTI_CLR		(0x1 << 6)
-#define BRIDGE_IRR_CRP_GRP_CLR		(0x1 << 5)
-#define BRIDGE_IRR_RESP_BUF_GRP_CLR	(0x1 << 4)
-#define BRIDGE_IRR_REQ_DSP_GRP_CLR	(0x1 << 3)
-#define BRIDGE_IRR_LLP_GRP_CLR		(0x1 << 2)
-#define BRIDGE_IRR_SSRAM_GRP_CLR	(0x1 << 1)
-#define BRIDGE_IRR_PCI_GRP_CLR		(0x1 << 0)
-#define BRIDGE_IRR_GIO_GRP_CLR		(0x1 << 0)
+#define BRIDGE_IRR_MULTI_CLR		BIT(6)
+#define BRIDGE_IRR_CRP_GRP_CLR		BIT(5)
+#define BRIDGE_IRR_RESP_BUF_GRP_CLR	BIT(4)
+#define BRIDGE_IRR_REQ_DSP_GRP_CLR	BIT(3)
+#define BRIDGE_IRR_LLP_GRP_CLR		BIT(2)
+#define BRIDGE_IRR_SSRAM_GRP_CLR	BIT(1)
+#define BRIDGE_IRR_PCI_GRP_CLR		BIT(0)
+#define BRIDGE_IRR_GIO_GRP_CLR		BRIDGE_IRR_PCI_GRP_CLR
 #define BRIDGE_IRR_ALL_CLR		0x7f
 
-#define BRIDGE_IRR_CRP_GRP		(BRIDGE_ISR_UNEXP_RESP | \
-					 BRIDGE_ISR_XREQ_FIFO_OFLOW)
-#define BRIDGE_IRR_RESP_BUF_GRP		(BRIDGE_ISR_BAD_XRESP_PKT | \
-					 BRIDGE_ISR_RESP_XTLK_ERR | \
-					 BRIDGE_ISR_XREAD_REQ_TIMEOUT)
-#define BRIDGE_IRR_REQ_DSP_GRP		(BRIDGE_ISR_UNSUPPORTED_XOP | \
-					 BRIDGE_ISR_BAD_XREQ_PKT | \
-					 BRIDGE_ISR_REQ_XTLK_ERR | \
-					 BRIDGE_ISR_INVLD_ADDR)
-#define BRIDGE_IRR_LLP_GRP		(BRIDGE_ISR_LLP_REC_SNERR | \
-					 BRIDGE_ISR_LLP_REC_CBERR | \
-					 BRIDGE_ISR_LLP_RCTY | \
-					 BRIDGE_ISR_LLP_TX_RETRY | \
-					 BRIDGE_ISR_LLP_TCTY)
-#define BRIDGE_IRR_SSRAM_GRP		(BRIDGE_ISR_SSRAM_PERR | \
-					 BRIDGE_ISR_PMU_ESIZE_FAULT)
-#define BRIDGE_IRR_PCI_GRP		(BRIDGE_ISR_PCI_ABORT | \
-					 BRIDGE_ISR_PCI_PARITY | \
-					 BRIDGE_ISR_PCI_SERR | \
-					 BRIDGE_ISR_PCI_PERR | \
-					 BRIDGE_ISR_PCI_MST_TIMEOUT | \
-					 BRIDGE_ISR_PCI_RETRY_CNT)
-
-#define BRIDGE_IRR_GIO_GRP		(BRIDGE_ISR_GIO_B_ENBL_ERR | \
-					 BRIDGE_ISR_GIO_MST_TIMEOUT)
+#define BRIDGE_IRR_CRP_GRP						\
+	(BRIDGE_ISR_UNEXP_RESP |					\
+	 BRIDGE_ISR_XREQ_FIFO_OFLOW)
+
+#define BRIDGE_IRR_RESP_BUF_GRP						\
+	(BRIDGE_ISR_BAD_XRESP_PKT |					\
+	 BRIDGE_ISR_RESP_XTLK_ERR |					\
+	 BRIDGE_ISR_XREAD_REQ_TIMEOUT)
+
+#define BRIDGE_IRR_REQ_DSP_GRP						\
+	(BRIDGE_ISR_UNSUPPORTED_XOP |					\
+	 BRIDGE_ISR_BAD_XREQ_PKT |					\
+	 BRIDGE_ISR_REQ_XTLK_ERR |					\
+	 BRIDGE_ISR_INVLD_ADDR)
+
+#define BRIDGE_IRR_LLP_GRP						\
+	(BRIDGE_ISR_LLP_REC_SNERR |					\
+	 BRIDGE_ISR_LLP_REC_CBERR |					\
+	 BRIDGE_ISR_LLP_RCTY |						\
+	 BRIDGE_ISR_LLP_TX_RETRY |					\
+	 BRIDGE_ISR_LLP_TCTY)
+
+#define BRIDGE_IRR_SSRAM_GRP						\
+	(BRIDGE_ISR_SSRAM_PERR |					\
+	 BRIDGE_ISR_PMU_ESIZE_FAULT)
+
+#define BRIDGE_IRR_PCI_GRP						\
+	(BRIDGE_ISR_PCI_ABORT |						\
+	 BRIDGE_ISR_PCI_PARITY |					\
+	 BRIDGE_ISR_PCI_SERR |						\
+	 BRIDGE_ISR_PCI_PERR |						\
+	 BRIDGE_ISR_PCI_MST_TIMEOUT |					\
+	 BRIDGE_ISR_PCI_RETRY_CNT)
+
+#define BRIDGE_IRR_GIO_GRP						\
+	(BRIDGE_ISR_GIO_B_ENBL_ERR |					\
+	 BRIDGE_ISR_GIO_MST_TIMEOUT)
 
 /* Bridge INT_DEV register bits definition */
-#define BRIDGE_INT_DEV_SHFT(n)		((n)*3)
+#define BRIDGE_INT_DEV_SHFT(n)		((n) * 3)
 #define BRIDGE_INT_DEV_MASK(n)		(0x7 << BRIDGE_INT_DEV_SHFT(n))
 #define BRIDGE_INT_DEV_SET(_dev, _line) (_dev << BRIDGE_INT_DEV_SHFT(_line))
 
 /* Bridge interrupt(x) register bits definition */
-#define BRIDGE_INT_ADDR_HOST		0x0003FF00
-#define BRIDGE_INT_ADDR_FLD		0x000000FF
+#define BRIDGE_INT_ADDR_HOST		GENMASK(17, 8)
+#define BRIDGE_INT_ADDR_FLD		GENMASK(7, 0)
 
-#define BRIDGE_TMO_PCI_RETRY_HLD_MASK	0x1f0000
-#define BRIDGE_TMO_GIO_TIMEOUT_MASK	0x001000
-#define BRIDGE_TMO_PCI_RETRY_CNT_MASK	0x0003ff
+/* Bridge timeout register bits definition */
+#define BRIDGE_TMO_PCI_RETRY_HLD_MASK	GENMASK(20, 16)
+#define BRIDGE_TMO_GIO_TIMEOUT_MASK	BIT(12)
+#define BRIDGE_TMO_PCI_RETRY_CNT_MASK	GENMASK(9, 0)
 
 #define BRIDGE_TMO_PCI_RETRY_CNT_MAX	0x3ff
 
 /*
  * The NASID should be shifted by this amount and stored into the
- * interrupt(x) register.
+ * interrupt(x) register. (IP27/IP35 only)
  */
 #define BRIDGE_INT_ADDR_NASID_SHFT	8
 
@@ -638,76 +665,82 @@ typedef struct bridge_err_cmdword_s {
  * The BRIDGE_INT_ADDR_DEST_IO bit should be set to send an interrupt to
  * memory.
  */
-#define BRIDGE_INT_ADDR_DEST_IO		(1 << 17)
+#define BRIDGE_INT_ADDR_DEST_IO		BIT(17)
 #define BRIDGE_INT_ADDR_DEST_MEM	0
-#define BRIDGE_INT_ADDR_MASK		(1 << 17)
+#define BRIDGE_INT_ADDR_MASK		BIT(17)
 
 /* Bridge device(x) register bits definition */
-#define BRIDGE_DEV_ERR_LOCK_EN		0x10000000
-#define BRIDGE_DEV_PAGE_CHK_DIS		0x08000000
-#define BRIDGE_DEV_FORCE_PCI_PAR	0x04000000
-#define BRIDGE_DEV_VIRTUAL_EN		0x02000000
-#define BRIDGE_DEV_PMU_WRGA_EN		0x01000000
-#define BRIDGE_DEV_DIR_WRGA_EN		0x00800000
-#define BRIDGE_DEV_DEV_SIZE		0x00400000
-#define BRIDGE_DEV_RT			0x00200000
-#define BRIDGE_DEV_SWAP_PMU		0x00100000
-#define BRIDGE_DEV_SWAP_DIR		0x00080000
-#define BRIDGE_DEV_PREF			0x00040000
-#define BRIDGE_DEV_PRECISE		0x00020000
-#define BRIDGE_DEV_COH			0x00010000
-#define BRIDGE_DEV_BARRIER		0x00008000
-#define BRIDGE_DEV_GBR			0x00004000
-#define BRIDGE_DEV_DEV_SWAP		0x00002000
-#define BRIDGE_DEV_DEV_IO_MEM		0x00001000
-#define BRIDGE_DEV_OFF_MASK		0x00000fff
+#define BRIDGE_DEV_ERR_LOCK_EN		BIT(28)
+#define BRIDGE_DEV_PAGE_CHK_DIS		BIT(27)
+#define BRIDGE_DEV_FORCE_PCI_PAR	BIT(26)
+#define BRIDGE_DEV_VIRTUAL_EN		BIT(25)
+#define BRIDGE_DEV_PMU_WRGA_EN		BIT(24)
+#define BRIDGE_DEV_DIR_WRGA_EN		BIT(23)
+#define BRIDGE_DEV_DEV_SIZE		BIT(22)
+#define BRIDGE_DEV_RT			BIT(21)
+#define BRIDGE_DEV_SWAP_PMU		BIT(20)
+#define BRIDGE_DEV_SWAP_DIR		BIT(19)
+#define BRIDGE_DEV_PREF			BIT(18)
+#define BRIDGE_DEV_PRECISE		BIT(17)
+#define BRIDGE_DEV_COH			BIT(16)
+#define BRIDGE_DEV_BARRIER		BIT(15)
+#define BRIDGE_DEV_GBR			BIT(14)
+#define BRIDGE_DEV_DEV_SWAP		BIT(13)
+#define BRIDGE_DEV_DEV_IO_MEM		BIT(12)
+#define BRIDGE_DEV_OFF_MASK		GENMASK(11, 0)
 #define BRIDGE_DEV_OFF_ADDR_SHFT	20
 
-#define BRIDGE_DEV_PMU_BITS		(BRIDGE_DEV_PMU_WRGA_EN		| \
-					 BRIDGE_DEV_SWAP_PMU)
-#define BRIDGE_DEV_D32_BITS		(BRIDGE_DEV_DIR_WRGA_EN		| \
-					 BRIDGE_DEV_SWAP_DIR		| \
-					 BRIDGE_DEV_PREF		| \
-					 BRIDGE_DEV_PRECISE		| \
-					 BRIDGE_DEV_COH			| \
-					 BRIDGE_DEV_BARRIER)
-#define BRIDGE_DEV_D64_BITS		(BRIDGE_DEV_DIR_WRGA_EN		| \
-					 BRIDGE_DEV_SWAP_DIR		| \
-					 BRIDGE_DEV_COH			| \
-					 BRIDGE_DEV_BARRIER)
+#define BRIDGE_DEV_PMU_BITS						\
+	(BRIDGE_DEV_PMU_WRGA_EN |					\
+	 BRIDGE_DEV_SWAP_PMU)
+
+#define BRIDGE_DEV_D32_BITS						\
+	(BRIDGE_DEV_DIR_WRGA_EN |					\
+	 BRIDGE_DEV_SWAP_DIR |						\
+	 BRIDGE_DEV_PREF |						\
+	 BRIDGE_DEV_PRECISE |						\
+	 BRIDGE_DEV_COH |						\
+	 BRIDGE_DEV_BARRIER)
+
+#define BRIDGE_DEV_D64_BITS						\
+	(BRIDGE_DEV_DIR_WRGA_EN |					\
+	 BRIDGE_DEV_SWAP_DIR |						\
+	 BRIDGE_DEV_COH |						\
+	 BRIDGE_DEV_BARRIER)
 
 /* Bridge Error Upper register bit field definition */
-#define BRIDGE_ERRUPPR_DEVMASTER	(0x1 << 20)	/* Device was master */
-#define BRIDGE_ERRUPPR_PCIVDEV		(0x1 << 19)	/* Virtual Req value */
+#define BRIDGE_ERRUPPR_DEVMASTER	BIT(20)	/* Device was master */
+#define BRIDGE_ERRUPPR_PCIVDEV		BIT(19)	/* Virtual Req value */
 #define BRIDGE_ERRUPPR_DEVNUM_SHFT	(16)
 #define BRIDGE_ERRUPPR_DEVNUM_MASK	(0x7 << BRIDGE_ERRUPPR_DEVNUM_SHFT)
-#define BRIDGE_ERRUPPR_DEVICE(err)	(((err) >> BRIDGE_ERRUPPR_DEVNUM_SHFT) & 0x7)
-#define BRIDGE_ERRUPPR_ADDRMASK		(0xFFFF)
+#define BRIDGE_ERRUPPR_DEVICE(err)					\
+	(((err) >> BRIDGE_ERRUPPR_DEVNUM_SHFT) & 0x7)
+#define BRIDGE_ERRUPPR_ADDRMASK		GENMASK(15, 0)
 
 /* Bridge interrupt mode register bits definition */
-#define BRIDGE_INTMODE_CLR_PKT_EN(x)	(0x1 << (x))
+#define BRIDGE_INTMODE_CLR_PKT_EN(x)	BIT((x))
 
-/* this should be written to the xbow's link_control(x) register */
-#define BRIDGE_CREDIT	3
+/* This should be written to the XBOW's link_control(x) register */
+#define BRIDGE_CREDIT			3
 
-/* RRB assignment register */
-#define BRIDGE_RRB_EN	0x8	/* after shifting down */
-#define BRIDGE_RRB_DEV	0x7	/* after shifting down */
-#define BRIDGE_RRB_VDEV 0x4	/* after shifting down */
-#define BRIDGE_RRB_PDEV 0x3	/* after shifting down */
+/* RRB assignment register -- value applies after shifting down */
+#define BRIDGE_RRB_EN			0x8
+#define BRIDGE_RRB_DEV			0x7
+#define BRIDGE_RRB_VDEV			0x4
+#define BRIDGE_RRB_PDEV			0x3
 
 /* RRB status register */
-#define BRIDGE_RRB_VALID(r)	(0x00010000<<(r))
-#define BRIDGE_RRB_INUSE(r)	(0x00000001<<(r))
+#define BRIDGE_RRB_VALID(r)		(0x00010000 << (r))
+#define BRIDGE_RRB_INUSE(r)		(0x00000001 << (r))
 
 /* RRB clear register */
-#define BRIDGE_RRB_CLEAR(r)	(0x00000001<<(r))
+#define BRIDGE_RRB_CLEAR(r)		(0x00000001 << (r))
 
-/* xbox system controller declarations */
-#define XBOX_BRIDGE_WID		8
-#define FLASH_PROM1_BASE	0xE00000 /* To read the xbox sysctlr status */
-#define XBOX_RPS_EXISTS		1 << 6	 /* RPS bit in status register */
-#define XBOX_RPS_FAIL		1 << 4	 /* RPS status bit in register */
+/* Xbox system controller declarations */
+#define XBOX_BRIDGE_WID			8
+#define FLASH_PROM1_BASE		0xe00000 /* Read Xbox sysctlr stat */
+#define XBOX_RPS_EXISTS			BIT(6)	 /* RPS bit in status reg */
+#define XBOX_RPS_FAIL			BIT(4)	 /* RPS status bit in reg */
 
 /* ========================================================================
  */
@@ -716,12 +749,12 @@ typedef struct bridge_err_cmdword_s {
  * refer to section 4.2.1 of Bridge Spec for xtalk to PCI/GIO PIO mappings
  */
 /* XTALK addresses that map into Bridge Bus addr space */
-#define BRIDGE_PIO32_XTALK_ALIAS_BASE	0x000040000000L
-#define BRIDGE_PIO32_XTALK_ALIAS_LIMIT	0x00007FFFFFFFL
-#define BRIDGE_PIO64_XTALK_ALIAS_BASE	0x000080000000L
-#define BRIDGE_PIO64_XTALK_ALIAS_LIMIT	0x0000BFFFFFFFL
-#define BRIDGE_PCIIO_XTALK_ALIAS_BASE	0x000100000000L
-#define BRIDGE_PCIIO_XTALK_ALIAS_LIMIT	0x0001FFFFFFFFL
+#define BRIDGE_PIO32_XTALK_ALIAS_BASE	0x000040000000UL
+#define BRIDGE_PIO32_XTALK_ALIAS_LIMIT	0x00007fffffffUL
+#define BRIDGE_PIO64_XTALK_ALIAS_BASE	0x000080000000UL
+#define BRIDGE_PIO64_XTALK_ALIAS_LIMIT	0x0000bfffffffUL
+#define BRIDGE_PCIIO_XTALK_ALIAS_BASE	0x000100000000UL
+#define BRIDGE_PCIIO_XTALK_ALIAS_LIMIT	0x0001ffffffffUL
 
 /* Ranges of PCI bus space that can be accessed via PIO from xtalk */
 #define BRIDGE_MIN_PIO_ADDR_MEM		0x00000000	/* 1G PCI memory space */
@@ -753,15 +786,17 @@ typedef struct bridge_err_cmdword_s {
 #define PCI32_MAPPED_BASE		BRIDGE_DMA_MAPPED_BASE
 #define PCI32_DIRECT_BASE		BRIDGE_DMA_DIRECT_BASE
 
-#define IS_PCI32_LOCAL(x)	((ulong_t)(x) < PCI32_MAPPED_BASE)
-#define IS_PCI32_MAPPED(x)	((ulong_t)(x) < PCI32_DIRECT_BASE && \
-					(ulong_t)(x) >= PCI32_MAPPED_BASE)
-#define IS_PCI32_DIRECT(x)	((ulong_t)(x) >= PCI32_MAPPED_BASE)
-#define IS_PCI64(x)		((ulong_t)(x) >= PCI64_BASE)
-
-/*
- * The GIO address space.
- */
+#define IS_PCI32_LOCAL(x)						\
+	((ulong_t)(x) < PCI32_MAPPED_BASE)
+#define IS_PCI32_MAPPED(x)						\
+	((ulong_t)(x) < PCI32_DIRECT_BASE && \
+	 (ulong_t)(x) >= PCI32_MAPPED_BASE)
+#define IS_PCI32_DIRECT(x)						\
+	((ulong_t)(x) >= PCI32_MAPPED_BASE)
+#define IS_PCI64(x)							\
+	((ulong_t)(x) >= PCI64_BASE)
+
+/* GIO address space. */
 /* Xtalk to GIO PIO */
 #define BRIDGE_GIO_MEM32_BASE		BRIDGE_PIO32_XTALK_ALIAS_BASE
 #define BRIDGE_GIO_MEM32_LIMIT		BRIDGE_PIO32_XTALK_ALIAS_LIMIT
@@ -772,29 +807,33 @@ typedef struct bridge_err_cmdword_s {
 #define GIO_MAPPED_BASE			BRIDGE_DMA_MAPPED_BASE
 #define GIO_DIRECT_BASE			BRIDGE_DMA_DIRECT_BASE
 
-#define IS_GIO_LOCAL(x)		((ulong_t)(x) < GIO_MAPPED_BASE)
-#define IS_GIO_MAPPED(x)	((ulong_t)(x) < GIO_DIRECT_BASE && \
-					(ulong_t)(x) >= GIO_MAPPED_BASE)
-#define IS_GIO_DIRECT(x)	((ulong_t)(x) >= GIO_MAPPED_BASE)
+#define IS_GIO_LOCAL(x)							\
+	((ulong_t)(x) < GIO_MAPPED_BASE)
+#define IS_GIO_MAPPED(x)						\
+	((ulong_t)(x) < GIO_DIRECT_BASE && \
+	 (ulong_t)(x) >= GIO_MAPPED_BASE)
+#define IS_GIO_DIRECT(x)						\
+	((ulong_t)(x) >= GIO_MAPPED_BASE)
+
 
-/* PCI to xtalk mapping */
+/* PCI-to-Xtalk mapping */
 
 /* given a DIR_OFF value and a pci/gio 32 bits direct address, determine
  * which xtalk address is accessed
  */
 #define BRIDGE_DIRECT_32_SEG_SIZE	BRIDGE_DMA_DIRECT_SIZE
-#define BRIDGE_DIRECT_32_TO_XTALK(dir_off,adr)		\
-	((dir_off) * BRIDGE_DIRECT_32_SEG_SIZE +	\
-		((adr) & (BRIDGE_DIRECT_32_SEG_SIZE - 1)) + PHYS_RAMBASE)
+#define BRIDGE_DIRECT_32_TO_XTALK(dir_off, adr)				\
+	((dir_off) * BRIDGE_DIRECT_32_SEG_SIZE +			\
+	 ((adr) & (BRIDGE_DIRECT_32_SEG_SIZE - 1)) + PHYS_RAMBASE)
 
 /* 64-bit address attribute masks */
-#define PCI64_ATTR_TARG_MASK	0xf000000000000000
+#define PCI64_ATTR_TARG_MASK	GENMASK_ULL(63, 60)
 #define PCI64_ATTR_TARG_SHFT	60
-#define PCI64_ATTR_PREF		0x0800000000000000
-#define PCI64_ATTR_PREC		0x0400000000000000
-#define PCI64_ATTR_VIRTUAL	0x0200000000000000
-#define PCI64_ATTR_BAR		0x0100000000000000
-#define PCI64_ATTR_RMF_MASK	0x00ff000000000000
+#define PCI64_ATTR_PREF		BIT_ULL(59)
+#define PCI64_ATTR_PREC		BIT_ULL(58)
+#define PCI64_ATTR_VIRTUAL	BIT_ULL(57)
+#define PCI64_ATTR_BAR		BIT_ULL(56)
+#define PCI64_ATTR_RMF_MASK	GENMASK_ULL(55, 48)
 #define PCI64_ATTR_RMF_SHFT	48
 
 #ifndef __ASSEMBLY__
@@ -815,33 +854,34 @@ typedef union ate_u {
 } ate_t;
 #endif /* !__ASSEMBLY__ */
 
-#define ATE_V		0x01
-#define ATE_CO		0x02
-#define ATE_PREC	0x04
-#define ATE_PREF	0x08
-#define ATE_BAR		0x10
+#define ATE_V			0x01
+#define ATE_CO			0x02
+#define ATE_PREC		0x04
+#define ATE_PREF		0x08
+#define ATE_BAR			0x10
 
 #define ATE_PFNSHIFT		12
 #define ATE_TIDSHIFT		8
 #define ATE_RMFSHIFT		48
 
-#define mkate(xaddr, xid, attr) ((xaddr) & 0x0000fffffffff000ULL) | \
-				((xid)<<ATE_TIDSHIFT) | \
-				(attr)
+#define mkate(xaddr, xid, attr)						\
+	(((xaddr) & 0x0000fffffffff000ULL) |				\
+	 ((xid) << ATE_TIDSHIFT) |					\
+	 (attr))
 
 #define BRIDGE_INTERNAL_ATES	128
 
 struct bridge_controller {
-	struct pci_controller	pc;
-	struct resource		mem;
-	struct resource		io;
-	struct resource		busn;
-	bridge_t		*base;
-	nasid_t			nasid;
-	unsigned int		widget_id;
-	unsigned int		irq_cpu;
-	u64			baddr;
-	unsigned int		pci_int[8];
+	struct pci_controller pc;
+	struct resource mem;
+	struct resource io;
+	struct resource busn;
+	bridge_t *base;
+	nasid_t nasid;
+	u32 widget_id;
+	u32 irq_cpu;
+	u64 baddr;
+	u32 pci_int[8];
 };
 
 #define BRIDGE_CONTROLLER(bus) \
-- 
2.11.1

[PATCH 06/12] MIPS: BRIDGE: Overhaul bridge.h header file

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Rewrite/replace the large bridge_t typedef volatile struct used to
access BRIDGE registers with a new struct that is comprised of several
smaller structs and unions.  Documentation for each struct, derived
from both the BRIDGE specs and XBRIDGE specs, is included in kernel-doc
format.

Register read and write macros are also included that simplify access
to BRIDGE/XBRIDGE registers:

  Old (given 'bridge_t *bridge;'):
    Read:  foo = bridge->b_int_enable;
    Write: bridge->b_int_enable = foo;

  New (given 'struct bridge_controller *bc;'):
    Read: foo = bridge_read_reg(bc, b_int_enable);
    Write: bridge_write_reg(foo, bc, b_int_enable);

Other changes in this overhaul patch include:
 - Whitespace/comment fixes
 - Update older comments with info from the BRIDGE/XBRIDGE specs
 - Remove old/unused macros that accessed various BRIDGE registers
 - Added 'alloc_irq' hook to struct bridge_controller (used by IP27)
 - Per IA64, reading the BRIDGE flush register is done in a while loop
 - 'bridge_root_dev' is defined here instead of in pci-bridge.c
 - A platform_data struct is defined for BRIDGE for later use by IP27

There are several temporary pieces of code left that are needed by the
current IP27 code to boot correctly under these changes.  They will
be removed in the future with the IP27 "mega update" patch series.

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
---
 arch/mips/include/asm/pci/bridge.h | 864 ++++++++++++++++-----------
 1 file changed, 511 insertions(+), 353 deletions(-)

diff --git a/arch/mips/include/asm/pci/bridge.h b/arch/mips/include/asm/pci/bridge.h
index 248a390e89b3..3c214feab772 100644
--- a/arch/mips/include/asm/pci/bridge.h
+++ b/arch/mips/include/asm/pci/bridge.h
@@ -8,385 +8,468 @@
  *
  * Copyright (C) 1996, 1999 Silcon Graphics, Inc.
  * Copyright (C) 1999 Ralf Baechle (ralf@gnu.org)
+ * Copyright (C) 2016 Joshua Kinard <kumba@gentoo.org>
  */
 #ifndef _ASM_PCI_BRIDGE_H
 #define _ASM_PCI_BRIDGE_H
 
 #include <linux/types.h>
 #include <linux/pci.h>
-#include <asm/xtalk/xwidget.h>		/* generic widget header */
+#include <linux/spinlock.h>
+
+#include <asm/xtalk/xwidget.h>		/* generic xtalk widget header */
+
+#ifdef CONFIG_SGI_IP27
 #include <asm/sn/types.h>
+#endif
 
-/* I/O page size */
+/* ----------------------------------------------------------------------- */
+/* Common BRIDGE/XBRIDGE defines. */
 
-#define IOPFNSHIFT		12	/* 4K per mapped page */
+/* XXX: XBRIDGE definitions/support is incomplete and still needs work! */
 
+/* I/O page size */
+#define IOPFNSHIFT		12	/* 4K per mapped page */
 #define IOPGSIZE		(1 << IOPFNSHIFT)
 #define IOPG(x)			((x) >> IOPFNSHIFT)
 #define IOPGOFF(x)		((x) & (IOPGSIZE-1))
 
-/* Bridge RAM sizes */
-
+/*
+ * ATE RAM is 1KB per entry.
+ * BRIDGE has 128KB of ATE.
+ * XBRIDGE has 1024KB of ATE.
+ */
 #define BRIDGE_ATE_RAM_SIZE	0x00000400	/* 1kB ATE RAM */
 
-#define BRIDGE_CONFIG_BASE	0x20000
-#define BRIDGE_CONFIG1_BASE	0x28000
-#define BRIDGE_CONFIG_END	0x30000
-#define BRIDGE_CONFIG_SLOT_SIZE 0x1000
-
+/* BRIDGE/XBRIDGE SSRAM comes in several sizes. */
 #define BRIDGE_SSRAM_512K	0x00080000	/* 512kB */
 #define BRIDGE_SSRAM_128K	0x00020000	/* 128kB */
 #define BRIDGE_SSRAM_64K	0x00010000	/* 64kB */
 #define BRIDGE_SSRAM_0K		0x00000000	/* 0kB */
 
-/* ========================================================================
- *    Bridge address map
+/*
+ * Up to 8 devices per BRIDGE/XBRIDGE
+ * XXX: PIC maxes out at 4 devices, but that can be added later.
  */
+#define BRIDGE_MAX_DEVS		8
+
+/* ----------------------------------------------------------------------- */
+
+
+/* ----------------------------------------------------------------------- */
+/* BRIDGE/XBRIDGE structures that map the address space. */
 
 #ifndef __ASSEMBLY__
 
 /*
- * All accesses to bridge hardware registers must be done
- * using 32-bit loads and stores.
+ * Similar to crosstalk registers, BRIDGE/XBRIDGE registers are 32-bits or
+ * less in size and are aligned to a 64-bit boundary.  Register data begins
+ * at bits 31:0 and can only be accessed by a Crosstalk double-word packet
+ * type.  As such, we can reuse 'union xbow_reg' from xwidget.h, even though
+ * there is no need to access BRIDGE/XBRIDGE registers in a 64-bit manner.
+ *
+ */
+#define bridge_reg		xbow_reg
+
+/**
+ * struct bridge_registers - BRIDGE/XBRIDGE hardware registers.
+ * @wid_cfg: struct xwidget_cmn_regs access to common widget config registers.
+ * @__pad0: 0x8 bytes of padding (u64 * 1).
+ * @dir_map: r/w, relocate a 2GB region for PCI-to-Xtalk transfers.
+ * @__pad1: 0x8 bytes of padding (u64 * 1).
+ * @ssram_perr: r/o, holds the address of an SSRAM parity error.
+ * @__pad2: 0x8 bytes of padding (u64 * 1).
+ * @arb_prio: r/w, sets priority/timeout timing for PCI/GIO arbitration.
+ * @__pad3: 0x8 bytes of padding (u64 * 1).
+ * @nic: r/w, access to the 1-Wire Number-In-a-Can device.
+ * @__pad4: 0x8 bytes of padding (u64 * 1).
+ * @pcigio_bus_timo: r/w, sets GIO timeout or PCI retry timeout/count.
+ * @pci_type1_cfg: r/w, selects secondary bus/dev during PCI type 1 accesses.
+ * @pcigio_err_upper: r/o, access to upper 32-bits of PCI/GIO error address.
+ * @pcigio_err_lower: r/o, access to lower 32-bits of PCI/GIO error address.
+ * @__pad5: 0x0020 bytes of padding (0x04 * 8).
+ * @int_status: r/o, BRIDGE/XBRIDGE interrupt status register.
+ * @int_enable: r/w, enables specific interrupts in 'int_status'.
+ * @int_reset_stat: w/o, clears all bits not tied to an int pin.
+ * @int_mode: r/w, sets the mode of the interrupt pins.
+ * @int_device: r/w, associates interrupt pins with devices.
+ * @int_host_err: r/w, describes which bit in the host ISR to reset on error.
+ * @int_addr: r/w, maps int pins to host addresses and int bits.
+ * @__pad6: 0x0090 bytes of padding (0x12 * 8).
+ * @device: r/w, generic per-device (DevIO) control register.
+ * @dev_w_req_buf: r/o, returns 0 after per-dev write buffer has been flushed.
+ * @dev_even_rrb: r/w, allocates read response buffers for slots 0, 2, 4, & 6.
+ * @dev_odd_rrb: r/w, allocates read response buffers for slots 1, 3, 5, & 7.
+ * @dev_rrb_status: r/o, holds current read response buffer status.
+ * @dev_rrb_clear: w/o, write to bits 15:0 to clear buffer after disable.
+ * @__pad7: 0xfc60 bytes of padding (0x1fac * 8).
+ *
+ * XXX: XBRIDGE may need a separate struct definition, as some things are
+ *	different, such as the number of ATEs is larger, and this alters the
+ *	address space layout.
  */
-typedef u32	bridgereg_t;
+struct bridge_registers {
+	/*  Widget config */
+	struct xwidget_cmn_regs wid_cfg;	/* 0x000000 */
+	u64 __pad0;				/*  +0x0078 */
+	/* PMU/Direct map */
+	union bridge_reg dir_map;		/*  +0x0080 */
+	u64 __pad1;				/*  +0x0088 */
+	/* SSRAM */
+	union bridge_reg ssram_perr;		/*  +0x0090 */
+	u64 __pad2;				/*  +0x0098 */
+	/* Arbitration */
+	union bridge_reg arb_prio;		/*  +0x00a0 */
+	u64 __pad3;				/*  +0x00a8 */
+	/* Number-In-a-Can */
+	union bridge_reg nic;			/*  +0x00b0 */
+	u64 __pad4;				/*  +0x00b8 */
+	/* PCI/GIO */
+	union bridge_reg pcigio_bus_timo;	/*  +0x00c0 */
+	union bridge_reg pci_type1_cfg;		/*  +0x00c8 */
+	union bridge_reg pcigio_err_upper;	/*  +0x00d0 */
+	union bridge_reg pcigio_err_lower;	/*  +0x00d8 */
+	u64 __pad5[0x0004];			/*  +0x00e0 + 0x20 */
+	/* Interrupts */
+	union bridge_reg int_status;		/*  +0x0100 */
+	union bridge_reg int_enable;		/*  +0x0108 */
+	union bridge_reg int_reset_stat;	/*  +0x0110 */
+	union bridge_reg int_mode;		/*  +0x0118 */
+	union bridge_reg int_device;		/*  +0x0120 */
+	union bridge_reg int_host_err;		/*  +0x0128 */
+	union bridge_reg int_addr[8];		/*  +0x0130 + 0x40 */
+	u64 __pad6[0x0012];			/*  +0x0170 + 0x90 */
+	/* Devices */
+	union bridge_reg device[8];		/*  +0x0200 + 0x40 */
+	union bridge_reg dev_w_req_buf[8];	/*  +0x0240 + 0x40 */
+	union bridge_reg dev_even_rrb;		/*  +0x0280 */
+	union bridge_reg dev_odd_rrb;		/*  +0x0288 */
+	union bridge_reg dev_rrb_status;	/*  +0x0290 */
+	union bridge_reg dev_rrb_clear;		/*  +0x0298 */
+	u64 __pad7[0x1fac];			/*  +0x02a0 + 0xfd60 */
+};
 
-typedef u64	bridge_ate_t;
+/*
+ * BRIDGE/XBRIDGE widget configuration registers mirror common Crosstalk
+ * widget registers up until 0x48 (xw_llp_ctrl).  BRIDGE/XBRIDGE-specific
+ * config registers begin at 0x50 and continue until 0x70.  These macros
+ * below are shortcut names that map to either the shared
+ * 'struct xwidget_cmn_regs' names or those unique to BRIDGE/XBRIDGE, via a
+ * 'union bridge_reg' type's ".l.data" member, and must be accessed off of a
+ * defined 'struct bridge_registers' variable.
+ */
+#define b_wid_id		wid_cfg.id.l.data		/* ro */
+#define b_wid_status		wid_cfg.status.l.data		/* ro */
+#define b_wid_err_upper		wid_cfg.err_upper.l.data	/* ro */
+#define b_wid_err_lower		wid_cfg.err_lower.l.data	/* ro */
+#define b_wid_ctrl		wid_cfg.wid_ctrl.l.data		/* rw */
+#define b_wid_pkt_timo		wid_cfg.pkt_timo.l.data		/* rw */
+#define b_wid_int_upper		wid_cfg.int_upper.l.data	/* rw */
+#define b_wid_int_lower		wid_cfg.int_lower.l.data	/* rw */
+#define b_wid_err_cmdword	wid_cfg.err_cmdword.l.data	/* ro */
+#define b_wid_llp_ctrl		wid_cfg.llp_ctrl.l.data		/* rw */
+#define b_wid_targ_flush	wid_cfg.stat_clr.l.data		/* ro */
+#define b_wid_aux_errcmd	wid_cfg.arb_reload.l.data	/* ro */
+#define b_wid_resp_buf_upper	wid_cfg.perf_ctr_a.l.data	/* ro */
+#define b_wid_resp_buf_lower	wid_cfg.perf_ctr_b.l.data	/* ro */
+#define b_wid_test_pinctrl	wid_cfg.nic.l.data		/* rw */
+#define b_dir_map		dir_map.l.data			/* rw */
+#define b_ssram_perr		ssram_perr.l.data		/* ro */
+#define b_arb_prio		arb_prio.l.data			/* rw */
+#define b_nic			nic.l.data			/* rw */
+#define b_pci_bus_timo		pcigio_bus_timo.l.data		/* rw */
+#define b_pci_type1_cfg		pci_type1_cfg.l.data		/* rw */
+#define b_pci_err_upper		pcigio_err_upper.l.data		/* ro */
+#define b_pci_err_lower		pcigio_err_lower.l.data		/* ro */
+#define b_int_status		int_status.l.data		/* ro */
+#define b_int_enable		int_enable.l.data		/* rw */
+#define b_int_reset_stat	int_reset_stat.l.data		/* wo */
+#define b_int_mode		int_mode.l.data			/* rw */
+#define b_int_device		int_device.l.data		/* rw */
+#define b_int_host_err		int_host_err.l.data		/* rw */
+#define b_int_addr(_x)		int_addr[(_x)].l.data		/* rw */
+#define b_device(_x)		device[(_x)].l.data		/* rw */
+#define b_dev_w_req_buf(_x)	dev_w_req_buf[(_x)].l.data	/* ro */
+#define b_dev_even_rrb		dev_even_rrb.l.data		/* rw */
+#define b_dev_odd_rrb		dev_odd_rrb.l.data		/* rw */
+#define b_dev_rrb_status	dev_rrb_status.l.data		/* ro */
+#define b_dev_rrb_clear		dev_rrb_clear.l.data		/* wo */
 
-/* pointers to bridge ATEs
- * are always "pointer to volatile"
+/*
+ * BRIDGE/XBRIDGE can support either a PCI Bus or a GIO Bus.  As of writing,
+ * there is no known SGI platform supported by Linux where we'll encounter an
+ * XIO->GIO setup via BRIDGE/XBRIDGE.  The below macros mirror several of the
+ * above 'b_pci_*' names for better code readability should such support ever
+ * be added.
  */
-typedef volatile bridge_ate_t  *bridge_ate_p;
+#define b_gio_bus_timo		b_pci_bus_timo			/* rw */
+#define b_gio_err_upper		b_pci_err_upper			/* ro */
+#define b_gio_err_lower		b_pci_err_lower			/* ro */
 
 /*
- * It is generally preferred that hardware registers on the bridge
- * are located from C code via this structure.
+ * Per include/asm-ia64/sn/pci/bridge.h in Linux-2.5.70:
+ *   In [X]bridge the internal ATE Ram is writen as double words only,
+ *   but due to internal design issues it is read back as single words.
+ *   i.e:
+ *     b_int_ate_ram[index].hi.rd << 32 | xb_int_ate_ram_lo[index].rd
  *
- * Generated from Bridge spec dated 04oct95
+ * It just so happens that 'union xbow_reg' fits this situation perfectly,
+ * and with a little macro magic, can be used to access BRIDGE/XBRIDGE ATE
+ * entries rather easily.  However, unlike the other macros, 'b_ate_read' and
+ * 'b_ate_write' are passed the struct variable name and index offset, instead
+ * of being accessed off of a defined struct variable.
  */
+#define bridge_ate		xbow_reg
+#define b_ate_read(_s, _x)	(_s.ate_ram[_x].l.data << 32 |		\
+				 _s.ate_ram_lo[_x].l.data)
+#define b_ate_write(_s, _x)	(_s.ate_ram[_x].q)
+
+/**
+ * struct bridge_pci_cfg_cmn - common fields of a PCI config space type.
+ * @dev_id: device ID.
+ * @vendor: vendor ID.
+ * @status: status register.
+ * @cmd: command register.
+ * @class_code: PCI class code.
+ * @rev_id: revision ID.
+ * @bist: built-in self-test.
+ * @hdr_type: header type.
+ * @latency_tmr: latency timer.
+ * @cache_ln_sz: cache line size.
+ */
+struct bridge_pci_cfg_cmn {
+	u16 dev_id;		/* 0x00 */
+	u16 vendor;		/* 0x02 */
+	u16 status;		/* 0x04 */
+	u16 cmd;		/* 0x06 */
+	u32 class_code_rev;	/* 0x08 */
+	u8 bist;		/* 0x0c */
+	u8 hdr_type;		/* 0x0d */
+	u8 latency_tmr;		/* 0x0e */
+	u8 cache_ln_sz;		/* 0x0f */
+};
 
-typedef volatile struct bridge_s {
-	/* Local Registers			       0x000000-0x00FFFF */
-
-	/* standard widget configuration	       0x000000-0x000057 */
-	widget_cfg_t	    b_widget;			/* 0x000000 */
-
-	/* helper fieldnames for accessing bridge widget */
-
-#define b_wid_id			b_widget.w_id
-#define b_wid_stat			b_widget.w_status
-#define b_wid_err_upper			b_widget.w_err_upper_addr
-#define b_wid_err_lower			b_widget.w_err_lower_addr
-#define b_wid_control			b_widget.w_control
-#define b_wid_req_timeout		b_widget.w_req_timeout
-#define b_wid_int_upper			b_widget.w_intdest_upper_addr
-#define b_wid_int_lower			b_widget.w_intdest_lower_addr
-#define b_wid_err_cmdword		b_widget.w_err_cmd_word
-#define b_wid_llp			b_widget.w_llp_cfg
-#define b_wid_tflush			b_widget.w_tflush
-
-	/* bridge-specific widget configuration 0x000058-0x00007F */
-	bridgereg_t	    _pad_000058;
-	bridgereg_t	    b_wid_aux_err;		/* 0x00005C */
-	bridgereg_t	    _pad_000060;
-	bridgereg_t	    b_wid_resp_upper;		/* 0x000064 */
-	bridgereg_t	    _pad_000068;
-	bridgereg_t	    b_wid_resp_lower;		/* 0x00006C */
-	bridgereg_t	    _pad_000070;
-	bridgereg_t	    b_wid_tst_pin_ctrl;		/* 0x000074 */
-	bridgereg_t	_pad_000078[2];
-
-	/* PMU & Map 0x000080-0x00008F */
-	bridgereg_t	_pad_000080;
-	bridgereg_t	b_dir_map;			/* 0x000084 */
-	bridgereg_t	_pad_000088[2];
-
-	/* SSRAM 0x000090-0x00009F */
-	bridgereg_t	_pad_000090;
-	bridgereg_t	b_ram_perr;			/* 0x000094 */
-	bridgereg_t	_pad_000098[2];
-
-	/* Arbitration 0x0000A0-0x0000AF */
-	bridgereg_t	_pad_0000A0;
-	bridgereg_t	b_arb;				/* 0x0000A4 */
-	bridgereg_t	_pad_0000A8[2];
-
-	/* Number In A Can 0x0000B0-0x0000BF */
-	bridgereg_t	_pad_0000B0;
-	bridgereg_t	b_nic;				/* 0x0000B4 */
-	bridgereg_t	_pad_0000B8[2];
-
-	/* PCI/GIO 0x0000C0-0x0000FF */
-	bridgereg_t	_pad_0000C0;
-	bridgereg_t	b_bus_timeout;			/* 0x0000C4 */
-#define b_pci_bus_timeout b_bus_timeout
-
-	bridgereg_t	_pad_0000C8;
-	bridgereg_t	b_pci_cfg;			/* 0x0000CC */
-	bridgereg_t	_pad_0000D0;
-	bridgereg_t	b_pci_err_upper;		/* 0x0000D4 */
-	bridgereg_t	_pad_0000D8;
-	bridgereg_t	b_pci_err_lower;		/* 0x0000DC */
-	bridgereg_t	_pad_0000E0[8];
-#define b_gio_err_lower b_pci_err_lower
-#define b_gio_err_upper b_pci_err_upper
-
-	/* Interrupt 0x000100-0x0001FF */
-	bridgereg_t	_pad_000100;
-	bridgereg_t	b_int_status;			/* 0x000104 */
-	bridgereg_t	_pad_000108;
-	bridgereg_t	b_int_enable;			/* 0x00010C */
-	bridgereg_t	_pad_000110;
-	bridgereg_t	b_int_rst_stat;			/* 0x000114 */
-	bridgereg_t	_pad_000118;
-	bridgereg_t	b_int_mode;			/* 0x00011C */
-	bridgereg_t	_pad_000120;
-	bridgereg_t	b_int_device;			/* 0x000124 */
-	bridgereg_t	_pad_000128;
-	bridgereg_t	b_int_host_err;			/* 0x00012C */
-
-	struct {
-		bridgereg_t	__pad;			/* 0x0001{30,,,68} */
-		bridgereg_t	addr;			/* 0x0001{34,,,6C} */
-	} b_int_addr[8];				/* 0x000130 */
-
-	bridgereg_t	_pad_000170[36];
-
-	/* Device 0x000200-0x0003FF */
-	struct {
-		bridgereg_t	__pad;			/* 0x0002{00,,,38} */
-		bridgereg_t	reg;			/* 0x0002{04,,,3C} */
-	} b_device[8];					/* 0x000200 */
 
+/**
+ * union bridge_pci_cfg_t0 - PCI configuration space type 0.
+ * @b: byte-array access to the start of the configuration space.
+ * @w: short/word-array access to the start of the configuration space.
+ * @l: dword-array access to the start of the configuration space.
+ * @q: qword-array access to the start of the configuration space.
+ * @func: nested multi-size union for specific func# access via PCI_FUNC().
+ * @fields.bar: PCI base address registers #0 to #5.
+ * @fields.cardbus_dis: cardbus card information structure.
+ * @fields.subsys_id: subsystem ID.
+ * @fields.subsys_vendor: subsystem vendor.
+ * @fields.x_rom_base: expansion ROM base address.
+ * @fields.rsvd: reserved fields (0x34, 0x38).
+ * @fields.max_lat: max latency.
+ * @fields.min_gnt: minimum grant.
+ * @fields.int_pin: interrupt pin.
+ * @fields.int_line: interrupt line.
+ * @fields.device_specific: device-specific config area (BRIDGE-specific?).
+ */
+union bridge_pci_cfg_t0 {
+	u8  b[0x1000 / 1];
+	u16 w[0x1000 / 2];
+	u32 l[0x1000 / 4];
+	u64 q[0x1000 / 8];
+	union {
+		u8  b[0x100 / 1];
+		u16 w[0x100 / 2];
+		u32 l[0x100 / 4];
+		u64 q[0x100 / 8];
+	} func[8];
 	struct {
-		bridgereg_t	__pad;			/* 0x0002{40,,,78} */
-		bridgereg_t	reg;			/* 0x0002{44,,,7C} */
-	} b_wr_req_buf[8];				/* 0x000240 */
+		struct bridge_pci_cfg_cmn common;	/* 0x00 */
+		u32 bar[6];				/* 0x10 */
+		u32 cardbus_cis;			/* 0x28 */
+		u16 subsys_id;				/* 0x2c */
+		u16 subsys_vendor;			/* 0x2e */
+		u32 x_rom_base;				/* 0x30 */
+		u32 rsvd[2];				/* 0x34 */
+		u8 max_lat;				/* 0x3c */
+		u8 min_gnt;				/* 0x3d */
+		u8 int_pin;				/* 0x3e */
+		u8 int_line;				/* 0x3f */
+		u32 device_specific[48];		/* 0x40 + 0xc0 */
+	} fields;
+};
 
+/**
+ * struct bridge_pci_cfg_t1 - PCI configuration space type 1.
+ * @b: byte-array access to the start of the configuration space.
+ * @w: short/word-array access to the start of the configuration space.
+ * @l: dword-array access to the start of the configuration space.
+ * @q: qword-array access to the start of the configuration space.
+ * @fields.bar: PCI base address registers #0 & #1.
+ * @fields.sec_lat_tmr: secondary latency timer.
+ * @fields.sub_bus_num: subordinate bus number.
+ * @fields.sec_bus_num: secondary bus number.
+ * @fields.pri_bus_num: primary bus number.
+ * @fields.sec_status: secondary status.
+ * @fields.io_limit: I/O limit lower 8-bits.
+ * @fields.io_base: I/O base lower 8-bits.
+ * @fields.mem_limit: memory limit lower 16-bits.
+ * @fields.mem_base: memory base lower 16-bits.
+ * @fields.pf_mem_limit: prefetchable memory limit lower 16-bits.
+ * @fields.pf_mem_base: prefetchable memory base lower 16-bits.
+ * @fields.pf_mem_base_upper: prefetchable memory base upper 32-bits.
+ * @fields.pf_mem_limit_upper: prefetchable memory limit upper 32-bits.
+ * @fields.io_limit_upper: I/O limit upper 16-bits.
+ * @fields.io_base_upper: I/O base upper 16-bits.
+ * @fields.rsvd: reserved field.
+ * @fields.x_rom_base: expansion ROM base address.
+ * @fields.bridge_ctrl: PCI bridge control.
+ * @fields.int_pin: interrupt pin.
+ * @fields.int_line: interrupt line.
+ * @fields.device_specific: device-specific config area (BRIDGE-specific?).
+ */
+union bridge_pci_cfg_t1 {
+	u8  b[0x1000 / 1];
+	u16 w[0x1000 / 2];
+	u32 l[0x1000 / 4];
+	u64 q[0x1000 / 8];
 	struct {
-		bridgereg_t	__pad;			/* 0x0002{80,,,88} */
-		bridgereg_t	reg;			/* 0x0002{84,,,8C} */
-	} b_rrb_map[2];					/* 0x000280 */
-#define b_even_resp	b_rrb_map[0].reg		/* 0x000284 */
-#define b_odd_resp	b_rrb_map[1].reg		/* 0x00028C */
-
-	bridgereg_t	_pad_000290;
-	bridgereg_t	b_resp_status;			/* 0x000294 */
-	bridgereg_t	_pad_000298;
-	bridgereg_t	b_resp_clear;			/* 0x00029C */
-
-	bridgereg_t	_pad_0002A0[24];
-
-	char		_pad_000300[0x10000 - 0x000300];
+		struct bridge_pci_cfg_cmn common;	/* 0x00 */
+		u32 bar[2];				/* 0x10 */
+		u8 sec_lat_tmr;				/* 0x18 */
+		u8 sub_bus_num;				/* 0x19 */
+		u8 sec_bus_num;				/* 0x1a */
+		u8 pri_bus_num;				/* 0x1b */
+		u16 sec_status;				/* 0x1c */
+		u8 io_limit;				/* 0x1e */
+		u8 io_base;				/* 0x1f */
+		u16 mem_limit;				/* 0x20 */
+		u16 mem_base;				/* 0x22 */
+		u16 pf_mem_limit;			/* 0x24 */
+		u16 pf_mem_base;			/* 0x26 */
+		u32 pf_mem_base_upper;			/* 0x28 */
+		u32 pf_mem_limit_upper;			/* 0x2c */
+		u16 io_limit_upper;			/* 0x30 */
+		u16 io_base_upper;			/* 0x32 */
+		u32 rsvd;				/* 0x34 */
+		u32 x_rom_base;				/* 0x38 */
+		u16 bridge_ctrl;			/* 0x3c */
+		u8 int_pin;				/* 0x3e */
+		u8 int_line;				/* 0x3f */
+		u32 device_specific[48];		/* 0x40 + 0xc0 */
+	} fields;
+};
 
-	/* Internal Address Translation Entry RAM 0x010000-0x0103FF */
-	union {
-		bridge_ate_t	wr;			/* write-only */
-		struct {
-			bridgereg_t	_p_pad;
-			bridgereg_t	rd;		/* read-only */
-		}			hi;
-	}			    b_int_ate_ram[128];
+/**
+ * union bridge_devio - multi-size access to BRIDGE/XBRIDGE DevIO windows.
+ * @b: u8 access.
+ * @w: u16 access.
+ * @l: u32 access.
+ * @q: u64 access.
+ */
+union bridge_devio {
+	u8  b[0x100000 / 1];
+	u16 w[0x100000 / 2];
+	u32 l[0x100000 / 4];
+	u64 q[0x100000 / 8];
+};
 
-	char	_pad_010400[0x11000 - 0x010400];
+/**
+ * union bridge_flash_prom - multi-size access to BRIDGE/XBRIDGE flash PROMs.
+ * @b: u8 access.
+ * @w: u16 access.
+ * @l: u32 access.
+ * @q: u64 access.
+ *
+ * Be careful when messing around with the flash PROM.
+ */
+union bridge_flash_prom {
+	u8  b[0x400000 / 1];	/* ro */
+	u16 w[0x400000 / 2];	/* rw */
+	u32 l[0x400000 / 4];	/* ro */
+	u64 q[0x400000 / 8];	/* ro */
+};
 
-	/* Internal Address Translation Entry RAM LOW 0x011000-0x0113FF */
-	struct {
-		bridgereg_t	_p_pad;
-		bridgereg_t	rd;		/* read-only */
-	} b_int_ate_ram_lo[128];
-
-	char	_pad_011400[0x20000 - 0x011400];
-
-	/* PCI Device Configuration Spaces 0x020000-0x027FFF */
-	union {				/* make all access sizes available. */
-		u8	c[0x1000 / 1];
-		u16	s[0x1000 / 2];
-		u32	l[0x1000 / 4];
-		u64	d[0x1000 / 8];
-		union {
-			u8	c[0x100 / 1];
-			u16	s[0x100 / 2];
-			u32	l[0x100 / 4];
-			u64	d[0x100 / 8];
-		} f[8];
-	} b_type0_cfg_dev[8];					/* 0x020000 */
-
-    /* PCI Type 1 Configuration Space 0x028000-0x028FFF */
-	union {				/* make all access sizes available. */
-		u8	c[0x1000 / 1];
-		u16	s[0x1000 / 2];
-		u32	l[0x1000 / 4];
-		u64	d[0x1000 / 8];
-	} b_type1_cfg;					/* 0x028000-0x029000 */
-
-	char	_pad_029000[0x007000];			/* 0x029000-0x030000 */
-
-	/* PCI Interrupt Acknowledge Cycle 0x030000 */
-	union {
-		u8	c[8 / 1];
-		u16	s[8 / 2];
-		u32	l[8 / 4];
-		u64	d[8 / 8];
-	} b_pci_iack;						/* 0x030000 */
-
-	u8	_pad_030007[0x04fff8];			/* 0x030008-0x07FFFF */
-
-	/* External Address Translation Entry RAM 0x080000-0x0FFFFF */
-	bridge_ate_t	b_ext_ate_ram[0x10000];
-
-	/* Reserved 0x100000-0x1FFFFF */
-	char	_pad_100000[0x200000-0x100000];
-
-	/* PCI/GIO Device Spaces 0x200000-0xBFFFFF */
-	union {				/* make all access sizes available. */
-		u8	c[0x100000 / 1];
-		u16	s[0x100000 / 2];
-		u32	l[0x100000 / 4];
-		u64	d[0x100000 / 8];
-	} b_devio_raw[10];				/* 0x200000 */
-
-	/* b_devio macro is a bit strange; it reflects the
-	 * fact that the Bridge ASIC provides 2M for the
-	 * first two DevIO windows and 1M for the other six.
-	 */
-#define b_devio(n)	b_devio_raw[((n)<2)?(n*2):(n+2)]
-
-	/* External Flash Proms 1,0 0xC00000-0xFFFFFF */
-	union {		/* make all access sizes available. */
-		u8	c[0x400000 / 1];	/* read-only */
-		u16	s[0x400000 / 2];	/* read-write */
-		u32	l[0x400000 / 4];	/* read-only */
-		u64	d[0x400000 / 8];	/* read-only */
-	} b_external_flash;			/* 0xC00000 */
-} bridge_t;
+/**
+ * struct bridge_widget - a BRIDGE widget on a Crosstalk bus.
+ * @regs: struct bridge_registers access to BRIDGE/XBRIDGE registers.
+ * @ate_ram: access to upper 32-bits of ATE RAM area.
+ * @__pad0: 0x000c00 bytes of padding (0x000180 * 8).
+ * @ate_ram_lo: access to lower 32-bits of ATE RAM area.
+ * @__pad1: 0x00ec00 bytes of padding (0x001d80 * 8).
+ * @pci_t0: access to type 0 PCI configuration space for each PCI slot.
+ * @pci_t1: access to type 1 PCI configuration space.
+ * @__pad2: 0x007000 bytes of padding (0x000e00 * 8).
+ * @pci_iack: PCI interrupt ack cycle field/register.
+ * @__pad3: 0x04fff8 bytes of padding (0x009fff * 8).
+ * @ext_ate_ram: access to any external ATE RAM (don't use; known HW bugs).
+ * @__pad4: 0x100000 bytes of padding (0x020000 * 8).
+ * @devio_raw: access to DevIO windows.  2MB windows to #0 & #1; else, 1MB.
+ * @ext_flash: access to flash PROM space.  BE CAREFUL!
+ */
+struct bridge_widget {
+	/* BRIDGE/XBRIDGE registers */
+	struct bridge_registers regs;		/* 0x000000 - 0x00ffff */
+	/* PMU Address Translation Entries, HIGH */
+	union bridge_ate ate_ram[128];		/* 0x010000 - 0x0103ff */
+	u64 __pad0[0x000180];			/* 0x010400 - 0x010fff */
+	/* PMU Address Translation Entries, LOW */
+	union bridge_ate ate_ram_lo[128];	/* 0x011000 - 0x0113ff */
+	u64 __pad1[0x001d80];			/* 0x011400 - 0x01ffff */
+	/* PCI configuration space type 0 */
+	union bridge_pci_cfg_t0 pci_t0[8];	/* 0x020000 - 0x027fff */
+	/* PCI configuration space type 1 */
+	union bridge_pci_cfg_t1 pci_t1;		/* 0x028000 - 0x028fff */
+	u64 __pad2[0x000e00];			/* 0x029000 - 0x02ffff */
+	/* PCI Interrupt Acknowledge Cycle */
+	union bridge_reg pci_iack;		/* 0x030000 - 0x030007 */
+	u64 __pad3[0x009fff];			/* 0x030008 - 0x07ffff */
+	/* External Address Translation Entry RAM */
+	u64 ext_ate_ram[0x10000];		/* 0x080000 - 0x0fffff */
+	u64 __pad4[0x020000];			/* 0x100000 - 0x1fffff */
+	/* PCI/GIO DevIO windows */
+	union bridge_devio devio_raw[10];	/* 0x200000 - 0xbfffff */
+	/* External flash PROMs 0 & 1 */
+	union bridge_flash_prom ext_flash;	/* 0xc00000 - 0xffffff */
+};
 
 /*
- * Field formats for Error Command Word and Auxiliary Error Command Word
- * of bridge.
+ * The b_devio() macro is a bit strange, because it reflects the fact that
+ * the BRIDGE/XBRIDGE ASIC provides 2MB for the first two DevIO windows and
+ * 1MB for the remaining six DevIO windows.
  */
-typedef struct bridge_err_cmdword_s {
-	union {
-		u32		cmd_word;
-		struct {
-			u32	didn:4,		/* Destination ID  */
-				sidn:4,		/* Source ID	   */
-				pactyp:4,	/* Packet type	   */
-				tnum:5,		/* Trans Number	   */
-				coh:1,		/* Coh Transaction */
-				ds:2,		/* Data size	   */
-				gbr:1,		/* GBR enable	   */
-				vbpm:1,		/* VBPM message	   */
-				error:1,	/* Error occurred  */
-				barr:1,		/* Barrier op	   */
-				rsvd:8;
-		} berr_st;
-	} berr_un;
-} bridge_err_cmdword_t;
+#define b_devio(_x)	devio_raw[((_x) < 2) ? ((_x) * 2) : ((_x) + 2)].l
+#define b_devio_0	b_devio(0)
+#define b_devio_1	b_devio(1)
+#define b_devio_2	b_devio(2)
+#define b_devio_3	b_devio(3)
+#define b_devio_4	b_devio(4)
+#define b_devio_5	b_devio(5)
+#define b_devio_6	b_devio(6)
+#define b_devio_7	b_devio(7)
+
+/* Standard 32-bit access to the flash PROMs - BE CAREFUL! */
+#define b_ext_flash	ext_flash.l
 
-#define berr_field	berr_un.berr_st
 #endif /* !__ASSEMBLY__ */
 
 /*
- * The values of these macros can and should be crosschecked
- * regularly against the offsets of the like-named fields
- * within the "bridge_t" structure above.
+ * Shortcut accessor macros for reading/writing BRIDGE/XBRIDGE bits.
+ * @_bc: pointer to a 'struct bridge_controller' reference.
+ * @_r: register or field name from one of the many macros defined above.
+ * @_v: for writes, the value to write back to the register.
  */
+#define bridge_read_reg(_bc, _r)					\
+	__raw_readl(&_bc->bridge->regs._r)
+#define bridge_write_reg(_v, _bc, _r)					\
+	__raw_writel(_v, &_bc->bridge->regs._r)
+#define bridge_read(_bc, _r)						\
+	__raw_readl(_bc->bridge->_r)
+#define bridge_write(_v, _bc, _r)					\
+	__raw_writel(_v, _bc->bridge->_r)
 
-/* Byte offset macros for Bridge internal registers */
-
-#define BRIDGE_WID_ID		WIDGET_ID
-#define BRIDGE_WID_STAT		WIDGET_STATUS
-#define BRIDGE_WID_ERR_UPPER	WIDGET_ERR_UPPER_ADDR
-#define BRIDGE_WID_ERR_LOWER	WIDGET_ERR_LOWER_ADDR
-#define BRIDGE_WID_CONTROL	WIDGET_CONTROL
-#define BRIDGE_WID_REQ_TIMEOUT	WIDGET_REQ_TIMEOUT
-#define BRIDGE_WID_INT_UPPER	WIDGET_INTDEST_UPPER_ADDR
-#define BRIDGE_WID_INT_LOWER	WIDGET_INTDEST_LOWER_ADDR
-#define BRIDGE_WID_ERR_CMDWORD	WIDGET_ERR_CMD_WORD
-#define BRIDGE_WID_LLP		WIDGET_LLP_CFG
-#define BRIDGE_WID_TFLUSH	WIDGET_TFLUSH
-
-#define BRIDGE_WID_AUX_ERR	0x00005C	/* Aux Error Command Word */
-#define BRIDGE_WID_RESP_UPPER	0x000064	/* Response Buf Upper Addr */
-#define BRIDGE_WID_RESP_LOWER	0x00006C	/* Response Buf Lower Addr */
-#define BRIDGE_WID_TST_PIN_CTRL 0x000074	/* Test pin control */
-
-#define BRIDGE_DIR_MAP		0x000084	/* Direct Map reg */
+/* ----------------------------------------------------------------------- */
 
-#define BRIDGE_RAM_PERR		0x000094	/* SSRAM Parity Error */
 
-#define BRIDGE_ARB		0x0000A4	/* Arbitration Priority reg */
-
-#define BRIDGE_NIC		0x0000B4	/* Number In A Can */
-
-#define BRIDGE_BUS_TIMEOUT	0x0000C4	/* Bus Timeout Register */
-#define BRIDGE_PCI_BUS_TIMEOUT	BRIDGE_BUS_TIMEOUT
-#define BRIDGE_PCI_CFG		0x0000CC	/* PCI Type 1 Config reg */
-#define BRIDGE_PCI_ERR_UPPER	0x0000D4	/* PCI error Upper Addr */
-#define BRIDGE_PCI_ERR_LOWER	0x0000DC	/* PCI error Lower Addr */
-
-#define BRIDGE_INT_STATUS	0x000104	/* Interrupt Status */
-#define BRIDGE_INT_ENABLE	0x00010C	/* Interrupt Enables */
-#define BRIDGE_INT_RST_STAT	0x000114	/* Reset Intr Status */
-#define BRIDGE_INT_MODE		0x00011C	/* Interrupt Mode */
-#define BRIDGE_INT_DEVICE	0x000124	/* Interrupt Device */
-#define BRIDGE_INT_HOST_ERR	0x00012C	/* Host Error Field */
-
-#define BRIDGE_INT_ADDR0	0x000134	/* Host Address Reg */
-#define BRIDGE_INT_ADDR_OFF	0x000008	/* Host Addr offset (1..7) */
-#define BRIDGE_INT_ADDR(x)	(BRIDGE_INT_ADDR0+(x)*BRIDGE_INT_ADDR_OFF)
-
-#define BRIDGE_DEVICE0		0x000204	/* Device 0 */
-#define BRIDGE_DEVICE_OFF	0x000008	/* Device offset (1..7) */
-#define BRIDGE_DEVICE(x)	(BRIDGE_DEVICE0+(x)*BRIDGE_DEVICE_OFF)
-
-#define BRIDGE_WR_REQ_BUF0	0x000244	/* Write Request Buffer 0 */
-#define BRIDGE_WR_REQ_BUF_OFF	0x000008	/* Buffer Offset (1..7) */
-#define BRIDGE_WR_REQ_BUF(x)	(BRIDGE_WR_REQ_BUF0+(x)*BRIDGE_WR_REQ_BUF_OFF)
-
-#define BRIDGE_EVEN_RESP	0x000284	/* Even Device Response Buf */
-#define BRIDGE_ODD_RESP		0x00028C	/* Odd Device Response Buf */
-
-#define BRIDGE_RESP_STATUS	0x000294	/* Read Response Status reg */
-#define BRIDGE_RESP_CLEAR	0x00029C	/* Read Response Clear reg */
-
-/* Byte offset macros for Bridge I/O space */
-
-#define BRIDGE_ATE_RAM		0x00010000	/* Internal Addr Xlat Ram */
-
-#define BRIDGE_TYPE0_CFG_DEV0	0x00020000	/* Type 0 Cfg, Device 0 */
-#define BRIDGE_TYPE0_CFG_SLOT_OFF	0x00001000	/* Type 0 Cfg Slot Offset (1..7) */
-#define BRIDGE_TYPE0_CFG_FUNC_OFF	0x00000100	/* Type 0 Cfg Func Offset (1..7) */
-#define BRIDGE_TYPE0_CFG_DEV(s)		(BRIDGE_TYPE0_CFG_DEV0+\
-					 (s)*BRIDGE_TYPE0_CFG_SLOT_OFF)
-#define BRIDGE_TYPE0_CFG_DEVF(s, f)	(BRIDGE_TYPE0_CFG_DEV0+\
-					 (s)*BRIDGE_TYPE0_CFG_SLOT_OFF+\
-					 (f)*BRIDGE_TYPE0_CFG_FUNC_OFF)
-
-#define BRIDGE_TYPE1_CFG	0x00028000	/* Type 1 Cfg space */
-
-#define BRIDGE_PCI_IACK		0x00030000	/* PCI Interrupt Ack */
-#define BRIDGE_EXT_SSRAM	0x00080000	/* Extern SSRAM (ATE) */
-
-/* Byte offset macros for Bridge device IO spaces */
-
-#define BRIDGE_DEV_CNT		8	/* Up to 8 devices per bridge */
-#define BRIDGE_DEVIO0		0x00200000	/* Device IO 0 Addr */
-#define BRIDGE_DEVIO1		0x00400000	/* Device IO 1 Addr */
-#define BRIDGE_DEVIO2		0x00600000	/* Device IO 2 Addr */
-#define BRIDGE_DEVIO_OFF	0x00100000	/* Device IO Offset (3..7) */
-
-#define BRIDGE_DEVIO_2MB	0x00200000	/* Device IO Offset (0..1) */
-#define BRIDGE_DEVIO_1MB	0x00100000	/* Device IO Offset (2..7) */
-
-#define BRIDGE_DEVIO(x)		((x)<=1 ? BRIDGE_DEVIO0+(x)*BRIDGE_DEVIO_2MB : BRIDGE_DEVIO2+((x)-2)*BRIDGE_DEVIO_1MB)
-
-#define BRIDGE_EXTERNAL_FLASH	0x00C00000	/* External Flash PROMS */
-
-/* ========================================================================
- *    Bridge register bit field definitions
- */
+/* ----------------------------------------------------------------------- */
+/* Various BRIDGE/XBRIDGE definitions/params/macros. */
 
 /* Widget part numbers. */
 #define BRIDGE_WIDGET_PART_NUM		0xc002
@@ -723,30 +806,51 @@ typedef struct bridge_err_cmdword_s {
 /* This should be written to the XBOW's link_control(x) register */
 #define BRIDGE_CREDIT			3
 
-/* RRB assignment register -- value applies after shifting down */
-#define BRIDGE_RRB_EN			0x8
-#define BRIDGE_RRB_DEV			0x7
-#define BRIDGE_RRB_VDEV			0x4
-#define BRIDGE_RRB_PDEV			0x3
-
-/* RRB status register */
-#define BRIDGE_RRB_VALID(r)		(0x00010000 << (r))
-#define BRIDGE_RRB_INUSE(r)		(0x00000001 << (r))
-
-/* RRB clear register */
-#define BRIDGE_RRB_CLEAR(r)		(0x00000001 << (r))
-
 /* Xbox system controller declarations */
 #define XBOX_BRIDGE_WID			8
 #define FLASH_PROM1_BASE		0xe00000 /* Read Xbox sysctlr stat */
 #define XBOX_RPS_EXISTS			BIT(6)	 /* RPS bit in status reg */
 #define XBOX_RPS_FAIL			BIT(4)	 /* RPS status bit in reg */
 
-/* ========================================================================
+#ifndef __ASSEMBLY__
+/*
+ * XXX: Convert below struct into bitfield macros.  Or do away with
+ *	entirely.  Unused in Linux, but useful documentation.
  */
 /*
- * Macros for Xtalk to Bridge bus (PCI/GIO) PIO
- * refer to section 4.2.1 of Bridge Spec for xtalk to PCI/GIO PIO mappings
+ * Field formats for Error Command Word and Auxiliary Error Command Word
+ * of bridge.
+ */
+struct bridge_err_cmdword_s {
+	union {
+		u32		cmd_word;
+		struct {
+			u32	didn:4,		/* Destination ID  */
+				sidn:4,		/* Source ID	   */
+				pactyp:4,	/* Packet type	   */
+				tnum:5,		/* Trans Number	   */
+				coh:1,		/* Coh Transaction */
+				ds:2,		/* Data size	   */
+				gbr:1,		/* GBR enable	   */
+				vbpm:1,		/* VBPM message	   */
+				error:1,	/* Error occurred  */
+				barr:1,		/* Barrier op	   */
+				rsvd:8;
+		} berr_st;
+	} berr_un;
+};
+
+#define berr_field	berr_un.berr_st
+#endif /* !__ASSEMBLY__ */
+
+/* ----------------------------------------------------------------------- */
+
+
+/* ----------------------------------------------------------------------- */
+
+/*
+ * Macros for Xtalk to BRIDGE bus (PCI/GIO) PIO
+ * refer to section 4.2.1 of the BRIDGE Spec for xtalk to PCI/GIO PIO mappings
  */
 /* XTALK addresses that map into Bridge Bus addr space */
 #define BRIDGE_PIO32_XTALK_ALIAS_BASE	0x000040000000UL
@@ -757,7 +861,7 @@ typedef struct bridge_err_cmdword_s {
 #define BRIDGE_PCIIO_XTALK_ALIAS_LIMIT	0x0001ffffffffUL
 
 /* Ranges of PCI bus space that can be accessed via PIO from xtalk */
-#define BRIDGE_MIN_PIO_ADDR_MEM		0x00000000	/* 1G PCI memory space */
+#define BRIDGE_MIN_PIO_ADDR_MEM		0x00000000	/* 1G PCI mem space */
 #define BRIDGE_MAX_PIO_ADDR_MEM		0x3fffffff
 #define BRIDGE_MIN_PIO_ADDR_IO		0x00000000	/* 4G PCI IO space */
 #define BRIDGE_MAX_PIO_ADDR_IO		0xffffffff
@@ -829,10 +933,10 @@ typedef struct bridge_err_cmdword_s {
 /* 64-bit address attribute masks */
 #define PCI64_ATTR_TARG_MASK	GENMASK_ULL(63, 60)
 #define PCI64_ATTR_TARG_SHFT	60
-#define PCI64_ATTR_PREF		BIT_ULL(59)
-#define PCI64_ATTR_PREC		BIT_ULL(58)
-#define PCI64_ATTR_VIRTUAL	BIT_ULL(57)
-#define PCI64_ATTR_BAR		BIT_ULL(56)
+#define PCI64_ATTR_PREF		BIT_ULL(59)	/* Prefetch */
+#define PCI64_ATTR_PREC		BIT_ULL(58)	/* Precise */
+#define PCI64_ATTR_VIRTUAL	BIT_ULL(57)	/* Virtual Request */
+#define PCI64_ATTR_BAR		BIT_ULL(56)	/* Barrier */
 #define PCI64_ATTR_RMF_MASK	GENMASK_ULL(55, 48)
 #define PCI64_ATTR_RMF_SHFT	48
 
@@ -876,20 +980,74 @@ struct bridge_controller {
 	struct resource mem;
 	struct resource io;
 	struct resource busn;
-	bridge_t *base;
-	nasid_t nasid;
+	struct bridge_widget __iomem *bridge;
+	s16 nasid;
 	u32 widget_id;
-	u32 irq_cpu;
 	u64 baddr;
 	u32 pci_int[8];
+	u32 irq_cpu;
+	int (*alloc_irq)(struct pci_dev *);
 };
 
 #define BRIDGE_CONTROLLER(bus) \
 	((struct bridge_controller *)((bus)->sysdata))
 
-extern void register_bridge_irq(unsigned int irq);
-extern int request_bridge_irq(struct bridge_controller *bc);
-
 extern struct pci_ops bridge_pci_ops;
 
+/*
+ * Device might live on a subordinate PCI bus.  Walk up the chain of buses
+ * to find the slot number in sense of the bridge device register.
+ * XXX: This also means multiple devices might rely on conflicting bridge
+ * settings.
+ */
+static inline struct pci_dev *
+bridge_root_dev(struct pci_dev *dev)
+{
+	/* Move up the chain of bridges. */
+	while (dev->bus->parent)
+		dev = dev->bus->self;
+
+	return dev;
+}
+
+/*
+ * Simple macro to flush all pending BRIDGE PIO ops.
+ *
+ * XXX: IA64 actually checks the return value of the b_wid_targ_flush register
+ * and panics if it returns a non-zero status.
+ */
+#define BRIDGE_FLUSH(_bc)						\
+	do {								\
+		while (bridge_read_reg(_bc, b_wid_targ_flush))		\
+			cpu_relax();					\
+	} while (0)
+
+/*
+ * Magic/cosmic value for BRIDGE's INT_DEV register.  This sets bits 31:24
+ * to all 1's.  The only odd thing about this is those bits are marked as
+ * "reserved" in both the BRIDGE and XBRIDGE docs.  Apparently, IRIX does
+ * this as well.
+ */
+#define BRIDGE_COSMIC_INT_DEV    0xff000000
+
+/**
+ * struct bridge_platform_data - BRIDGE-specific data for IP27/IP30/IP35.
+ * @xio_target_addr: HUB/HEART/BEDROCK address in Crosstalk space.
+ * @baseio_widget_id: crosstalk widget ID for the BaseIO BRIDGE.
+ * @iomem_swap: bool that enables mem and I/O byteswapping on BRIDGE.
+ * @add_512: bool that if set, adds BRIDGE_DIRMAP_ADD512 to bridge->b_dir_map.
+ * @setup_baseio_rrbs: platform function to config the Read Response Buffers.
+ * @alloc_irq: platform function used by BRIDGE to allocate an IRQ.
+ * @pre_enable: platform function to setup BRIDGE before probing it.
+ */
+struct bridge_platform_data {
+	u32 xio_target_addr;
+	s8 baseio_widget_id;
+	bool iomem_swap;
+	bool add_512;
+	void (*setup_baseio_rrbs)(struct bridge_controller *, const bool *);
+	int (*alloc_irq)(struct pci_dev *);
+	int (*pre_enable)(struct pci_controller *, struct pci_dev *, int);
+};
+
 #endif /* _ASM_PCI_BRIDGE_H */
-- 
2.11.1

[PATCH 07/12] MIPS: BRIDGE: Add XBRIDGE revs and SWAP bit

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Add macros for the two known XBRIDGE revisions commonly found on IP35
hardware.  Both macros are sourced from the IA64 copy of bridge.h from
Linux-2.4.18.  Additionally, per the XBRIDGE ASIC specification, bit 55
of the Crosstalk address selects whether to use byte-swapping or not on
the XBRIDGE.

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
---
 arch/mips/include/asm/pci/bridge.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/mips/include/asm/pci/bridge.h b/arch/mips/include/asm/pci/bridge.h
index 3c214feab772..f173875b4233 100644
--- a/arch/mips/include/asm/pci/bridge.h
+++ b/arch/mips/include/asm/pci/bridge.h
@@ -484,6 +484,8 @@ struct bridge_widget {
 #define BRIDGE_REV_B			0x2
 #define BRIDGE_REV_C			0x3
 #define BRIDGE_REV_D			0x4
+#define XBRIDGE_REV_A			0x1
+#define XBRIDGE_REV_B			0x2
 
 /* Bridge widget status register bits definition */
 #define BRIDGE_STAT_LLP_REC_CNT		GENMASK(31, 24)
@@ -937,6 +939,7 @@ struct bridge_err_cmdword_s {
 #define PCI64_ATTR_PREC		BIT_ULL(58)	/* Precise */
 #define PCI64_ATTR_VIRTUAL	BIT_ULL(57)	/* Virtual Request */
 #define PCI64_ATTR_BAR		BIT_ULL(56)	/* Barrier */
+#define PCI64_ATTR_SWAP		BIT_ULL(55)	/* Byte swap (XBRIDGE) */
 #define PCI64_ATTR_RMF_MASK	GENMASK_ULL(55, 48)
 #define PCI64_ATTR_RMF_SHFT	48
 
-- 
2.11.1

[PATCH 08/12] MIPS: BRIDGE: Update ops-bridge.c

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Update arch/mips/pci/ops-bridge.c to use the new access structs added
to bridge.h in the BRIDGE overhaul patch.  Additional changes include
various fixes to comments, whitespace, and marking the 'bridge_pci_ops'
struct as __read_mostly.

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
---
 arch/mips/pci/ops-bridge.c | 125 ++++++++++++++++++-----------------
 1 file changed, 65 insertions(+), 60 deletions(-)

diff --git a/arch/mips/pci/ops-bridge.c b/arch/mips/pci/ops-bridge.c
index 57e1463fcd02..6793a72edec2 100644
--- a/arch/mips/pci/ops-bridge.c
+++ b/arch/mips/pci/ops-bridge.c
@@ -8,16 +8,20 @@
  */
 #include <linux/pci.h>
 #include <asm/paccess.h>
+
+#if defined(CONFIG_SGI_IP27)
 #include <asm/pci/bridge.h>
 #include <asm/sn/arch.h>
 #include <asm/sn/intr.h>
 #include <asm/sn/sn0/hub.h>
+#endif
 
 /*
  * Most of the IOC3 PCI config register aren't present
  * we emulate what is needed for a normal PCI enumeration
  */
-static u32 emulate_ioc3_cfg(int where, int size)
+static inline u32
+emulate_ioc3_cfg(int where, int size)
 {
 	if (size == 1 && where == 0x3d)
 		return 0x01;
@@ -40,29 +44,29 @@ static u32 emulate_ioc3_cfg(int where, int size)
  * accesses and does only decode parts of it's address space.
  */
 
-static int pci_conf0_read_config(struct pci_bus *bus, unsigned int devfn,
-				 int where, int size, u32 * value)
+static int
+pci_conf0_read_config(struct pci_bus *bus, u32 devfn, int where, int size,
+		      u32 *value)
 {
 	struct bridge_controller *bc = BRIDGE_CONTROLLER(bus);
-	bridge_t *bridge = bc->base;
 	int slot = PCI_SLOT(devfn);
 	int fn = PCI_FUNC(devfn);
 	volatile void *addr;
 	u32 cf, shift, mask;
 	int res;
 
-	addr = &bridge->b_type0_cfg_dev[slot].f[fn].c[PCI_VENDOR_ID];
+	addr = &bc->bridge->pci_t0[slot].func[fn].b[PCI_VENDOR_ID];
 	if (get_dbe(cf, (u32 *) addr))
 		return PCIBIOS_DEVICE_NOT_FOUND;
 
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't even give the
-	 * generic PCI code a chance to look at it for real ...
+	 * IOC3 is fucking fucked beyond belief.  Don't even give the
+	 * generic PCI code a chance to look at it for real.
 	 */
 	if (cf == (PCI_VENDOR_ID_SGI | (PCI_DEVICE_ID_SGI_IOC3 << 16)))
 		goto oh_my_gawd;
 
-	addr = &bridge->b_type0_cfg_dev[slot].f[fn].c[where ^ (4 - size)];
+	addr = &bc->bridge->pci_t0[slot].func[fn].b[where ^ (4 - size)];
 
 	if (size == 1)
 		res = get_dbe(*value, (u8 *) addr);
@@ -74,9 +78,8 @@ static int pci_conf0_read_config(struct pci_bus *bus, unsigned int devfn,
 	return res ? PCIBIOS_DEVICE_NOT_FOUND : PCIBIOS_SUCCESSFUL;
 
 oh_my_gawd:
-
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't even give the
+	 * IOC3 is fucking fucked beyond belief.  Don't even give the
 	 * generic PCI code a chance to look at the wrong register.
 	 */
 	if ((where >= 0x14 && where < 0x40) || (where >= 0x48)) {
@@ -85,10 +88,10 @@ static int pci_conf0_read_config(struct pci_bus *bus, unsigned int devfn,
 	}
 
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't try to access
-	 * anything but 32-bit words ...
+	 * IOC3 is fucking fucked beyond belief.  Don't try to access
+	 * anything but 32-bit words.
 	 */
-	addr = &bridge->b_type0_cfg_dev[slot].f[fn].l[where >> 2];
+	addr = &bc->bridge->pci_t0[slot].func[fn].l[where >> 2];
 
 	if (get_dbe(cf, (u32 *) addr))
 		return PCIBIOS_DEVICE_NOT_FOUND;
@@ -100,11 +103,11 @@ static int pci_conf0_read_config(struct pci_bus *bus, unsigned int devfn,
 	return PCIBIOS_SUCCESSFUL;
 }
 
-static int pci_conf1_read_config(struct pci_bus *bus, unsigned int devfn,
-				 int where, int size, u32 * value)
+static int
+pci_conf1_read_config(struct pci_bus *bus, u32 devfn, int where, int size,
+		      u32 *value)
 {
 	struct bridge_controller *bc = BRIDGE_CONTROLLER(bus);
-	bridge_t *bridge = bc->base;
 	int busno = bus->number;
 	int slot = PCI_SLOT(devfn);
 	int fn = PCI_FUNC(devfn);
@@ -112,20 +115,20 @@ static int pci_conf1_read_config(struct pci_bus *bus, unsigned int devfn,
 	u32 cf, shift, mask;
 	int res;
 
-	bridge->b_pci_cfg = (busno << 16) | (slot << 11);
-	addr = &bridge->b_type1_cfg.c[(fn << 8) | PCI_VENDOR_ID];
+	bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg);
+	addr = &bc->bridge->pci_t1.b[(fn << 8) | PCI_VENDOR_ID];
 	if (get_dbe(cf, (u32 *) addr))
 		return PCIBIOS_DEVICE_NOT_FOUND;
 
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't even give the
-	 * generic PCI code a chance to look at it for real ...
+	 * IOC3 is fucking fucked beyond belief.  Don't even give the
+	 * generic PCI code a chance to look at it for real.
 	 */
 	if (cf == (PCI_VENDOR_ID_SGI | (PCI_DEVICE_ID_SGI_IOC3 << 16)))
 		goto oh_my_gawd;
 
-	bridge->b_pci_cfg = (busno << 16) | (slot << 11);
-	addr = &bridge->b_type1_cfg.c[(fn << 8) | (where ^ (4 - size))];
+	bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg);
+	addr = &bc->bridge->pci_t1.b[(fn << 8) | (where ^ (4 - size))];
 
 	if (size == 1)
 		res = get_dbe(*value, (u8 *) addr);
@@ -137,9 +140,8 @@ static int pci_conf1_read_config(struct pci_bus *bus, unsigned int devfn,
 	return res ? PCIBIOS_DEVICE_NOT_FOUND : PCIBIOS_SUCCESSFUL;
 
 oh_my_gawd:
-
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't even give the
+	 * IOC3 is fucking fucked beyond belief.  Don't even give the
 	 * generic PCI code a chance to look at the wrong register.
 	 */
 	if ((where >= 0x14 && where < 0x40) || (where >= 0x48)) {
@@ -148,11 +150,11 @@ static int pci_conf1_read_config(struct pci_bus *bus, unsigned int devfn,
 	}
 
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't try to access
-	 * anything but 32-bit words ...
+	 * IOC3 is fucking fucked beyond belief.  Don't try to access
+	 * anything but 32-bit words.
 	 */
-	bridge->b_pci_cfg = (busno << 16) | (slot << 11);
-	addr = &bridge->b_type1_cfg.c[(fn << 8) | where];
+	bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg);
+	addr = &bc->bridge->pci_t1.b[(fn << 8) | where];
 
 	if (get_dbe(cf, (u32 *) addr))
 		return PCIBIOS_DEVICE_NOT_FOUND;
@@ -164,8 +166,9 @@ static int pci_conf1_read_config(struct pci_bus *bus, unsigned int devfn,
 	return PCIBIOS_SUCCESSFUL;
 }
 
-static int pci_read_config(struct pci_bus *bus, unsigned int devfn,
-			   int where, int size, u32 * value)
+static int
+pci_read_config(struct pci_bus *bus, u32 devfn, int where, int size,
+		u32 *value)
 {
 	if (bus->number > 0)
 		return pci_conf1_read_config(bus, devfn, where, size, value);
@@ -173,29 +176,29 @@ static int pci_read_config(struct pci_bus *bus, unsigned int devfn,
 	return pci_conf0_read_config(bus, devfn, where, size, value);
 }
 
-static int pci_conf0_write_config(struct pci_bus *bus, unsigned int devfn,
-				  int where, int size, u32 value)
+static int
+pci_conf0_write_config(struct pci_bus *bus, u32 devfn, int where, int size,
+		       u32 value)
 {
 	struct bridge_controller *bc = BRIDGE_CONTROLLER(bus);
-	bridge_t *bridge = bc->base;
 	int slot = PCI_SLOT(devfn);
 	int fn = PCI_FUNC(devfn);
 	volatile void *addr;
 	u32 cf, shift, mask, smask;
 	int res;
 
-	addr = &bridge->b_type0_cfg_dev[slot].f[fn].c[PCI_VENDOR_ID];
+	addr = &bc->bridge->pci_t0[slot].func[fn].b[PCI_VENDOR_ID];
 	if (get_dbe(cf, (u32 *) addr))
 		return PCIBIOS_DEVICE_NOT_FOUND;
 
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't even give the
-	 * generic PCI code a chance to look at it for real ...
+	 * IOC3 is fucking fucked beyond belief.  Don't even give the
+	 * generic PCI code a chance to look at it for real.
 	 */
 	if (cf == (PCI_VENDOR_ID_SGI | (PCI_DEVICE_ID_SGI_IOC3 << 16)))
 		goto oh_my_gawd;
 
-	addr = &bridge->b_type0_cfg_dev[slot].f[fn].c[where ^ (4 - size)];
+	addr = &bc->bridge->pci_t0[slot].func[fn].b[where ^ (4 - size)];
 
 	if (size == 1) {
 		res = put_dbe(value, (u8 *) addr);
@@ -211,19 +214,18 @@ static int pci_conf0_write_config(struct pci_bus *bus, unsigned int devfn,
 	return PCIBIOS_SUCCESSFUL;
 
 oh_my_gawd:
-
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't even give the
-	 * generic PCI code a chance to touch the wrong register.
+	 * IOC3 is fucking fucked beyond belief.  Don't even give the
+	 * generic PCI code a chance to look at the wrong register.
 	 */
 	if ((where >= 0x14 && where < 0x40) || (where >= 0x48))
 		return PCIBIOS_SUCCESSFUL;
 
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't try to access
-	 * anything but 32-bit words ...
+	 * IOC3 is fucking fucked beyond belief.  Don't try to access
+	 * anything but 32-bit words.
 	 */
-	addr = &bridge->b_type0_cfg_dev[slot].f[fn].l[where >> 2];
+	addr = &bc->bridge->pci_t0[slot].func[fn].l[where >> 2];
 
 	if (get_dbe(cf, (u32 *) addr))
 		return PCIBIOS_DEVICE_NOT_FOUND;
@@ -239,11 +241,11 @@ static int pci_conf0_write_config(struct pci_bus *bus, unsigned int devfn,
 	return PCIBIOS_SUCCESSFUL;
 }
 
-static int pci_conf1_write_config(struct pci_bus *bus, unsigned int devfn,
-				  int where, int size, u32 value)
+static int
+pci_conf1_write_config(struct pci_bus *bus, u32 devfn, int where, int size,
+		       u32 value)
 {
 	struct bridge_controller *bc = BRIDGE_CONTROLLER(bus);
-	bridge_t *bridge = bc->base;
 	int slot = PCI_SLOT(devfn);
 	int fn = PCI_FUNC(devfn);
 	int busno = bus->number;
@@ -251,19 +253,20 @@ static int pci_conf1_write_config(struct pci_bus *bus, unsigned int devfn,
 	u32 cf, shift, mask, smask;
 	int res;
 
-	bridge->b_pci_cfg = (busno << 16) | (slot << 11);
-	addr = &bridge->b_type1_cfg.c[(fn << 8) | PCI_VENDOR_ID];
+	bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg);
+	addr = &bc->bridge->pci_t1.b[(fn << 8) | PCI_VENDOR_ID];
 	if (get_dbe(cf, (u32 *) addr))
 		return PCIBIOS_DEVICE_NOT_FOUND;
 
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't even give the
-	 * generic PCI code a chance to look at it for real ...
+	 * IOC3 is fucking fucked beyond belief.  Don't even give the
+	 * generic PCI code a chance to look at it for real.
 	 */
 	if (cf == (PCI_VENDOR_ID_SGI | (PCI_DEVICE_ID_SGI_IOC3 << 16)))
 		goto oh_my_gawd;
 
-	addr = &bridge->b_type1_cfg.c[(fn << 8) | (where ^ (4 - size))];
+	bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg);
+	addr = &bc->bridge->pci_t1.b[(fn << 8) | (where ^ (4 - size))];
 
 	if (size == 1) {
 		res = put_dbe(value, (u8 *) addr);
@@ -279,19 +282,19 @@ static int pci_conf1_write_config(struct pci_bus *bus, unsigned int devfn,
 	return PCIBIOS_SUCCESSFUL;
 
 oh_my_gawd:
-
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't even give the
-	 * generic PCI code a chance to touch the wrong register.
+	 * IOC3 is fucking fucked beyond belief.  Don't even give the
+	 * generic PCI code a chance to look at the wrong register.
 	 */
 	if ((where >= 0x14 && where < 0x40) || (where >= 0x48))
 		return PCIBIOS_SUCCESSFUL;
 
 	/*
-	 * IOC3 is fucking fucked beyond belief ...  Don't try to access
-	 * anything but 32-bit words ...
+	 * IOC3 is fucking fucked beyond belief.  Don't try to access
+	 * anything but 32-bit words.
 	 */
-	addr = &bridge->b_type0_cfg_dev[slot].f[fn].l[where >> 2];
+	bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg);
+	addr = &bc->bridge->pci_t1.b[(fn << 8) | where];
 
 	if (get_dbe(cf, (u32 *) addr))
 		return PCIBIOS_DEVICE_NOT_FOUND;
@@ -307,8 +310,9 @@ static int pci_conf1_write_config(struct pci_bus *bus, unsigned int devfn,
 	return PCIBIOS_SUCCESSFUL;
 }
 
-static int pci_write_config(struct pci_bus *bus, unsigned int devfn,
-	int where, int size, u32 value)
+static int
+pci_write_config(struct pci_bus *bus, u32 devfn, int where, int size,
+		 u32 value)
 {
 	if (bus->number > 0)
 		return pci_conf1_write_config(bus, devfn, where, size, value);
@@ -316,7 +320,8 @@ static int pci_write_config(struct pci_bus *bus, unsigned int devfn,
 	return pci_conf0_write_config(bus, devfn, where, size, value);
 }
 
-struct pci_ops bridge_pci_ops = {
+struct pci_ops __read_mostly
+bridge_pci_ops = {
 	.read	= pci_read_config,
 	.write	= pci_write_config,
 };
-- 
2.11.1

[PATCH 09/12] MIPS: BRIDGE: Use !pci_is_root_bus(bus) to check for bus->number > 0

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

This is a manual cherrypick of commit c7ddc3d137b7 from Alastair
Bridgewater's IP35 tree.  In arch/mips/pci/ops-bridge.c, replace:
    if (bus->number > 0)
        foo();

With:
    if (!pci_is_root_bus(bus))
        foo();

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
Cc: Alastair Bridgewater <alastair.bridgewater@gmail.com>
---
 arch/mips/pci/ops-bridge.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/mips/pci/ops-bridge.c b/arch/mips/pci/ops-bridge.c
index 6793a72edec2..28efce2dcb7d 100644
--- a/arch/mips/pci/ops-bridge.c
+++ b/arch/mips/pci/ops-bridge.c
@@ -170,7 +170,7 @@ static int
 pci_read_config(struct pci_bus *bus, u32 devfn, int where, int size,
 		u32 *value)
 {
-	if (bus->number > 0)
+	if (!pci_is_root_bus(bus))
 		return pci_conf1_read_config(bus, devfn, where, size, value);
 
 	return pci_conf0_read_config(bus, devfn, where, size, value);
@@ -314,7 +314,7 @@ static int
 pci_write_config(struct pci_bus *bus, u32 devfn, int where, int size,
 		 u32 value)
 {
-	if (bus->number > 0)
+	if (!pci_is_root_bus(bus))
 		return pci_conf1_write_config(bus, devfn, where, size, value);
 
 	return pci_conf0_write_config(bus, devfn, where, size, value);
-- 
2.11.1

[PATCH 10/12] MIPS: BRIDGE: Overhaul pci-bridge.c driver

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Rewrite the BRIDGE code in pci-bridge.c to address a number of issues:
 - Partially re-write to support platform_driver in the future
 - Use the new access structs from the bridge.h overhaul
 - Handle several BRIDGE WARs found in old IA64 code from 2.4/2.5
 - Document what the driver is doing during probe
 - Update copyrights

A lot of the changes are built up over the years from the IP30 patch
series as well as other contributions, along with work done on the
IP27 code that will be part of a forth-coming patch series.

There are several temporary pieces of code left that are needed by the
current IP27 code to boot correctly under these changes.  They will
be removed in the future with the IP27 "mega update" patch series and
this driver will become a full platform_driver module.

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
Cc: Stanislaw Skowronek <skylark@unaligned.org>
Cc: Johannes Dickgreber <tanzy@gmx.de>
---
 arch/mips/pci/pci-bridge.c | 307 +++++++++++++++++++++--------------
 1 file changed, 186 insertions(+), 121 deletions(-)

diff --git a/arch/mips/pci/pci-bridge.c b/arch/mips/pci/pci-bridge.c
index 0f09eafa5e3a..28b0eb5b844e 100644
--- a/arch/mips/pci/pci-bridge.c
+++ b/arch/mips/pci/pci-bridge.c
@@ -3,162 +3,236 @@
  * License.  See the file "COPYING" in the main directory of this archive
  * for more details.
  *
- * Copyright (C) 2003 Christoph Hellwig (hch@lst.de)
- * Copyright (C) 1999, 2000, 04 Ralf Baechle (ralf@linux-mips.org)
- * Copyright (C) 1999, 2000 Silicon Graphics, Inc.
+ * arch/mips/pci/pci-bridge.c
+ * platform_driver for SGI BRIDGE/XBRIDGE (and in the future, SGI PIC) ASICs.
+ *
+ * Originally called pci-ip27.c, which is:
+ *   Copyright (C) 2003 Christoph Hellwig (hch@lst.de)
+ *   Copyright (C) 1999, 2000, 04 Ralf Baechle (ralf@linux-mips.org)
+ *   Copyright (C) 1999, 2000 Silicon Graphics, Inc.
+ *
+ * Modifications sourced from pci-ip30.c in the IP30 patchset are:
+ *   Copyright (C) 2004-2007 Stanislaw Skowronek <skylark@unaligned.org>
+ *   Copyright (C) 2009 Johannes Dickgreber <tanzy@gmx.de>
+ *   Copyright (C) 2016 Joshua Kinard <kumba@gentoo.org>
+ *
+ * Functions/info/insight sourced from old IA64 code are:
+ *   Copyright (C) 1992 - 1997, 2000-2002 Silicon Graphics, Inc.
+ *                 All rights reserved.
  */
+
 #include <linux/kernel.h>
-#include <linux/export.h>
 #include <linux/pci.h>
-#include <linux/smp.h>
-#include <asm/sn/arch.h>
-#include <asm/pci/bridge.h>
+#include <linux/platform_device.h>
+
 #include <asm/paccess.h>
-#include <asm/sn/intr.h>
-#include <asm/sn/sn0/hub.h>
+#include <asm/pci/bridge.h>
+#include <asm/xtalk/xtalk.h>
 
-/*
- * Max #PCI busses we can handle; ie, max #PCI bridges.
- */
-#define MAX_PCI_BUSSES		40
+#if defined(CONFIG_SGI_IP27)
+#include <asm/mach-ip27/pcibr.h>
+#else
+#error "Unknown CONFIG_SGI_IP??"
+#endif
 
-/*
- * Max #PCI devices (like scsi controllers) we handle on a bus.
- */
-#define MAX_DEVICES_PER_PCIBUS	8
+/* Increments for each additional bridge. */
+static int num_bridges;
 
+
+/* XXX: Temporary until the IP27 "mega update". */
 /*
  * XXX: No kmalloc available when we do our crosstalk scan,
- *	we should try to move it later in the boot process.
+ *     we should try to move it later in the boot process.
  */
-static struct bridge_controller bridges[MAX_PCI_BUSSES];
+static struct bridge_controller bridges[PCIBR_MAX_NUM_PCIBUS];
 
-/*
- * Translate from irq to software PCI bus number and PCI slot.
+/**
+ * bridge_probe - probes a BRIDGE chip and configures it.
+ * @nasid: NUMA Address Space Identifier.
+ * @widget_id: s8 value of this BRIDGE's xtalk widget ID.
+ * @masterwid: widget ID of HUB.
+ *
+ * Always returns '0'.
  */
-struct bridge_controller *irq_to_bridge[MAX_PCI_BUSSES * MAX_DEVICES_PER_PCIBUS];
-int irq_to_slot[MAX_PCI_BUSSES * MAX_DEVICES_PER_PCIBUS];
-
-extern struct pci_ops bridge_pci_ops;
-
-int bridge_probe(nasid_t nasid, int widget_id, int masterwid)
+int
+bridge_probe(nasid_t nasid, int widget_id, int masterwid)
 {
+	u32 slot, reg;
+	u32 wid_ctrl; /* BRIDGE WAR */
 	unsigned long offset = NODE_OFFSET(nasid);
 	struct bridge_controller *bc;
-	static int num_bridges = 0;
-	bridge_t *bridge;
-	int slot;
-
-	pci_set_flags(PCI_PROBE_ONLY);
 
-	printk("a bridge\n");
-
-	/* XXX: kludge alert.. */
+	/* XXX: Temporary until the IP27 "mega update". */
+	bc = &bridges[num_bridges];
 	if (!num_bridges)
 		ioport_resource.end = ~0UL;
 
-	bc = &bridges[num_bridges];
-
-	bc->pc.pci_ops		= &bridge_pci_ops;
-	bc->pc.mem_resource	= &bc->mem;
-	bc->pc.io_resource	= &bc->io;
-
-	bc->pc.index		= num_bridges;
+	/* Set bridge_controller parameters. */
+	bc->pc.pci_ops = &bridge_pci_ops;
+	bc->pc.mem_resource = &bc->mem;
+	bc->pc.mem_offset = offset;
+	bc->pc.io_resource = &bc->io;
+	bc->pc.io_offset = offset;
+	bc->pc.busn_resource = &bc->busn;
+	bc->pc.busn_offset = offset;
+	bc->pc.index = num_bridges;
+	bc->pc.io_map_base = NODE_SWIN_BASE(nasid, widget_id);
+
+	bc->mem.name = "Bridge MEM";
+	bc->mem.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_MEM);
+	bc->mem.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO - 1);
+	bc->mem.flags = IORESOURCE_MEM;
+
+	bc->io.name = "Bridge IO";
+	bc->io.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO);
+	bc->io.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_END - 1);
+	bc->io.flags = IORESOURCE_IO;
+
+	bc->busn.name = "Bridge BUSN";
+	bc->busn.start = num_bridges;
+	bc->busn.end = 255;
+	bc->busn.flags = IORESOURCE_BUS;
 
-	bc->mem.name		= "Bridge PCI MEM";
-	bc->pc.mem_offset	= offset;
-	bc->mem.start		= 0;
-	bc->mem.end		= ~0UL;
-	bc->mem.flags		= IORESOURCE_MEM;
-
-	bc->io.name		= "Bridge IO MEM";
-	bc->pc.io_offset	= offset;
-	bc->io.start		= 0UL;
-	bc->io.end		= ~0UL;
-	bc->io.flags		= IORESOURCE_IO;
-
-	bc->irq_cpu = smp_processor_id();
 	bc->widget_id = widget_id;
 	bc->nasid = nasid;
+	bc->baddr = ((u64)masterwid << PCI64_ATTR_TARG_SHFT);
+
+	/* XXX: Kill */
+	bc->irq_cpu = smp_processor_id();
 
-	bc->baddr = (u64)masterwid << 60 | PCI64_ATTR_BAR;
+	/*
+	 * Point to this bridge
+	 */
+	bc->bridge = (struct bridge_widget __iomem *)
+		RAW_NODE_SWIN_BASE(nasid, widget_id);
 
 	/*
-	 * point to this bridge
+	 * On BRIDGEs prior to Rev D, set the PCI_RETRY_CNT to zero to avoid
+	 * dropping stores (WAR #475347).
+	 * Sourced from Linux-2.5.70/arch/ia64/sn/io/sn1/pcibr.c
 	 */
-	bridge = (bridge_t *) RAW_NODE_SWIN_BASE(nasid, widget_id);
+	if (XWIDGET_REV_NUM(bridge_read_reg(bc, b_wid_id)) < BRIDGE_REV_D) {
+		reg = bridge_read_reg(bc, b_pci_bus_timo);
+		bridge_write_reg((reg & ~(BRIDGE_BUS_PCI_RETRY_MASK)), bc,
+				 b_pci_bus_timo);
+	}
 
 	/*
 	 * Clear all pending interrupts.
 	 */
-	bridge->b_int_rst_stat = BRIDGE_IRR_ALL_CLR;
+	bridge_write_reg(BRIDGE_IRR_ALL_CLR, bc, b_int_reset_stat);
 
 	/*
 	 * Until otherwise set up, assume all interrupts are from slot 0
 	 */
-	bridge->b_int_device = 0x0;
+	bridge_write_reg(0, bc, b_int_device);
+
+	/* Configure BRIDGE widget control ... */
+	wid_ctrl = bridge_read_reg(bc, b_wid_ctrl);
 
 	/*
-	 * swap pio's to pci mem and io space (big windows)
+	 * IP27 & IP35 need I/O and Mem swapping enabled.
+	 * IP30 needs it disabled.
 	 */
-	bridge->b_wid_control |= BRIDGE_CTRL_IO_SWAP |
-				 BRIDGE_CTRL_MEM_SWAP;
+	wid_ctrl |= (BRIDGE_CTRL_IO_SWAP | BRIDGE_CTRL_MEM_SWAP);
+
+	/* Set the BRIDGE PAGE_SIZE */
 #ifdef CONFIG_PAGE_SIZE_4KB
-	bridge->b_wid_control &= ~BRIDGE_CTRL_PAGE_SIZE;
+	wid_ctrl &= ~BRIDGE_CTRL_PAGE_SIZE;
 #else /* 16kB or larger */
-	bridge->b_wid_control |= BRIDGE_CTRL_PAGE_SIZE;
+	wid_ctrl |= BRIDGE_CTRL_PAGE_SIZE;
 #endif
 
 	/*
-	 * Hmm...  IRIX sets additional bits in the address which
-	 * are documented as reserved in the bridge docs.
+	 * Another BRIDGE WAR, read the BRIDGE widget control register
+	 * back after writing it to avoid an invalid address bug.
+	 */
+	bridge_write_reg(wid_ctrl, bc, b_wid_ctrl);
+	wid_ctrl = bridge_read_reg(bc, b_wid_ctrl);
+
+	/*
+	 * Set per-device properties.
+	 *
+	 * XXX: Setup per-slot configuration at some point.  Different devices
+	 * need different properties.
 	 */
-	bridge->b_wid_int_upper = 0x8000 | (masterwid << 16);
-	bridge->b_wid_int_lower = 0x01800090;	/* PI_INT_PEND_MOD off*/
-	bridge->b_dir_map = (masterwid << 20);	/* DMA */
-	bridge->b_int_enable = 0;
+	for (slot = 0; slot < BRIDGE_MAX_DEVS; slot++) {
+		reg = bridge_read_reg(bc, b_device(slot));
+		reg &= ~BRIDGE_DEV_PAGE_CHK_DIS;
+		reg |= (BRIDGE_DEV_ERR_LOCK_EN | BRIDGE_DEV_D64_BITS);
 
-	for (slot = 0; slot < 8; slot ++) {
-		bridge->b_device[slot].reg |= BRIDGE_DEV_SWAP_DIR;
+		bridge_write_reg(reg, bc, b_device(slot));
 		bc->pci_int[slot] = -1;
 	}
-	bridge->b_wid_tflush;	  /* wait until Bridge PIO complete */
 
-	bc->base = bridge;
-
-	register_pci_controller(&bc->pc);
+	/* Configure direct-mapped DMA */
+	reg = (masterwid << BRIDGE_DIRMAP_W_ID_SHFT);
+	bridge_write_reg(reg, bc, b_dir_map);
 
+	/*
+	 * Route all PCI bridge interrupts to the appropriate ASIC responsible
+	 * for handling IRQs (HUB in IP27, HEART in IP30, BEDROCK in IP35).
+	 * The actual IRQ support and masking is done elsewhere.
+	 */
+	/*
+	 * XXX: IRIX sets additional bits (0x8000) in the address which are
+	 * marked as reserved in the BRIDGE docs.
+	 */
+	bridge_write_reg(((masterwid << WIDGET_TARGET_ID_SHFT) | 0x8000), bc,
+			 b_wid_int_upper);
+	bridge_write_reg(PCIBR_XIO_SEES_HUB, bc, b_wid_int_lower);
+	bridge_write_reg(BRIDGE_COSMIC_INT_DEV, bc, b_int_device);
+	bridge_write_reg(0, bc, b_int_mode);
+	reg = bridge_read_reg(bc, b_int_enable);
+	bridge_write_reg((reg | BRIDGE_ISR_ERRORS), bc, b_int_enable);
+
+	/* Wait until Bridge PIO completes. */
+	BRIDGE_FLUSH(bc);
+
+	/* Increment number of discovered BRIDGE/XBRIDGE widgets. */
 	num_bridges++;
 
+	register_pci_controller(&bc->pc);
+
 	return 0;
 }
 
+/**
+ * bridge_alloc_irq - allocate the next available IRQ for a BRIDGE slot.
+ * @dev: pointer to struct pci_dev of PCI device info.
+ *
+ * Casts dev->bus->sysdata to struct bridge_controller and returns the
+ * outcome of the platform_data-defined 'alloc_irq' function.
+ */
+inline int
+bridge_alloc_irq(struct pci_dev *dev)
+{
+	const struct bridge_controller *bc = BRIDGE_CONTROLLER(dev->bus);
+
+	if (bc->alloc_irq)
+		return bc->alloc_irq(dev);
+
+	return -1;
+}
+
 /*
  * All observed requests have pin == 1. We could have a global here, that
  * gets incremented and returned every time - unfortunately, pci_map_irq
  * may be called on the same device over and over, and need to return the
  * same value. On O2000, pin can be 0 or 1, and PCI slots can be [0..7].
  *
- * A given PCI device, in general, should be able to intr any of the cpus
- * on any one of the hubs connected to its xbow.
+ * A given PCI device, in general, should be able to interrupt any of the
+ * cpus on any one of the HUBs or HEART connected to its xbow.
  */
-int pcibios_map_irq(const struct pci_dev *dev, u8 slot, u8 pin)
+int __init
+pcibios_map_irq(const struct pci_dev *dev, u8 slot, u8 pin)
 {
 	return 0;
 }
 
-static inline struct pci_dev *bridge_root_dev(struct pci_dev *dev)
-{
-	while (dev->bus->parent) {
-		/* Move up the chain of bridges. */
-		dev = dev->bus->self;
-	}
-
-	return dev;
-}
-
 /* Do platform specific device initialization at pci_enable_device() time */
-int pcibios_plat_dev_init(struct pci_dev *dev)
+int
+pcibios_plat_dev_init(struct pci_dev *dev)
 {
 	struct bridge_controller *bc = BRIDGE_CONTROLLER(dev->bus);
 	struct pci_dev *rdev = bridge_root_dev(dev);
@@ -182,42 +256,35 @@ int pcibios_plat_dev_init(struct pci_dev *dev)
 	return 0;
 }
 
-/*
- * Device might live on a subordinate PCI bus.	XXX Walk up the chain of buses
- * to find the slot number in sense of the bridge device register.
- * XXX This also means multiple devices might rely on conflicting bridge
- * settings.
- */
-
-static inline void pci_disable_swapping(struct pci_dev *dev)
+static void
+bridge_disable_swapping_dma(struct pci_dev *dev)
 {
-	struct bridge_controller *bc = BRIDGE_CONTROLLER(dev->bus);
-	bridge_t *bridge = bc->base;
+	u32 reg;
 	int slot = PCI_SLOT(dev->devfn);
-
-	/* Turn off byte swapping */
-	bridge->b_device[slot].reg &= ~BRIDGE_DEV_SWAP_DIR;
-	bridge->b_widget.w_tflush;	/* Flush */
-}
-
-static inline void pci_enable_swapping(struct pci_dev *dev)
-{
 	struct bridge_controller *bc = BRIDGE_CONTROLLER(dev->bus);
-	bridge_t *bridge = bc->base;
-	int slot = PCI_SLOT(dev->devfn);
 
-	/* Turn on byte swapping */
-	bridge->b_device[slot].reg |= BRIDGE_DEV_SWAP_DIR;
-	bridge->b_widget.w_tflush;	/* Flush */
+	/* Turn off byte swapping */
+	reg = bridge_read_reg(bc, b_device(slot));
+	reg &= ~(BRIDGE_DEV_DEV_SWAP | BRIDGE_DEV_SWAP_PMU |
+		 BRIDGE_DEV_SWAP_DIR);
+	bridge_write_reg(reg, bc, b_device(slot));
+	BRIDGE_FLUSH(bc);
 }
 
-static void pci_fixup_ioc3(struct pci_dev *d)
-{
-	pci_disable_swapping(d);
-}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SGI, PCI_DEVICE_ID_SGI_IOC3,
+			bridge_disable_swapping_dma);
 
+/* XXX: Temporarily defined here until the IP27 "mega update". */
 #ifdef CONFIG_NUMA
-int pcibus_to_node(struct pci_bus *bus)
+/**
+ * pcibus_to_node - fetch the nasid that the passed struct pci_bus lives on.
+ * @bus: struct pci_bus pointer for a given PCI bus.
+ *
+ * casts bus->sysdata to struct bridge_controller and returns the nasid
+ * member that references the specific node this PCI bus lives on.
+ */
+int
+pcibus_to_node(struct pci_bus *bus)
 {
 	struct bridge_controller *bc = BRIDGE_CONTROLLER(bus);
 
@@ -226,5 +293,3 @@ int pcibus_to_node(struct pci_bus *bus)
 EXPORT_SYMBOL(pcibus_to_node);
 #endif /* CONFIG_NUMA */
 
-DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_SGI, PCI_DEVICE_ID_SGI_IOC3,
-	pci_fixup_ioc3);
-- 
2.11.1

[PATCH 11/12] MIPS: IP27: Update IRQ code to work with the new BRIDGE code

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Sparsely update just enough of the existing IP27 code so that the
machine will boot against the newer Xtalk/BRIDGE code.   A future
patch series will completely rewrite a majority of the existing IP27
code.

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
---
 arch/mips/pci/pci-bridge.c        | 11 +++++
 arch/mips/sgi-ip27/ip27-irq-pci.c | 72 +++++++++++++++++------------
 arch/mips/sgi-ip27/ip27-irq.c     |  5 ++
 3 files changed, 58 insertions(+), 30 deletions(-)

diff --git a/arch/mips/pci/pci-bridge.c b/arch/mips/pci/pci-bridge.c
index 28b0eb5b844e..9df13ce313b5 100644
--- a/arch/mips/pci/pci-bridge.c
+++ b/arch/mips/pci/pci-bridge.c
@@ -274,6 +274,17 @@ bridge_disable_swapping_dma(struct pci_dev *dev)
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SGI, PCI_DEVICE_ID_SGI_IOC3,
 			bridge_disable_swapping_dma);
 
+#if 0
+/*
+ * XXX: Onyx2 systems have a RAD1 on their IO6-G board, but the RAD1 driver
+ * and corresponding defines are part of the IP30 patchset, so we need to
+ * make this part conditional for now until the IP30 patches are merged
+ * upstream.
+ */
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SGI, PCI_DEVICE_ID_SGI_RAD1,
+			bridge_disable_swapping_dma);
+#endif
+
 /* XXX: Temporarily defined here until the IP27 "mega update". */
 #ifdef CONFIG_NUMA
 /**
diff --git a/arch/mips/sgi-ip27/ip27-irq-pci.c b/arch/mips/sgi-ip27/ip27-irq-pci.c
index 2a1c40784bd9..957b689b35f4 100644
--- a/arch/mips/sgi-ip27/ip27-irq-pci.c
+++ b/arch/mips/sgi-ip27/ip27-irq-pci.c
@@ -134,45 +134,56 @@ static int intr_disconnect_level(int cpu, int bit)
 /* Startup one of the (PCI ...) IRQs routes over a bridge.  */
 static unsigned int startup_bridge_irq(struct irq_data *d)
 {
+	u32 slot, reg;
 	struct bridge_controller *bc;
-	bridgereg_t device;
-	bridge_t *bridge;
-	int pin, swlevel;
+	int swlevel;
 	cpuid_t cpu;
 
-	pin = SLOT_FROM_PCI_IRQ(d->irq);
+	slot = SLOT_FROM_PCI_IRQ(d->irq);
 	bc = IRQ_TO_BRIDGE(d->irq);
-	bridge = bc->base;
 
-	pr_debug("bridge_startup(): irq= 0x%x  pin=%d\n", d->irq, pin);
+	pr_debug("bridge_startup(): irq= 0x%x  pin=%d\n", d->irq, slot);
 	/*
 	 * "map" irq to a swlevel greater than 6 since the first 6 bits
 	 * of INT_PEND0 are taken
 	 */
 	swlevel = find_level(&cpu, d->irq);
-	bridge->b_int_addr[pin].addr = (0x20000 | swlevel | (bc->nasid << 8));
-	bridge->b_int_enable |= (1 << pin);
-	bridge->b_int_enable |= 0x7ffffe00;	/* more stuff in int_enable */
 
 	/*
-	 * Enable sending of an interrupt clear packt to the hub on a high to
-	 * low transition of the interrupt pin.
+	 * Handle BRIDGE IRQs.
 	 *
-	 * IRIX sets additional bits in the address which are documented as
-	 * reserved in the bridge docs.
-	 */
-	bridge->b_int_mode |= (1UL << pin);
-
-	/*
-	 * We assume the bridge to have a 1:1 mapping between devices
-	 * (slots) and intr pins.
+	 * b_int_addr(slot) - Points to the HUB that this BRIDGE
+	 *	is assigned to, so that BRIDGE can frob the PI_INT_PEND_MOD
+	 *	register directly, without having to know which nasid the
+	 *	specific HUB is on.
+	 *
+	 * b_int_mode - Enable the sending of an interrupt clear
+	 *	packet to the HUB on a high-to-low transition of the
+	 *	interrupt pin.
+	 *
+	 * b_int_device - We assume the bridge to have a 1:1 mapping
+	 *	between devices (slots) and interrupt numbers.
+	 *
+	 * XXX: Replace the magic values with readable macros at some point.
 	 */
-	device = bridge->b_int_device;
-	device &= ~(7 << (pin*3));
-	device |= (pin << (pin*3));
-	bridge->b_int_device = device;
-
-	bridge->b_wid_tflush;
+	/* b_int_addr */
+	bridge_write_reg((0x20000 | swlevel | (bc->nasid << 8)), bc,
+			 b_int_addr(slot));
+	/* b_int_enable */
+	reg = bridge_read_reg(bc, b_int_enable);
+	bridge_write_reg((reg | BIT(slot) | 0x7ffffe00), bc, b_int_enable);
+
+	/* b_int_mode */
+	reg = bridge_read_reg(bc, b_int_mode);
+	bridge_write_reg((reg | BIT(slot)), bc, b_int_mode);
+
+	/* b_int_device */
+	reg = bridge_read_reg(bc, b_int_device);
+	reg &= ~BRIDGE_INT_DEV_MASK(slot);
+	reg |= BRIDGE_INT_DEV_SET(slot, slot);
+	bridge_write_reg(reg, bc, b_int_device);
+	/* flush */
+	BRIDGE_FLUSH(bc);
 
 	intr_connect_level(cpu, swlevel);
 
@@ -182,13 +193,13 @@ static unsigned int startup_bridge_irq(struct irq_data *d)
 /* Shutdown one of the (PCI ...) IRQs routes over a bridge.  */
 static void shutdown_bridge_irq(struct irq_data *d)
 {
+	u32 slot, reg;
 	struct bridge_controller *bc = IRQ_TO_BRIDGE(d->irq);
-	bridge_t *bridge = bc->base;
-	int pin, swlevel;
+	int swlevel;
 	cpuid_t cpu;
 
 	pr_debug("bridge_shutdown: irq 0x%x\n", d->irq);
-	pin = SLOT_FROM_PCI_IRQ(d->irq);
+	slot = SLOT_FROM_PCI_IRQ(d->irq);
 
 	/*
 	 * map irq to a swlevel greater than 6 since the first 6 bits
@@ -197,8 +208,9 @@ static void shutdown_bridge_irq(struct irq_data *d)
 	swlevel = find_level(&cpu, d->irq);
 	intr_disconnect_level(cpu, swlevel);
 
-	bridge->b_int_enable &= ~(1 << pin);
-	bridge->b_wid_tflush;
+	reg = bridge_read_reg(bc, b_int_enable);
+	bridge_write_reg((reg & ~(BIT(slot))), bc, b_int_enable);
+	BRIDGE_FLUSH(bc);
 }
 
 static inline void enable_bridge_irq(struct irq_data *d)
diff --git a/arch/mips/sgi-ip27/ip27-irq.c b/arch/mips/sgi-ip27/ip27-irq.c
index 16ec4e12daa3..812b23c126ef 100644
--- a/arch/mips/sgi-ip27/ip27-irq.c
+++ b/arch/mips/sgi-ip27/ip27-irq.c
@@ -35,6 +35,11 @@
 #include <asm/sn/hub.h>
 #include <asm/sn/intr.h>
 
+/* XXX: Part of the IP27 "mega update" */
+#include <asm/mach-ip27/pcibr.h>
+struct bridge_controller *irq_to_bridge[PCIBR_MAX_BUS_X_DEV];
+u32 irq_to_slot[PCIBR_MAX_BUS_X_DEV];
+
 /*
  * Linux has a controller-independent x86 interrupt architecture.
  * every controller has a 'controller-template', that is used
-- 
2.11.1

[PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Joshua Kinard]

From: Joshua Kinard <kumba@gentoo.org>

Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY
will now explicitly claim PCI resources instead of skipping the sizing
of the bridges and assigning resources.  This is okay for IP30's PCI
code, which doesn't use physical address space to access I/O resources.

However, IP27 is completely different in this regard.  Instead of using
ioremapped addresses for I/O, IP27 has a dedicated address range,
0x92xxxxxxxxxxxxxx, that is used for all I/O access.  Since this is
uncached physical address space, the generic MIPS PCI code will not
probe it correctly and thus, the original behavior of PCI_PROBE_ONLY
needs to be restored only for the IP27 platform to bypass this logic
and have working PCI, at least for the IO6/IO6G board that houses the
base devices, until a better solution is found.

Fixes: 046136170a56 ("MIPS/PCI: Claim bus resources on PCI_PROBE_ONLY set-ups")
Signed-off-by: Joshua Kinard <kumba@gentoo.org>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
---
 arch/mips/pci/pci-bridge.c | 15 +++++++++++++++
 arch/mips/pci/pci-legacy.c | 15 +++++++++++++++
 2 files changed, 30 insertions(+)

diff --git a/arch/mips/pci/pci-bridge.c b/arch/mips/pci/pci-bridge.c
index 9df13ce313b5..af7073dba36b 100644
--- a/arch/mips/pci/pci-bridge.c
+++ b/arch/mips/pci/pci-bridge.c
@@ -62,6 +62,21 @@ bridge_probe(nasid_t nasid, int widget_id, int masterwid)
 	unsigned long offset = NODE_OFFSET(nasid);
 	struct bridge_controller *bc;
 
+#ifdef CONFIG_SGI_IP27
+	/*
+	 * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY
+	 * will now explicitly claim PCI resources instead of skipping the
+	 * sizing of the bridges and assigning resources.  This is okay for
+	 * the IP30's PCI code, which uses normal, ioremapped addresses to
+	 * do I/O.  IP27, however, is different and uses a hardware-specific
+	 * address range of 0x92xxxxxxxxxxxxxx for all I/O access.  As such,
+	 * the generic MIPS PCI code will not probe correctly and thus make
+	 * PCI on IP27 completely unusable.  Thus, we must restore the
+	 * original logic only for IP27 until a better solution can be found.
+	 */
+	pci_set_flags(PCI_PROBE_ONLY);
+#endif
+
 	/* XXX: Temporary until the IP27 "mega update". */
 	bc = &bridges[num_bridges];
 	if (!num_bridges)
diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c
index 68268bbb15b8..5590af4f367f 100644
--- a/arch/mips/pci/pci-legacy.c
+++ b/arch/mips/pci/pci-legacy.c
@@ -107,6 +107,20 @@ static void pcibios_scanbus(struct pci_controller *hose)
 		need_domain_info = 1;
 	}
 
+#ifdef CONFIG_SGI_IP27
+	/*
+	 * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY
+	 * will now explicitly claim PCI resources instead of skipping the
+	 * sizing of the bridges and assigning resources.  This is okay for
+	 * the IP30's PCI code, which uses normal, ioremapped addresses to
+	 * do I/O.  IP27, however, is different and uses a hardware-specific
+	 * address range of 0x92xxxxxxxxxxxxxx for all I/O access.  As such,
+	 * the generic MIPS PCI code will not probe correctly and thus make
+	 * PCI on IP27 completely unusable.  Thus, we must restore the
+	 * original logic only for IP27 until a better solution can be found.
+	 */
+	if (!pci_has_flag(PCI_PROBE_ONLY)) {
+#else
 	/*
 	 * We insert PCI resources into the iomem_resource and
 	 * ioport_resource trees in either pci_bus_claim_resources()
@@ -115,6 +129,7 @@ static void pcibios_scanbus(struct pci_controller *hose)
 	if (pci_has_flag(PCI_PROBE_ONLY)) {
 		pci_bus_claim_resources(bus);
 	} else {
+#endif
 		pci_bus_size_bridges(bus);
 		pci_bus_assign_resources(bus);
 	}
-- 
2.11.1

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Bjorn Helgaas]

Hi Joshua,

First of all, apologies for breaking IP27 and the inconvenience this
caused you.  And thanks a lot for tracking down the problem and
proposing a solution!

On Tue, Feb 7, 2017 at 12:13 AM, Joshua Kinard <kumba@gentoo.org> wrote:
> From: Joshua Kinard <kumba@gentoo.org>
>
> Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY
> will now explicitly claim PCI resources instead of skipping the sizing
> of the bridges and assigning resources.  This is okay for IP30's PCI
> code, which doesn't use physical address space to access I/O resources.
>
> However, IP27 is completely different in this regard.  Instead of using
> ioremapped addresses for I/O, IP27 has a dedicated address range,
> 0x92xxxxxxxxxxxxxx, that is used for all I/O access.  Since this is
> uncached physical address space, the generic MIPS PCI code will not
> probe it correctly and thus, the original behavior of PCI_PROBE_ONLY
> needs to be restored only for the IP27 platform to bypass this logic
> and have working PCI, at least for the IO6/IO6G board that houses the
> base devices, until a better solution is found.

It sounds like there's something different about how ioremap() works
on these platforms and PCI probing is tripping over that.  I'd really
like to understand more about this difference to see if we can
converge that instead of adding back the PCI_PROBE_ONLY usage.

Drivers shouldn't know whether they're running on IP27 or IP30, and
they should be using ioremap() in both cases.  Does ioremap() work
differently on IP27 and IP30?  Does this have something to do with
plat_ioremap() or fixup_bigphys_addr()?

Is there any chance you can collect complete dmesg logs and
/proc/iomem contents from IP27 and IP30?  Maybe "lspci -vv" output,
too?  I'm not sure where to look to understand the ioremap() behavior.

What exactly is the PCI probe failure without this patch?  If you have
a console log (with "ignore_loglevel") it might have a clue, too.

> Fixes: 046136170a56 ("MIPS/PCI: Claim bus resources on PCI_PROBE_ONLY set-ups")
> Signed-off-by: Joshua Kinard <kumba@gentoo.org>
> Cc: Bjorn Helgaas <bhelgaas@google.com>
> Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> ---
>  arch/mips/pci/pci-bridge.c | 15 +++++++++++++++
>  arch/mips/pci/pci-legacy.c | 15 +++++++++++++++
>  2 files changed, 30 insertions(+)
>
> diff --git a/arch/mips/pci/pci-bridge.c b/arch/mips/pci/pci-bridge.c
> index 9df13ce313b5..af7073dba36b 100644
> --- a/arch/mips/pci/pci-bridge.c
> +++ b/arch/mips/pci/pci-bridge.c
> @@ -62,6 +62,21 @@ bridge_probe(nasid_t nasid, int widget_id, int masterwid)
>         unsigned long offset = NODE_OFFSET(nasid);
>         struct bridge_controller *bc;
>
> +#ifdef CONFIG_SGI_IP27

I don't know how MIPS multi-platform support works.  If you enable
CONFIG_SGI_IP27, does that mean the resulting kernel will *only* run
on IP27?  Or can you enable several platforms, e.g., SGI_IP22,
SGI_IP27, SGI_IP28, SGI_IP32, etc., and end up with a kernel that will
boot on any of those platforms?  From Kconfig, it looks like these
options are not mutually exclusive, so my guess is maybe the latter?

If so, I would think whatever we do should be based on a run-time test
for SGI_IP27 instead of a compile-time test.

> +       /*
> +        * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY
> +        * will now explicitly claim PCI resources instead of skipping the
> +        * sizing of the bridges and assigning resources.  This is okay for
> +        * the IP30's PCI code, which uses normal, ioremapped addresses to
> +        * do I/O.  IP27, however, is different and uses a hardware-specific
> +        * address range of 0x92xxxxxxxxxxxxxx for all I/O access.  As such,
> +        * the generic MIPS PCI code will not probe correctly and thus make
> +        * PCI on IP27 completely unusable.  Thus, we must restore the
> +        * original logic only for IP27 until a better solution can be found.
> +        */
> +       pci_set_flags(PCI_PROBE_ONLY);
> +#endif
> +
>         /* XXX: Temporary until the IP27 "mega update". */
>         bc = &bridges[num_bridges];
>         if (!num_bridges)
> diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c
> index 68268bbb15b8..5590af4f367f 100644
> --- a/arch/mips/pci/pci-legacy.c
> +++ b/arch/mips/pci/pci-legacy.c
> @@ -107,6 +107,20 @@ static void pcibios_scanbus(struct pci_controller *hose)
>                 need_domain_info = 1;
>         }
>
> +#ifdef CONFIG_SGI_IP27
> +       /*
> +        * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY
> +        * will now explicitly claim PCI resources instead of skipping the
> +        * sizing of the bridges and assigning resources.  This is okay for
> +        * the IP30's PCI code, which uses normal, ioremapped addresses to
> +        * do I/O.  IP27, however, is different and uses a hardware-specific
> +        * address range of 0x92xxxxxxxxxxxxxx for all I/O access.  As such,
> +        * the generic MIPS PCI code will not probe correctly and thus make
> +        * PCI on IP27 completely unusable.  Thus, we must restore the
> +        * original logic only for IP27 until a better solution can be found.
> +        */
> +       if (!pci_has_flag(PCI_PROBE_ONLY)) {
> +#else
>         /*
>          * We insert PCI resources into the iomem_resource and
>          * ioport_resource trees in either pci_bus_claim_resources()
> @@ -115,6 +129,7 @@ static void pcibios_scanbus(struct pci_controller *hose)
>         if (pci_has_flag(PCI_PROBE_ONLY)) {
>                 pci_bus_claim_resources(bus);
>         } else {
> +#endif
>                 pci_bus_size_bridges(bus);
>                 pci_bus_assign_resources(bus);
>         }
> --
> 2.11.1
>

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Joshua Kinard]

[-- Attachment #1: Type: text/plain, Size: 12598 bytes --]

On 02/07/2017 13:29, Bjorn Helgaas wrote:
> Hi Joshua,
> 
> First of all, apologies for breaking IP27 and the inconvenience this
> caused you.  And thanks a lot for tracking down the problem and
> proposing a solution!

Eh, don't worry, IP27's been broken and unbroken over the years.  These systems
are becoming difficult to find intact, so these breaks tend to go unnoticed.


> On Tue, Feb 7, 2017 at 12:13 AM, Joshua Kinard <kumba@gentoo.org> wrote:
>> From: Joshua Kinard <kumba@gentoo.org>
>>
>> Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY
>> will now explicitly claim PCI resources instead of skipping the sizing
>> of the bridges and assigning resources.  This is okay for IP30's PCI
>> code, which doesn't use physical address space to access I/O resources.
>>
>> However, IP27 is completely different in this regard.  Instead of using
>> ioremapped addresses for I/O, IP27 has a dedicated address range,
>> 0x92xxxxxxxxxxxxxx, that is used for all I/O access.  Since this is
>> uncached physical address space, the generic MIPS PCI code will not
>> probe it correctly and thus, the original behavior of PCI_PROBE_ONLY
>> needs to be restored only for the IP27 platform to bypass this logic
>> and have working PCI, at least for the IO6/IO6G board that houses the
>> base devices, until a better solution is found.
> 
> It sounds like there's something different about how ioremap() works
> on these platforms and PCI probing is tripping over that.  I'd really
> like to understand more about this difference to see if we can
> converge that instead of adding back the PCI_PROBE_ONLY usage.

I'd need to go and dig around in the IP27 headers again for this machine to see
what ioremap() is actually doing, but I *think* it returns uncached physical
addresses in most instances because of a special feature of the CPU, the
R10000-family, which makes uncached access very fast.  I think only the IP27
platform uses this capability.  Other R1x0-based systems don't (although, I
might be wrong about IP27's successor in IP35).


> Drivers shouldn't know whether they're running on IP27 or IP30, and
> they should be using ioremap() in both cases.  Does ioremap() work
> differently on IP27 and IP30?  Does this have something to do with
> plat_ioremap() or fixup_bigphys_addr()?

I believe most drivers won't know about this change.  The patch I proposed
modifies two files very specific to MIPS and SGI systems, one being the PCI
module for "legacy" MIPS systems, and the newly-proposed generic BRIDGE driver
(created by an earlier patch in this patch series).

The BRIDGE ASIC (and its cousin, the XBRIDGE) are only used on SGI systems and
they provide a translation layer from Crosstalk address space (Xtalk, the
system interconnect bus) to PCI devices by translating PCI ops into something
understood by the rest of the system.  BRIDGE is the only thing that needs to
get setup correctly, and then, I think, other drivers should be unaware of the
magic going on behind the scenes.

That said, it's still a hack that I proposed.  IP27 has long had issues with
PCI because it uses addresses differently.  I'm no expert in these systems --
just a hobby of mine I do in my free time -- but I'll refer you to these two
documents, which are archive copies since SGI retired Techpubs last year:

Origin™ and Onyx2™ Theory of Operations Manual:
http://csweb.cs.wfu.edu/~torgerse/Kokua/SGI/007-3439-002/sgi_html/index.html

Origin™ and Onyx2™ Programmer's Reference Manual
http://csweb.cs.wfu.edu/~torgerse/Kokua/SGI/007-3410-001/sgi_html/ch02.html#id5437809

The whole setup is extremely modular, so each node gets a static, fixed portion
of physical address space, regardless if that node is actually present or not.
By setting various bits in the upper part of a memory address, you can do
various things on specific nodes.  See Chapter 1 in the programmer's manual for
the details.

The address space in question here, 0x9200000000000000, is how you tell the HUB
(system controller for each node board) to access I/O space and talk to the
device peripherals.  Since this is physical address space, there's no need to
ioremap, so I think deep in the IP27 headers, ioremap just falls through and
returns one of these physical address, then you add in the needed offsets (node
number, crosstalk port, and PCI address) to get to the device you're trying to
reach.

IP30, SGI Octane, is similar in that it's like a single-node IP27 system.  It
has a HEART as a system controller instead of a HUB, but also uses Crosstalk
devices.  Documentation for the system doesn't really exist -- work on it has
been done via manual reverse engineering by people way better than I at this.
I've just been maintaining the code in external patches over the years.

I think the addresses on that platform *might* also be physical addresses, but
with no documentation on HEART, it's unknown if HEART treats certain address
spaces in special ways like HUB does.


> Is there any chance you can collect complete dmesg logs and
> /proc/iomem contents from IP27 and IP30?  Maybe "lspci -vv" output,
> too?  I'm not sure where to look to understand the ioremap() behavior.

Due to the size, I'll attach the dmesg and lspci outputs from both platforms as
text files, and inline /proc/iomem.  There are three dmesg files for IP27.  One
is a normal boot with enough patches to make PCI work.  Another is without the
PCI_PROBE_ONLY flag and my hack, so it will try to size/assign the resources,
and the last is with PCI_PROBE_ONLY and my hack, so it will try to "claim" the
resources only.  For the last two, each results in different error paths.

A note about IP30, since that's part of an external patch set, the dmesg output
is a little different, as it is using the newer BRIDGE/Xtalk code I just sent
in as patches.  IP27's dmesg output won't be using my full patchset in this
test, so it'll more closely resemble what you'd get out of the existing
mainline code.

Also, IP30 doesn't use the PCI_PROBE_ONLY thing anymore.  This causes
/proc/iomem to be far more detailed than it used to be.

IP30's /proc/iomem:
    00000000-00003fff : reserved
    1d200000-1d9fffff : Bridge MEM
      d080000000-d080003fff : 0001:00:03.0
      1d204000-1d204fff : 0001:00:02.0
      1d205000-1d205fff : 0001:00:02.1
      1d206000-1d206fff : 0001:00:02.2
      1d207000-1d2070ff : 0001:00:02.3
      1d207100-1d20717f : 0001:00:01.0
    1f200000-1f9fffff : Bridge MEM
      f080100000-f0801fffff : 0000:00:02.0
      f080010000-f08001ffff : 0000:00:00.0
      f080030000-f08003ffff : 0000:00:01.0
      f080000000-f080000fff : 0000:00:00.0
      f080020000-f080020fff : 0000:00:01.0
    20004000-209b8fff : reserved
      20004000-206acb13 : Kernel code
      206acb14-208affff : Kernel data
    209b9000-20efffff : System RAM
    20f00000-20ffffff : System RAM
    21000000-9fffffff : System RAM
    f080100000-f0801fffff : ioc3

IP27's /proc/iomem (from the working boot):
    0f600000-0f6fffff : ioc3
      0f680000-0f687fff : rtc-m48t35
    0fa00000-0fafffff : ioc3
    920000000b200000-920000000b9fffff : Bridge MEM
    920000000c200000-920000000c9fffff : Bridge MEM
    920000000f200000-920000000f9fffff : Bridge MEM
      920000000f620170-920000000f620177 : serial
      920000000f620178-920000000f62017f : serial
    920000000fa20170-920000000fa20177 : serial
    920000000fa20178-920000000fa2017f : serial


> What exactly is the PCI probe failure without this patch?  If you have
> a console log (with "ignore_loglevel") it might have a clue, too.
> 
>> Fixes: 046136170a56 ("MIPS/PCI: Claim bus resources on PCI_PROBE_ONLY set-ups")
>> Signed-off-by: Joshua Kinard <kumba@gentoo.org>
>> Cc: Bjorn Helgaas <bhelgaas@google.com>
>> Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
>> Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
>> ---
>>  arch/mips/pci/pci-bridge.c | 15 +++++++++++++++
>>  arch/mips/pci/pci-legacy.c | 15 +++++++++++++++
>>  2 files changed, 30 insertions(+)
>>
>> diff --git a/arch/mips/pci/pci-bridge.c b/arch/mips/pci/pci-bridge.c
>> index 9df13ce313b5..af7073dba36b 100644
>> --- a/arch/mips/pci/pci-bridge.c
>> +++ b/arch/mips/pci/pci-bridge.c
>> @@ -62,6 +62,21 @@ bridge_probe(nasid_t nasid, int widget_id, int masterwid)
>>         unsigned long offset = NODE_OFFSET(nasid);
>>         struct bridge_controller *bc;
>>
>> +#ifdef CONFIG_SGI_IP27
> 
> I don't know how MIPS multi-platform support works.  If you enable
> CONFIG_SGI_IP27, does that mean the resulting kernel will *only* run
> on IP27?  Or can you enable several platforms, e.g., SGI_IP22,
> SGI_IP27, SGI_IP28, SGI_IP32, etc., and end up with a kernel that will
> boot on any of those platforms?  From Kconfig, it looks like these
> options are not mutually exclusive, so my guess is maybe the latter?
> 
> If so, I would think whatever we do should be based on a run-time test
> for SGI_IP27 instead of a compile-time test.

Kernels are platform-specific for the SGI machines.  So you need a different
kernel for IP22, IP27, IP30, IP32, etc.  Sometimes, different CPU's in a system
needs a different kernel as well (e.g., IP32 w/ R5000 versus IP32 w/ RM7000,
different kernels for each).

The one *slight* exception is Indy and Indigo2, where one Indigo2 model is just
a larger Indy (both are IP22).  There's two other Indigo2 variants out there
though, the rare Indigo2 Power (IP26) and the Indigo2 Impact (IP28).  These
classify as distinctly-different systems, mostly, and thus need their own
kernels (not that anyone's written code for IP26 yet, but I digress).

Other MIPS platforms may be a bit different.  I can only speak for the SGI
platforms right now.  You can see where all the fun is at...


>> +       /*
>> +        * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY
>> +        * will now explicitly claim PCI resources instead of skipping the
>> +        * sizing of the bridges and assigning resources.  This is okay for
>> +        * the IP30's PCI code, which uses normal, ioremapped addresses to
>> +        * do I/O.  IP27, however, is different and uses a hardware-specific
>> +        * address range of 0x92xxxxxxxxxxxxxx for all I/O access.  As such,
>> +        * the generic MIPS PCI code will not probe correctly and thus make
>> +        * PCI on IP27 completely unusable.  Thus, we must restore the
>> +        * original logic only for IP27 until a better solution can be found.
>> +        */
>> +       pci_set_flags(PCI_PROBE_ONLY);
>> +#endif
>> +
>>         /* XXX: Temporary until the IP27 "mega update". */
>>         bc = &bridges[num_bridges];
>>         if (!num_bridges)
>> diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c
>> index 68268bbb15b8..5590af4f367f 100644
>> --- a/arch/mips/pci/pci-legacy.c
>> +++ b/arch/mips/pci/pci-legacy.c
>> @@ -107,6 +107,20 @@ static void pcibios_scanbus(struct pci_controller *hose)
>>                 need_domain_info = 1;
>>         }
>>
>> +#ifdef CONFIG_SGI_IP27
>> +       /*
>> +        * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY
>> +        * will now explicitly claim PCI resources instead of skipping the
>> +        * sizing of the bridges and assigning resources.  This is okay for
>> +        * the IP30's PCI code, which uses normal, ioremapped addresses to
>> +        * do I/O.  IP27, however, is different and uses a hardware-specific
>> +        * address range of 0x92xxxxxxxxxxxxxx for all I/O access.  As such,
>> +        * the generic MIPS PCI code will not probe correctly and thus make
>> +        * PCI on IP27 completely unusable.  Thus, we must restore the
>> +        * original logic only for IP27 until a better solution can be found.
>> +        */
>> +       if (!pci_has_flag(PCI_PROBE_ONLY)) {
>> +#else
>>         /*
>>          * We insert PCI resources into the iomem_resource and
>>          * ioport_resource trees in either pci_bus_claim_resources()
>> @@ -115,6 +129,7 @@ static void pcibios_scanbus(struct pci_controller *hose)
>>         if (pci_has_flag(PCI_PROBE_ONLY)) {
>>                 pci_bus_claim_resources(bus);
>>         } else {
>> +#endif
>>                 pci_bus_size_bridges(bus);
>>                 pci_bus_assign_resources(bus);
>>         }
>> --
>> 2.11.1
>>
> 


-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

[-- Attachment #2: ip30-dmesg-20170208.txt --]
[-- Type: text/plain, Size: 27352 bytes --]

[    0.000000] Linux version 4.9.8-mipsgit-20161216 (root@helcaraxe) (gcc version 6.3.0 (Gentoo Hardened 6.3.0 p1.0) ) #1 SMP Sat Feb 4 21:01:37 EST 2017
[    0.000000] ARCH: SGI-IP30
[    0.000000] PROMLIB: ARC firmware Version 64 Revision 0
[    0.000000] bootconsole [early_impact0] enabled
[    0.000000] CPU0 revision is: 00000f24 (R14000)
[    0.000000] FPU revision is: 00000900
[    0.000000] Checking for the multiply/shift bug...
[    0.000000] no.
[    0.000000] Checking for the daddiu bug...
[    0.000000] no.
[    0.000000] Detected 2048MB of physical memory.
[    0.000000] IP30: CPU0: 600 MHz CPU detected.
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 0000000000004000 @ 0000000000000000 (reserved)
[    0.000000]  memory: 00000000009b5000 @ 0000000020004000 (reserved)
[    0.000000]  memory: 0000000000547000 @ 00000000209b9000 (usable)
[    0.000000]  memory: 0000000000100000 @ 0000000020f00000 (ROM data)
[    0.000000]  memory: 000000007f000000 @ 0000000021000000 (usable)
[    0.000000]  memory: 00000000008b0000 @ 0000000020000000 (usable)
[    0.000000] cma: Reserved 512 MiB at 0x0000000080000000
[    0.000000] IP30: Slot: 0, PrID: 00000f24, PhyID: 0, VirtID: 0
[    0.000000] IP30: Slot: 1, PrID: 00000f24, PhyID: 1, VirtID: 1
[    0.000000] IP30: Detected 2 CPU(s) present.
[    0.000000] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    0.000000] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    0.000000] Unified secondary cache 2048kB 2-way, linesize 128 bytes.
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000020000000-0x00000000ffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000020000000-0x00000000209affff]
[    0.000000]   node   0: [mem 0x00000000209c0000-0x000000009fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000020000000-0x000000009fffffff]
[    0.000000] On node 0 totalpages: 32767
[    0.000000] free_area_init_node: node 0, pgdat a80000002086a300, node_mem_map a8000000209d0300
[    0.000000]   DMA32 zone: 32 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 32767 pages, LIFO batch:1
[    0.000000] percpu: Embedded 1 pages/cpu @a800000020c00000 s34336 r0 d31200 u65536
[    0.000000] pcpu-alloc: s34336 r0 d31200 u65536 alloc=1*65536
[    0.000000] pcpu-alloc: [0] 0 [0] 1
[    0.000000] Built 1 zonelists in Zone order, mobility grouping off.  Total pages: 32735
[    0.000000] Kernel command line: root=xio(0)pci(15)scsi(0)disk(1)rdisk(0)partition(0) console=tty0 root=/dev/md0 consoleblank=0
[    0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes)
[    0.000000] Dentry cache hash table entries: 262144 (order: 5, 2097152 bytes)
[    0.000000] Inode-cache hash table entries: 131072 (order: 4, 1048576 bytes)
[    0.000000] Memory: 1556288K/2097088K available (6817K kernel code, 687K rwdata, 1348K rodata, 320K init, 739K bss, 16512K reserved, 524288K cma-reserved)
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000] NR_IRQS:128
[    0.000000] IP30: HEART interrupt controller initialized.
[    0.000000] clocksource: HEART: mask: 0xfffffffffffff max_cycles: 0x2e2049cda, max_idle_ns: 440795202628 ns
[    0.000006] sched_clock: 52 bits at 12MHz, resolution 80ns, wraps every 4398046511080ns
[    0.000550] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6369645396 ns
[    0.001086] sched_clock: 32 bits at 300MHz, resolution 3ns, wraps every 7156904958ns
[    0.002520] Console: colour dummy device 80x25
[    0.002875] console [tty0] enabled
[    0.003185] bootconsole [early_impact0] disabled
[    0.003573] Calibrating delay loop... 898.66 BogoMIPS (lpj=4493312)
[    0.090689] pid_max: default: 32768 minimum: 301
[    0.091554] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes)
[    0.091573] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes)
[    0.093852] Checking for the daddi bug...
[    0.093871] no.
[    6.760385] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    6.760397] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    6.760402] Unified secondary cache 2048kB 2-way, linesize 128 bytes.
[   14.264191] IP30: CPU1: 600 MHz CPU detected.
[   14.264235] CPU1 revision is: 00000f24 (R14000)
[   14.264238] FPU revision is: 00000900
[    0.230963] Synchronize counters for CPU 1:
[    0.230963] done.
[    0.231234] Brought up 2 CPUs
[    0.232749] devtmpfs: initialized
[    0.233747] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.236205] xor: measuring software checksum speed
[    0.330951]    8regs     :   851.200 MB/sec
[    0.431036]    8regs_prefetch:   806.400 MB/sec
[    0.531198]    32regs    :   729.600 MB/sec
[    0.631269]    32regs_prefetch:   748.800 MB/sec
[    0.631299] xor: using function: 8regs (851.200 MB/sec)
[    0.632356] NET: Registered protocol family 16
[    0.671300] cpuidle: using governor ladder
[    0.711323] cpuidle: using governor menu
[    0.711467] xtalk:n0/f bridge widget (rev D) registered as a platform device.
[    0.711544] xtalk:n0/d bridge widget (rev C) registered as a platform device.
[    0.711617] xtalk:n0/c impact widget (rev B) registered as a platform device.
[    0.711690] xtalk:n0/8 heart widget (rev F) registered as a platform device.
[    0.961640] raid6: int64x1  gen()   323 MB/s
[    1.131840] raid6: int64x1  xor()   120 MB/s
[    1.301753] raid6: int64x2  gen()   468 MB/s
[    1.341955] random: fast init done
[    1.472023] raid6: int64x2  xor()   161 MB/s
[    1.642080] raid6: int64x4  gen()   416 MB/s
[    1.812194] raid6: int64x4  xor()   163 MB/s
[    1.982253] raid6: int64x8  gen()   342 MB/s
[    2.152536] raid6: int64x8  xor()   133 MB/s
[    2.152558] raid6: using algorithm int64x2 gen() 468 MB/s
[    2.152578] raid6: .... xor() 161 MB/s, rmw enabled
[    2.152597] raid6: using intx1 recovery algorithm
[    2.152988] SCSI subsystem initialized
[    2.153116] pps_core: LinuxPPS API ver. 1 registered
[    2.153138] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    2.153188] PTP clock support registered
[    2.154071] clocksource: Switched to clocksource HEART
[    2.154483] FS-Cache: Loaded
[    2.154723] CacheFiles: Loaded
[    2.162091] NET: Registered protocol family 2
[    2.162753] TCP established hash table entries: 16384 (order: 1, 131072 bytes)
[    2.163396] TCP bind hash table entries: 16384 (order: 2, 262144 bytes)
[    2.164403] TCP: Hash tables configured (established 16384 bind 16384)
[    2.164619] UDP hash table entries: 2048 (order: 0, 65536 bytes)
[    2.164892] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes)
[    2.165412] NET: Registered protocol family 1
[    2.166001] RPC: Registered named UNIX socket transport module.
[    2.166027] RPC: Registered udp transport module.
[    2.166047] RPC: Registered tcp transport module.
[    2.166066] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    2.166093] PCI: CLS 0 bytes, default 128
[    2.166728] futex hash table entries: 512 (order: 0, 65536 bytes)
[    2.167250] workingset: timestamp_bits=46 max_order=15 bucket_order=0
[    2.167409] zbud: loaded
[    2.168725] DLM installed
[    2.168790] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    2.169364] NFS: Registering the id_resolver key type
[    2.169420] Key type id_resolver registered
[    2.169442] Key type id_legacy registered
[    2.169472] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    2.170214] efs: 1.0a - http://aeschi.ch.eu.org/efs/
[    2.170279] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[    2.181274] NET: Registered protocol family 38
[    2.181383] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    2.181422] io scheduler noop registered
[    2.181443] io scheduler deadline registered
[    2.181468] io scheduler bfq registered (default)
[    2.181489] BFQ I/O-scheduler: v8r7
[    2.181668] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    2.223223] Console: switching to colour frame buffer device 160x64
[    2.255766] fb0: ImpactSR 1RSS frame buffer device
[    2.317384] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    2.327988] loop: module loaded
[    2.328851] mousedev: PS/2 mouse device common for all mice
[    2.329536] md: raid6 personality registered for level 6
[    2.329752] md: raid5 personality registered for level 5
[    2.329961] md: raid4 personality registered for level 4
[    2.330336] PCI host bridge to bus 0000:00
[    2.330521] pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff]
[    2.330794] pci_bus 0000:00: root bus resource [io  0x1fa00000-0x1fbfffff]
[    2.331068] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.331330] pci 0000:00:00.0: [1077:1020] type 00 class 0x010000
[    2.331378] pci 0000:00:00.0: reg 0x10: [io  0x200000-0x2000ff]
[    2.331402] pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff]
[    2.331483] pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref]
[    2.331704] pci 0000:00:01.0: [1077:1020] type 00 class 0x010000
[    2.331738] pci 0000:00:01.0: reg 0x10: [io  0x400000-0x4000ff]
[    2.331761] pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff]
[    2.331843] pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref]
[    2.332030] pci 0000:00:02.0: [10a9:0003] type 00 class 0xff0000
[    2.332068] pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff]
[    2.332345] pci 0000:00:03.0: [10a9:0005] type 00 class 0x000000
[    2.332377] pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff]
[    2.332727] pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff]
[    2.333011] pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref]
[    2.333298] pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref]
[    2.333582] pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff]
[    2.333856] pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff]
[    2.334181] pci 0000:00:00.0: BAR 0: assigned [io  0x1fa00000-0x1fa000ff]
[    2.334456] pci 0000:00:01.0: BAR 0: assigned [io  0x1fa00400-0x1fa004ff]
[    2.334795] qla1280: QLA1040 found on PCI bus 0, dev 0
[    2.335016] ip30-bridge: 0000:00:00.0 Bar 0 with size 0x00000100 at bus 0x00000000 vma 0x000000f100000000 is Direct I/O.
[    2.335435] ip30-bridge: 0000:00:00.0 Bar 1 with size 0x00001000 at bus 0x00000000 vma 0x000000f080000000 is Direct 64-bit.
[    2.335862] ip30-bridge: 0000:00:00.0 Bar 6 with size 0x00010000 at bus 0x00010000 vma 0x000000f080010000 is Direct 64-bit.
[    2.336287] PCI: Enabling device 0000:00:00.0 (0006 -> 0007)
[    2.337044] scsi(0): Enabling vchannel on BRIDGE for SGI/MIPS
[    2.738704] random: crng init done
[    2.923416] scsi(0:0): Resetting SCSI BUS
[    6.004087] scsi host0: QLogic QLA1040 PCI to SCSI Host Adapter
                      Firmware version:  7.65.06, Driver version 3.27.1
[    6.010788] qla1280: QLA1040 found on PCI bus 0, dev 1
[    6.011039] ip30-bridge: 0000:00:01.0 Bar 0 with size 0x00000100 at bus 0x00000100 vma 0x000000f100000100 is Direct I/O.
[    6.011464] ip30-bridge: 0000:00:01.0 Bar 1 with size 0x00001000 at bus 0x00020000 vma 0x000000f080020000 is Direct 64-bit.
[    6.011895] ip30-bridge: 0000:00:01.0 Bar 6 with size 0x00010000 at bus 0x00030000 vma 0x000000f080030000 is Direct 64-bit.
[    6.012324] PCI: Enabling device 0000:00:01.0 (0006 -> 0007)
[   13.185643] scsi 0:0:1:0: Direct-Access     IBM-ESXS ST973401LC    FN B41D PQ: 0 ANSI: 4
[   13.186771] scsi(0:0:1:0):
[   13.186889]  Sync: period 10, offset 12
[   13.187053] , Wide
[   13.187089] , Tagged queuing: depth 31

[   13.962105] scsi 0:0:2:0: Direct-Access     IBM-ESXS ST973401LC    FN B41D PQ: 0 ANSI: 4
[   13.963095] scsi(0:0:2:0):
[   13.963212]  Sync: period 10, offset 12
[   13.972934] , Wide
[   13.973098] , Tagged queuing: depth 31

[   14.175153] scsi 0:0:3:0: Direct-Access     IBM-ESXS ST973401LC    FN B41D PQ: 0 ANSI: 4
[   14.186715] scsi(0:0:3:0):
[   14.186837]  Sync: period 10, offset 12
[   14.197546] , Wide
[   14.197712] , Tagged queuing: depth 31

[   14.511621] scsi(1:0): Resetting SCSI BUS
[   19.044315] scsi host1: QLogic QLA1040 PCI to SCSI Host Adapter
                      Firmware version:  7.65.06, Driver version 3.27.1
[   19.070925] ip30-bridge: 0000:00:02.0 Bar 0 with size 0x00100000 at bus 0x00100000 vma 0x000000f080100000 is Direct 64-bit.
[   21.194039] ioc3: part: [030-0891-003], serial: [KBB731] => class IP30 System Board
[   21.207013] ioc3-eth: Ethernet address is 08:00:69:0e:11:0c.
[   21.218711] IOC3 0000:00:02.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1
[   21.229449] eth0: Using PHY 1, vendor 0x15f42, model 2, rev 3.
[   21.240215] eth0: IOC3 SSRAM has 128 kbyte.
[   21.251666] rtc-ds1685 rtc-ds1685: rtc core: registered rtc-ds1685 as rtc0
[   21.287565] 0000:00:02.0: ttyS0 at IOC3 0xf080120178 (irq = 0, base_baud = 458333) is a 16550A
[   21.319087] 0000:00:02.0: ttyS1 at IOC3 0xf080120170 (irq = 0, base_baud = 458333) is a 16550A
[   21.330079] IOC3 Master Driver loaded for 0000:00:02.0
[   21.340951] PCI host bridge to bus 0001:00
[   21.351657] pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff]
[   21.362655] pci_bus 0001:00: root bus resource [io  0x1da00000-0x1dbfffff]
[   21.373502] pci_bus 0001:00: root bus resource [bus 01-ff]
[   21.384204] pci 0001:00:01.0: [11fe:080e] type 00 class 0x078000
[   21.384251] pci 0001:00:01.0: reg 0x10: [mem 0x00200000-0x0020007f]
[   21.384274] pci 0001:00:01.0: reg 0x14: [io  0x200000-0x20007f]
[   21.384297] pci 0001:00:01.0: reg 0x18: [io  0x204000-0x2040ff]
[   21.384578] pci 0001:00:02.0: [10b9:5237] type 00 class 0x0c0310
[   21.384618] pci 0001:00:02.0: reg 0x10: [mem 0x00300000-0x00300fff]
[   21.384778] pci 0001:00:02.0: PME# supported from D0 D1 D3hot D3cold
[   21.384951] pci 0001:00:02.1: [10b9:5237] type 00 class 0x0c0310
[   21.384987] pci 0001:00:02.1: reg 0x10: [mem 0x00000000-0x00000fff]
[   21.385126] pci 0001:00:02.1: PME# supported from D0 D1 D3hot D3cold
[   21.385291] pci 0001:00:02.2: [10b9:5237] type 00 class 0x0c0310
[   21.385327] pci 0001:00:02.2: reg 0x10: [mem 0x00000000-0x00000fff]
[   21.385466] pci 0001:00:02.2: PME# supported from D0 D1 D3hot D3cold
[   21.385636] pci 0001:00:02.3: [10b9:5239] type 00 class 0x0c0320
[   21.385676] pci 0001:00:02.3: reg 0x10: [mem 0x00000000-0x000000ff]
[   21.385828] pci 0001:00:02.3: PME# supported from D0 D3hot D3cold
[   21.386004] pci 0001:00:03.0: [10a9:0009] type 00 class 0x020000
[   21.386039] pci 0001:00:03.0: reg 0x10: [mem 0x00400000-0x00403fff]
[   21.386395] pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff]
[   21.397018] pci 0001:00:02.0: BAR 0: assigned [mem 0x1d204000-0x1d204fff]
[   21.407552] pci 0001:00:02.1: BAR 0: assigned [mem 0x1d205000-0x1d205fff]
[   21.417945] pci 0001:00:02.2: BAR 0: assigned [mem 0x1d206000-0x1d206fff]
[   21.428185] pci 0001:00:01.0: BAR 2: assigned [io  0x1da00000-0x1da000ff]
[   21.438321] pci 0001:00:02.3: BAR 0: assigned [mem 0x1d207000-0x1d2070ff]
[   21.448373] pci 0001:00:01.0: BAR 0: assigned [mem 0x1d207100-0x1d20717f]
[   21.458433] pci 0001:00:01.0: BAR 1: assigned [io  0x1da00400-0x1da0047f]
[   21.468646] acenic.c: v0.92 08/05/2002  Jes Sorensen, linux-acenic@SunSITE.dk
                                           http://home.cern.ch/~jes/gige/acenic.html
[   21.488947] ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit.
[   21.499592] 0001:00:03.0: SGI AceNIC
[   21.499804] Gigabit Ethernet at 0xd080000000,
[   21.510369] irq 4
[   21.521387]   Tigon II (Rev. 6), Firmware: 0.0.0,
[   21.526945] MAC: 08:00:69:14:69:9c
[   21.537682]   PCI cache line size set incorrectly (0 bytes) by BIOS/FW,
[   21.538031] correcting to 128
[   21.549022]   PCI bus width: 64 bits, speed: 33MHz, latency: 64 clks
[   21.609244] 0001:00:03.0: Firmware up and running
[   21.621834] NET: Registered protocol family 26
[   21.633741] NET: Registered protocol family 10
[   21.646997] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[   21.659228] NET: Registered protocol family 17
[   21.670539] NET: Registered protocol family 4
[   21.681995] 8021q: 802.1Q VLAN Support v1.8
[   21.693639] sctp: Hash tables configured (bind 4096/4096)
[   21.705107] Key type dns_resolver registered
[   21.717647] rtc-ds1685 rtc-ds1685: setting system clock to 2017-02-08 06:26:13 UTC (1486535173)
[   21.847090] input: AT Raw Set 2 keyboard as /devices/pci0000:00/0000:00:02.0/serio0/input/input0
[   22.077817] input: ImPS/2 Logitech Wheel Mouse as /devices/pci0000:00/0000:00:02.0/serio1/input/input2
[   23.593306] sd 0:0:1:0: [sda] 143374000 512-byte logical blocks: (73.4 GB/68.4 GiB)
[   23.605279] sd 0:0:2:0: [sdb] 143374000 512-byte logical blocks: (73.4 GB/68.4 GiB)
[   23.605350] sd 0:0:3:0: [sdc] 143374000 512-byte logical blocks: (73.4 GB/68.4 GiB)
[   23.608552] sd 0:0:1:0: [sda] Write Protect is off
[   23.608562] sd 0:0:1:0: [sda] Mode Sense: b3 00 10 08
[   23.608736] sd 0:0:3:0: [sdc] Write Protect is off
[   23.608746] sd 0:0:3:0: [sdc] Mode Sense: b3 00 10 08
[   23.610832] sd 0:0:1:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   23.611019] sd 0:0:3:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   23.630520]  sdc: sdc1 sdc2 sdc3 sdc4 sdc5 sdc6 sdc9 sdc11
[   23.634527]  sda: sda1 sda2 sda3 sda4 sda5 sda6 sda9 sda11
[   23.641757] sd 0:0:3:0: [sdc] Attached SCSI disk
[   23.643655] sd 0:0:1:0: [sda] Attached SCSI disk
[   23.720715] sd 0:0:2:0: [sdb] Write Protect is off
[   23.731276] sd 0:0:2:0: [sdb] Mode Sense: b3 00 10 08
[   23.735226] sd 0:0:2:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   23.767686]  sdb: sdb1 sdb2 sdb3 sdb4 sdb5 sdb6 sdb9 sdb11
[   23.786987] sd 0:0:2:0: [sdb] Attached SCSI disk
[   27.778038] eth1: Optical link UP (Full Duplex, Flow Control: TX RX)
[   29.454240] md: Waiting for all devices to be available before autodetect
[   29.464960] md: If you don't use raid, use raid=noautodetect
[   29.476420] md: Autodetecting RAID arrays.
[   29.618865] md: Scanned 15 and added 15 devices.
[   29.629579] md: autorun ...
[   29.640073] md: considering sdb6 ...
[   29.650393] md:  adding sdb6 ...
[   29.660352] md: sdb5 has different UUID to sdb6
[   29.670272] md: sdb4 has different UUID to sdb6
[   29.679984] md: sdb3 has different UUID to sdb6
[   29.689514] md: sdb1 has different UUID to sdb6
[   29.698907] md:  adding sda6 ...
[   29.708151] md: sda5 has different UUID to sdb6
[   29.717293] md: sda4 has different UUID to sdb6
[   29.726434] md: sda3 has different UUID to sdb6
[   29.735366] md: sda1 has different UUID to sdb6
[   29.744150] md:  adding sdc6 ...
[   29.752762] md: sdc5 has different UUID to sdb6
[   29.761561] md: sdc4 has different UUID to sdb6
[   29.770264] md: sdc3 has different UUID to sdb6
[   29.778788] md: sdc1 has different UUID to sdb6
[   29.788125] md: created md4
[   29.796582] md: bind<sdc6>
[   29.805174] md: bind<sda6>
[   29.813663] md: bind<sdb6>
[   29.822159] md: running:
[   29.822307] <sdb6>
[   29.830737] <sda6>
[   29.830883] <sdc6>

[   29.848811] md/raid:md4: device sdb6 operational as raid disk 1
[   29.857310] md/raid:md4: device sda6 operational as raid disk 0
[   29.865556] md/raid:md4: device sdc6 operational as raid disk 2
[   29.875656] md/raid:md4: allocated 49386kB
[   29.883462] md/raid:md4: raid level 5 active with 3 out of 3 devices, algorithm 2
[   29.891223] RAID conf printout:
[   29.891230]  --- level:5 rd:3 wd:3
[   29.891236]  disk 0, o:1, dev:sda6
[   29.891241]  disk 1, o:1, dev:sdb6
[   29.891246]  disk 2, o:1, dev:sdc6
[   29.930444] created bitmap (1 pages) for device md4
[   29.942220] md4: bitmap initialized from disk: read 1 pages, set 0 of 5 bits
[   29.955188] md4: detected capacity change from 0 to 2263613440
[   29.962791] md: considering sdb5 ...
[   29.970412] md:  adding sdb5 ...
[   29.978015] md: sdb4 has different UUID to sdb5
[   29.985631] md: sdb3 has different UUID to sdb5
[   29.993100] md: sdb1 has different UUID to sdb5
[   30.000602] md:  adding sda5 ...
[   30.008021] md: sda4 has different UUID to sdb5
[   30.015457] md: sda3 has different UUID to sdb5
[   30.022727] md: sda1 has different UUID to sdb5
[   30.029792] md:  adding sdc5 ...
[   30.036613] md: sdc4 has different UUID to sdb5
[   30.043326] md: sdc3 has different UUID to sdb5
[   30.049785] md: sdc1 has different UUID to sdb5
[   30.056974] md: created md3
[   30.063332] md: bind<sdc5>
[   30.069777] md: bind<sda5>
[   30.076087] md: bind<sdb5>
[   30.082235] md: running:
[   30.082403] <sdb5>
[   30.088532] <sda5>
[   30.088677] <sdc5>

[   30.101736] md/raid:md3: device sdb5 operational as raid disk 1
[   30.107954] md/raid:md3: device sda5 operational as raid disk 0
[   30.114238] md/raid:md3: device sdc5 operational as raid disk 2
[   30.122570] md/raid:md3: allocated 49386kB
[   30.128738] md/raid:md3: raid level 5 active with 3 out of 3 devices, algorithm 2
[   30.134803] RAID conf printout:
[   30.134809]  --- level:5 rd:3 wd:3
[   30.134814]  disk 0, o:1, dev:sda5
[   30.134819]  disk 1, o:1, dev:sdb5
[   30.134824]  disk 2, o:1, dev:sdc5
[   30.135368] created bitmap (1 pages) for device md3
[   30.141770] md3: bitmap initialized from disk: read 1 pages, set 0 of 9 bits
[   30.151821] md3: detected capacity change from 0 to 4714135552
[   30.157929] md: considering sdb4 ...
[   30.163894] md:  adding sdb4 ...
[   30.169855] md: sdb3 has different UUID to sdb4
[   30.175818] md: sdb1 has different UUID to sdb4
[   30.181669] md:  adding sda4 ...
[   30.187556] md: sda3 has different UUID to sdb4
[   30.193472] md: sda1 has different UUID to sdb4
[   30.199386] md:  adding sdc4 ...
[   30.205261] md: sdc3 has different UUID to sdb4
[   30.211266] md: sdc1 has different UUID to sdb4
[   30.218075] md: created md2
[   30.223942] md: bind<sdc4>
[   30.229963] md: bind<sda4>
[   30.235856] md: bind<sdb4>
[   30.241738] md: running:
[   30.241910] <sdb4>
[   30.247783] <sda4>
[   30.247927] <sdc4>

[   30.260739] md/raid:md2: device sdb4 operational as raid disk 1
[   30.266922] md/raid:md2: device sda4 operational as raid disk 0
[   30.273147] md/raid:md2: device sdc4 operational as raid disk 2
[   30.281542] md/raid:md2: allocated 49386kB
[   30.287799] md/raid:md2: raid level 5 active with 3 out of 3 devices, algorithm 2
[   30.294168] RAID conf printout:
[   30.294173]  --- level:5 rd:3 wd:3
[   30.294179]  disk 0, o:1, dev:sda4
[   30.294185]  disk 1, o:1, dev:sdb4
[   30.294190]  disk 2, o:1, dev:sdc4
[   30.294759] created bitmap (1 pages) for device md2
[   30.301603] md2: bitmap initialized from disk: read 1 pages, set 0 of 204 bits
[   30.314853] md2: detected capacity change from 0 to 109521403904
[   30.321657] md: considering sdb3 ...
[   30.328478] md:  adding sdb3 ...
[   30.335220] md: sdb1 has different UUID to sdb3
[   30.341980] md:  adding sda3 ...
[   30.348792] md: sda1 has different UUID to sdb3
[   30.355738] md:  adding sdc3 ...
[   30.362566] md: sdc1 has different UUID to sdb3
[   30.370189] md: created md1
[   30.376794] md: bind<sdc3>
[   30.383346] md: bind<sda3>
[   30.389765] md: bind<sdb3>
[   30.395951] md: running:
[   30.396102] <sdb3>
[   30.402210] <sda3>
[   30.402374] <sdc3>

[   30.415789] md/raid:md1: device sdb3 operational as raid disk 1
[   30.422276] md/raid:md1: device sda3 operational as raid disk 0
[   30.428822] md/raid:md1: device sdc3 operational as raid disk 2
[   30.437351] md/raid:md1: allocated 49386kB
[   30.443851] md/raid:md1: raid level 5 active with 3 out of 3 devices, algorithm 2
[   30.450498] RAID conf printout:
[   30.450504]  --- level:5 rd:3 wd:3
[   30.450509]  disk 0, o:1, dev:sda3
[   30.450514]  disk 1, o:1, dev:sdb3
[   30.450519]  disk 2, o:1, dev:sdc3
[   30.451066] created bitmap (1 pages) for device md1
[   30.458150] md1: bitmap initialized from disk: read 1 pages, set 0 of 44 bits
[   30.478655] md1: detected capacity change from 0 to 23622057984
[   30.485668] md: considering sdb1 ...
[   30.492746] md:  adding sdb1 ...
[   30.499820] md:  adding sda1 ...
[   30.506833] md:  adding sdc1 ...
[   30.513823] md: created md0
[   30.520890] md: bind<sdc1>
[   30.527969] md: bind<sda1>
[   30.534841] md: bind<sdb1>
[   30.541634] md: running:
[   30.541804] <sdb1>
[   30.548437] <sda1>
[   30.548581] <sdc1>

[   30.563131] md/raid:md0: device sdb1 operational as raid disk 1
[   30.570154] md/raid:md0: device sda1 operational as raid disk 0
[   30.577066] md/raid:md0: device sdc1 operational as raid disk 2
[   30.585924] md/raid:md0: allocated 49386kB
[   30.592560] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2
[   30.599202] RAID conf printout:
[   30.599207]  --- level:5 rd:3 wd:3
[   30.599213]  disk 0, o:1, dev:sda1
[   30.599218]  disk 1, o:1, dev:sdb1
[   30.599223]  disk 2, o:1, dev:sdc1
[   30.599790] created bitmap (1 pages) for device md0
[   30.606979] md0: bitmap initialized from disk: read 1 pages, set 7 of 8 bits
[   30.622820] md0: detected capacity change from 0 to 4294705152
[   30.629693] md: ... autorun DONE.
[   31.634010] XFS (md0): Mounting V4 Filesystem
[   34.170302] XFS (md0): Ending clean mount
[   34.177482] VFS: Mounted root (xfs filesystem) readonly on device 9:0.
[   34.187355] devtmpfs: mounted
[   34.195485] Freeing unused kernel memory: 320K (a8000000208b0000 - a800000020900000)
[   34.202859] This architecture does not have kernel memory protection.
[   46.575416] udevd[593]: starting version 3.2.1
[   47.260061] udevd[593]: specified group 'video' unknown
[   47.261061] udevd[593]: specified group 'tape' unknown
[   47.378114] udevd[594]: starting eudev-3.2.1
[   47.849194] udevd[594]: specified group 'video' unknown
[   47.850206] udevd[594]: specified group 'tape' unknown
[   51.867298] Adding 1048512k swap on /dev/sda2.  Priority:1 extents:1 across:1048512k
[   51.883342] Adding 1048512k swap on /dev/sdb2.  Priority:1 extents:1 across:1048512k
[   51.906315] Adding 1048512k swap on /dev/sdc2.  Priority:1 extents:1 across:1048512k
[   52.273998] XFS (md1): Mounting V5 Filesystem
[   54.356290] XFS (md1): Ending clean mount
[   54.378647] XFS (md2): Mounting V5 Filesystem
[   54.877762] XFS (md2): Ending clean mount
[   54.934648] XFS (md3): Mounting V5 Filesystem
[   55.402402] XFS (md3): Ending clean mount
[   55.418250] XFS (md4): Mounting V5 Filesystem
[   55.950290] XFS (md4): Ending clean mount

[-- Attachment #3: ip30-lspci-20170208.txt --]
[-- Type: text/plain, Size: 10744 bytes --]

0000:00:00.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 0
        Region 0: I/O ports at f100000000 [size=257]
        Region 1: [virtual] Memory at f080000000 (32-bit, non-prefetchable) [size=4097]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at f080010000 [disabled] [size=65537]
        Kernel driver in use: qla1280
lspci: Unable to load libkmod resources: error -12

0000:00:01.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 1
        Region 0: I/O ports at f100000100 [size=257]
        Region 1: Memory at f080020000 (32-bit, non-prefetchable) [size=4097]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at f080030000 [disabled] [size=65537]
        Kernel driver in use: qla1280

0000:00:02.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. IOC3 I/O controller (rev 01)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Interrupt: pin A routed to IRQ 2
        Region 0: Memory at f080100000 (32-bit, non-prefetchable) [size=1048577]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Kernel driver in use: IOC3

0000:00:03.0 Non-VGA unclassified device: Silicon Graphics Intl. Corp. RAD Audio (rev c0)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=slow >TAbort+ <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 255
        Region 0: Memory at 00600000 (32-bit, non-prefetchable) [size=8193]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]

0001:00:01.0 Communication controller: Comtrol Corporation Device 080e (rev 01)
        Subsystem: Comtrol Corporation Device 080e
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 0
        Region 0: Memory at 1d207100 (32-bit, non-prefetchable) [size=129]
        Region 1: I/O ports at 1da00400 [size=129]
        Region 2: I/O ports at 1da00000 [size=257]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]

0001:00:02.0 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 16 (20000ns max)
        Interrupt: pin B routed to IRQ 0
        Region 0: Memory at 1d204000 (32-bit, non-prefetchable) [size=4097]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [60] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

0001:00:02.1 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin C routed to IRQ 0
        Region 0: Memory at 1d205000 (32-bit, non-prefetchable) [disabled] [size=4097]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [60] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

0001:00:02.2 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin D routed to IRQ 0
        Region 0: Memory at 1d206000 (32-bit, non-prefetchable) [disabled] [size=4097]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [60] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

0001:00:02.3 USB controller: ULi Electronics Inc. USB 2.0 Controller (rev 01) (prog-if 20 [EHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 0
        Region 0: Memory at 1d207000 (32-bit, non-prefetchable) [disabled] [size=257]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Debug port: BAR=1 offset=0090

0001:00:03.0 Ethernet controller: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet (rev 01)
        Subsystem: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64 (16000ns min), Cache Line Size: 128 bytes
        Interrupt: pin A routed to IRQ 4
        Region 0: [virtual] Memory at d080000000 (32-bit, non-prefetchable) [size=16385]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Kernel driver in use: acenic

[-- Attachment #4: ip27-lspci-20170208.txt --]
[-- Type: text/plain, Size: 6736 bytes --]

0001:00:00.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. Linc I/O controller
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        NUMA node: 0
        Region 0: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=134217729]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
lspci: Unable to load libkmod resources: error -12

0001:00:01.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. Linc I/O controller
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        NUMA node: 0
        Region 0: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=134217729]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]

0002:00:00.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 1
        NUMA node: 0
        Region 0: I/O ports at f200000 [size=257]
        Region 1: Memory at 0f200000 (32-bit, non-prefetchable) [size=4097]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=65537]
        Kernel driver in use: qla1280

0002:00:01.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 2
        NUMA node: 0
        Region 0: I/O ports at f400000 [size=257]
        Region 1: Memory at 0f400000 (32-bit, non-prefetchable) [size=4097]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=65537]
        Kernel driver in use: qla1280

0002:00:02.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. IOC3 I/O controller (rev 01)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 255
        Interrupt: pin A routed to IRQ 3
        NUMA node: 0
        Region 0: Memory at 0f600000 (32-bit, non-prefetchable) [size=1048577]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Kernel driver in use: ioc3-eth

0002:00:06.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. IOC3 I/O controller (rev 01)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 255
        Interrupt: pin A routed to IRQ 4
        NUMA node: 0
        Region 0: Memory at 0fa00000 (32-bit, non-prefetchable) [size=1048577]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Kernel driver in use: ioc3-eth

0002:00:07.0 Non-VGA unclassified device: Silicon Graphics Intl. Corp. RAD Audio (rev 70)
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=slow >TAbort+ <TAbort- <MAbort- >SERR- <PERR- INTx-
        NUMA node: 0
        Region 0: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=8193]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]

[-- Attachment #5: ip27-dmesg-working_pci-20170208.txt --]
[-- Type: text/plain, Size: 22685 bytes --]

[    0.000000] Linux version 4.10.0-rc7 (root@helcaraxe) (gcc version 6.3.0 (Gentoo Hardened 6.3.0 p1.0) ) #1 SMP Wed Feb 8 03:55:04 EST 2017
[    0.000000] ARCH: SGI-IP27
[    0.000000] PROMLIB: ARC firmware Version 64 Revision 0
[    0.000000] Discovered 4 cpus on 2 nodes
[    0.000000] node_distance: router_a NULL
[    0.000000] node_distance: router_a NULL
[    0.000000] node_distance: router_a NULL
[    0.000000] node_distance: router_a NULL
[    0.000000] ************** Topology ********************
[    0.000000]
[    0.000000] 00
[    0.000000] 01

[    0.000000] 00
[    0.000000] 255
[    0.000000] 255

[    0.000000] 01
[    0.000000] 255
[    0.000000] 255

[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00000f14 (R14000)
[    0.000000] FPU revision is: 00000900
[    0.000000] Checking for the multiply/shift bug... no.
[    0.000000] Checking for the daddiu bug... no.
[    0.000000] IP27: Running on node 0.
[    0.000000] Node 0 has a primary CPU, CPU is running.
[    0.000000] Node 0 has a secondary CPU, CPU is running.
[    0.000000] Machine is in M mode.
[    0.000000] Cpu 0, Nasid 0x0: partnum 0x0 is
[    0.000000] is xbow
[    0.000000] Cpu 0, Nasid 0x0, widget 0x8 (partnum 0xc102) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xb (partnum 0xc002) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xc (partnum 0xc002) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xd (partnum 0xc003) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xf (partnum 0xc002) is
[    0.000000] CPU 0 clock is 500MHz.
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 00000000006e0000 @ 0000000000010000 (usable)
[    0.000000]  memory: 0000000000050000 @ 00000000006f0000 (usable after init)
[    0.000000] cma: Reserved 512 MiB at 0x00000001e0000000
[    0.000000] REPLICATION: ON nasid 0, ktext from nasid 0, kdata from nasid 0
[    0.000000] REPLICATION: ON nasid 1, ktext from nasid 0, kdata from nasid 0
[    0.000000] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    0.000000] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    0.000000] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x00000001ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   node   1: [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000] On node 0 totalpages: 65536
[    0.000000] free_area_init_node: node 0, pgdat a8000000007f0000, node_mem_map a800000001000000
[    0.000000]   Normal zone: 64 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 65536 pages, LIFO batch:1
[    0.000000] Initmem setup node 1 [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] On node 1 totalpages: 65536
[    0.000000] free_area_init_node: node 1, pgdat a800000100030000, node_mem_map a800000100050000
[    0.000000]   Normal zone: 64 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 65536 pages, LIFO batch:1
[    0.000000] percpu: Embedded 2 pages/cpu @a800000001430000 s60064 r0 d71008 u131072
[    0.000000] pcpu-alloc: s60064 r0 d71008 u131072 alloc=2*65536
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 130944
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=dksc(0,1,0) console=ttyS0,9600 root=/dev/md0
[    0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes)
[    0.000000] Memory: 7846784K/8388608K available (5172K kernel code, 678K rwdata, 1116K rodata, 320K init, 700K bss, 17536K reserved, 524288K cma-reserved)
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000]  RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4
[    0.000000] NR_IRQS:256
[    0.000000] clocksource: HUB-RT: mask: 0xfffffffffffff max_cycles: 0x127350b88, max_idle_ns: 1763180808480 ns
[    0.000012] sched_clock: 52 bits at 1250kHz, resolution 800ns, wraps every 4398046510800ns
[    0.103175] Console: colour dummy device 80x25
[    0.156244] Calibrating delay loop... 749.56 BogoMIPS (lpj=3747840)
[    0.318496] pid_max: default: 32768 minimum: 301
[    0.378324] Dentry cache hash table entries: 1048576 (order: 7, 8388608 bytes)
[    0.577930] Inode-cache hash table entries: 524288 (order: 6, 4194304 bytes)
[    0.720390] Mount-cache hash table entries: 16384 (order: 1, 131072 bytes)
[    0.801233] Mountpoint-cache hash table entries: 16384 (order: 1, 131072 bytes)
[    0.900780] Checking for the daddi bug... no.
[    0.957828] smp: Bringing up secondary CPUs ...
[    1.012770] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    1.012786] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    1.012794] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    1.015959] CPU 1 clock is 500MHz.
[    1.015981] CPU1 revision is: 00000f14 (R14000)
[    1.015984] FPU revision is: 00000900
[    1.054076] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    1.054096] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    1.054105] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    1.057333] Cpu 2, Nasid 0x1: partnum 0x0 is
[    1.057356] is xbow
[    1.057400] CPU 2 clock is 500MHz.
[    1.057417] CPU2 revision is: 00000f14 (R14000)
[    1.057421] FPU revision is: 00000900
[    1.094440] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    1.094459] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    1.094468] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    1.097704] CPU 3 clock is 500MHz.
[    1.097732] CPU3 revision is: 00000f14 (R14000)
[    1.097736] FPU revision is: 00000900
[    1.131286] smp: Brought up 2 nodes, 4 CPUs
[    2.425108] devtmpfs: initialized
[    2.464264] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    2.585991] xor: measuring software checksum speed
[    2.741920]    8regs     :   710.400 MB/sec
[    2.890564]    8regs_prefetch:   684.800 MB/sec
[    3.043377]    32regs    :   659.200 MB/sec
[    3.171273] random: fast init done
[    3.191971]    32regs_prefetch:   640.000 MB/sec
[    3.191979] xor: using function: 8regs (710.400 MB/sec)
[    3.329553] NET: Registered protocol family 16
[    3.442067] cpuidle: using governor ladder
[    3.571538] cpuidle: using governor menu
[    3.907402] raid6: int64x1  gen()   309 MB/s
[    4.127084] raid6: int64x1  xor()    62 MB/s
[    4.346824] raid6: int64x2  gen()   444 MB/s
[    4.566655] raid6: int64x2  xor()   152 MB/s
[    4.786336] raid6: int64x4  gen()   387 MB/s
[    5.006111] raid6: int64x4  xor()   141 MB/s
[    5.225866] raid6: int64x8  gen()   315 MB/s
[    5.445840] raid6: int64x8  xor()   116 MB/s
[    5.495273] raid6: using algorithm int64x2 gen() 444 MB/s
[    5.560206] raid6: .... xor() 152 MB/s, rmw enabled
[    5.618856] raid6: using intx1 recovery algorithm
[    5.675924] SCSI subsystem initialized
[    5.721236] PCI host bridge to bus 0002:00
[    5.769704] pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff]
[    5.869182] pci_bus 0002:00: root bus resource [io  0x920000000fa00000-0x920000000fbfffff]
[    5.968680] pci_bus 0002:00: root bus resource [bus 02-ff]
[    6.034717] pci 0002:00:00.0: [1077:1020] type 00 class 0x010000
[    6.034773] pci 0002:00:00.0: reg 0x10: [io  0xf200000-0xf2000ff]
[    6.034800] pci 0002:00:00.0: reg 0x14: [mem 0x0f200000-0x0f200fff]
[    6.034893] pci 0002:00:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    6.035175] pci 0002:00:01.0: [1077:1020] type 00 class 0x010000
[    6.035213] pci 0002:00:01.0: reg 0x10: [io  0xf400000-0xf4000ff]
[    6.035240] pci 0002:00:01.0: reg 0x14: [mem 0x0f400000-0x0f400fff]
[    6.035333] pci 0002:00:01.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    6.035599] pci 0002:00:02.0: [10a9:0003] type 00 class 0xff0000
[    6.035641] pci 0002:00:02.0: reg 0x10: [mem 0x0f600000-0x0f6fffff]
[    6.036013] pci 0002:00:06.0: [10a9:0003] type 00 class 0xff0000
[    6.036056] pci 0002:00:06.0: reg 0x10: [mem 0x0fa00000-0x0fafffff]
[    6.036401] pci 0002:00:07.0: [10a9:0005] type 00 class 0x000000
[    6.036433] pci 0002:00:07.0: reg 0x10: [mem 0x00000000-0x00001fff]
[    6.036933] PCI host bridge to bus 0001:00
[    6.084311] pci_bus 0001:00: root bus resource [mem 0x920000000c200000-0x920000000c9fffff]
[    6.183803] pci_bus 0001:00: root bus resource [io  0x920000000ca00000-0x920000000cbfffff]
[    6.283297] pci_bus 0001:00: root bus resource [bus 01-ff]
[    6.349329] pci 0001:00:00.0: [10a9:0002] type 00 class 0xff0000
[    6.349369] pci 0001:00:00.0: reg 0x10: [mem 0x00000000-0x07ffffff]
[    6.349771] pci 0001:00:01.0: [10a9:0002] type 00 class 0xff0000
[    6.349811] pci 0001:00:01.0: reg 0x10: [mem 0x00000000-0x07ffffff]
[    6.350376] PCI host bridge to bus 0000:00
[    6.398511] pci_bus 0000:00: root bus resource [mem 0x920000000b200000-0x920000000b9fffff]
[    6.498002] pci_bus 0000:00: root bus resource [io  0x920000000ba00000-0x920000000bbfffff]
[    6.597500] pci_bus 0000:00: root bus resource [bus 00-ff]
[    6.664580] clocksource: Switched to clocksource HUB-RT
[    6.737340] NET: Registered protocol family 2
[    6.788631] TCP established hash table entries: 65536 (order: 3, 524288 bytes)
[    6.877860] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes)
[    6.961938] TCP: Hash tables configured (established 65536 bind 65536)
[    7.038796] UDP hash table entries: 4096 (order: 1, 131072 bytes)
[    7.112666] UDP-Lite hash table entries: 4096 (order: 1, 131072 bytes)
[    7.191678] NET: Registered protocol family 1
[    7.243000] PCI: CLS 256 bytes, default 128
[    7.244032] futex hash table entries: 1024 (order: 1, 131072 bytes)
[    7.319896] workingset: timestamp_bits=40 max_order=17 bucket_order=0
[    7.396183] zbud: loaded
[    7.428650] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[    7.519969] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    7.607150] io scheduler noop registered
[    7.654268] io scheduler deadline registered
[    7.705641] io scheduler cfq registered (default)
[    7.825919] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    7.915337] loop: module loaded
[    7.951330] qla1280: QLA1040 found on PCI bus 0, dev 0
[    8.013015] PCI: Enabling device 0002:00:00.0 (0006 -> 0007)
[    8.583489] random: crng init done
[    8.843500] scsi(0:0): Resetting SCSI BUS
[   11.914788] scsi host0: QLogic QLA1040 PCI to SCSI Host Adapter
                      Firmware version:  7.65.06, Driver version 3.27.1
[   12.068902] qla1280: QLA1040 found on PCI bus 0, dev 1
[   12.128824] PCI: Enabling device 0002:00:01.0 (0006 -> 0007)
[   12.240023] scsi 0:0:1:0: Direct-Access     SEAGATE  SX150176LC       BA12 PQ: 0 ANSI: 2
[   12.336821] scsi(0:0:1:0):
[   12.336828]  Sync: period 10, offset 12
[   12.367970] , Wide
[   12.414054] , Tagged queuing: depth 31

[   12.507613] scsi 0:0:2:0: Direct-Access     SEAGATE  SX150176LC       BA08 PQ: 0 ANSI: 2
[   12.604289] scsi(0:0:2:0):
[   12.604296]  Sync: period 10, offset 12
[   12.635564] , Wide
[   12.681648] , Tagged queuing: depth 31

[   12.775169] scsi 0:0:3:0: Direct-Access     SEAGATE  SX150176LC       BA08 PQ: 0 ANSI: 2
[   12.871848] scsi(0:0:3:0):
[   12.871856]  Sync: period 10, offset 12
[   12.903159] , Wide
[   12.949242] , Tagged queuing: depth 31
[   12.961432] scsi(1:0): Resetting SCSI BUS

[   15.858292] scsi 0:0:4:0: Direct-Access     SEAGATE  SX150176LC       BA11 PQ: 0 ANSI: 2
[   15.954656] scsi(0:0:4:0):
[   15.954664]  Sync: period 10, offset 12
[   15.986201] , Wide
[   16.032286] , Tagged queuing: depth 31

[   16.119472] scsi host1: QLogic QLA1040 PCI to SCSI Host Adapter
                      Firmware version:  7.65.06, Driver version 3.27.1
[   16.275065] console [ttyS0] disabled
[   16.336528] serial8250: ttyS0 at MMIO 0x920000000f620178 (irq = 0, base_baud = 458333) is a 16550A
[   16.442612] console [ttyS0] enabled
[   16.525366] bootconsole [early0] disabled
[   16.646656] serial8250: ttyS1 at MMIO 0x920000000f620170 (irq = 0, base_baud = 458333) is a 16550A
[   16.754744] scsi 0:0:5:0: Direct-Access     SEAGATE  SX150176LC       BA11 PQ: 0 ANSI: 2
[   16.855313] scsi(0:0:5:0):
[   16.855329]  Sync: period 10, offset 12
[   16.887944] , Wide
[   16.889739] Found DS1981U NIC
[   16.889748]  registration number 5e:57:03:00:70:5e, CRC 52
[   16.889750] .
[   16.905995] Ethernet address is 08:00:69:05:64:74.
[   16.907374] ioc3-eth 0002:00:02.0 eth0: link up, 100Mbps, full-duplex, lpa 0x8DE1
[   16.907381] eth0: Using PHY 31, vendor 0x20005c0, model 0, rev 0.
[   16.907384] eth0: IOC3 SSRAM has 128 kbyte.
[   16.929432] serial8250: ttyS2 at MMIO 0x920000000fa20178 (irq = 0, base_baud = 458333) is a 16550A
[   16.950337] serial8250: ttyS3 at MMIO 0x920000000fa20170 (irq = 0, base_baud = 458333) is a 16550A
[   16.954048] NIC search failed (not fatal).
[   16.956507] NIC search failed (not fatal).
[   16.957008] Failed to read MAC address
[   16.957012] Ethernet address is 00:00:00:00:00:00.
[   16.958135] ioc3-eth 0002:00:06.0 eth1: link down
[   16.958142] eth1: Using PHY 31, vendor 0x0, model 0, rev 0.
[   16.958145] eth1: IOC3 SSRAM has 64 kbyte.
[   16.959185] NET: Registered protocol family 17
[   16.959732] rtc-m48t35 rtc-m48t35: rtc core: registered m48t35 as rtc0
[   18.073532] , Tagged queuing: depth 31

[   18.136683] rtc-m48t35 rtc-m48t35: setting system clock to 2017-02-08 08:57:44 UTC (1486544264)
[   18.249152] scsi 0:0:6:0: CD-ROM            TOSHIBA  CD-ROM XM-5701TA 0167 PQ: 0 ANSI: 2
[   18.347391] scsi(0:0:6:0):
[   18.347396]  Sync: period 10, offset 12

[   19.841973] sd 0:0:3:0: [sdc] 97693755 512-byte logical blocks: (50.0 GB/46.6 GiB)
[   19.844598] sr 0:0:6:0: [sr0] scsi-1 drive
[   19.844606] cdrom: Uniform CD-ROM driver Revision: 3.20
[   19.845324] sr 0:0:6:0: Attached scsi CD-ROM sr0
[   19.934698] sd 0:0:2:0: [sdb] 97693755 512-byte logical blocks: (50.0 GB/46.6 GiB)
[   19.934807] sd 0:0:1:0: [sda] 97693755 512-byte logical blocks: (50.0 GB/46.6 GiB)
[   19.934986] sd 0:0:4:0: [sdd] 97693755 512-byte logical blocks: (50.0 GB/46.6 GiB)
[   19.935572] sd 0:0:5:0: [sde] 97693755 512-byte logical blocks: (50.0 GB/46.6 GiB)
[   19.936993] sd 0:0:2:0: [sdb] Write Protect is off
[   19.937004] sd 0:0:2:0: [sdb] Mode Sense: cb 00 10 08
[   19.937556] sd 0:0:1:0: [sda] Write Protect is off
[   19.937565] sd 0:0:1:0: [sda] Mode Sense: cb 00 10 08
[   19.937803] sd 0:0:4:0: [sdd] Write Protect is off
[   19.937812] sd 0:0:4:0: [sdd] Mode Sense: cb 00 10 08
[   19.938310] sd 0:0:5:0: [sde] Write Protect is off
[   19.938320] sd 0:0:5:0: [sde] Mode Sense: cb 00 10 08
[   19.939948] sd 0:0:2:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   19.940496] sd 0:0:1:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   19.940944] sd 0:0:4:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   19.941212] sd 0:0:5:0: [sde] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   20.002748]  sda: sda1 sda2 sda3 sda4 sda5 sda6 sda9 sda11
[   20.013020]  sdd: sdd1 sdd2 sdd3 sdd4 sdd5 sdd6 sdd9 sdd11
[   20.014354]  sdb: sdb1 sdb2 sdb3 sdb4 sdb5 sdb6 sdb9 sdb11
[   20.024588] sd 0:0:5:0: [sde] Attached SCSI disk
[   20.025291] sd 0:0:1:0: [sda] Attached SCSI disk
[   20.026150] sd 0:0:2:0: [sdb] Attached SCSI disk
[   20.026569] sd 0:0:4:0: [sdd] Attached SCSI disk
[   21.481104] sd 0:0:3:0: [sdc] Write Protect is off
[   21.538895] sd 0:0:3:0: [sdc] Mode Sense: cb 00 10 08
[   21.541780] sd 0:0:3:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   21.694266]  sdc: sdc1 sdc2 sdc3 sdc4 sdc5 sdc6 sdc9 sdc11
[   21.770364] sd 0:0:3:0: [sdc] Attached SCSI disk
[   21.826524] md: Waiting for all devices to be available before autodetect
[   21.908398] md: If you don't use raid, use raid=noautodetect
[   21.977995] md: Autodetecting RAID arrays.
[   22.112008] md: invalid raid superblock magic on sdd3
[   22.172984] md: sdd3 does not have a valid v0.90 superblock, not importing!
[   22.272565] md: invalid raid superblock magic on sdd4
[   22.333508] md: sdd4 does not have a valid v0.90 superblock, not importing!
[   22.543056] md: autorun ...
[   22.576739] md: sdc5 has different UUID to sdc6
[   22.631360] md: sdc4 has different UUID to sdc6
[   22.686052] md: sdc3 has different UUID to sdc6
[   22.740709] md: sdc1 has different UUID to sdc6
[   22.795447] md: sdd5 has different UUID to sdc6
[   22.850064] md: sdb5 has different UUID to sdc6
[   22.904724] md: sdb4 has different UUID to sdc6
[   22.959392] md: sdb3 has different UUID to sdc6
[   23.014086] md: sdb1 has different UUID to sdc6
[   23.068748] md: sdd1 has different UUID to sdc6
[   23.123434] md: sda5 has different UUID to sdc6
[   23.178092] md: sda4 has different UUID to sdc6
[   23.232752] md: sda3 has different UUID to sdc6
[   23.287422] md: sda1 has different UUID to sdc6
[   23.343672] md: running: <sdc6><sdd6><sdb6><sda6>
[   23.402609] md/raid:md4: device sdc6 operational as raid disk 2
[   23.473960] md/raid:md4: device sdb6 operational as raid disk 1
[   23.545358] md/raid:md4: device sda6 operational as raid disk 0
[   23.620651] md/raid:md4: raid level 5 active with 3 out of 3 devices, algorithm 2
[   23.731897] md4: detected capacity change from 0 to 1940914176
[   23.802401] md: sdc4 has different UUID to sdc5
[   23.857115] md: sdc3 has different UUID to sdc5
[   23.911764] md: sdc1 has different UUID to sdc5
[   23.966428] md: sdd5 has different UUID to sdc5
[   24.021133] md: sdb4 has different UUID to sdc5
[   24.075753] md: sdb3 has different UUID to sdc5
[   24.130442] md: sdb1 has different UUID to sdc5
[   24.185126] md: sdd1 has different UUID to sdc5
[   24.239785] md: sda4 has different UUID to sdc5
[   24.294452] md: sda3 has different UUID to sdc5
[   24.349137] md: sda1 has different UUID to sdc5
[   24.405311] md: running: <sdc5><sdb5><sda5>
[   24.457636] md/raid:md3: device sdc5 operational as raid disk 2
[   24.529063] md/raid:md3: device sdb5 operational as raid disk 1
[   24.600456] md/raid:md3: device sda5 operational as raid disk 0
[   24.675431] md/raid:md3: raid level 5 active with 3 out of 3 devices, algorithm 2
[   24.789229] md3: detected capacity change from 0 to 4013686784
[   24.859652] md: sdc3 has different UUID to sdc4
[   24.914292] md: sdc1 has different UUID to sdc4
[   24.968933] md: sdd5 has different UUID to sdc4
[   25.023626] md: sdb3 has different UUID to sdc4
[   25.078273] md: sdb1 has different UUID to sdc4
[   25.132940] md: sdd1 has different UUID to sdc4
[   25.187633] md: sda3 has different UUID to sdc4
[   25.242286] md: sda1 has different UUID to sdc4
[   25.298457] md: running: <sdc4><sdb4><sda4>
[   25.350924] md/raid:md2: device sdc4 operational as raid disk 2
[   25.422336] md/raid:md2: device sdb4 operational as raid disk 1
[   25.493740] md/raid:md2: device sda4 operational as raid disk 0
[   25.568663] md/raid:md2: raid level 5 active with 3 out of 3 devices, algorithm 2
[   25.681053] md2: detected capacity change from 0 to 80015261696
[   25.752544] md: sdc1 has different UUID to sdc3
[   25.807252] md: sdd5 has different UUID to sdc3
[   25.861928] md: sdb1 has different UUID to sdc3
[   25.916728] md: sdd1 has different UUID to sdc3
[   25.971376] md: sda1 has different UUID to sdc3
[   26.027416] md: running: <sdc3><sdb3><sda3>
[   26.079824] md/raid:md1: device sdc3 operational as raid disk 2
[   26.151284] md/raid:md1: device sdb3 operational as raid disk 1
[   26.222664] md/raid:md1: device sda3 operational as raid disk 0
[   26.297448] md/raid:md1: raid level 5 active with 3 out of 3 devices, algorithm 2
[   26.400761] md1: detected capacity change from 0 to 12008554496
[   26.472253] md: sdd5 has different UUID to sdc1
[   26.527422] md: running: <sdc1><sdb1><sdd1><sda1>
[   26.586161] md/raid:md0: device sdc1 operational as raid disk 2
[   26.657569] md/raid:md0: device sdb1 operational as raid disk 1
[   26.728956] md/raid:md0: device sda1 operational as raid disk 0
[   26.804247] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2
[   26.909265] md0: detected capacity change from 0 to 1019740160
[   26.979722] md: md3 already running, cannot run sdd5
[   27.039662] md: ... autorun DONE.
[   27.151464] UDF-fs: warning (device md0): udf_fill_super: No partition found (2)
[   27.268270] XFS (md0): Mounting V5 Filesystem
[   30.003679] XFS (md0): Ending clean mount
[   30.052192] VFS: Mounted root (xfs filesystem) readonly on device 9:0.
[   30.136790] devtmpfs: mounted
[   30.174297] Freeing unused kernel memory: 320K
[   30.227891] This architecture does not have kernel memory protection.
[   40.863143] udevd[585]: starting version 3.2.1
[   40.938693] udevd[585]: specified group 'video' unknown
[   40.939862] udevd[585]: specified group 'tape' unknown
[   40.997900] udevd[586]: starting eudev-3.2.1
[   41.517564] udevd[586]: specified group 'video' unknown
[   41.518745] udevd[586]: specified group 'tape' unknown
[   48.072540] Adding 449728k swap on /dev/sda2.  Priority:1 extents:1 across:449728k
[   48.111252] Adding 449728k swap on /dev/sdb2.  Priority:1 extents:1 across:449728k
[   48.219132] Adding 449728k swap on /dev/sdc2.  Priority:1 extents:1 across:449728k
[   48.685496] XFS (md1): Mounting V5 Filesystem
[   51.388111] XFS (md1): Ending clean mount
[   51.577772] XFS (md2): Mounting V5 Filesystem
[   54.300352] XFS (md2): Ending clean mount
[   54.600708] XFS (md3): Mounting V5 Filesystem
[   57.382781] XFS (md3): Ending clean mount
[   57.485516] XFS (md4): Mounting V5 Filesystem
[   60.223068] XFS (md4): Ending clean mount

[-- Attachment #6: ip27-dmesg-size_assign_pci-20170208.txt --]
[-- Type: text/plain, Size: 14335 bytes --]

[    0.000000] Linux version 4.10.0-rc7 (root@helcaraxe) (gcc version 6.3.0 (Gentoo Hardened 6.3.0 p1.0) ) #2 SMP Wed Feb 8 04:07:11 EST 2017
[    0.000000] ARCH: SGI-IP27
[    0.000000] PROMLIB: ARC firmware Version 64 Revision 0
[    0.000000] Discovered 4 cpus on 2 nodes
[    0.000000] node_distance: router_a NULL
[    0.000000] node_distance: router_a NULL
[    0.000000] node_distance: router_a NULL
[    0.000000] node_distance: router_a NULL
[    0.000000] ************** Topology ********************
[    0.000000]
[    0.000000] 00
[    0.000000] 01
[    0.000000]
[    0.000000] 00
[    0.000000] 255
[    0.000000] 255
[    0.000000]
[    0.000000] 01
[    0.000000] 255
[    0.000000] 255
[    0.000000]
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00000f14 (R14000)
[    0.000000] FPU revision is: 00000900
[    0.000000] Checking for the multiply/shift bug... no.
[    0.000000] Checking for the daddiu bug... no.
[    0.000000] IP27: Running on node 0.
[    0.000000] Node 0 has a primary CPU, CPU is running.
[    0.000000] Node 0 has a secondary CPU, CPU is running.
[    0.000000] Machine is in M mode.
[    0.000000] Cpu 0, Nasid 0x0: partnum 0x0 is
[    0.000000] is xbow
[    0.000000] Cpu 0, Nasid 0x0, widget 0x8 (partnum 0xc102) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xb (partnum 0xc002) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xc (partnum 0xc002) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xd (partnum 0xc003) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xf (partnum 0xc002) is
[    0.000000] CPU 0 clock is 500MHz.
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 00000000006e0000 @ 0000000000010000 (usable)
[    0.000000]  memory: 0000000000050000 @ 00000000006f0000 (usable after init)
[    0.000000] cma: Reserved 512 MiB at 0x00000001e0000000
[    0.000000] REPLICATION: ON nasid 0, ktext from nasid 0, kdata from nasid 0
[    0.000000] REPLICATION: ON nasid 1, ktext from nasid 0, kdata from nasid 0
[    0.000000] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    0.000000] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    0.000000] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x00000001ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   node   1: [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000] Initmem setup node 1 [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] percpu: Embedded 2 pages/cpu @a800000001430000 s60064 r0 d71008 u131072
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 130944
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=dksc(0,1,0) console=ttyS0,9600 root=/dev/md0
[    0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes)
[    0.000000] Memory: 7846784K/8388608K available (5172K kernel code, 678K rwdata, 1116K rodata, 320K init, 700K bss, 17536K reserved, 524288K cma-reserved)
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000]  RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4
[    0.000000] NR_IRQS:256
[    0.000000] clocksource: HUB-RT: mask: 0xfffffffffffff max_cycles: 0x127350b88, max_idle_ns: 1763180808480 ns
[    0.000012] sched_clock: 52 bits at 1250kHz, resolution 800ns, wraps every 4398046510800ns
[    0.103183] Console: colour dummy device 80x25
[    0.156258] Calibrating delay loop... 749.56 BogoMIPS (lpj=3747840)
[    0.318497] pid_max: default: 32768 minimum: 301
[    0.378347] Dentry cache hash table entries: 1048576 (order: 7, 8388608 bytes)
[    0.577889] Inode-cache hash table entries: 524288 (order: 6, 4194304 bytes)
[    0.720380] Mount-cache hash table entries: 16384 (order: 1, 131072 bytes)
[    0.801235] Mountpoint-cache hash table entries: 16384 (order: 1, 131072 bytes)
[    0.900788] Checking for the daddi bug... no.
[    0.957852] smp: Bringing up secondary CPUs ...
[    1.012850] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    1.012866] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    1.012873] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    1.016040] CPU 1 clock is 500MHz.
[    1.016063] CPU1 revision is: 00000f14 (R14000)
[    1.016067] FPU revision is: 00000900
[    1.054189] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    1.054209] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    1.054219] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    1.057447] Cpu 2, Nasid 0x1: partnum 0x0 is
[    1.057471] is xbow
[    1.057513] CPU 2 clock is 500MHz.
[    1.057532] CPU2 revision is: 00000f14 (R14000)
[    1.057536] FPU revision is: 00000900
[    1.094532] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    1.094550] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    1.094560] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    1.097795] CPU 3 clock is 500MHz.
[    1.097823] CPU3 revision is: 00000f14 (R14000)
[    1.097827] FPU revision is: 00000900
[    1.131387] smp: Brought up 2 nodes, 4 CPUs
[    2.425207] devtmpfs: initialized
[    2.464376] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    2.586105] xor: measuring software checksum speed
[    2.742023]    8regs     :   710.400 MB/sec
[    2.890670]    8regs_prefetch:   684.800 MB/sec
[    3.043481]    32regs    :   659.200 MB/sec
[    3.171390] random: fast init done
[    3.192078]    32regs_prefetch:   640.000 MB/sec
[    3.192086] xor: using function: 8regs (710.400 MB/sec)
[    3.329769] NET: Registered protocol family 16
[    3.442281] cpuidle: using governor ladder
[    3.549724] cpuidle: using governor menu
[    3.902207] raid6: int64x1  gen()   304 MB/s
[    4.122196] raid6: int64x1  xor()    64 MB/s
[    4.341936] raid6: int64x2  gen()   446 MB/s
[    4.561720] raid6: int64x2  xor()   152 MB/s
[    4.781428] raid6: int64x4  gen()   386 MB/s
[    5.001245] raid6: int64x4  xor()   141 MB/s
[    5.221081] raid6: int64x8  gen()   315 MB/s
[    5.440814] raid6: int64x8  xor()   116 MB/s
[    5.490252] raid6: using algorithm int64x2 gen() 446 MB/s
[    5.555184] raid6: .... xor() 152 MB/s, rmw enabled
[    5.613836] raid6: using intx1 recovery algorithm
[    5.670909] SCSI subsystem initialized
[    5.716212] PCI host bridge to bus 0002:00
[    5.764684] pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff]
[    5.864163] pci_bus 0002:00: root bus resource [io  0x920000000fa00000-0x920000000fbfffff]
[    5.963663] pci_bus 0002:00: root bus resource [bus 02-ff]
[    6.031825] pci 0002:00:02.0: BAR 0: no space for [mem size 0x00100000]
[    6.109557] pci 0002:00:02.0: BAR 0: failed to assign [mem size 0x00100000]
[    6.193341] pci 0002:00:06.0: BAR 0: no space for [mem size 0x00100000]
[    6.272936] pci 0002:00:06.0: BAR 0: failed to assign [mem size 0x00100000]
[    6.356740] pci 0002:00:00.0: BAR 6: no space for [mem size 0x00010000 pref]
[    6.441558] pci 0002:00:00.0: BAR 6: failed to assign [mem size 0x00010000 pref]
[    6.530583] pci 0002:00:01.0: BAR 6: no space for [mem size 0x00010000 pref]
[    6.615415] pci 0002:00:01.0: BAR 6: failed to assign [mem size 0x00010000 pref]
[    6.704440] pci 0002:00:00.0: BAR 1: no space for [mem size 0x00001000]
[    6.784036] pci 0002:00:00.0: BAR 1: failed to assign [mem size 0x00001000]
[    6.867824] pci 0002:00:01.0: BAR 1: no space for [mem size 0x00001000]
[    6.947420] pci 0002:00:01.0: BAR 1: failed to assign [mem size 0x00001000]
[    7.031210] pci 0002:00:00.0: BAR 0: no space for [io  size 0x0100]
[    7.106623] pci 0002:00:00.0: BAR 0: failed to assign [io  size 0x0100]
[    7.186215] pci 0002:00:01.0: BAR 0: no space for [io  size 0x0100]
[    7.261620] pci 0002:00:01.0: BAR 0: failed to assign [io  size 0x0100]
[    7.341358] PCI host bridge to bus 0001:00
[    7.390455] pci_bus 0001:00: root bus resource [mem 0x920000000c200000-0x920000000c9fffff]
[    7.489945] pci_bus 0001:00: root bus resource [io  0x920000000ca00000-0x920000000cbfffff]
[    7.589441] pci_bus 0001:00: root bus resource [bus 01-ff]
[    7.656392] pci 0001:00:00.0: BAR 0: no space for [mem size 0x08000000]
[    7.735022] pci 0001:00:00.0: BAR 0: failed to assign [mem size 0x08000000]
[    7.818808] pci 0001:00:01.0: BAR 0: no space for [mem size 0x08000000]
[    7.898403] pci 0001:00:01.0: BAR 0: failed to assign [mem size 0x08000000]
[    7.982315] PCI host bridge to bus 0000:00
[    8.031416] pci_bus 0000:00: root bus resource [mem 0x920000000b200000-0x920000000b9fffff]
[    8.130914] pci_bus 0000:00: root bus resource [io  0x920000000ba00000-0x920000000bbfffff]
[    8.230410] pci_bus 0000:00: root bus resource [bus 00-ff]
[    8.297487] clocksource: Switched to clocksource HUB-RT
[    8.370358] NET: Registered protocol family 2
[    8.421632] TCP established hash table entries: 65536 (order: 3, 524288 bytes)
[    8.510880] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes)
[    8.594960] TCP: Hash tables configured (established 65536 bind 65536)
[    8.671821] UDP hash table entries: 4096 (order: 1, 131072 bytes)
[    8.745673] UDP-Lite hash table entries: 4096 (order: 1, 131072 bytes)
[    8.824692] NET: Registered protocol family 1
[    8.877054] futex hash table entries: 1024 (order: 1, 131072 bytes)
[    8.953001] workingset: timestamp_bits=40 max_order=17 bucket_order=0
[    9.029206] zbud: loaded
[    9.061660] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[    9.153560] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    9.240707] io scheduler noop registered
[    9.287809] io scheduler deadline registered
[    9.339188] io scheduler cfq registered (default)
[    9.460018] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[   13.353990] loop: module loaded
[   13.389999] qla1280: QLA1040 found on PCI bus 0, dev 0
[   13.452304] qla1280 0002:00:00.0: can't ioremap BAR 1: [??? 0x00000000 flags 0x0]
[   13.541691] qla1280: Unable to map I/O memory
[   13.594332] qla1280: QLA1040 found on PCI bus 0, dev 1
[   13.656402] qla1280 0002:00:01.0: can't ioremap BAR 1: [??? 0x00000000 flags 0x0]
[   13.745916] qla1280: Unable to map I/O memory
[   13.809256] Data bus error, epc == a8000000003fdb74, ra == a8000000003fe4f0
[   13.891137] Oops[#1]:
[   13.918372] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.10.0-rc7 #2
[   13.993771] task: a8000000e077dc80 task.stack: a8000000e0780000
[   14.064988] $ 0   : 0000000000000000 ffffffff94001ce0 00000000000800c0 ffffffffefffffff
[   14.161342] $ 4   : 9200000000020178 ffffffff94001ce1 ffffffffa8000000 0000000000000000
[   14.257697] $ 8   : 0000000000000001 a8000001c09d4070 a8000001c09d4060 ffffffffffffffff
[   14.354051] $12   : 0000000000000000 a80000000032f1ec 0000000000000000 a8000000e078fd74
[   14.450406] $16   : 9200000000000000 a8000001c08b4800 0000000000020000 a8000001c09d4700
[   14.546761] $20   : 0000000000000000 a8000000007e0000 a8000001c09d4000 a8000000006f045c
[   14.643116] $24   : 0000000000000000 0000000000680000
[   14.739472] $28   : a8000000e0780000 a8000000e078fc30 a800000000760000 a8000000003fe4f0
[   14.835827] Hi    : 0000000000000000
[   14.878768] Lo    : 000000000000b400
[   14.921724] epc   : a8000000003fdb74 ioc3_probe+0x1dc/0xda0
[   14.988744] ra    : a8000000003fe4f0 ioc3_probe+0xb58/0xda0
[   15.055767] Status: 94001ce3 KX SX UX KERNEL EXL IE
[   15.115467] Cause : 0000d01c (ExcCode 07)
[   15.163644] PrId  : 00000f14 (R14000)
[   15.207636] Process swapper/0 (pid: 1, threadinfo=a8000000e0780000, task=a8000000e077dc80, tls=0000000000000000)
[   15.330169] Stack : a8000001c06b6978 a8000001c0d6cc60 a8000001c08b4800 a8000001c08b4800
[   15.426524]         a8000000006a2dc0 a800000000696bd0 0000000000000000 a8000000007e0000
[   15.522879]         a8000000006a2e28 a8000000006f045c a800000000760000 a800000000371f94
[   15.619234]         a8000001c08b4898 a800000000372a28 a8000000006a2dc0 a8000001c08b4800
[   15.715588]         a800000000588550 a8000000003c2824 a80000000062a410 a8000000007d0000
[   15.811944]         a8000001c08b4898 a8000000007e0000 0000000000000000 a8000000003c2e58
[   15.908298]         a8000001c08b4898 a8000000006a2e28 a8000001c08b48f8 a800000000696bd0
[   16.004653]         a80000000071bf30 a800000000760000 0000000000000007 a8000000003c30c8
[   16.101008]         0000000000000000 a8000000006a2e28 a8000000003c2fd8 a8000000003c0ca4
[   16.197364]         a8000000e093a0a0 a8000001c0708568 a8000000006a2e28 a8000001c0d19f00
[   16.293718]         ...
[   16.323046] Call Trace:
[   16.352376] [<a8000000003fdb74>] ioc3_probe+0x1dc/0xda0
[   16.415232] [<a800000000371f94>] local_pci_probe+0x2c/0x98
[   16.481201] [<a800000000372a28>] pci_device_probe+0x140/0x1b8
[   16.550325] [<a8000000003c2e58>] really_probe+0x1b0/0x330
[   16.615259] [<a8000000003c30c8>] __driver_attach+0xf0/0xf8
[   16.681248] [<a8000000003c0ca4>] bus_for_each_dev+0x6c/0xb8
[   16.748272] [<a8000000003c1960>] bus_add_driver+0x1c8/0x330
[   16.815301] [<a8000000003c3d34>] driver_register+0x84/0x150
[   16.882340] [<a8000000006f109c>] do_one_initcall+0xc4/0x1b4
[   16.949360] [<a8000000006f1478>] kernel_init_freeable+0x2ec/0x3e8
[   17.022675] [<a80000000051fdd0>] kernel_init+0x10/0x1c0
[   17.085519] [<a800000000025968>] ret_from_kernel_thread+0x14/0x1c
[   17.158823] Code: 8e020034  0204202d  ae000058 <8e020058> ae00005c  8e02005c  8e0200b8  00431024  ae0200b8
[   17.276122]
[   17.294049] ---[ end trace 027164ad3883b4c1 ]---
[   17.349485] Kernel panic - not syncing: Fatal exception
[   17.412320] Reboot started from CPU 0

[-- Attachment #7: ip27-dmesg-claim_pci-20170208.txt --]
[-- Type: text/plain, Size: 15400 bytes --]

[    0.000000] Linux version 4.10.0-rc7 (root@helcaraxe) (gcc version 6.3.0 (Gentoo Hardened 6.3.0 p1.0) ) #3 SMP Wed Feb 8 04:13:21 EST 2017
[    0.000000] ARCH: SGI-IP27
[    0.000000] PROMLIB: ARC firmware Version 64 Revision 0
[    0.000000] Discovered 4 cpus on 2 nodes
[    0.000000] node_distance: router_a NULL
[    0.000000] node_distance: router_a NULL
[    0.000000] node_distance: router_a NULL
[    0.000000] node_distance: router_a NULL
[    0.000000] ************** Topology ********************
[    0.000000]
[    0.000000] 00
[    0.000000] 01
[    0.000000]
[    0.000000] 00
[    0.000000] 255
[    0.000000] 255
[    0.000000]
[    0.000000] 01
[    0.000000] 255
[    0.000000] 255
[    0.000000]
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00000f14 (R14000)
[    0.000000] FPU revision is: 00000900
[    0.000000] Checking for the multiply/shift bug... no.
[    0.000000] Checking for the daddiu bug... no.
[    0.000000] IP27: Running on node 0.
[    0.000000] Node 0 has a primary CPU, CPU is running.
[    0.000000] Node 0 has a secondary CPU, CPU is running.
[    0.000000] Machine is in M mode.
[    0.000000] Cpu 0, Nasid 0x0: partnum 0x0 is
[    0.000000] is xbow
[    0.000000] Cpu 0, Nasid 0x0, widget 0x8 (partnum 0xc102) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xb (partnum 0xc002) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xc (partnum 0xc002) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xd (partnum 0xc003) is
[    0.000000] Cpu 0, Nasid 0x0, widget 0xf (partnum 0xc002) is
[    0.000000] CPU 0 clock is 500MHz.
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 00000000006e0000 @ 0000000000010000 (usable)
[    0.000000]  memory: 0000000000050000 @ 00000000006f0000 (usable after init)
[    0.000000] cma: Reserved 512 MiB at 0x00000001e0000000
[    0.000000] REPLICATION: ON nasid 0, ktext from nasid 0, kdata from nasid 0
[    0.000000] REPLICATION: ON nasid 1, ktext from nasid 0, kdata from nasid 0
[    0.000000] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    0.000000] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    0.000000] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x00000001ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   node   1: [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000] Initmem setup node 1 [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] percpu: Embedded 2 pages/cpu @a800000001430000 s60064 r0 d71008 u131072
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 130944
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=dksc(0,1,0) console=ttyS0,9600 root=/dev/md0
[    0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes)
[    0.000000] Memory: 7846784K/8388608K available (5172K kernel code, 678K rwdata, 1116K rodata, 320K init, 700K bss, 17536K reserved, 524288K cma-reserved)
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000]  RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4
[    0.000000] NR_IRQS:256
[    0.000000] clocksource: HUB-RT: mask: 0xfffffffffffff max_cycles: 0x127350b88, max_idle_ns: 1763180808480 ns
[    0.000012] sched_clock: 52 bits at 1250kHz, resolution 800ns, wraps every 4398046510800ns
[    0.103168] Console: colour dummy device 80x25
[    0.156248] Calibrating delay loop... 749.56 BogoMIPS (lpj=3747840)
[    0.318496] pid_max: default: 32768 minimum: 301
[    0.378314] Dentry cache hash table entries: 1048576 (order: 7, 8388608 bytes)
[    0.577879] Inode-cache hash table entries: 524288 (order: 6, 4194304 bytes)
[    0.720390] Mount-cache hash table entries: 16384 (order: 1, 131072 bytes)
[    0.801234] Mountpoint-cache hash table entries: 16384 (order: 1, 131072 bytes)
[    0.900782] Checking for the daddi bug... no.
[    0.957853] smp: Bringing up secondary CPUs ...
[    1.012864] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    1.012880] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    1.012888] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    1.016053] CPU 1 clock is 500MHz.
[    1.016076] CPU1 revision is: 00000f14 (R14000)
[    1.016080] FPU revision is: 00000900
[    1.054200] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    1.054220] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    1.054229] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    1.057457] Cpu 2, Nasid 0x1: partnum 0x0 is
[    1.057480] is xbow
[    1.057523] CPU 2 clock is 500MHz.
[    1.057540] CPU2 revision is: 00000f14 (R14000)
[    1.057544] FPU revision is: 00000900
[    1.094540] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    1.094559] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    1.094568] Unified secondary cache 8192kB 2-way, linesize 128 bytes.
[    1.097801] CPU 3 clock is 500MHz.
[    1.097830] CPU3 revision is: 00000f14 (R14000)
[    1.097833] FPU revision is: 00000900
[    1.131379] smp: Brought up 2 nodes, 4 CPUs
[    2.425205] devtmpfs: initialized
[    2.464371] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    2.586100] xor: measuring software checksum speed
[    2.741957]    8regs     :   710.400 MB/sec
[    2.890575]    8regs_prefetch:   684.800 MB/sec
[    3.043334]    32regs    :   659.200 MB/sec
[    3.171378] random: fast init done
[    3.191971]    32regs_prefetch:   640.000 MB/sec
[    3.191979] xor: using function: 8regs (710.400 MB/sec)
[    3.329660] NET: Registered protocol family 16
[    3.442177] cpuidle: using governor ladder
[    3.539610] cpuidle: using governor menu
[    3.862130] raid6: int64x1  gen()   304 MB/s
[    4.082259] raid6: int64x1  xor()    65 MB/s
[    4.301996] raid6: int64x2  gen()   445 MB/s
[    4.521803] raid6: int64x2  xor()   152 MB/s
[    4.741596] raid6: int64x4  gen()   387 MB/s
[    4.961463] raid6: int64x4  xor()   141 MB/s
[    5.181241] raid6: int64x8  gen()   315 MB/s
[    5.401104] raid6: int64x8  xor()   116 MB/s
[    5.450556] raid6: using algorithm int64x2 gen() 445 MB/s
[    5.515490] raid6: .... xor() 152 MB/s, rmw enabled
[    5.574141] raid6: using intx1 recovery algorithm
[    5.631217] SCSI subsystem initialized
[    5.676508] PCI host bridge to bus 0002:00
[    5.724989] pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff]
[    5.824467] pci_bus 0002:00: root bus resource [io  0x920000000fa00000-0x920000000fbfffff]
[    5.923966] pci_bus 0002:00: root bus resource [bus 02-ff]
[    5.992099] pci 0002:00:00.0: can't claim BAR 0 [io  0xf200000-0xf2000ff]: no compatible bridge window
[    6.102330] pci 0002:00:00.0: can't claim BAR 1 [mem 0x0f200000-0x0f200fff]: no compatible bridge window
[    6.216484] pci 0002:00:00.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window
[    6.335894] pci 0002:00:01.0: can't claim BAR 0 [io  0xf400000-0xf4000ff]: no compatible bridge window
[    6.447946] pci 0002:00:01.0: can't claim BAR 1 [mem 0x0f400000-0x0f400fff]: no compatible bridge window
[    6.562105] pci 0002:00:01.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window
[    6.681501] pci 0002:00:02.0: can't claim BAR 0 [mem 0x0f600000-0x0f6fffff]: no compatible bridge window
[    6.795661] pci 0002:00:06.0: can't claim BAR 0 [mem 0x0fa00000-0x0fafffff]: no compatible bridge window
[    6.909821] pci 0002:00:07.0: can't claim BAR 0 [mem 0x00000000-0x00001fff]: no compatible bridge window
[    7.024114] PCI host bridge to bus 0001:00
[    7.073200] pci_bus 0001:00: root bus resource [mem 0x920000000c200000-0x920000000c9fffff]
[    7.172704] pci_bus 0001:00: root bus resource [io  0x920000000ca00000-0x920000000cbfffff]
[    7.272200] pci_bus 0001:00: root bus resource [bus 01-ff]
[    7.339148] pci 0001:00:00.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window
[    7.452346] pci 0001:00:01.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window
[    7.566625] PCI host bridge to bus 0000:00
[    7.615733] pci_bus 0000:00: root bus resource [mem 0x920000000b200000-0x920000000b9fffff]
[    7.715232] pci_bus 0000:00: root bus resource [io  0x920000000ba00000-0x920000000bbfffff]
[    7.814720] pci_bus 0000:00: root bus resource [bus 00-ff]
[    7.881790] clocksource: Switched to clocksource HUB-RT
[    7.954615] NET: Registered protocol family 2
[    8.005935] TCP established hash table entries: 65536 (order: 3, 524288 bytes)
[    8.095191] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes)
[    8.179264] TCP: Hash tables configured (established 65536 bind 65536)
[    8.256130] UDP hash table entries: 4096 (order: 1, 131072 bytes)
[    8.329975] UDP-Lite hash table entries: 4096 (order: 1, 131072 bytes)
[    8.409000] NET: Registered protocol family 1
[    8.461360] futex hash table entries: 1024 (order: 1, 131072 bytes)
[    8.537336] workingset: timestamp_bits=40 max_order=17 bucket_order=0
[    8.613510] zbud: loaded
[    8.645971] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[    8.737724] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    8.824909] io scheduler noop registered
[    8.872006] io scheduler deadline registered
[    8.923377] io scheduler cfq registered (default)
[    9.043624] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    9.133084] loop: module loaded
[    9.169123] qla1280: QLA1040 found on PCI bus 0, dev 0
[    9.230776] PCI: Enabling device 0002:00:00.0 (0006 -> 0007)
[    9.299489] qla1280 0002:00:00.0: can't ioremap BAR 1: [mem size 0x00001000]
[    9.383644] qla1280: Unable to map I/O memory
[    9.436303] qla1280: QLA1040 found on PCI bus 0, dev 1
[    9.497807] PCI: Enabling device 0002:00:01.0 (0006 -> 0007)
[    9.566389] qla1280 0002:00:01.0: can't ioremap BAR 1: [mem size 0x00001000]
[    9.650713] qla1280: Unable to map I/O memory
[    9.872302] console [ttyS0] enabledd¦[    9.766238] serial8250: ttyS0 at MMIO 0x920000000f620178 (irq = 0, base_baud = 458333) is a 16550A
[    9.872302] console [ttyS0] enabled
[    9.955067] bootconsole [early0] disabled
[    9.955067] bootconsole [early0] disabled
[   10.073000] serial8250: ttyS1 at MMIO 0x920000000f620170 (irq = 0, base_baud = 458333) is a 16550A
[   10.216100] Found DS1981U NIC
[   10.216112]  registration number 5e:57:03:00:70:5e, CRC 52
[   10.251922] .
[   10.353129] Ethernet address is 08:00:69:05:64:74.
[   10.412166] ioc3-eth 0002:00:02.0 eth0: link up, 100Mbps, full-duplex, lpa 0x8DE1
[   10.502356] eth0: Using PHY 31, vendor 0x20005c0, model 0, rev 0.
[   10.575863] eth0: IOC3 SSRAM has 128 kbyte.
[   10.646991] serial8250: ttyS2 at MMIO 0x920000000fa20178 (irq = 0, base_baud = 458333) is a 16550A
[   10.775950] serial8250: ttyS3 at MMIO 0x920000000fa20170 (irq = 0, base_baud = 458333) is a 16550A
[   10.887716] NIC search failed (not fatal).
[   10.939561] NIC search failed (not fatal).
[   10.989434] Failed to read MAC address
[   11.034612] Ethernet address is 00:00:00:00:00:00.
[   11.093536] ioc3-eth 0002:00:06.0 eth1: link down
[   11.150237] eth1: Using PHY 31, vendor 0x0, model 0, rev 0.
[   11.217496] eth1: IOC3 SSRAM has 64 kbyte.
[   11.267928] NET: Registered protocol family 17
[   11.322106] rtc-m48t35 rtc-m48t35: rtc core: registered m48t35 as rtc0
[   11.402360] rtc-m48t35 rtc-m48t35: setting system clock to 2017-02-08 09:13:55 UTC (1486545235)
[   11.509220] md: Waiting for all devices to be available before autodetect
[   11.591060] md: If you don't use raid, use raid=noautodetect
[   11.660780] md: Autodetecting RAID arrays.
[   11.710168] md: autorun ...
[   11.743888] md: ... autorun DONE.
[   11.784804] isofs_fill_super: bread failed, dev=md0, iso_blknum=16, block=32
[   11.870695] UDF-fs: error (device md0): udf_read_tagged: read failed, block=256, location=256
[   11.973747] UDF-fs: error (device md0): udf_read_tagged: read failed, block=512, location=512
[   12.077091] UDF-fs: error (device md0): udf_read_tagged: read failed, block=256, location=256
[   12.179999] UDF-fs: error (device md0): udf_read_tagged: read failed, block=512, location=512
[   12.282824] UDF-fs: warning (device md0): udf_fill_super: No partition found (1)
[   12.375128] VFS: Cannot open root device "md0" or unknown-block(9,0): error -5
[   12.462245] Please append a correct "root=" boot option; here are the available partitions:
[   12.563018] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(9,0)
[   12.662695] Reboot started from CPU 0
[   12.706844] ------------[ cut here ]------------
[   12.762483] WARNING: CPU: 0 PID: 1 at kernel/smp.c:405 smp_call_function_many+0x18c/0x430
[   12.861020] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.10.0-rc7 #3
[   12.936526] Stack : 0000000000000006 a8000000007714a2 0000000000000000 a80000000009ed58
[   13.032980]         ffffffff94001ce0 0000000000000000 0000000000000000 a800000000770000
[   13.129440]         a80000000067bcf7 a800000000610a50 a8000000e0772880 0000000000000000
[   13.225900]         0000000000000001 a800000000769670 a800000000030460 0000000000000000
[   13.322360]         a800000000670000 a800000000102298 a8000000e078fb78 a8000000000494dc
[   13.418820]         ffffffff94001ce0 a8000000000a09a0 00000000000000f3 a800000000610a50
[   13.515279]         0000000000000000 0000000000000001 0000000000000000 0000000000000000
[   13.611740]         0000000000000000 a8000000e078fac0 0000000000000000 a80000000031e6ec
[   13.708199]         0000000000000000 0000000000000000 0000000000000000 00ffffff94001ce0
[   13.804660]         a800000000690000 a80000000002bb88 0000000000000000 a80000000031e6ec
[   13.901120]         ...
[   13.930552] Call Trace:
[   13.959995] [<a80000000002bb88>] show_stack+0x88/0xb8
[   14.020854] [<a80000000031e6ec>] dump_stack+0xac/0xe0
[   14.081687] [<a8000000000498a4>] __warn+0x144/0x168
[   14.140444] [<a8000000000ce86c>] smp_call_function_many+0x18c/0x430
[   14.215957] [<a8000000000ceb48>] smp_call_function+0x38/0x48
[   14.284137] [<a80000000001ec18>] ip27_machine_restart+0x30/0x50
[   14.355462] [<a800000000029ea4>] machine_restart+0x34/0x90
[   14.421552] [<a8000000001020cc>] panic+0x29c/0x324
[   14.479268] [<a8000000006f1c1c>] mount_block_root+0x418/0x444
[   14.548488] [<a8000000006f1f2c>] prepare_namespace+0x21c/0x294
[   14.618764] [<a8000000006f1544>] kernel_init_freeable+0x3b8/0x3e8
[   14.692187] [<a80000000051fdf0>] kernel_init+0x10/0x1c0
[   14.755124] [<a800000000025968>] ret_from_kernel_thread+0x14/0x1c
[   14.828542] ---[ end trace 6891c236c4efea54 ]---

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Joshua Kinard]

On 02/07/2017 13:29, Bjorn Helgaas wrote:

[snip]

>>
>> However, IP27 is completely different in this regard.  Instead of using
>> ioremapped addresses for I/O, IP27 has a dedicated address range,
>> 0x92xxxxxxxxxxxxxx, that is used for all I/O access.  Since this is
>> uncached physical address space, the generic MIPS PCI code will not
>> probe it correctly and thus, the original behavior of PCI_PROBE_ONLY
>> needs to be restored only for the IP27 platform to bypass this logic
>> and have working PCI, at least for the IO6/IO6G board that houses the
>> base devices, until a better solution is found.
> 
> It sounds like there's something different about how ioremap() works
> on these platforms and PCI probing is tripping over that.  I'd really
> like to understand more about this difference to see if we can
> converge that instead of adding back the PCI_PROBE_ONLY usage.
> 
> Drivers shouldn't know whether they're running on IP27 or IP30, and
> they should be using ioremap() in both cases.  Does ioremap() work
> differently on IP27 and IP30?  Does this have something to do with
> plat_ioremap() or fixup_bigphys_addr()?

Okay, I think I have a rough idea on the differences with the I/O addresses on
IP27 and IP30.  It all boils down to the return value of a macro called
NODE_SWIN_BASE, which operates differently on each platform.

I don't fuilly understand //why// SGI did it a certain way, but HUB in IP27
gives you two sets of "windows" into Crosstalk space, "Big" (BWIN) and "Little"
(LWIN).  IP30's HEART chip has three sets of these windows (as I recall now
from and appendix in the BRIDGE docs), Big (BWIN), medium (MWIN) and small (SWIN).

So on IP27, HUB has seven BWIN spaces at 512MB each, and one LWIN space at
256MB (with a second LWIN that's aliased to the first).  The LWIN is broken
into sixteen 16MB spaces, which, I think, is what we're calling "Small windows"
or SWIN's (Ralf, correct me if wrong).

IP30's HEART allocates 64G of space per Xtalk widget for BWIN, 2G/widget for
MWIN, and 16MB/widget for SWIN.  As such, since HUB doesn't have an MWIN, you
only find definitions of NODE_BWIN_BASE and NODE_SWIN_BASE in either IP27 or
IP30's sources.  Specifically, our focus is really only on NODE_SWIN_BASE,
because the BWIN spaces need to be mapped into translation tables somewhere,
and Linux lacks that capability.  There is also a corresponding
RAW_NODE_SWIN_BASE for both platforms that gives back the full address in
uncached physical address space.

IP30's definition of NODE_SWIN_BASE is:
#define NODE_SWIN_BASE(nasid, widget) \
	(0x0000000010000000 | (((unsigned long)(widget)) << 24))

And IP30's RAW_NODE_SWIN_BASE:
#define RAW_NODE_SWIN_BASE(nasid, widget) \
	(0x9000000010000000 | (((unsigned long)(widget)) << 24))


Contrast that to IP27's NODE_SWIN_BASE (for the __ASSEMBLY__ case)
in arch/mips/include/asm/sn/sn0/addrs.h:
#define NODE_SWIN_BASE(nasid, widget) \
	(NODE_IO_BASE(nasid) + (UINT64_CAST(widget) << SWIN_SIZE_BITS))

(IP27's RAW_NODE_SWIN_BASE is the same as above)

SWIN_SIZE_BITS is '24', so it matches IP30's.  The difference is the value of
NODE_IO_BASE that takes a nasid (NUMA address space identifier or node id) and
returns the correct address in IP27's memory space to get to a specific node's
I/O Base, which is 0x9200000000000000 plus the node ID.  Merged with the widget
ID shifted by 24 bits, and that's why we're getting the 0x92xx addresses.

Basically, IP30 is returning the widget offset //without// the I/O base
factored in, and that's why RAW_NODE_SWIN_BASE is provided to get the widget
address in uncached physical address space.  Whereas IP27's NODE_SWIN_BASE is
an alias to its RAW_NODE_SWIN_BASE definition and turning 0x92xxx addresses.

---

Then we look in arch/mips/include/asm/io.h at the function __ioremap_mode and
see this construct near the top:

        if (cpu_has_64bit_addresses) {
                u64 base = UNCAC_BASE;

                /*
                 * R10000 supports a 2 bit uncached attribute therefore
                 * UNCAC_BASE may not equal IO_BASE.
                 */
                if (flags == _CACHE_UNCACHED)
                        base = (u64) IO_BASE;
                return (void __iomem *) (unsigned long) (base + offset);

For IP30, when we set up the BRIDGE parameters, we use NODE_SWIN_BASE to set
the BRIDGE's MEM and IO start/end ranges and thus get back an address where the
upper bits are all zero's.  Which the PCI layer then fiddles with to yield an
address like 0x000000f100000000.  When that gets run through ioremap() by a
driver's probe, we should be getting something like 0x900000f100000000, because
IP30 uses the generic definition of IO_BASE defined in MIPS
asm-generic/spaces.h.  And the devices probe correctly.

For IP27, I program BRIDGE the same way, but because IP27's NODE_SWIN_BASE is
returning the 0x92xxx addresses in uncached space, my guess is ioremap(), as
called by the drivers, is doubling that, yielding addresses like 0x24xxx, which
unprotected, should generate an unhandled kernel unaligned access or such.
Properly protected, attempts to access such an address probably just fail outright.

---

So my thinking, which I'll test out over the weekend, is whether we can change
IP27's NODE_SWIN_BASE to behave more like IP30's and just return the widget id
shifted by 24 bits, then let ioremap() calls actually do the conversion to the
0x92xxx IO_BASE address space.  There's only a handful of places in IP27's code
and headers where NODE_SWIN_BASE is actually used, so I'll have to see which
ones need to become RAW_NODE_SWIN_BASE (e.g., for direct HUB or XBOW access),
and which can stay as NODE_SWIN_BASE.

That would then allow the correct operation of ioremap() to return the 0x92xxx
address space for drivers and I guess PCI to probe correctly.  We'll find out!

--J

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Joshua Kinard]

On 02/09/2017 16:29, Bjorn Helgaas wrote:

[snip]

>>>> However, IP27 is completely different in this regard.  Instead of 
>>>> using ioremapped addresses for I/O, IP27 has a dedicated address 
>>>> range, 0x92xxxxxxxxxxxxxx, that is used for all I/O access.  Since 
>>>> this is uncached physical address space, the generic MIPS PCI code 
>>>> will not probe it correctly and thus, the original behavior of 
>>>> PCI_PROBE_ONLY needs to be restored only for the IP27 platform to 
>>>> bypass this logic and have working PCI, at least for the IO6/IO6G 
>>>> board that houses the base devices, until a better solution is found.
>>> 
>>> It sounds like there's something different about how ioremap() works on
>>>  these platforms and PCI probing is tripping over that.  I'd really like
>>>  to understand more about this difference to see if we can converge that
>>>  instead of adding back the PCI_PROBE_ONLY usage.
>> 
>> I'd need to go and dig around in the IP27 headers again for this machine 
>> to see what ioremap() is actually doing, but I *think* it returns uncached
>> physical addresses in most instances because of a special feature of the
>> CPU, the R10000-family, which makes uncached access very fast.  I think
>> only the IP27 platform uses this capability.  Other R1x0-based systems
>> don't (although, I might be wrong about IP27's successor in IP35).
> 
> ioremap() must return a CPU virtual address.  It can take advantage of 
> arch-specific features, special address ranges that are uncached, special 
> identity mappings, or whatever, but from the caller's point of view, the 
> return value must be usable as a virtual address without any special 
> handling.

Apparently, MIPS's implementation of ioremap does not guarantee that a virtual
address is always returned.  Quoting from arch/mips/include/asm/io.h around
line #236:

/*
 * ioremap     -   map bus memory into CPU space
 * @offset:    bus address of the memory
 * @size:      size of the resource to map
 *
 * ioremap performs a platform specific sequence of operations to
 * make bus memory CPU accessible via the readb/readw/readl/writeb/
 * writew/writel functions and the other mmio helpers. The returned
 * address is not guaranteed to be usable directly as a virtual
 * address.
 */

It looks like the qla1280.c driver calls pci_ioremap_bar(), which calls
ioremap_nocache(), and on MIPS, that's just an alias (as far as I can tell) to
ioremap(), and it has the same limitation, that the address returned may not
be usable as a CPU virtual address.


> This is from ip27-dmesg-working_pci-20170208.txt:
> 
>   PCI host bridge to bus 0002:00
>   pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff]
>   pci_bus 0002:00: root bus resource [io  0x920000000fa00000-0x920000000fbfffff]
>   pci_bus 0002:00: root bus resource [bus 02-ff]
>   pci 0002:00:00.0: [1077:1020] type 00 class 0x010000
>   pci 0002:00:00.0: reg 0x10: [io  0xf200000-0xf2000ff]
>   pci 0002:00:00.0: reg 0x14: [mem 0x0f200000-0x0f200fff]
>   pci 0002:00:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
> 
> There's something wrong here: all the resources printed above, including the
> BARs, should be CPU physical addresses (as are all the /proc/iomem 
> addresses).  The "root bus resources" are windows through the host bridge to
> PCI.  The BAR resources should be inside those windows.
> 
> I'm sure the bridge translates between CPU physical addresses and PCI bus 
> addresses, probably by chopping off all those high-order bits.  The PCI core
> needs to know about this, and it looks like pcibios_scanbus() is trying to
> tell it by using pci_add_resource_offset().  But the PCI core isn't getting
> the message because we don't see the "(bus address [%#010llx-%#010llx])"
> being printed by pci_register_host_bridge().
> 
> A correct dmesg log would look something like this:
> 
>   PCI host bridge to bus 0002:00
>   pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff]
> (bus address [0x0f200000-0x0f9fffff])
>   pci 0002:00:00.0: reg 0x14: [mem 0x920000000f200000-0x920000000f200fff]
> 
> This would mean that for 0002:00:00.0, pdev->resource[1] contains the CPU 
> physical address 0x920000000f200000, and the host bridge translates that to 
> 0x0f200000, which is what the 32-bit BAR1 would contain.
> 
> I think "lspci" should show you the CPU addresses (0x920000000f200000) and 
> "lspci -b" should show you the actual BAR values (0x0f200000).
> 
> Your dmesg log is showing you 0x0f200000, which is probably the BAR value. 
> __pci_read_base() reads the BAR value and runs it through 
> pcibios_bus_to_resource(), which *should* be converting it to 
> 0x920000000f200000 (the reverse of the translation done by the host bridge).
> But since the PCI core doesn't know about that translation, 
> pcibios_bus_to_resource() does nothing.
> 
> So I think there's something wrong with the way you're using 
> pci_add_resource_offset().

This is basically the same conclusion I've come to.  I've found a "partial"
solution, but I'm pretty sure it's wrong.

It seems the reason IP30 works without PCI_PROBE_ONLY now is because it doesn't
need special casing on the addresses in order to work.  IP27, on the other
hand, needs all I/O accesses to go through the 0x92xxx range or you'll run into
hardware-enforced barriers that will panic the kernel.

First attempt I did was to simply subtract NODE_IO_BASE(_n) from the address
values that ultimately get fed to pci_add_resource_offset().  That yields this
kind of output:

[    8.065146] PCI host bridge to bus 0000:00
[    8.113572] pci_bus 0000:00: root bus resource [mem 0x0b200000-0x0b9fffff]
[    8.196284] pci_bus 0000:00: root bus resource [io  0xba00000-0xbbfffff]
[    8.276934] pci_bus 0000:00: root bus resource [bus 00-ff]
[    8.343199] PCI host bridge to bus 0001:00
[    8.392144] pci_bus 0001:00: root bus resource [mem 0x0c200000-0x0c9fffff]
[    8.474899] pci_bus 0001:00: root bus resource [io  0xca00000-0xcbfffff]
[    8.555520] pci_bus 0001:00: root bus resource [bus 01-ff]
[    8.622536] pci 0001:00:00.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window
[    8.735665] pci 0001:00:01.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window
[    8.850128] PCI host bridge to bus 0002:00
[    8.899049] pci_bus 0002:00: root bus resource [mem 0x0f200000-0x0f9fffff]
[    8.981794] pci_bus 0002:00: root bus resource [io  0xfa00000-0xfbfffff]
[    9.062413] pci_bus 0002:00: root bus resource [bus 02-ff]
[    9.130425] pci 0002:00:00.0: can't claim BAR 0 [io  0xf200000-0xf2000ff]: no compatible bridge window
[    9.240674] pci 0002:00:00.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window
[    9.360079] pci 0002:00:01.0: can't claim BAR 0 [io  0xf400000-0xf4000ff]: no compatible bridge window
[    9.472148] pci 0002:00:01.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window
[    9.591555] pci 0002:00:06.0: can't claim BAR 0 [mem 0x0fa00000-0x0fafffff]: no compatible bridge window
[    9.705685] pci 0002:00:07.0: can't claim BAR 0 [mem 0x00000000-0x00001fff]: no compatible bridge window
[    9.819933] qla1280: QLA1040 found on PCI bus 0, dev 0
[    9.881644] PCI: Enabling device 0002:00:00.0 (0006 -> 0007)

Despite the above errors, it succeeds in probing and finds devices and boots
into userland.  The error message above stems from 'pci_claim_resource' in
drivers/pci/setup-res.c because it can't seem to find the parent bridge.  Not
sure if that's good or bad.  But it does boot, and it would remove one of the
two #ifdef hacks in my patch.

So I next looked at the offset bit you mentioned, and after some fiddling
around in pci-bridge.c, got this dmesg output, which matches what you say
should be the correct dmesg output:

[    8.095560] PCI host bridge to bus 0000:00
[    8.143949] pci_bus 0000:00: root bus resource [mem 0x920000000b200000-0x920000000b9fffff] (bus address [0x0b200000-0x0b9fffff])
[    8.283216] pci_bus 0000:00: root bus resource [io  0x920000000ba00000-0x920000000bbfffff] (bus address [0xba00000-0xbbfffff])
[    8.420442] pci_bus 0000:00: root bus resource [bus 00-ff]
[    8.486711] PCI host bridge to bus 0001:00
[    8.535656] pci_bus 0001:00: root bus resource [mem 0x920000000c200000-0x920000000c9fffff] (bus address [0x0c200000-0x0c9fffff])
[    8.674965] pci_bus 0001:00: root bus resource [io  0x920000000ca00000-0x920000000cbfffff] (bus address [0xca00000-0xcbfffff])
[    8.812144] pci_bus 0001:00: root bus resource [bus 01-ff]
[    8.879140] pci 0001:00:00.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window
[    8.992298] pci 0001:00:01.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window
[    9.106732] PCI host bridge to bus 0002:00
[    9.155679] pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff] (bus address [0x0f200000-0x0f9fffff])
[    9.294974] pci_bus 0002:00: root bus resource [io  0x920000000fa00000-0x920000000fbfffff] (bus address [0xfa00000-0xfbfffff])
[    9.432168] pci_bus 0002:00: root bus resource [bus 02-ff]
[    9.500118] pci 0002:00:00.0: can't claim BAR 0 [io  0xf200000-0xf2000ff]: no compatible bridge window
[    9.610326] pci 0002:00:00.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window
[    9.729688] pci 0002:00:01.0: can't claim BAR 0 [io  0xf400000-0xf4000ff]: no compatible bridge window
[    9.841764] pci 0002:00:01.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window
[    9.961151] pci 0002:00:06.0: can't claim BAR 0 [mem 0x0fa00000-0x0fafffff]: no compatible bridge window
[   10.075344] pci 0002:00:07.0: can't claim BAR 0 [mem 0x00000000-0x00001fff]: no compatible bridge window
[   10.189528] qla1280: QLA1040 found on PCI bus 0, dev 0
[   10.251300] PCI: Enabling device 0002:00:00.0 (0006 -> 0007)
[   10.320129] Unhandled kernel unaligned access[#1]:
[   10.376908] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.10.0-rc7-mipsgit-20170208 #24
[   10.471163] task: a8000000e077c880 task.stack: a8000000e0780000
[   10.542378] $ 0   : 0000000000000000 0000000000024680 0000000000000000 0000000000000000
[   10.638730] $ 4   : 000000000065e0c0 0000000000000010 a800000000450a18 240000000f20000a
[   10.735084] $ 8   : a8000000007a3978 a8000000e00c05b0 0000000000000000 a80000000145ef30
[   10.831440] $12   : ffffffff94005ce1 000000001000001e ffffffffffffff80 00000000007e0000
[   10.927796] $16   : a8000000e078f920 a800000000450a14 0000000000000000 ffffffffa4600000
[   11.024149] $20   : 240000000f20000a a8000000004509b8 a8000000006b9930 a8000000007eb1b8
[   11.120506] $24   : 0000000000940000 00000000007e0000
[   11.216860] $28   : a8000000e0780000 a8000000e078f8d0 a8000000e00c1650 a800000000024680
[   11.313216] Hi    : 0000000000000000
[   11.356156] Lo    : 0000000000002800
[   11.399117] epc   : a80000000002de44 do_ade+0x4f4/0x960
[   11.461952] ra    : a800000000024680 ret_from_exception+0x0/0x18
[   11.534204] Status: 94005ce3 KX SX UX KERNEL EXL IE
[   11.593903] Cause : 00008014 (ExcCode 05)
[   11.642080] BadVA : 240000000f20000b
[   11.685020] PrId  : 00000f14 (R14000)

And you can see that while the dmesg output is correct, the end result is not.
The issue goes back to what MIPS's ioremap() is doing when it encounters the
specialized cached-uncached attribute of the R1x0k CPUs on IP27, it tries
adding 0x9200000000000000, which is IP27's IO_BASE, to one of the addresses
during qla1280's probe, which wraps around to 0x240000000xxxxxxx and is
invalid in MIPS.  This leads to the unhandled kernel unaligned access panic.

I tried implementing a custom plat_ioremap() that would subtract the IO_BASE
off of the address being ioremapped by qla1280.  MIPS has a generic template
available to do this in arch/mips/include/asm/mach-generic/ioremap.h, but it
appears my gcc version is recent enough to have some new checks enabled that
choke on casting a phys_addr_t to void __iomem * (-Wint-to-pointer-cast), and
I've given up finding a workaround that doesn't involve invasive changes to
MIPS's ioremapping core.

--J

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Bjorn Helgaas]

It looks like IP30 is using some code that's not upstream yet, so I'll
just point out some things that don't look right.  I'm ignoring the
IP27 stuff here in case that has different issues.

Joshua wrote:
> On 02/07/2017 13:29, Bjorn Helgaas wrote:
>> Is there any chance you can collect complete dmesg logs and
>> /proc/iomem contents from IP27 and IP30?  Maybe "lspci -vv" output,
>> too?  I'm not sure where to look to understand the ioremap() behavior.
> 
> ...
> A note about IP30, since that's part of an external patch set, the
> dmesg output is a little different, as it is using the newer
> BRIDGE/Xtalk code I just sent in as patches.  ...
> 
> Also, IP30 doesn't use the PCI_PROBE_ONLY thing anymore.  This causes
> /proc/iomem to be far more detailed than it used to be.
> 
> IP30's /proc/iomem:
>     00000000-00003fff : reserved
>     1d200000-1d9fffff : Bridge MEM
>       d080000000-d080003fff : 0001:00:03.0
>       1d204000-1d204fff : 0001:00:02.0
>       1d205000-1d205fff : 0001:00:02.1
>       1d206000-1d206fff : 0001:00:02.2
>       1d207000-1d2070ff : 0001:00:02.3
>       1d207100-1d20717f : 0001:00:01.0
>     1f200000-1f9fffff : Bridge MEM
>       f080100000-f0801fffff : 0000:00:02.0
>       f080010000-f08001ffff : 0000:00:00.0
>       f080030000-f08003ffff : 0000:00:01.0
>       f080000000-f080000fff : 0000:00:00.0
>       f080020000-f080020fff : 0000:00:01.0
>     20004000-209b8fff : reserved
>       20004000-206acb13 : Kernel code
>       206acb14-208affff : Kernel data
>     209b9000-20efffff : System RAM
>     20f00000-20ffffff : System RAM
>     21000000-9fffffff : System RAM
>     f080100000-f0801fffff : ioc3

>From the dmesg you attached (ip30-dmesg-20170208.txt, PCI parts
appended below):

  pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff]
  pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff]

These are shown correctly in /proc/iomem.  It would be nice if the
/proc/iomem string identified which bridge was which (x86 uses "PCI
Bus 0000:00", "PCI Bus 0001:00", etc.) but that's not essential.

  pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff]
  ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit.

This is funky.  We read 0x1d200000 from the BAR (PCI bus addresses are
apparently identical to CPU physical addresses).  We claimed that
space from the host bridge window, so /proc/iomem would have looked
like this at that point:

  1d200000-1d9fffff : Bridge MEM
    1d200000-1d203fff : 0001:00:03.0

But it looks like whatever this ip30-bridge thing is overwrote the
0001:00:03.0 resource, which makes /proc/iomem wrong.

All the resources under the bridge to 0000:00 are wrong, too:

  1f200000-1f9fffff : Bridge MEM
    f080100000-f0801fffff : 0000:00:02.0
    f080010000-f08001ffff : 0000:00:00.0
    f080030000-f08003ffff : 0000:00:01.0
    f080000000-f080000fff : 0000:00:00.0
    f080020000-f080020fff : 0000:00:01.0

We read values from the BARs:

  pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff]
  pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff]
  pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref]
  pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff]
  pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref]
  pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff]
  pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff]

These aren't zero, so somebody apparently has assigned them, but they
aren't inside the host bridge window.  Therefore the PCI core assumes
they will not work, and it reassigns space from the window:

  pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff]
  pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref]
  pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref]
  pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff]
  pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff]

I don't know why we didn't assign space to 0000:00:03.0 BAR 1.

Anyway, it looks like they got inserted in /proc/iomem, then
overwritten by the ip30-bridge stuff again.

>From /proc/iomem, I would expect that only the USB devices (0001:00:02
functions 0, 1, 2, 3) would work.  But apparently your system does
actually work, so there must be corresponding funkiness somewhere else
that compensates for these resources.

I'm attaching the PCI parts of your ip30-dmesg-20170208.txt and the
complete ip30-lspci-20170208.txt below for context.

  pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff]
  pci_bus 0000:00: root bus resource [io  0x1fa00000-0x1fbfffff]
  pci_bus 0000:00: root bus resource [bus 00-ff]
  pci 0000:00:00.0: [1077:1020] type 00 class 0x010000
  pci 0000:00:00.0: reg 0x10: [io  0x200000-0x2000ff]
  pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff]
  pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref]
  pci 0000:00:01.0: [1077:1020] type 00 class 0x010000
  pci 0000:00:01.0: reg 0x10: [io  0x400000-0x4000ff]
  pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff]
  pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref]
  pci 0000:00:02.0: [10a9:0003] type 00 class 0xff0000
  pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff]
  pci 0000:00:03.0: [10a9:0005] type 00 class 0x000000
  pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff]
  pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff]
  pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref]
  pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref]
  pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff]
  pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff]
  pci 0000:00:00.0: BAR 0: assigned [io  0x1fa00000-0x1fa000ff]
  pci 0000:00:01.0: BAR 0: assigned [io  0x1fa00400-0x1fa004ff]

  ip30-bridge: 0000:00:00.0 Bar 0 with size 0x00000100 at bus 0x00000000 vma 0x000000f100000000 is Direct I/O.
  ip30-bridge: 0000:00:00.0 Bar 1 with size 0x00001000 at bus 0x00000000 vma 0x000000f080000000 is Direct 64-bit.
  ip30-bridge: 0000:00:00.0 Bar 6 with size 0x00010000 at bus 0x00010000 vma 0x000000f080010000 is Direct 64-bit.

  ip30-bridge: 0000:00:01.0 Bar 0 with size 0x00000100 at bus 0x00000100 vma 0x000000f100000100 is Direct I/O.
  ip30-bridge: 0000:00:01.0 Bar 1 with size 0x00001000 at bus 0x00020000 vma 0x000000f080020000 is Direct 64-bit.
  ip30-bridge: 0000:00:01.0 Bar 6 with size 0x00010000 at bus 0x00030000 vma 0x000000f080030000 is Direct 64-bit.

  ip30-bridge: 0000:00:02.0 Bar 0 with size 0x00100000 at bus 0x00100000 vma 0x000000f080100000 is Direct 64-bit.

  PCI host bridge to bus 0001:00
  pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff]
  pci_bus 0001:00: root bus resource [io  0x1da00000-0x1dbfffff]
  pci_bus 0001:00: root bus resource [bus 01-ff]
  pci 0001:00:01.0: [11fe:080e] type 00 class 0x078000
  pci 0001:00:01.0: reg 0x10: [mem 0x00200000-0x0020007f]
  pci 0001:00:01.0: reg 0x14: [io  0x200000-0x20007f]
  pci 0001:00:01.0: reg 0x18: [io  0x204000-0x2040ff]
  pci 0001:00:02.0: [10b9:5237] type 00 class 0x0c0310
  pci 0001:00:02.0: reg 0x10: [mem 0x00300000-0x00300fff]
  pci 0001:00:02.0: PME# supported from D0 D1 D3hot D3cold
  pci 0001:00:02.1: [10b9:5237] type 00 class 0x0c0310
  pci 0001:00:02.1: reg 0x10: [mem 0x00000000-0x00000fff]
  pci 0001:00:02.1: PME# supported from D0 D1 D3hot D3cold
  pci 0001:00:02.2: [10b9:5237] type 00 class 0x0c0310
  pci 0001:00:02.2: reg 0x10: [mem 0x00000000-0x00000fff]
  pci 0001:00:02.2: PME# supported from D0 D1 D3hot D3cold
  pci 0001:00:02.3: [10b9:5239] type 00 class 0x0c0320
  pci 0001:00:02.3: reg 0x10: [mem 0x00000000-0x000000ff]
  pci 0001:00:02.3: PME# supported from D0 D3hot D3cold
  pci 0001:00:03.0: [10a9:0009] type 00 class 0x020000
  pci 0001:00:03.0: reg 0x10: [mem 0x00400000-0x00403fff]
  pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff]
  pci 0001:00:02.0: BAR 0: assigned [mem 0x1d204000-0x1d204fff]
  pci 0001:00:02.1: BAR 0: assigned [mem 0x1d205000-0x1d205fff]
  pci 0001:00:02.2: BAR 0: assigned [mem 0x1d206000-0x1d206fff]
  pci 0001:00:01.0: BAR 2: assigned [io  0x1da00000-0x1da000ff]
  pci 0001:00:02.3: BAR 0: assigned [mem 0x1d207000-0x1d2070ff]
  pci 0001:00:01.0: BAR 0: assigned [mem 0x1d207100-0x1d20717f]
  pci 0001:00:01.0: BAR 1: assigned [io  0x1da00400-0x1da0047f]

  ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit.


Here's the ip30-lspci-20170208.txt attachment from your mail:


0000:00:00.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 0
        Region 0: I/O ports at f100000000 [size=257]
        Region 1: [virtual] Memory at f080000000 (32-bit, non-prefetchable) [size=4097]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at f080010000 [disabled] [size=65537]
        Kernel driver in use: qla1280
lspci: Unable to load libkmod resources: error -12

0000:00:01.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 1
        Region 0: I/O ports at f100000100 [size=257]
        Region 1: Memory at f080020000 (32-bit, non-prefetchable) [size=4097]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at f080030000 [disabled] [size=65537]
        Kernel driver in use: qla1280

0000:00:02.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. IOC3 I/O controller (rev 01)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Interrupt: pin A routed to IRQ 2
        Region 0: Memory at f080100000 (32-bit, non-prefetchable) [size=1048577]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Kernel driver in use: IOC3

0000:00:03.0 Non-VGA unclassified device: Silicon Graphics Intl. Corp. RAD Audio (rev c0)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=slow >TAbort+ <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 255
        Region 0: Memory at 00600000 (32-bit, non-prefetchable) [size=8193]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]

0001:00:01.0 Communication controller: Comtrol Corporation Device 080e (rev 01)
        Subsystem: Comtrol Corporation Device 080e
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 0
        Region 0: Memory at 1d207100 (32-bit, non-prefetchable) [size=129]
        Region 1: I/O ports at 1da00400 [size=129]
        Region 2: I/O ports at 1da00000 [size=257]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]

0001:00:02.0 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 16 (20000ns max)
        Interrupt: pin B routed to IRQ 0
        Region 0: Memory at 1d204000 (32-bit, non-prefetchable) [size=4097]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [60] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

0001:00:02.1 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin C routed to IRQ 0
        Region 0: Memory at 1d205000 (32-bit, non-prefetchable) [disabled] [size=4097]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [60] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

0001:00:02.2 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin D routed to IRQ 0
        Region 0: Memory at 1d206000 (32-bit, non-prefetchable) [disabled] [size=4097]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [60] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

0001:00:02.3 USB controller: ULi Electronics Inc. USB 2.0 Controller (rev 01) (prog-if 20 [EHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 0
        Region 0: Memory at 1d207000 (32-bit, non-prefetchable) [disabled] [size=257]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Debug port: BAR=1 offset=0090

0001:00:03.0 Ethernet controller: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet (rev 01)
        Subsystem: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64 (16000ns min), Cache Line Size: 128 bytes
        Interrupt: pin A routed to IRQ 4
        Region 0: [virtual] Memory at d080000000 (32-bit, non-prefetchable) [size=16385]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Kernel driver in use: acenic

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Bjorn Helgaas]

It looks like IP30 is using some code that's not upstream yet, so I'll
just point out some things that don't look right.  I'm ignoring the
IP27 stuff here in case that has different issues.

Joshua wrote:
> On 02/07/2017 13:29, Bjorn Helgaas wrote:
>> Is there any chance you can collect complete dmesg logs and
>> /proc/iomem contents from IP27 and IP30?  Maybe "lspci -vv" output,
>> too?  I'm not sure where to look to understand the ioremap() behavior.
> 
> ...
> A note about IP30, since that's part of an external patch set, the
> dmesg output is a little different, as it is using the newer
> BRIDGE/Xtalk code I just sent in as patches.  ...
> 
> Also, IP30 doesn't use the PCI_PROBE_ONLY thing anymore.  This causes
> /proc/iomem to be far more detailed than it used to be.
> 
> IP30's /proc/iomem:
>     00000000-00003fff : reserved
>     1d200000-1d9fffff : Bridge MEM
>       d080000000-d080003fff : 0001:00:03.0
>       1d204000-1d204fff : 0001:00:02.0
>       1d205000-1d205fff : 0001:00:02.1
>       1d206000-1d206fff : 0001:00:02.2
>       1d207000-1d2070ff : 0001:00:02.3
>       1d207100-1d20717f : 0001:00:01.0
>     1f200000-1f9fffff : Bridge MEM
>       f080100000-f0801fffff : 0000:00:02.0
>       f080010000-f08001ffff : 0000:00:00.0
>       f080030000-f08003ffff : 0000:00:01.0
>       f080000000-f080000fff : 0000:00:00.0
>       f080020000-f080020fff : 0000:00:01.0
>     20004000-209b8fff : reserved
>       20004000-206acb13 : Kernel code
>       206acb14-208affff : Kernel data
>     209b9000-20efffff : System RAM
>     20f00000-20ffffff : System RAM
>     21000000-9fffffff : System RAM
>     f080100000-f0801fffff : ioc3

From the dmesg you attached (ip30-dmesg-20170208.txt, PCI parts
appended below):

  pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff]
  pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff]

These are shown correctly in /proc/iomem.  It would be nice if the
/proc/iomem string identified which bridge was which (x86 uses "PCI
Bus 0000:00", "PCI Bus 0001:00", etc.) but that's not essential.

  pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff]
  ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit.

This is funky.  We read 0x1d200000 from the BAR (PCI bus addresses are
apparently identical to CPU physical addresses).  We claimed that
space from the host bridge window, so /proc/iomem would have looked
like this at that point:

  1d200000-1d9fffff : Bridge MEM
    1d200000-1d203fff : 0001:00:03.0

But it looks like whatever this ip30-bridge thing is overwrote the
0001:00:03.0 resource, which makes /proc/iomem wrong.

All the resources under the bridge to 0000:00 are wrong, too:

  1f200000-1f9fffff : Bridge MEM
    f080100000-f0801fffff : 0000:00:02.0
    f080010000-f08001ffff : 0000:00:00.0
    f080030000-f08003ffff : 0000:00:01.0
    f080000000-f080000fff : 0000:00:00.0
    f080020000-f080020fff : 0000:00:01.0

We read values from the BARs:

  pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff]
  pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff]
  pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref]
  pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff]
  pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref]
  pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff]
  pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff]

These aren't zero, so somebody apparently has assigned them, but they
aren't inside the host bridge window.  Therefore the PCI core assumes
they will not work, and it reassigns space from the window:

  pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff]
  pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref]
  pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref]
  pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff]
  pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff]

I don't know why we didn't assign space to 0000:00:03.0 BAR 1.

Anyway, it looks like they got inserted in /proc/iomem, then
overwritten by the ip30-bridge stuff again.

From /proc/iomem, I would expect that only the USB devices (0001:00:02
functions 0, 1, 2, 3) would work.  But apparently your system does
actually work, so there must be corresponding funkiness somewhere else
that compensates for these resources.

I'm attaching the PCI parts of your ip30-dmesg-20170208.txt and the
complete ip30-lspci-20170208.txt below for context.

  pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff]
  pci_bus 0000:00: root bus resource [io  0x1fa00000-0x1fbfffff]
  pci_bus 0000:00: root bus resource [bus 00-ff]
  pci 0000:00:00.0: [1077:1020] type 00 class 0x010000
  pci 0000:00:00.0: reg 0x10: [io  0x200000-0x2000ff]
  pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff]
  pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref]
  pci 0000:00:01.0: [1077:1020] type 00 class 0x010000
  pci 0000:00:01.0: reg 0x10: [io  0x400000-0x4000ff]
  pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff]
  pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref]
  pci 0000:00:02.0: [10a9:0003] type 00 class 0xff0000
  pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff]
  pci 0000:00:03.0: [10a9:0005] type 00 class 0x000000
  pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff]
  pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff]
  pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref]
  pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref]
  pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff]
  pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff]
  pci 0000:00:00.0: BAR 0: assigned [io  0x1fa00000-0x1fa000ff]
  pci 0000:00:01.0: BAR 0: assigned [io  0x1fa00400-0x1fa004ff]

  ip30-bridge: 0000:00:00.0 Bar 0 with size 0x00000100 at bus 0x00000000 vma 0x000000f100000000 is Direct I/O.
  ip30-bridge: 0000:00:00.0 Bar 1 with size 0x00001000 at bus 0x00000000 vma 0x000000f080000000 is Direct 64-bit.
  ip30-bridge: 0000:00:00.0 Bar 6 with size 0x00010000 at bus 0x00010000 vma 0x000000f080010000 is Direct 64-bit.

  ip30-bridge: 0000:00:01.0 Bar 0 with size 0x00000100 at bus 0x00000100 vma 0x000000f100000100 is Direct I/O.
  ip30-bridge: 0000:00:01.0 Bar 1 with size 0x00001000 at bus 0x00020000 vma 0x000000f080020000 is Direct 64-bit.
  ip30-bridge: 0000:00:01.0 Bar 6 with size 0x00010000 at bus 0x00030000 vma 0x000000f080030000 is Direct 64-bit.

  ip30-bridge: 0000:00:02.0 Bar 0 with size 0x00100000 at bus 0x00100000 vma 0x000000f080100000 is Direct 64-bit.

  PCI host bridge to bus 0001:00
  pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff]
  pci_bus 0001:00: root bus resource [io  0x1da00000-0x1dbfffff]
  pci_bus 0001:00: root bus resource [bus 01-ff]
  pci 0001:00:01.0: [11fe:080e] type 00 class 0x078000
  pci 0001:00:01.0: reg 0x10: [mem 0x00200000-0x0020007f]
  pci 0001:00:01.0: reg 0x14: [io  0x200000-0x20007f]
  pci 0001:00:01.0: reg 0x18: [io  0x204000-0x2040ff]
  pci 0001:00:02.0: [10b9:5237] type 00 class 0x0c0310
  pci 0001:00:02.0: reg 0x10: [mem 0x00300000-0x00300fff]
  pci 0001:00:02.0: PME# supported from D0 D1 D3hot D3cold
  pci 0001:00:02.1: [10b9:5237] type 00 class 0x0c0310
  pci 0001:00:02.1: reg 0x10: [mem 0x00000000-0x00000fff]
  pci 0001:00:02.1: PME# supported from D0 D1 D3hot D3cold
  pci 0001:00:02.2: [10b9:5237] type 00 class 0x0c0310
  pci 0001:00:02.2: reg 0x10: [mem 0x00000000-0x00000fff]
  pci 0001:00:02.2: PME# supported from D0 D1 D3hot D3cold
  pci 0001:00:02.3: [10b9:5239] type 00 class 0x0c0320
  pci 0001:00:02.3: reg 0x10: [mem 0x00000000-0x000000ff]
  pci 0001:00:02.3: PME# supported from D0 D3hot D3cold
  pci 0001:00:03.0: [10a9:0009] type 00 class 0x020000
  pci 0001:00:03.0: reg 0x10: [mem 0x00400000-0x00403fff]
  pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff]
  pci 0001:00:02.0: BAR 0: assigned [mem 0x1d204000-0x1d204fff]
  pci 0001:00:02.1: BAR 0: assigned [mem 0x1d205000-0x1d205fff]
  pci 0001:00:02.2: BAR 0: assigned [mem 0x1d206000-0x1d206fff]
  pci 0001:00:01.0: BAR 2: assigned [io  0x1da00000-0x1da000ff]
  pci 0001:00:02.3: BAR 0: assigned [mem 0x1d207000-0x1d2070ff]
  pci 0001:00:01.0: BAR 0: assigned [mem 0x1d207100-0x1d20717f]
  pci 0001:00:01.0: BAR 1: assigned [io  0x1da00400-0x1da0047f]

  ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit.


Here's the ip30-lspci-20170208.txt attachment from your mail:


0000:00:00.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 0
        Region 0: I/O ports at f100000000 [size=257]
        Region 1: [virtual] Memory at f080000000 (32-bit, non-prefetchable) [size=4097]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at f080010000 [disabled] [size=65537]
        Kernel driver in use: qla1280
lspci: Unable to load libkmod resources: error -12

0000:00:01.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 1
        Region 0: I/O ports at f100000100 [size=257]
        Region 1: Memory at f080020000 (32-bit, non-prefetchable) [size=4097]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at f080030000 [disabled] [size=65537]
        Kernel driver in use: qla1280

0000:00:02.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. IOC3 I/O controller (rev 01)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Interrupt: pin A routed to IRQ 2
        Region 0: Memory at f080100000 (32-bit, non-prefetchable) [size=1048577]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Kernel driver in use: IOC3

0000:00:03.0 Non-VGA unclassified device: Silicon Graphics Intl. Corp. RAD Audio (rev c0)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=slow >TAbort+ <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 255
        Region 0: Memory at 00600000 (32-bit, non-prefetchable) [size=8193]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]

0001:00:01.0 Communication controller: Comtrol Corporation Device 080e (rev 01)
        Subsystem: Comtrol Corporation Device 080e
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 0
        Region 0: Memory at 1d207100 (32-bit, non-prefetchable) [size=129]
        Region 1: I/O ports at 1da00400 [size=129]
        Region 2: I/O ports at 1da00000 [size=257]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]

0001:00:02.0 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 16 (20000ns max)
        Interrupt: pin B routed to IRQ 0
        Region 0: Memory at 1d204000 (32-bit, non-prefetchable) [size=4097]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [60] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

0001:00:02.1 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin C routed to IRQ 0
        Region 0: Memory at 1d205000 (32-bit, non-prefetchable) [disabled] [size=4097]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [60] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

0001:00:02.2 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin D routed to IRQ 0
        Region 0: Memory at 1d206000 (32-bit, non-prefetchable) [disabled] [size=4097]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [60] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

0001:00:02.3 USB controller: ULi Electronics Inc. USB 2.0 Controller (rev 01) (prog-if 20 [EHCI])
        Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 0
        Region 0: Memory at 1d207000 (32-bit, non-prefetchable) [disabled] [size=257]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Debug port: BAR=1 offset=0090

0001:00:03.0 Ethernet controller: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet (rev 01)
        Subsystem: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64 (16000ns min), Cache Line Size: 128 bytes
        Interrupt: pin A routed to IRQ 4
        Region 0: [virtual] Memory at d080000000 (32-bit, non-prefetchable) [size=16385]
        Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2]
        Expansion ROM at <unassigned> [disabled] [size=2]
        Kernel driver in use: acenic

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Bjorn Helgaas]

It looks like IP30 is using some code that's not upstream yet, so I'll
just point out some things that don't look right.  I'm ignoring the
IP27 stuff here in case that has different issues.

Joshua wrote:
> On 02/07/2017 13:29, Bjorn Helgaas wrote:
>> Is there any chance you can collect complete dmesg logs and
>> /proc/iomem contents from IP27 and IP30?  Maybe "lspci -vv" output,
>> too?  I'm not sure where to look to understand the ioremap() behavior.
> 
> ...
> A note about IP30, since that's part of an external patch set, the
> dmesg output is a little different, as it is using the newer
> BRIDGE/Xtalk code I just sent in as patches.  ...
> 
> Also, IP30 doesn't use the PCI_PROBE_ONLY thing anymore.  This causes
> /proc/iomem to be far more detailed than it used to be.
> 
> IP30's /proc/iomem:
>     00000000-00003fff : reserved
>     1d200000-1d9fffff : Bridge MEM
>       d080000000-d080003fff : 0001:00:03.0
>       1d204000-1d204fff : 0001:00:02.0
>       1d205000-1d205fff : 0001:00:02.1
>       1d206000-1d206fff : 0001:00:02.2
>       1d207000-1d2070ff : 0001:00:02.3
>       1d207100-1d20717f : 0001:00:01.0
>     1f200000-1f9fffff : Bridge MEM
>       f080100000-f0801fffff : 0000:00:02.0
>       f080010000-f08001ffff : 0000:00:00.0
>       f080030000-f08003ffff : 0000:00:01.0
>       f080000000-f080000fff : 0000:00:00.0
>       f080020000-f080020fff : 0000:00:01.0
>     20004000-209b8fff : reserved
>       20004000-206acb13 : Kernel code
>       206acb14-208affff : Kernel data
>     209b9000-20efffff : System RAM
>     20f00000-20ffffff : System RAM
>     21000000-9fffffff : System RAM
>     f080100000-f0801fffff : ioc3

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Joshua Kinard]

On 02/13/2017 17:45, Bjorn Helgaas wrote:
> It looks like IP30 is using some code that's not upstream yet, so I'll
> just point out some things that don't look right.  I'm ignoring the
> IP27 stuff here in case that has different issues.

Sorry about that.  It's an external set of patches I've been maintaining for
the last decade (I'm not the original author).  Being that I do this for a
hobby, sometimes I get waylaid by other priorities.  I've spent some time
re-organizing my patch collection lately so that I can start sending these
things in.  This series and an earlier one to the Linux/MIPS list are the
beginnings of that, as time permits.  However, I've got a ways to go before
getting to the IP30 patches.  E.g., IP27 comes before IP30, so that's where my
focus has been lately.

I've put an 4.10-rc7 tree with all patches applied here (including the IOC3,
IP27, and IP30 patches):
http://dev.gentoo.org/~kumba/mips/ip30/linux-4.10_rc7-20170208.ip30/

The patchset itself, if interested:
http://dev.gentoo.org/~kumba/mips/ip30/mips-patches-4.10.0/

You can find the IP30 code in:
  arch/mips/sgi-ip30/*
  arch/mips/include/asm/mach-ip30/*

IP27's code is in these folders:
  arch/mips/sgi-ip27/*
  arch/mips/include/asm/mach-ip27/*
  arch/mips/include/asm/sn/*

PCI code:
  arch/mips/pci/pci-bridge.c
  arch/mips/pci/ops-bridge.c
  arch/mips/include/asm/pci/bridge.h


> Joshua wrote:
>> On 02/07/2017 13:29, Bjorn Helgaas wrote:
>>> Is there any chance you can collect complete dmesg logs and
>>> /proc/iomem contents from IP27 and IP30?  Maybe "lspci -vv" output,
>>> too?  I'm not sure where to look to understand the ioremap() behavior.
>>
>> ...
>> A note about IP30, since that's part of an external patch set, the
>> dmesg output is a little different, as it is using the newer
>> BRIDGE/Xtalk code I just sent in as patches.  ...
>>
>> Also, IP30 doesn't use the PCI_PROBE_ONLY thing anymore.  This causes
>> /proc/iomem to be far more detailed than it used to be.
>>
>> IP30's /proc/iomem:
>>     00000000-00003fff : reserved
>>     1d200000-1d9fffff : Bridge MEM
>>       d080000000-d080003fff : 0001:00:03.0
>>       1d204000-1d204fff : 0001:00:02.0
>>       1d205000-1d205fff : 0001:00:02.1
>>       1d206000-1d206fff : 0001:00:02.2
>>       1d207000-1d2070ff : 0001:00:02.3
>>       1d207100-1d20717f : 0001:00:01.0
>>     1f200000-1f9fffff : Bridge MEM
>>       f080100000-f0801fffff : 0000:00:02.0
>>       f080010000-f08001ffff : 0000:00:00.0
>>       f080030000-f08003ffff : 0000:00:01.0
>>       f080000000-f080000fff : 0000:00:00.0
>>       f080020000-f080020fff : 0000:00:01.0
>>     20004000-209b8fff : reserved
>>       20004000-206acb13 : Kernel code
>>       206acb14-208affff : Kernel data
>>     209b9000-20efffff : System RAM
>>     20f00000-20ffffff : System RAM
>>     21000000-9fffffff : System RAM
>>     f080100000-f0801fffff : ioc3
> 
>>From the dmesg you attached (ip30-dmesg-20170208.txt, PCI parts
> appended below):
> 
>   pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff]
>   pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff]
> 
> These are shown correctly in /proc/iomem.  It would be nice if the
> /proc/iomem string identified which bridge was which (x86 uses "PCI
> Bus 0000:00", "PCI Bus 0001:00", etc.) but that's not essential.
> 
>   pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff]
>   ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit.
> 
> This is funky.  We read 0x1d200000 from the BAR (PCI bus addresses are
> apparently identical to CPU physical addresses).  We claimed that
> space from the host bridge window, so /proc/iomem would have looked
> like this at that point:
> 
>   1d200000-1d9fffff : Bridge MEM
>     1d200000-1d203fff : 0001:00:03.0
> 
> But it looks like whatever this ip30-bridge thing is overwrote the
> 0001:00:03.0 resource, which makes /proc/iomem wrong.
> 
> All the resources under the bridge to 0000:00 are wrong, too:
> 
>   1f200000-1f9fffff : Bridge MEM
>     f080100000-f0801fffff : 0000:00:02.0
>     f080010000-f08001ffff : 0000:00:00.0
>     f080030000-f08003ffff : 0000:00:01.0
>     f080000000-f080000fff : 0000:00:00.0
>     f080020000-f080020fff : 0000:00:01.0

Actually, that's not wrong.  It took me a long time to figure this out, too,
but once I stumbled upon the BRIDGE ASIC specification on an anonymous FTP
site, via Google, by complete accident (hint), it finally made sense.

BRIDGE is what SGI calls the chip that interfaces the Crosstalk (Xtalk) bus to
a PCI bus.  Most known "XIO" (Xtalk I/O) boards contain a BRIDGE chip under a
heatsink at the edge of the board, next to a "compression connector".  Octane
can hold up to four XIO boards, with a fifth slot dedicated to an optional "PCI
Shoebox" (literally a metal box that can hold up to three PCI or PCI-X cards).
One exception to this are the two graphics boards, Impact/MGRAS and
Odyssey/VPro, which are "pure" XIO devices, as far as anyone's figured out.

Each BRIDGE chip can address up to eight different PCI devices.

Crosstalk on IP30 is managed by the Crossbow (XBOW) ASIC.  It implements a
crossbar switch that's not totally unlike a networking switch.  A single XBOW
supports up to 16 XIO widgets, numbered 0x0 to 0xf.  Widgets 0x1 to 0x7 are for
internal use or unused.  XBOW itself is widget 0x0.  HEART, Octane's system
controller, is widget 0x8.  The four standard XIO slots are numbered (I think)
as 0x9 to 0xc.  I think widget 0xe is unused.

The system board has a BRIDGE chip on it to provide the "BaseIO" devices, such
as two SCSI (QL1040B), IOC3 Ethernet, KB/Mouse, Serial/Parallel, Audio, and an
RTC.  This BaseIO BRIDGE is widget 0xf, which is what you're seeing with the
first set of addresses starting with 0x1fxxx.  The second BRIDGE that you're
seeing is the PCI Shoebox I have installed.  It's widget 0xd, and has addresses
starting with 0x1dxxx (second nibble from left is the widget), with three
random PCI cards I found in a storage bin plugged into it.  The Xtalk scan
happens in reverse order, starting at widget 0xf down to 0x0.

Both 0x1fxxx and 0x1dxxx addresses are "small window" addresses.  The HEART
system controller gives you three "windows" into Xtalk space, which uses 48-bit
addressing:

0x0000_1000_0000 - 0x0000_1fff_ffff - Small windows, x16 @ 16MB each
0x0008_0000_0000 - 0x000f_ffff_ffff - Medium windows, x16 @ 2GB each
0x0010_0000_0000 - 0x00ff_ffff_ffff - Big windows, x15 @ 64GB each

So a device on widget 0xf mapped into a small window at 0x0000_1fxx_xxxx could
also be referred to by a big window at address 0x00f0_8xxx_xxxx.  The only
difference is the big window gives you a larger address space to play around
with.  Medium windows are not currently used on the IP30 platform in Linux.
Widget 0x0 (XBOW) is only accessible via small or medium windows.

As such, this layout in /proc/iomem is oddly correct:

>   1f200000-1f9fffff : Bridge MEM
>     f080100000-f0801fffff : 0000:00:02.0
>     f080010000-f08001ffff : 0000:00:00.0
>     f080030000-f08003ffff : 0000:00:01.0
>     f080000000-f080000fff : 0000:00:00.0
>     f080020000-f080020fff : 0000:00:01.0

My bug is probably that I do the initial BRIDGE scan with small windows in the
main BRIDGE driver (arch/mips/pci/pci-bridge.c), and then IP30's BRIDGE glue
driver (arch/mips/sgi-ip30/ip30-bridge.c) is switching to big windows to probe
each device.  So, yes, different address ranges, but still looking at the same
device.  I'll look into fixing that.  It's partially tied to how I was kludging
the IP27 to also start doing PCI correctly within its own BRIDGE glue driver
(ip27-bridge.c), but I just learned over the weekend that there's missing
functionality I have to port over from old 2.5.x-era IA64 code.  In IP30, big
windows are available by default, but on IP27, what it calls "big windows" need
to go through some kind of translation table, and I'm trying to figure out how
to set that up.

There's also an issue with DMA on BRIDGE on IP30 that limits the system's
memory to 2GB to avoid kernel panics.  OpenBSD's figured it out -- BRIDGE's
IOMMU is busted so it can only do 31-bit DMA max.  I just haven't figured out
how to set that limit in Linux for the entire BRIDGE, not just for individual
devices.


> We read values from the BARs:
> 
>   pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff]
>   pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff]
>   pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref]
>   pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff]
>   pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref]
>   pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff]
>   pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff]
> 
> These aren't zero, so somebody apparently has assigned them, but they
> aren't inside the host bridge window.  Therefore the PCI core assumes
> they will not work, and it reassigns space from the window:
> 
>   pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff]
>   pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref]
>   pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref]
>   pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff]
>   pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff]

Hmm, from the PCI device's point of view, 0x00200000-0x00200fff would be
correct.  The same address beginning with 0x1fxxx is the Crosstalk view of the
device.  Odds are likely I am not masking something off somewhere, but somehow,
it all still finds a way to work.


> I don't know why we didn't assign space to 0000:00:03.0 BAR 1.

The PCI device sitting at 0000:00:03.0 is the "RAD Audio" chip.  There's an
issue with the driver that I keep forgetting to try and fix, so I've left it
out of my recent kernel builds.  That's probably why no space is being assigned
to it.


>>From /proc/iomem, I would expect that only the USB devices (0001:00:02
> functions 0, 1, 2, 3) would work.  But apparently your system does
> actually work, so there must be corresponding funkiness somewhere else
> that compensates for these resources.

USB is...interesting on this platform.  Some USB devices will work, but not
many have been tested.  If my memory recalls, I think UHCI was known to play
nice, but OHCI had issues -- might be the other way around.  EHCI is dependent
on what you plug in and the position of several planets in the sky.  xHCI has
endianess issues with one card that I tried.  It's hard to find PCI or PCI-X
xHCI cards, though.  Most everything now is PCIe.  And due to chassis space
restrictions, using PCIe-to-PCI-X adapters is a no-go.

PCI-to-PCI bridges and USB hubs are totally busted, because BRIDGE's PCI Type 1
configuration space isn't very well understood.  The BRIDGE Specification I
think finally cleared that up, but I have not tried to learn how to actually
set the space up and find a spare USB Hub to play with.

All-in-all, the entire machine is as much a work of art in its inner workings
as how it looks on the outside.  As with all art, it's up to the eye of the
beholder on whether that's a good thing or not </smirk>

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Bjorn Helgaas]

On Tue, Feb 14, 2017 at 02:39:45AM -0500, Joshua Kinard wrote:
> On 02/13/2017 17:45, Bjorn Helgaas wrote:
> > It looks like IP30 is using some code that's not upstream yet, so I'll
> > just point out some things that don't look right.  I'm ignoring the
> > IP27 stuff here in case that has different issues.
> 
> Sorry about that.  It's an external set of patches I've been maintaining for
> the last decade (I'm not the original author).  Being that I do this for a
> hobby, sometimes I get waylaid by other priorities.  I've spent some time
> re-organizing my patch collection lately so that I can start sending these
> things in.  This series and an earlier one to the Linux/MIPS list are the
> beginnings of that, as time permits.  However, I've got a ways to go before
> getting to the IP30 patches.  E.g., IP27 comes before IP30, so that's where my
> focus has been lately.
> 
> I've put an 4.10-rc7 tree with all patches applied here (including the IOC3,
> IP27, and IP30 patches):
> http://dev.gentoo.org/~kumba/mips/ip30/linux-4.10_rc7-20170208.ip30/
> 
> The patchset itself, if interested:
> http://dev.gentoo.org/~kumba/mips/ip30/mips-patches-4.10.0/

Thanks for the pointers.

> > All the resources under the bridge to 0000:00 are wrong, too:
> > 
> >   1f200000-1f9fffff : Bridge MEM
> >     f080100000-f0801fffff : 0000:00:02.0
> >     f080010000-f08001ffff : 0000:00:00.0
> >     f080030000-f08003ffff : 0000:00:01.0
> >     f080000000-f080000fff : 0000:00:00.0
> >     f080020000-f080020fff : 0000:00:01.0
> 
> Actually, that's not wrong.  It took me a long time to figure this out, too,
> but once I stumbled upon the BRIDGE ASIC specification on an anonymous FTP
> site, via Google, by complete accident (hint), it finally made sense.
> 
> BRIDGE is what SGI calls the chip that interfaces the Crosstalk (Xtalk) bus to
> a PCI bus.  Most known "XIO" (Xtalk I/O) boards contain a BRIDGE chip under a
> heatsink at the edge of the board, next to a "compression connector".  Octane
> can hold up to four XIO boards, with a fifth slot dedicated to an optional "PCI
> Shoebox" (literally a metal box that can hold up to three PCI or PCI-X cards).
> One exception to this are the two graphics boards, Impact/MGRAS and
> Odyssey/VPro, which are "pure" XIO devices, as far as anyone's figured out.
> 
> Each BRIDGE chip can address up to eight different PCI devices.
> 
> Crosstalk on IP30 is managed by the Crossbow (XBOW) ASIC.  It implements a
> crossbar switch that's not totally unlike a networking switch.  A single XBOW
> supports up to 16 XIO widgets, numbered 0x0 to 0xf.  Widgets 0x1 to 0x7 are for
> internal use or unused.  XBOW itself is widget 0x0.  HEART, Octane's system
> controller, is widget 0x8.  The four standard XIO slots are numbered (I think)
> as 0x9 to 0xc.  I think widget 0xe is unused.
> 
> The system board has a BRIDGE chip on it to provide the "BaseIO" devices, such
> as two SCSI (QL1040B), IOC3 Ethernet, KB/Mouse, Serial/Parallel, Audio, and an
> RTC.  This BaseIO BRIDGE is widget 0xf, which is what you're seeing with the
> first set of addresses starting with 0x1fxxx.  The second BRIDGE that you're
> seeing is the PCI Shoebox I have installed.  It's widget 0xd, and has addresses
> starting with 0x1dxxx (second nibble from left is the widget), with three
> random PCI cards I found in a storage bin plugged into it.  The Xtalk scan
> happens in reverse order, starting at widget 0xf down to 0x0.
> 
> Both 0x1fxxx and 0x1dxxx addresses are "small window" addresses.  The HEART
> system controller gives you three "windows" into Xtalk space, which uses 48-bit
> addressing:
> 
> 0x0000_1000_0000 - 0x0000_1fff_ffff - Small windows, x16 @ 16MB each
> 0x0008_0000_0000 - 0x000f_ffff_ffff - Medium windows, x16 @ 2GB each
> 0x0010_0000_0000 - 0x00ff_ffff_ffff - Big windows, x15 @ 64GB each
> 
> So a device on widget 0xf mapped into a small window at 0x0000_1fxx_xxxx could
> also be referred to by a big window at address 0x00f0_8xxx_xxxx.  The only
> difference is the big window gives you a larger address space to play around
> with.  Medium windows are not currently used on the IP30 platform in Linux.
> Widget 0x0 (XBOW) is only accessible via small or medium windows.
> 
> As such, this layout in /proc/iomem is oddly correct:
> 
> >   1f200000-1f9fffff : Bridge MEM
> >     f080100000-f0801fffff : 0000:00:02.0
> >     f080010000-f08001ffff : 0000:00:00.0
> >     f080030000-f08003ffff : 0000:00:01.0
> >     f080000000-f080000fff : 0000:00:00.0
> >     f080020000-f080020fff : 0000:00:01.0
> 
> My bug is probably that I do the initial BRIDGE scan with small windows in the
> main BRIDGE driver (arch/mips/pci/pci-bridge.c), and then IP30's BRIDGE glue
> driver (arch/mips/sgi-ip30/ip30-bridge.c) is switching to big windows to probe
> each device.  So, yes, different address ranges, but still looking at the same
> device.  I'll look into fixing that.  It's partially tied to how I was kludging
> the IP27 to also start doing PCI correctly within its own BRIDGE glue driver
> (ip27-bridge.c), but I just learned over the weekend that there's missing
> functionality I have to port over from old 2.5.x-era IA64 code.  In IP30, big
> windows are available by default, but on IP27, what it calls "big windows" need
> to go through some kind of translation table, and I'm trying to figure out how
> to set that up.

I agree, it sounds like the problem is the switch from small windows
to big ones.  In order for the PCI core to work correctly, the host
bridge window ("1f200000-1f9fffff : Bridge MEM") must enclose the BARs
of the devices below the bridge.

If you use the big windows for the device BARs, you should use big
windows for the host bridge windows.  I think we're talking about
widget 0xf here, so the 0xf big window would be 0xf0_0000_0000 -
0xff_ffff_ffff.  This should all be set up *before* calling
pci_scan_root_bus() instead of after, as it seems to be today.

The scan would look like this:

  pci_bus 0000:00: root bus resource [mem 0xf000000000-0xffffffffff] (bus addresses [0x00000000-0xfffffffff])
  pci 0000:00:00.0: reg 0x14: [mem 0xf000200000-0xf000200fff]
  pci 0000:00:00.0: reg 0x30: [mem 0xf000210000-0xf00021ffff pref]
  pci 0000:00:01.0: reg 0x14: [mem 0xf000400000-0xf000400fff]
  pci 0000:00:01.0: reg 0x30: [mem 0xf000410000-0xf00041ffff pref]
  pci 0000:00:02.0: reg 0x10: [mem 0xf000500000-0xf0005fffff]
  pci 0000:00:03.0: reg 0x10: [mem 0xf000600000-0xf000601fff]

This would make /proc/iomem look like this:

  f000000000-ffffffffff : Bridge MEM
    f000500000-f0005fffff : 0000:00:02.0
    f000210000-f00021ffff : 0000:00:00.0
    f000410000-f00041ffff : 0000:00:01.0
    f000200000-f000200fff : 0000:00:00.0
    f000400000-f000400fff : 0000:00:01.0

This doesn't match the device BARs in your /proc/iomem, so there must
be some other transformation going on as well.

As long as you tell the PCI core about the host bridge windows you're
going to use, along with offsets that include *all* these
transformations, the core should just work, and /proc/iomem should
also make sense.  The details of small/medium/big windows, widgets,
etc., are immaterial to the core.

Bjorn

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Joshua Kinard]

On 02/14/2017 09:56, Bjorn Helgaas wrote:
> On Tue, Feb 14, 2017 at 02:39:45AM -0500, Joshua Kinard wrote:
>> On 02/13/2017 17:45, Bjorn Helgaas wrote:
>>> It looks like IP30 is using some code that's not upstream yet, so I'll
>>> just point out some things that don't look right.  I'm ignoring the
>>> IP27 stuff here in case that has different issues.
>>
>> Sorry about that.  It's an external set of patches I've been maintaining for
>> the last decade (I'm not the original author).  Being that I do this for a
>> hobby, sometimes I get waylaid by other priorities.  I've spent some time
>> re-organizing my patch collection lately so that I can start sending these
>> things in.  This series and an earlier one to the Linux/MIPS list are the
>> beginnings of that, as time permits.  However, I've got a ways to go before
>> getting to the IP30 patches.  E.g., IP27 comes before IP30, so that's where my
>> focus has been lately.
>>
>> I've put an 4.10-rc7 tree with all patches applied here (including the IOC3,
>> IP27, and IP30 patches):
>> http://dev.gentoo.org/~kumba/mips/ip30/linux-4.10_rc7-20170208.ip30/
>>
>> The patchset itself, if interested:
>> http://dev.gentoo.org/~kumba/mips/ip30/mips-patches-4.10.0/
> 
> Thanks for the pointers.
> 
>>> All the resources under the bridge to 0000:00 are wrong, too:
>>>
>>>   1f200000-1f9fffff : Bridge MEM
>>>     f080100000-f0801fffff : 0000:00:02.0
>>>     f080010000-f08001ffff : 0000:00:00.0
>>>     f080030000-f08003ffff : 0000:00:01.0
>>>     f080000000-f080000fff : 0000:00:00.0
>>>     f080020000-f080020fff : 0000:00:01.0
>>
>> Actually, that's not wrong.  It took me a long time to figure this out, too,
>> but once I stumbled upon the BRIDGE ASIC specification on an anonymous FTP
>> site, via Google, by complete accident (hint), it finally made sense.
>>
>> BRIDGE is what SGI calls the chip that interfaces the Crosstalk (Xtalk) bus to
>> a PCI bus.  Most known "XIO" (Xtalk I/O) boards contain a BRIDGE chip under a
>> heatsink at the edge of the board, next to a "compression connector".  Octane
>> can hold up to four XIO boards, with a fifth slot dedicated to an optional "PCI
>> Shoebox" (literally a metal box that can hold up to three PCI or PCI-X cards).
>> One exception to this are the two graphics boards, Impact/MGRAS and
>> Odyssey/VPro, which are "pure" XIO devices, as far as anyone's figured out.
>>
>> Each BRIDGE chip can address up to eight different PCI devices.
>>
>> Crosstalk on IP30 is managed by the Crossbow (XBOW) ASIC.  It implements a
>> crossbar switch that's not totally unlike a networking switch.  A single XBOW
>> supports up to 16 XIO widgets, numbered 0x0 to 0xf.  Widgets 0x1 to 0x7 are for
>> internal use or unused.  XBOW itself is widget 0x0.  HEART, Octane's system
>> controller, is widget 0x8.  The four standard XIO slots are numbered (I think)
>> as 0x9 to 0xc.  I think widget 0xe is unused.
>>
>> The system board has a BRIDGE chip on it to provide the "BaseIO" devices, such
>> as two SCSI (QL1040B), IOC3 Ethernet, KB/Mouse, Serial/Parallel, Audio, and an
>> RTC.  This BaseIO BRIDGE is widget 0xf, which is what you're seeing with the
>> first set of addresses starting with 0x1fxxx.  The second BRIDGE that you're
>> seeing is the PCI Shoebox I have installed.  It's widget 0xd, and has addresses
>> starting with 0x1dxxx (second nibble from left is the widget), with three
>> random PCI cards I found in a storage bin plugged into it.  The Xtalk scan
>> happens in reverse order, starting at widget 0xf down to 0x0.
>>
>> Both 0x1fxxx and 0x1dxxx addresses are "small window" addresses.  The HEART
>> system controller gives you three "windows" into Xtalk space, which uses 48-bit
>> addressing:
>>
>> 0x0000_1000_0000 - 0x0000_1fff_ffff - Small windows, x16 @ 16MB each
>> 0x0008_0000_0000 - 0x000f_ffff_ffff - Medium windows, x16 @ 2GB each
>> 0x0010_0000_0000 - 0x00ff_ffff_ffff - Big windows, x15 @ 64GB each
>>
>> So a device on widget 0xf mapped into a small window at 0x0000_1fxx_xxxx could
>> also be referred to by a big window at address 0x00f0_8xxx_xxxx.  The only
>> difference is the big window gives you a larger address space to play around
>> with.  Medium windows are not currently used on the IP30 platform in Linux.
>> Widget 0x0 (XBOW) is only accessible via small or medium windows.
>>
>> As such, this layout in /proc/iomem is oddly correct:
>>
>>>   1f200000-1f9fffff : Bridge MEM
>>>     f080100000-f0801fffff : 0000:00:02.0
>>>     f080010000-f08001ffff : 0000:00:00.0
>>>     f080030000-f08003ffff : 0000:00:01.0
>>>     f080000000-f080000fff : 0000:00:00.0
>>>     f080020000-f080020fff : 0000:00:01.0
>>
>> My bug is probably that I do the initial BRIDGE scan with small windows in the
>> main BRIDGE driver (arch/mips/pci/pci-bridge.c), and then IP30's BRIDGE glue
>> driver (arch/mips/sgi-ip30/ip30-bridge.c) is switching to big windows to probe
>> each device.  So, yes, different address ranges, but still looking at the same
>> device.  I'll look into fixing that.  It's partially tied to how I was kludging
>> the IP27 to also start doing PCI correctly within its own BRIDGE glue driver
>> (ip27-bridge.c), but I just learned over the weekend that there's missing
>> functionality I have to port over from old 2.5.x-era IA64 code.  In IP30, big
>> windows are available by default, but on IP27, what it calls "big windows" need
>> to go through some kind of translation table, and I'm trying to figure out how
>> to set that up.
> 
> I agree, it sounds like the problem is the switch from small windows
> to big ones.  In order for the PCI core to work correctly, the host
> bridge window ("1f200000-1f9fffff : Bridge MEM") must enclose the BARs
> of the devices below the bridge.
> 
> If you use the big windows for the device BARs, you should use big
> windows for the host bridge windows.  I think we're talking about
> widget 0xf here, so the 0xf big window would be 0xf0_0000_0000 -
> 0xff_ffff_ffff.  This should all be set up *before* calling
> pci_scan_root_bus() instead of after, as it seems to be today.

Okay, so after a week of downtime due to hardware failure on another machine, I
started looking at this again, and it looks like it's more than just getting
the windows wrong.  The existing BRIDGE code (pci-bridge.c) is limiting the PCI
window to the first five devices on BaseIO in the small window (my fault), and
the IP30 glue code (ip30-bridge.c) is re-writing the PCI BARs to access the
BRIDGE PCI Memory and PCI I/O spaces via big windows, without updating the
original mapping.  We're not teaching the Linux PCI core at all, I think, about
how to generically access PCI Memory Space and PCI I/O Space on the BRIDGE.
The fact this actually works appears to be pure luck.

Quoting from the BRIDGE docs somewhat, from the point of view of generic
Crosstalk space and not specific to IP30 or IP27, there are several views into
PCI space.  Each address is 48-bits, the size of a Crosstalk address.

    0000_0000_0000 to 0000_00ff_ffff is "Widget space".  It seems the BRIDGE
    docs use the term "widget" here to refer to the eight possible PCI devices
    addressable by a single BRIDGE ASIC.  This plus the small windows is how
    the code is currently getting things to work.  I think.

    0000_4000_0000 to 0000_7fff_ffff is BRIDGE's view into PCI Memory Space
    for direct-mapped 32-bit devices.  It is 1GB in size.  My take is this is
    normally a 4GB space?  Can the Linux PCI core be taught that only 1GB
    is usable?

    0000_8000_0000 to 0000_bfff_ffff is an alias view by BRIDGE into PCI
    Memory space for 64-bit PCI devices, immediately after the 32-bit space.
    It is also 1GB in size.

    0000_c000_0000 to 0000_ffff_ffff is 2GB adjacent to PCI Memory space, but
    is either unused or used for Crosstalk/BRIDGE DMA operations
    (the docs aren't very clear on this point or I am simply not
    understanding).

    0001_0000_0000 to 0001_ffff_ffff is BRIDGE's view into PCI I/O Space.
    It has the entire 4GB range available to use.


I suspect the existing BRIDGE driver is mapping through "widget space" first to
probe the PCI device slots, then the ip30-bridge.c code is re-writing the PCI
BARs to go through the big window on IP30, but it never updates the original
BRIDGE setup (if that's even possible).  As such, as you pointed out, there's
some kind of logic in the generic PCI core that's re-assigning space from the
window and we're just getting lucky and everything still works.

I dug up some old debugging code given to me by the original author of the IP30
port and built up a set of macros that get me to the PCI Configuration Space on
Widget 0xf (BaseIO Bridge), Device #0 (the first QLA1040B SCSI controller).
All three addresses access the same PCI device and return the same config space
info.

    Small window:  0x900000001f02xxxx
    Medium window: 0x9000000f8002xxxx
    Big window:    0x900000f00002xxxx


Each additional device's configuration space is offset 0x1000, which gives me
the following addresses (dev's 4 to 7 are special-cased for IRQ trickery, so
can't probe them):

    - Dev #0: scsi0/qla1040b: 0x90xxxxxxxxx20000
    - Dev #1: scsi1/qla1040b: 0x90xxxxxxxxx21000
    - Dev #2: io/ioc3       : 0x90xxxxxxxxx22000
    - Dev #3: audio/rad1    : 0x90xxxxxxxxx23000


If I probe and dump the config space data for each, I get the following:

    PCI slot 0 information:
        vendor ID :              0x1077
        device ID :              0x1020
        command :                0x0006
        status :                 0x0200
        revision :               0x05
        prog if :                0x00
        class :                  0x0100
        cache line :             0x40
        latency :                0x40
        hdr type :               0x00
        BIST :                   0x00
        region 0 :               0x00200001
        region 1 :               0x00200000
        region 2 :               0x00000000
        region 3 :               0x00000000
        region 4 :               0x00000000
        region 5 :               0x00000000
        IRQ line :               0x00
        IRQ pin :                0x01

    PCI slot 1 information:
        vendor ID :              0x1077
        device ID :              0x1020
        command :                0x0006
        status :                 0x0200
        revision :               0x05
        prog if :                0x00
        class :                  0x0100
        cache line :             0x40
        latency :                0x40
        hdr type :               0x00
        BIST :                   0x00
        region 0 :               0x00400001
        region 1 :               0x00400000
        region 2 :               0x00000000
        region 3 :               0x00000000
        region 4 :               0x00000000
        region 5 :               0x00000000
        IRQ line :               0x00
        IRQ pin :                0x01

    PCI slot 2 information:
        vendor ID :              0x10a9
        device ID :              0x0003
        command :                0x0146
        status :                 0x0280
        revision :               0x01
        prog if :                0x00
        class :                  0xff00
        cache line :             0x00
        latency :                0x28
        hdr type :               0x00
        BIST :                   0x00
        region 0 :               0x00500000
        region 1 :               0x00000000
        region 2 :               0x00000000
        region 3 :               0x00500000
        region 4 :               0x000310a9
        region 5 :               0x02800146
        IRQ line :               0x00
        IRQ pin :                0x00

    PCI slot 3 information:
        vendor ID :              0x10a9
        device ID :              0x0005
        command :                0x0006
        status :                 0x0480
        revision :               0xc0
        prog if :                0x00
        class :                  0x0000
        cache line :             0x00
        latency :                0xff
        hdr type :               0x00
        BIST :                   0x00
        region 0 :               0x00600000
        region 1 :               0x00000000
        region 2 :               0x00000000
        region 3 :               0x00000000
        region 4 :               0x00000000
        region 5 :               0x00000000
        IRQ line :               0x00
        IRQ pin :                0x00

This is where my knowledge runs short -- I'm not fluent in PCI device
programming, so I'm not sure what is really supposed to be going on with these
different BAR regions.  I dug up a copy of the PCI Spec 2.2, but it's 322 pages
and I'm not sure where I should start reading from.  Are these regions
device-specific?  E.g., do I need the QLA1040B programming manual to understand
why region 0 is 0x00200001 and region 1 is 0x00200000?

I am also not yet considering how PCI views Crosstalk space for DMA operations
-- BRIDGE has several mechanisms for that, depending on if the PCI device is a
32-bit device or a 64-bit device.  It can do direct-mapping for 32-bit or
64-bit, or built-in page-mapping hardware is available (but apparently to be
avoided due to numerous hardware quirks/bugs that would make the driver
overly-complicated).

There's a good write-up in the OpenBSD "xbridge" driver, which handles BRIDGE
(IP27/IP30), XBRIDGE (IP35), and PIC (IP35/IA64 Altix):

http://bxr.su/OpenBSD/sys/arch/sgi/xbow/xbridge.c

It's starting to make some sense to me, but I am still uncertain how to work
with the 32-bit and 64-bit 1GB PCI memory spaces on BRIDGE as well as the 4GB
PCI I/O space from a Linux point of view.  Do they only matter when dealing
with DMA?

--J



> The scan would look like this:
> 
>   pci_bus 0000:00: root bus resource [mem 0xf000000000-0xffffffffff] (bus addresses [0x00000000-0xfffffffff])
>   pci 0000:00:00.0: reg 0x14: [mem 0xf000200000-0xf000200fff]
>   pci 0000:00:00.0: reg 0x30: [mem 0xf000210000-0xf00021ffff pref]
>   pci 0000:00:01.0: reg 0x14: [mem 0xf000400000-0xf000400fff]
>   pci 0000:00:01.0: reg 0x30: [mem 0xf000410000-0xf00041ffff pref]
>   pci 0000:00:02.0: reg 0x10: [mem 0xf000500000-0xf0005fffff]
>   pci 0000:00:03.0: reg 0x10: [mem 0xf000600000-0xf000601fff]
> 
> This would make /proc/iomem look like this:
> 
>   f000000000-ffffffffff : Bridge MEM
>     f000500000-f0005fffff : 0000:00:02.0
>     f000210000-f00021ffff : 0000:00:00.0
>     f000410000-f00041ffff : 0000:00:01.0
>     f000200000-f000200fff : 0000:00:00.0
>     f000400000-f000400fff : 0000:00:01.0
> 
> This doesn't match the device BARs in your /proc/iomem, so there must
> be some other transformation going on as well.
> 
> As long as you tell the PCI core about the host bridge windows you're
> going to use, along with offsets that include *all* these
> transformations, the core should just work, and /proc/iomem should
> also make sense.  The details of small/medium/big windows, widgets,
> etc., are immaterial to the core.
> 
> Bjorn
> 

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Bjorn Helgaas]

On Fri, Feb 24, 2017 at 03:50:26AM -0500, Joshua Kinard wrote:
> On 02/14/2017 09:56, Bjorn Helgaas wrote:
> > On Tue, Feb 14, 2017 at 02:39:45AM -0500, Joshua Kinard wrote:
> >> On 02/13/2017 17:45, Bjorn Helgaas wrote:
> >>> It looks like IP30 is using some code that's not upstream yet, so I'll
> >>> just point out some things that don't look right.  I'm ignoring the
> >>> IP27 stuff here in case that has different issues.
> >>
> >> Sorry about that.  It's an external set of patches I've been maintaining for
> >> the last decade (I'm not the original author).  Being that I do this for a
> >> hobby, sometimes I get waylaid by other priorities.  I've spent some time
> >> re-organizing my patch collection lately so that I can start sending these
> >> things in.  This series and an earlier one to the Linux/MIPS list are the
> >> beginnings of that, as time permits.  However, I've got a ways to go before
> >> getting to the IP30 patches.  E.g., IP27 comes before IP30, so that's where my
> >> focus has been lately.
> >>
> >> I've put an 4.10-rc7 tree with all patches applied here (including the IOC3,
> >> IP27, and IP30 patches):
> >> http://dev.gentoo.org/~kumba/mips/ip30/linux-4.10_rc7-20170208.ip30/
> >>
> >> The patchset itself, if interested:
> >> http://dev.gentoo.org/~kumba/mips/ip30/mips-patches-4.10.0/
> > 
> > Thanks for the pointers.
> > 
> >>> All the resources under the bridge to 0000:00 are wrong, too:
> >>>
> >>>   1f200000-1f9fffff : Bridge MEM
> >>>     f080100000-f0801fffff : 0000:00:02.0
> >>>     f080010000-f08001ffff : 0000:00:00.0
> >>>     f080030000-f08003ffff : 0000:00:01.0
> >>>     f080000000-f080000fff : 0000:00:00.0
> >>>     f080020000-f080020fff : 0000:00:01.0
> >>
> >> Actually, that's not wrong.  It took me a long time to figure this out, too,
> >> but once I stumbled upon the BRIDGE ASIC specification on an anonymous FTP
> >> site, via Google, by complete accident (hint), it finally made sense.
> >>
> >> BRIDGE is what SGI calls the chip that interfaces the Crosstalk (Xtalk) bus to
> >> a PCI bus.  Most known "XIO" (Xtalk I/O) boards contain a BRIDGE chip under a
> >> heatsink at the edge of the board, next to a "compression connector".  Octane
> >> can hold up to four XIO boards, with a fifth slot dedicated to an optional "PCI
> >> Shoebox" (literally a metal box that can hold up to three PCI or PCI-X cards).
> >> One exception to this are the two graphics boards, Impact/MGRAS and
> >> Odyssey/VPro, which are "pure" XIO devices, as far as anyone's figured out.
> >>
> >> Each BRIDGE chip can address up to eight different PCI devices.
> >>
> >> Crosstalk on IP30 is managed by the Crossbow (XBOW) ASIC.  It implements a
> >> crossbar switch that's not totally unlike a networking switch.  A single XBOW
> >> supports up to 16 XIO widgets, numbered 0x0 to 0xf.  Widgets 0x1 to 0x7 are for
> >> internal use or unused.  XBOW itself is widget 0x0.  HEART, Octane's system
> >> controller, is widget 0x8.  The four standard XIO slots are numbered (I think)
> >> as 0x9 to 0xc.  I think widget 0xe is unused.
> >>
> >> The system board has a BRIDGE chip on it to provide the "BaseIO" devices, such
> >> as two SCSI (QL1040B), IOC3 Ethernet, KB/Mouse, Serial/Parallel, Audio, and an
> >> RTC.  This BaseIO BRIDGE is widget 0xf, which is what you're seeing with the
> >> first set of addresses starting with 0x1fxxx.  The second BRIDGE that you're
> >> seeing is the PCI Shoebox I have installed.  It's widget 0xd, and has addresses
> >> starting with 0x1dxxx (second nibble from left is the widget), with three
> >> random PCI cards I found in a storage bin plugged into it.  The Xtalk scan
> >> happens in reverse order, starting at widget 0xf down to 0x0.
> >>
> >> Both 0x1fxxx and 0x1dxxx addresses are "small window" addresses.  The HEART
> >> system controller gives you three "windows" into Xtalk space, which uses 48-bit
> >> addressing:
> >>
> >> 0x0000_1000_0000 - 0x0000_1fff_ffff - Small windows, x16 @ 16MB each
> >> 0x0008_0000_0000 - 0x000f_ffff_ffff - Medium windows, x16 @ 2GB each
> >> 0x0010_0000_0000 - 0x00ff_ffff_ffff - Big windows, x15 @ 64GB each
> >>
> >> So a device on widget 0xf mapped into a small window at 0x0000_1fxx_xxxx could
> >> also be referred to by a big window at address 0x00f0_8xxx_xxxx.  The only
> >> difference is the big window gives you a larger address space to play around
> >> with.  Medium windows are not currently used on the IP30 platform in Linux.
> >> Widget 0x0 (XBOW) is only accessible via small or medium windows.
> >>
> >> As such, this layout in /proc/iomem is oddly correct:
> >>
> >>>   1f200000-1f9fffff : Bridge MEM
> >>>     f080100000-f0801fffff : 0000:00:02.0
> >>>     f080010000-f08001ffff : 0000:00:00.0
> >>>     f080030000-f08003ffff : 0000:00:01.0
> >>>     f080000000-f080000fff : 0000:00:00.0
> >>>     f080020000-f080020fff : 0000:00:01.0
> >>
> >> My bug is probably that I do the initial BRIDGE scan with small windows in the
> >> main BRIDGE driver (arch/mips/pci/pci-bridge.c), and then IP30's BRIDGE glue
> >> driver (arch/mips/sgi-ip30/ip30-bridge.c) is switching to big windows to probe
> >> each device.  So, yes, different address ranges, but still looking at the same
> >> device.  I'll look into fixing that.  It's partially tied to how I was kludging
> >> the IP27 to also start doing PCI correctly within its own BRIDGE glue driver
> >> (ip27-bridge.c), but I just learned over the weekend that there's missing
> >> functionality I have to port over from old 2.5.x-era IA64 code.  In IP30, big
> >> windows are available by default, but on IP27, what it calls "big windows" need
> >> to go through some kind of translation table, and I'm trying to figure out how
> >> to set that up.
> > 
> > I agree, it sounds like the problem is the switch from small windows
> > to big ones.  In order for the PCI core to work correctly, the host
> > bridge window ("1f200000-1f9fffff : Bridge MEM") must enclose the BARs
> > of the devices below the bridge.
> > 
> > If you use the big windows for the device BARs, you should use big
> > windows for the host bridge windows.  I think we're talking about
> > widget 0xf here, so the 0xf big window would be 0xf0_0000_0000 -
> > 0xff_ffff_ffff.  This should all be set up *before* calling
> > pci_scan_root_bus() instead of after, as it seems to be today.
> 
> Okay, so after a week of downtime due to hardware failure on another machine, I
> started looking at this again, and it looks like it's more than just getting
> the windows wrong.  The existing BRIDGE code (pci-bridge.c) is limiting the PCI
> window to the first five devices on BaseIO in the small window (my fault), and
> the IP30 glue code (ip30-bridge.c) is re-writing the PCI BARs to access the
> BRIDGE PCI Memory and PCI I/O spaces via big windows, without updating the
> original mapping.  We're not teaching the Linux PCI core at all, I think, about
> how to generically access PCI Memory Space and PCI I/O Space on the BRIDGE.
> The fact this actually works appears to be pure luck.

I agree about the luck part :)

> Quoting from the BRIDGE docs somewhat, from the point of view of generic
> Crosstalk space and not specific to IP30 or IP27, there are several views into
> PCI space.  Each address is 48-bits, the size of a Crosstalk address.
> 
>     0000_0000_0000 to 0000_00ff_ffff is "Widget space".  It seems the BRIDGE
>     docs use the term "widget" here to refer to the eight possible PCI devices
>     addressable by a single BRIDGE ASIC.  This plus the small windows is how
>     the code is currently getting things to work.  I think.
> 
>     0000_4000_0000 to 0000_7fff_ffff is BRIDGE's view into PCI Memory Space
>     for direct-mapped 32-bit devices.  It is 1GB in size.  My take is this is
>     normally a 4GB space?  Can the Linux PCI core be taught that only 1GB
>     is usable?

Bridges normally have a window that contains some 32-bit PCI memory
space, because many PCI devices have 32-bit BARs that have to be
located below 4GB.

This window is usually smaller than 4GB (1GB would be typical) because
these devices likely can only generate 32-bit DMA as well, and the DMA
has to use 32-bit PCI addresses that are outside the host bridge
window.

The host bridge code, i.e., the pcibios_scanbus() path, tells the core
how big the window is.  In this case, "hose->mem_resource" contains
the CPU physical address range (and size), and "host->mem_offset"
contains the offset between the CPU physical address and the PCI bus
address.  So if "hose->mem_resource" is only 1GB, that's all the PCI
core will use.

>     0000_8000_0000 to 0000_bfff_ffff is an alias view by BRIDGE into PCI
>     Memory space for 64-bit PCI devices, immediately after the 32-bit space.
>     It is also 1GB in size.
> 
>     0000_c000_0000 to 0000_ffff_ffff is 2GB adjacent to PCI Memory space, but
>     is either unused or used for Crosstalk/BRIDGE DMA operations
>     (the docs aren't very clear on this point or I am simply not
>     understanding).
> 
>     0001_0000_0000 to 0001_ffff_ffff is BRIDGE's view into PCI I/O Space.
>     It has the entire 4GB range available to use.
> 
> 
> I suspect the existing BRIDGE driver is mapping through "widget space" first to
> probe the PCI device slots, then the ip30-bridge.c code is re-writing the PCI
> BARs to go through the big window on IP30, but it never updates the original
> BRIDGE setup (if that's even possible).  As such, as you pointed out, there's
> some kind of logic in the generic PCI core that's re-assigning space from the
> window and we're just getting lucky and everything still works.
> 
> I dug up some old debugging code given to me by the original author of the IP30
> port and built up a set of macros that get me to the PCI Configuration Space on
> Widget 0xf (BaseIO Bridge), Device #0 (the first QLA1040B SCSI controller).
> All three addresses access the same PCI device and return the same config space
> info.
> 
>     Small window:  0x900000001f02xxxx
>     Medium window: 0x9000000f8002xxxx
>     Big window:    0x900000f00002xxxx
> 
> 
> Each additional device's configuration space is offset 0x1000, which gives me
> the following addresses (dev's 4 to 7 are special-cased for IRQ trickery, so
> can't probe them):
> 
>     - Dev #0: scsi0/qla1040b: 0x90xxxxxxxxx20000
>     - Dev #1: scsi1/qla1040b: 0x90xxxxxxxxx21000
>     - Dev #2: io/ioc3       : 0x90xxxxxxxxx22000
>     - Dev #3: audio/rad1    : 0x90xxxxxxxxx23000

PCIe functions have 4K of config space each, so a 0x1000 offset makes
sense.  Whether a platform can access all of it is a separate
question.

It looks like the type 0 config accessors (pci_conf0_read_config(),
pci_conf0_write_config()) are basically like ECAM, where config space
is memory-mapped.

Per spec, the ECAM window for a bridge that could have buses 00-ff
below it would be 256MB (256 buses * 32 devices/bus * 8
functions/device * 4096 bytes config space/function).

But based on b_type0_cfg_dev[], it looks like SGI only made space for
8 devices on the root bus, each with 8 functions.  I think that's OK,
because they can control how many devices can be on the root bus.

The type 1 config accessors (pci_conf1_read_config(),
pci_conf1_write_config()) are for devices below a PCI-to-PCI bridge.
It looks like there's a single 4K memory-mapped window, and you point
it at a specific bus & device with bridge->b_pci_cfg.

The Linux type 0 accessors look like they support 4K config space per
function, while the type 1 accessors put the function number in bits
8-10, so it looks like they only support 256 bytes per function.

The OpenBSD accessors (xbridge_conf_read() and xbridge_conf_write())
look like they only support 256 bytes per function regardless of
whether it's type 0 or type 1.

Supporting 4K of space for type 0 seems like a potential Linux
problem.  Also, pci_conf1_write_config() uses b_type0_cfg_dev in one
place where it looks like it should be using b_type1_cfg.  But that's
in the IOC3 path, and I don't know if that even makes sense for
non-root bus devices -- I doubt you can put an IOC3 behind a
PCI-to-PCI bridge.

If the hardware only supports 256 bytes of config space on non-root
bus devices, that's not a disaster.  We should still be able to
enumerate them and use all the conventional PCI features and even
basic PCIe features.  But the extended config space (offsets
0x100-0xfff) would be inaccessible, and we wouldn't see any PCIe
extended capabilities (AER, VC, SR-IOV, etc., see PCI_EXT_CAP_ID_ERR
and subsequent definitions) because they live in that space.

> If I probe and dump the config space data for each, I get the following:
> 
>     PCI slot 0 information:
>         vendor ID :              0x1077
>         device ID :              0x1020
>         command :                0x0006
>         status :                 0x0200
>         revision :               0x05
>         prog if :                0x00
>         class :                  0x0100
>         cache line :             0x40
>         latency :                0x40
>         hdr type :               0x00
>         BIST :                   0x00
>         region 0 :               0x00200001
>         region 1 :               0x00200000
>         region 2 :               0x00000000
>         region 3 :               0x00000000
>         region 4 :               0x00000000
>         region 5 :               0x00000000
>         IRQ line :               0x00
>         IRQ pin :                0x01
> 
>     PCI slot 1 information:
>         vendor ID :              0x1077
>         device ID :              0x1020
>         command :                0x0006
>         status :                 0x0200
>         revision :               0x05
>         prog if :                0x00
>         class :                  0x0100
>         cache line :             0x40
>         latency :                0x40
>         hdr type :               0x00
>         BIST :                   0x00
>         region 0 :               0x00400001
>         region 1 :               0x00400000
>         region 2 :               0x00000000
>         region 3 :               0x00000000
>         region 4 :               0x00000000
>         region 5 :               0x00000000
>         IRQ line :               0x00
>         IRQ pin :                0x01
> 
>     PCI slot 2 information:
>         vendor ID :              0x10a9
>         device ID :              0x0003
>         command :                0x0146
>         status :                 0x0280
>         revision :               0x01
>         prog if :                0x00
>         class :                  0xff00
>         cache line :             0x00
>         latency :                0x28
>         hdr type :               0x00
>         BIST :                   0x00
>         region 0 :               0x00500000
>         region 1 :               0x00000000
>         region 2 :               0x00000000
>         region 3 :               0x00500000
>         region 4 :               0x000310a9
>         region 5 :               0x02800146
>         IRQ line :               0x00
>         IRQ pin :                0x00
> 
>     PCI slot 3 information:
>         vendor ID :              0x10a9
>         device ID :              0x0005
>         command :                0x0006
>         status :                 0x0480
>         revision :               0xc0
>         prog if :                0x00
>         class :                  0x0000
>         cache line :             0x00
>         latency :                0xff
>         hdr type :               0x00
>         BIST :                   0x00
>         region 0 :               0x00600000
>         region 1 :               0x00000000
>         region 2 :               0x00000000
>         region 3 :               0x00000000
>         region 4 :               0x00000000
>         region 5 :               0x00000000
>         IRQ line :               0x00
>         IRQ pin :                0x00
> 
> This is where my knowledge runs short -- I'm not fluent in PCI device
> programming, so I'm not sure what is really supposed to be going on with these
> different BAR regions.  I dug up a copy of the PCI Spec 2.2, but it's 322 pages
> and I'm not sure where I should start reading from.  Are these regions
> device-specific?  E.g., do I need the QLA1040B programming manual to understand
> why region 0 is 0x00200001 and region 1 is 0x00200000?

The regions (BARs) are definitely device-specific.  The low order bit
in some of them indicates an I/O BAR.  In the PCI r3.0 spec, sec 6.2.5
covers "Base Addresses" and tells you how to interpret the low bits.
For most of the BARs above, they're either 0b01, indicating an I/O
BAR, or 0b00000, indicating a non-prefetchable 32-bit memory BAR.

> I am also not yet considering how PCI views Crosstalk space for DMA operations
> -- BRIDGE has several mechanisms for that, depending on if the PCI device is a
> 32-bit device or a 64-bit device.  It can do direct-mapping for 32-bit or
> 64-bit, or built-in page-mapping hardware is available (but apparently to be
> avoided due to numerous hardware quirks/bugs that would make the driver
> overly-complicated).
> 
> There's a good write-up in the OpenBSD "xbridge" driver, which handles BRIDGE
> (IP27/IP30), XBRIDGE (IP35), and PIC (IP35/IA64 Altix):
> 
> http://bxr.su/OpenBSD/sys/arch/sgi/xbow/xbridge.c
> 
> It's starting to make some sense to me, but I am still uncertain how to work
> with the 32-bit and 64-bit 1GB PCI memory spaces on BRIDGE as well as the 4GB
> PCI I/O space from a Linux point of view.  Do they only matter when dealing
> with DMA?

The host bridge windows (the "root bus resource" lines in dmesg) are
only for PIO, i.e., a driver running on the CPU reading or writing
registers on the PCI device (it could be actual registers, or a frame
buffer, etc., but it resides on the device and its PCI bus address is
determined by a BAR).  The driver uses ioremap(), pci_iomap(),
pcim_iomap_regions(), etc. to map these into the kernel virtual space.

DMA is coming the other direction and the windows are irrelevant
except that the target PCI bus address must be outside all the windows
(if the DMA target address were *inside* a host bridge window, the
bridge would assume it is intended for a PCI device, e.g., for
peer-to-peer DMA).

I think the biggest problem is that you need to set up the BRIDGE
*before* calling pcibios_scan_bus().  That way the windows
("hose->mem_resource", "hose->io_resource", etc.) will be correct when
the PCI core enumerates the devices.  Note that you can have as many
windows in the "&resources" list as you need -- the current code there
only has one memory and one I/O window, but you can add more.

> > The scan would look like this:
> > 
> >   pci_bus 0000:00: root bus resource [mem 0xf000000000-0xffffffffff] (bus addresses [0x00000000-0xfffffffff])
> >   pci 0000:00:00.0: reg 0x14: [mem 0xf000200000-0xf000200fff]
> >   pci 0000:00:00.0: reg 0x30: [mem 0xf000210000-0xf00021ffff pref]
> >   pci 0000:00:01.0: reg 0x14: [mem 0xf000400000-0xf000400fff]
> >   pci 0000:00:01.0: reg 0x30: [mem 0xf000410000-0xf00041ffff pref]
> >   pci 0000:00:02.0: reg 0x10: [mem 0xf000500000-0xf0005fffff]
> >   pci 0000:00:03.0: reg 0x10: [mem 0xf000600000-0xf000601fff]
> > 
> > This would make /proc/iomem look like this:
> > 
> >   f000000000-ffffffffff : Bridge MEM
> >     f000500000-f0005fffff : 0000:00:02.0
> >     f000210000-f00021ffff : 0000:00:00.0
> >     f000410000-f00041ffff : 0000:00:01.0
> >     f000200000-f000200fff : 0000:00:00.0
> >     f000400000-f000400fff : 0000:00:01.0
> > 
> > This doesn't match the device BARs in your /proc/iomem, so there must
> > be some other transformation going on as well.
> > 
> > As long as you tell the PCI core about the host bridge windows you're
> > going to use, along with offsets that include *all* these
> > transformations, the core should just work, and /proc/iomem should
> > also make sense.  The details of small/medium/big windows, widgets,
> > etc., are immaterial to the core.
> > 
> > Bjorn
> > 
> 

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Joshua Kinard]

On 02/24/2017 13:38, Bjorn Helgaas wrote:
> On Fri, Feb 24, 2017 at 03:50:26AM -0500, Joshua Kinard wrote:
>> Quoting from the BRIDGE docs somewhat, from the point of view of generic
>> Crosstalk space and not specific to IP30 or IP27, there are several views into
>> PCI space.  Each address is 48-bits, the size of a Crosstalk address.
>>
>>     0000_0000_0000 to 0000_00ff_ffff is "Widget space".  It seems the BRIDGE
>>     docs use the term "widget" here to refer to the eight possible PCI devices
>>     addressable by a single BRIDGE ASIC.  This plus the small windows is how
>>     the code is currently getting things to work.  I think.
>>
>>     0000_4000_0000 to 0000_7fff_ffff is BRIDGE's view into PCI Memory Space
>>     for direct-mapped 32-bit devices.  It is 1GB in size.  My take is this is
>>     normally a 4GB space?  Can the Linux PCI core be taught that only 1GB
>>     is usable?
> 
> Bridges normally have a window that contains some 32-bit PCI memory
> space, because many PCI devices have 32-bit BARs that have to be
> located below 4GB.
> 
> This window is usually smaller than 4GB (1GB would be typical) because
> these devices likely can only generate 32-bit DMA as well, and the DMA
> has to use 32-bit PCI addresses that are outside the host bridge
> window.

Okay, this fits then.  I wasn't sure, because the verbiage in the BRIDGE docs
kinda-suggests that the 1GB window for 32/64-bit memory space isn't common.
Likely, what they're getting at is the fact the lower 1GB for 32-bit is aliased
in the next 1GB for use by 64-bit PCI devices.  It looks like the remaining 2GB
in memory space is available for DMA, and I'll have to try and wrap my head
around that at later date.


> The host bridge code, i.e., the pcibios_scanbus() path, tells the core
> how big the window is.  In this case, "hose->mem_resource" contains
> the CPU physical address range (and size), and "host->mem_offset"
> contains the offset between the CPU physical address and the PCI bus
> address.  So if "hose->mem_resource" is only 1GB, that's all the PCI
> core will use.

So basically, this bit of code from the proposed pci-bridge.c needs additional
work:

	bc->mem.name = "Bridge MEM";
	bc->mem.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_MEM);
	bc->mem.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO - 1);
	bc->mem.flags = IORESOURCE_MEM;

	bc->io.name = "Bridge IO";
	bc->io.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO);
	bc->io.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_END - 1);
	bc->io.flags = IORESOURCE_IO;

The values I've been using for PCIBR_OFFSET_MEM and PCIBR_OFFSET_IO are
obviously wrong.  For IP30, they ultimately direct "BRIDGE MEM" to the first
five device slots via small windows, and then "Bridge IO" to the last two
slots.  This is one of those "luck" moments that happened to work on BaseIO,
but is probably why devices in the secondary PCI shoebox are hit-or-miss.

Per your comment further down that I can tell the PCI core about //multiple//
windows, probably what I want to do is hardwire access to the small window
spaces in the main BRIDGE driver, with machine-specific offsets passed via
platform_data.  Small windows, for both IP30 and IP27, are always guaranteed to
be available without any additional magic.  This window range can be used to do
the initial slot probes to query devices for their desired BAR values.

Then, in IP30-specific or IP27-specific code, we add in additional memory
windows as needed.  IP30 would simply pass along the hardwired address ranges
for BRIDGE MEM or BRIDGE IO via both medium and big window addresses.

IP27 is trickier -- we'll need some logic that attempts to use a small window
(16MB) mapping first, but if that is too small, we'll have to setup an entry in
the HUB's IOTTE buffer that maps a specific block of physical memory to
Crosstalk address space.  This is what IP27 calls "big windows", of which there
are seven entries maximum, on 512MB boundaries, per HUB chip.  However, only
six entries are available because one IOTTE is used to handle a known hardware
bug on early HUB revisions, leaving only six left.

As such, the additional window information will go into the machine-specific
BRIDGE glue drivers (ip30-bridge.c or ip27-bridge.c).

Sound sane?


>>     0000_8000_0000 to 0000_bfff_ffff is an alias view by BRIDGE into PCI
>>     Memory space for 64-bit PCI devices, immediately after the 32-bit space.
>>     It is also 1GB in size.
>>
>>     0000_c000_0000 to 0000_ffff_ffff is 2GB adjacent to PCI Memory space, but
>>     is either unused or used for Crosstalk/BRIDGE DMA operations
>>     (the docs aren't very clear on this point or I am simply not
>>     understanding).
>>
>>     0001_0000_0000 to 0001_ffff_ffff is BRIDGE's view into PCI I/O Space.
>>     It has the entire 4GB range available to use.
>>
>>
>> I suspect the existing BRIDGE driver is mapping through "widget space" first to
>> probe the PCI device slots, then the ip30-bridge.c code is re-writing the PCI
>> BARs to go through the big window on IP30, but it never updates the original
>> BRIDGE setup (if that's even possible).  As such, as you pointed out, there's
>> some kind of logic in the generic PCI core that's re-assigning space from the
>> window and we're just getting lucky and everything still works.
>>
>> I dug up some old debugging code given to me by the original author of the IP30
>> port and built up a set of macros that get me to the PCI Configuration Space on
>> Widget 0xf (BaseIO Bridge), Device #0 (the first QLA1040B SCSI controller).
>> All three addresses access the same PCI device and return the same config space
>> info.
>>
>>     Small window:  0x900000001f02xxxx
>>     Medium window: 0x9000000f8002xxxx
>>     Big window:    0x900000f00002xxxx
>>
>>
>> Each additional device's configuration space is offset 0x1000, which gives me
>> the following addresses (dev's 4 to 7 are special-cased for IRQ trickery, so
>> can't probe them):
>>
>>     - Dev #0: scsi0/qla1040b: 0x90xxxxxxxxx20000
>>     - Dev #1: scsi1/qla1040b: 0x90xxxxxxxxx21000
>>     - Dev #2: io/ioc3       : 0x90xxxxxxxxx22000
>>     - Dev #3: audio/rad1    : 0x90xxxxxxxxx23000
> 
> PCIe functions have 4K of config space each, so a 0x1000 offset makes
> sense.  Whether a platform can access all of it is a separate
> question.

No need to worry about PCIe on this platform.  You're talking about 15+-year
old hardware that was built to last.  I believe SGI only implemented up to PCI
2.1 in the BRIDGE and XBRIDGE ASICs.  It's possible the PIC ASIC might do 2.2
or 3.0.  But I don't have to worry about PIC for a good while, as that's only
found in the IP35-class of hardware (Origin 300/350/3000, Fuel, Tezro, Onyx4,
etc).  We need fully-working IP27 and IP30 first, because much of the working
logic can then be re-purposed for bringing up IP35 hardware.  Especially since
IA64 did a lot of the work already for the Altix systems, which is just IP35
with an IA64 CPU, but I digress...


> It looks like the type 0 config accessors (pci_conf0_read_config(),
> pci_conf0_write_config()) are basically like ECAM, where config space
> is memory-mapped.
> 
> Per spec, the ECAM window for a bridge that could have buses 00-ff
> below it would be 256MB (256 buses * 32 devices/bus * 8
> functions/device * 4096 bytes config space/function).
> 
> But based on b_type0_cfg_dev[], it looks like SGI only made space for
> 8 devices on the root bus, each with 8 functions.  I think that's OK,
> because they can control how many devices can be on the root bus.

Not sure what ECAM is, but yes, SGI wired each BRIDGE ASIC for a maximum of 8
PCI devices (or slots).  There can be multiple BRIDGEs in an SGI system,
though, and for several of their XIO add-in boards, you have one PCI device
hardwired into a BRIDGE ASIC, so whatever logic I cook up is going to need to
handle BRIDGE's with one device or eight.  So much for this being easy...


> The type 1 config accessors (pci_conf1_read_config(),
> pci_conf1_write_config()) are for devices below a PCI-to-PCI bridge.
> It looks like there's a single 4K memory-mapped window, and you point
> it at a specific bus & device with bridge->b_pci_cfg.

Yeah, PCI-to-PCI bridges are a known-broken item right now.  Since there
appears to only be a single Type 1 config space, if I have a PCI Shoebox with
three PCI-X slots, and I stick three USB 2.0 boards in them, I have to pick
which one can handle PCI-to-PCI bridges, like USB hubs, and program that into
this Type 1 register, right?


> The Linux type 0 accessors look like they support 4K config space per
> function, while the type 1 accessors put the function number in bits
> 8-10, so it looks like they only support 256 bytes per function.
> 
> The OpenBSD accessors (xbridge_conf_read() and xbridge_conf_write())
> look like they only support 256 bytes per function regardless of
> whether it's type 0 or type 1.

I'll have to pass this question to the OpenBSD dev who maintains the xbridge
driver.  He understands this hardware far better than I do, and there might be
a reason why they do it that way.


> Supporting 4K of space for type 0 seems like a potential Linux
> problem.  Also, pci_conf1_write_config() uses b_type0_cfg_dev in one
> place where it looks like it should be using b_type1_cfg.  But that's
> in the IOC3 path, and I don't know if that even makes sense for
> non-root bus devices -- I doubt you can put an IOC3 behind a
> PCI-to-PCI bridge.

IOC3 is....special.  You'll never see one of those beind a PCI-to-PCI bridge,
thankfully.  That device is evil enough just sitting exposed on the main PCI
bus.  Just look at the comments in mainline arch/mips/pci/ops-bridge.c for some
rather flavourful language regarding IOC3.  Basically, IOC3:

  - Claims it's a single-function device but is really multi-function.
  - Implements only half of the PCI configuration space.
  - Read/write ops to the unimplemented PCI config registers are undefined.

Living "behind" the IOC3 device is your ethernet, serial ports (RS232 and
RS422), parallel port, keyboard/mouse PS/2 ports, and the RTC chip.  The IA64
people have an IOC3 "metadriver" in drivers/sn/ioc3.c that takes care of
dealing with that monster, and one of the later patches I've yet to send in
moves that driver to drivers/misc/ioc3.c (next to ioc4.c) for use by the SGI
MIPS platforms.  But that is a ways off, as we need working Xtalk code and
working BRIDGE code before the IOC3 metadriver can be worked on.


> If the hardware only supports 256 bytes of config space on non-root
> bus devices, that's not a disaster.  We should still be able to
> enumerate them and use all the conventional PCI features and even
> basic PCIe features.  But the extended config space (offsets
> 0x100-0xfff) would be inaccessible, and we wouldn't see any PCIe
> extended capabilities (AER, VC, SR-IOV, etc., see PCI_EXT_CAP_ID_ERR
> and subsequent definitions) because they live in that space.

I do not think that the BRIDGE supports AER, VC, or SR-IOV at all, primarily
due to age of the hardware.  Even if a PCI device that does support those is
plugged into a BRIDGE, I am not 100% certain those features would even be
usable if the BRIDGE doesn't know about them.  Or is this a case of where the
BRIDGE wouldn't care and once it programs the BARs correctly, these features
would be available to the Linux drivers?


>> If I probe and dump the config space data for each, I get the following:
>>
>>     PCI slot 0 information:
>>         vendor ID :              0x1077
>>         device ID :              0x1020
>>         command :                0x0006
>>         status :                 0x0200
>>         revision :               0x05
>>         prog if :                0x00
>>         class :                  0x0100
>>         cache line :             0x40
>>         latency :                0x40
>>         hdr type :               0x00
>>         BIST :                   0x00
>>         region 0 :               0x00200001
>>         region 1 :               0x00200000
>>         region 2 :               0x00000000
>>         region 3 :               0x00000000
>>         region 4 :               0x00000000
>>         region 5 :               0x00000000
>>         IRQ line :               0x00
>>         IRQ pin :                0x01
>>
>>     PCI slot 1 information:
>>         vendor ID :              0x1077
>>         device ID :              0x1020
>>         command :                0x0006
>>         status :                 0x0200
>>         revision :               0x05
>>         prog if :                0x00
>>         class :                  0x0100
>>         cache line :             0x40
>>         latency :                0x40
>>         hdr type :               0x00
>>         BIST :                   0x00
>>         region 0 :               0x00400001
>>         region 1 :               0x00400000
>>         region 2 :               0x00000000
>>         region 3 :               0x00000000
>>         region 4 :               0x00000000
>>         region 5 :               0x00000000
>>         IRQ line :               0x00
>>         IRQ pin :                0x01
>>
>>     PCI slot 2 information:
>>         vendor ID :              0x10a9
>>         device ID :              0x0003
>>         command :                0x0146
>>         status :                 0x0280
>>         revision :               0x01
>>         prog if :                0x00
>>         class :                  0xff00
>>         cache line :             0x00
>>         latency :                0x28
>>         hdr type :               0x00
>>         BIST :                   0x00
>>         region 0 :               0x00500000
>>         region 1 :               0x00000000
>>         region 2 :               0x00000000
>>         region 3 :               0x00500000
>>         region 4 :               0x000310a9
>>         region 5 :               0x02800146
>>         IRQ line :               0x00
>>         IRQ pin :                0x00
>>
>>     PCI slot 3 information:
>>         vendor ID :              0x10a9
>>         device ID :              0x0005
>>         command :                0x0006
>>         status :                 0x0480
>>         revision :               0xc0
>>         prog if :                0x00
>>         class :                  0x0000
>>         cache line :             0x00
>>         latency :                0xff
>>         hdr type :               0x00
>>         BIST :                   0x00
>>         region 0 :               0x00600000
>>         region 1 :               0x00000000
>>         region 2 :               0x00000000
>>         region 3 :               0x00000000
>>         region 4 :               0x00000000
>>         region 5 :               0x00000000
>>         IRQ line :               0x00
>>         IRQ pin :                0x00
>>
>> This is where my knowledge runs short -- I'm not fluent in PCI device
>> programming, so I'm not sure what is really supposed to be going on with these
>> different BAR regions.  I dug up a copy of the PCI Spec 2.2, but it's 322 pages
>> and I'm not sure where I should start reading from.  Are these regions
>> device-specific?  E.g., do I need the QLA1040B programming manual to understand
>> why region 0 is 0x00200001 and region 1 is 0x00200000?
> 
> The regions (BARs) are definitely device-specific.  The low order bit
> in some of them indicates an I/O BAR.  In the PCI r3.0 spec, sec 6.2.5
> covers "Base Addresses" and tells you how to interpret the low bits.
> For most of the BARs above, they're either 0b01, indicating an I/O
> BAR, or 0b00000, indicating a non-prefetchable 32-bit memory BAR.

Okay, I stumbled across this URL:
http://moi.vonos.net/linux/the-pci-bus/

Which gives a good, plain-english breakdown of BARs and whatnot.

I remember from the OpenBSD xbridge driver, it states that it currently uses
the BAR mappings setup by the ARCS firmware, but the plan is to eventually move
away from those mappings and calculate its own.  I didn't quite get that, but
after reading that site, now I do.  ARCS is writing ~0 to the BARs to query the
device to figure out how much memory each BAR wants, then programs in some
ranges for us.

By tallying up the total requested size of all BAR mappings on a given device,
we can then make a determination on whether we can use a small, medium, or
large window and then program the BAR with the correct address.  Right?

Could also do my own probes (or maybe Linux already does this?), but for the
short-term, using what ARCS has already setup might not be a bad idea to get
things working again.


>> I am also not yet considering how PCI views Crosstalk space for DMA operations
>> -- BRIDGE has several mechanisms for that, depending on if the PCI device is a
>> 32-bit device or a 64-bit device.  It can do direct-mapping for 32-bit or
>> 64-bit, or built-in page-mapping hardware is available (but apparently to be
>> avoided due to numerous hardware quirks/bugs that would make the driver
>> overly-complicated).
>>
>> There's a good write-up in the OpenBSD "xbridge" driver, which handles BRIDGE
>> (IP27/IP30), XBRIDGE (IP35), and PIC (IP35/IA64 Altix):
>>
>> http://bxr.su/OpenBSD/sys/arch/sgi/xbow/xbridge.c
>>
>> It's starting to make some sense to me, but I am still uncertain how to work
>> with the 32-bit and 64-bit 1GB PCI memory spaces on BRIDGE as well as the 4GB
>> PCI I/O space from a Linux point of view.  Do they only matter when dealing
>> with DMA?
> 
> The host bridge windows (the "root bus resource" lines in dmesg) are
> only for PIO, i.e., a driver running on the CPU reading or writing
> registers on the PCI device (it could be actual registers, or a frame
> buffer, etc., but it resides on the device and its PCI bus address is
> determined by a BAR).  The driver uses ioremap(), pci_iomap(),
> pcim_iomap_regions(), etc. to map these into the kernel virtual space.

So for the MIPS case, knowing that ioremap() and friends is not guaranteed to
return a workable virtual address, I need to be careful of what addresses I
program into each BAR.  E.g., given that on IP30, if a physical address starts
with 0x9000000f800xxxxx, it is using medium windows to talk to a device on the
BaseIO BRIDGE (Xtalk widget 0xf).  As such, knowing MIPS' ioremap() returns,
for the R10000 CPU case, the requested address OR'ed with IO_BASE
(0x9000000000000000), I want to tell the PCI core to use 0x0000000f800xxxxx so
that ioremap() will return back 0x9000000f800xxxxx?

And since I can apparently specify multiple window ranges for memory space and
I/O space, I probably want to, as stated earlier, specify all three of IP30's
known window ranges for each BRIDGE so that the Linux PCI core can walk each
resource struct and find a matching window?

If that thinking is correct, then I have some idea of how to set that up and
then see if things start working again on IP30 and then eventually, IP27.


> DMA is coming the other direction and the windows are irrelevant
> except that the target PCI bus address must be outside all the windows
> (if the DMA target address were *inside* a host bridge window, the
> bridge would assume it is intended for a PCI device, e.g., for
> peer-to-peer DMA).

DMA is the tricky one.  OpenBSD's xbridge driver implies that BRIDGE's IOMMU
has a bug of some kind in it that restricts our DMA-able address ranges to 31
bits, or 0x00000000 to 0x7fffffff.  IP27 doesn't seem to be bothered by this in
the current code, and as such, I can use all 8GB of RAM on that platform.

But IP30 has its physical memory offset 512MB, so physical memory maps begin at
0x20000000 and run to 0x9fffffff.  This means someone playing with Linux on an
IP30 platform can only install up to 2GB of RAM right now, if they want the
machine to be stable.  Though, there are other issues going on that imply that
"stable" is a very subjective term...

In either event, how do I teach the Linux PCI core that the BRIDGE itself is
limited to 31bits of DMA-able address space?  There seems to be plenty of
documentation on having an individual PCI device set this up by setting its DMA
mask, but I can't seem to find any wording on how to do this for an entire PCI bus.

Or is this an architecture-specific thing?  MIPS has the dma-coherence.h header
for defining system-specific DMA quirks, but that code seems heavily biased
towards PCI devices, and on IP30 at least, the Impact and Odyssey video cards
are NOT PCI devices at all, instead being native XIO devices that support DMA
operations.


> I think the biggest problem is that you need to set up the BRIDGE
> *before* calling pcibios_scan_bus().  That way the windows
> ("hose->mem_resource", "hose->io_resource", etc.) will be correct when
> the PCI core enumerates the devices.  Note that you can have as many
> windows in the "&resources" list as you need -- the current code there
> only has one memory and one I/O window, but you can add more.

Agreed, however, I think this is actually being done to some extent already.
We're configuring BRIDGE-specific properties and writing some values to BRIDGE
registers and the per-slot registers in bridge_probe() in the generic
pci-bridge.c driver, then, at the very end, calling register_pci_controller(),
which I believe is what kicks off the PCI bus scan.

The IP30 BRIDGE glue code adds a new function hook called "pre_enable" where it
does, in ip30-bridge.c, some additional PCI device tweaking, but this code will
run after the PCI bus scan has happened.  My guess is that I want to reverse
this and have the IP30 glue code twiddle the PCI devices on a given BRIDGE
before the PCI bus scan, right?  That code uses pci_read_config_dword() and
pci_write_config_dword() -- are these functions safe to use if we haven't
already done a PCI bus scan?

Thanks!

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Bjorn Helgaas]

On Sat, Feb 25, 2017 at 04:34:12AM -0500, Joshua Kinard wrote:
> On 02/24/2017 13:38, Bjorn Helgaas wrote:
> > On Fri, Feb 24, 2017 at 03:50:26AM -0500, Joshua Kinard wrote:

> > The host bridge code, i.e., the pcibios_scanbus() path, tells the
> > core how big the window is.  In this case, "hose->mem_resource"
> > contains the CPU physical address range (and size), and
> > "host->mem_offset" contains the offset between the CPU physical
> > address and the PCI bus address.  So if "hose->mem_resource" is
> > only 1GB, that's all the PCI core will use.
> 
> So basically, this bit of code from the proposed pci-bridge.c needs
> additional work:
> 
> 	bc->mem.name = "Bridge MEM";
> 	bc->mem.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_MEM);
> 	bc->mem.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO - 1);
> 	bc->mem.flags = IORESOURCE_MEM;
> 
> 	bc->io.name = "Bridge IO";
> 	bc->io.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO);
> 	bc->io.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_END - 1);
> 	bc->io.flags = IORESOURCE_IO;
> 
> The values I've been using for PCIBR_OFFSET_MEM and PCIBR_OFFSET_IO
> are obviously wrong.  For IP30, they ultimately direct "BRIDGE MEM"
> to the first five device slots via small windows, and then "Bridge
> IO" to the last two slots.  This is one of those "luck" moments that
> happened to work on BaseIO, but is probably why devices in the
> secondary PCI shoebox are hit-or-miss.
> 
> Per your comment further down that I can tell the PCI core about
> //multiple// windows, probably what I want to do is hardwire access
> to the small window spaces in the main BRIDGE driver, with
> machine-specific offsets passed via platform_data.  Small windows,
> for both IP30 and IP27, are always guaranteed to be available
> without any additional magic.  This window range can be used to do
> the initial slot probes to query devices for their desired BAR
> values.
> 
> Then, in IP30-specific or IP27-specific code, we add in additional
> memory windows as needed.  IP30 would simply pass along the
> hardwired address ranges for BRIDGE MEM or BRIDGE IO via both medium
> and big window addresses.
> 
> IP27 is trickier -- we'll need some logic that attempts to use a
> small window (16MB) mapping first, but if that is too small, we'll
> have to setup an entry in the HUB's IOTTE buffer that maps a
> specific block of physical memory to Crosstalk address space.  

pcibios_scanbus() builds the &resources list of host bridge windows,
then calls pci_scan_root_bus().  If you're proposing to change the
list of host bridge windows *after* calling pci_scan_root_bus(), that
sounds a little problematic because we do not have a PCI core
interface to do that.

> This is what IP27 calls "big windows", of which there are seven
> entries maximum, on 512MB boundaries, per HUB chip.  However, only
> six entries are available because one IOTTE is used to handle a
> known hardware bug on early HUB revisions, leaving only six left.
> 
> As such, the additional window information will go into the
> machine-specific BRIDGE glue drivers (ip30-bridge.c or
> ip27-bridge.c).
> 
> Sound sane?

Do you need to allocate the big windows based on what PCI devices you
find?  If so, that is going to be a hard problem.

> > PCIe functions have 4K of config space each, so a 0x1000 offset
> > makes sense.  Whether a platform can access all of it is a
> > separate question.
> 
> No need to worry about PCIe on this platform.  You're talking about
> 15+-year old hardware that was built to last.  I believe SGI only
> implemented up to PCI 2.1 in the BRIDGE and XBRIDGE ASICs.  It's
> possible the PIC ASIC might do 2.2 or 3.0.  But I don't have to
> worry about PIC for a good while, as that's only found in the
> IP35-class of hardware (Origin 300/350/3000, Fuel, Tezro, Onyx4,
> etc).  We need fully-working IP27 and IP30 first, because much of
> the working logic can then be re-purposed for bringing up IP35
> hardware.  Especially since IA64 did a lot of the work already for
> the Altix systems, which is just IP35 with an IA64 CPU, but I
> digress...

OK.  If you only plug in PCI devices, there should be no problem.  If
you did plug in PCIe devices (via a PCI-to-PCIe bridge, for example),
the core should still enumerate them correctly and they should be
functional, though some PCIe-only features like AER won't work.

> > The type 1 config accessors (pci_conf1_read_config(),
> > pci_conf1_write_config()) are for devices below a PCI-to-PCI
> > bridge.  It looks like there's a single 4K memory-mapped window,
> > and you point it at a specific bus & device with
> > bridge->b_pci_cfg.
> 
> Yeah, PCI-to-PCI bridges are a known-broken item right now.  Since
> there appears to only be a single Type 1 config space, if I have a
> PCI Shoebox with three PCI-X slots, and I stick three USB 2.0 boards
> in them, I have to pick which one can handle PCI-to-PCI bridges,
> like USB hubs, and program that into this Type 1 register, right?

I don't think USB hubs are relevant here because they are completely
in the USB domain.  The USB 2.0 board is a USB host controller with a
PCI interface on it.  The USB hub is on the USB side and is invisible
to the PCI core.

The way I read the BSD code, it looks like you should be able to use
the Type 1 config interface to generate accesses to any PCI bus, as
long as you serialize them, e.g., with a lock around the use of
b_type1_cfg.

> > The Linux type 0 accessors look like they support 4K config space
> > per function, while the type 1 accessors put the function number
> > in bits 8-10, so it looks like they only support 256 bytes per
> > function.
> > 
> > The OpenBSD accessors (xbridge_conf_read() and
> > xbridge_conf_write()) look like they only support 256 bytes per
> > function regardless of whether it's type 0 or type 1.
> 
> I'll have to pass this question to the OpenBSD dev who maintains the
> xbridge driver.  He understands this hardware far better than I do,
> and there might be a reason why they do it that way.

If you only support PCI devices, 256 bytes of config space is enough.
The 4K config space is only supported for PCI-X Mode 2 and PCIe
devices.  Even those PCI-X Mode 2 and PCIe devices should be
functional with only 256 bytes.

> > Supporting 4K of space for type 0 seems like a potential Linux
> > problem.  Also, pci_conf1_write_config() uses b_type0_cfg_dev in
> > one place where it looks like it should be using b_type1_cfg.

I didn't word this well.  I was trying to point out things that look
like bugs in pci_conf1_write_config() (the use of b_type0_cfg_dev) and
maybe pci_conf0_read_config() (the fact that it allows 4K config space
but the hardware probably only supports 256 bytes).

> > If the hardware only supports 256 bytes of config space on
> > non-root bus devices, that's not a disaster.  We should still be
> > able to enumerate them and use all the conventional PCI features
> > and even basic PCIe features.  But the extended config space
> > (offsets 0x100-0xfff) would be inaccessible, and we wouldn't see
> > any PCIe extended capabilities (AER, VC, SR-IOV, etc., see
> > PCI_EXT_CAP_ID_ERR and subsequent definitions) because they live
> > in that space.
> 
> I do not think that the BRIDGE supports AER, VC, or SR-IOV at all,
> primarily due to age of the hardware.  Even if a PCI device that
> does support those is plugged into a BRIDGE, I am not 100% certain
> those features would even be usable if the BRIDGE doesn't know about
> them.  Or is this a case of where the BRIDGE wouldn't care and once
> it programs the BARs correctly, these features would be available to
> the Linux drivers?

These are mostly PCIe features and I don't think you should worry
about them, at least for now.

> I remember from the OpenBSD xbridge driver, it states that it
> currently uses the BAR mappings setup by the ARCS firmware, but the
> plan is to eventually move away from those mappings and calculate
> its own.  I didn't quite get that, but after reading that site, now
> I do.  ARCS is writing ~0 to the BARs to query the device to figure
> out how much memory each BAR wants, then programs in some ranges for
> us.
> 
> By tallying up the total requested size of all BAR mappings on a
> given device, we can then make a determination on whether we can use
> a small, medium, or large window and then program the BAR with the
> correct address.  Right?

This is the problem I alluded to above: Linux does not currently
support anything like this.  We assume the Linux host bridge driver
knows the window sizes at the beginning of time (it may learn them
from firmware or it may have hard-coded knowledge of the address map).

Linux does enumerate the devices and tally up all the BAR sizes (see
pci_bus_assign_resources()), but it does not have a way to change the
host bridge windows based on how much BAR space we need.

The common thing is to use whatever host bridge windows and device BAR
values were set up by firmware.  If those are all sensible, Linux
won't change anything.

> > The host bridge windows (the "root bus resource" lines in dmesg)
> > are only for PIO, i.e., a driver running on the CPU reading or
> > writing registers on the PCI device (it could be actual registers,
> > or a frame buffer, etc., but it resides on the device and its PCI
> > bus address is determined by a BAR).  The driver uses ioremap(),
> > pci_iomap(), pcim_iomap_regions(), etc. to map these into the
> > kernel virtual space.
> 
> So for the MIPS case, knowing that ioremap() and friends is not
> guaranteed to return a workable virtual address, I need to be
> careful of what addresses I program into each BAR.  E.g., given that
> on IP30, if a physical address starts with 0x9000000f800xxxxx, it is
> using medium windows to talk to a device on the BaseIO BRIDGE (Xtalk
> widget 0xf).  As such, knowing MIPS' ioremap() returns, for the
> R10000 CPU case, the requested address OR'ed with IO_BASE
> (0x9000000000000000), I want to tell the PCI core to use
> 0x0000000f800xxxxx so that ioremap() will return back
> 0x9000000f800xxxxx?

If ioremap() doesn't return a virtual address, or at least something
that can be used like a virtual address, I think that's fundamentally
broken.  All drivers assume they should ioremap() a BAR resource,
i.e., the CPU physical address that maps to a PCI BAR value, and use
the result as a virtual address.

Documentation/DMA-API-HOWTO.txt has a picture that may help clarify
the different address spaces.

> And since I can apparently specify multiple window ranges for memory
> space and I/O space, I probably want to, as stated earlier, specify
> all three of IP30's known window ranges for each BRIDGE so that the
> Linux PCI core can walk each resource struct and find a matching
> window?

If all three windows are disjoint, you can specify them all.  If a
range in the small window is aliased and can also be reached via a
medium or large window, you should not specify both.

> > DMA is coming the other direction and the windows are irrelevant
> > except that the target PCI bus address must be outside all the
> > windows (if the DMA target address were *inside* a host bridge
> > window, the bridge would assume it is intended for a PCI device,
> > e.g., for peer-to-peer DMA).
> 
> DMA is the tricky one.  OpenBSD's xbridge driver implies that
> BRIDGE's IOMMU has a bug of some kind in it that restricts our
> DMA-able address ranges to 31 bits, or 0x00000000 to 0x7fffffff.
> IP27 doesn't seem to be bothered by this in the current code, and as
> such, I can use all 8GB of RAM on that platform.
> 
> But IP30 has its physical memory offset 512MB, so physical memory
> maps begin at 0x20000000 and run to 0x9fffffff.  This means someone
> playing with Linux on an IP30 platform can only install up to 2GB of
> RAM right now, if they want the machine to be stable.  Though, there
> are other issues going on that imply that "stable" is a very
> subjective term...
> 
> In either event, how do I teach the Linux PCI core that the BRIDGE
> itself is limited to 31bits of DMA-able address space?  There seems
> to be plenty of documentation on having an individual PCI device set
> this up by setting its DMA mask, but I can't seem to find any
> wording on how to do this for an entire PCI bus.
> 
> Or is this an architecture-specific thing?  MIPS has the
> dma-coherence.h header for defining system-specific DMA quirks, but
> that code seems heavily biased towards PCI devices, and on IP30 at
> least, the Impact and Odyssey video cards are NOT PCI devices at
> all, instead being native XIO devices that support DMA operations.

The PCI core isn't involved much with DMA, so I don't know this off
the top of my head.

> > I think the biggest problem is that you need to set up the BRIDGE
> > *before* calling pcibios_scan_bus().  That way the windows
> > ("hose->mem_resource", "hose->io_resource", etc.) will be correct
> > when the PCI core enumerates the devices.  Note that you can have
> > as many windows in the "&resources" list as you need -- the
> > current code there only has one memory and one I/O window, but you
> > can add more.
> 
> Agreed, however, I think this is actually being done to some extent
> already.  We're configuring BRIDGE-specific properties and writing
> some values to BRIDGE registers and the per-slot registers in
> bridge_probe() in the generic pci-bridge.c driver, then, at the very
> end, calling register_pci_controller(), which I believe is what
> kicks off the PCI bus scan.
> 
> The IP30 BRIDGE glue code adds a new function hook called
> "pre_enable" where it does, in ip30-bridge.c, some additional PCI
> device tweaking, but this code will run after the PCI bus scan has
> happened.  My guess is that I want to reverse this and have the IP30
> glue code twiddle the PCI devices on a given BRIDGE before the PCI
> bus scan, right?  

Theoretically you should not need to do any PCI device tweaking: PCI
devices are basically arch-independent and shouldn't need
arch-specific tweaks.  All the arch stuff should be encapsulated in
the host bridge driver, i.e., the code that sets up the window list
and calls pci_scan_root_bus().  In the MIPS case, this code is kind of
spread out and doesn't really look like a single "driver".

The ideal thing would be if you can set up the bridge to always use
large windows.  Then the PCI core will enumerate the devices and set
up their resources automatically.

I assume the small/medium windows exist because there's not enough
address space to always use the large windows.  I don't have any good
suggestions for that -- we don't have support for adjusting the window
sizes based on what devices we find.

> That code uses pci_read_config_dword() and
> pci_write_config_dword() -- are these functions safe to use if we
> haven't already done a PCI bus scan?

You can't use pci_read_config_dword() before we scan the bus because
it requires a "struct pci_dev *", and those are created during the bus
scan.

Bjorn

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Joshua Kinard]

On 02/27/2017 11:36, Bjorn Helgaas wrote:
> On Sat, Feb 25, 2017 at 04:34:12AM -0500, Joshua Kinard wrote:
>> On 02/24/2017 13:38, Bjorn Helgaas wrote:
>>> On Fri, Feb 24, 2017 at 03:50:26AM -0500, Joshua Kinard wrote:
> 
>>> The host bridge code, i.e., the pcibios_scanbus() path, tells the
>>> core how big the window is.  In this case, "hose->mem_resource"
>>> contains the CPU physical address range (and size), and
>>> "host->mem_offset" contains the offset between the CPU physical
>>> address and the PCI bus address.  So if "hose->mem_resource" is
>>> only 1GB, that's all the PCI core will use.
>>
>> So basically, this bit of code from the proposed pci-bridge.c needs
>> additional work:
>>
>> 	bc->mem.name = "Bridge MEM";
>> 	bc->mem.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_MEM);
>> 	bc->mem.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO - 1);
>> 	bc->mem.flags = IORESOURCE_MEM;
>>
>> 	bc->io.name = "Bridge IO";
>> 	bc->io.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO);
>> 	bc->io.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_END - 1);
>> 	bc->io.flags = IORESOURCE_IO;
>>
>> The values I've been using for PCIBR_OFFSET_MEM and PCIBR_OFFSET_IO
>> are obviously wrong.  For IP30, they ultimately direct "BRIDGE MEM"
>> to the first five device slots via small windows, and then "Bridge
>> IO" to the last two slots.  This is one of those "luck" moments that
>> happened to work on BaseIO, but is probably why devices in the
>> secondary PCI shoebox are hit-or-miss.
>>
>> Per your comment further down that I can tell the PCI core about
>> //multiple// windows, probably what I want to do is hardwire access
>> to the small window spaces in the main BRIDGE driver, with
>> machine-specific offsets passed via platform_data.  Small windows,
>> for both IP30 and IP27, are always guaranteed to be available
>> without any additional magic.  This window range can be used to do
>> the initial slot probes to query devices for their desired BAR
>> values.
>>
>> Then, in IP30-specific or IP27-specific code, we add in additional
>> memory windows as needed.  IP30 would simply pass along the
>> hardwired address ranges for BRIDGE MEM or BRIDGE IO via both medium
>> and big window addresses.
>>
>> IP27 is trickier -- we'll need some logic that attempts to use a
>> small window (16MB) mapping first, but if that is too small, we'll
>> have to setup an entry in the HUB's IOTTE buffer that maps a
>> specific block of physical memory to Crosstalk address space.  
> 
> pcibios_scanbus() builds the &resources list of host bridge windows,
> then calls pci_scan_root_bus().  If you're proposing to change the
> list of host bridge windows *after* calling pci_scan_root_bus(), that
> sounds a little problematic because we do not have a PCI core
> interface to do that.

I believe I can determine the size of the windows on BRIDGE before hitting the
generic PCI code.  The bridge_probe() function calls register_pci_controller()
at the very bottom, even in the current mainline file (pci-ip27.c), after it's
twiddled a number of BRIDGE register bits.  I'll first have to try and
understand the 'pre_enable' code from the IP30 patch that changes the window
mappings in ip30-bridge.c, then find a way to move that code to execute earlier
before any of the generic PCI code, since it contains some of the logic to size
out the window mappings.

I'll see about getting it to work on IP30 (Octane) first, because all three
window spaces are available without any special magic.  That way, I'll know
what can be done in the generic BRIDGE driver and what needs to happen in
platform-specific files.  That'll make figuring out the IP27 logic a lot
easier....I hope.


>> This is what IP27 calls "big windows", of which there are seven
>> entries maximum, on 512MB boundaries, per HUB chip.  However, only
>> six entries are available because one IOTTE is used to handle a
>> known hardware bug on early HUB revisions, leaving only six left.
>>
>> As such, the additional window information will go into the
>> machine-specific BRIDGE glue drivers (ip30-bridge.c or
>> ip27-bridge.c).
>>
>> Sound sane?
> 
> Do you need to allocate the big windows based on what PCI devices you
> find?  If so, that is going to be a hard problem.

I believe so.  The IOTTE stuff in the HUB ASIC on IP27 appears to behave like
an extremely simplified TLB, and once you work out how much space a given PCI
device needs, then you can calculate out the matching Crosstalk address, and
you write that to an unused IOTTE slot.  It doesn't have to be accurate -- each
IOTTE entry needs to be aligned on a 512MB boundary, so once you've set up a
mapping, if another PCI device (or BAR?) needs a Crosstalk address within an
already-mapped boundary, you just point it to the existing IOTTE entry that
matches.

That's my understanding anyways after looking at really old IA64 code in 2.5.70
that used virtually the same hardware when SGI was bringing up the Altix
platform.  Specifically, hub_piomap_alloc() on Line 117 in
Linux-2.5.70/arch/ia64/sn/io/io.c:
https://git.linux-mips.org/cgit/ralf/linux.git/tree/arch/ia64/sn/io/io.c?h=linux-2.5.70

So I just need to make sure all of this happens during the initial
bridge_probe() function, so that everything is ready before calling
register_pci_controller().  We'll find out!


>>> PCIe functions have 4K of config space each, so a 0x1000 offset
>>> makes sense.  Whether a platform can access all of it is a
>>> separate question.
>>
>> No need to worry about PCIe on this platform.  You're talking about
>> 15+-year old hardware that was built to last.  I believe SGI only
>> implemented up to PCI 2.1 in the BRIDGE and XBRIDGE ASICs.  It's
>> possible the PIC ASIC might do 2.2 or 3.0.  But I don't have to
>> worry about PIC for a good while, as that's only found in the
>> IP35-class of hardware (Origin 300/350/3000, Fuel, Tezro, Onyx4,
>> etc).  We need fully-working IP27 and IP30 first, because much of
>> the working logic can then be re-purposed for bringing up IP35
>> hardware.  Especially since IA64 did a lot of the work already for
>> the Altix systems, which is just IP35 with an IA64 CPU, but I
>> digress...
> 
> OK.  If you only plug in PCI devices, there should be no problem.  If
> you did plug in PCIe devices (via a PCI-to-PCIe bridge, for example),
> the core should still enumerate them correctly and they should be
> functional, though some PCIe-only features like AER won't work.

There might be physical limitations on using any kind of PCIe bridge device.
Getting a PCI device into an SGI Octane (IP30) or Origin 2000/Onyx2 (IP27) that
isn't either the BaseIO on IP30 or IO6 on IP27 requires an XIO Shoehorn or PCI
Shoebox.  There's a little bit of leeway on how wide of a PCI card you can
stuff into either one, but it's not much.  I'll have to measure a shoehorn if I
can find one of my spares in a bin someplace.  The Origin 200 tower machine
might let you get away with it...but that's a corner case in my book and not
really worth worrying about it.  It's hard to find one of those completely
intact these days, let alone with a working power supply that doesn't double as
a fireworks display.


>>> The type 1 config accessors (pci_conf1_read_config(),
>>> pci_conf1_write_config()) are for devices below a PCI-to-PCI
>>> bridge.  It looks like there's a single 4K memory-mapped window,
>>> and you point it at a specific bus & device with
>>> bridge->b_pci_cfg.
>>
>> Yeah, PCI-to-PCI bridges are a known-broken item right now.  Since
>> there appears to only be a single Type 1 config space, if I have a
>> PCI Shoebox with three PCI-X slots, and I stick three USB 2.0 boards
>> in them, I have to pick which one can handle PCI-to-PCI bridges,
>> like USB hubs, and program that into this Type 1 register, right?
> 
> I don't think USB hubs are relevant here because they are completely
> in the USB domain.  The USB 2.0 board is a USB host controller with a
> PCI interface on it.  The USB hub is on the USB side and is invisible
> to the PCI core.
> 
> The way I read the BSD code, it looks like you should be able to use
> the Type 1 config interface to generate accesses to any PCI bus, as
> long as you serialize them, e.g., with a lock around the use of
> b_type1_cfg.

Huh, well, I've never actually tried a USB Hub device on my Octane.  If I can
fix PCI up properly and get a 2.0 card to actually work, I'll try it out before
looking too deeply into the Type 1 stuff.


>>> The Linux type 0 accessors look like they support 4K config space
>>> per function, while the type 1 accessors put the function number
>>> in bits 8-10, so it looks like they only support 256 bytes per
>>> function.
>>>
>>> The OpenBSD accessors (xbridge_conf_read() and
>>> xbridge_conf_write()) look like they only support 256 bytes per
>>> function regardless of whether it's type 0 or type 1.
>>
>> I'll have to pass this question to the OpenBSD dev who maintains the
>> xbridge driver.  He understands this hardware far better than I do,
>> and there might be a reason why they do it that way.
> 
> If you only support PCI devices, 256 bytes of config space is enough.
> The 4K config space is only supported for PCI-X Mode 2 and PCIe
> devices.  Even those PCI-X Mode 2 and PCIe devices should be
> functional with only 256 bytes.

It's unknown why SGI left that much space between the registers on the BRIDGE.
Could've been future-proofing, or maybe there really is a PCI or PCI-X device
out there that currently only works under IRIX that needed it all.  If I can
get things to work better than they are now, I've got a bit of a collection of
PCI devices to test things out with.  Might expose any wrong assumptions.


>>> Supporting 4K of space for type 0 seems like a potential Linux
>>> problem.  Also, pci_conf1_write_config() uses b_type0_cfg_dev in
>>> one place where it looks like it should be using b_type1_cfg.
> 
> I didn't word this well.  I was trying to point out things that look
> like bugs in pci_conf1_write_config() (the use of b_type0_cfg_dev) and
> maybe pci_conf0_read_config() (the fact that it allows 4K config space
> but the hardware probably only supports 256 bytes).

If you're referring to those uses in ops-bridge.c, that file probably needs
some additional TLC.  The patchset I've got fixed it up a little, but looking
at it made my head hurt (and not just because of the IOC3 voodoo in it), so I
kinda passed on spending too much time on it.  I'll probably have to double
back on it in the future after fixing the other areas, so I'll keep this in mind.


>>> If the hardware only supports 256 bytes of config space on
>>> non-root bus devices, that's not a disaster.  We should still be
>>> able to enumerate them and use all the conventional PCI features
>>> and even basic PCIe features.  But the extended config space
>>> (offsets 0x100-0xfff) would be inaccessible, and we wouldn't see
>>> any PCIe extended capabilities (AER, VC, SR-IOV, etc., see
>>> PCI_EXT_CAP_ID_ERR and subsequent definitions) because they live
>>> in that space.
>>
>> I do not think that the BRIDGE supports AER, VC, or SR-IOV at all,
>> primarily due to age of the hardware.  Even if a PCI device that
>> does support those is plugged into a BRIDGE, I am not 100% certain
>> those features would even be usable if the BRIDGE doesn't know about
>> them.  Or is this a case of where the BRIDGE wouldn't care and once
>> it programs the BARs correctly, these features would be available to
>> the Linux drivers?
> 
> These are mostly PCIe features and I don't think you should worry
> about them, at least for now.

Good to know then.  Hopefully never :)


>> I remember from the OpenBSD xbridge driver, it states that it
>> currently uses the BAR mappings setup by the ARCS firmware, but the
>> plan is to eventually move away from those mappings and calculate
>> its own.  I didn't quite get that, but after reading that site, now
>> I do.  ARCS is writing ~0 to the BARs to query the device to figure
>> out how much memory each BAR wants, then programs in some ranges for
>> us.
>>
>> By tallying up the total requested size of all BAR mappings on a
>> given device, we can then make a determination on whether we can use
>> a small, medium, or large window and then program the BAR with the
>> correct address.  Right?
> 
> This is the problem I alluded to above: Linux does not currently
> support anything like this.  We assume the Linux host bridge driver
> knows the window sizes at the beginning of time (it may learn them
> from firmware or it may have hard-coded knowledge of the address map).
> 
> Linux does enumerate the devices and tally up all the BAR sizes (see
> pci_bus_assign_resources()), but it does not have a way to change the
> host bridge windows based on how much BAR space we need.
> 
> The common thing is to use whatever host bridge windows and device BAR
> values were set up by firmware.  If those are all sensible, Linux
> won't change anything.

Well, as I stated earlier above, it looks like the bridge_probe() function in
the new driver will have to do some probing of its own.  One of the patches I
have in a different series does a minimal probe to read each device slot on
BRIDGE to get the vendor ID and device ID to see if the slot is actually
populated.  This is because not all slots are "populated"; e.g., Octane wires
one of the PCI interrupt pins on slot 6 on BaseIO to its power button, for example.

So the logic used to read the vendor/dev IDs can probably be extended to poke
the BARs via the ~0 trick OR read out the pre-defined mappings from ARCS
firmware, then use that info to size out the BRIDGE windows into Crosstalk
space and then set up our IORESOURCE_MEM and IORESOURCE_IO structs with the
right information to pass into register_pci_controller().

I dunno, I'll probably try reading the ARCS mappings idea first, as that seems
easier.  I won't have a lot of time to play with things in March, so getting
something to work, even if it is sub-optimal, is better than nothing working at
all.


>>> The host bridge windows (the "root bus resource" lines in dmesg)
>>> are only for PIO, i.e., a driver running on the CPU reading or
>>> writing registers on the PCI device (it could be actual registers,
>>> or a frame buffer, etc., but it resides on the device and its PCI
>>> bus address is determined by a BAR).  The driver uses ioremap(),
>>> pci_iomap(), pcim_iomap_regions(), etc. to map these into the
>>> kernel virtual space.
>>
>> So for the MIPS case, knowing that ioremap() and friends is not
>> guaranteed to return a workable virtual address, I need to be
>> careful of what addresses I program into each BAR.  E.g., given that
>> on IP30, if a physical address starts with 0x9000000f800xxxxx, it is
>> using medium windows to talk to a device on the BaseIO BRIDGE (Xtalk
>> widget 0xf).  As such, knowing MIPS' ioremap() returns, for the
>> R10000 CPU case, the requested address OR'ed with IO_BASE
>> (0x9000000000000000), I want to tell the PCI core to use
>> 0x0000000f800xxxxx so that ioremap() will return back
>> 0x9000000f800xxxxx?
> 
> If ioremap() doesn't return a virtual address, or at least something
> that can be used like a virtual address, I think that's fundamentally
> broken.  All drivers assume they should ioremap() a BAR resource,
> i.e., the CPU physical address that maps to a PCI BAR value, and use
> the result as a virtual address.

This is an issue that the core Linux/MIPS people will have to work out.  Ralf
probably knows the precise history on why ioremap*() and related functions
aren't guaranteed to be usable as virtual addresses.  I suspect it's tied to
the other issues with IP27, as that was one of the first MIPS platforms that
Linux ran on in the early 2000's.


>> And since I can apparently specify multiple window ranges for memory
>> space and I/O space, I probably want to, as stated earlier, specify
>> all three of IP30's known window ranges for each BRIDGE so that the
>> Linux PCI core can walk each resource struct and find a matching
>> window?
> 
> If all three windows are disjoint, you can specify them all.  If a
> range in the small window is aliased and can also be reached via a
> medium or large window, you should not specify both.

Each window, at least on Octane, lives at distinct addresses in Crosstalk
space.  A specific address within each window can point to the same device on a
subordinate PCI bus, but I think that will be transparent to the Linux PCI
core.  My thinking is, if I setup IORESOURCE_MEM structs for each window for
each BRIDGE widget, then the PCI code will check each struct to see which one
contains the address range that the PCI device's BAR wants.

E.g., Using widget 0xd as an example for readability, if I know my three
windows in Crosstalk space are:
    Small:  0x0000_1d00_0000 - 0x0000_1dff_ffff
    Medium: 0x000e_8000_0000 - 0x000e_ffff_ffff
    Large:  0x00d0_0000_0000 - 0x00df_ffff_ffff

And for that specific BRIDGE, if I pass those three ranges as IORESOURCE_MEM
structs, then shouldn't the PCI core, if it is told that a specific devices BAR
wants range 1d200000 to 1d203fff, select the small window mapping?  E,g., is it
going to try to find the smallest possible window to fit first?  Or will it
instead try to match using the large window?

I guess I'll find out when I can get some time to actually test the idea out...


[snip]

>>> I think the biggest problem is that you need to set up the BRIDGE
>>> *before* calling pcibios_scan_bus().  That way the windows
>>> ("hose->mem_resource", "hose->io_resource", etc.) will be correct
>>> when the PCI core enumerates the devices.  Note that you can have
>>> as many windows in the "&resources" list as you need -- the
>>> current code there only has one memory and one I/O window, but you
>>> can add more.
>>
>> Agreed, however, I think this is actually being done to some extent
>> already.  We're configuring BRIDGE-specific properties and writing
>> some values to BRIDGE registers and the per-slot registers in
>> bridge_probe() in the generic pci-bridge.c driver, then, at the very
>> end, calling register_pci_controller(), which I believe is what
>> kicks off the PCI bus scan.
>>
>> The IP30 BRIDGE glue code adds a new function hook called
>> "pre_enable" where it does, in ip30-bridge.c, some additional PCI
>> device tweaking, but this code will run after the PCI bus scan has
>> happened.  My guess is that I want to reverse this and have the IP30
>> glue code twiddle the PCI devices on a given BRIDGE before the PCI
>> bus scan, right?  
> 
> Theoretically you should not need to do any PCI device tweaking: PCI
> devices are basically arch-independent and shouldn't need
> arch-specific tweaks.  All the arch stuff should be encapsulated in
> the host bridge driver, i.e., the code that sets up the window list
> and calls pci_scan_root_bus().  In the MIPS case, this code is kind of
> spread out and doesn't really look like a single "driver".

In an ideal world, this would be correct.  However, owning SGI equipment must
have an effect on the curvature of local spacetime, and things seem to operate
a bit differently.  The IOC3 device is a perfect example of this phenomenon.


> The ideal thing would be if you can set up the bridge to always use
> large windows.  Then the PCI core will enumerate the devices and set
> up their resources automatically.

I'll keep this in mind when I can start testing.  I'll initially try feeding
three IORESOURCE_MEM structs to the PCI core to describe the three windows in
order from smallest to largest and see what it does.  Then try reversing the
order and see if it still finds the right window or not.  And whether that
actually leads to correct probing of the PCI devices or not.


> I assume the small/medium windows exist because there's not enough
> address space to always use the large windows.  I don't have any good
> suggestions for that -- we don't have support for adjusting the window
> sizes based on what devices we find.

IMHO, "Use a bigger hammer" seems most appropriate when it comes to this hardware.


>> That code uses pci_read_config_dword() and
>> pci_write_config_dword() -- are these functions safe to use if we
>> haven't already done a PCI bus scan?
> 
> You can't use pci_read_config_dword() before we scan the bus because
> it requires a "struct pci_dev *", and those are created during the bus
> scan.

Okay, that's good to know then.  There's other accessors that are MIPS-specific
that I think I can use.  I don't think I specifically need a 'struct pci_dev *'
at that point anyways.


-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case

[Bjorn Helgaas]

On Mon, Feb 27, 2017 at 07:25:58PM -0500, Joshua Kinard wrote:
> On 02/27/2017 11:36, Bjorn Helgaas wrote:
> > On Sat, Feb 25, 2017 at 04:34:12AM -0500, Joshua Kinard wrote:
> >> On 02/24/2017 13:38, Bjorn Helgaas wrote:
> >>> On Fri, Feb 24, 2017 at 03:50:26AM -0500, Joshua Kinard wrote:

> So the logic used to read the vendor/dev IDs can probably be
> extended to poke the BARs via the ~0 trick OR read out the
> pre-defined mappings from ARCS firmware, then use that info to size
> out the BRIDGE windows into Crosstalk space and then set up our
> IORESOURCE_MEM and IORESOURCE_IO structs with the right information
> to pass into register_pci_controller().

As you probably know, you can determine the size of a BAR regardless of
whether it has been assigned.  So it doesn't matter whether ARCS has
already assigned anything.  See __pci_read_base().

> I dunno, I'll probably try reading the ARCS mappings idea first, as
> that seems easier.  I won't have a lot of time to play with things
> in March, so getting something to work, even if it is sub-optimal,
> is better than nothing working at all.

The simplest possible thing is to enable some pre-defined set of
windows and accept that this may lead to configuration restrictions,
e.g., a device with large BARs may work only in certain slots.

> Each window, at least on Octane, lives at distinct addresses in
> Crosstalk space.  A specific address within each window can point to
> the same device on a subordinate PCI bus, but I think that will be
> transparent to the Linux PCI core.  My thinking is, if I setup
> IORESOURCE_MEM structs for each window for each BRIDGE widget, then
> the PCI code will check each struct to see which one contains the
> address range that the PCI device's BAR wants.
> 
> E.g., Using widget 0xd as an example for readability, if I know my
> three windows in Crosstalk space are:
>     Small:  0x0000_1d00_0000 - 0x0000_1dff_ffff
>     Medium: 0x000e_8000_0000 - 0x000e_ffff_ffff
>     Large:  0x00d0_0000_0000 - 0x00df_ffff_ffff
> 
> And for that specific BRIDGE, if I pass those three ranges as
> IORESOURCE_MEM structs, then shouldn't the PCI core, if it is told
> that a specific devices BAR wants range 1d200000 to 1d203fff, select
> the small window mapping?  E,g., is it going to try to find the
> smallest possible window to fit first?  Or will it instead try to
> match using the large window?

No, when matching a BAR value to a host bridge window, the PCI core
will definitely not look for the smallest (or largest) window that
contains the BAR.  

The core reads the BAR value (a bus address), then searches the host
bridge windows for one that maps to a region that contains the bus
address.  This happens in pcibios_bus_to_resource(), which is called
by __pci_read_base().

The search order is undefined because the core assumes the windows do
not overlap in PCI bus address space.  It can't deal with two windows
that map to the same PCI space because when we allocate space when
assigning a BAR, we search for available CPU address space, not for
available PCI bus address space.

If window A and window B both map to PCI bus address X, we may assign
space from window A to PCI BAR 1 and space from window B to PCI BAR 2.
Then we have two BARs at the same PCI bus address, which will cause
conflicts.

I don't think we currently check in the PCI core for host bridge
windows that map to overlapping PCI bus space.  Maybe we should, and
just reject anything that overlaps.

Bjorn