* [PATCH 00/12] MIPS: BRIDGE Updates
@ 2017-02-07 6:13 Joshua Kinard
2017-02-07 6:13 ` [PATCH 01/12] MIPS: BRIDGE: Rename pci-ip27.c to pci-bridge.c Joshua Kinard
` (11 more replies)
0 siblings, 12 replies; 28+ messages in thread
From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw)
To: Ralf Baechle; +Cc: Linux/MIPS
From: Joshua Kinard <kumba@gentoo.org>
This series performs a number of clean-ups on the code for the BRIDGE
ASIC found in several SGI platforms. It must be applied after the
Xtalk clean-up patch series is applied.
Notable changes include:
- Genericize the IP27 BRIDGE driver so future platforms can use it
- Prepare the generic BRIDGE driver to become a platform_driver
- Clean-up/replace IRIX-era structures and macros in bridge.h
- Fix the recent PCI_PROBE_ONLY change to work for IP27 again.
Joshua Kinard (12):
MIPS: BRIDGE: Rename pci-ip27.c to pci-bridge.c
MIPS: IP27: Add pcibr.h header for IP27
MIPS: PCI: Minor clean-ups to pci-legacy.c
MIPS: PCI: Add BRIDGE 'pre_enable' hook
MIPS: BRIDGE: Clean-up bridge.h header file
MIPS: BRIDGE: Overhaul bridge.h header file
MIPS: BRIDGE: Add XBRIDGE revs and SWAP bit
MIPS: BRIDGE: Update ops-bridge.c
MIPS: BRIDGE: Use !pci_is_root_bus(bus) to check for bus->number > 0
MIPS: BRIDGE: Overhaul pci-bridge.c driver
MIPS: IP27: Update IRQ code to work with the new BRIDGE code
MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case
arch/mips/include/asm/mach-ip27/pcibr.h | 50 +
arch/mips/include/asm/pci.h | 3 +
arch/mips/include/asm/pci/bridge.h | 1353 ++++++++++++---------
arch/mips/pci/Makefile | 2 +-
arch/mips/pci/ops-bridge.c | 129 +-
arch/mips/pci/pci-bridge.c | 321 +++++
arch/mips/pci/pci-ip27.c | 230 ----
arch/mips/pci/pci-legacy.c | 74 +-
arch/mips/sgi-ip27/ip27-irq-pci.c | 72 +-
arch/mips/sgi-ip27/ip27-irq.c | 5 +
10 files changed, 1313 insertions(+), 926 deletions(-)
create mode 100644 arch/mips/include/asm/mach-ip27/pcibr.h
create mode 100644 arch/mips/pci/pci-bridge.c
delete mode 100644 arch/mips/pci/pci-ip27.c
--
2.11.1
^ permalink raw reply [flat|nested] 28+ messages in thread* [PATCH 01/12] MIPS: BRIDGE: Rename pci-ip27.c to pci-bridge.c 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 02/12] MIPS: IP27: Add pcibr.h header for IP27 Joshua Kinard ` (10 subsequent siblings) 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS From: Joshua Kinard <kumba@gentoo.org> Rename arch/mips/pci/pci-ip27.c to arch/mips/pci/pci-bridge.c so that it can become a generic driver for all BRIDGE/XBRIDGE-based systems and update the Makefile for this change. Signed-off-by: Joshua Kinard <kumba@gentoo.org> --- arch/mips/pci/Makefile | 2 +- arch/mips/pci/{pci-ip27.c => pci-bridge.c} | 0 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile index 4b821481dd44..7ff66cce2cf7 100644 --- a/arch/mips/pci/Makefile +++ b/arch/mips/pci/Makefile @@ -37,7 +37,7 @@ obj-$(CONFIG_MIPS_MALTA) += fixup-malta.o pci-malta.o obj-$(CONFIG_PMC_MSP7120_GW) += fixup-pmcmsp.o ops-pmcmsp.o obj-$(CONFIG_PMC_MSP7120_EVAL) += fixup-pmcmsp.o ops-pmcmsp.o obj-$(CONFIG_PMC_MSP7120_FPGA) += fixup-pmcmsp.o ops-pmcmsp.o -obj-$(CONFIG_SGI_IP27) += ops-bridge.o pci-ip27.o +obj-$(CONFIG_SGI_IP27) += ops-bridge.o pci-bridge.o obj-$(CONFIG_SGI_IP32) += fixup-ip32.o ops-mace.o pci-ip32.o obj-$(CONFIG_SIBYTE_SB1250) += fixup-sb1250.o pci-sb1250.o obj-$(CONFIG_SIBYTE_BCM112X) += fixup-sb1250.o pci-sb1250.o diff --git a/arch/mips/pci/pci-ip27.c b/arch/mips/pci/pci-bridge.c similarity index 100% rename from arch/mips/pci/pci-ip27.c rename to arch/mips/pci/pci-bridge.c -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 02/12] MIPS: IP27: Add pcibr.h header for IP27 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard 2017-02-07 6:13 ` [PATCH 01/12] MIPS: BRIDGE: Rename pci-ip27.c to pci-bridge.c Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 03/12] MIPS: PCI: Minor clean-ups to pci-legacy.c Joshua Kinard ` (9 subsequent siblings) 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS From: Joshua Kinard <kumba@gentoo.org> Add a new header 'pcibr.h' to arch/mips/include/asm/mach-ip27 which contains IP27-specific definitions for the BRIDGE ASIC used to connect PCI devices to the Crosstalk bus. Signed-off-by: Joshua Kinard <kumba@gentoo.org> --- arch/mips/include/asm/mach-ip27/pcibr.h | 50 +++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/arch/mips/include/asm/mach-ip27/pcibr.h b/arch/mips/include/asm/mach-ip27/pcibr.h new file mode 100644 index 000000000000..6664843c30f9 --- /dev/null +++ b/arch/mips/include/asm/mach-ip27/pcibr.h @@ -0,0 +1,50 @@ +/* + * Definitions for PCI bridges in IP27. Derived from pcibr.h in the + * IP30 port. + * + * Copyright (C) 2004-2007 Stanislaw Skowronek <skylark@unaligned.org> + * Copyright (C) 2015-2016 Joshua Kinard <kumba@gentoo.org> + * + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + */ + +#ifndef __ASM_MACH_IP27_PCIBR_H +#define __ASM_MACH_IP27_PCIBR_H + +#include <asm/pci/bridge.h> +#include <asm/mach-ip27/irq.h> + +/* Xtalk */ +#define PCIBR_OFFSET_MEM 0x200000 +#define PCIBR_OFFSET_IO 0xa00000 +#define PCIBR_OFFSET_END 0xc00000 + +/* + * This is how XIO sees HUB's PI_INT_PEND_MOD register. + */ +#define PCIBR_XIO_SEES_HUB 0x01800090 + +/* + * Max # of PCI buses we can handle; ie, max #PCI bridges. + */ +#define PCIBR_MAX_NUM_PCIBUS 40 + +/* + * Max # of PCI devices (like scsi controllers) we handle on a bus. + */ +#define PCIBR_MAX_DEV_PCIBUS 8 + +/* + * Used by ip27-bridge.c and ip27-irq.c. + */ +#define PCIBR_MAX_BUS_X_DEV (PCIBR_MAX_NUM_PCIBUS * PCIBR_MAX_DEV_PCIBUS) +extern struct bridge_controller *irq_to_bridge[PCIBR_MAX_BUS_X_DEV]; +extern u32 irq_to_slot[PCIBR_MAX_BUS_X_DEV]; + +/* XXX: Temporary until IP27 "mega update". */ +extern int request_bridge_irq(struct bridge_controller *bc); + +#endif /* __ASM_MACH_IP27_PCIBR_H */ + -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 03/12] MIPS: PCI: Minor clean-ups to pci-legacy.c 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard 2017-02-07 6:13 ` [PATCH 01/12] MIPS: BRIDGE: Rename pci-ip27.c to pci-bridge.c Joshua Kinard 2017-02-07 6:13 ` [PATCH 02/12] MIPS: IP27: Add pcibr.h header for IP27 Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 04/12] MIPS: PCI: Add BRIDGE 'pre_enable' hook Joshua Kinard ` (8 subsequent siblings) 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS From: Joshua Kinard <kumba@gentoo.org> Apply several minor clean-ups to arch/mips/pci/pci-legacy.c: - Collapse one-line comments to use a single /* ... */ - Move 2nd arg to pci_add_resource_offset up a line - Put a pr_info message all on one line (checkpatch) - Replace several printk's with pr_info/pr_warn/pr_err - Add whitespace around operators in for loop and conditional Signed-off-by: Joshua Kinard <kumba@gentoo.org> --- arch/mips/pci/pci-legacy.c | 48 ++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 27 deletions(-) diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c index 014649be158d..98f36ed8f7da 100644 --- a/arch/mips/pci/pci-legacy.c +++ b/arch/mips/pci/pci-legacy.c @@ -58,9 +58,7 @@ pcibios_align_resource(void *data, const struct resource *res, if (start < PCIBIOS_MIN_IO + hose->io_resource->start) start = PCIBIOS_MIN_IO + hose->io_resource->start; - /* - * Put everything into 0x00-0xff region modulo 0x400 - */ + /* Put everything into 0x00-0xff region modulo 0x400 */ if (start & 0x300) start = (start + 0x3ff) & ~0x3ff; } else if (res->flags & IORESOURCE_MEM) { @@ -82,12 +80,12 @@ static void pcibios_scanbus(struct pci_controller *hose) if (hose->get_busno && pci_has_flag(PCI_PROBE_ONLY)) next_busno = (*hose->get_busno)(); - pci_add_resource_offset(&resources, - hose->mem_resource, hose->mem_offset); - pci_add_resource_offset(&resources, - hose->io_resource, hose->io_offset); - pci_add_resource_offset(&resources, - hose->busn_resource, hose->busn_offset); + pci_add_resource_offset(&resources, hose->mem_resource, + hose->mem_offset); + pci_add_resource_offset(&resources, hose->io_resource, + hose->io_offset); + pci_add_resource_offset(&resources, hose->busn_resource, + hose->busn_offset); bus = pci_scan_root_bus(NULL, next_busno, hose->pci_ops, hose, &resources); hose->bus = bus; @@ -139,18 +137,16 @@ void pci_load_of_ranges(struct pci_controller *hose, struct device_node *node) switch (range.flags & IORESOURCE_TYPE_BITS) { case IORESOURCE_IO: - pr_info(" IO 0x%016llx..0x%016llx\n", - range.cpu_addr, - range.cpu_addr + range.size - 1); + pr_info(" IO 0x%016llx..0x%016llx\n", range.cpu_addr, + (range.cpu_addr + range.size - 1)); hose->io_map_base = (unsigned long)ioremap(range.cpu_addr, range.size); res = hose->io_resource; break; case IORESOURCE_MEM: - pr_info(" MEM 0x%016llx..0x%016llx\n", - range.cpu_addr, - range.cpu_addr + range.size - 1); + pr_info(" MEM 0x%016llx..0x%016llx\n", range.cpu_addr, + (range.cpu_addr + range.size - 1)); res = hose->mem_resource; break; } @@ -195,10 +191,8 @@ void register_pci_controller(struct pci_controller *hose) /* * Do not panic here but later - this might happen before console init. */ - if (!hose->io_map_base) { - printk(KERN_WARNING - "registering PCI controller with io_map_base unset\n"); - } + if (!hose->io_map_base) + pr_warn("registering PCI controller with io_map_base unset\n"); /* * Scan the bus if it is register after the PCI subsystem @@ -213,8 +207,7 @@ void register_pci_controller(struct pci_controller *hose) return; out: - printk(KERN_WARNING - "Skipping PCI bus scan due to resource conflict\n"); + pr_warn("Skipping PCI bus scan due to resource conflict\n"); } static int __init pcibios_init(void) @@ -242,9 +235,9 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask) pci_read_config_word(dev, PCI_COMMAND, &cmd); old_cmd = cmd; - for (idx=0; idx < PCI_NUM_RESOURCES; idx++) { + for (idx = 0; idx < PCI_NUM_RESOURCES; idx++) { /* Only set up the requested stuff */ - if (!(mask & (1<<idx))) + if (!(mask & (1 << idx))) continue; r = &dev->resource[idx]; @@ -254,8 +247,7 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask) (!(r->flags & IORESOURCE_ROM_ENABLE))) continue; if (!r->start && r->end) { - printk(KERN_ERR "PCI: Device %s not available " - "because of resource collisions\n", + pr_err("PCI: Device %s not available because of resource collisions\n", pci_name(dev)); return -EINVAL; } @@ -265,10 +257,11 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask) cmd |= PCI_COMMAND_MEMORY; } if (cmd != old_cmd) { - printk("PCI: Enabling device %s (%04x -> %04x)\n", - pci_name(dev), old_cmd, cmd); + pr_info("PCI: Enabling device %s (%04x -> %04x)\n", + pci_name(dev), old_cmd, cmd); pci_write_config_word(dev, PCI_COMMAND, cmd); } + return 0; } @@ -298,5 +291,6 @@ char *__init pcibios_setup(char *str) { if (pcibios_plat_setup) return pcibios_plat_setup(str); + return str; } -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 04/12] MIPS: PCI: Add BRIDGE 'pre_enable' hook 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard ` (2 preceding siblings ...) 2017-02-07 6:13 ` [PATCH 03/12] MIPS: PCI: Minor clean-ups to pci-legacy.c Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 05/12] MIPS: BRIDGE: Clean-up bridge.h header file Joshua Kinard ` (7 subsequent siblings) 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS, Stanislaw Skowronek From: Joshua Kinard <kumba@gentoo.org> Add a hook for a 'pre_enable' function in struct pci_controller in arch/mips/include/asm/pci.h and arch/mips/pci/pci-legacy.c. This is used to do any needed housekeeping on the BRIDGE/XBRIDGE ASICs prior to PCI detection, such setting up DevIO windows or applying any known BRIDGE WARs. This change originates from the earliest days of the IP30 patchset by Stanislaw Skowronek. Signed-off-by: Joshua Kinard <kumba@gentoo.org> Cc: Stanislaw Skowronek <skylark@unaligned.org> --- arch/mips/include/asm/pci.h | 3 +++ arch/mips/pci/pci-legacy.c | 11 +++++++++++ 2 files changed, 14 insertions(+) diff --git a/arch/mips/include/asm/pci.h b/arch/mips/include/asm/pci.h index 30d1129d8624..0f1e381a79e3 100644 --- a/arch/mips/include/asm/pci.h +++ b/arch/mips/include/asm/pci.h @@ -48,6 +48,9 @@ struct pci_controller { unsigned int need_domain_info; #endif + /* BRIDGE/XBRIDGE may need to config things before bringup */ + int (*pre_enable)(struct pci_controller *, struct pci_dev *, int); + /* Optional access methods for reading/writing the bus number of the PCI controller */ int (*get_busno)(void); diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c index 98f36ed8f7da..68268bbb15b8 100644 --- a/arch/mips/pci/pci-legacy.c +++ b/arch/mips/pci/pci-legacy.c @@ -7,6 +7,7 @@ * Copyright (C) 2003, 04, 11 Ralf Baechle (ralf@linux-mips.org) * Copyright (C) 2011 Wind River Systems, * written by Ralf Baechle (ralf@linux-mips.org) + * Copyright (C) 2004-2007 Stanislaw Skowronek <skylark@unaligned.org> */ #include <linux/bug.h> #include <linux/kernel.h> @@ -232,6 +233,7 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask) u16 cmd, old_cmd; int idx; struct resource *r; + struct pci_controller *hose = (struct pci_controller *)dev->sysdata; pci_read_config_word(dev, PCI_COMMAND, &cmd); old_cmd = cmd; @@ -240,6 +242,15 @@ static int pcibios_enable_resources(struct pci_dev *dev, int mask) if (!(mask & (1 << idx))) continue; + /* + * Some devices, like BRIDGE/XBRIDGE, may need to do a little + * housekeeping prior to the generic code setting up the PCI + * resources. + */ + if (hose->pre_enable) + if (hose->pre_enable(hose, dev, idx) < 0) + return -EINVAL; + r = &dev->resource[idx]; if (!(r->flags & (IORESOURCE_IO | IORESOURCE_MEM))) continue; -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 05/12] MIPS: BRIDGE: Clean-up bridge.h header file 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard ` (3 preceding siblings ...) 2017-02-07 6:13 ` [PATCH 04/12] MIPS: PCI: Add BRIDGE 'pre_enable' hook Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 06/12] MIPS: BRIDGE: Overhaul " Joshua Kinard ` (6 subsequent siblings) 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS From: Joshua Kinard <kumba@gentoo.org> Apply a number of clean-ups to arch/mips/include/asm/pci/bridge.h: - Replace (0x1 << x) shifts with BIT(x) or BIT_ULL - Replace (mask << x) shifts with GENMASK or GENMASK_ULL - Clean-up several multi-line macros for readability - Other whitespace and comment fixes as needed Signed-off-by: Joshua Kinard <kumba@gentoo.org> --- arch/mips/include/asm/pci/bridge.h | 520 ++++++++++++++------------- 1 file changed, 280 insertions(+), 240 deletions(-) diff --git a/arch/mips/include/asm/pci/bridge.h b/arch/mips/include/asm/pci/bridge.h index 3206245d1ed6..248a390e89b3 100644 --- a/arch/mips/include/asm/pci/bridge.h +++ b/arch/mips/include/asm/pci/bridge.h @@ -388,79 +388,80 @@ typedef struct bridge_err_cmdword_s { * Bridge register bit field definitions */ -/* Widget part number of bridge */ +/* Widget part numbers. */ #define BRIDGE_WIDGET_PART_NUM 0xc002 #define XBRIDGE_WIDGET_PART_NUM 0xd002 -/* Manufacturer of bridge */ +/* Widget manufacturer numbers. */ #define BRIDGE_WIDGET_MFGR_NUM 0x036 #define XBRIDGE_WIDGET_MFGR_NUM 0x024 -/* Revision numbers for known Bridge revisions */ +/* Widget revision numbers. */ #define BRIDGE_REV_A 0x1 #define BRIDGE_REV_B 0x2 #define BRIDGE_REV_C 0x3 #define BRIDGE_REV_D 0x4 /* Bridge widget status register bits definition */ - -#define BRIDGE_STAT_LLP_REC_CNT (0xFFu << 24) -#define BRIDGE_STAT_LLP_TX_CNT (0xFF << 16) -#define BRIDGE_STAT_FLASH_SELECT (0x1 << 6) -#define BRIDGE_STAT_PCI_GIO_N (0x1 << 5) -#define BRIDGE_STAT_PENDING (0x1F << 0) +#define BRIDGE_STAT_LLP_REC_CNT GENMASK(31, 24) +#define BRIDGE_STAT_LLP_TX_CNT GENMASK(23, 16) +#define BRIDGE_STAT_FLASH_SELECT BIT(6) +#define BRIDGE_STAT_PCI_GIO_N BIT(5) +#define BRIDGE_STAT_PENDING GENMASK(4, 0) /* Bridge widget control register bits definition */ -#define BRIDGE_CTRL_FLASH_WR_EN (0x1ul << 31) -#define BRIDGE_CTRL_EN_CLK50 (0x1 << 30) -#define BRIDGE_CTRL_EN_CLK40 (0x1 << 29) -#define BRIDGE_CTRL_EN_CLK33 (0x1 << 28) +#define BRIDGE_CTRL_FLASH_WR_EN BIT(31) +#define BRIDGE_CTRL_EN_CLK50 BIT(30) +#define BRIDGE_CTRL_EN_CLK40 BIT(29) +#define BRIDGE_CTRL_EN_CLK33 BIT(28) #define BRIDGE_CTRL_RST(n) ((n) << 24) -#define BRIDGE_CTRL_RST_MASK (BRIDGE_CTRL_RST(0xF)) -#define BRIDGE_CTRL_RST_PIN(x) (BRIDGE_CTRL_RST(0x1 << (x))) -#define BRIDGE_CTRL_IO_SWAP (0x1 << 23) -#define BRIDGE_CTRL_MEM_SWAP (0x1 << 22) -#define BRIDGE_CTRL_PAGE_SIZE (0x1 << 21) -#define BRIDGE_CTRL_SS_PAR_BAD (0x1 << 20) -#define BRIDGE_CTRL_SS_PAR_EN (0x1 << 19) +#define BRIDGE_CTRL_RST_MASK GENMASK(27, 24) +#define BRIDGE_CTRL_RST_PIN(x) (BRIDGE_CTRL_RST(1UL << (x))) +#define BRIDGE_CTRL_IO_SWAP BIT(23) +#define BRIDGE_CTRL_MEM_SWAP BIT(22) +#define BRIDGE_CTRL_PAGE_SIZE BIT(21) +#define BRIDGE_CTRL_SS_PAR_BAD BIT(20) +#define BRIDGE_CTRL_SS_PAR_EN BIT(19) #define BRIDGE_CTRL_SSRAM_SIZE(n) ((n) << 17) #define BRIDGE_CTRL_SSRAM_SIZE_MASK (BRIDGE_CTRL_SSRAM_SIZE(0x3)) #define BRIDGE_CTRL_SSRAM_512K (BRIDGE_CTRL_SSRAM_SIZE(0x3)) #define BRIDGE_CTRL_SSRAM_128K (BRIDGE_CTRL_SSRAM_SIZE(0x2)) #define BRIDGE_CTRL_SSRAM_64K (BRIDGE_CTRL_SSRAM_SIZE(0x1)) #define BRIDGE_CTRL_SSRAM_1K (BRIDGE_CTRL_SSRAM_SIZE(0x0)) -#define BRIDGE_CTRL_F_BAD_PKT (0x1 << 16) +#define BRIDGE_CTRL_F_BAD_PKT BIT(16) #define BRIDGE_CTRL_LLP_XBAR_CRD(n) ((n) << 12) #define BRIDGE_CTRL_LLP_XBAR_CRD_MASK (BRIDGE_CTRL_LLP_XBAR_CRD(0xf)) -#define BRIDGE_CTRL_CLR_RLLP_CNT (0x1 << 11) -#define BRIDGE_CTRL_CLR_TLLP_CNT (0x1 << 10) -#define BRIDGE_CTRL_SYS_END (0x1 << 9) +#define BRIDGE_CTRL_CLR_RLLP_CNT BIT(11) +#define BRIDGE_CTRL_CLR_TLLP_CNT BIT(10) +#define BRIDGE_CTRL_SYS_END BIT(9) #define BRIDGE_CTRL_MAX_TRANS(n) ((n) << 4) #define BRIDGE_CTRL_MAX_TRANS_MASK (BRIDGE_CTRL_MAX_TRANS(0x1f)) #define BRIDGE_CTRL_WIDGET_ID(n) ((n) << 0) #define BRIDGE_CTRL_WIDGET_ID_MASK (BRIDGE_CTRL_WIDGET_ID(0xf)) /* Bridge Response buffer Error Upper Register bit fields definition */ -#define BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT (20) -#define BRIDGE_RESP_ERRUPPR_DEVNUM_MASK (0x7 << BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT) -#define BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT (16) -#define BRIDGE_RESP_ERRUPPR_BUFNUM_MASK (0xF << BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT) -#define BRIDGE_RESP_ERRRUPPR_BUFMASK (0xFFFF) - -#define BRIDGE_RESP_ERRUPPR_BUFNUM(x) \ - (((x) & BRIDGE_RESP_ERRUPPR_BUFNUM_MASK) >> \ - BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT) - -#define BRIDGE_RESP_ERRUPPR_DEVICE(x) \ - (((x) & BRIDGE_RESP_ERRUPPR_DEVNUM_MASK) >> \ - BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT) +#define BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT (20) +#define BRIDGE_RESP_ERRUPPR_DEVNUM_MASK \ + (0x7 << BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT) +#define BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT (16) +#define BRIDGE_RESP_ERRUPPR_BUFNUM_MASK \ + (0xf << BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT) +#define BRIDGE_RESP_ERRRUPPR_BUFMASK (0xffff) + +#define BRIDGE_RESP_ERRUPPR_BUFNUM(x) \ + (((x) & BRIDGE_RESP_ERRUPPR_BUFNUM_MASK) >> \ + BRIDGE_RESP_ERRUPPR_BUFNUM_SHFT) + +#define BRIDGE_RESP_ERRUPPR_DEVICE(x) \ + (((x) & BRIDGE_RESP_ERRUPPR_DEVNUM_MASK) >> \ + BRIDGE_RESP_ERRUPPR_DEVNUM_SHFT) /* Bridge direct mapping register bits definition */ #define BRIDGE_DIRMAP_W_ID_SHFT 20 -#define BRIDGE_DIRMAP_W_ID (0xf << BRIDGE_DIRMAP_W_ID_SHFT) -#define BRIDGE_DIRMAP_RMF_64 (0x1 << 18) -#define BRIDGE_DIRMAP_ADD512 (0x1 << 17) -#define BRIDGE_DIRMAP_OFF (0x1ffff << 0) +#define BRIDGE_DIRMAP_W_ID GENMASK(23, BRIDGE_DIRMAP_W_ID_SHFT) +#define BRIDGE_DIRMAP_RMF_64 BIT(18) +#define BRIDGE_DIRMAP_ADD512 BIT(17) +#define BRIDGE_DIRMAP_OFF GENMASK(16, 0) #define BRIDGE_DIRMAP_OFF_ADDRSHFT (31) /* lsbit of DIRMAP_OFF is xtalk address bit 31 */ /* Bridge Arbitration register bits definition */ @@ -468,85 +469,98 @@ typedef struct bridge_err_cmdword_s { #define BRIDGE_ARB_REQ_WAIT_TICK_MASK BRIDGE_ARB_REQ_WAIT_TICK(0x3) #define BRIDGE_ARB_REQ_WAIT_EN(x) ((x) << 8) #define BRIDGE_ARB_REQ_WAIT_EN_MASK BRIDGE_ARB_REQ_WAIT_EN(0xff) -#define BRIDGE_ARB_FREEZE_GNT (1 << 6) -#define BRIDGE_ARB_HPRI_RING_B2 (1 << 5) -#define BRIDGE_ARB_HPRI_RING_B1 (1 << 4) -#define BRIDGE_ARB_HPRI_RING_B0 (1 << 3) -#define BRIDGE_ARB_LPRI_RING_B2 (1 << 2) -#define BRIDGE_ARB_LPRI_RING_B1 (1 << 1) -#define BRIDGE_ARB_LPRI_RING_B0 (1 << 0) +#define BRIDGE_ARB_FREEZE_GNT BIT(6) +#define BRIDGE_ARB_HPRI_RING_B2 BIT(5) +#define BRIDGE_ARB_HPRI_RING_B1 BIT(4) +#define BRIDGE_ARB_HPRI_RING_B0 BIT(3) +#define BRIDGE_ARB_LPRI_RING_B2 BIT(2) +#define BRIDGE_ARB_LPRI_RING_B1 BIT(1) +#define BRIDGE_ARB_LPRI_RING_B0 BIT(0) /* Bridge Bus time-out register bits definition */ #define BRIDGE_BUS_PCI_RETRY_HLD(x) ((x) << 16) #define BRIDGE_BUS_PCI_RETRY_HLD_MASK BRIDGE_BUS_PCI_RETRY_HLD(0x1f) -#define BRIDGE_BUS_GIO_TIMEOUT (1 << 12) +#define BRIDGE_BUS_GIO_TIMEOUT BIT(12) #define BRIDGE_BUS_PCI_RETRY_CNT(x) ((x) << 0) #define BRIDGE_BUS_PCI_RETRY_MASK BRIDGE_BUS_PCI_RETRY_CNT(0x3ff) /* Bridge interrupt status register bits definition */ -#define BRIDGE_ISR_MULTI_ERR (0x1u << 31) -#define BRIDGE_ISR_PMU_ESIZE_FAULT (0x1 << 30) -#define BRIDGE_ISR_UNEXP_RESP (0x1 << 29) -#define BRIDGE_ISR_BAD_XRESP_PKT (0x1 << 28) -#define BRIDGE_ISR_BAD_XREQ_PKT (0x1 << 27) -#define BRIDGE_ISR_RESP_XTLK_ERR (0x1 << 26) -#define BRIDGE_ISR_REQ_XTLK_ERR (0x1 << 25) -#define BRIDGE_ISR_INVLD_ADDR (0x1 << 24) -#define BRIDGE_ISR_UNSUPPORTED_XOP (0x1 << 23) -#define BRIDGE_ISR_XREQ_FIFO_OFLOW (0x1 << 22) -#define BRIDGE_ISR_LLP_REC_SNERR (0x1 << 21) -#define BRIDGE_ISR_LLP_REC_CBERR (0x1 << 20) -#define BRIDGE_ISR_LLP_RCTY (0x1 << 19) -#define BRIDGE_ISR_LLP_TX_RETRY (0x1 << 18) -#define BRIDGE_ISR_LLP_TCTY (0x1 << 17) -#define BRIDGE_ISR_SSRAM_PERR (0x1 << 16) -#define BRIDGE_ISR_PCI_ABORT (0x1 << 15) -#define BRIDGE_ISR_PCI_PARITY (0x1 << 14) -#define BRIDGE_ISR_PCI_SERR (0x1 << 13) -#define BRIDGE_ISR_PCI_PERR (0x1 << 12) -#define BRIDGE_ISR_PCI_MST_TIMEOUT (0x1 << 11) +#define BRIDGE_ISR_MULTI_ERR BIT(31) +#define BRIDGE_ISR_PMU_ESIZE_FAULT BIT(30) +#define BRIDGE_ISR_UNEXP_RESP BIT(29) +#define BRIDGE_ISR_BAD_XRESP_PKT BIT(28) +#define BRIDGE_ISR_BAD_XREQ_PKT BIT(27) +#define BRIDGE_ISR_RESP_XTLK_ERR BIT(26) +#define BRIDGE_ISR_REQ_XTLK_ERR BIT(25) +#define BRIDGE_ISR_INVLD_ADDR BIT(24) +#define BRIDGE_ISR_UNSUPPORTED_XOP BIT(23) +#define BRIDGE_ISR_XREQ_FIFO_OFLOW BIT(22) +#define BRIDGE_ISR_LLP_REC_SNERR BIT(21) +#define BRIDGE_ISR_LLP_REC_CBERR BIT(20) +#define BRIDGE_ISR_LLP_RCTY BIT(19) +#define BRIDGE_ISR_LLP_TX_RETRY BIT(18) +#define BRIDGE_ISR_LLP_TCTY BIT(17) +#define BRIDGE_ISR_SSRAM_PERR BIT(16) +#define BRIDGE_ISR_PCI_ABORT BIT(15) +#define BRIDGE_ISR_PCI_PARITY BIT(14) +#define BRIDGE_ISR_PCI_SERR BIT(13) +#define BRIDGE_ISR_PCI_PERR BIT(12) +#define BRIDGE_ISR_PCI_MST_TIMEOUT BIT(11) #define BRIDGE_ISR_GIO_MST_TIMEOUT BRIDGE_ISR_PCI_MST_TIMEOUT -#define BRIDGE_ISR_PCI_RETRY_CNT (0x1 << 10) -#define BRIDGE_ISR_XREAD_REQ_TIMEOUT (0x1 << 9) -#define BRIDGE_ISR_GIO_B_ENBL_ERR (0x1 << 8) -#define BRIDGE_ISR_INT_MSK (0xff << 0) -#define BRIDGE_ISR_INT(x) (0x1 << (x)) - -#define BRIDGE_ISR_LINK_ERROR \ - (BRIDGE_ISR_LLP_REC_SNERR|BRIDGE_ISR_LLP_REC_CBERR| \ - BRIDGE_ISR_LLP_RCTY|BRIDGE_ISR_LLP_TX_RETRY| \ +#define BRIDGE_ISR_PCI_RETRY_CNT BIT(10) +#define BRIDGE_ISR_XREAD_REQ_TIMEOUT BIT(9) +#define BRIDGE_ISR_GIO_B_ENBL_ERR BIT(8) +#define BRIDGE_ISR_INT_MSK GENMASK(7, 0) +#define BRIDGE_ISR_INT(x) BIT((x)) + +#define BRIDGE_ISR_LINK_ERROR \ + (BRIDGE_ISR_LLP_REC_SNERR | \ + BRIDGE_ISR_LLP_REC_CBERR | \ + BRIDGE_ISR_LLP_RCTY | \ + BRIDGE_ISR_LLP_TX_RETRY | \ BRIDGE_ISR_LLP_TCTY) -#define BRIDGE_ISR_PCIBUS_PIOERR \ - (BRIDGE_ISR_PCI_MST_TIMEOUT|BRIDGE_ISR_PCI_ABORT) +#define BRIDGE_ISR_PCIBUS_PIOERR \ + (BRIDGE_ISR_PCI_MST_TIMEOUT | \ + BRIDGE_ISR_PCI_ABORT) -#define BRIDGE_ISR_PCIBUS_ERROR \ - (BRIDGE_ISR_PCIBUS_PIOERR|BRIDGE_ISR_PCI_PERR| \ - BRIDGE_ISR_PCI_SERR|BRIDGE_ISR_PCI_RETRY_CNT| \ +#define BRIDGE_ISR_PCIBUS_ERROR \ + (BRIDGE_ISR_PCIBUS_PIOERR | \ + BRIDGE_ISR_PCI_PERR | \ + BRIDGE_ISR_PCI_SERR | \ + BRIDGE_ISR_PCI_RETRY_CNT | \ BRIDGE_ISR_PCI_PARITY) -#define BRIDGE_ISR_XTALK_ERROR \ - (BRIDGE_ISR_XREAD_REQ_TIMEOUT|BRIDGE_ISR_XREQ_FIFO_OFLOW|\ - BRIDGE_ISR_UNSUPPORTED_XOP|BRIDGE_ISR_INVLD_ADDR| \ - BRIDGE_ISR_REQ_XTLK_ERR|BRIDGE_ISR_RESP_XTLK_ERR| \ - BRIDGE_ISR_BAD_XREQ_PKT|BRIDGE_ISR_BAD_XRESP_PKT| \ +#define BRIDGE_ISR_XTALK_ERROR \ + (BRIDGE_ISR_XREAD_REQ_TIMEOUT | \ + BRIDGE_ISR_XREQ_FIFO_OFLOW | \ + BRIDGE_ISR_UNSUPPORTED_XOP | \ + BRIDGE_ISR_INVLD_ADDR | \ + BRIDGE_ISR_REQ_XTLK_ERR | \ + BRIDGE_ISR_RESP_XTLK_ERR | \ + BRIDGE_ISR_BAD_XREQ_PKT | \ + BRIDGE_ISR_BAD_XRESP_PKT | \ BRIDGE_ISR_UNEXP_RESP) -#define BRIDGE_ISR_ERRORS \ - (BRIDGE_ISR_LINK_ERROR|BRIDGE_ISR_PCIBUS_ERROR| \ - BRIDGE_ISR_XTALK_ERROR|BRIDGE_ISR_SSRAM_PERR| \ +#define BRIDGE_ISR_ERRORS \ + (BRIDGE_ISR_LINK_ERROR | \ + BRIDGE_ISR_PCIBUS_ERROR | \ + BRIDGE_ISR_XTALK_ERROR | \ + BRIDGE_ISR_SSRAM_PERR | \ BRIDGE_ISR_PMU_ESIZE_FAULT) -/* - * List of Errors which are fatal and kill the system - */ -#define BRIDGE_ISR_ERROR_FATAL \ - ((BRIDGE_ISR_XTALK_ERROR & ~BRIDGE_ISR_XREAD_REQ_TIMEOUT)|\ - BRIDGE_ISR_PCI_SERR|BRIDGE_ISR_PCI_PARITY ) +/* List of Errors which are fatal and kill the system */ +#define BRIDGE_ISR_ERROR_FATAL \ + ((BRIDGE_ISR_XTALK_ERROR & \ + ~BRIDGE_ISR_XREAD_REQ_TIMEOUT) | \ + BRIDGE_ISR_PCI_SERR | \ + BRIDGE_ISR_PCI_PARITY) -#define BRIDGE_ISR_ERROR_DUMP \ - (BRIDGE_ISR_PCIBUS_ERROR|BRIDGE_ISR_PMU_ESIZE_FAULT| \ - BRIDGE_ISR_XTALK_ERROR|BRIDGE_ISR_SSRAM_PERR) +#define BRIDGE_ISR_ERROR_DUMP \ + (BRIDGE_ISR_PCIBUS_ERROR | \ + BRIDGE_ISR_PMU_ESIZE_FAULT | \ + BRIDGE_ISR_XTALK_ERROR | \ + BRIDGE_ISR_SSRAM_PERR) /* Bridge interrupt enable register bits definition */ #define BRIDGE_IMR_UNEXP_RESP BRIDGE_ISR_UNEXP_RESP @@ -577,60 +591,73 @@ typedef struct bridge_err_cmdword_s { #define BRIDGE_IMR_INT(x) BRIDGE_ISR_INT(x) /* Bridge interrupt reset register bits definition */ -#define BRIDGE_IRR_MULTI_CLR (0x1 << 6) -#define BRIDGE_IRR_CRP_GRP_CLR (0x1 << 5) -#define BRIDGE_IRR_RESP_BUF_GRP_CLR (0x1 << 4) -#define BRIDGE_IRR_REQ_DSP_GRP_CLR (0x1 << 3) -#define BRIDGE_IRR_LLP_GRP_CLR (0x1 << 2) -#define BRIDGE_IRR_SSRAM_GRP_CLR (0x1 << 1) -#define BRIDGE_IRR_PCI_GRP_CLR (0x1 << 0) -#define BRIDGE_IRR_GIO_GRP_CLR (0x1 << 0) +#define BRIDGE_IRR_MULTI_CLR BIT(6) +#define BRIDGE_IRR_CRP_GRP_CLR BIT(5) +#define BRIDGE_IRR_RESP_BUF_GRP_CLR BIT(4) +#define BRIDGE_IRR_REQ_DSP_GRP_CLR BIT(3) +#define BRIDGE_IRR_LLP_GRP_CLR BIT(2) +#define BRIDGE_IRR_SSRAM_GRP_CLR BIT(1) +#define BRIDGE_IRR_PCI_GRP_CLR BIT(0) +#define BRIDGE_IRR_GIO_GRP_CLR BRIDGE_IRR_PCI_GRP_CLR #define BRIDGE_IRR_ALL_CLR 0x7f -#define BRIDGE_IRR_CRP_GRP (BRIDGE_ISR_UNEXP_RESP | \ - BRIDGE_ISR_XREQ_FIFO_OFLOW) -#define BRIDGE_IRR_RESP_BUF_GRP (BRIDGE_ISR_BAD_XRESP_PKT | \ - BRIDGE_ISR_RESP_XTLK_ERR | \ - BRIDGE_ISR_XREAD_REQ_TIMEOUT) -#define BRIDGE_IRR_REQ_DSP_GRP (BRIDGE_ISR_UNSUPPORTED_XOP | \ - BRIDGE_ISR_BAD_XREQ_PKT | \ - BRIDGE_ISR_REQ_XTLK_ERR | \ - BRIDGE_ISR_INVLD_ADDR) -#define BRIDGE_IRR_LLP_GRP (BRIDGE_ISR_LLP_REC_SNERR | \ - BRIDGE_ISR_LLP_REC_CBERR | \ - BRIDGE_ISR_LLP_RCTY | \ - BRIDGE_ISR_LLP_TX_RETRY | \ - BRIDGE_ISR_LLP_TCTY) -#define BRIDGE_IRR_SSRAM_GRP (BRIDGE_ISR_SSRAM_PERR | \ - BRIDGE_ISR_PMU_ESIZE_FAULT) -#define BRIDGE_IRR_PCI_GRP (BRIDGE_ISR_PCI_ABORT | \ - BRIDGE_ISR_PCI_PARITY | \ - BRIDGE_ISR_PCI_SERR | \ - BRIDGE_ISR_PCI_PERR | \ - BRIDGE_ISR_PCI_MST_TIMEOUT | \ - BRIDGE_ISR_PCI_RETRY_CNT) - -#define BRIDGE_IRR_GIO_GRP (BRIDGE_ISR_GIO_B_ENBL_ERR | \ - BRIDGE_ISR_GIO_MST_TIMEOUT) +#define BRIDGE_IRR_CRP_GRP \ + (BRIDGE_ISR_UNEXP_RESP | \ + BRIDGE_ISR_XREQ_FIFO_OFLOW) + +#define BRIDGE_IRR_RESP_BUF_GRP \ + (BRIDGE_ISR_BAD_XRESP_PKT | \ + BRIDGE_ISR_RESP_XTLK_ERR | \ + BRIDGE_ISR_XREAD_REQ_TIMEOUT) + +#define BRIDGE_IRR_REQ_DSP_GRP \ + (BRIDGE_ISR_UNSUPPORTED_XOP | \ + BRIDGE_ISR_BAD_XREQ_PKT | \ + BRIDGE_ISR_REQ_XTLK_ERR | \ + BRIDGE_ISR_INVLD_ADDR) + +#define BRIDGE_IRR_LLP_GRP \ + (BRIDGE_ISR_LLP_REC_SNERR | \ + BRIDGE_ISR_LLP_REC_CBERR | \ + BRIDGE_ISR_LLP_RCTY | \ + BRIDGE_ISR_LLP_TX_RETRY | \ + BRIDGE_ISR_LLP_TCTY) + +#define BRIDGE_IRR_SSRAM_GRP \ + (BRIDGE_ISR_SSRAM_PERR | \ + BRIDGE_ISR_PMU_ESIZE_FAULT) + +#define BRIDGE_IRR_PCI_GRP \ + (BRIDGE_ISR_PCI_ABORT | \ + BRIDGE_ISR_PCI_PARITY | \ + BRIDGE_ISR_PCI_SERR | \ + BRIDGE_ISR_PCI_PERR | \ + BRIDGE_ISR_PCI_MST_TIMEOUT | \ + BRIDGE_ISR_PCI_RETRY_CNT) + +#define BRIDGE_IRR_GIO_GRP \ + (BRIDGE_ISR_GIO_B_ENBL_ERR | \ + BRIDGE_ISR_GIO_MST_TIMEOUT) /* Bridge INT_DEV register bits definition */ -#define BRIDGE_INT_DEV_SHFT(n) ((n)*3) +#define BRIDGE_INT_DEV_SHFT(n) ((n) * 3) #define BRIDGE_INT_DEV_MASK(n) (0x7 << BRIDGE_INT_DEV_SHFT(n)) #define BRIDGE_INT_DEV_SET(_dev, _line) (_dev << BRIDGE_INT_DEV_SHFT(_line)) /* Bridge interrupt(x) register bits definition */ -#define BRIDGE_INT_ADDR_HOST 0x0003FF00 -#define BRIDGE_INT_ADDR_FLD 0x000000FF +#define BRIDGE_INT_ADDR_HOST GENMASK(17, 8) +#define BRIDGE_INT_ADDR_FLD GENMASK(7, 0) -#define BRIDGE_TMO_PCI_RETRY_HLD_MASK 0x1f0000 -#define BRIDGE_TMO_GIO_TIMEOUT_MASK 0x001000 -#define BRIDGE_TMO_PCI_RETRY_CNT_MASK 0x0003ff +/* Bridge timeout register bits definition */ +#define BRIDGE_TMO_PCI_RETRY_HLD_MASK GENMASK(20, 16) +#define BRIDGE_TMO_GIO_TIMEOUT_MASK BIT(12) +#define BRIDGE_TMO_PCI_RETRY_CNT_MASK GENMASK(9, 0) #define BRIDGE_TMO_PCI_RETRY_CNT_MAX 0x3ff /* * The NASID should be shifted by this amount and stored into the - * interrupt(x) register. + * interrupt(x) register. (IP27/IP35 only) */ #define BRIDGE_INT_ADDR_NASID_SHFT 8 @@ -638,76 +665,82 @@ typedef struct bridge_err_cmdword_s { * The BRIDGE_INT_ADDR_DEST_IO bit should be set to send an interrupt to * memory. */ -#define BRIDGE_INT_ADDR_DEST_IO (1 << 17) +#define BRIDGE_INT_ADDR_DEST_IO BIT(17) #define BRIDGE_INT_ADDR_DEST_MEM 0 -#define BRIDGE_INT_ADDR_MASK (1 << 17) +#define BRIDGE_INT_ADDR_MASK BIT(17) /* Bridge device(x) register bits definition */ -#define BRIDGE_DEV_ERR_LOCK_EN 0x10000000 -#define BRIDGE_DEV_PAGE_CHK_DIS 0x08000000 -#define BRIDGE_DEV_FORCE_PCI_PAR 0x04000000 -#define BRIDGE_DEV_VIRTUAL_EN 0x02000000 -#define BRIDGE_DEV_PMU_WRGA_EN 0x01000000 -#define BRIDGE_DEV_DIR_WRGA_EN 0x00800000 -#define BRIDGE_DEV_DEV_SIZE 0x00400000 -#define BRIDGE_DEV_RT 0x00200000 -#define BRIDGE_DEV_SWAP_PMU 0x00100000 -#define BRIDGE_DEV_SWAP_DIR 0x00080000 -#define BRIDGE_DEV_PREF 0x00040000 -#define BRIDGE_DEV_PRECISE 0x00020000 -#define BRIDGE_DEV_COH 0x00010000 -#define BRIDGE_DEV_BARRIER 0x00008000 -#define BRIDGE_DEV_GBR 0x00004000 -#define BRIDGE_DEV_DEV_SWAP 0x00002000 -#define BRIDGE_DEV_DEV_IO_MEM 0x00001000 -#define BRIDGE_DEV_OFF_MASK 0x00000fff +#define BRIDGE_DEV_ERR_LOCK_EN BIT(28) +#define BRIDGE_DEV_PAGE_CHK_DIS BIT(27) +#define BRIDGE_DEV_FORCE_PCI_PAR BIT(26) +#define BRIDGE_DEV_VIRTUAL_EN BIT(25) +#define BRIDGE_DEV_PMU_WRGA_EN BIT(24) +#define BRIDGE_DEV_DIR_WRGA_EN BIT(23) +#define BRIDGE_DEV_DEV_SIZE BIT(22) +#define BRIDGE_DEV_RT BIT(21) +#define BRIDGE_DEV_SWAP_PMU BIT(20) +#define BRIDGE_DEV_SWAP_DIR BIT(19) +#define BRIDGE_DEV_PREF BIT(18) +#define BRIDGE_DEV_PRECISE BIT(17) +#define BRIDGE_DEV_COH BIT(16) +#define BRIDGE_DEV_BARRIER BIT(15) +#define BRIDGE_DEV_GBR BIT(14) +#define BRIDGE_DEV_DEV_SWAP BIT(13) +#define BRIDGE_DEV_DEV_IO_MEM BIT(12) +#define BRIDGE_DEV_OFF_MASK GENMASK(11, 0) #define BRIDGE_DEV_OFF_ADDR_SHFT 20 -#define BRIDGE_DEV_PMU_BITS (BRIDGE_DEV_PMU_WRGA_EN | \ - BRIDGE_DEV_SWAP_PMU) -#define BRIDGE_DEV_D32_BITS (BRIDGE_DEV_DIR_WRGA_EN | \ - BRIDGE_DEV_SWAP_DIR | \ - BRIDGE_DEV_PREF | \ - BRIDGE_DEV_PRECISE | \ - BRIDGE_DEV_COH | \ - BRIDGE_DEV_BARRIER) -#define BRIDGE_DEV_D64_BITS (BRIDGE_DEV_DIR_WRGA_EN | \ - BRIDGE_DEV_SWAP_DIR | \ - BRIDGE_DEV_COH | \ - BRIDGE_DEV_BARRIER) +#define BRIDGE_DEV_PMU_BITS \ + (BRIDGE_DEV_PMU_WRGA_EN | \ + BRIDGE_DEV_SWAP_PMU) + +#define BRIDGE_DEV_D32_BITS \ + (BRIDGE_DEV_DIR_WRGA_EN | \ + BRIDGE_DEV_SWAP_DIR | \ + BRIDGE_DEV_PREF | \ + BRIDGE_DEV_PRECISE | \ + BRIDGE_DEV_COH | \ + BRIDGE_DEV_BARRIER) + +#define BRIDGE_DEV_D64_BITS \ + (BRIDGE_DEV_DIR_WRGA_EN | \ + BRIDGE_DEV_SWAP_DIR | \ + BRIDGE_DEV_COH | \ + BRIDGE_DEV_BARRIER) /* Bridge Error Upper register bit field definition */ -#define BRIDGE_ERRUPPR_DEVMASTER (0x1 << 20) /* Device was master */ -#define BRIDGE_ERRUPPR_PCIVDEV (0x1 << 19) /* Virtual Req value */ +#define BRIDGE_ERRUPPR_DEVMASTER BIT(20) /* Device was master */ +#define BRIDGE_ERRUPPR_PCIVDEV BIT(19) /* Virtual Req value */ #define BRIDGE_ERRUPPR_DEVNUM_SHFT (16) #define BRIDGE_ERRUPPR_DEVNUM_MASK (0x7 << BRIDGE_ERRUPPR_DEVNUM_SHFT) -#define BRIDGE_ERRUPPR_DEVICE(err) (((err) >> BRIDGE_ERRUPPR_DEVNUM_SHFT) & 0x7) -#define BRIDGE_ERRUPPR_ADDRMASK (0xFFFF) +#define BRIDGE_ERRUPPR_DEVICE(err) \ + (((err) >> BRIDGE_ERRUPPR_DEVNUM_SHFT) & 0x7) +#define BRIDGE_ERRUPPR_ADDRMASK GENMASK(15, 0) /* Bridge interrupt mode register bits definition */ -#define BRIDGE_INTMODE_CLR_PKT_EN(x) (0x1 << (x)) +#define BRIDGE_INTMODE_CLR_PKT_EN(x) BIT((x)) -/* this should be written to the xbow's link_control(x) register */ -#define BRIDGE_CREDIT 3 +/* This should be written to the XBOW's link_control(x) register */ +#define BRIDGE_CREDIT 3 -/* RRB assignment register */ -#define BRIDGE_RRB_EN 0x8 /* after shifting down */ -#define BRIDGE_RRB_DEV 0x7 /* after shifting down */ -#define BRIDGE_RRB_VDEV 0x4 /* after shifting down */ -#define BRIDGE_RRB_PDEV 0x3 /* after shifting down */ +/* RRB assignment register -- value applies after shifting down */ +#define BRIDGE_RRB_EN 0x8 +#define BRIDGE_RRB_DEV 0x7 +#define BRIDGE_RRB_VDEV 0x4 +#define BRIDGE_RRB_PDEV 0x3 /* RRB status register */ -#define BRIDGE_RRB_VALID(r) (0x00010000<<(r)) -#define BRIDGE_RRB_INUSE(r) (0x00000001<<(r)) +#define BRIDGE_RRB_VALID(r) (0x00010000 << (r)) +#define BRIDGE_RRB_INUSE(r) (0x00000001 << (r)) /* RRB clear register */ -#define BRIDGE_RRB_CLEAR(r) (0x00000001<<(r)) +#define BRIDGE_RRB_CLEAR(r) (0x00000001 << (r)) -/* xbox system controller declarations */ -#define XBOX_BRIDGE_WID 8 -#define FLASH_PROM1_BASE 0xE00000 /* To read the xbox sysctlr status */ -#define XBOX_RPS_EXISTS 1 << 6 /* RPS bit in status register */ -#define XBOX_RPS_FAIL 1 << 4 /* RPS status bit in register */ +/* Xbox system controller declarations */ +#define XBOX_BRIDGE_WID 8 +#define FLASH_PROM1_BASE 0xe00000 /* Read Xbox sysctlr stat */ +#define XBOX_RPS_EXISTS BIT(6) /* RPS bit in status reg */ +#define XBOX_RPS_FAIL BIT(4) /* RPS status bit in reg */ /* ======================================================================== */ @@ -716,12 +749,12 @@ typedef struct bridge_err_cmdword_s { * refer to section 4.2.1 of Bridge Spec for xtalk to PCI/GIO PIO mappings */ /* XTALK addresses that map into Bridge Bus addr space */ -#define BRIDGE_PIO32_XTALK_ALIAS_BASE 0x000040000000L -#define BRIDGE_PIO32_XTALK_ALIAS_LIMIT 0x00007FFFFFFFL -#define BRIDGE_PIO64_XTALK_ALIAS_BASE 0x000080000000L -#define BRIDGE_PIO64_XTALK_ALIAS_LIMIT 0x0000BFFFFFFFL -#define BRIDGE_PCIIO_XTALK_ALIAS_BASE 0x000100000000L -#define BRIDGE_PCIIO_XTALK_ALIAS_LIMIT 0x0001FFFFFFFFL +#define BRIDGE_PIO32_XTALK_ALIAS_BASE 0x000040000000UL +#define BRIDGE_PIO32_XTALK_ALIAS_LIMIT 0x00007fffffffUL +#define BRIDGE_PIO64_XTALK_ALIAS_BASE 0x000080000000UL +#define BRIDGE_PIO64_XTALK_ALIAS_LIMIT 0x0000bfffffffUL +#define BRIDGE_PCIIO_XTALK_ALIAS_BASE 0x000100000000UL +#define BRIDGE_PCIIO_XTALK_ALIAS_LIMIT 0x0001ffffffffUL /* Ranges of PCI bus space that can be accessed via PIO from xtalk */ #define BRIDGE_MIN_PIO_ADDR_MEM 0x00000000 /* 1G PCI memory space */ @@ -753,15 +786,17 @@ typedef struct bridge_err_cmdword_s { #define PCI32_MAPPED_BASE BRIDGE_DMA_MAPPED_BASE #define PCI32_DIRECT_BASE BRIDGE_DMA_DIRECT_BASE -#define IS_PCI32_LOCAL(x) ((ulong_t)(x) < PCI32_MAPPED_BASE) -#define IS_PCI32_MAPPED(x) ((ulong_t)(x) < PCI32_DIRECT_BASE && \ - (ulong_t)(x) >= PCI32_MAPPED_BASE) -#define IS_PCI32_DIRECT(x) ((ulong_t)(x) >= PCI32_MAPPED_BASE) -#define IS_PCI64(x) ((ulong_t)(x) >= PCI64_BASE) - -/* - * The GIO address space. - */ +#define IS_PCI32_LOCAL(x) \ + ((ulong_t)(x) < PCI32_MAPPED_BASE) +#define IS_PCI32_MAPPED(x) \ + ((ulong_t)(x) < PCI32_DIRECT_BASE && \ + (ulong_t)(x) >= PCI32_MAPPED_BASE) +#define IS_PCI32_DIRECT(x) \ + ((ulong_t)(x) >= PCI32_MAPPED_BASE) +#define IS_PCI64(x) \ + ((ulong_t)(x) >= PCI64_BASE) + +/* GIO address space. */ /* Xtalk to GIO PIO */ #define BRIDGE_GIO_MEM32_BASE BRIDGE_PIO32_XTALK_ALIAS_BASE #define BRIDGE_GIO_MEM32_LIMIT BRIDGE_PIO32_XTALK_ALIAS_LIMIT @@ -772,29 +807,33 @@ typedef struct bridge_err_cmdword_s { #define GIO_MAPPED_BASE BRIDGE_DMA_MAPPED_BASE #define GIO_DIRECT_BASE BRIDGE_DMA_DIRECT_BASE -#define IS_GIO_LOCAL(x) ((ulong_t)(x) < GIO_MAPPED_BASE) -#define IS_GIO_MAPPED(x) ((ulong_t)(x) < GIO_DIRECT_BASE && \ - (ulong_t)(x) >= GIO_MAPPED_BASE) -#define IS_GIO_DIRECT(x) ((ulong_t)(x) >= GIO_MAPPED_BASE) +#define IS_GIO_LOCAL(x) \ + ((ulong_t)(x) < GIO_MAPPED_BASE) +#define IS_GIO_MAPPED(x) \ + ((ulong_t)(x) < GIO_DIRECT_BASE && \ + (ulong_t)(x) >= GIO_MAPPED_BASE) +#define IS_GIO_DIRECT(x) \ + ((ulong_t)(x) >= GIO_MAPPED_BASE) + -/* PCI to xtalk mapping */ +/* PCI-to-Xtalk mapping */ /* given a DIR_OFF value and a pci/gio 32 bits direct address, determine * which xtalk address is accessed */ #define BRIDGE_DIRECT_32_SEG_SIZE BRIDGE_DMA_DIRECT_SIZE -#define BRIDGE_DIRECT_32_TO_XTALK(dir_off,adr) \ - ((dir_off) * BRIDGE_DIRECT_32_SEG_SIZE + \ - ((adr) & (BRIDGE_DIRECT_32_SEG_SIZE - 1)) + PHYS_RAMBASE) +#define BRIDGE_DIRECT_32_TO_XTALK(dir_off, adr) \ + ((dir_off) * BRIDGE_DIRECT_32_SEG_SIZE + \ + ((adr) & (BRIDGE_DIRECT_32_SEG_SIZE - 1)) + PHYS_RAMBASE) /* 64-bit address attribute masks */ -#define PCI64_ATTR_TARG_MASK 0xf000000000000000 +#define PCI64_ATTR_TARG_MASK GENMASK_ULL(63, 60) #define PCI64_ATTR_TARG_SHFT 60 -#define PCI64_ATTR_PREF 0x0800000000000000 -#define PCI64_ATTR_PREC 0x0400000000000000 -#define PCI64_ATTR_VIRTUAL 0x0200000000000000 -#define PCI64_ATTR_BAR 0x0100000000000000 -#define PCI64_ATTR_RMF_MASK 0x00ff000000000000 +#define PCI64_ATTR_PREF BIT_ULL(59) +#define PCI64_ATTR_PREC BIT_ULL(58) +#define PCI64_ATTR_VIRTUAL BIT_ULL(57) +#define PCI64_ATTR_BAR BIT_ULL(56) +#define PCI64_ATTR_RMF_MASK GENMASK_ULL(55, 48) #define PCI64_ATTR_RMF_SHFT 48 #ifndef __ASSEMBLY__ @@ -815,33 +854,34 @@ typedef union ate_u { } ate_t; #endif /* !__ASSEMBLY__ */ -#define ATE_V 0x01 -#define ATE_CO 0x02 -#define ATE_PREC 0x04 -#define ATE_PREF 0x08 -#define ATE_BAR 0x10 +#define ATE_V 0x01 +#define ATE_CO 0x02 +#define ATE_PREC 0x04 +#define ATE_PREF 0x08 +#define ATE_BAR 0x10 #define ATE_PFNSHIFT 12 #define ATE_TIDSHIFT 8 #define ATE_RMFSHIFT 48 -#define mkate(xaddr, xid, attr) ((xaddr) & 0x0000fffffffff000ULL) | \ - ((xid)<<ATE_TIDSHIFT) | \ - (attr) +#define mkate(xaddr, xid, attr) \ + (((xaddr) & 0x0000fffffffff000ULL) | \ + ((xid) << ATE_TIDSHIFT) | \ + (attr)) #define BRIDGE_INTERNAL_ATES 128 struct bridge_controller { - struct pci_controller pc; - struct resource mem; - struct resource io; - struct resource busn; - bridge_t *base; - nasid_t nasid; - unsigned int widget_id; - unsigned int irq_cpu; - u64 baddr; - unsigned int pci_int[8]; + struct pci_controller pc; + struct resource mem; + struct resource io; + struct resource busn; + bridge_t *base; + nasid_t nasid; + u32 widget_id; + u32 irq_cpu; + u64 baddr; + u32 pci_int[8]; }; #define BRIDGE_CONTROLLER(bus) \ -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 06/12] MIPS: BRIDGE: Overhaul bridge.h header file 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard ` (4 preceding siblings ...) 2017-02-07 6:13 ` [PATCH 05/12] MIPS: BRIDGE: Clean-up bridge.h header file Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 07/12] MIPS: BRIDGE: Add XBRIDGE revs and SWAP bit Joshua Kinard ` (5 subsequent siblings) 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS From: Joshua Kinard <kumba@gentoo.org> Rewrite/replace the large bridge_t typedef volatile struct used to access BRIDGE registers with a new struct that is comprised of several smaller structs and unions. Documentation for each struct, derived from both the BRIDGE specs and XBRIDGE specs, is included in kernel-doc format. Register read and write macros are also included that simplify access to BRIDGE/XBRIDGE registers: Old (given 'bridge_t *bridge;'): Read: foo = bridge->b_int_enable; Write: bridge->b_int_enable = foo; New (given 'struct bridge_controller *bc;'): Read: foo = bridge_read_reg(bc, b_int_enable); Write: bridge_write_reg(foo, bc, b_int_enable); Other changes in this overhaul patch include: - Whitespace/comment fixes - Update older comments with info from the BRIDGE/XBRIDGE specs - Remove old/unused macros that accessed various BRIDGE registers - Added 'alloc_irq' hook to struct bridge_controller (used by IP27) - Per IA64, reading the BRIDGE flush register is done in a while loop - 'bridge_root_dev' is defined here instead of in pci-bridge.c - A platform_data struct is defined for BRIDGE for later use by IP27 There are several temporary pieces of code left that are needed by the current IP27 code to boot correctly under these changes. They will be removed in the future with the IP27 "mega update" patch series. Signed-off-by: Joshua Kinard <kumba@gentoo.org> --- arch/mips/include/asm/pci/bridge.h | 864 ++++++++++++++++----------- 1 file changed, 511 insertions(+), 353 deletions(-) diff --git a/arch/mips/include/asm/pci/bridge.h b/arch/mips/include/asm/pci/bridge.h index 248a390e89b3..3c214feab772 100644 --- a/arch/mips/include/asm/pci/bridge.h +++ b/arch/mips/include/asm/pci/bridge.h @@ -8,385 +8,468 @@ * * Copyright (C) 1996, 1999 Silcon Graphics, Inc. * Copyright (C) 1999 Ralf Baechle (ralf@gnu.org) + * Copyright (C) 2016 Joshua Kinard <kumba@gentoo.org> */ #ifndef _ASM_PCI_BRIDGE_H #define _ASM_PCI_BRIDGE_H #include <linux/types.h> #include <linux/pci.h> -#include <asm/xtalk/xwidget.h> /* generic widget header */ +#include <linux/spinlock.h> + +#include <asm/xtalk/xwidget.h> /* generic xtalk widget header */ + +#ifdef CONFIG_SGI_IP27 #include <asm/sn/types.h> +#endif -/* I/O page size */ +/* ----------------------------------------------------------------------- */ +/* Common BRIDGE/XBRIDGE defines. */ -#define IOPFNSHIFT 12 /* 4K per mapped page */ +/* XXX: XBRIDGE definitions/support is incomplete and still needs work! */ +/* I/O page size */ +#define IOPFNSHIFT 12 /* 4K per mapped page */ #define IOPGSIZE (1 << IOPFNSHIFT) #define IOPG(x) ((x) >> IOPFNSHIFT) #define IOPGOFF(x) ((x) & (IOPGSIZE-1)) -/* Bridge RAM sizes */ - +/* + * ATE RAM is 1KB per entry. + * BRIDGE has 128KB of ATE. + * XBRIDGE has 1024KB of ATE. + */ #define BRIDGE_ATE_RAM_SIZE 0x00000400 /* 1kB ATE RAM */ -#define BRIDGE_CONFIG_BASE 0x20000 -#define BRIDGE_CONFIG1_BASE 0x28000 -#define BRIDGE_CONFIG_END 0x30000 -#define BRIDGE_CONFIG_SLOT_SIZE 0x1000 - +/* BRIDGE/XBRIDGE SSRAM comes in several sizes. */ #define BRIDGE_SSRAM_512K 0x00080000 /* 512kB */ #define BRIDGE_SSRAM_128K 0x00020000 /* 128kB */ #define BRIDGE_SSRAM_64K 0x00010000 /* 64kB */ #define BRIDGE_SSRAM_0K 0x00000000 /* 0kB */ -/* ======================================================================== - * Bridge address map +/* + * Up to 8 devices per BRIDGE/XBRIDGE + * XXX: PIC maxes out at 4 devices, but that can be added later. */ +#define BRIDGE_MAX_DEVS 8 + +/* ----------------------------------------------------------------------- */ + + +/* ----------------------------------------------------------------------- */ +/* BRIDGE/XBRIDGE structures that map the address space. */ #ifndef __ASSEMBLY__ /* - * All accesses to bridge hardware registers must be done - * using 32-bit loads and stores. + * Similar to crosstalk registers, BRIDGE/XBRIDGE registers are 32-bits or + * less in size and are aligned to a 64-bit boundary. Register data begins + * at bits 31:0 and can only be accessed by a Crosstalk double-word packet + * type. As such, we can reuse 'union xbow_reg' from xwidget.h, even though + * there is no need to access BRIDGE/XBRIDGE registers in a 64-bit manner. + * + */ +#define bridge_reg xbow_reg + +/** + * struct bridge_registers - BRIDGE/XBRIDGE hardware registers. + * @wid_cfg: struct xwidget_cmn_regs access to common widget config registers. + * @__pad0: 0x8 bytes of padding (u64 * 1). + * @dir_map: r/w, relocate a 2GB region for PCI-to-Xtalk transfers. + * @__pad1: 0x8 bytes of padding (u64 * 1). + * @ssram_perr: r/o, holds the address of an SSRAM parity error. + * @__pad2: 0x8 bytes of padding (u64 * 1). + * @arb_prio: r/w, sets priority/timeout timing for PCI/GIO arbitration. + * @__pad3: 0x8 bytes of padding (u64 * 1). + * @nic: r/w, access to the 1-Wire Number-In-a-Can device. + * @__pad4: 0x8 bytes of padding (u64 * 1). + * @pcigio_bus_timo: r/w, sets GIO timeout or PCI retry timeout/count. + * @pci_type1_cfg: r/w, selects secondary bus/dev during PCI type 1 accesses. + * @pcigio_err_upper: r/o, access to upper 32-bits of PCI/GIO error address. + * @pcigio_err_lower: r/o, access to lower 32-bits of PCI/GIO error address. + * @__pad5: 0x0020 bytes of padding (0x04 * 8). + * @int_status: r/o, BRIDGE/XBRIDGE interrupt status register. + * @int_enable: r/w, enables specific interrupts in 'int_status'. + * @int_reset_stat: w/o, clears all bits not tied to an int pin. + * @int_mode: r/w, sets the mode of the interrupt pins. + * @int_device: r/w, associates interrupt pins with devices. + * @int_host_err: r/w, describes which bit in the host ISR to reset on error. + * @int_addr: r/w, maps int pins to host addresses and int bits. + * @__pad6: 0x0090 bytes of padding (0x12 * 8). + * @device: r/w, generic per-device (DevIO) control register. + * @dev_w_req_buf: r/o, returns 0 after per-dev write buffer has been flushed. + * @dev_even_rrb: r/w, allocates read response buffers for slots 0, 2, 4, & 6. + * @dev_odd_rrb: r/w, allocates read response buffers for slots 1, 3, 5, & 7. + * @dev_rrb_status: r/o, holds current read response buffer status. + * @dev_rrb_clear: w/o, write to bits 15:0 to clear buffer after disable. + * @__pad7: 0xfc60 bytes of padding (0x1fac * 8). + * + * XXX: XBRIDGE may need a separate struct definition, as some things are + * different, such as the number of ATEs is larger, and this alters the + * address space layout. */ -typedef u32 bridgereg_t; +struct bridge_registers { + /* Widget config */ + struct xwidget_cmn_regs wid_cfg; /* 0x000000 */ + u64 __pad0; /* +0x0078 */ + /* PMU/Direct map */ + union bridge_reg dir_map; /* +0x0080 */ + u64 __pad1; /* +0x0088 */ + /* SSRAM */ + union bridge_reg ssram_perr; /* +0x0090 */ + u64 __pad2; /* +0x0098 */ + /* Arbitration */ + union bridge_reg arb_prio; /* +0x00a0 */ + u64 __pad3; /* +0x00a8 */ + /* Number-In-a-Can */ + union bridge_reg nic; /* +0x00b0 */ + u64 __pad4; /* +0x00b8 */ + /* PCI/GIO */ + union bridge_reg pcigio_bus_timo; /* +0x00c0 */ + union bridge_reg pci_type1_cfg; /* +0x00c8 */ + union bridge_reg pcigio_err_upper; /* +0x00d0 */ + union bridge_reg pcigio_err_lower; /* +0x00d8 */ + u64 __pad5[0x0004]; /* +0x00e0 + 0x20 */ + /* Interrupts */ + union bridge_reg int_status; /* +0x0100 */ + union bridge_reg int_enable; /* +0x0108 */ + union bridge_reg int_reset_stat; /* +0x0110 */ + union bridge_reg int_mode; /* +0x0118 */ + union bridge_reg int_device; /* +0x0120 */ + union bridge_reg int_host_err; /* +0x0128 */ + union bridge_reg int_addr[8]; /* +0x0130 + 0x40 */ + u64 __pad6[0x0012]; /* +0x0170 + 0x90 */ + /* Devices */ + union bridge_reg device[8]; /* +0x0200 + 0x40 */ + union bridge_reg dev_w_req_buf[8]; /* +0x0240 + 0x40 */ + union bridge_reg dev_even_rrb; /* +0x0280 */ + union bridge_reg dev_odd_rrb; /* +0x0288 */ + union bridge_reg dev_rrb_status; /* +0x0290 */ + union bridge_reg dev_rrb_clear; /* +0x0298 */ + u64 __pad7[0x1fac]; /* +0x02a0 + 0xfd60 */ +}; -typedef u64 bridge_ate_t; +/* + * BRIDGE/XBRIDGE widget configuration registers mirror common Crosstalk + * widget registers up until 0x48 (xw_llp_ctrl). BRIDGE/XBRIDGE-specific + * config registers begin at 0x50 and continue until 0x70. These macros + * below are shortcut names that map to either the shared + * 'struct xwidget_cmn_regs' names or those unique to BRIDGE/XBRIDGE, via a + * 'union bridge_reg' type's ".l.data" member, and must be accessed off of a + * defined 'struct bridge_registers' variable. + */ +#define b_wid_id wid_cfg.id.l.data /* ro */ +#define b_wid_status wid_cfg.status.l.data /* ro */ +#define b_wid_err_upper wid_cfg.err_upper.l.data /* ro */ +#define b_wid_err_lower wid_cfg.err_lower.l.data /* ro */ +#define b_wid_ctrl wid_cfg.wid_ctrl.l.data /* rw */ +#define b_wid_pkt_timo wid_cfg.pkt_timo.l.data /* rw */ +#define b_wid_int_upper wid_cfg.int_upper.l.data /* rw */ +#define b_wid_int_lower wid_cfg.int_lower.l.data /* rw */ +#define b_wid_err_cmdword wid_cfg.err_cmdword.l.data /* ro */ +#define b_wid_llp_ctrl wid_cfg.llp_ctrl.l.data /* rw */ +#define b_wid_targ_flush wid_cfg.stat_clr.l.data /* ro */ +#define b_wid_aux_errcmd wid_cfg.arb_reload.l.data /* ro */ +#define b_wid_resp_buf_upper wid_cfg.perf_ctr_a.l.data /* ro */ +#define b_wid_resp_buf_lower wid_cfg.perf_ctr_b.l.data /* ro */ +#define b_wid_test_pinctrl wid_cfg.nic.l.data /* rw */ +#define b_dir_map dir_map.l.data /* rw */ +#define b_ssram_perr ssram_perr.l.data /* ro */ +#define b_arb_prio arb_prio.l.data /* rw */ +#define b_nic nic.l.data /* rw */ +#define b_pci_bus_timo pcigio_bus_timo.l.data /* rw */ +#define b_pci_type1_cfg pci_type1_cfg.l.data /* rw */ +#define b_pci_err_upper pcigio_err_upper.l.data /* ro */ +#define b_pci_err_lower pcigio_err_lower.l.data /* ro */ +#define b_int_status int_status.l.data /* ro */ +#define b_int_enable int_enable.l.data /* rw */ +#define b_int_reset_stat int_reset_stat.l.data /* wo */ +#define b_int_mode int_mode.l.data /* rw */ +#define b_int_device int_device.l.data /* rw */ +#define b_int_host_err int_host_err.l.data /* rw */ +#define b_int_addr(_x) int_addr[(_x)].l.data /* rw */ +#define b_device(_x) device[(_x)].l.data /* rw */ +#define b_dev_w_req_buf(_x) dev_w_req_buf[(_x)].l.data /* ro */ +#define b_dev_even_rrb dev_even_rrb.l.data /* rw */ +#define b_dev_odd_rrb dev_odd_rrb.l.data /* rw */ +#define b_dev_rrb_status dev_rrb_status.l.data /* ro */ +#define b_dev_rrb_clear dev_rrb_clear.l.data /* wo */ -/* pointers to bridge ATEs - * are always "pointer to volatile" +/* + * BRIDGE/XBRIDGE can support either a PCI Bus or a GIO Bus. As of writing, + * there is no known SGI platform supported by Linux where we'll encounter an + * XIO->GIO setup via BRIDGE/XBRIDGE. The below macros mirror several of the + * above 'b_pci_*' names for better code readability should such support ever + * be added. */ -typedef volatile bridge_ate_t *bridge_ate_p; +#define b_gio_bus_timo b_pci_bus_timo /* rw */ +#define b_gio_err_upper b_pci_err_upper /* ro */ +#define b_gio_err_lower b_pci_err_lower /* ro */ /* - * It is generally preferred that hardware registers on the bridge - * are located from C code via this structure. + * Per include/asm-ia64/sn/pci/bridge.h in Linux-2.5.70: + * In [X]bridge the internal ATE Ram is writen as double words only, + * but due to internal design issues it is read back as single words. + * i.e: + * b_int_ate_ram[index].hi.rd << 32 | xb_int_ate_ram_lo[index].rd * - * Generated from Bridge spec dated 04oct95 + * It just so happens that 'union xbow_reg' fits this situation perfectly, + * and with a little macro magic, can be used to access BRIDGE/XBRIDGE ATE + * entries rather easily. However, unlike the other macros, 'b_ate_read' and + * 'b_ate_write' are passed the struct variable name and index offset, instead + * of being accessed off of a defined struct variable. */ +#define bridge_ate xbow_reg +#define b_ate_read(_s, _x) (_s.ate_ram[_x].l.data << 32 | \ + _s.ate_ram_lo[_x].l.data) +#define b_ate_write(_s, _x) (_s.ate_ram[_x].q) + +/** + * struct bridge_pci_cfg_cmn - common fields of a PCI config space type. + * @dev_id: device ID. + * @vendor: vendor ID. + * @status: status register. + * @cmd: command register. + * @class_code: PCI class code. + * @rev_id: revision ID. + * @bist: built-in self-test. + * @hdr_type: header type. + * @latency_tmr: latency timer. + * @cache_ln_sz: cache line size. + */ +struct bridge_pci_cfg_cmn { + u16 dev_id; /* 0x00 */ + u16 vendor; /* 0x02 */ + u16 status; /* 0x04 */ + u16 cmd; /* 0x06 */ + u32 class_code_rev; /* 0x08 */ + u8 bist; /* 0x0c */ + u8 hdr_type; /* 0x0d */ + u8 latency_tmr; /* 0x0e */ + u8 cache_ln_sz; /* 0x0f */ +}; -typedef volatile struct bridge_s { - /* Local Registers 0x000000-0x00FFFF */ - - /* standard widget configuration 0x000000-0x000057 */ - widget_cfg_t b_widget; /* 0x000000 */ - - /* helper fieldnames for accessing bridge widget */ - -#define b_wid_id b_widget.w_id -#define b_wid_stat b_widget.w_status -#define b_wid_err_upper b_widget.w_err_upper_addr -#define b_wid_err_lower b_widget.w_err_lower_addr -#define b_wid_control b_widget.w_control -#define b_wid_req_timeout b_widget.w_req_timeout -#define b_wid_int_upper b_widget.w_intdest_upper_addr -#define b_wid_int_lower b_widget.w_intdest_lower_addr -#define b_wid_err_cmdword b_widget.w_err_cmd_word -#define b_wid_llp b_widget.w_llp_cfg -#define b_wid_tflush b_widget.w_tflush - - /* bridge-specific widget configuration 0x000058-0x00007F */ - bridgereg_t _pad_000058; - bridgereg_t b_wid_aux_err; /* 0x00005C */ - bridgereg_t _pad_000060; - bridgereg_t b_wid_resp_upper; /* 0x000064 */ - bridgereg_t _pad_000068; - bridgereg_t b_wid_resp_lower; /* 0x00006C */ - bridgereg_t _pad_000070; - bridgereg_t b_wid_tst_pin_ctrl; /* 0x000074 */ - bridgereg_t _pad_000078[2]; - - /* PMU & Map 0x000080-0x00008F */ - bridgereg_t _pad_000080; - bridgereg_t b_dir_map; /* 0x000084 */ - bridgereg_t _pad_000088[2]; - - /* SSRAM 0x000090-0x00009F */ - bridgereg_t _pad_000090; - bridgereg_t b_ram_perr; /* 0x000094 */ - bridgereg_t _pad_000098[2]; - - /* Arbitration 0x0000A0-0x0000AF */ - bridgereg_t _pad_0000A0; - bridgereg_t b_arb; /* 0x0000A4 */ - bridgereg_t _pad_0000A8[2]; - - /* Number In A Can 0x0000B0-0x0000BF */ - bridgereg_t _pad_0000B0; - bridgereg_t b_nic; /* 0x0000B4 */ - bridgereg_t _pad_0000B8[2]; - - /* PCI/GIO 0x0000C0-0x0000FF */ - bridgereg_t _pad_0000C0; - bridgereg_t b_bus_timeout; /* 0x0000C4 */ -#define b_pci_bus_timeout b_bus_timeout - - bridgereg_t _pad_0000C8; - bridgereg_t b_pci_cfg; /* 0x0000CC */ - bridgereg_t _pad_0000D0; - bridgereg_t b_pci_err_upper; /* 0x0000D4 */ - bridgereg_t _pad_0000D8; - bridgereg_t b_pci_err_lower; /* 0x0000DC */ - bridgereg_t _pad_0000E0[8]; -#define b_gio_err_lower b_pci_err_lower -#define b_gio_err_upper b_pci_err_upper - - /* Interrupt 0x000100-0x0001FF */ - bridgereg_t _pad_000100; - bridgereg_t b_int_status; /* 0x000104 */ - bridgereg_t _pad_000108; - bridgereg_t b_int_enable; /* 0x00010C */ - bridgereg_t _pad_000110; - bridgereg_t b_int_rst_stat; /* 0x000114 */ - bridgereg_t _pad_000118; - bridgereg_t b_int_mode; /* 0x00011C */ - bridgereg_t _pad_000120; - bridgereg_t b_int_device; /* 0x000124 */ - bridgereg_t _pad_000128; - bridgereg_t b_int_host_err; /* 0x00012C */ - - struct { - bridgereg_t __pad; /* 0x0001{30,,,68} */ - bridgereg_t addr; /* 0x0001{34,,,6C} */ - } b_int_addr[8]; /* 0x000130 */ - - bridgereg_t _pad_000170[36]; - - /* Device 0x000200-0x0003FF */ - struct { - bridgereg_t __pad; /* 0x0002{00,,,38} */ - bridgereg_t reg; /* 0x0002{04,,,3C} */ - } b_device[8]; /* 0x000200 */ +/** + * union bridge_pci_cfg_t0 - PCI configuration space type 0. + * @b: byte-array access to the start of the configuration space. + * @w: short/word-array access to the start of the configuration space. + * @l: dword-array access to the start of the configuration space. + * @q: qword-array access to the start of the configuration space. + * @func: nested multi-size union for specific func# access via PCI_FUNC(). + * @fields.bar: PCI base address registers #0 to #5. + * @fields.cardbus_dis: cardbus card information structure. + * @fields.subsys_id: subsystem ID. + * @fields.subsys_vendor: subsystem vendor. + * @fields.x_rom_base: expansion ROM base address. + * @fields.rsvd: reserved fields (0x34, 0x38). + * @fields.max_lat: max latency. + * @fields.min_gnt: minimum grant. + * @fields.int_pin: interrupt pin. + * @fields.int_line: interrupt line. + * @fields.device_specific: device-specific config area (BRIDGE-specific?). + */ +union bridge_pci_cfg_t0 { + u8 b[0x1000 / 1]; + u16 w[0x1000 / 2]; + u32 l[0x1000 / 4]; + u64 q[0x1000 / 8]; + union { + u8 b[0x100 / 1]; + u16 w[0x100 / 2]; + u32 l[0x100 / 4]; + u64 q[0x100 / 8]; + } func[8]; struct { - bridgereg_t __pad; /* 0x0002{40,,,78} */ - bridgereg_t reg; /* 0x0002{44,,,7C} */ - } b_wr_req_buf[8]; /* 0x000240 */ + struct bridge_pci_cfg_cmn common; /* 0x00 */ + u32 bar[6]; /* 0x10 */ + u32 cardbus_cis; /* 0x28 */ + u16 subsys_id; /* 0x2c */ + u16 subsys_vendor; /* 0x2e */ + u32 x_rom_base; /* 0x30 */ + u32 rsvd[2]; /* 0x34 */ + u8 max_lat; /* 0x3c */ + u8 min_gnt; /* 0x3d */ + u8 int_pin; /* 0x3e */ + u8 int_line; /* 0x3f */ + u32 device_specific[48]; /* 0x40 + 0xc0 */ + } fields; +}; +/** + * struct bridge_pci_cfg_t1 - PCI configuration space type 1. + * @b: byte-array access to the start of the configuration space. + * @w: short/word-array access to the start of the configuration space. + * @l: dword-array access to the start of the configuration space. + * @q: qword-array access to the start of the configuration space. + * @fields.bar: PCI base address registers #0 & #1. + * @fields.sec_lat_tmr: secondary latency timer. + * @fields.sub_bus_num: subordinate bus number. + * @fields.sec_bus_num: secondary bus number. + * @fields.pri_bus_num: primary bus number. + * @fields.sec_status: secondary status. + * @fields.io_limit: I/O limit lower 8-bits. + * @fields.io_base: I/O base lower 8-bits. + * @fields.mem_limit: memory limit lower 16-bits. + * @fields.mem_base: memory base lower 16-bits. + * @fields.pf_mem_limit: prefetchable memory limit lower 16-bits. + * @fields.pf_mem_base: prefetchable memory base lower 16-bits. + * @fields.pf_mem_base_upper: prefetchable memory base upper 32-bits. + * @fields.pf_mem_limit_upper: prefetchable memory limit upper 32-bits. + * @fields.io_limit_upper: I/O limit upper 16-bits. + * @fields.io_base_upper: I/O base upper 16-bits. + * @fields.rsvd: reserved field. + * @fields.x_rom_base: expansion ROM base address. + * @fields.bridge_ctrl: PCI bridge control. + * @fields.int_pin: interrupt pin. + * @fields.int_line: interrupt line. + * @fields.device_specific: device-specific config area (BRIDGE-specific?). + */ +union bridge_pci_cfg_t1 { + u8 b[0x1000 / 1]; + u16 w[0x1000 / 2]; + u32 l[0x1000 / 4]; + u64 q[0x1000 / 8]; struct { - bridgereg_t __pad; /* 0x0002{80,,,88} */ - bridgereg_t reg; /* 0x0002{84,,,8C} */ - } b_rrb_map[2]; /* 0x000280 */ -#define b_even_resp b_rrb_map[0].reg /* 0x000284 */ -#define b_odd_resp b_rrb_map[1].reg /* 0x00028C */ - - bridgereg_t _pad_000290; - bridgereg_t b_resp_status; /* 0x000294 */ - bridgereg_t _pad_000298; - bridgereg_t b_resp_clear; /* 0x00029C */ - - bridgereg_t _pad_0002A0[24]; - - char _pad_000300[0x10000 - 0x000300]; + struct bridge_pci_cfg_cmn common; /* 0x00 */ + u32 bar[2]; /* 0x10 */ + u8 sec_lat_tmr; /* 0x18 */ + u8 sub_bus_num; /* 0x19 */ + u8 sec_bus_num; /* 0x1a */ + u8 pri_bus_num; /* 0x1b */ + u16 sec_status; /* 0x1c */ + u8 io_limit; /* 0x1e */ + u8 io_base; /* 0x1f */ + u16 mem_limit; /* 0x20 */ + u16 mem_base; /* 0x22 */ + u16 pf_mem_limit; /* 0x24 */ + u16 pf_mem_base; /* 0x26 */ + u32 pf_mem_base_upper; /* 0x28 */ + u32 pf_mem_limit_upper; /* 0x2c */ + u16 io_limit_upper; /* 0x30 */ + u16 io_base_upper; /* 0x32 */ + u32 rsvd; /* 0x34 */ + u32 x_rom_base; /* 0x38 */ + u16 bridge_ctrl; /* 0x3c */ + u8 int_pin; /* 0x3e */ + u8 int_line; /* 0x3f */ + u32 device_specific[48]; /* 0x40 + 0xc0 */ + } fields; +}; - /* Internal Address Translation Entry RAM 0x010000-0x0103FF */ - union { - bridge_ate_t wr; /* write-only */ - struct { - bridgereg_t _p_pad; - bridgereg_t rd; /* read-only */ - } hi; - } b_int_ate_ram[128]; +/** + * union bridge_devio - multi-size access to BRIDGE/XBRIDGE DevIO windows. + * @b: u8 access. + * @w: u16 access. + * @l: u32 access. + * @q: u64 access. + */ +union bridge_devio { + u8 b[0x100000 / 1]; + u16 w[0x100000 / 2]; + u32 l[0x100000 / 4]; + u64 q[0x100000 / 8]; +}; - char _pad_010400[0x11000 - 0x010400]; +/** + * union bridge_flash_prom - multi-size access to BRIDGE/XBRIDGE flash PROMs. + * @b: u8 access. + * @w: u16 access. + * @l: u32 access. + * @q: u64 access. + * + * Be careful when messing around with the flash PROM. + */ +union bridge_flash_prom { + u8 b[0x400000 / 1]; /* ro */ + u16 w[0x400000 / 2]; /* rw */ + u32 l[0x400000 / 4]; /* ro */ + u64 q[0x400000 / 8]; /* ro */ +}; - /* Internal Address Translation Entry RAM LOW 0x011000-0x0113FF */ - struct { - bridgereg_t _p_pad; - bridgereg_t rd; /* read-only */ - } b_int_ate_ram_lo[128]; - - char _pad_011400[0x20000 - 0x011400]; - - /* PCI Device Configuration Spaces 0x020000-0x027FFF */ - union { /* make all access sizes available. */ - u8 c[0x1000 / 1]; - u16 s[0x1000 / 2]; - u32 l[0x1000 / 4]; - u64 d[0x1000 / 8]; - union { - u8 c[0x100 / 1]; - u16 s[0x100 / 2]; - u32 l[0x100 / 4]; - u64 d[0x100 / 8]; - } f[8]; - } b_type0_cfg_dev[8]; /* 0x020000 */ - - /* PCI Type 1 Configuration Space 0x028000-0x028FFF */ - union { /* make all access sizes available. */ - u8 c[0x1000 / 1]; - u16 s[0x1000 / 2]; - u32 l[0x1000 / 4]; - u64 d[0x1000 / 8]; - } b_type1_cfg; /* 0x028000-0x029000 */ - - char _pad_029000[0x007000]; /* 0x029000-0x030000 */ - - /* PCI Interrupt Acknowledge Cycle 0x030000 */ - union { - u8 c[8 / 1]; - u16 s[8 / 2]; - u32 l[8 / 4]; - u64 d[8 / 8]; - } b_pci_iack; /* 0x030000 */ - - u8 _pad_030007[0x04fff8]; /* 0x030008-0x07FFFF */ - - /* External Address Translation Entry RAM 0x080000-0x0FFFFF */ - bridge_ate_t b_ext_ate_ram[0x10000]; - - /* Reserved 0x100000-0x1FFFFF */ - char _pad_100000[0x200000-0x100000]; - - /* PCI/GIO Device Spaces 0x200000-0xBFFFFF */ - union { /* make all access sizes available. */ - u8 c[0x100000 / 1]; - u16 s[0x100000 / 2]; - u32 l[0x100000 / 4]; - u64 d[0x100000 / 8]; - } b_devio_raw[10]; /* 0x200000 */ - - /* b_devio macro is a bit strange; it reflects the - * fact that the Bridge ASIC provides 2M for the - * first two DevIO windows and 1M for the other six. - */ -#define b_devio(n) b_devio_raw[((n)<2)?(n*2):(n+2)] - - /* External Flash Proms 1,0 0xC00000-0xFFFFFF */ - union { /* make all access sizes available. */ - u8 c[0x400000 / 1]; /* read-only */ - u16 s[0x400000 / 2]; /* read-write */ - u32 l[0x400000 / 4]; /* read-only */ - u64 d[0x400000 / 8]; /* read-only */ - } b_external_flash; /* 0xC00000 */ -} bridge_t; +/** + * struct bridge_widget - a BRIDGE widget on a Crosstalk bus. + * @regs: struct bridge_registers access to BRIDGE/XBRIDGE registers. + * @ate_ram: access to upper 32-bits of ATE RAM area. + * @__pad0: 0x000c00 bytes of padding (0x000180 * 8). + * @ate_ram_lo: access to lower 32-bits of ATE RAM area. + * @__pad1: 0x00ec00 bytes of padding (0x001d80 * 8). + * @pci_t0: access to type 0 PCI configuration space for each PCI slot. + * @pci_t1: access to type 1 PCI configuration space. + * @__pad2: 0x007000 bytes of padding (0x000e00 * 8). + * @pci_iack: PCI interrupt ack cycle field/register. + * @__pad3: 0x04fff8 bytes of padding (0x009fff * 8). + * @ext_ate_ram: access to any external ATE RAM (don't use; known HW bugs). + * @__pad4: 0x100000 bytes of padding (0x020000 * 8). + * @devio_raw: access to DevIO windows. 2MB windows to #0 & #1; else, 1MB. + * @ext_flash: access to flash PROM space. BE CAREFUL! + */ +struct bridge_widget { + /* BRIDGE/XBRIDGE registers */ + struct bridge_registers regs; /* 0x000000 - 0x00ffff */ + /* PMU Address Translation Entries, HIGH */ + union bridge_ate ate_ram[128]; /* 0x010000 - 0x0103ff */ + u64 __pad0[0x000180]; /* 0x010400 - 0x010fff */ + /* PMU Address Translation Entries, LOW */ + union bridge_ate ate_ram_lo[128]; /* 0x011000 - 0x0113ff */ + u64 __pad1[0x001d80]; /* 0x011400 - 0x01ffff */ + /* PCI configuration space type 0 */ + union bridge_pci_cfg_t0 pci_t0[8]; /* 0x020000 - 0x027fff */ + /* PCI configuration space type 1 */ + union bridge_pci_cfg_t1 pci_t1; /* 0x028000 - 0x028fff */ + u64 __pad2[0x000e00]; /* 0x029000 - 0x02ffff */ + /* PCI Interrupt Acknowledge Cycle */ + union bridge_reg pci_iack; /* 0x030000 - 0x030007 */ + u64 __pad3[0x009fff]; /* 0x030008 - 0x07ffff */ + /* External Address Translation Entry RAM */ + u64 ext_ate_ram[0x10000]; /* 0x080000 - 0x0fffff */ + u64 __pad4[0x020000]; /* 0x100000 - 0x1fffff */ + /* PCI/GIO DevIO windows */ + union bridge_devio devio_raw[10]; /* 0x200000 - 0xbfffff */ + /* External flash PROMs 0 & 1 */ + union bridge_flash_prom ext_flash; /* 0xc00000 - 0xffffff */ +}; /* - * Field formats for Error Command Word and Auxiliary Error Command Word - * of bridge. + * The b_devio() macro is a bit strange, because it reflects the fact that + * the BRIDGE/XBRIDGE ASIC provides 2MB for the first two DevIO windows and + * 1MB for the remaining six DevIO windows. */ -typedef struct bridge_err_cmdword_s { - union { - u32 cmd_word; - struct { - u32 didn:4, /* Destination ID */ - sidn:4, /* Source ID */ - pactyp:4, /* Packet type */ - tnum:5, /* Trans Number */ - coh:1, /* Coh Transaction */ - ds:2, /* Data size */ - gbr:1, /* GBR enable */ - vbpm:1, /* VBPM message */ - error:1, /* Error occurred */ - barr:1, /* Barrier op */ - rsvd:8; - } berr_st; - } berr_un; -} bridge_err_cmdword_t; +#define b_devio(_x) devio_raw[((_x) < 2) ? ((_x) * 2) : ((_x) + 2)].l +#define b_devio_0 b_devio(0) +#define b_devio_1 b_devio(1) +#define b_devio_2 b_devio(2) +#define b_devio_3 b_devio(3) +#define b_devio_4 b_devio(4) +#define b_devio_5 b_devio(5) +#define b_devio_6 b_devio(6) +#define b_devio_7 b_devio(7) + +/* Standard 32-bit access to the flash PROMs - BE CAREFUL! */ +#define b_ext_flash ext_flash.l -#define berr_field berr_un.berr_st #endif /* !__ASSEMBLY__ */ /* - * The values of these macros can and should be crosschecked - * regularly against the offsets of the like-named fields - * within the "bridge_t" structure above. + * Shortcut accessor macros for reading/writing BRIDGE/XBRIDGE bits. + * @_bc: pointer to a 'struct bridge_controller' reference. + * @_r: register or field name from one of the many macros defined above. + * @_v: for writes, the value to write back to the register. */ +#define bridge_read_reg(_bc, _r) \ + __raw_readl(&_bc->bridge->regs._r) +#define bridge_write_reg(_v, _bc, _r) \ + __raw_writel(_v, &_bc->bridge->regs._r) +#define bridge_read(_bc, _r) \ + __raw_readl(_bc->bridge->_r) +#define bridge_write(_v, _bc, _r) \ + __raw_writel(_v, _bc->bridge->_r) -/* Byte offset macros for Bridge internal registers */ - -#define BRIDGE_WID_ID WIDGET_ID -#define BRIDGE_WID_STAT WIDGET_STATUS -#define BRIDGE_WID_ERR_UPPER WIDGET_ERR_UPPER_ADDR -#define BRIDGE_WID_ERR_LOWER WIDGET_ERR_LOWER_ADDR -#define BRIDGE_WID_CONTROL WIDGET_CONTROL -#define BRIDGE_WID_REQ_TIMEOUT WIDGET_REQ_TIMEOUT -#define BRIDGE_WID_INT_UPPER WIDGET_INTDEST_UPPER_ADDR -#define BRIDGE_WID_INT_LOWER WIDGET_INTDEST_LOWER_ADDR -#define BRIDGE_WID_ERR_CMDWORD WIDGET_ERR_CMD_WORD -#define BRIDGE_WID_LLP WIDGET_LLP_CFG -#define BRIDGE_WID_TFLUSH WIDGET_TFLUSH - -#define BRIDGE_WID_AUX_ERR 0x00005C /* Aux Error Command Word */ -#define BRIDGE_WID_RESP_UPPER 0x000064 /* Response Buf Upper Addr */ -#define BRIDGE_WID_RESP_LOWER 0x00006C /* Response Buf Lower Addr */ -#define BRIDGE_WID_TST_PIN_CTRL 0x000074 /* Test pin control */ - -#define BRIDGE_DIR_MAP 0x000084 /* Direct Map reg */ +/* ----------------------------------------------------------------------- */ -#define BRIDGE_RAM_PERR 0x000094 /* SSRAM Parity Error */ -#define BRIDGE_ARB 0x0000A4 /* Arbitration Priority reg */ - -#define BRIDGE_NIC 0x0000B4 /* Number In A Can */ - -#define BRIDGE_BUS_TIMEOUT 0x0000C4 /* Bus Timeout Register */ -#define BRIDGE_PCI_BUS_TIMEOUT BRIDGE_BUS_TIMEOUT -#define BRIDGE_PCI_CFG 0x0000CC /* PCI Type 1 Config reg */ -#define BRIDGE_PCI_ERR_UPPER 0x0000D4 /* PCI error Upper Addr */ -#define BRIDGE_PCI_ERR_LOWER 0x0000DC /* PCI error Lower Addr */ - -#define BRIDGE_INT_STATUS 0x000104 /* Interrupt Status */ -#define BRIDGE_INT_ENABLE 0x00010C /* Interrupt Enables */ -#define BRIDGE_INT_RST_STAT 0x000114 /* Reset Intr Status */ -#define BRIDGE_INT_MODE 0x00011C /* Interrupt Mode */ -#define BRIDGE_INT_DEVICE 0x000124 /* Interrupt Device */ -#define BRIDGE_INT_HOST_ERR 0x00012C /* Host Error Field */ - -#define BRIDGE_INT_ADDR0 0x000134 /* Host Address Reg */ -#define BRIDGE_INT_ADDR_OFF 0x000008 /* Host Addr offset (1..7) */ -#define BRIDGE_INT_ADDR(x) (BRIDGE_INT_ADDR0+(x)*BRIDGE_INT_ADDR_OFF) - -#define BRIDGE_DEVICE0 0x000204 /* Device 0 */ -#define BRIDGE_DEVICE_OFF 0x000008 /* Device offset (1..7) */ -#define BRIDGE_DEVICE(x) (BRIDGE_DEVICE0+(x)*BRIDGE_DEVICE_OFF) - -#define BRIDGE_WR_REQ_BUF0 0x000244 /* Write Request Buffer 0 */ -#define BRIDGE_WR_REQ_BUF_OFF 0x000008 /* Buffer Offset (1..7) */ -#define BRIDGE_WR_REQ_BUF(x) (BRIDGE_WR_REQ_BUF0+(x)*BRIDGE_WR_REQ_BUF_OFF) - -#define BRIDGE_EVEN_RESP 0x000284 /* Even Device Response Buf */ -#define BRIDGE_ODD_RESP 0x00028C /* Odd Device Response Buf */ - -#define BRIDGE_RESP_STATUS 0x000294 /* Read Response Status reg */ -#define BRIDGE_RESP_CLEAR 0x00029C /* Read Response Clear reg */ - -/* Byte offset macros for Bridge I/O space */ - -#define BRIDGE_ATE_RAM 0x00010000 /* Internal Addr Xlat Ram */ - -#define BRIDGE_TYPE0_CFG_DEV0 0x00020000 /* Type 0 Cfg, Device 0 */ -#define BRIDGE_TYPE0_CFG_SLOT_OFF 0x00001000 /* Type 0 Cfg Slot Offset (1..7) */ -#define BRIDGE_TYPE0_CFG_FUNC_OFF 0x00000100 /* Type 0 Cfg Func Offset (1..7) */ -#define BRIDGE_TYPE0_CFG_DEV(s) (BRIDGE_TYPE0_CFG_DEV0+\ - (s)*BRIDGE_TYPE0_CFG_SLOT_OFF) -#define BRIDGE_TYPE0_CFG_DEVF(s, f) (BRIDGE_TYPE0_CFG_DEV0+\ - (s)*BRIDGE_TYPE0_CFG_SLOT_OFF+\ - (f)*BRIDGE_TYPE0_CFG_FUNC_OFF) - -#define BRIDGE_TYPE1_CFG 0x00028000 /* Type 1 Cfg space */ - -#define BRIDGE_PCI_IACK 0x00030000 /* PCI Interrupt Ack */ -#define BRIDGE_EXT_SSRAM 0x00080000 /* Extern SSRAM (ATE) */ - -/* Byte offset macros for Bridge device IO spaces */ - -#define BRIDGE_DEV_CNT 8 /* Up to 8 devices per bridge */ -#define BRIDGE_DEVIO0 0x00200000 /* Device IO 0 Addr */ -#define BRIDGE_DEVIO1 0x00400000 /* Device IO 1 Addr */ -#define BRIDGE_DEVIO2 0x00600000 /* Device IO 2 Addr */ -#define BRIDGE_DEVIO_OFF 0x00100000 /* Device IO Offset (3..7) */ - -#define BRIDGE_DEVIO_2MB 0x00200000 /* Device IO Offset (0..1) */ -#define BRIDGE_DEVIO_1MB 0x00100000 /* Device IO Offset (2..7) */ - -#define BRIDGE_DEVIO(x) ((x)<=1 ? BRIDGE_DEVIO0+(x)*BRIDGE_DEVIO_2MB : BRIDGE_DEVIO2+((x)-2)*BRIDGE_DEVIO_1MB) - -#define BRIDGE_EXTERNAL_FLASH 0x00C00000 /* External Flash PROMS */ - -/* ======================================================================== - * Bridge register bit field definitions - */ +/* ----------------------------------------------------------------------- */ +/* Various BRIDGE/XBRIDGE definitions/params/macros. */ /* Widget part numbers. */ #define BRIDGE_WIDGET_PART_NUM 0xc002 @@ -723,30 +806,51 @@ typedef struct bridge_err_cmdword_s { /* This should be written to the XBOW's link_control(x) register */ #define BRIDGE_CREDIT 3 -/* RRB assignment register -- value applies after shifting down */ -#define BRIDGE_RRB_EN 0x8 -#define BRIDGE_RRB_DEV 0x7 -#define BRIDGE_RRB_VDEV 0x4 -#define BRIDGE_RRB_PDEV 0x3 - -/* RRB status register */ -#define BRIDGE_RRB_VALID(r) (0x00010000 << (r)) -#define BRIDGE_RRB_INUSE(r) (0x00000001 << (r)) - -/* RRB clear register */ -#define BRIDGE_RRB_CLEAR(r) (0x00000001 << (r)) - /* Xbox system controller declarations */ #define XBOX_BRIDGE_WID 8 #define FLASH_PROM1_BASE 0xe00000 /* Read Xbox sysctlr stat */ #define XBOX_RPS_EXISTS BIT(6) /* RPS bit in status reg */ #define XBOX_RPS_FAIL BIT(4) /* RPS status bit in reg */ -/* ======================================================================== +#ifndef __ASSEMBLY__ +/* + * XXX: Convert below struct into bitfield macros. Or do away with + * entirely. Unused in Linux, but useful documentation. */ /* - * Macros for Xtalk to Bridge bus (PCI/GIO) PIO - * refer to section 4.2.1 of Bridge Spec for xtalk to PCI/GIO PIO mappings + * Field formats for Error Command Word and Auxiliary Error Command Word + * of bridge. + */ +struct bridge_err_cmdword_s { + union { + u32 cmd_word; + struct { + u32 didn:4, /* Destination ID */ + sidn:4, /* Source ID */ + pactyp:4, /* Packet type */ + tnum:5, /* Trans Number */ + coh:1, /* Coh Transaction */ + ds:2, /* Data size */ + gbr:1, /* GBR enable */ + vbpm:1, /* VBPM message */ + error:1, /* Error occurred */ + barr:1, /* Barrier op */ + rsvd:8; + } berr_st; + } berr_un; +}; + +#define berr_field berr_un.berr_st +#endif /* !__ASSEMBLY__ */ + +/* ----------------------------------------------------------------------- */ + + +/* ----------------------------------------------------------------------- */ + +/* + * Macros for Xtalk to BRIDGE bus (PCI/GIO) PIO + * refer to section 4.2.1 of the BRIDGE Spec for xtalk to PCI/GIO PIO mappings */ /* XTALK addresses that map into Bridge Bus addr space */ #define BRIDGE_PIO32_XTALK_ALIAS_BASE 0x000040000000UL @@ -757,7 +861,7 @@ typedef struct bridge_err_cmdword_s { #define BRIDGE_PCIIO_XTALK_ALIAS_LIMIT 0x0001ffffffffUL /* Ranges of PCI bus space that can be accessed via PIO from xtalk */ -#define BRIDGE_MIN_PIO_ADDR_MEM 0x00000000 /* 1G PCI memory space */ +#define BRIDGE_MIN_PIO_ADDR_MEM 0x00000000 /* 1G PCI mem space */ #define BRIDGE_MAX_PIO_ADDR_MEM 0x3fffffff #define BRIDGE_MIN_PIO_ADDR_IO 0x00000000 /* 4G PCI IO space */ #define BRIDGE_MAX_PIO_ADDR_IO 0xffffffff @@ -829,10 +933,10 @@ typedef struct bridge_err_cmdword_s { /* 64-bit address attribute masks */ #define PCI64_ATTR_TARG_MASK GENMASK_ULL(63, 60) #define PCI64_ATTR_TARG_SHFT 60 -#define PCI64_ATTR_PREF BIT_ULL(59) -#define PCI64_ATTR_PREC BIT_ULL(58) -#define PCI64_ATTR_VIRTUAL BIT_ULL(57) -#define PCI64_ATTR_BAR BIT_ULL(56) +#define PCI64_ATTR_PREF BIT_ULL(59) /* Prefetch */ +#define PCI64_ATTR_PREC BIT_ULL(58) /* Precise */ +#define PCI64_ATTR_VIRTUAL BIT_ULL(57) /* Virtual Request */ +#define PCI64_ATTR_BAR BIT_ULL(56) /* Barrier */ #define PCI64_ATTR_RMF_MASK GENMASK_ULL(55, 48) #define PCI64_ATTR_RMF_SHFT 48 @@ -876,20 +980,74 @@ struct bridge_controller { struct resource mem; struct resource io; struct resource busn; - bridge_t *base; - nasid_t nasid; + struct bridge_widget __iomem *bridge; + s16 nasid; u32 widget_id; - u32 irq_cpu; u64 baddr; u32 pci_int[8]; + u32 irq_cpu; + int (*alloc_irq)(struct pci_dev *); }; #define BRIDGE_CONTROLLER(bus) \ ((struct bridge_controller *)((bus)->sysdata)) -extern void register_bridge_irq(unsigned int irq); -extern int request_bridge_irq(struct bridge_controller *bc); - extern struct pci_ops bridge_pci_ops; +/* + * Device might live on a subordinate PCI bus. Walk up the chain of buses + * to find the slot number in sense of the bridge device register. + * XXX: This also means multiple devices might rely on conflicting bridge + * settings. + */ +static inline struct pci_dev * +bridge_root_dev(struct pci_dev *dev) +{ + /* Move up the chain of bridges. */ + while (dev->bus->parent) + dev = dev->bus->self; + + return dev; +} + +/* + * Simple macro to flush all pending BRIDGE PIO ops. + * + * XXX: IA64 actually checks the return value of the b_wid_targ_flush register + * and panics if it returns a non-zero status. + */ +#define BRIDGE_FLUSH(_bc) \ + do { \ + while (bridge_read_reg(_bc, b_wid_targ_flush)) \ + cpu_relax(); \ + } while (0) + +/* + * Magic/cosmic value for BRIDGE's INT_DEV register. This sets bits 31:24 + * to all 1's. The only odd thing about this is those bits are marked as + * "reserved" in both the BRIDGE and XBRIDGE docs. Apparently, IRIX does + * this as well. + */ +#define BRIDGE_COSMIC_INT_DEV 0xff000000 + +/** + * struct bridge_platform_data - BRIDGE-specific data for IP27/IP30/IP35. + * @xio_target_addr: HUB/HEART/BEDROCK address in Crosstalk space. + * @baseio_widget_id: crosstalk widget ID for the BaseIO BRIDGE. + * @iomem_swap: bool that enables mem and I/O byteswapping on BRIDGE. + * @add_512: bool that if set, adds BRIDGE_DIRMAP_ADD512 to bridge->b_dir_map. + * @setup_baseio_rrbs: platform function to config the Read Response Buffers. + * @alloc_irq: platform function used by BRIDGE to allocate an IRQ. + * @pre_enable: platform function to setup BRIDGE before probing it. + */ +struct bridge_platform_data { + u32 xio_target_addr; + s8 baseio_widget_id; + bool iomem_swap; + bool add_512; + void (*setup_baseio_rrbs)(struct bridge_controller *, const bool *); + int (*alloc_irq)(struct pci_dev *); + int (*pre_enable)(struct pci_controller *, struct pci_dev *, int); +}; + #endif /* _ASM_PCI_BRIDGE_H */ -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 07/12] MIPS: BRIDGE: Add XBRIDGE revs and SWAP bit 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard ` (5 preceding siblings ...) 2017-02-07 6:13 ` [PATCH 06/12] MIPS: BRIDGE: Overhaul " Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 08/12] MIPS: BRIDGE: Update ops-bridge.c Joshua Kinard ` (4 subsequent siblings) 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS From: Joshua Kinard <kumba@gentoo.org> Add macros for the two known XBRIDGE revisions commonly found on IP35 hardware. Both macros are sourced from the IA64 copy of bridge.h from Linux-2.4.18. Additionally, per the XBRIDGE ASIC specification, bit 55 of the Crosstalk address selects whether to use byte-swapping or not on the XBRIDGE. Signed-off-by: Joshua Kinard <kumba@gentoo.org> --- arch/mips/include/asm/pci/bridge.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/mips/include/asm/pci/bridge.h b/arch/mips/include/asm/pci/bridge.h index 3c214feab772..f173875b4233 100644 --- a/arch/mips/include/asm/pci/bridge.h +++ b/arch/mips/include/asm/pci/bridge.h @@ -484,6 +484,8 @@ struct bridge_widget { #define BRIDGE_REV_B 0x2 #define BRIDGE_REV_C 0x3 #define BRIDGE_REV_D 0x4 +#define XBRIDGE_REV_A 0x1 +#define XBRIDGE_REV_B 0x2 /* Bridge widget status register bits definition */ #define BRIDGE_STAT_LLP_REC_CNT GENMASK(31, 24) @@ -937,6 +939,7 @@ struct bridge_err_cmdword_s { #define PCI64_ATTR_PREC BIT_ULL(58) /* Precise */ #define PCI64_ATTR_VIRTUAL BIT_ULL(57) /* Virtual Request */ #define PCI64_ATTR_BAR BIT_ULL(56) /* Barrier */ +#define PCI64_ATTR_SWAP BIT_ULL(55) /* Byte swap (XBRIDGE) */ #define PCI64_ATTR_RMF_MASK GENMASK_ULL(55, 48) #define PCI64_ATTR_RMF_SHFT 48 -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 08/12] MIPS: BRIDGE: Update ops-bridge.c 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard ` (6 preceding siblings ...) 2017-02-07 6:13 ` [PATCH 07/12] MIPS: BRIDGE: Add XBRIDGE revs and SWAP bit Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 09/12] MIPS: BRIDGE: Use !pci_is_root_bus(bus) to check for bus->number > 0 Joshua Kinard ` (3 subsequent siblings) 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS From: Joshua Kinard <kumba@gentoo.org> Update arch/mips/pci/ops-bridge.c to use the new access structs added to bridge.h in the BRIDGE overhaul patch. Additional changes include various fixes to comments, whitespace, and marking the 'bridge_pci_ops' struct as __read_mostly. Signed-off-by: Joshua Kinard <kumba@gentoo.org> --- arch/mips/pci/ops-bridge.c | 125 ++++++++++++++++++----------------- 1 file changed, 65 insertions(+), 60 deletions(-) diff --git a/arch/mips/pci/ops-bridge.c b/arch/mips/pci/ops-bridge.c index 57e1463fcd02..6793a72edec2 100644 --- a/arch/mips/pci/ops-bridge.c +++ b/arch/mips/pci/ops-bridge.c @@ -8,16 +8,20 @@ */ #include <linux/pci.h> #include <asm/paccess.h> + +#if defined(CONFIG_SGI_IP27) #include <asm/pci/bridge.h> #include <asm/sn/arch.h> #include <asm/sn/intr.h> #include <asm/sn/sn0/hub.h> +#endif /* * Most of the IOC3 PCI config register aren't present * we emulate what is needed for a normal PCI enumeration */ -static u32 emulate_ioc3_cfg(int where, int size) +static inline u32 +emulate_ioc3_cfg(int where, int size) { if (size == 1 && where == 0x3d) return 0x01; @@ -40,29 +44,29 @@ static u32 emulate_ioc3_cfg(int where, int size) * accesses and does only decode parts of it's address space. */ -static int pci_conf0_read_config(struct pci_bus *bus, unsigned int devfn, - int where, int size, u32 * value) +static int +pci_conf0_read_config(struct pci_bus *bus, u32 devfn, int where, int size, + u32 *value) { struct bridge_controller *bc = BRIDGE_CONTROLLER(bus); - bridge_t *bridge = bc->base; int slot = PCI_SLOT(devfn); int fn = PCI_FUNC(devfn); volatile void *addr; u32 cf, shift, mask; int res; - addr = &bridge->b_type0_cfg_dev[slot].f[fn].c[PCI_VENDOR_ID]; + addr = &bc->bridge->pci_t0[slot].func[fn].b[PCI_VENDOR_ID]; if (get_dbe(cf, (u32 *) addr)) return PCIBIOS_DEVICE_NOT_FOUND; /* - * IOC3 is fucking fucked beyond belief ... Don't even give the - * generic PCI code a chance to look at it for real ... + * IOC3 is fucking fucked beyond belief. Don't even give the + * generic PCI code a chance to look at it for real. */ if (cf == (PCI_VENDOR_ID_SGI | (PCI_DEVICE_ID_SGI_IOC3 << 16))) goto oh_my_gawd; - addr = &bridge->b_type0_cfg_dev[slot].f[fn].c[where ^ (4 - size)]; + addr = &bc->bridge->pci_t0[slot].func[fn].b[where ^ (4 - size)]; if (size == 1) res = get_dbe(*value, (u8 *) addr); @@ -74,9 +78,8 @@ static int pci_conf0_read_config(struct pci_bus *bus, unsigned int devfn, return res ? PCIBIOS_DEVICE_NOT_FOUND : PCIBIOS_SUCCESSFUL; oh_my_gawd: - /* - * IOC3 is fucking fucked beyond belief ... Don't even give the + * IOC3 is fucking fucked beyond belief. Don't even give the * generic PCI code a chance to look at the wrong register. */ if ((where >= 0x14 && where < 0x40) || (where >= 0x48)) { @@ -85,10 +88,10 @@ static int pci_conf0_read_config(struct pci_bus *bus, unsigned int devfn, } /* - * IOC3 is fucking fucked beyond belief ... Don't try to access - * anything but 32-bit words ... + * IOC3 is fucking fucked beyond belief. Don't try to access + * anything but 32-bit words. */ - addr = &bridge->b_type0_cfg_dev[slot].f[fn].l[where >> 2]; + addr = &bc->bridge->pci_t0[slot].func[fn].l[where >> 2]; if (get_dbe(cf, (u32 *) addr)) return PCIBIOS_DEVICE_NOT_FOUND; @@ -100,11 +103,11 @@ static int pci_conf0_read_config(struct pci_bus *bus, unsigned int devfn, return PCIBIOS_SUCCESSFUL; } -static int pci_conf1_read_config(struct pci_bus *bus, unsigned int devfn, - int where, int size, u32 * value) +static int +pci_conf1_read_config(struct pci_bus *bus, u32 devfn, int where, int size, + u32 *value) { struct bridge_controller *bc = BRIDGE_CONTROLLER(bus); - bridge_t *bridge = bc->base; int busno = bus->number; int slot = PCI_SLOT(devfn); int fn = PCI_FUNC(devfn); @@ -112,20 +115,20 @@ static int pci_conf1_read_config(struct pci_bus *bus, unsigned int devfn, u32 cf, shift, mask; int res; - bridge->b_pci_cfg = (busno << 16) | (slot << 11); - addr = &bridge->b_type1_cfg.c[(fn << 8) | PCI_VENDOR_ID]; + bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg); + addr = &bc->bridge->pci_t1.b[(fn << 8) | PCI_VENDOR_ID]; if (get_dbe(cf, (u32 *) addr)) return PCIBIOS_DEVICE_NOT_FOUND; /* - * IOC3 is fucking fucked beyond belief ... Don't even give the - * generic PCI code a chance to look at it for real ... + * IOC3 is fucking fucked beyond belief. Don't even give the + * generic PCI code a chance to look at it for real. */ if (cf == (PCI_VENDOR_ID_SGI | (PCI_DEVICE_ID_SGI_IOC3 << 16))) goto oh_my_gawd; - bridge->b_pci_cfg = (busno << 16) | (slot << 11); - addr = &bridge->b_type1_cfg.c[(fn << 8) | (where ^ (4 - size))]; + bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg); + addr = &bc->bridge->pci_t1.b[(fn << 8) | (where ^ (4 - size))]; if (size == 1) res = get_dbe(*value, (u8 *) addr); @@ -137,9 +140,8 @@ static int pci_conf1_read_config(struct pci_bus *bus, unsigned int devfn, return res ? PCIBIOS_DEVICE_NOT_FOUND : PCIBIOS_SUCCESSFUL; oh_my_gawd: - /* - * IOC3 is fucking fucked beyond belief ... Don't even give the + * IOC3 is fucking fucked beyond belief. Don't even give the * generic PCI code a chance to look at the wrong register. */ if ((where >= 0x14 && where < 0x40) || (where >= 0x48)) { @@ -148,11 +150,11 @@ static int pci_conf1_read_config(struct pci_bus *bus, unsigned int devfn, } /* - * IOC3 is fucking fucked beyond belief ... Don't try to access - * anything but 32-bit words ... + * IOC3 is fucking fucked beyond belief. Don't try to access + * anything but 32-bit words. */ - bridge->b_pci_cfg = (busno << 16) | (slot << 11); - addr = &bridge->b_type1_cfg.c[(fn << 8) | where]; + bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg); + addr = &bc->bridge->pci_t1.b[(fn << 8) | where]; if (get_dbe(cf, (u32 *) addr)) return PCIBIOS_DEVICE_NOT_FOUND; @@ -164,8 +166,9 @@ static int pci_conf1_read_config(struct pci_bus *bus, unsigned int devfn, return PCIBIOS_SUCCESSFUL; } -static int pci_read_config(struct pci_bus *bus, unsigned int devfn, - int where, int size, u32 * value) +static int +pci_read_config(struct pci_bus *bus, u32 devfn, int where, int size, + u32 *value) { if (bus->number > 0) return pci_conf1_read_config(bus, devfn, where, size, value); @@ -173,29 +176,29 @@ static int pci_read_config(struct pci_bus *bus, unsigned int devfn, return pci_conf0_read_config(bus, devfn, where, size, value); } -static int pci_conf0_write_config(struct pci_bus *bus, unsigned int devfn, - int where, int size, u32 value) +static int +pci_conf0_write_config(struct pci_bus *bus, u32 devfn, int where, int size, + u32 value) { struct bridge_controller *bc = BRIDGE_CONTROLLER(bus); - bridge_t *bridge = bc->base; int slot = PCI_SLOT(devfn); int fn = PCI_FUNC(devfn); volatile void *addr; u32 cf, shift, mask, smask; int res; - addr = &bridge->b_type0_cfg_dev[slot].f[fn].c[PCI_VENDOR_ID]; + addr = &bc->bridge->pci_t0[slot].func[fn].b[PCI_VENDOR_ID]; if (get_dbe(cf, (u32 *) addr)) return PCIBIOS_DEVICE_NOT_FOUND; /* - * IOC3 is fucking fucked beyond belief ... Don't even give the - * generic PCI code a chance to look at it for real ... + * IOC3 is fucking fucked beyond belief. Don't even give the + * generic PCI code a chance to look at it for real. */ if (cf == (PCI_VENDOR_ID_SGI | (PCI_DEVICE_ID_SGI_IOC3 << 16))) goto oh_my_gawd; - addr = &bridge->b_type0_cfg_dev[slot].f[fn].c[where ^ (4 - size)]; + addr = &bc->bridge->pci_t0[slot].func[fn].b[where ^ (4 - size)]; if (size == 1) { res = put_dbe(value, (u8 *) addr); @@ -211,19 +214,18 @@ static int pci_conf0_write_config(struct pci_bus *bus, unsigned int devfn, return PCIBIOS_SUCCESSFUL; oh_my_gawd: - /* - * IOC3 is fucking fucked beyond belief ... Don't even give the - * generic PCI code a chance to touch the wrong register. + * IOC3 is fucking fucked beyond belief. Don't even give the + * generic PCI code a chance to look at the wrong register. */ if ((where >= 0x14 && where < 0x40) || (where >= 0x48)) return PCIBIOS_SUCCESSFUL; /* - * IOC3 is fucking fucked beyond belief ... Don't try to access - * anything but 32-bit words ... + * IOC3 is fucking fucked beyond belief. Don't try to access + * anything but 32-bit words. */ - addr = &bridge->b_type0_cfg_dev[slot].f[fn].l[where >> 2]; + addr = &bc->bridge->pci_t0[slot].func[fn].l[where >> 2]; if (get_dbe(cf, (u32 *) addr)) return PCIBIOS_DEVICE_NOT_FOUND; @@ -239,11 +241,11 @@ static int pci_conf0_write_config(struct pci_bus *bus, unsigned int devfn, return PCIBIOS_SUCCESSFUL; } -static int pci_conf1_write_config(struct pci_bus *bus, unsigned int devfn, - int where, int size, u32 value) +static int +pci_conf1_write_config(struct pci_bus *bus, u32 devfn, int where, int size, + u32 value) { struct bridge_controller *bc = BRIDGE_CONTROLLER(bus); - bridge_t *bridge = bc->base; int slot = PCI_SLOT(devfn); int fn = PCI_FUNC(devfn); int busno = bus->number; @@ -251,19 +253,20 @@ static int pci_conf1_write_config(struct pci_bus *bus, unsigned int devfn, u32 cf, shift, mask, smask; int res; - bridge->b_pci_cfg = (busno << 16) | (slot << 11); - addr = &bridge->b_type1_cfg.c[(fn << 8) | PCI_VENDOR_ID]; + bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg); + addr = &bc->bridge->pci_t1.b[(fn << 8) | PCI_VENDOR_ID]; if (get_dbe(cf, (u32 *) addr)) return PCIBIOS_DEVICE_NOT_FOUND; /* - * IOC3 is fucking fucked beyond belief ... Don't even give the - * generic PCI code a chance to look at it for real ... + * IOC3 is fucking fucked beyond belief. Don't even give the + * generic PCI code a chance to look at it for real. */ if (cf == (PCI_VENDOR_ID_SGI | (PCI_DEVICE_ID_SGI_IOC3 << 16))) goto oh_my_gawd; - addr = &bridge->b_type1_cfg.c[(fn << 8) | (where ^ (4 - size))]; + bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg); + addr = &bc->bridge->pci_t1.b[(fn << 8) | (where ^ (4 - size))]; if (size == 1) { res = put_dbe(value, (u8 *) addr); @@ -279,19 +282,19 @@ static int pci_conf1_write_config(struct pci_bus *bus, unsigned int devfn, return PCIBIOS_SUCCESSFUL; oh_my_gawd: - /* - * IOC3 is fucking fucked beyond belief ... Don't even give the - * generic PCI code a chance to touch the wrong register. + * IOC3 is fucking fucked beyond belief. Don't even give the + * generic PCI code a chance to look at the wrong register. */ if ((where >= 0x14 && where < 0x40) || (where >= 0x48)) return PCIBIOS_SUCCESSFUL; /* - * IOC3 is fucking fucked beyond belief ... Don't try to access - * anything but 32-bit words ... + * IOC3 is fucking fucked beyond belief. Don't try to access + * anything but 32-bit words. */ - addr = &bridge->b_type0_cfg_dev[slot].f[fn].l[where >> 2]; + bridge_write_reg(((busno << 16) | (slot << 11)), bc, b_pci_type1_cfg); + addr = &bc->bridge->pci_t1.b[(fn << 8) | where]; if (get_dbe(cf, (u32 *) addr)) return PCIBIOS_DEVICE_NOT_FOUND; @@ -307,8 +310,9 @@ static int pci_conf1_write_config(struct pci_bus *bus, unsigned int devfn, return PCIBIOS_SUCCESSFUL; } -static int pci_write_config(struct pci_bus *bus, unsigned int devfn, - int where, int size, u32 value) +static int +pci_write_config(struct pci_bus *bus, u32 devfn, int where, int size, + u32 value) { if (bus->number > 0) return pci_conf1_write_config(bus, devfn, where, size, value); @@ -316,7 +320,8 @@ static int pci_write_config(struct pci_bus *bus, unsigned int devfn, return pci_conf0_write_config(bus, devfn, where, size, value); } -struct pci_ops bridge_pci_ops = { +struct pci_ops __read_mostly +bridge_pci_ops = { .read = pci_read_config, .write = pci_write_config, }; -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 09/12] MIPS: BRIDGE: Use !pci_is_root_bus(bus) to check for bus->number > 0 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard ` (7 preceding siblings ...) 2017-02-07 6:13 ` [PATCH 08/12] MIPS: BRIDGE: Update ops-bridge.c Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 10/12] MIPS: BRIDGE: Overhaul pci-bridge.c driver Joshua Kinard ` (2 subsequent siblings) 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS, Alastair Bridgewater From: Joshua Kinard <kumba@gentoo.org> This is a manual cherrypick of commit c7ddc3d137b7 from Alastair Bridgewater's IP35 tree. In arch/mips/pci/ops-bridge.c, replace: if (bus->number > 0) foo(); With: if (!pci_is_root_bus(bus)) foo(); Signed-off-by: Joshua Kinard <kumba@gentoo.org> Cc: Alastair Bridgewater <alastair.bridgewater@gmail.com> --- arch/mips/pci/ops-bridge.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/mips/pci/ops-bridge.c b/arch/mips/pci/ops-bridge.c index 6793a72edec2..28efce2dcb7d 100644 --- a/arch/mips/pci/ops-bridge.c +++ b/arch/mips/pci/ops-bridge.c @@ -170,7 +170,7 @@ static int pci_read_config(struct pci_bus *bus, u32 devfn, int where, int size, u32 *value) { - if (bus->number > 0) + if (!pci_is_root_bus(bus)) return pci_conf1_read_config(bus, devfn, where, size, value); return pci_conf0_read_config(bus, devfn, where, size, value); @@ -314,7 +314,7 @@ static int pci_write_config(struct pci_bus *bus, u32 devfn, int where, int size, u32 value) { - if (bus->number > 0) + if (!pci_is_root_bus(bus)) return pci_conf1_write_config(bus, devfn, where, size, value); return pci_conf0_write_config(bus, devfn, where, size, value); -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 10/12] MIPS: BRIDGE: Overhaul pci-bridge.c driver 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard ` (8 preceding siblings ...) 2017-02-07 6:13 ` [PATCH 09/12] MIPS: BRIDGE: Use !pci_is_root_bus(bus) to check for bus->number > 0 Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 11/12] MIPS: IP27: Update IRQ code to work with the new BRIDGE code Joshua Kinard 2017-02-07 6:13 ` [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case Joshua Kinard 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS, Stanislaw Skowronek, Johannes Dickgreber From: Joshua Kinard <kumba@gentoo.org> Rewrite the BRIDGE code in pci-bridge.c to address a number of issues: - Partially re-write to support platform_driver in the future - Use the new access structs from the bridge.h overhaul - Handle several BRIDGE WARs found in old IA64 code from 2.4/2.5 - Document what the driver is doing during probe - Update copyrights A lot of the changes are built up over the years from the IP30 patch series as well as other contributions, along with work done on the IP27 code that will be part of a forth-coming patch series. There are several temporary pieces of code left that are needed by the current IP27 code to boot correctly under these changes. They will be removed in the future with the IP27 "mega update" patch series and this driver will become a full platform_driver module. Signed-off-by: Joshua Kinard <kumba@gentoo.org> Cc: Stanislaw Skowronek <skylark@unaligned.org> Cc: Johannes Dickgreber <tanzy@gmx.de> --- arch/mips/pci/pci-bridge.c | 307 +++++++++++++++++++++-------------- 1 file changed, 186 insertions(+), 121 deletions(-) diff --git a/arch/mips/pci/pci-bridge.c b/arch/mips/pci/pci-bridge.c index 0f09eafa5e3a..28b0eb5b844e 100644 --- a/arch/mips/pci/pci-bridge.c +++ b/arch/mips/pci/pci-bridge.c @@ -3,162 +3,236 @@ * License. See the file "COPYING" in the main directory of this archive * for more details. * - * Copyright (C) 2003 Christoph Hellwig (hch@lst.de) - * Copyright (C) 1999, 2000, 04 Ralf Baechle (ralf@linux-mips.org) - * Copyright (C) 1999, 2000 Silicon Graphics, Inc. + * arch/mips/pci/pci-bridge.c + * platform_driver for SGI BRIDGE/XBRIDGE (and in the future, SGI PIC) ASICs. + * + * Originally called pci-ip27.c, which is: + * Copyright (C) 2003 Christoph Hellwig (hch@lst.de) + * Copyright (C) 1999, 2000, 04 Ralf Baechle (ralf@linux-mips.org) + * Copyright (C) 1999, 2000 Silicon Graphics, Inc. + * + * Modifications sourced from pci-ip30.c in the IP30 patchset are: + * Copyright (C) 2004-2007 Stanislaw Skowronek <skylark@unaligned.org> + * Copyright (C) 2009 Johannes Dickgreber <tanzy@gmx.de> + * Copyright (C) 2016 Joshua Kinard <kumba@gentoo.org> + * + * Functions/info/insight sourced from old IA64 code are: + * Copyright (C) 1992 - 1997, 2000-2002 Silicon Graphics, Inc. + * All rights reserved. */ + #include <linux/kernel.h> -#include <linux/export.h> #include <linux/pci.h> -#include <linux/smp.h> -#include <asm/sn/arch.h> -#include <asm/pci/bridge.h> +#include <linux/platform_device.h> + #include <asm/paccess.h> -#include <asm/sn/intr.h> -#include <asm/sn/sn0/hub.h> +#include <asm/pci/bridge.h> +#include <asm/xtalk/xtalk.h> -/* - * Max #PCI busses we can handle; ie, max #PCI bridges. - */ -#define MAX_PCI_BUSSES 40 +#if defined(CONFIG_SGI_IP27) +#include <asm/mach-ip27/pcibr.h> +#else +#error "Unknown CONFIG_SGI_IP??" +#endif -/* - * Max #PCI devices (like scsi controllers) we handle on a bus. - */ -#define MAX_DEVICES_PER_PCIBUS 8 +/* Increments for each additional bridge. */ +static int num_bridges; + +/* XXX: Temporary until the IP27 "mega update". */ /* * XXX: No kmalloc available when we do our crosstalk scan, - * we should try to move it later in the boot process. + * we should try to move it later in the boot process. */ -static struct bridge_controller bridges[MAX_PCI_BUSSES]; +static struct bridge_controller bridges[PCIBR_MAX_NUM_PCIBUS]; -/* - * Translate from irq to software PCI bus number and PCI slot. +/** + * bridge_probe - probes a BRIDGE chip and configures it. + * @nasid: NUMA Address Space Identifier. + * @widget_id: s8 value of this BRIDGE's xtalk widget ID. + * @masterwid: widget ID of HUB. + * + * Always returns '0'. */ -struct bridge_controller *irq_to_bridge[MAX_PCI_BUSSES * MAX_DEVICES_PER_PCIBUS]; -int irq_to_slot[MAX_PCI_BUSSES * MAX_DEVICES_PER_PCIBUS]; - -extern struct pci_ops bridge_pci_ops; - -int bridge_probe(nasid_t nasid, int widget_id, int masterwid) +int +bridge_probe(nasid_t nasid, int widget_id, int masterwid) { + u32 slot, reg; + u32 wid_ctrl; /* BRIDGE WAR */ unsigned long offset = NODE_OFFSET(nasid); struct bridge_controller *bc; - static int num_bridges = 0; - bridge_t *bridge; - int slot; - - pci_set_flags(PCI_PROBE_ONLY); - printk("a bridge\n"); - - /* XXX: kludge alert.. */ + /* XXX: Temporary until the IP27 "mega update". */ + bc = &bridges[num_bridges]; if (!num_bridges) ioport_resource.end = ~0UL; - bc = &bridges[num_bridges]; - - bc->pc.pci_ops = &bridge_pci_ops; - bc->pc.mem_resource = &bc->mem; - bc->pc.io_resource = &bc->io; - - bc->pc.index = num_bridges; + /* Set bridge_controller parameters. */ + bc->pc.pci_ops = &bridge_pci_ops; + bc->pc.mem_resource = &bc->mem; + bc->pc.mem_offset = offset; + bc->pc.io_resource = &bc->io; + bc->pc.io_offset = offset; + bc->pc.busn_resource = &bc->busn; + bc->pc.busn_offset = offset; + bc->pc.index = num_bridges; + bc->pc.io_map_base = NODE_SWIN_BASE(nasid, widget_id); + + bc->mem.name = "Bridge MEM"; + bc->mem.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_MEM); + bc->mem.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO - 1); + bc->mem.flags = IORESOURCE_MEM; + + bc->io.name = "Bridge IO"; + bc->io.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO); + bc->io.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_END - 1); + bc->io.flags = IORESOURCE_IO; + + bc->busn.name = "Bridge BUSN"; + bc->busn.start = num_bridges; + bc->busn.end = 255; + bc->busn.flags = IORESOURCE_BUS; - bc->mem.name = "Bridge PCI MEM"; - bc->pc.mem_offset = offset; - bc->mem.start = 0; - bc->mem.end = ~0UL; - bc->mem.flags = IORESOURCE_MEM; - - bc->io.name = "Bridge IO MEM"; - bc->pc.io_offset = offset; - bc->io.start = 0UL; - bc->io.end = ~0UL; - bc->io.flags = IORESOURCE_IO; - - bc->irq_cpu = smp_processor_id(); bc->widget_id = widget_id; bc->nasid = nasid; + bc->baddr = ((u64)masterwid << PCI64_ATTR_TARG_SHFT); + + /* XXX: Kill */ + bc->irq_cpu = smp_processor_id(); - bc->baddr = (u64)masterwid << 60 | PCI64_ATTR_BAR; + /* + * Point to this bridge + */ + bc->bridge = (struct bridge_widget __iomem *) + RAW_NODE_SWIN_BASE(nasid, widget_id); /* - * point to this bridge + * On BRIDGEs prior to Rev D, set the PCI_RETRY_CNT to zero to avoid + * dropping stores (WAR #475347). + * Sourced from Linux-2.5.70/arch/ia64/sn/io/sn1/pcibr.c */ - bridge = (bridge_t *) RAW_NODE_SWIN_BASE(nasid, widget_id); + if (XWIDGET_REV_NUM(bridge_read_reg(bc, b_wid_id)) < BRIDGE_REV_D) { + reg = bridge_read_reg(bc, b_pci_bus_timo); + bridge_write_reg((reg & ~(BRIDGE_BUS_PCI_RETRY_MASK)), bc, + b_pci_bus_timo); + } /* * Clear all pending interrupts. */ - bridge->b_int_rst_stat = BRIDGE_IRR_ALL_CLR; + bridge_write_reg(BRIDGE_IRR_ALL_CLR, bc, b_int_reset_stat); /* * Until otherwise set up, assume all interrupts are from slot 0 */ - bridge->b_int_device = 0x0; + bridge_write_reg(0, bc, b_int_device); + + /* Configure BRIDGE widget control ... */ + wid_ctrl = bridge_read_reg(bc, b_wid_ctrl); /* - * swap pio's to pci mem and io space (big windows) + * IP27 & IP35 need I/O and Mem swapping enabled. + * IP30 needs it disabled. */ - bridge->b_wid_control |= BRIDGE_CTRL_IO_SWAP | - BRIDGE_CTRL_MEM_SWAP; + wid_ctrl |= (BRIDGE_CTRL_IO_SWAP | BRIDGE_CTRL_MEM_SWAP); + + /* Set the BRIDGE PAGE_SIZE */ #ifdef CONFIG_PAGE_SIZE_4KB - bridge->b_wid_control &= ~BRIDGE_CTRL_PAGE_SIZE; + wid_ctrl &= ~BRIDGE_CTRL_PAGE_SIZE; #else /* 16kB or larger */ - bridge->b_wid_control |= BRIDGE_CTRL_PAGE_SIZE; + wid_ctrl |= BRIDGE_CTRL_PAGE_SIZE; #endif /* - * Hmm... IRIX sets additional bits in the address which - * are documented as reserved in the bridge docs. + * Another BRIDGE WAR, read the BRIDGE widget control register + * back after writing it to avoid an invalid address bug. + */ + bridge_write_reg(wid_ctrl, bc, b_wid_ctrl); + wid_ctrl = bridge_read_reg(bc, b_wid_ctrl); + + /* + * Set per-device properties. + * + * XXX: Setup per-slot configuration at some point. Different devices + * need different properties. */ - bridge->b_wid_int_upper = 0x8000 | (masterwid << 16); - bridge->b_wid_int_lower = 0x01800090; /* PI_INT_PEND_MOD off*/ - bridge->b_dir_map = (masterwid << 20); /* DMA */ - bridge->b_int_enable = 0; + for (slot = 0; slot < BRIDGE_MAX_DEVS; slot++) { + reg = bridge_read_reg(bc, b_device(slot)); + reg &= ~BRIDGE_DEV_PAGE_CHK_DIS; + reg |= (BRIDGE_DEV_ERR_LOCK_EN | BRIDGE_DEV_D64_BITS); - for (slot = 0; slot < 8; slot ++) { - bridge->b_device[slot].reg |= BRIDGE_DEV_SWAP_DIR; + bridge_write_reg(reg, bc, b_device(slot)); bc->pci_int[slot] = -1; } - bridge->b_wid_tflush; /* wait until Bridge PIO complete */ - bc->base = bridge; - - register_pci_controller(&bc->pc); + /* Configure direct-mapped DMA */ + reg = (masterwid << BRIDGE_DIRMAP_W_ID_SHFT); + bridge_write_reg(reg, bc, b_dir_map); + /* + * Route all PCI bridge interrupts to the appropriate ASIC responsible + * for handling IRQs (HUB in IP27, HEART in IP30, BEDROCK in IP35). + * The actual IRQ support and masking is done elsewhere. + */ + /* + * XXX: IRIX sets additional bits (0x8000) in the address which are + * marked as reserved in the BRIDGE docs. + */ + bridge_write_reg(((masterwid << WIDGET_TARGET_ID_SHFT) | 0x8000), bc, + b_wid_int_upper); + bridge_write_reg(PCIBR_XIO_SEES_HUB, bc, b_wid_int_lower); + bridge_write_reg(BRIDGE_COSMIC_INT_DEV, bc, b_int_device); + bridge_write_reg(0, bc, b_int_mode); + reg = bridge_read_reg(bc, b_int_enable); + bridge_write_reg((reg | BRIDGE_ISR_ERRORS), bc, b_int_enable); + + /* Wait until Bridge PIO completes. */ + BRIDGE_FLUSH(bc); + + /* Increment number of discovered BRIDGE/XBRIDGE widgets. */ num_bridges++; + register_pci_controller(&bc->pc); + return 0; } +/** + * bridge_alloc_irq - allocate the next available IRQ for a BRIDGE slot. + * @dev: pointer to struct pci_dev of PCI device info. + * + * Casts dev->bus->sysdata to struct bridge_controller and returns the + * outcome of the platform_data-defined 'alloc_irq' function. + */ +inline int +bridge_alloc_irq(struct pci_dev *dev) +{ + const struct bridge_controller *bc = BRIDGE_CONTROLLER(dev->bus); + + if (bc->alloc_irq) + return bc->alloc_irq(dev); + + return -1; +} + /* * All observed requests have pin == 1. We could have a global here, that * gets incremented and returned every time - unfortunately, pci_map_irq * may be called on the same device over and over, and need to return the * same value. On O2000, pin can be 0 or 1, and PCI slots can be [0..7]. * - * A given PCI device, in general, should be able to intr any of the cpus - * on any one of the hubs connected to its xbow. + * A given PCI device, in general, should be able to interrupt any of the + * cpus on any one of the HUBs or HEART connected to its xbow. */ -int pcibios_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) +int __init +pcibios_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) { return 0; } -static inline struct pci_dev *bridge_root_dev(struct pci_dev *dev) -{ - while (dev->bus->parent) { - /* Move up the chain of bridges. */ - dev = dev->bus->self; - } - - return dev; -} - /* Do platform specific device initialization at pci_enable_device() time */ -int pcibios_plat_dev_init(struct pci_dev *dev) +int +pcibios_plat_dev_init(struct pci_dev *dev) { struct bridge_controller *bc = BRIDGE_CONTROLLER(dev->bus); struct pci_dev *rdev = bridge_root_dev(dev); @@ -182,42 +256,35 @@ int pcibios_plat_dev_init(struct pci_dev *dev) return 0; } -/* - * Device might live on a subordinate PCI bus. XXX Walk up the chain of buses - * to find the slot number in sense of the bridge device register. - * XXX This also means multiple devices might rely on conflicting bridge - * settings. - */ - -static inline void pci_disable_swapping(struct pci_dev *dev) +static void +bridge_disable_swapping_dma(struct pci_dev *dev) { - struct bridge_controller *bc = BRIDGE_CONTROLLER(dev->bus); - bridge_t *bridge = bc->base; + u32 reg; int slot = PCI_SLOT(dev->devfn); - - /* Turn off byte swapping */ - bridge->b_device[slot].reg &= ~BRIDGE_DEV_SWAP_DIR; - bridge->b_widget.w_tflush; /* Flush */ -} - -static inline void pci_enable_swapping(struct pci_dev *dev) -{ struct bridge_controller *bc = BRIDGE_CONTROLLER(dev->bus); - bridge_t *bridge = bc->base; - int slot = PCI_SLOT(dev->devfn); - /* Turn on byte swapping */ - bridge->b_device[slot].reg |= BRIDGE_DEV_SWAP_DIR; - bridge->b_widget.w_tflush; /* Flush */ + /* Turn off byte swapping */ + reg = bridge_read_reg(bc, b_device(slot)); + reg &= ~(BRIDGE_DEV_DEV_SWAP | BRIDGE_DEV_SWAP_PMU | + BRIDGE_DEV_SWAP_DIR); + bridge_write_reg(reg, bc, b_device(slot)); + BRIDGE_FLUSH(bc); } -static void pci_fixup_ioc3(struct pci_dev *d) -{ - pci_disable_swapping(d); -} +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SGI, PCI_DEVICE_ID_SGI_IOC3, + bridge_disable_swapping_dma); +/* XXX: Temporarily defined here until the IP27 "mega update". */ #ifdef CONFIG_NUMA -int pcibus_to_node(struct pci_bus *bus) +/** + * pcibus_to_node - fetch the nasid that the passed struct pci_bus lives on. + * @bus: struct pci_bus pointer for a given PCI bus. + * + * casts bus->sysdata to struct bridge_controller and returns the nasid + * member that references the specific node this PCI bus lives on. + */ +int +pcibus_to_node(struct pci_bus *bus) { struct bridge_controller *bc = BRIDGE_CONTROLLER(bus); @@ -226,5 +293,3 @@ int pcibus_to_node(struct pci_bus *bus) EXPORT_SYMBOL(pcibus_to_node); #endif /* CONFIG_NUMA */ -DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_SGI, PCI_DEVICE_ID_SGI_IOC3, - pci_fixup_ioc3); -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 11/12] MIPS: IP27: Update IRQ code to work with the new BRIDGE code 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard ` (9 preceding siblings ...) 2017-02-07 6:13 ` [PATCH 10/12] MIPS: BRIDGE: Overhaul pci-bridge.c driver Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 6:13 ` [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case Joshua Kinard 11 siblings, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle; +Cc: Linux/MIPS From: Joshua Kinard <kumba@gentoo.org> Sparsely update just enough of the existing IP27 code so that the machine will boot against the newer Xtalk/BRIDGE code. A future patch series will completely rewrite a majority of the existing IP27 code. Signed-off-by: Joshua Kinard <kumba@gentoo.org> --- arch/mips/pci/pci-bridge.c | 11 +++++ arch/mips/sgi-ip27/ip27-irq-pci.c | 72 +++++++++++++++++------------ arch/mips/sgi-ip27/ip27-irq.c | 5 ++ 3 files changed, 58 insertions(+), 30 deletions(-) diff --git a/arch/mips/pci/pci-bridge.c b/arch/mips/pci/pci-bridge.c index 28b0eb5b844e..9df13ce313b5 100644 --- a/arch/mips/pci/pci-bridge.c +++ b/arch/mips/pci/pci-bridge.c @@ -274,6 +274,17 @@ bridge_disable_swapping_dma(struct pci_dev *dev) DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SGI, PCI_DEVICE_ID_SGI_IOC3, bridge_disable_swapping_dma); +#if 0 +/* + * XXX: Onyx2 systems have a RAD1 on their IO6-G board, but the RAD1 driver + * and corresponding defines are part of the IP30 patchset, so we need to + * make this part conditional for now until the IP30 patches are merged + * upstream. + */ +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_SGI, PCI_DEVICE_ID_SGI_RAD1, + bridge_disable_swapping_dma); +#endif + /* XXX: Temporarily defined here until the IP27 "mega update". */ #ifdef CONFIG_NUMA /** diff --git a/arch/mips/sgi-ip27/ip27-irq-pci.c b/arch/mips/sgi-ip27/ip27-irq-pci.c index 2a1c40784bd9..957b689b35f4 100644 --- a/arch/mips/sgi-ip27/ip27-irq-pci.c +++ b/arch/mips/sgi-ip27/ip27-irq-pci.c @@ -134,45 +134,56 @@ static int intr_disconnect_level(int cpu, int bit) /* Startup one of the (PCI ...) IRQs routes over a bridge. */ static unsigned int startup_bridge_irq(struct irq_data *d) { + u32 slot, reg; struct bridge_controller *bc; - bridgereg_t device; - bridge_t *bridge; - int pin, swlevel; + int swlevel; cpuid_t cpu; - pin = SLOT_FROM_PCI_IRQ(d->irq); + slot = SLOT_FROM_PCI_IRQ(d->irq); bc = IRQ_TO_BRIDGE(d->irq); - bridge = bc->base; - pr_debug("bridge_startup(): irq= 0x%x pin=%d\n", d->irq, pin); + pr_debug("bridge_startup(): irq= 0x%x pin=%d\n", d->irq, slot); /* * "map" irq to a swlevel greater than 6 since the first 6 bits * of INT_PEND0 are taken */ swlevel = find_level(&cpu, d->irq); - bridge->b_int_addr[pin].addr = (0x20000 | swlevel | (bc->nasid << 8)); - bridge->b_int_enable |= (1 << pin); - bridge->b_int_enable |= 0x7ffffe00; /* more stuff in int_enable */ /* - * Enable sending of an interrupt clear packt to the hub on a high to - * low transition of the interrupt pin. + * Handle BRIDGE IRQs. * - * IRIX sets additional bits in the address which are documented as - * reserved in the bridge docs. - */ - bridge->b_int_mode |= (1UL << pin); - - /* - * We assume the bridge to have a 1:1 mapping between devices - * (slots) and intr pins. + * b_int_addr(slot) - Points to the HUB that this BRIDGE + * is assigned to, so that BRIDGE can frob the PI_INT_PEND_MOD + * register directly, without having to know which nasid the + * specific HUB is on. + * + * b_int_mode - Enable the sending of an interrupt clear + * packet to the HUB on a high-to-low transition of the + * interrupt pin. + * + * b_int_device - We assume the bridge to have a 1:1 mapping + * between devices (slots) and interrupt numbers. + * + * XXX: Replace the magic values with readable macros at some point. */ - device = bridge->b_int_device; - device &= ~(7 << (pin*3)); - device |= (pin << (pin*3)); - bridge->b_int_device = device; - - bridge->b_wid_tflush; + /* b_int_addr */ + bridge_write_reg((0x20000 | swlevel | (bc->nasid << 8)), bc, + b_int_addr(slot)); + /* b_int_enable */ + reg = bridge_read_reg(bc, b_int_enable); + bridge_write_reg((reg | BIT(slot) | 0x7ffffe00), bc, b_int_enable); + + /* b_int_mode */ + reg = bridge_read_reg(bc, b_int_mode); + bridge_write_reg((reg | BIT(slot)), bc, b_int_mode); + + /* b_int_device */ + reg = bridge_read_reg(bc, b_int_device); + reg &= ~BRIDGE_INT_DEV_MASK(slot); + reg |= BRIDGE_INT_DEV_SET(slot, slot); + bridge_write_reg(reg, bc, b_int_device); + /* flush */ + BRIDGE_FLUSH(bc); intr_connect_level(cpu, swlevel); @@ -182,13 +193,13 @@ static unsigned int startup_bridge_irq(struct irq_data *d) /* Shutdown one of the (PCI ...) IRQs routes over a bridge. */ static void shutdown_bridge_irq(struct irq_data *d) { + u32 slot, reg; struct bridge_controller *bc = IRQ_TO_BRIDGE(d->irq); - bridge_t *bridge = bc->base; - int pin, swlevel; + int swlevel; cpuid_t cpu; pr_debug("bridge_shutdown: irq 0x%x\n", d->irq); - pin = SLOT_FROM_PCI_IRQ(d->irq); + slot = SLOT_FROM_PCI_IRQ(d->irq); /* * map irq to a swlevel greater than 6 since the first 6 bits @@ -197,8 +208,9 @@ static void shutdown_bridge_irq(struct irq_data *d) swlevel = find_level(&cpu, d->irq); intr_disconnect_level(cpu, swlevel); - bridge->b_int_enable &= ~(1 << pin); - bridge->b_wid_tflush; + reg = bridge_read_reg(bc, b_int_enable); + bridge_write_reg((reg & ~(BIT(slot))), bc, b_int_enable); + BRIDGE_FLUSH(bc); } static inline void enable_bridge_irq(struct irq_data *d) diff --git a/arch/mips/sgi-ip27/ip27-irq.c b/arch/mips/sgi-ip27/ip27-irq.c index 16ec4e12daa3..812b23c126ef 100644 --- a/arch/mips/sgi-ip27/ip27-irq.c +++ b/arch/mips/sgi-ip27/ip27-irq.c @@ -35,6 +35,11 @@ #include <asm/sn/hub.h> #include <asm/sn/intr.h> +/* XXX: Part of the IP27 "mega update" */ +#include <asm/mach-ip27/pcibr.h> +struct bridge_controller *irq_to_bridge[PCIBR_MAX_BUS_X_DEV]; +u32 irq_to_slot[PCIBR_MAX_BUS_X_DEV]; + /* * Linux has a controller-independent x86 interrupt architecture. * every controller has a 'controller-template', that is used -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard ` (10 preceding siblings ...) 2017-02-07 6:13 ` [PATCH 11/12] MIPS: IP27: Update IRQ code to work with the new BRIDGE code Joshua Kinard @ 2017-02-07 6:13 ` Joshua Kinard 2017-02-07 18:29 ` Bjorn Helgaas 11 siblings, 1 reply; 28+ messages in thread From: Joshua Kinard @ 2017-02-07 6:13 UTC (permalink / raw) To: Ralf Baechle Cc: Linux/MIPS, Bjorn Helgaas, Lorenzo Pieralisi, Thomas Bogendoerfer From: Joshua Kinard <kumba@gentoo.org> Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY will now explicitly claim PCI resources instead of skipping the sizing of the bridges and assigning resources. This is okay for IP30's PCI code, which doesn't use physical address space to access I/O resources. However, IP27 is completely different in this regard. Instead of using ioremapped addresses for I/O, IP27 has a dedicated address range, 0x92xxxxxxxxxxxxxx, that is used for all I/O access. Since this is uncached physical address space, the generic MIPS PCI code will not probe it correctly and thus, the original behavior of PCI_PROBE_ONLY needs to be restored only for the IP27 platform to bypass this logic and have working PCI, at least for the IO6/IO6G board that houses the base devices, until a better solution is found. Fixes: 046136170a56 ("MIPS/PCI: Claim bus resources on PCI_PROBE_ONLY set-ups") Signed-off-by: Joshua Kinard <kumba@gentoo.org> Cc: Bjorn Helgaas <bhelgaas@google.com> Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de> --- arch/mips/pci/pci-bridge.c | 15 +++++++++++++++ arch/mips/pci/pci-legacy.c | 15 +++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/mips/pci/pci-bridge.c b/arch/mips/pci/pci-bridge.c index 9df13ce313b5..af7073dba36b 100644 --- a/arch/mips/pci/pci-bridge.c +++ b/arch/mips/pci/pci-bridge.c @@ -62,6 +62,21 @@ bridge_probe(nasid_t nasid, int widget_id, int masterwid) unsigned long offset = NODE_OFFSET(nasid); struct bridge_controller *bc; +#ifdef CONFIG_SGI_IP27 + /* + * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY + * will now explicitly claim PCI resources instead of skipping the + * sizing of the bridges and assigning resources. This is okay for + * the IP30's PCI code, which uses normal, ioremapped addresses to + * do I/O. IP27, however, is different and uses a hardware-specific + * address range of 0x92xxxxxxxxxxxxxx for all I/O access. As such, + * the generic MIPS PCI code will not probe correctly and thus make + * PCI on IP27 completely unusable. Thus, we must restore the + * original logic only for IP27 until a better solution can be found. + */ + pci_set_flags(PCI_PROBE_ONLY); +#endif + /* XXX: Temporary until the IP27 "mega update". */ bc = &bridges[num_bridges]; if (!num_bridges) diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c index 68268bbb15b8..5590af4f367f 100644 --- a/arch/mips/pci/pci-legacy.c +++ b/arch/mips/pci/pci-legacy.c @@ -107,6 +107,20 @@ static void pcibios_scanbus(struct pci_controller *hose) need_domain_info = 1; } +#ifdef CONFIG_SGI_IP27 + /* + * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY + * will now explicitly claim PCI resources instead of skipping the + * sizing of the bridges and assigning resources. This is okay for + * the IP30's PCI code, which uses normal, ioremapped addresses to + * do I/O. IP27, however, is different and uses a hardware-specific + * address range of 0x92xxxxxxxxxxxxxx for all I/O access. As such, + * the generic MIPS PCI code will not probe correctly and thus make + * PCI on IP27 completely unusable. Thus, we must restore the + * original logic only for IP27 until a better solution can be found. + */ + if (!pci_has_flag(PCI_PROBE_ONLY)) { +#else /* * We insert PCI resources into the iomem_resource and * ioport_resource trees in either pci_bus_claim_resources() @@ -115,6 +129,7 @@ static void pcibios_scanbus(struct pci_controller *hose) if (pci_has_flag(PCI_PROBE_ONLY)) { pci_bus_claim_resources(bus); } else { +#endif pci_bus_size_bridges(bus); pci_bus_assign_resources(bus); } -- 2.11.1 ^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-07 6:13 ` [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case Joshua Kinard @ 2017-02-07 18:29 ` Bjorn Helgaas 2017-02-08 9:39 ` Joshua Kinard 2017-02-09 18:36 ` Joshua Kinard 0 siblings, 2 replies; 28+ messages in thread From: Bjorn Helgaas @ 2017-02-07 18:29 UTC (permalink / raw) To: Joshua Kinard Cc: Ralf Baechle, Linux/MIPS, Lorenzo Pieralisi, Thomas Bogendoerfer Hi Joshua, First of all, apologies for breaking IP27 and the inconvenience this caused you. And thanks a lot for tracking down the problem and proposing a solution! On Tue, Feb 7, 2017 at 12:13 AM, Joshua Kinard <kumba@gentoo.org> wrote: > From: Joshua Kinard <kumba@gentoo.org> > > Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY > will now explicitly claim PCI resources instead of skipping the sizing > of the bridges and assigning resources. This is okay for IP30's PCI > code, which doesn't use physical address space to access I/O resources. > > However, IP27 is completely different in this regard. Instead of using > ioremapped addresses for I/O, IP27 has a dedicated address range, > 0x92xxxxxxxxxxxxxx, that is used for all I/O access. Since this is > uncached physical address space, the generic MIPS PCI code will not > probe it correctly and thus, the original behavior of PCI_PROBE_ONLY > needs to be restored only for the IP27 platform to bypass this logic > and have working PCI, at least for the IO6/IO6G board that houses the > base devices, until a better solution is found. It sounds like there's something different about how ioremap() works on these platforms and PCI probing is tripping over that. I'd really like to understand more about this difference to see if we can converge that instead of adding back the PCI_PROBE_ONLY usage. Drivers shouldn't know whether they're running on IP27 or IP30, and they should be using ioremap() in both cases. Does ioremap() work differently on IP27 and IP30? Does this have something to do with plat_ioremap() or fixup_bigphys_addr()? Is there any chance you can collect complete dmesg logs and /proc/iomem contents from IP27 and IP30? Maybe "lspci -vv" output, too? I'm not sure where to look to understand the ioremap() behavior. What exactly is the PCI probe failure without this patch? If you have a console log (with "ignore_loglevel") it might have a clue, too. > Fixes: 046136170a56 ("MIPS/PCI: Claim bus resources on PCI_PROBE_ONLY set-ups") > Signed-off-by: Joshua Kinard <kumba@gentoo.org> > Cc: Bjorn Helgaas <bhelgaas@google.com> > Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> > Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de> > --- > arch/mips/pci/pci-bridge.c | 15 +++++++++++++++ > arch/mips/pci/pci-legacy.c | 15 +++++++++++++++ > 2 files changed, 30 insertions(+) > > diff --git a/arch/mips/pci/pci-bridge.c b/arch/mips/pci/pci-bridge.c > index 9df13ce313b5..af7073dba36b 100644 > --- a/arch/mips/pci/pci-bridge.c > +++ b/arch/mips/pci/pci-bridge.c > @@ -62,6 +62,21 @@ bridge_probe(nasid_t nasid, int widget_id, int masterwid) > unsigned long offset = NODE_OFFSET(nasid); > struct bridge_controller *bc; > > +#ifdef CONFIG_SGI_IP27 I don't know how MIPS multi-platform support works. If you enable CONFIG_SGI_IP27, does that mean the resulting kernel will *only* run on IP27? Or can you enable several platforms, e.g., SGI_IP22, SGI_IP27, SGI_IP28, SGI_IP32, etc., and end up with a kernel that will boot on any of those platforms? From Kconfig, it looks like these options are not mutually exclusive, so my guess is maybe the latter? If so, I would think whatever we do should be based on a run-time test for SGI_IP27 instead of a compile-time test. > + /* > + * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY > + * will now explicitly claim PCI resources instead of skipping the > + * sizing of the bridges and assigning resources. This is okay for > + * the IP30's PCI code, which uses normal, ioremapped addresses to > + * do I/O. IP27, however, is different and uses a hardware-specific > + * address range of 0x92xxxxxxxxxxxxxx for all I/O access. As such, > + * the generic MIPS PCI code will not probe correctly and thus make > + * PCI on IP27 completely unusable. Thus, we must restore the > + * original logic only for IP27 until a better solution can be found. > + */ > + pci_set_flags(PCI_PROBE_ONLY); > +#endif > + > /* XXX: Temporary until the IP27 "mega update". */ > bc = &bridges[num_bridges]; > if (!num_bridges) > diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c > index 68268bbb15b8..5590af4f367f 100644 > --- a/arch/mips/pci/pci-legacy.c > +++ b/arch/mips/pci/pci-legacy.c > @@ -107,6 +107,20 @@ static void pcibios_scanbus(struct pci_controller *hose) > need_domain_info = 1; > } > > +#ifdef CONFIG_SGI_IP27 > + /* > + * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY > + * will now explicitly claim PCI resources instead of skipping the > + * sizing of the bridges and assigning resources. This is okay for > + * the IP30's PCI code, which uses normal, ioremapped addresses to > + * do I/O. IP27, however, is different and uses a hardware-specific > + * address range of 0x92xxxxxxxxxxxxxx for all I/O access. As such, > + * the generic MIPS PCI code will not probe correctly and thus make > + * PCI on IP27 completely unusable. Thus, we must restore the > + * original logic only for IP27 until a better solution can be found. > + */ > + if (!pci_has_flag(PCI_PROBE_ONLY)) { > +#else > /* > * We insert PCI resources into the iomem_resource and > * ioport_resource trees in either pci_bus_claim_resources() > @@ -115,6 +129,7 @@ static void pcibios_scanbus(struct pci_controller *hose) > if (pci_has_flag(PCI_PROBE_ONLY)) { > pci_bus_claim_resources(bus); > } else { > +#endif > pci_bus_size_bridges(bus); > pci_bus_assign_resources(bus); > } > -- > 2.11.1 > ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-07 18:29 ` Bjorn Helgaas @ 2017-02-08 9:39 ` Joshua Kinard [not found] ` <CAErSpo4LsrPCtdZwp6CyT0jKhXLt3j=fGSiFjpRRTPUjFoKHtQ@mail.gmail.com> 2017-02-09 18:36 ` Joshua Kinard 1 sibling, 1 reply; 28+ messages in thread From: Joshua Kinard @ 2017-02-08 9:39 UTC (permalink / raw) To: Bjorn Helgaas Cc: Ralf Baechle, Linux/MIPS, Lorenzo Pieralisi, Thomas Bogendoerfer [-- Attachment #1: Type: text/plain, Size: 12598 bytes --] On 02/07/2017 13:29, Bjorn Helgaas wrote: > Hi Joshua, > > First of all, apologies for breaking IP27 and the inconvenience this > caused you. And thanks a lot for tracking down the problem and > proposing a solution! Eh, don't worry, IP27's been broken and unbroken over the years. These systems are becoming difficult to find intact, so these breaks tend to go unnoticed. > On Tue, Feb 7, 2017 at 12:13 AM, Joshua Kinard <kumba@gentoo.org> wrote: >> From: Joshua Kinard <kumba@gentoo.org> >> >> Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY >> will now explicitly claim PCI resources instead of skipping the sizing >> of the bridges and assigning resources. This is okay for IP30's PCI >> code, which doesn't use physical address space to access I/O resources. >> >> However, IP27 is completely different in this regard. Instead of using >> ioremapped addresses for I/O, IP27 has a dedicated address range, >> 0x92xxxxxxxxxxxxxx, that is used for all I/O access. Since this is >> uncached physical address space, the generic MIPS PCI code will not >> probe it correctly and thus, the original behavior of PCI_PROBE_ONLY >> needs to be restored only for the IP27 platform to bypass this logic >> and have working PCI, at least for the IO6/IO6G board that houses the >> base devices, until a better solution is found. > > It sounds like there's something different about how ioremap() works > on these platforms and PCI probing is tripping over that. I'd really > like to understand more about this difference to see if we can > converge that instead of adding back the PCI_PROBE_ONLY usage. I'd need to go and dig around in the IP27 headers again for this machine to see what ioremap() is actually doing, but I *think* it returns uncached physical addresses in most instances because of a special feature of the CPU, the R10000-family, which makes uncached access very fast. I think only the IP27 platform uses this capability. Other R1x0-based systems don't (although, I might be wrong about IP27's successor in IP35). > Drivers shouldn't know whether they're running on IP27 or IP30, and > they should be using ioremap() in both cases. Does ioremap() work > differently on IP27 and IP30? Does this have something to do with > plat_ioremap() or fixup_bigphys_addr()? I believe most drivers won't know about this change. The patch I proposed modifies two files very specific to MIPS and SGI systems, one being the PCI module for "legacy" MIPS systems, and the newly-proposed generic BRIDGE driver (created by an earlier patch in this patch series). The BRIDGE ASIC (and its cousin, the XBRIDGE) are only used on SGI systems and they provide a translation layer from Crosstalk address space (Xtalk, the system interconnect bus) to PCI devices by translating PCI ops into something understood by the rest of the system. BRIDGE is the only thing that needs to get setup correctly, and then, I think, other drivers should be unaware of the magic going on behind the scenes. That said, it's still a hack that I proposed. IP27 has long had issues with PCI because it uses addresses differently. I'm no expert in these systems -- just a hobby of mine I do in my free time -- but I'll refer you to these two documents, which are archive copies since SGI retired Techpubs last year: Origin™ and Onyx2™ Theory of Operations Manual: http://csweb.cs.wfu.edu/~torgerse/Kokua/SGI/007-3439-002/sgi_html/index.html Origin™ and Onyx2™ Programmer's Reference Manual http://csweb.cs.wfu.edu/~torgerse/Kokua/SGI/007-3410-001/sgi_html/ch02.html#id5437809 The whole setup is extremely modular, so each node gets a static, fixed portion of physical address space, regardless if that node is actually present or not. By setting various bits in the upper part of a memory address, you can do various things on specific nodes. See Chapter 1 in the programmer's manual for the details. The address space in question here, 0x9200000000000000, is how you tell the HUB (system controller for each node board) to access I/O space and talk to the device peripherals. Since this is physical address space, there's no need to ioremap, so I think deep in the IP27 headers, ioremap just falls through and returns one of these physical address, then you add in the needed offsets (node number, crosstalk port, and PCI address) to get to the device you're trying to reach. IP30, SGI Octane, is similar in that it's like a single-node IP27 system. It has a HEART as a system controller instead of a HUB, but also uses Crosstalk devices. Documentation for the system doesn't really exist -- work on it has been done via manual reverse engineering by people way better than I at this. I've just been maintaining the code in external patches over the years. I think the addresses on that platform *might* also be physical addresses, but with no documentation on HEART, it's unknown if HEART treats certain address spaces in special ways like HUB does. > Is there any chance you can collect complete dmesg logs and > /proc/iomem contents from IP27 and IP30? Maybe "lspci -vv" output, > too? I'm not sure where to look to understand the ioremap() behavior. Due to the size, I'll attach the dmesg and lspci outputs from both platforms as text files, and inline /proc/iomem. There are three dmesg files for IP27. One is a normal boot with enough patches to make PCI work. Another is without the PCI_PROBE_ONLY flag and my hack, so it will try to size/assign the resources, and the last is with PCI_PROBE_ONLY and my hack, so it will try to "claim" the resources only. For the last two, each results in different error paths. A note about IP30, since that's part of an external patch set, the dmesg output is a little different, as it is using the newer BRIDGE/Xtalk code I just sent in as patches. IP27's dmesg output won't be using my full patchset in this test, so it'll more closely resemble what you'd get out of the existing mainline code. Also, IP30 doesn't use the PCI_PROBE_ONLY thing anymore. This causes /proc/iomem to be far more detailed than it used to be. IP30's /proc/iomem: 00000000-00003fff : reserved 1d200000-1d9fffff : Bridge MEM d080000000-d080003fff : 0001:00:03.0 1d204000-1d204fff : 0001:00:02.0 1d205000-1d205fff : 0001:00:02.1 1d206000-1d206fff : 0001:00:02.2 1d207000-1d2070ff : 0001:00:02.3 1d207100-1d20717f : 0001:00:01.0 1f200000-1f9fffff : Bridge MEM f080100000-f0801fffff : 0000:00:02.0 f080010000-f08001ffff : 0000:00:00.0 f080030000-f08003ffff : 0000:00:01.0 f080000000-f080000fff : 0000:00:00.0 f080020000-f080020fff : 0000:00:01.0 20004000-209b8fff : reserved 20004000-206acb13 : Kernel code 206acb14-208affff : Kernel data 209b9000-20efffff : System RAM 20f00000-20ffffff : System RAM 21000000-9fffffff : System RAM f080100000-f0801fffff : ioc3 IP27's /proc/iomem (from the working boot): 0f600000-0f6fffff : ioc3 0f680000-0f687fff : rtc-m48t35 0fa00000-0fafffff : ioc3 920000000b200000-920000000b9fffff : Bridge MEM 920000000c200000-920000000c9fffff : Bridge MEM 920000000f200000-920000000f9fffff : Bridge MEM 920000000f620170-920000000f620177 : serial 920000000f620178-920000000f62017f : serial 920000000fa20170-920000000fa20177 : serial 920000000fa20178-920000000fa2017f : serial > What exactly is the PCI probe failure without this patch? If you have > a console log (with "ignore_loglevel") it might have a clue, too. > >> Fixes: 046136170a56 ("MIPS/PCI: Claim bus resources on PCI_PROBE_ONLY set-ups") >> Signed-off-by: Joshua Kinard <kumba@gentoo.org> >> Cc: Bjorn Helgaas <bhelgaas@google.com> >> Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> >> Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de> >> --- >> arch/mips/pci/pci-bridge.c | 15 +++++++++++++++ >> arch/mips/pci/pci-legacy.c | 15 +++++++++++++++ >> 2 files changed, 30 insertions(+) >> >> diff --git a/arch/mips/pci/pci-bridge.c b/arch/mips/pci/pci-bridge.c >> index 9df13ce313b5..af7073dba36b 100644 >> --- a/arch/mips/pci/pci-bridge.c >> +++ b/arch/mips/pci/pci-bridge.c >> @@ -62,6 +62,21 @@ bridge_probe(nasid_t nasid, int widget_id, int masterwid) >> unsigned long offset = NODE_OFFSET(nasid); >> struct bridge_controller *bc; >> >> +#ifdef CONFIG_SGI_IP27 > > I don't know how MIPS multi-platform support works. If you enable > CONFIG_SGI_IP27, does that mean the resulting kernel will *only* run > on IP27? Or can you enable several platforms, e.g., SGI_IP22, > SGI_IP27, SGI_IP28, SGI_IP32, etc., and end up with a kernel that will > boot on any of those platforms? From Kconfig, it looks like these > options are not mutually exclusive, so my guess is maybe the latter? > > If so, I would think whatever we do should be based on a run-time test > for SGI_IP27 instead of a compile-time test. Kernels are platform-specific for the SGI machines. So you need a different kernel for IP22, IP27, IP30, IP32, etc. Sometimes, different CPU's in a system needs a different kernel as well (e.g., IP32 w/ R5000 versus IP32 w/ RM7000, different kernels for each). The one *slight* exception is Indy and Indigo2, where one Indigo2 model is just a larger Indy (both are IP22). There's two other Indigo2 variants out there though, the rare Indigo2 Power (IP26) and the Indigo2 Impact (IP28). These classify as distinctly-different systems, mostly, and thus need their own kernels (not that anyone's written code for IP26 yet, but I digress). Other MIPS platforms may be a bit different. I can only speak for the SGI platforms right now. You can see where all the fun is at... >> + /* >> + * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY >> + * will now explicitly claim PCI resources instead of skipping the >> + * sizing of the bridges and assigning resources. This is okay for >> + * the IP30's PCI code, which uses normal, ioremapped addresses to >> + * do I/O. IP27, however, is different and uses a hardware-specific >> + * address range of 0x92xxxxxxxxxxxxxx for all I/O access. As such, >> + * the generic MIPS PCI code will not probe correctly and thus make >> + * PCI on IP27 completely unusable. Thus, we must restore the >> + * original logic only for IP27 until a better solution can be found. >> + */ >> + pci_set_flags(PCI_PROBE_ONLY); >> +#endif >> + >> /* XXX: Temporary until the IP27 "mega update". */ >> bc = &bridges[num_bridges]; >> if (!num_bridges) >> diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c >> index 68268bbb15b8..5590af4f367f 100644 >> --- a/arch/mips/pci/pci-legacy.c >> +++ b/arch/mips/pci/pci-legacy.c >> @@ -107,6 +107,20 @@ static void pcibios_scanbus(struct pci_controller *hose) >> need_domain_info = 1; >> } >> >> +#ifdef CONFIG_SGI_IP27 >> + /* >> + * Commit 046136170a56 changed things such that setting PCI_PROBE_ONLY >> + * will now explicitly claim PCI resources instead of skipping the >> + * sizing of the bridges and assigning resources. This is okay for >> + * the IP30's PCI code, which uses normal, ioremapped addresses to >> + * do I/O. IP27, however, is different and uses a hardware-specific >> + * address range of 0x92xxxxxxxxxxxxxx for all I/O access. As such, >> + * the generic MIPS PCI code will not probe correctly and thus make >> + * PCI on IP27 completely unusable. Thus, we must restore the >> + * original logic only for IP27 until a better solution can be found. >> + */ >> + if (!pci_has_flag(PCI_PROBE_ONLY)) { >> +#else >> /* >> * We insert PCI resources into the iomem_resource and >> * ioport_resource trees in either pci_bus_claim_resources() >> @@ -115,6 +129,7 @@ static void pcibios_scanbus(struct pci_controller *hose) >> if (pci_has_flag(PCI_PROBE_ONLY)) { >> pci_bus_claim_resources(bus); >> } else { >> +#endif >> pci_bus_size_bridges(bus); >> pci_bus_assign_resources(bus); >> } >> -- >> 2.11.1 >> > -- Joshua Kinard Gentoo/MIPS kumba@gentoo.org 6144R/F5C6C943 2015-04-27 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic [-- Attachment #2: ip30-dmesg-20170208.txt --] [-- Type: text/plain, Size: 27352 bytes --] [ 0.000000] Linux version 4.9.8-mipsgit-20161216 (root@helcaraxe) (gcc version 6.3.0 (Gentoo Hardened 6.3.0 p1.0) ) #1 SMP Sat Feb 4 21:01:37 EST 2017 [ 0.000000] ARCH: SGI-IP30 [ 0.000000] PROMLIB: ARC firmware Version 64 Revision 0 [ 0.000000] bootconsole [early_impact0] enabled [ 0.000000] CPU0 revision is: 00000f24 (R14000) [ 0.000000] FPU revision is: 00000900 [ 0.000000] Checking for the multiply/shift bug... [ 0.000000] no. [ 0.000000] Checking for the daddiu bug... [ 0.000000] no. [ 0.000000] Detected 2048MB of physical memory. [ 0.000000] IP30: CPU0: 600 MHz CPU detected. [ 0.000000] Determined physical RAM map: [ 0.000000] memory: 0000000000004000 @ 0000000000000000 (reserved) [ 0.000000] memory: 00000000009b5000 @ 0000000020004000 (reserved) [ 0.000000] memory: 0000000000547000 @ 00000000209b9000 (usable) [ 0.000000] memory: 0000000000100000 @ 0000000020f00000 (ROM data) [ 0.000000] memory: 000000007f000000 @ 0000000021000000 (usable) [ 0.000000] memory: 00000000008b0000 @ 0000000020000000 (usable) [ 0.000000] cma: Reserved 512 MiB at 0x0000000080000000 [ 0.000000] IP30: Slot: 0, PrID: 00000f24, PhyID: 0, VirtID: 0 [ 0.000000] IP30: Slot: 1, PrID: 00000f24, PhyID: 1, VirtID: 1 [ 0.000000] IP30: Detected 2 CPU(s) present. [ 0.000000] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 0.000000] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 0.000000] Unified secondary cache 2048kB 2-way, linesize 128 bytes. [ 0.000000] Zone ranges: [ 0.000000] DMA32 [mem 0x0000000020000000-0x00000000ffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000020000000-0x00000000209affff] [ 0.000000] node 0: [mem 0x00000000209c0000-0x000000009fffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000020000000-0x000000009fffffff] [ 0.000000] On node 0 totalpages: 32767 [ 0.000000] free_area_init_node: node 0, pgdat a80000002086a300, node_mem_map a8000000209d0300 [ 0.000000] DMA32 zone: 32 pages used for memmap [ 0.000000] DMA32 zone: 0 pages reserved [ 0.000000] DMA32 zone: 32767 pages, LIFO batch:1 [ 0.000000] percpu: Embedded 1 pages/cpu @a800000020c00000 s34336 r0 d31200 u65536 [ 0.000000] pcpu-alloc: s34336 r0 d31200 u65536 alloc=1*65536 [ 0.000000] pcpu-alloc: [0] 0 [0] 1 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping off. Total pages: 32735 [ 0.000000] Kernel command line: root=xio(0)pci(15)scsi(0)disk(1)rdisk(0)partition(0) console=tty0 root=/dev/md0 consoleblank=0 [ 0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes) [ 0.000000] Dentry cache hash table entries: 262144 (order: 5, 2097152 bytes) [ 0.000000] Inode-cache hash table entries: 131072 (order: 4, 1048576 bytes) [ 0.000000] Memory: 1556288K/2097088K available (6817K kernel code, 687K rwdata, 1348K rodata, 320K init, 739K bss, 16512K reserved, 524288K cma-reserved) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] NR_IRQS:128 [ 0.000000] IP30: HEART interrupt controller initialized. [ 0.000000] clocksource: HEART: mask: 0xfffffffffffff max_cycles: 0x2e2049cda, max_idle_ns: 440795202628 ns [ 0.000006] sched_clock: 52 bits at 12MHz, resolution 80ns, wraps every 4398046511080ns [ 0.000550] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6369645396 ns [ 0.001086] sched_clock: 32 bits at 300MHz, resolution 3ns, wraps every 7156904958ns [ 0.002520] Console: colour dummy device 80x25 [ 0.002875] console [tty0] enabled [ 0.003185] bootconsole [early_impact0] disabled [ 0.003573] Calibrating delay loop... 898.66 BogoMIPS (lpj=4493312) [ 0.090689] pid_max: default: 32768 minimum: 301 [ 0.091554] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes) [ 0.091573] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes) [ 0.093852] Checking for the daddi bug... [ 0.093871] no. [ 6.760385] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 6.760397] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 6.760402] Unified secondary cache 2048kB 2-way, linesize 128 bytes. [ 14.264191] IP30: CPU1: 600 MHz CPU detected. [ 14.264235] CPU1 revision is: 00000f24 (R14000) [ 14.264238] FPU revision is: 00000900 [ 0.230963] Synchronize counters for CPU 1: [ 0.230963] done. [ 0.231234] Brought up 2 CPUs [ 0.232749] devtmpfs: initialized [ 0.233747] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 0.236205] xor: measuring software checksum speed [ 0.330951] 8regs : 851.200 MB/sec [ 0.431036] 8regs_prefetch: 806.400 MB/sec [ 0.531198] 32regs : 729.600 MB/sec [ 0.631269] 32regs_prefetch: 748.800 MB/sec [ 0.631299] xor: using function: 8regs (851.200 MB/sec) [ 0.632356] NET: Registered protocol family 16 [ 0.671300] cpuidle: using governor ladder [ 0.711323] cpuidle: using governor menu [ 0.711467] xtalk:n0/f bridge widget (rev D) registered as a platform device. [ 0.711544] xtalk:n0/d bridge widget (rev C) registered as a platform device. [ 0.711617] xtalk:n0/c impact widget (rev B) registered as a platform device. [ 0.711690] xtalk:n0/8 heart widget (rev F) registered as a platform device. [ 0.961640] raid6: int64x1 gen() 323 MB/s [ 1.131840] raid6: int64x1 xor() 120 MB/s [ 1.301753] raid6: int64x2 gen() 468 MB/s [ 1.341955] random: fast init done [ 1.472023] raid6: int64x2 xor() 161 MB/s [ 1.642080] raid6: int64x4 gen() 416 MB/s [ 1.812194] raid6: int64x4 xor() 163 MB/s [ 1.982253] raid6: int64x8 gen() 342 MB/s [ 2.152536] raid6: int64x8 xor() 133 MB/s [ 2.152558] raid6: using algorithm int64x2 gen() 468 MB/s [ 2.152578] raid6: .... xor() 161 MB/s, rmw enabled [ 2.152597] raid6: using intx1 recovery algorithm [ 2.152988] SCSI subsystem initialized [ 2.153116] pps_core: LinuxPPS API ver. 1 registered [ 2.153138] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 2.153188] PTP clock support registered [ 2.154071] clocksource: Switched to clocksource HEART [ 2.154483] FS-Cache: Loaded [ 2.154723] CacheFiles: Loaded [ 2.162091] NET: Registered protocol family 2 [ 2.162753] TCP established hash table entries: 16384 (order: 1, 131072 bytes) [ 2.163396] TCP bind hash table entries: 16384 (order: 2, 262144 bytes) [ 2.164403] TCP: Hash tables configured (established 16384 bind 16384) [ 2.164619] UDP hash table entries: 2048 (order: 0, 65536 bytes) [ 2.164892] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes) [ 2.165412] NET: Registered protocol family 1 [ 2.166001] RPC: Registered named UNIX socket transport module. [ 2.166027] RPC: Registered udp transport module. [ 2.166047] RPC: Registered tcp transport module. [ 2.166066] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.166093] PCI: CLS 0 bytes, default 128 [ 2.166728] futex hash table entries: 512 (order: 0, 65536 bytes) [ 2.167250] workingset: timestamp_bits=46 max_order=15 bucket_order=0 [ 2.167409] zbud: loaded [ 2.168725] DLM installed [ 2.168790] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 2.169364] NFS: Registering the id_resolver key type [ 2.169420] Key type id_resolver registered [ 2.169442] Key type id_legacy registered [ 2.169472] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 2.170214] efs: 1.0a - http://aeschi.ch.eu.org/efs/ [ 2.170279] SGI XFS with ACLs, security attributes, realtime, no debug enabled [ 2.181274] NET: Registered protocol family 38 [ 2.181383] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 2.181422] io scheduler noop registered [ 2.181443] io scheduler deadline registered [ 2.181468] io scheduler bfq registered (default) [ 2.181489] BFQ I/O-scheduler: v8r7 [ 2.181668] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.223223] Console: switching to colour frame buffer device 160x64 [ 2.255766] fb0: ImpactSR 1RSS frame buffer device [ 2.317384] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.327988] loop: module loaded [ 2.328851] mousedev: PS/2 mouse device common for all mice [ 2.329536] md: raid6 personality registered for level 6 [ 2.329752] md: raid5 personality registered for level 5 [ 2.329961] md: raid4 personality registered for level 4 [ 2.330336] PCI host bridge to bus 0000:00 [ 2.330521] pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff] [ 2.330794] pci_bus 0000:00: root bus resource [io 0x1fa00000-0x1fbfffff] [ 2.331068] pci_bus 0000:00: root bus resource [bus 00-ff] [ 2.331330] pci 0000:00:00.0: [1077:1020] type 00 class 0x010000 [ 2.331378] pci 0000:00:00.0: reg 0x10: [io 0x200000-0x2000ff] [ 2.331402] pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff] [ 2.331483] pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref] [ 2.331704] pci 0000:00:01.0: [1077:1020] type 00 class 0x010000 [ 2.331738] pci 0000:00:01.0: reg 0x10: [io 0x400000-0x4000ff] [ 2.331761] pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff] [ 2.331843] pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref] [ 2.332030] pci 0000:00:02.0: [10a9:0003] type 00 class 0xff0000 [ 2.332068] pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff] [ 2.332345] pci 0000:00:03.0: [10a9:0005] type 00 class 0x000000 [ 2.332377] pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff] [ 2.332727] pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff] [ 2.333011] pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref] [ 2.333298] pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref] [ 2.333582] pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff] [ 2.333856] pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff] [ 2.334181] pci 0000:00:00.0: BAR 0: assigned [io 0x1fa00000-0x1fa000ff] [ 2.334456] pci 0000:00:01.0: BAR 0: assigned [io 0x1fa00400-0x1fa004ff] [ 2.334795] qla1280: QLA1040 found on PCI bus 0, dev 0 [ 2.335016] ip30-bridge: 0000:00:00.0 Bar 0 with size 0x00000100 at bus 0x00000000 vma 0x000000f100000000 is Direct I/O. [ 2.335435] ip30-bridge: 0000:00:00.0 Bar 1 with size 0x00001000 at bus 0x00000000 vma 0x000000f080000000 is Direct 64-bit. [ 2.335862] ip30-bridge: 0000:00:00.0 Bar 6 with size 0x00010000 at bus 0x00010000 vma 0x000000f080010000 is Direct 64-bit. [ 2.336287] PCI: Enabling device 0000:00:00.0 (0006 -> 0007) [ 2.337044] scsi(0): Enabling vchannel on BRIDGE for SGI/MIPS [ 2.738704] random: crng init done [ 2.923416] scsi(0:0): Resetting SCSI BUS [ 6.004087] scsi host0: QLogic QLA1040 PCI to SCSI Host Adapter Firmware version: 7.65.06, Driver version 3.27.1 [ 6.010788] qla1280: QLA1040 found on PCI bus 0, dev 1 [ 6.011039] ip30-bridge: 0000:00:01.0 Bar 0 with size 0x00000100 at bus 0x00000100 vma 0x000000f100000100 is Direct I/O. [ 6.011464] ip30-bridge: 0000:00:01.0 Bar 1 with size 0x00001000 at bus 0x00020000 vma 0x000000f080020000 is Direct 64-bit. [ 6.011895] ip30-bridge: 0000:00:01.0 Bar 6 with size 0x00010000 at bus 0x00030000 vma 0x000000f080030000 is Direct 64-bit. [ 6.012324] PCI: Enabling device 0000:00:01.0 (0006 -> 0007) [ 13.185643] scsi 0:0:1:0: Direct-Access IBM-ESXS ST973401LC FN B41D PQ: 0 ANSI: 4 [ 13.186771] scsi(0:0:1:0): [ 13.186889] Sync: period 10, offset 12 [ 13.187053] , Wide [ 13.187089] , Tagged queuing: depth 31 [ 13.962105] scsi 0:0:2:0: Direct-Access IBM-ESXS ST973401LC FN B41D PQ: 0 ANSI: 4 [ 13.963095] scsi(0:0:2:0): [ 13.963212] Sync: period 10, offset 12 [ 13.972934] , Wide [ 13.973098] , Tagged queuing: depth 31 [ 14.175153] scsi 0:0:3:0: Direct-Access IBM-ESXS ST973401LC FN B41D PQ: 0 ANSI: 4 [ 14.186715] scsi(0:0:3:0): [ 14.186837] Sync: period 10, offset 12 [ 14.197546] , Wide [ 14.197712] , Tagged queuing: depth 31 [ 14.511621] scsi(1:0): Resetting SCSI BUS [ 19.044315] scsi host1: QLogic QLA1040 PCI to SCSI Host Adapter Firmware version: 7.65.06, Driver version 3.27.1 [ 19.070925] ip30-bridge: 0000:00:02.0 Bar 0 with size 0x00100000 at bus 0x00100000 vma 0x000000f080100000 is Direct 64-bit. [ 21.194039] ioc3: part: [030-0891-003], serial: [KBB731] => class IP30 System Board [ 21.207013] ioc3-eth: Ethernet address is 08:00:69:0e:11:0c. [ 21.218711] IOC3 0000:00:02.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 21.229449] eth0: Using PHY 1, vendor 0x15f42, model 2, rev 3. [ 21.240215] eth0: IOC3 SSRAM has 128 kbyte. [ 21.251666] rtc-ds1685 rtc-ds1685: rtc core: registered rtc-ds1685 as rtc0 [ 21.287565] 0000:00:02.0: ttyS0 at IOC3 0xf080120178 (irq = 0, base_baud = 458333) is a 16550A [ 21.319087] 0000:00:02.0: ttyS1 at IOC3 0xf080120170 (irq = 0, base_baud = 458333) is a 16550A [ 21.330079] IOC3 Master Driver loaded for 0000:00:02.0 [ 21.340951] PCI host bridge to bus 0001:00 [ 21.351657] pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff] [ 21.362655] pci_bus 0001:00: root bus resource [io 0x1da00000-0x1dbfffff] [ 21.373502] pci_bus 0001:00: root bus resource [bus 01-ff] [ 21.384204] pci 0001:00:01.0: [11fe:080e] type 00 class 0x078000 [ 21.384251] pci 0001:00:01.0: reg 0x10: [mem 0x00200000-0x0020007f] [ 21.384274] pci 0001:00:01.0: reg 0x14: [io 0x200000-0x20007f] [ 21.384297] pci 0001:00:01.0: reg 0x18: [io 0x204000-0x2040ff] [ 21.384578] pci 0001:00:02.0: [10b9:5237] type 00 class 0x0c0310 [ 21.384618] pci 0001:00:02.0: reg 0x10: [mem 0x00300000-0x00300fff] [ 21.384778] pci 0001:00:02.0: PME# supported from D0 D1 D3hot D3cold [ 21.384951] pci 0001:00:02.1: [10b9:5237] type 00 class 0x0c0310 [ 21.384987] pci 0001:00:02.1: reg 0x10: [mem 0x00000000-0x00000fff] [ 21.385126] pci 0001:00:02.1: PME# supported from D0 D1 D3hot D3cold [ 21.385291] pci 0001:00:02.2: [10b9:5237] type 00 class 0x0c0310 [ 21.385327] pci 0001:00:02.2: reg 0x10: [mem 0x00000000-0x00000fff] [ 21.385466] pci 0001:00:02.2: PME# supported from D0 D1 D3hot D3cold [ 21.385636] pci 0001:00:02.3: [10b9:5239] type 00 class 0x0c0320 [ 21.385676] pci 0001:00:02.3: reg 0x10: [mem 0x00000000-0x000000ff] [ 21.385828] pci 0001:00:02.3: PME# supported from D0 D3hot D3cold [ 21.386004] pci 0001:00:03.0: [10a9:0009] type 00 class 0x020000 [ 21.386039] pci 0001:00:03.0: reg 0x10: [mem 0x00400000-0x00403fff] [ 21.386395] pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff] [ 21.397018] pci 0001:00:02.0: BAR 0: assigned [mem 0x1d204000-0x1d204fff] [ 21.407552] pci 0001:00:02.1: BAR 0: assigned [mem 0x1d205000-0x1d205fff] [ 21.417945] pci 0001:00:02.2: BAR 0: assigned [mem 0x1d206000-0x1d206fff] [ 21.428185] pci 0001:00:01.0: BAR 2: assigned [io 0x1da00000-0x1da000ff] [ 21.438321] pci 0001:00:02.3: BAR 0: assigned [mem 0x1d207000-0x1d2070ff] [ 21.448373] pci 0001:00:01.0: BAR 0: assigned [mem 0x1d207100-0x1d20717f] [ 21.458433] pci 0001:00:01.0: BAR 1: assigned [io 0x1da00400-0x1da0047f] [ 21.468646] acenic.c: v0.92 08/05/2002 Jes Sorensen, linux-acenic@SunSITE.dk http://home.cern.ch/~jes/gige/acenic.html [ 21.488947] ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit. [ 21.499592] 0001:00:03.0: SGI AceNIC [ 21.499804] Gigabit Ethernet at 0xd080000000, [ 21.510369] irq 4 [ 21.521387] Tigon II (Rev. 6), Firmware: 0.0.0, [ 21.526945] MAC: 08:00:69:14:69:9c [ 21.537682] PCI cache line size set incorrectly (0 bytes) by BIOS/FW, [ 21.538031] correcting to 128 [ 21.549022] PCI bus width: 64 bits, speed: 33MHz, latency: 64 clks [ 21.609244] 0001:00:03.0: Firmware up and running [ 21.621834] NET: Registered protocol family 26 [ 21.633741] NET: Registered protocol family 10 [ 21.646997] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver [ 21.659228] NET: Registered protocol family 17 [ 21.670539] NET: Registered protocol family 4 [ 21.681995] 8021q: 802.1Q VLAN Support v1.8 [ 21.693639] sctp: Hash tables configured (bind 4096/4096) [ 21.705107] Key type dns_resolver registered [ 21.717647] rtc-ds1685 rtc-ds1685: setting system clock to 2017-02-08 06:26:13 UTC (1486535173) [ 21.847090] input: AT Raw Set 2 keyboard as /devices/pci0000:00/0000:00:02.0/serio0/input/input0 [ 22.077817] input: ImPS/2 Logitech Wheel Mouse as /devices/pci0000:00/0000:00:02.0/serio1/input/input2 [ 23.593306] sd 0:0:1:0: [sda] 143374000 512-byte logical blocks: (73.4 GB/68.4 GiB) [ 23.605279] sd 0:0:2:0: [sdb] 143374000 512-byte logical blocks: (73.4 GB/68.4 GiB) [ 23.605350] sd 0:0:3:0: [sdc] 143374000 512-byte logical blocks: (73.4 GB/68.4 GiB) [ 23.608552] sd 0:0:1:0: [sda] Write Protect is off [ 23.608562] sd 0:0:1:0: [sda] Mode Sense: b3 00 10 08 [ 23.608736] sd 0:0:3:0: [sdc] Write Protect is off [ 23.608746] sd 0:0:3:0: [sdc] Mode Sense: b3 00 10 08 [ 23.610832] sd 0:0:1:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 23.611019] sd 0:0:3:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 23.630520] sdc: sdc1 sdc2 sdc3 sdc4 sdc5 sdc6 sdc9 sdc11 [ 23.634527] sda: sda1 sda2 sda3 sda4 sda5 sda6 sda9 sda11 [ 23.641757] sd 0:0:3:0: [sdc] Attached SCSI disk [ 23.643655] sd 0:0:1:0: [sda] Attached SCSI disk [ 23.720715] sd 0:0:2:0: [sdb] Write Protect is off [ 23.731276] sd 0:0:2:0: [sdb] Mode Sense: b3 00 10 08 [ 23.735226] sd 0:0:2:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 23.767686] sdb: sdb1 sdb2 sdb3 sdb4 sdb5 sdb6 sdb9 sdb11 [ 23.786987] sd 0:0:2:0: [sdb] Attached SCSI disk [ 27.778038] eth1: Optical link UP (Full Duplex, Flow Control: TX RX) [ 29.454240] md: Waiting for all devices to be available before autodetect [ 29.464960] md: If you don't use raid, use raid=noautodetect [ 29.476420] md: Autodetecting RAID arrays. [ 29.618865] md: Scanned 15 and added 15 devices. [ 29.629579] md: autorun ... [ 29.640073] md: considering sdb6 ... [ 29.650393] md: adding sdb6 ... [ 29.660352] md: sdb5 has different UUID to sdb6 [ 29.670272] md: sdb4 has different UUID to sdb6 [ 29.679984] md: sdb3 has different UUID to sdb6 [ 29.689514] md: sdb1 has different UUID to sdb6 [ 29.698907] md: adding sda6 ... [ 29.708151] md: sda5 has different UUID to sdb6 [ 29.717293] md: sda4 has different UUID to sdb6 [ 29.726434] md: sda3 has different UUID to sdb6 [ 29.735366] md: sda1 has different UUID to sdb6 [ 29.744150] md: adding sdc6 ... [ 29.752762] md: sdc5 has different UUID to sdb6 [ 29.761561] md: sdc4 has different UUID to sdb6 [ 29.770264] md: sdc3 has different UUID to sdb6 [ 29.778788] md: sdc1 has different UUID to sdb6 [ 29.788125] md: created md4 [ 29.796582] md: bind<sdc6> [ 29.805174] md: bind<sda6> [ 29.813663] md: bind<sdb6> [ 29.822159] md: running: [ 29.822307] <sdb6> [ 29.830737] <sda6> [ 29.830883] <sdc6> [ 29.848811] md/raid:md4: device sdb6 operational as raid disk 1 [ 29.857310] md/raid:md4: device sda6 operational as raid disk 0 [ 29.865556] md/raid:md4: device sdc6 operational as raid disk 2 [ 29.875656] md/raid:md4: allocated 49386kB [ 29.883462] md/raid:md4: raid level 5 active with 3 out of 3 devices, algorithm 2 [ 29.891223] RAID conf printout: [ 29.891230] --- level:5 rd:3 wd:3 [ 29.891236] disk 0, o:1, dev:sda6 [ 29.891241] disk 1, o:1, dev:sdb6 [ 29.891246] disk 2, o:1, dev:sdc6 [ 29.930444] created bitmap (1 pages) for device md4 [ 29.942220] md4: bitmap initialized from disk: read 1 pages, set 0 of 5 bits [ 29.955188] md4: detected capacity change from 0 to 2263613440 [ 29.962791] md: considering sdb5 ... [ 29.970412] md: adding sdb5 ... [ 29.978015] md: sdb4 has different UUID to sdb5 [ 29.985631] md: sdb3 has different UUID to sdb5 [ 29.993100] md: sdb1 has different UUID to sdb5 [ 30.000602] md: adding sda5 ... [ 30.008021] md: sda4 has different UUID to sdb5 [ 30.015457] md: sda3 has different UUID to sdb5 [ 30.022727] md: sda1 has different UUID to sdb5 [ 30.029792] md: adding sdc5 ... [ 30.036613] md: sdc4 has different UUID to sdb5 [ 30.043326] md: sdc3 has different UUID to sdb5 [ 30.049785] md: sdc1 has different UUID to sdb5 [ 30.056974] md: created md3 [ 30.063332] md: bind<sdc5> [ 30.069777] md: bind<sda5> [ 30.076087] md: bind<sdb5> [ 30.082235] md: running: [ 30.082403] <sdb5> [ 30.088532] <sda5> [ 30.088677] <sdc5> [ 30.101736] md/raid:md3: device sdb5 operational as raid disk 1 [ 30.107954] md/raid:md3: device sda5 operational as raid disk 0 [ 30.114238] md/raid:md3: device sdc5 operational as raid disk 2 [ 30.122570] md/raid:md3: allocated 49386kB [ 30.128738] md/raid:md3: raid level 5 active with 3 out of 3 devices, algorithm 2 [ 30.134803] RAID conf printout: [ 30.134809] --- level:5 rd:3 wd:3 [ 30.134814] disk 0, o:1, dev:sda5 [ 30.134819] disk 1, o:1, dev:sdb5 [ 30.134824] disk 2, o:1, dev:sdc5 [ 30.135368] created bitmap (1 pages) for device md3 [ 30.141770] md3: bitmap initialized from disk: read 1 pages, set 0 of 9 bits [ 30.151821] md3: detected capacity change from 0 to 4714135552 [ 30.157929] md: considering sdb4 ... [ 30.163894] md: adding sdb4 ... [ 30.169855] md: sdb3 has different UUID to sdb4 [ 30.175818] md: sdb1 has different UUID to sdb4 [ 30.181669] md: adding sda4 ... [ 30.187556] md: sda3 has different UUID to sdb4 [ 30.193472] md: sda1 has different UUID to sdb4 [ 30.199386] md: adding sdc4 ... [ 30.205261] md: sdc3 has different UUID to sdb4 [ 30.211266] md: sdc1 has different UUID to sdb4 [ 30.218075] md: created md2 [ 30.223942] md: bind<sdc4> [ 30.229963] md: bind<sda4> [ 30.235856] md: bind<sdb4> [ 30.241738] md: running: [ 30.241910] <sdb4> [ 30.247783] <sda4> [ 30.247927] <sdc4> [ 30.260739] md/raid:md2: device sdb4 operational as raid disk 1 [ 30.266922] md/raid:md2: device sda4 operational as raid disk 0 [ 30.273147] md/raid:md2: device sdc4 operational as raid disk 2 [ 30.281542] md/raid:md2: allocated 49386kB [ 30.287799] md/raid:md2: raid level 5 active with 3 out of 3 devices, algorithm 2 [ 30.294168] RAID conf printout: [ 30.294173] --- level:5 rd:3 wd:3 [ 30.294179] disk 0, o:1, dev:sda4 [ 30.294185] disk 1, o:1, dev:sdb4 [ 30.294190] disk 2, o:1, dev:sdc4 [ 30.294759] created bitmap (1 pages) for device md2 [ 30.301603] md2: bitmap initialized from disk: read 1 pages, set 0 of 204 bits [ 30.314853] md2: detected capacity change from 0 to 109521403904 [ 30.321657] md: considering sdb3 ... [ 30.328478] md: adding sdb3 ... [ 30.335220] md: sdb1 has different UUID to sdb3 [ 30.341980] md: adding sda3 ... [ 30.348792] md: sda1 has different UUID to sdb3 [ 30.355738] md: adding sdc3 ... [ 30.362566] md: sdc1 has different UUID to sdb3 [ 30.370189] md: created md1 [ 30.376794] md: bind<sdc3> [ 30.383346] md: bind<sda3> [ 30.389765] md: bind<sdb3> [ 30.395951] md: running: [ 30.396102] <sdb3> [ 30.402210] <sda3> [ 30.402374] <sdc3> [ 30.415789] md/raid:md1: device sdb3 operational as raid disk 1 [ 30.422276] md/raid:md1: device sda3 operational as raid disk 0 [ 30.428822] md/raid:md1: device sdc3 operational as raid disk 2 [ 30.437351] md/raid:md1: allocated 49386kB [ 30.443851] md/raid:md1: raid level 5 active with 3 out of 3 devices, algorithm 2 [ 30.450498] RAID conf printout: [ 30.450504] --- level:5 rd:3 wd:3 [ 30.450509] disk 0, o:1, dev:sda3 [ 30.450514] disk 1, o:1, dev:sdb3 [ 30.450519] disk 2, o:1, dev:sdc3 [ 30.451066] created bitmap (1 pages) for device md1 [ 30.458150] md1: bitmap initialized from disk: read 1 pages, set 0 of 44 bits [ 30.478655] md1: detected capacity change from 0 to 23622057984 [ 30.485668] md: considering sdb1 ... [ 30.492746] md: adding sdb1 ... [ 30.499820] md: adding sda1 ... [ 30.506833] md: adding sdc1 ... [ 30.513823] md: created md0 [ 30.520890] md: bind<sdc1> [ 30.527969] md: bind<sda1> [ 30.534841] md: bind<sdb1> [ 30.541634] md: running: [ 30.541804] <sdb1> [ 30.548437] <sda1> [ 30.548581] <sdc1> [ 30.563131] md/raid:md0: device sdb1 operational as raid disk 1 [ 30.570154] md/raid:md0: device sda1 operational as raid disk 0 [ 30.577066] md/raid:md0: device sdc1 operational as raid disk 2 [ 30.585924] md/raid:md0: allocated 49386kB [ 30.592560] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2 [ 30.599202] RAID conf printout: [ 30.599207] --- level:5 rd:3 wd:3 [ 30.599213] disk 0, o:1, dev:sda1 [ 30.599218] disk 1, o:1, dev:sdb1 [ 30.599223] disk 2, o:1, dev:sdc1 [ 30.599790] created bitmap (1 pages) for device md0 [ 30.606979] md0: bitmap initialized from disk: read 1 pages, set 7 of 8 bits [ 30.622820] md0: detected capacity change from 0 to 4294705152 [ 30.629693] md: ... autorun DONE. [ 31.634010] XFS (md0): Mounting V4 Filesystem [ 34.170302] XFS (md0): Ending clean mount [ 34.177482] VFS: Mounted root (xfs filesystem) readonly on device 9:0. [ 34.187355] devtmpfs: mounted [ 34.195485] Freeing unused kernel memory: 320K (a8000000208b0000 - a800000020900000) [ 34.202859] This architecture does not have kernel memory protection. [ 46.575416] udevd[593]: starting version 3.2.1 [ 47.260061] udevd[593]: specified group 'video' unknown [ 47.261061] udevd[593]: specified group 'tape' unknown [ 47.378114] udevd[594]: starting eudev-3.2.1 [ 47.849194] udevd[594]: specified group 'video' unknown [ 47.850206] udevd[594]: specified group 'tape' unknown [ 51.867298] Adding 1048512k swap on /dev/sda2. Priority:1 extents:1 across:1048512k [ 51.883342] Adding 1048512k swap on /dev/sdb2. Priority:1 extents:1 across:1048512k [ 51.906315] Adding 1048512k swap on /dev/sdc2. Priority:1 extents:1 across:1048512k [ 52.273998] XFS (md1): Mounting V5 Filesystem [ 54.356290] XFS (md1): Ending clean mount [ 54.378647] XFS (md2): Mounting V5 Filesystem [ 54.877762] XFS (md2): Ending clean mount [ 54.934648] XFS (md3): Mounting V5 Filesystem [ 55.402402] XFS (md3): Ending clean mount [ 55.418250] XFS (md4): Mounting V5 Filesystem [ 55.950290] XFS (md4): Ending clean mount [-- Attachment #3: ip30-lspci-20170208.txt --] [-- Type: text/plain, Size: 10744 bytes --] 0000:00:00.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64, Cache Line Size: 256 bytes Interrupt: pin A routed to IRQ 0 Region 0: I/O ports at f100000000 [size=257] Region 1: [virtual] Memory at f080000000 (32-bit, non-prefetchable) [size=4097] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at f080010000 [disabled] [size=65537] Kernel driver in use: qla1280 lspci: Unable to load libkmod resources: error -12 0000:00:01.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64, Cache Line Size: 256 bytes Interrupt: pin A routed to IRQ 1 Region 0: I/O ports at f100000100 [size=257] Region 1: Memory at f080020000 (32-bit, non-prefetchable) [size=4097] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at f080030000 [disabled] [size=65537] Kernel driver in use: qla1280 0000:00:02.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. IOC3 I/O controller (rev 01) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin A routed to IRQ 2 Region 0: Memory at f080100000 (32-bit, non-prefetchable) [size=1048577] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Kernel driver in use: IOC3 0000:00:03.0 Non-VGA unclassified device: Silicon Graphics Intl. Corp. RAD Audio (rev c0) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=slow >TAbort+ <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 255 Region 0: Memory at 00600000 (32-bit, non-prefetchable) [size=8193] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] 0001:00:01.0 Communication controller: Comtrol Corporation Device 080e (rev 01) Subsystem: Comtrol Corporation Device 080e Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 0 Region 0: Memory at 1d207100 (32-bit, non-prefetchable) [size=129] Region 1: I/O ports at 1da00400 [size=129] Region 2: I/O ports at 1da00000 [size=257] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] 0001:00:02.0 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 16 (20000ns max) Interrupt: pin B routed to IRQ 0 Region 0: Memory at 1d204000 (32-bit, non-prefetchable) [size=4097] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- 0001:00:02.1 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin C routed to IRQ 0 Region 0: Memory at 1d205000 (32-bit, non-prefetchable) [disabled] [size=4097] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- 0001:00:02.2 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin D routed to IRQ 0 Region 0: Memory at 1d206000 (32-bit, non-prefetchable) [disabled] [size=4097] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- 0001:00:02.3 USB controller: ULi Electronics Inc. USB 2.0 Controller (rev 01) (prog-if 20 [EHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 0 Region 0: Memory at 1d207000 (32-bit, non-prefetchable) [disabled] [size=257] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Debug port: BAR=1 offset=0090 0001:00:03.0 Ethernet controller: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet (rev 01) Subsystem: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 (16000ns min), Cache Line Size: 128 bytes Interrupt: pin A routed to IRQ 4 Region 0: [virtual] Memory at d080000000 (32-bit, non-prefetchable) [size=16385] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Kernel driver in use: acenic [-- Attachment #4: ip27-lspci-20170208.txt --] [-- Type: text/plain, Size: 6736 bytes --] 0001:00:00.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. Linc I/O controller Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- NUMA node: 0 Region 0: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=134217729] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] lspci: Unable to load libkmod resources: error -12 0001:00:01.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. Linc I/O controller Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- NUMA node: 0 Region 0: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=134217729] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] 0002:00:00.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64, Cache Line Size: 256 bytes Interrupt: pin A routed to IRQ 1 NUMA node: 0 Region 0: I/O ports at f200000 [size=257] Region 1: Memory at 0f200000 (32-bit, non-prefetchable) [size=4097] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=65537] Kernel driver in use: qla1280 0002:00:01.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64, Cache Line Size: 256 bytes Interrupt: pin A routed to IRQ 2 NUMA node: 0 Region 0: I/O ports at f400000 [size=257] Region 1: Memory at 0f400000 (32-bit, non-prefetchable) [size=4097] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=65537] Kernel driver in use: qla1280 0002:00:02.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. IOC3 I/O controller (rev 01) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 255 Interrupt: pin A routed to IRQ 3 NUMA node: 0 Region 0: Memory at 0f600000 (32-bit, non-prefetchable) [size=1048577] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Kernel driver in use: ioc3-eth 0002:00:06.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. IOC3 I/O controller (rev 01) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 255 Interrupt: pin A routed to IRQ 4 NUMA node: 0 Region 0: Memory at 0fa00000 (32-bit, non-prefetchable) [size=1048577] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Kernel driver in use: ioc3-eth 0002:00:07.0 Non-VGA unclassified device: Silicon Graphics Intl. Corp. RAD Audio (rev 70) Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=slow >TAbort+ <TAbort- <MAbort- >SERR- <PERR- INTx- NUMA node: 0 Region 0: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=8193] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] [-- Attachment #5: ip27-dmesg-working_pci-20170208.txt --] [-- Type: text/plain, Size: 22685 bytes --] [ 0.000000] Linux version 4.10.0-rc7 (root@helcaraxe) (gcc version 6.3.0 (Gentoo Hardened 6.3.0 p1.0) ) #1 SMP Wed Feb 8 03:55:04 EST 2017 [ 0.000000] ARCH: SGI-IP27 [ 0.000000] PROMLIB: ARC firmware Version 64 Revision 0 [ 0.000000] Discovered 4 cpus on 2 nodes [ 0.000000] node_distance: router_a NULL [ 0.000000] node_distance: router_a NULL [ 0.000000] node_distance: router_a NULL [ 0.000000] node_distance: router_a NULL [ 0.000000] ************** Topology ******************** [ 0.000000] [ 0.000000] 00 [ 0.000000] 01 [ 0.000000] 00 [ 0.000000] 255 [ 0.000000] 255 [ 0.000000] 01 [ 0.000000] 255 [ 0.000000] 255 [ 0.000000] bootconsole [early0] enabled [ 0.000000] CPU0 revision is: 00000f14 (R14000) [ 0.000000] FPU revision is: 00000900 [ 0.000000] Checking for the multiply/shift bug... no. [ 0.000000] Checking for the daddiu bug... no. [ 0.000000] IP27: Running on node 0. [ 0.000000] Node 0 has a primary CPU, CPU is running. [ 0.000000] Node 0 has a secondary CPU, CPU is running. [ 0.000000] Machine is in M mode. [ 0.000000] Cpu 0, Nasid 0x0: partnum 0x0 is [ 0.000000] is xbow [ 0.000000] Cpu 0, Nasid 0x0, widget 0x8 (partnum 0xc102) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xb (partnum 0xc002) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xc (partnum 0xc002) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xd (partnum 0xc003) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xf (partnum 0xc002) is [ 0.000000] CPU 0 clock is 500MHz. [ 0.000000] Determined physical RAM map: [ 0.000000] memory: 00000000006e0000 @ 0000000000010000 (usable) [ 0.000000] memory: 0000000000050000 @ 00000000006f0000 (usable after init) [ 0.000000] cma: Reserved 512 MiB at 0x00000001e0000000 [ 0.000000] REPLICATION: ON nasid 0, ktext from nasid 0, kdata from nasid 0 [ 0.000000] REPLICATION: ON nasid 1, ktext from nasid 0, kdata from nasid 0 [ 0.000000] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 0.000000] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 0.000000] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 0.000000] Zone ranges: [ 0.000000] Normal [mem 0x0000000000000000-0x00000001ffffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000000000-0x00000000ffffffff] [ 0.000000] node 1: [mem 0x0000000100000000-0x00000001ffffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000ffffffff] [ 0.000000] On node 0 totalpages: 65536 [ 0.000000] free_area_init_node: node 0, pgdat a8000000007f0000, node_mem_map a800000001000000 [ 0.000000] Normal zone: 64 pages used for memmap [ 0.000000] Normal zone: 0 pages reserved [ 0.000000] Normal zone: 65536 pages, LIFO batch:1 [ 0.000000] Initmem setup node 1 [mem 0x0000000100000000-0x00000001ffffffff] [ 0.000000] On node 1 totalpages: 65536 [ 0.000000] free_area_init_node: node 1, pgdat a800000100030000, node_mem_map a800000100050000 [ 0.000000] Normal zone: 64 pages used for memmap [ 0.000000] Normal zone: 0 pages reserved [ 0.000000] Normal zone: 65536 pages, LIFO batch:1 [ 0.000000] percpu: Embedded 2 pages/cpu @a800000001430000 s60064 r0 d71008 u131072 [ 0.000000] pcpu-alloc: s60064 r0 d71008 u131072 alloc=2*65536 [ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [ 0.000000] Built 2 zonelists in Node order, mobility grouping on. Total pages: 130944 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: root=dksc(0,1,0) console=ttyS0,9600 root=/dev/md0 [ 0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes) [ 0.000000] Memory: 7846784K/8388608K available (5172K kernel code, 678K rwdata, 1116K rodata, 320K init, 700K bss, 17536K reserved, 524288K cma-reserved) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4 [ 0.000000] NR_IRQS:256 [ 0.000000] clocksource: HUB-RT: mask: 0xfffffffffffff max_cycles: 0x127350b88, max_idle_ns: 1763180808480 ns [ 0.000012] sched_clock: 52 bits at 1250kHz, resolution 800ns, wraps every 4398046510800ns [ 0.103175] Console: colour dummy device 80x25 [ 0.156244] Calibrating delay loop... 749.56 BogoMIPS (lpj=3747840) [ 0.318496] pid_max: default: 32768 minimum: 301 [ 0.378324] Dentry cache hash table entries: 1048576 (order: 7, 8388608 bytes) [ 0.577930] Inode-cache hash table entries: 524288 (order: 6, 4194304 bytes) [ 0.720390] Mount-cache hash table entries: 16384 (order: 1, 131072 bytes) [ 0.801233] Mountpoint-cache hash table entries: 16384 (order: 1, 131072 bytes) [ 0.900780] Checking for the daddi bug... no. [ 0.957828] smp: Bringing up secondary CPUs ... [ 1.012770] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 1.012786] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 1.012794] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 1.015959] CPU 1 clock is 500MHz. [ 1.015981] CPU1 revision is: 00000f14 (R14000) [ 1.015984] FPU revision is: 00000900 [ 1.054076] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 1.054096] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 1.054105] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 1.057333] Cpu 2, Nasid 0x1: partnum 0x0 is [ 1.057356] is xbow [ 1.057400] CPU 2 clock is 500MHz. [ 1.057417] CPU2 revision is: 00000f14 (R14000) [ 1.057421] FPU revision is: 00000900 [ 1.094440] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 1.094459] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 1.094468] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 1.097704] CPU 3 clock is 500MHz. [ 1.097732] CPU3 revision is: 00000f14 (R14000) [ 1.097736] FPU revision is: 00000900 [ 1.131286] smp: Brought up 2 nodes, 4 CPUs [ 2.425108] devtmpfs: initialized [ 2.464264] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 2.585991] xor: measuring software checksum speed [ 2.741920] 8regs : 710.400 MB/sec [ 2.890564] 8regs_prefetch: 684.800 MB/sec [ 3.043377] 32regs : 659.200 MB/sec [ 3.171273] random: fast init done [ 3.191971] 32regs_prefetch: 640.000 MB/sec [ 3.191979] xor: using function: 8regs (710.400 MB/sec) [ 3.329553] NET: Registered protocol family 16 [ 3.442067] cpuidle: using governor ladder [ 3.571538] cpuidle: using governor menu [ 3.907402] raid6: int64x1 gen() 309 MB/s [ 4.127084] raid6: int64x1 xor() 62 MB/s [ 4.346824] raid6: int64x2 gen() 444 MB/s [ 4.566655] raid6: int64x2 xor() 152 MB/s [ 4.786336] raid6: int64x4 gen() 387 MB/s [ 5.006111] raid6: int64x4 xor() 141 MB/s [ 5.225866] raid6: int64x8 gen() 315 MB/s [ 5.445840] raid6: int64x8 xor() 116 MB/s [ 5.495273] raid6: using algorithm int64x2 gen() 444 MB/s [ 5.560206] raid6: .... xor() 152 MB/s, rmw enabled [ 5.618856] raid6: using intx1 recovery algorithm [ 5.675924] SCSI subsystem initialized [ 5.721236] PCI host bridge to bus 0002:00 [ 5.769704] pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff] [ 5.869182] pci_bus 0002:00: root bus resource [io 0x920000000fa00000-0x920000000fbfffff] [ 5.968680] pci_bus 0002:00: root bus resource [bus 02-ff] [ 6.034717] pci 0002:00:00.0: [1077:1020] type 00 class 0x010000 [ 6.034773] pci 0002:00:00.0: reg 0x10: [io 0xf200000-0xf2000ff] [ 6.034800] pci 0002:00:00.0: reg 0x14: [mem 0x0f200000-0x0f200fff] [ 6.034893] pci 0002:00:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref] [ 6.035175] pci 0002:00:01.0: [1077:1020] type 00 class 0x010000 [ 6.035213] pci 0002:00:01.0: reg 0x10: [io 0xf400000-0xf4000ff] [ 6.035240] pci 0002:00:01.0: reg 0x14: [mem 0x0f400000-0x0f400fff] [ 6.035333] pci 0002:00:01.0: reg 0x30: [mem 0x00000000-0x0000ffff pref] [ 6.035599] pci 0002:00:02.0: [10a9:0003] type 00 class 0xff0000 [ 6.035641] pci 0002:00:02.0: reg 0x10: [mem 0x0f600000-0x0f6fffff] [ 6.036013] pci 0002:00:06.0: [10a9:0003] type 00 class 0xff0000 [ 6.036056] pci 0002:00:06.0: reg 0x10: [mem 0x0fa00000-0x0fafffff] [ 6.036401] pci 0002:00:07.0: [10a9:0005] type 00 class 0x000000 [ 6.036433] pci 0002:00:07.0: reg 0x10: [mem 0x00000000-0x00001fff] [ 6.036933] PCI host bridge to bus 0001:00 [ 6.084311] pci_bus 0001:00: root bus resource [mem 0x920000000c200000-0x920000000c9fffff] [ 6.183803] pci_bus 0001:00: root bus resource [io 0x920000000ca00000-0x920000000cbfffff] [ 6.283297] pci_bus 0001:00: root bus resource [bus 01-ff] [ 6.349329] pci 0001:00:00.0: [10a9:0002] type 00 class 0xff0000 [ 6.349369] pci 0001:00:00.0: reg 0x10: [mem 0x00000000-0x07ffffff] [ 6.349771] pci 0001:00:01.0: [10a9:0002] type 00 class 0xff0000 [ 6.349811] pci 0001:00:01.0: reg 0x10: [mem 0x00000000-0x07ffffff] [ 6.350376] PCI host bridge to bus 0000:00 [ 6.398511] pci_bus 0000:00: root bus resource [mem 0x920000000b200000-0x920000000b9fffff] [ 6.498002] pci_bus 0000:00: root bus resource [io 0x920000000ba00000-0x920000000bbfffff] [ 6.597500] pci_bus 0000:00: root bus resource [bus 00-ff] [ 6.664580] clocksource: Switched to clocksource HUB-RT [ 6.737340] NET: Registered protocol family 2 [ 6.788631] TCP established hash table entries: 65536 (order: 3, 524288 bytes) [ 6.877860] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes) [ 6.961938] TCP: Hash tables configured (established 65536 bind 65536) [ 7.038796] UDP hash table entries: 4096 (order: 1, 131072 bytes) [ 7.112666] UDP-Lite hash table entries: 4096 (order: 1, 131072 bytes) [ 7.191678] NET: Registered protocol family 1 [ 7.243000] PCI: CLS 256 bytes, default 128 [ 7.244032] futex hash table entries: 1024 (order: 1, 131072 bytes) [ 7.319896] workingset: timestamp_bits=40 max_order=17 bucket_order=0 [ 7.396183] zbud: loaded [ 7.428650] SGI XFS with ACLs, security attributes, realtime, no debug enabled [ 7.519969] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) [ 7.607150] io scheduler noop registered [ 7.654268] io scheduler deadline registered [ 7.705641] io scheduler cfq registered (default) [ 7.825919] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.915337] loop: module loaded [ 7.951330] qla1280: QLA1040 found on PCI bus 0, dev 0 [ 8.013015] PCI: Enabling device 0002:00:00.0 (0006 -> 0007) [ 8.583489] random: crng init done [ 8.843500] scsi(0:0): Resetting SCSI BUS [ 11.914788] scsi host0: QLogic QLA1040 PCI to SCSI Host Adapter Firmware version: 7.65.06, Driver version 3.27.1 [ 12.068902] qla1280: QLA1040 found on PCI bus 0, dev 1 [ 12.128824] PCI: Enabling device 0002:00:01.0 (0006 -> 0007) [ 12.240023] scsi 0:0:1:0: Direct-Access SEAGATE SX150176LC BA12 PQ: 0 ANSI: 2 [ 12.336821] scsi(0:0:1:0): [ 12.336828] Sync: period 10, offset 12 [ 12.367970] , Wide [ 12.414054] , Tagged queuing: depth 31 [ 12.507613] scsi 0:0:2:0: Direct-Access SEAGATE SX150176LC BA08 PQ: 0 ANSI: 2 [ 12.604289] scsi(0:0:2:0): [ 12.604296] Sync: period 10, offset 12 [ 12.635564] , Wide [ 12.681648] , Tagged queuing: depth 31 [ 12.775169] scsi 0:0:3:0: Direct-Access SEAGATE SX150176LC BA08 PQ: 0 ANSI: 2 [ 12.871848] scsi(0:0:3:0): [ 12.871856] Sync: period 10, offset 12 [ 12.903159] , Wide [ 12.949242] , Tagged queuing: depth 31 [ 12.961432] scsi(1:0): Resetting SCSI BUS [ 15.858292] scsi 0:0:4:0: Direct-Access SEAGATE SX150176LC BA11 PQ: 0 ANSI: 2 [ 15.954656] scsi(0:0:4:0): [ 15.954664] Sync: period 10, offset 12 [ 15.986201] , Wide [ 16.032286] , Tagged queuing: depth 31 [ 16.119472] scsi host1: QLogic QLA1040 PCI to SCSI Host Adapter Firmware version: 7.65.06, Driver version 3.27.1 [ 16.275065] console [ttyS0] disabled [ 16.336528] serial8250: ttyS0 at MMIO 0x920000000f620178 (irq = 0, base_baud = 458333) is a 16550A [ 16.442612] console [ttyS0] enabled [ 16.525366] bootconsole [early0] disabled [ 16.646656] serial8250: ttyS1 at MMIO 0x920000000f620170 (irq = 0, base_baud = 458333) is a 16550A [ 16.754744] scsi 0:0:5:0: Direct-Access SEAGATE SX150176LC BA11 PQ: 0 ANSI: 2 [ 16.855313] scsi(0:0:5:0): [ 16.855329] Sync: period 10, offset 12 [ 16.887944] , Wide [ 16.889739] Found DS1981U NIC [ 16.889748] registration number 5e:57:03:00:70:5e, CRC 52 [ 16.889750] . [ 16.905995] Ethernet address is 08:00:69:05:64:74. [ 16.907374] ioc3-eth 0002:00:02.0 eth0: link up, 100Mbps, full-duplex, lpa 0x8DE1 [ 16.907381] eth0: Using PHY 31, vendor 0x20005c0, model 0, rev 0. [ 16.907384] eth0: IOC3 SSRAM has 128 kbyte. [ 16.929432] serial8250: ttyS2 at MMIO 0x920000000fa20178 (irq = 0, base_baud = 458333) is a 16550A [ 16.950337] serial8250: ttyS3 at MMIO 0x920000000fa20170 (irq = 0, base_baud = 458333) is a 16550A [ 16.954048] NIC search failed (not fatal). [ 16.956507] NIC search failed (not fatal). [ 16.957008] Failed to read MAC address [ 16.957012] Ethernet address is 00:00:00:00:00:00. [ 16.958135] ioc3-eth 0002:00:06.0 eth1: link down [ 16.958142] eth1: Using PHY 31, vendor 0x0, model 0, rev 0. [ 16.958145] eth1: IOC3 SSRAM has 64 kbyte. [ 16.959185] NET: Registered protocol family 17 [ 16.959732] rtc-m48t35 rtc-m48t35: rtc core: registered m48t35 as rtc0 [ 18.073532] , Tagged queuing: depth 31 [ 18.136683] rtc-m48t35 rtc-m48t35: setting system clock to 2017-02-08 08:57:44 UTC (1486544264) [ 18.249152] scsi 0:0:6:0: CD-ROM TOSHIBA CD-ROM XM-5701TA 0167 PQ: 0 ANSI: 2 [ 18.347391] scsi(0:0:6:0): [ 18.347396] Sync: period 10, offset 12 [ 19.841973] sd 0:0:3:0: [sdc] 97693755 512-byte logical blocks: (50.0 GB/46.6 GiB) [ 19.844598] sr 0:0:6:0: [sr0] scsi-1 drive [ 19.844606] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 19.845324] sr 0:0:6:0: Attached scsi CD-ROM sr0 [ 19.934698] sd 0:0:2:0: [sdb] 97693755 512-byte logical blocks: (50.0 GB/46.6 GiB) [ 19.934807] sd 0:0:1:0: [sda] 97693755 512-byte logical blocks: (50.0 GB/46.6 GiB) [ 19.934986] sd 0:0:4:0: [sdd] 97693755 512-byte logical blocks: (50.0 GB/46.6 GiB) [ 19.935572] sd 0:0:5:0: [sde] 97693755 512-byte logical blocks: (50.0 GB/46.6 GiB) [ 19.936993] sd 0:0:2:0: [sdb] Write Protect is off [ 19.937004] sd 0:0:2:0: [sdb] Mode Sense: cb 00 10 08 [ 19.937556] sd 0:0:1:0: [sda] Write Protect is off [ 19.937565] sd 0:0:1:0: [sda] Mode Sense: cb 00 10 08 [ 19.937803] sd 0:0:4:0: [sdd] Write Protect is off [ 19.937812] sd 0:0:4:0: [sdd] Mode Sense: cb 00 10 08 [ 19.938310] sd 0:0:5:0: [sde] Write Protect is off [ 19.938320] sd 0:0:5:0: [sde] Mode Sense: cb 00 10 08 [ 19.939948] sd 0:0:2:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 19.940496] sd 0:0:1:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 19.940944] sd 0:0:4:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 19.941212] sd 0:0:5:0: [sde] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 20.002748] sda: sda1 sda2 sda3 sda4 sda5 sda6 sda9 sda11 [ 20.013020] sdd: sdd1 sdd2 sdd3 sdd4 sdd5 sdd6 sdd9 sdd11 [ 20.014354] sdb: sdb1 sdb2 sdb3 sdb4 sdb5 sdb6 sdb9 sdb11 [ 20.024588] sd 0:0:5:0: [sde] Attached SCSI disk [ 20.025291] sd 0:0:1:0: [sda] Attached SCSI disk [ 20.026150] sd 0:0:2:0: [sdb] Attached SCSI disk [ 20.026569] sd 0:0:4:0: [sdd] Attached SCSI disk [ 21.481104] sd 0:0:3:0: [sdc] Write Protect is off [ 21.538895] sd 0:0:3:0: [sdc] Mode Sense: cb 00 10 08 [ 21.541780] sd 0:0:3:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 21.694266] sdc: sdc1 sdc2 sdc3 sdc4 sdc5 sdc6 sdc9 sdc11 [ 21.770364] sd 0:0:3:0: [sdc] Attached SCSI disk [ 21.826524] md: Waiting for all devices to be available before autodetect [ 21.908398] md: If you don't use raid, use raid=noautodetect [ 21.977995] md: Autodetecting RAID arrays. [ 22.112008] md: invalid raid superblock magic on sdd3 [ 22.172984] md: sdd3 does not have a valid v0.90 superblock, not importing! [ 22.272565] md: invalid raid superblock magic on sdd4 [ 22.333508] md: sdd4 does not have a valid v0.90 superblock, not importing! [ 22.543056] md: autorun ... [ 22.576739] md: sdc5 has different UUID to sdc6 [ 22.631360] md: sdc4 has different UUID to sdc6 [ 22.686052] md: sdc3 has different UUID to sdc6 [ 22.740709] md: sdc1 has different UUID to sdc6 [ 22.795447] md: sdd5 has different UUID to sdc6 [ 22.850064] md: sdb5 has different UUID to sdc6 [ 22.904724] md: sdb4 has different UUID to sdc6 [ 22.959392] md: sdb3 has different UUID to sdc6 [ 23.014086] md: sdb1 has different UUID to sdc6 [ 23.068748] md: sdd1 has different UUID to sdc6 [ 23.123434] md: sda5 has different UUID to sdc6 [ 23.178092] md: sda4 has different UUID to sdc6 [ 23.232752] md: sda3 has different UUID to sdc6 [ 23.287422] md: sda1 has different UUID to sdc6 [ 23.343672] md: running: <sdc6><sdd6><sdb6><sda6> [ 23.402609] md/raid:md4: device sdc6 operational as raid disk 2 [ 23.473960] md/raid:md4: device sdb6 operational as raid disk 1 [ 23.545358] md/raid:md4: device sda6 operational as raid disk 0 [ 23.620651] md/raid:md4: raid level 5 active with 3 out of 3 devices, algorithm 2 [ 23.731897] md4: detected capacity change from 0 to 1940914176 [ 23.802401] md: sdc4 has different UUID to sdc5 [ 23.857115] md: sdc3 has different UUID to sdc5 [ 23.911764] md: sdc1 has different UUID to sdc5 [ 23.966428] md: sdd5 has different UUID to sdc5 [ 24.021133] md: sdb4 has different UUID to sdc5 [ 24.075753] md: sdb3 has different UUID to sdc5 [ 24.130442] md: sdb1 has different UUID to sdc5 [ 24.185126] md: sdd1 has different UUID to sdc5 [ 24.239785] md: sda4 has different UUID to sdc5 [ 24.294452] md: sda3 has different UUID to sdc5 [ 24.349137] md: sda1 has different UUID to sdc5 [ 24.405311] md: running: <sdc5><sdb5><sda5> [ 24.457636] md/raid:md3: device sdc5 operational as raid disk 2 [ 24.529063] md/raid:md3: device sdb5 operational as raid disk 1 [ 24.600456] md/raid:md3: device sda5 operational as raid disk 0 [ 24.675431] md/raid:md3: raid level 5 active with 3 out of 3 devices, algorithm 2 [ 24.789229] md3: detected capacity change from 0 to 4013686784 [ 24.859652] md: sdc3 has different UUID to sdc4 [ 24.914292] md: sdc1 has different UUID to sdc4 [ 24.968933] md: sdd5 has different UUID to sdc4 [ 25.023626] md: sdb3 has different UUID to sdc4 [ 25.078273] md: sdb1 has different UUID to sdc4 [ 25.132940] md: sdd1 has different UUID to sdc4 [ 25.187633] md: sda3 has different UUID to sdc4 [ 25.242286] md: sda1 has different UUID to sdc4 [ 25.298457] md: running: <sdc4><sdb4><sda4> [ 25.350924] md/raid:md2: device sdc4 operational as raid disk 2 [ 25.422336] md/raid:md2: device sdb4 operational as raid disk 1 [ 25.493740] md/raid:md2: device sda4 operational as raid disk 0 [ 25.568663] md/raid:md2: raid level 5 active with 3 out of 3 devices, algorithm 2 [ 25.681053] md2: detected capacity change from 0 to 80015261696 [ 25.752544] md: sdc1 has different UUID to sdc3 [ 25.807252] md: sdd5 has different UUID to sdc3 [ 25.861928] md: sdb1 has different UUID to sdc3 [ 25.916728] md: sdd1 has different UUID to sdc3 [ 25.971376] md: sda1 has different UUID to sdc3 [ 26.027416] md: running: <sdc3><sdb3><sda3> [ 26.079824] md/raid:md1: device sdc3 operational as raid disk 2 [ 26.151284] md/raid:md1: device sdb3 operational as raid disk 1 [ 26.222664] md/raid:md1: device sda3 operational as raid disk 0 [ 26.297448] md/raid:md1: raid level 5 active with 3 out of 3 devices, algorithm 2 [ 26.400761] md1: detected capacity change from 0 to 12008554496 [ 26.472253] md: sdd5 has different UUID to sdc1 [ 26.527422] md: running: <sdc1><sdb1><sdd1><sda1> [ 26.586161] md/raid:md0: device sdc1 operational as raid disk 2 [ 26.657569] md/raid:md0: device sdb1 operational as raid disk 1 [ 26.728956] md/raid:md0: device sda1 operational as raid disk 0 [ 26.804247] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2 [ 26.909265] md0: detected capacity change from 0 to 1019740160 [ 26.979722] md: md3 already running, cannot run sdd5 [ 27.039662] md: ... autorun DONE. [ 27.151464] UDF-fs: warning (device md0): udf_fill_super: No partition found (2) [ 27.268270] XFS (md0): Mounting V5 Filesystem [ 30.003679] XFS (md0): Ending clean mount [ 30.052192] VFS: Mounted root (xfs filesystem) readonly on device 9:0. [ 30.136790] devtmpfs: mounted [ 30.174297] Freeing unused kernel memory: 320K [ 30.227891] This architecture does not have kernel memory protection. [ 40.863143] udevd[585]: starting version 3.2.1 [ 40.938693] udevd[585]: specified group 'video' unknown [ 40.939862] udevd[585]: specified group 'tape' unknown [ 40.997900] udevd[586]: starting eudev-3.2.1 [ 41.517564] udevd[586]: specified group 'video' unknown [ 41.518745] udevd[586]: specified group 'tape' unknown [ 48.072540] Adding 449728k swap on /dev/sda2. Priority:1 extents:1 across:449728k [ 48.111252] Adding 449728k swap on /dev/sdb2. Priority:1 extents:1 across:449728k [ 48.219132] Adding 449728k swap on /dev/sdc2. Priority:1 extents:1 across:449728k [ 48.685496] XFS (md1): Mounting V5 Filesystem [ 51.388111] XFS (md1): Ending clean mount [ 51.577772] XFS (md2): Mounting V5 Filesystem [ 54.300352] XFS (md2): Ending clean mount [ 54.600708] XFS (md3): Mounting V5 Filesystem [ 57.382781] XFS (md3): Ending clean mount [ 57.485516] XFS (md4): Mounting V5 Filesystem [ 60.223068] XFS (md4): Ending clean mount [-- Attachment #6: ip27-dmesg-size_assign_pci-20170208.txt --] [-- Type: text/plain, Size: 14335 bytes --] [ 0.000000] Linux version 4.10.0-rc7 (root@helcaraxe) (gcc version 6.3.0 (Gentoo Hardened 6.3.0 p1.0) ) #2 SMP Wed Feb 8 04:07:11 EST 2017 [ 0.000000] ARCH: SGI-IP27 [ 0.000000] PROMLIB: ARC firmware Version 64 Revision 0 [ 0.000000] Discovered 4 cpus on 2 nodes [ 0.000000] node_distance: router_a NULL [ 0.000000] node_distance: router_a NULL [ 0.000000] node_distance: router_a NULL [ 0.000000] node_distance: router_a NULL [ 0.000000] ************** Topology ******************** [ 0.000000] [ 0.000000] 00 [ 0.000000] 01 [ 0.000000] [ 0.000000] 00 [ 0.000000] 255 [ 0.000000] 255 [ 0.000000] [ 0.000000] 01 [ 0.000000] 255 [ 0.000000] 255 [ 0.000000] [ 0.000000] bootconsole [early0] enabled [ 0.000000] CPU0 revision is: 00000f14 (R14000) [ 0.000000] FPU revision is: 00000900 [ 0.000000] Checking for the multiply/shift bug... no. [ 0.000000] Checking for the daddiu bug... no. [ 0.000000] IP27: Running on node 0. [ 0.000000] Node 0 has a primary CPU, CPU is running. [ 0.000000] Node 0 has a secondary CPU, CPU is running. [ 0.000000] Machine is in M mode. [ 0.000000] Cpu 0, Nasid 0x0: partnum 0x0 is [ 0.000000] is xbow [ 0.000000] Cpu 0, Nasid 0x0, widget 0x8 (partnum 0xc102) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xb (partnum 0xc002) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xc (partnum 0xc002) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xd (partnum 0xc003) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xf (partnum 0xc002) is [ 0.000000] CPU 0 clock is 500MHz. [ 0.000000] Determined physical RAM map: [ 0.000000] memory: 00000000006e0000 @ 0000000000010000 (usable) [ 0.000000] memory: 0000000000050000 @ 00000000006f0000 (usable after init) [ 0.000000] cma: Reserved 512 MiB at 0x00000001e0000000 [ 0.000000] REPLICATION: ON nasid 0, ktext from nasid 0, kdata from nasid 0 [ 0.000000] REPLICATION: ON nasid 1, ktext from nasid 0, kdata from nasid 0 [ 0.000000] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 0.000000] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 0.000000] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 0.000000] Zone ranges: [ 0.000000] Normal [mem 0x0000000000000000-0x00000001ffffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000000000-0x00000000ffffffff] [ 0.000000] node 1: [mem 0x0000000100000000-0x00000001ffffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000ffffffff] [ 0.000000] Initmem setup node 1 [mem 0x0000000100000000-0x00000001ffffffff] [ 0.000000] percpu: Embedded 2 pages/cpu @a800000001430000 s60064 r0 d71008 u131072 [ 0.000000] Built 2 zonelists in Node order, mobility grouping on. Total pages: 130944 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: root=dksc(0,1,0) console=ttyS0,9600 root=/dev/md0 [ 0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes) [ 0.000000] Memory: 7846784K/8388608K available (5172K kernel code, 678K rwdata, 1116K rodata, 320K init, 700K bss, 17536K reserved, 524288K cma-reserved) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4 [ 0.000000] NR_IRQS:256 [ 0.000000] clocksource: HUB-RT: mask: 0xfffffffffffff max_cycles: 0x127350b88, max_idle_ns: 1763180808480 ns [ 0.000012] sched_clock: 52 bits at 1250kHz, resolution 800ns, wraps every 4398046510800ns [ 0.103183] Console: colour dummy device 80x25 [ 0.156258] Calibrating delay loop... 749.56 BogoMIPS (lpj=3747840) [ 0.318497] pid_max: default: 32768 minimum: 301 [ 0.378347] Dentry cache hash table entries: 1048576 (order: 7, 8388608 bytes) [ 0.577889] Inode-cache hash table entries: 524288 (order: 6, 4194304 bytes) [ 0.720380] Mount-cache hash table entries: 16384 (order: 1, 131072 bytes) [ 0.801235] Mountpoint-cache hash table entries: 16384 (order: 1, 131072 bytes) [ 0.900788] Checking for the daddi bug... no. [ 0.957852] smp: Bringing up secondary CPUs ... [ 1.012850] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 1.012866] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 1.012873] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 1.016040] CPU 1 clock is 500MHz. [ 1.016063] CPU1 revision is: 00000f14 (R14000) [ 1.016067] FPU revision is: 00000900 [ 1.054189] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 1.054209] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 1.054219] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 1.057447] Cpu 2, Nasid 0x1: partnum 0x0 is [ 1.057471] is xbow [ 1.057513] CPU 2 clock is 500MHz. [ 1.057532] CPU2 revision is: 00000f14 (R14000) [ 1.057536] FPU revision is: 00000900 [ 1.094532] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 1.094550] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 1.094560] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 1.097795] CPU 3 clock is 500MHz. [ 1.097823] CPU3 revision is: 00000f14 (R14000) [ 1.097827] FPU revision is: 00000900 [ 1.131387] smp: Brought up 2 nodes, 4 CPUs [ 2.425207] devtmpfs: initialized [ 2.464376] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 2.586105] xor: measuring software checksum speed [ 2.742023] 8regs : 710.400 MB/sec [ 2.890670] 8regs_prefetch: 684.800 MB/sec [ 3.043481] 32regs : 659.200 MB/sec [ 3.171390] random: fast init done [ 3.192078] 32regs_prefetch: 640.000 MB/sec [ 3.192086] xor: using function: 8regs (710.400 MB/sec) [ 3.329769] NET: Registered protocol family 16 [ 3.442281] cpuidle: using governor ladder [ 3.549724] cpuidle: using governor menu [ 3.902207] raid6: int64x1 gen() 304 MB/s [ 4.122196] raid6: int64x1 xor() 64 MB/s [ 4.341936] raid6: int64x2 gen() 446 MB/s [ 4.561720] raid6: int64x2 xor() 152 MB/s [ 4.781428] raid6: int64x4 gen() 386 MB/s [ 5.001245] raid6: int64x4 xor() 141 MB/s [ 5.221081] raid6: int64x8 gen() 315 MB/s [ 5.440814] raid6: int64x8 xor() 116 MB/s [ 5.490252] raid6: using algorithm int64x2 gen() 446 MB/s [ 5.555184] raid6: .... xor() 152 MB/s, rmw enabled [ 5.613836] raid6: using intx1 recovery algorithm [ 5.670909] SCSI subsystem initialized [ 5.716212] PCI host bridge to bus 0002:00 [ 5.764684] pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff] [ 5.864163] pci_bus 0002:00: root bus resource [io 0x920000000fa00000-0x920000000fbfffff] [ 5.963663] pci_bus 0002:00: root bus resource [bus 02-ff] [ 6.031825] pci 0002:00:02.0: BAR 0: no space for [mem size 0x00100000] [ 6.109557] pci 0002:00:02.0: BAR 0: failed to assign [mem size 0x00100000] [ 6.193341] pci 0002:00:06.0: BAR 0: no space for [mem size 0x00100000] [ 6.272936] pci 0002:00:06.0: BAR 0: failed to assign [mem size 0x00100000] [ 6.356740] pci 0002:00:00.0: BAR 6: no space for [mem size 0x00010000 pref] [ 6.441558] pci 0002:00:00.0: BAR 6: failed to assign [mem size 0x00010000 pref] [ 6.530583] pci 0002:00:01.0: BAR 6: no space for [mem size 0x00010000 pref] [ 6.615415] pci 0002:00:01.0: BAR 6: failed to assign [mem size 0x00010000 pref] [ 6.704440] pci 0002:00:00.0: BAR 1: no space for [mem size 0x00001000] [ 6.784036] pci 0002:00:00.0: BAR 1: failed to assign [mem size 0x00001000] [ 6.867824] pci 0002:00:01.0: BAR 1: no space for [mem size 0x00001000] [ 6.947420] pci 0002:00:01.0: BAR 1: failed to assign [mem size 0x00001000] [ 7.031210] pci 0002:00:00.0: BAR 0: no space for [io size 0x0100] [ 7.106623] pci 0002:00:00.0: BAR 0: failed to assign [io size 0x0100] [ 7.186215] pci 0002:00:01.0: BAR 0: no space for [io size 0x0100] [ 7.261620] pci 0002:00:01.0: BAR 0: failed to assign [io size 0x0100] [ 7.341358] PCI host bridge to bus 0001:00 [ 7.390455] pci_bus 0001:00: root bus resource [mem 0x920000000c200000-0x920000000c9fffff] [ 7.489945] pci_bus 0001:00: root bus resource [io 0x920000000ca00000-0x920000000cbfffff] [ 7.589441] pci_bus 0001:00: root bus resource [bus 01-ff] [ 7.656392] pci 0001:00:00.0: BAR 0: no space for [mem size 0x08000000] [ 7.735022] pci 0001:00:00.0: BAR 0: failed to assign [mem size 0x08000000] [ 7.818808] pci 0001:00:01.0: BAR 0: no space for [mem size 0x08000000] [ 7.898403] pci 0001:00:01.0: BAR 0: failed to assign [mem size 0x08000000] [ 7.982315] PCI host bridge to bus 0000:00 [ 8.031416] pci_bus 0000:00: root bus resource [mem 0x920000000b200000-0x920000000b9fffff] [ 8.130914] pci_bus 0000:00: root bus resource [io 0x920000000ba00000-0x920000000bbfffff] [ 8.230410] pci_bus 0000:00: root bus resource [bus 00-ff] [ 8.297487] clocksource: Switched to clocksource HUB-RT [ 8.370358] NET: Registered protocol family 2 [ 8.421632] TCP established hash table entries: 65536 (order: 3, 524288 bytes) [ 8.510880] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes) [ 8.594960] TCP: Hash tables configured (established 65536 bind 65536) [ 8.671821] UDP hash table entries: 4096 (order: 1, 131072 bytes) [ 8.745673] UDP-Lite hash table entries: 4096 (order: 1, 131072 bytes) [ 8.824692] NET: Registered protocol family 1 [ 8.877054] futex hash table entries: 1024 (order: 1, 131072 bytes) [ 8.953001] workingset: timestamp_bits=40 max_order=17 bucket_order=0 [ 9.029206] zbud: loaded [ 9.061660] SGI XFS with ACLs, security attributes, realtime, no debug enabled [ 9.153560] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) [ 9.240707] io scheduler noop registered [ 9.287809] io scheduler deadline registered [ 9.339188] io scheduler cfq registered (default) [ 9.460018] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 13.353990] loop: module loaded [ 13.389999] qla1280: QLA1040 found on PCI bus 0, dev 0 [ 13.452304] qla1280 0002:00:00.0: can't ioremap BAR 1: [??? 0x00000000 flags 0x0] [ 13.541691] qla1280: Unable to map I/O memory [ 13.594332] qla1280: QLA1040 found on PCI bus 0, dev 1 [ 13.656402] qla1280 0002:00:01.0: can't ioremap BAR 1: [??? 0x00000000 flags 0x0] [ 13.745916] qla1280: Unable to map I/O memory [ 13.809256] Data bus error, epc == a8000000003fdb74, ra == a8000000003fe4f0 [ 13.891137] Oops[#1]: [ 13.918372] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.10.0-rc7 #2 [ 13.993771] task: a8000000e077dc80 task.stack: a8000000e0780000 [ 14.064988] $ 0 : 0000000000000000 ffffffff94001ce0 00000000000800c0 ffffffffefffffff [ 14.161342] $ 4 : 9200000000020178 ffffffff94001ce1 ffffffffa8000000 0000000000000000 [ 14.257697] $ 8 : 0000000000000001 a8000001c09d4070 a8000001c09d4060 ffffffffffffffff [ 14.354051] $12 : 0000000000000000 a80000000032f1ec 0000000000000000 a8000000e078fd74 [ 14.450406] $16 : 9200000000000000 a8000001c08b4800 0000000000020000 a8000001c09d4700 [ 14.546761] $20 : 0000000000000000 a8000000007e0000 a8000001c09d4000 a8000000006f045c [ 14.643116] $24 : 0000000000000000 0000000000680000 [ 14.739472] $28 : a8000000e0780000 a8000000e078fc30 a800000000760000 a8000000003fe4f0 [ 14.835827] Hi : 0000000000000000 [ 14.878768] Lo : 000000000000b400 [ 14.921724] epc : a8000000003fdb74 ioc3_probe+0x1dc/0xda0 [ 14.988744] ra : a8000000003fe4f0 ioc3_probe+0xb58/0xda0 [ 15.055767] Status: 94001ce3 KX SX UX KERNEL EXL IE [ 15.115467] Cause : 0000d01c (ExcCode 07) [ 15.163644] PrId : 00000f14 (R14000) [ 15.207636] Process swapper/0 (pid: 1, threadinfo=a8000000e0780000, task=a8000000e077dc80, tls=0000000000000000) [ 15.330169] Stack : a8000001c06b6978 a8000001c0d6cc60 a8000001c08b4800 a8000001c08b4800 [ 15.426524] a8000000006a2dc0 a800000000696bd0 0000000000000000 a8000000007e0000 [ 15.522879] a8000000006a2e28 a8000000006f045c a800000000760000 a800000000371f94 [ 15.619234] a8000001c08b4898 a800000000372a28 a8000000006a2dc0 a8000001c08b4800 [ 15.715588] a800000000588550 a8000000003c2824 a80000000062a410 a8000000007d0000 [ 15.811944] a8000001c08b4898 a8000000007e0000 0000000000000000 a8000000003c2e58 [ 15.908298] a8000001c08b4898 a8000000006a2e28 a8000001c08b48f8 a800000000696bd0 [ 16.004653] a80000000071bf30 a800000000760000 0000000000000007 a8000000003c30c8 [ 16.101008] 0000000000000000 a8000000006a2e28 a8000000003c2fd8 a8000000003c0ca4 [ 16.197364] a8000000e093a0a0 a8000001c0708568 a8000000006a2e28 a8000001c0d19f00 [ 16.293718] ... [ 16.323046] Call Trace: [ 16.352376] [<a8000000003fdb74>] ioc3_probe+0x1dc/0xda0 [ 16.415232] [<a800000000371f94>] local_pci_probe+0x2c/0x98 [ 16.481201] [<a800000000372a28>] pci_device_probe+0x140/0x1b8 [ 16.550325] [<a8000000003c2e58>] really_probe+0x1b0/0x330 [ 16.615259] [<a8000000003c30c8>] __driver_attach+0xf0/0xf8 [ 16.681248] [<a8000000003c0ca4>] bus_for_each_dev+0x6c/0xb8 [ 16.748272] [<a8000000003c1960>] bus_add_driver+0x1c8/0x330 [ 16.815301] [<a8000000003c3d34>] driver_register+0x84/0x150 [ 16.882340] [<a8000000006f109c>] do_one_initcall+0xc4/0x1b4 [ 16.949360] [<a8000000006f1478>] kernel_init_freeable+0x2ec/0x3e8 [ 17.022675] [<a80000000051fdd0>] kernel_init+0x10/0x1c0 [ 17.085519] [<a800000000025968>] ret_from_kernel_thread+0x14/0x1c [ 17.158823] Code: 8e020034 0204202d ae000058 <8e020058> ae00005c 8e02005c 8e0200b8 00431024 ae0200b8 [ 17.276122] [ 17.294049] ---[ end trace 027164ad3883b4c1 ]--- [ 17.349485] Kernel panic - not syncing: Fatal exception [ 17.412320] Reboot started from CPU 0 [-- Attachment #7: ip27-dmesg-claim_pci-20170208.txt --] [-- Type: text/plain, Size: 15400 bytes --] [ 0.000000] Linux version 4.10.0-rc7 (root@helcaraxe) (gcc version 6.3.0 (Gentoo Hardened 6.3.0 p1.0) ) #3 SMP Wed Feb 8 04:13:21 EST 2017 [ 0.000000] ARCH: SGI-IP27 [ 0.000000] PROMLIB: ARC firmware Version 64 Revision 0 [ 0.000000] Discovered 4 cpus on 2 nodes [ 0.000000] node_distance: router_a NULL [ 0.000000] node_distance: router_a NULL [ 0.000000] node_distance: router_a NULL [ 0.000000] node_distance: router_a NULL [ 0.000000] ************** Topology ******************** [ 0.000000] [ 0.000000] 00 [ 0.000000] 01 [ 0.000000] [ 0.000000] 00 [ 0.000000] 255 [ 0.000000] 255 [ 0.000000] [ 0.000000] 01 [ 0.000000] 255 [ 0.000000] 255 [ 0.000000] [ 0.000000] bootconsole [early0] enabled [ 0.000000] CPU0 revision is: 00000f14 (R14000) [ 0.000000] FPU revision is: 00000900 [ 0.000000] Checking for the multiply/shift bug... no. [ 0.000000] Checking for the daddiu bug... no. [ 0.000000] IP27: Running on node 0. [ 0.000000] Node 0 has a primary CPU, CPU is running. [ 0.000000] Node 0 has a secondary CPU, CPU is running. [ 0.000000] Machine is in M mode. [ 0.000000] Cpu 0, Nasid 0x0: partnum 0x0 is [ 0.000000] is xbow [ 0.000000] Cpu 0, Nasid 0x0, widget 0x8 (partnum 0xc102) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xb (partnum 0xc002) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xc (partnum 0xc002) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xd (partnum 0xc003) is [ 0.000000] Cpu 0, Nasid 0x0, widget 0xf (partnum 0xc002) is [ 0.000000] CPU 0 clock is 500MHz. [ 0.000000] Determined physical RAM map: [ 0.000000] memory: 00000000006e0000 @ 0000000000010000 (usable) [ 0.000000] memory: 0000000000050000 @ 00000000006f0000 (usable after init) [ 0.000000] cma: Reserved 512 MiB at 0x00000001e0000000 [ 0.000000] REPLICATION: ON nasid 0, ktext from nasid 0, kdata from nasid 0 [ 0.000000] REPLICATION: ON nasid 1, ktext from nasid 0, kdata from nasid 0 [ 0.000000] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 0.000000] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 0.000000] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 0.000000] Zone ranges: [ 0.000000] Normal [mem 0x0000000000000000-0x00000001ffffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000000000-0x00000000ffffffff] [ 0.000000] node 1: [mem 0x0000000100000000-0x00000001ffffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000ffffffff] [ 0.000000] Initmem setup node 1 [mem 0x0000000100000000-0x00000001ffffffff] [ 0.000000] percpu: Embedded 2 pages/cpu @a800000001430000 s60064 r0 d71008 u131072 [ 0.000000] Built 2 zonelists in Node order, mobility grouping on. Total pages: 130944 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: root=dksc(0,1,0) console=ttyS0,9600 root=/dev/md0 [ 0.000000] PID hash table entries: 4096 (order: -1, 32768 bytes) [ 0.000000] Memory: 7846784K/8388608K available (5172K kernel code, 678K rwdata, 1116K rodata, 320K init, 700K bss, 17536K reserved, 524288K cma-reserved) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4 [ 0.000000] NR_IRQS:256 [ 0.000000] clocksource: HUB-RT: mask: 0xfffffffffffff max_cycles: 0x127350b88, max_idle_ns: 1763180808480 ns [ 0.000012] sched_clock: 52 bits at 1250kHz, resolution 800ns, wraps every 4398046510800ns [ 0.103168] Console: colour dummy device 80x25 [ 0.156248] Calibrating delay loop... 749.56 BogoMIPS (lpj=3747840) [ 0.318496] pid_max: default: 32768 minimum: 301 [ 0.378314] Dentry cache hash table entries: 1048576 (order: 7, 8388608 bytes) [ 0.577879] Inode-cache hash table entries: 524288 (order: 6, 4194304 bytes) [ 0.720390] Mount-cache hash table entries: 16384 (order: 1, 131072 bytes) [ 0.801234] Mountpoint-cache hash table entries: 16384 (order: 1, 131072 bytes) [ 0.900782] Checking for the daddi bug... no. [ 0.957853] smp: Bringing up secondary CPUs ... [ 1.012864] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 1.012880] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 1.012888] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 1.016053] CPU 1 clock is 500MHz. [ 1.016076] CPU1 revision is: 00000f14 (R14000) [ 1.016080] FPU revision is: 00000900 [ 1.054200] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 1.054220] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 1.054229] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 1.057457] Cpu 2, Nasid 0x1: partnum 0x0 is [ 1.057480] is xbow [ 1.057523] CPU 2 clock is 500MHz. [ 1.057540] CPU2 revision is: 00000f14 (R14000) [ 1.057544] FPU revision is: 00000900 [ 1.094540] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes. [ 1.094559] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes [ 1.094568] Unified secondary cache 8192kB 2-way, linesize 128 bytes. [ 1.097801] CPU 3 clock is 500MHz. [ 1.097830] CPU3 revision is: 00000f14 (R14000) [ 1.097833] FPU revision is: 00000900 [ 1.131379] smp: Brought up 2 nodes, 4 CPUs [ 2.425205] devtmpfs: initialized [ 2.464371] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 2.586100] xor: measuring software checksum speed [ 2.741957] 8regs : 710.400 MB/sec [ 2.890575] 8regs_prefetch: 684.800 MB/sec [ 3.043334] 32regs : 659.200 MB/sec [ 3.171378] random: fast init done [ 3.191971] 32regs_prefetch: 640.000 MB/sec [ 3.191979] xor: using function: 8regs (710.400 MB/sec) [ 3.329660] NET: Registered protocol family 16 [ 3.442177] cpuidle: using governor ladder [ 3.539610] cpuidle: using governor menu [ 3.862130] raid6: int64x1 gen() 304 MB/s [ 4.082259] raid6: int64x1 xor() 65 MB/s [ 4.301996] raid6: int64x2 gen() 445 MB/s [ 4.521803] raid6: int64x2 xor() 152 MB/s [ 4.741596] raid6: int64x4 gen() 387 MB/s [ 4.961463] raid6: int64x4 xor() 141 MB/s [ 5.181241] raid6: int64x8 gen() 315 MB/s [ 5.401104] raid6: int64x8 xor() 116 MB/s [ 5.450556] raid6: using algorithm int64x2 gen() 445 MB/s [ 5.515490] raid6: .... xor() 152 MB/s, rmw enabled [ 5.574141] raid6: using intx1 recovery algorithm [ 5.631217] SCSI subsystem initialized [ 5.676508] PCI host bridge to bus 0002:00 [ 5.724989] pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff] [ 5.824467] pci_bus 0002:00: root bus resource [io 0x920000000fa00000-0x920000000fbfffff] [ 5.923966] pci_bus 0002:00: root bus resource [bus 02-ff] [ 5.992099] pci 0002:00:00.0: can't claim BAR 0 [io 0xf200000-0xf2000ff]: no compatible bridge window [ 6.102330] pci 0002:00:00.0: can't claim BAR 1 [mem 0x0f200000-0x0f200fff]: no compatible bridge window [ 6.216484] pci 0002:00:00.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window [ 6.335894] pci 0002:00:01.0: can't claim BAR 0 [io 0xf400000-0xf4000ff]: no compatible bridge window [ 6.447946] pci 0002:00:01.0: can't claim BAR 1 [mem 0x0f400000-0x0f400fff]: no compatible bridge window [ 6.562105] pci 0002:00:01.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window [ 6.681501] pci 0002:00:02.0: can't claim BAR 0 [mem 0x0f600000-0x0f6fffff]: no compatible bridge window [ 6.795661] pci 0002:00:06.0: can't claim BAR 0 [mem 0x0fa00000-0x0fafffff]: no compatible bridge window [ 6.909821] pci 0002:00:07.0: can't claim BAR 0 [mem 0x00000000-0x00001fff]: no compatible bridge window [ 7.024114] PCI host bridge to bus 0001:00 [ 7.073200] pci_bus 0001:00: root bus resource [mem 0x920000000c200000-0x920000000c9fffff] [ 7.172704] pci_bus 0001:00: root bus resource [io 0x920000000ca00000-0x920000000cbfffff] [ 7.272200] pci_bus 0001:00: root bus resource [bus 01-ff] [ 7.339148] pci 0001:00:00.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window [ 7.452346] pci 0001:00:01.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window [ 7.566625] PCI host bridge to bus 0000:00 [ 7.615733] pci_bus 0000:00: root bus resource [mem 0x920000000b200000-0x920000000b9fffff] [ 7.715232] pci_bus 0000:00: root bus resource [io 0x920000000ba00000-0x920000000bbfffff] [ 7.814720] pci_bus 0000:00: root bus resource [bus 00-ff] [ 7.881790] clocksource: Switched to clocksource HUB-RT [ 7.954615] NET: Registered protocol family 2 [ 8.005935] TCP established hash table entries: 65536 (order: 3, 524288 bytes) [ 8.095191] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes) [ 8.179264] TCP: Hash tables configured (established 65536 bind 65536) [ 8.256130] UDP hash table entries: 4096 (order: 1, 131072 bytes) [ 8.329975] UDP-Lite hash table entries: 4096 (order: 1, 131072 bytes) [ 8.409000] NET: Registered protocol family 1 [ 8.461360] futex hash table entries: 1024 (order: 1, 131072 bytes) [ 8.537336] workingset: timestamp_bits=40 max_order=17 bucket_order=0 [ 8.613510] zbud: loaded [ 8.645971] SGI XFS with ACLs, security attributes, realtime, no debug enabled [ 8.737724] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) [ 8.824909] io scheduler noop registered [ 8.872006] io scheduler deadline registered [ 8.923377] io scheduler cfq registered (default) [ 9.043624] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.133084] loop: module loaded [ 9.169123] qla1280: QLA1040 found on PCI bus 0, dev 0 [ 9.230776] PCI: Enabling device 0002:00:00.0 (0006 -> 0007) [ 9.299489] qla1280 0002:00:00.0: can't ioremap BAR 1: [mem size 0x00001000] [ 9.383644] qla1280: Unable to map I/O memory [ 9.436303] qla1280: QLA1040 found on PCI bus 0, dev 1 [ 9.497807] PCI: Enabling device 0002:00:01.0 (0006 -> 0007) [ 9.566389] qla1280 0002:00:01.0: can't ioremap BAR 1: [mem size 0x00001000] [ 9.650713] qla1280: Unable to map I/O memory [ 9.872302] console [ttyS0] enabledd¦[ 9.766238] serial8250: ttyS0 at MMIO 0x920000000f620178 (irq = 0, base_baud = 458333) is a 16550A [ 9.872302] console [ttyS0] enabled [ 9.955067] bootconsole [early0] disabled [ 9.955067] bootconsole [early0] disabled [ 10.073000] serial8250: ttyS1 at MMIO 0x920000000f620170 (irq = 0, base_baud = 458333) is a 16550A [ 10.216100] Found DS1981U NIC [ 10.216112] registration number 5e:57:03:00:70:5e, CRC 52 [ 10.251922] . [ 10.353129] Ethernet address is 08:00:69:05:64:74. [ 10.412166] ioc3-eth 0002:00:02.0 eth0: link up, 100Mbps, full-duplex, lpa 0x8DE1 [ 10.502356] eth0: Using PHY 31, vendor 0x20005c0, model 0, rev 0. [ 10.575863] eth0: IOC3 SSRAM has 128 kbyte. [ 10.646991] serial8250: ttyS2 at MMIO 0x920000000fa20178 (irq = 0, base_baud = 458333) is a 16550A [ 10.775950] serial8250: ttyS3 at MMIO 0x920000000fa20170 (irq = 0, base_baud = 458333) is a 16550A [ 10.887716] NIC search failed (not fatal). [ 10.939561] NIC search failed (not fatal). [ 10.989434] Failed to read MAC address [ 11.034612] Ethernet address is 00:00:00:00:00:00. [ 11.093536] ioc3-eth 0002:00:06.0 eth1: link down [ 11.150237] eth1: Using PHY 31, vendor 0x0, model 0, rev 0. [ 11.217496] eth1: IOC3 SSRAM has 64 kbyte. [ 11.267928] NET: Registered protocol family 17 [ 11.322106] rtc-m48t35 rtc-m48t35: rtc core: registered m48t35 as rtc0 [ 11.402360] rtc-m48t35 rtc-m48t35: setting system clock to 2017-02-08 09:13:55 UTC (1486545235) [ 11.509220] md: Waiting for all devices to be available before autodetect [ 11.591060] md: If you don't use raid, use raid=noautodetect [ 11.660780] md: Autodetecting RAID arrays. [ 11.710168] md: autorun ... [ 11.743888] md: ... autorun DONE. [ 11.784804] isofs_fill_super: bread failed, dev=md0, iso_blknum=16, block=32 [ 11.870695] UDF-fs: error (device md0): udf_read_tagged: read failed, block=256, location=256 [ 11.973747] UDF-fs: error (device md0): udf_read_tagged: read failed, block=512, location=512 [ 12.077091] UDF-fs: error (device md0): udf_read_tagged: read failed, block=256, location=256 [ 12.179999] UDF-fs: error (device md0): udf_read_tagged: read failed, block=512, location=512 [ 12.282824] UDF-fs: warning (device md0): udf_fill_super: No partition found (1) [ 12.375128] VFS: Cannot open root device "md0" or unknown-block(9,0): error -5 [ 12.462245] Please append a correct "root=" boot option; here are the available partitions: [ 12.563018] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(9,0) [ 12.662695] Reboot started from CPU 0 [ 12.706844] ------------[ cut here ]------------ [ 12.762483] WARNING: CPU: 0 PID: 1 at kernel/smp.c:405 smp_call_function_many+0x18c/0x430 [ 12.861020] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.10.0-rc7 #3 [ 12.936526] Stack : 0000000000000006 a8000000007714a2 0000000000000000 a80000000009ed58 [ 13.032980] ffffffff94001ce0 0000000000000000 0000000000000000 a800000000770000 [ 13.129440] a80000000067bcf7 a800000000610a50 a8000000e0772880 0000000000000000 [ 13.225900] 0000000000000001 a800000000769670 a800000000030460 0000000000000000 [ 13.322360] a800000000670000 a800000000102298 a8000000e078fb78 a8000000000494dc [ 13.418820] ffffffff94001ce0 a8000000000a09a0 00000000000000f3 a800000000610a50 [ 13.515279] 0000000000000000 0000000000000001 0000000000000000 0000000000000000 [ 13.611740] 0000000000000000 a8000000e078fac0 0000000000000000 a80000000031e6ec [ 13.708199] 0000000000000000 0000000000000000 0000000000000000 00ffffff94001ce0 [ 13.804660] a800000000690000 a80000000002bb88 0000000000000000 a80000000031e6ec [ 13.901120] ... [ 13.930552] Call Trace: [ 13.959995] [<a80000000002bb88>] show_stack+0x88/0xb8 [ 14.020854] [<a80000000031e6ec>] dump_stack+0xac/0xe0 [ 14.081687] [<a8000000000498a4>] __warn+0x144/0x168 [ 14.140444] [<a8000000000ce86c>] smp_call_function_many+0x18c/0x430 [ 14.215957] [<a8000000000ceb48>] smp_call_function+0x38/0x48 [ 14.284137] [<a80000000001ec18>] ip27_machine_restart+0x30/0x50 [ 14.355462] [<a800000000029ea4>] machine_restart+0x34/0x90 [ 14.421552] [<a8000000001020cc>] panic+0x29c/0x324 [ 14.479268] [<a8000000006f1c1c>] mount_block_root+0x418/0x444 [ 14.548488] [<a8000000006f1f2c>] prepare_namespace+0x21c/0x294 [ 14.618764] [<a8000000006f1544>] kernel_init_freeable+0x3b8/0x3e8 [ 14.692187] [<a80000000051fdf0>] kernel_init+0x10/0x1c0 [ 14.755124] [<a800000000025968>] ret_from_kernel_thread+0x14/0x1c [ 14.828542] ---[ end trace 6891c236c4efea54 ]--- ^ permalink raw reply [flat|nested] 28+ messages in thread
[parent not found: <CAErSpo4LsrPCtdZwp6CyT0jKhXLt3j=fGSiFjpRRTPUjFoKHtQ@mail.gmail.com>]
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case [not found] ` <CAErSpo4LsrPCtdZwp6CyT0jKhXLt3j=fGSiFjpRRTPUjFoKHtQ@mail.gmail.com> @ 2017-02-12 4:09 ` Joshua Kinard 2017-02-13 22:45 ` Bjorn Helgaas 0 siblings, 1 reply; 28+ messages in thread From: Joshua Kinard @ 2017-02-12 4:09 UTC (permalink / raw) To: Bjorn Helgaas Cc: Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci On 02/09/2017 16:29, Bjorn Helgaas wrote: [snip] >>>> However, IP27 is completely different in this regard. Instead of >>>> using ioremapped addresses for I/O, IP27 has a dedicated address >>>> range, 0x92xxxxxxxxxxxxxx, that is used for all I/O access. Since >>>> this is uncached physical address space, the generic MIPS PCI code >>>> will not probe it correctly and thus, the original behavior of >>>> PCI_PROBE_ONLY needs to be restored only for the IP27 platform to >>>> bypass this logic and have working PCI, at least for the IO6/IO6G >>>> board that houses the base devices, until a better solution is found. >>> >>> It sounds like there's something different about how ioremap() works on >>> these platforms and PCI probing is tripping over that. I'd really like >>> to understand more about this difference to see if we can converge that >>> instead of adding back the PCI_PROBE_ONLY usage. >> >> I'd need to go and dig around in the IP27 headers again for this machine >> to see what ioremap() is actually doing, but I *think* it returns uncached >> physical addresses in most instances because of a special feature of the >> CPU, the R10000-family, which makes uncached access very fast. I think >> only the IP27 platform uses this capability. Other R1x0-based systems >> don't (although, I might be wrong about IP27's successor in IP35). > > ioremap() must return a CPU virtual address. It can take advantage of > arch-specific features, special address ranges that are uncached, special > identity mappings, or whatever, but from the caller's point of view, the > return value must be usable as a virtual address without any special > handling. Apparently, MIPS's implementation of ioremap does not guarantee that a virtual address is always returned. Quoting from arch/mips/include/asm/io.h around line #236: /* * ioremap - map bus memory into CPU space * @offset: bus address of the memory * @size: size of the resource to map * * ioremap performs a platform specific sequence of operations to * make bus memory CPU accessible via the readb/readw/readl/writeb/ * writew/writel functions and the other mmio helpers. The returned * address is not guaranteed to be usable directly as a virtual * address. */ It looks like the qla1280.c driver calls pci_ioremap_bar(), which calls ioremap_nocache(), and on MIPS, that's just an alias (as far as I can tell) to ioremap(), and it has the same limitation, that the address returned may not be usable as a CPU virtual address. > This is from ip27-dmesg-working_pci-20170208.txt: > > PCI host bridge to bus 0002:00 > pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff] > pci_bus 0002:00: root bus resource [io 0x920000000fa00000-0x920000000fbfffff] > pci_bus 0002:00: root bus resource [bus 02-ff] > pci 0002:00:00.0: [1077:1020] type 00 class 0x010000 > pci 0002:00:00.0: reg 0x10: [io 0xf200000-0xf2000ff] > pci 0002:00:00.0: reg 0x14: [mem 0x0f200000-0x0f200fff] > pci 0002:00:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref] > > There's something wrong here: all the resources printed above, including the > BARs, should be CPU physical addresses (as are all the /proc/iomem > addresses). The "root bus resources" are windows through the host bridge to > PCI. The BAR resources should be inside those windows. > > I'm sure the bridge translates between CPU physical addresses and PCI bus > addresses, probably by chopping off all those high-order bits. The PCI core > needs to know about this, and it looks like pcibios_scanbus() is trying to > tell it by using pci_add_resource_offset(). But the PCI core isn't getting > the message because we don't see the "(bus address [%#010llx-%#010llx])" > being printed by pci_register_host_bridge(). > > A correct dmesg log would look something like this: > > PCI host bridge to bus 0002:00 > pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff] > (bus address [0x0f200000-0x0f9fffff]) > pci 0002:00:00.0: reg 0x14: [mem 0x920000000f200000-0x920000000f200fff] > > This would mean that for 0002:00:00.0, pdev->resource[1] contains the CPU > physical address 0x920000000f200000, and the host bridge translates that to > 0x0f200000, which is what the 32-bit BAR1 would contain. > > I think "lspci" should show you the CPU addresses (0x920000000f200000) and > "lspci -b" should show you the actual BAR values (0x0f200000). > > Your dmesg log is showing you 0x0f200000, which is probably the BAR value. > __pci_read_base() reads the BAR value and runs it through > pcibios_bus_to_resource(), which *should* be converting it to > 0x920000000f200000 (the reverse of the translation done by the host bridge). > But since the PCI core doesn't know about that translation, > pcibios_bus_to_resource() does nothing. > > So I think there's something wrong with the way you're using > pci_add_resource_offset(). This is basically the same conclusion I've come to. I've found a "partial" solution, but I'm pretty sure it's wrong. It seems the reason IP30 works without PCI_PROBE_ONLY now is because it doesn't need special casing on the addresses in order to work. IP27, on the other hand, needs all I/O accesses to go through the 0x92xxx range or you'll run into hardware-enforced barriers that will panic the kernel. First attempt I did was to simply subtract NODE_IO_BASE(_n) from the address values that ultimately get fed to pci_add_resource_offset(). That yields this kind of output: [ 8.065146] PCI host bridge to bus 0000:00 [ 8.113572] pci_bus 0000:00: root bus resource [mem 0x0b200000-0x0b9fffff] [ 8.196284] pci_bus 0000:00: root bus resource [io 0xba00000-0xbbfffff] [ 8.276934] pci_bus 0000:00: root bus resource [bus 00-ff] [ 8.343199] PCI host bridge to bus 0001:00 [ 8.392144] pci_bus 0001:00: root bus resource [mem 0x0c200000-0x0c9fffff] [ 8.474899] pci_bus 0001:00: root bus resource [io 0xca00000-0xcbfffff] [ 8.555520] pci_bus 0001:00: root bus resource [bus 01-ff] [ 8.622536] pci 0001:00:00.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window [ 8.735665] pci 0001:00:01.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window [ 8.850128] PCI host bridge to bus 0002:00 [ 8.899049] pci_bus 0002:00: root bus resource [mem 0x0f200000-0x0f9fffff] [ 8.981794] pci_bus 0002:00: root bus resource [io 0xfa00000-0xfbfffff] [ 9.062413] pci_bus 0002:00: root bus resource [bus 02-ff] [ 9.130425] pci 0002:00:00.0: can't claim BAR 0 [io 0xf200000-0xf2000ff]: no compatible bridge window [ 9.240674] pci 0002:00:00.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window [ 9.360079] pci 0002:00:01.0: can't claim BAR 0 [io 0xf400000-0xf4000ff]: no compatible bridge window [ 9.472148] pci 0002:00:01.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window [ 9.591555] pci 0002:00:06.0: can't claim BAR 0 [mem 0x0fa00000-0x0fafffff]: no compatible bridge window [ 9.705685] pci 0002:00:07.0: can't claim BAR 0 [mem 0x00000000-0x00001fff]: no compatible bridge window [ 9.819933] qla1280: QLA1040 found on PCI bus 0, dev 0 [ 9.881644] PCI: Enabling device 0002:00:00.0 (0006 -> 0007) Despite the above errors, it succeeds in probing and finds devices and boots into userland. The error message above stems from 'pci_claim_resource' in drivers/pci/setup-res.c because it can't seem to find the parent bridge. Not sure if that's good or bad. But it does boot, and it would remove one of the two #ifdef hacks in my patch. So I next looked at the offset bit you mentioned, and after some fiddling around in pci-bridge.c, got this dmesg output, which matches what you say should be the correct dmesg output: [ 8.095560] PCI host bridge to bus 0000:00 [ 8.143949] pci_bus 0000:00: root bus resource [mem 0x920000000b200000-0x920000000b9fffff] (bus address [0x0b200000-0x0b9fffff]) [ 8.283216] pci_bus 0000:00: root bus resource [io 0x920000000ba00000-0x920000000bbfffff] (bus address [0xba00000-0xbbfffff]) [ 8.420442] pci_bus 0000:00: root bus resource [bus 00-ff] [ 8.486711] PCI host bridge to bus 0001:00 [ 8.535656] pci_bus 0001:00: root bus resource [mem 0x920000000c200000-0x920000000c9fffff] (bus address [0x0c200000-0x0c9fffff]) [ 8.674965] pci_bus 0001:00: root bus resource [io 0x920000000ca00000-0x920000000cbfffff] (bus address [0xca00000-0xcbfffff]) [ 8.812144] pci_bus 0001:00: root bus resource [bus 01-ff] [ 8.879140] pci 0001:00:00.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window [ 8.992298] pci 0001:00:01.0: can't claim BAR 0 [mem 0x00000000-0x07ffffff]: no compatible bridge window [ 9.106732] PCI host bridge to bus 0002:00 [ 9.155679] pci_bus 0002:00: root bus resource [mem 0x920000000f200000-0x920000000f9fffff] (bus address [0x0f200000-0x0f9fffff]) [ 9.294974] pci_bus 0002:00: root bus resource [io 0x920000000fa00000-0x920000000fbfffff] (bus address [0xfa00000-0xfbfffff]) [ 9.432168] pci_bus 0002:00: root bus resource [bus 02-ff] [ 9.500118] pci 0002:00:00.0: can't claim BAR 0 [io 0xf200000-0xf2000ff]: no compatible bridge window [ 9.610326] pci 0002:00:00.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window [ 9.729688] pci 0002:00:01.0: can't claim BAR 0 [io 0xf400000-0xf4000ff]: no compatible bridge window [ 9.841764] pci 0002:00:01.0: can't claim BAR 6 [mem 0x00000000-0x0000ffff pref]: no compatible bridge window [ 9.961151] pci 0002:00:06.0: can't claim BAR 0 [mem 0x0fa00000-0x0fafffff]: no compatible bridge window [ 10.075344] pci 0002:00:07.0: can't claim BAR 0 [mem 0x00000000-0x00001fff]: no compatible bridge window [ 10.189528] qla1280: QLA1040 found on PCI bus 0, dev 0 [ 10.251300] PCI: Enabling device 0002:00:00.0 (0006 -> 0007) [ 10.320129] Unhandled kernel unaligned access[#1]: [ 10.376908] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.10.0-rc7-mipsgit-20170208 #24 [ 10.471163] task: a8000000e077c880 task.stack: a8000000e0780000 [ 10.542378] $ 0 : 0000000000000000 0000000000024680 0000000000000000 0000000000000000 [ 10.638730] $ 4 : 000000000065e0c0 0000000000000010 a800000000450a18 240000000f20000a [ 10.735084] $ 8 : a8000000007a3978 a8000000e00c05b0 0000000000000000 a80000000145ef30 [ 10.831440] $12 : ffffffff94005ce1 000000001000001e ffffffffffffff80 00000000007e0000 [ 10.927796] $16 : a8000000e078f920 a800000000450a14 0000000000000000 ffffffffa4600000 [ 11.024149] $20 : 240000000f20000a a8000000004509b8 a8000000006b9930 a8000000007eb1b8 [ 11.120506] $24 : 0000000000940000 00000000007e0000 [ 11.216860] $28 : a8000000e0780000 a8000000e078f8d0 a8000000e00c1650 a800000000024680 [ 11.313216] Hi : 0000000000000000 [ 11.356156] Lo : 0000000000002800 [ 11.399117] epc : a80000000002de44 do_ade+0x4f4/0x960 [ 11.461952] ra : a800000000024680 ret_from_exception+0x0/0x18 [ 11.534204] Status: 94005ce3 KX SX UX KERNEL EXL IE [ 11.593903] Cause : 00008014 (ExcCode 05) [ 11.642080] BadVA : 240000000f20000b [ 11.685020] PrId : 00000f14 (R14000) And you can see that while the dmesg output is correct, the end result is not. The issue goes back to what MIPS's ioremap() is doing when it encounters the specialized cached-uncached attribute of the R1x0k CPUs on IP27, it tries adding 0x9200000000000000, which is IP27's IO_BASE, to one of the addresses during qla1280's probe, which wraps around to 0x240000000xxxxxxx and is invalid in MIPS. This leads to the unhandled kernel unaligned access panic. I tried implementing a custom plat_ioremap() that would subtract the IO_BASE off of the address being ioremapped by qla1280. MIPS has a generic template available to do this in arch/mips/include/asm/mach-generic/ioremap.h, but it appears my gcc version is recent enough to have some new checks enabled that choke on casting a phys_addr_t to void __iomem * (-Wint-to-pointer-cast), and I've given up finding a workaround that doesn't involve invasive changes to MIPS's ioremapping core. --J ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-12 4:09 ` Joshua Kinard 2017-02-13 22:45 ` Bjorn Helgaas @ 2017-02-13 22:45 ` Bjorn Helgaas 0 siblings, 0 replies; 28+ messages in thread From: Bjorn Helgaas @ 2017-02-13 22:45 UTC (permalink / raw) To: Joshua Kinard Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci It looks like IP30 is using some code that's not upstream yet, so I'll just point out some things that don't look right. I'm ignoring the IP27 stuff here in case that has different issues. Joshua wrote: > On 02/07/2017 13:29, Bjorn Helgaas wrote: >> Is there any chance you can collect complete dmesg logs and >> /proc/iomem contents from IP27 and IP30? Maybe "lspci -vv" output, >> too? I'm not sure where to look to understand the ioremap() behavior. > > ... > A note about IP30, since that's part of an external patch set, the > dmesg output is a little different, as it is using the newer > BRIDGE/Xtalk code I just sent in as patches. ... > > Also, IP30 doesn't use the PCI_PROBE_ONLY thing anymore. This causes > /proc/iomem to be far more detailed than it used to be. > > IP30's /proc/iomem: > 00000000-00003fff : reserved > 1d200000-1d9fffff : Bridge MEM > d080000000-d080003fff : 0001:00:03.0 > 1d204000-1d204fff : 0001:00:02.0 > 1d205000-1d205fff : 0001:00:02.1 > 1d206000-1d206fff : 0001:00:02.2 > 1d207000-1d2070ff : 0001:00:02.3 > 1d207100-1d20717f : 0001:00:01.0 > 1f200000-1f9fffff : Bridge MEM > f080100000-f0801fffff : 0000:00:02.0 > f080010000-f08001ffff : 0000:00:00.0 > f080030000-f08003ffff : 0000:00:01.0 > f080000000-f080000fff : 0000:00:00.0 > f080020000-f080020fff : 0000:00:01.0 > 20004000-209b8fff : reserved > 20004000-206acb13 : Kernel code > 206acb14-208affff : Kernel data > 209b9000-20efffff : System RAM > 20f00000-20ffffff : System RAM > 21000000-9fffffff : System RAM > f080100000-f0801fffff : ioc3 >From the dmesg you attached (ip30-dmesg-20170208.txt, PCI parts appended below): pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff] pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff] These are shown correctly in /proc/iomem. It would be nice if the /proc/iomem string identified which bridge was which (x86 uses "PCI Bus 0000:00", "PCI Bus 0001:00", etc.) but that's not essential. pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff] ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit. This is funky. We read 0x1d200000 from the BAR (PCI bus addresses are apparently identical to CPU physical addresses). We claimed that space from the host bridge window, so /proc/iomem would have looked like this at that point: 1d200000-1d9fffff : Bridge MEM 1d200000-1d203fff : 0001:00:03.0 But it looks like whatever this ip30-bridge thing is overwrote the 0001:00:03.0 resource, which makes /proc/iomem wrong. All the resources under the bridge to 0000:00 are wrong, too: 1f200000-1f9fffff : Bridge MEM f080100000-f0801fffff : 0000:00:02.0 f080010000-f08001ffff : 0000:00:00.0 f080030000-f08003ffff : 0000:00:01.0 f080000000-f080000fff : 0000:00:00.0 f080020000-f080020fff : 0000:00:01.0 We read values from the BARs: pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff] pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff] pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref] pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff] pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref] pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff] pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff] These aren't zero, so somebody apparently has assigned them, but they aren't inside the host bridge window. Therefore the PCI core assumes they will not work, and it reassigns space from the window: pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff] pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref] pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref] pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff] pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff] I don't know why we didn't assign space to 0000:00:03.0 BAR 1. Anyway, it looks like they got inserted in /proc/iomem, then overwritten by the ip30-bridge stuff again. >From /proc/iomem, I would expect that only the USB devices (0001:00:02 functions 0, 1, 2, 3) would work. But apparently your system does actually work, so there must be corresponding funkiness somewhere else that compensates for these resources. I'm attaching the PCI parts of your ip30-dmesg-20170208.txt and the complete ip30-lspci-20170208.txt below for context. pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff] pci_bus 0000:00: root bus resource [io 0x1fa00000-0x1fbfffff] pci_bus 0000:00: root bus resource [bus 00-ff] pci 0000:00:00.0: [1077:1020] type 00 class 0x010000 pci 0000:00:00.0: reg 0x10: [io 0x200000-0x2000ff] pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff] pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref] pci 0000:00:01.0: [1077:1020] type 00 class 0x010000 pci 0000:00:01.0: reg 0x10: [io 0x400000-0x4000ff] pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff] pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref] pci 0000:00:02.0: [10a9:0003] type 00 class 0xff0000 pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff] pci 0000:00:03.0: [10a9:0005] type 00 class 0x000000 pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff] pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff] pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref] pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref] pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff] pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff] pci 0000:00:00.0: BAR 0: assigned [io 0x1fa00000-0x1fa000ff] pci 0000:00:01.0: BAR 0: assigned [io 0x1fa00400-0x1fa004ff] ip30-bridge: 0000:00:00.0 Bar 0 with size 0x00000100 at bus 0x00000000 vma 0x000000f100000000 is Direct I/O. ip30-bridge: 0000:00:00.0 Bar 1 with size 0x00001000 at bus 0x00000000 vma 0x000000f080000000 is Direct 64-bit. ip30-bridge: 0000:00:00.0 Bar 6 with size 0x00010000 at bus 0x00010000 vma 0x000000f080010000 is Direct 64-bit. ip30-bridge: 0000:00:01.0 Bar 0 with size 0x00000100 at bus 0x00000100 vma 0x000000f100000100 is Direct I/O. ip30-bridge: 0000:00:01.0 Bar 1 with size 0x00001000 at bus 0x00020000 vma 0x000000f080020000 is Direct 64-bit. ip30-bridge: 0000:00:01.0 Bar 6 with size 0x00010000 at bus 0x00030000 vma 0x000000f080030000 is Direct 64-bit. ip30-bridge: 0000:00:02.0 Bar 0 with size 0x00100000 at bus 0x00100000 vma 0x000000f080100000 is Direct 64-bit. PCI host bridge to bus 0001:00 pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff] pci_bus 0001:00: root bus resource [io 0x1da00000-0x1dbfffff] pci_bus 0001:00: root bus resource [bus 01-ff] pci 0001:00:01.0: [11fe:080e] type 00 class 0x078000 pci 0001:00:01.0: reg 0x10: [mem 0x00200000-0x0020007f] pci 0001:00:01.0: reg 0x14: [io 0x200000-0x20007f] pci 0001:00:01.0: reg 0x18: [io 0x204000-0x2040ff] pci 0001:00:02.0: [10b9:5237] type 00 class 0x0c0310 pci 0001:00:02.0: reg 0x10: [mem 0x00300000-0x00300fff] pci 0001:00:02.0: PME# supported from D0 D1 D3hot D3cold pci 0001:00:02.1: [10b9:5237] type 00 class 0x0c0310 pci 0001:00:02.1: reg 0x10: [mem 0x00000000-0x00000fff] pci 0001:00:02.1: PME# supported from D0 D1 D3hot D3cold pci 0001:00:02.2: [10b9:5237] type 00 class 0x0c0310 pci 0001:00:02.2: reg 0x10: [mem 0x00000000-0x00000fff] pci 0001:00:02.2: PME# supported from D0 D1 D3hot D3cold pci 0001:00:02.3: [10b9:5239] type 00 class 0x0c0320 pci 0001:00:02.3: reg 0x10: [mem 0x00000000-0x000000ff] pci 0001:00:02.3: PME# supported from D0 D3hot D3cold pci 0001:00:03.0: [10a9:0009] type 00 class 0x020000 pci 0001:00:03.0: reg 0x10: [mem 0x00400000-0x00403fff] pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff] pci 0001:00:02.0: BAR 0: assigned [mem 0x1d204000-0x1d204fff] pci 0001:00:02.1: BAR 0: assigned [mem 0x1d205000-0x1d205fff] pci 0001:00:02.2: BAR 0: assigned [mem 0x1d206000-0x1d206fff] pci 0001:00:01.0: BAR 2: assigned [io 0x1da00000-0x1da000ff] pci 0001:00:02.3: BAR 0: assigned [mem 0x1d207000-0x1d2070ff] pci 0001:00:01.0: BAR 0: assigned [mem 0x1d207100-0x1d20717f] pci 0001:00:01.0: BAR 1: assigned [io 0x1da00400-0x1da0047f] ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit. Here's the ip30-lspci-20170208.txt attachment from your mail: 0000:00:00.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64, Cache Line Size: 256 bytes Interrupt: pin A routed to IRQ 0 Region 0: I/O ports at f100000000 [size=257] Region 1: [virtual] Memory at f080000000 (32-bit, non-prefetchable) [size=4097] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at f080010000 [disabled] [size=65537] Kernel driver in use: qla1280 lspci: Unable to load libkmod resources: error -12 0000:00:01.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64, Cache Line Size: 256 bytes Interrupt: pin A routed to IRQ 1 Region 0: I/O ports at f100000100 [size=257] Region 1: Memory at f080020000 (32-bit, non-prefetchable) [size=4097] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at f080030000 [disabled] [size=65537] Kernel driver in use: qla1280 0000:00:02.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. IOC3 I/O controller (rev 01) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin A routed to IRQ 2 Region 0: Memory at f080100000 (32-bit, non-prefetchable) [size=1048577] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Kernel driver in use: IOC3 0000:00:03.0 Non-VGA unclassified device: Silicon Graphics Intl. Corp. RAD Audio (rev c0) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=slow >TAbort+ <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 255 Region 0: Memory at 00600000 (32-bit, non-prefetchable) [size=8193] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] 0001:00:01.0 Communication controller: Comtrol Corporation Device 080e (rev 01) Subsystem: Comtrol Corporation Device 080e Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 0 Region 0: Memory at 1d207100 (32-bit, non-prefetchable) [size=129] Region 1: I/O ports at 1da00400 [size=129] Region 2: I/O ports at 1da00000 [size=257] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] 0001:00:02.0 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 16 (20000ns max) Interrupt: pin B routed to IRQ 0 Region 0: Memory at 1d204000 (32-bit, non-prefetchable) [size=4097] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- 0001:00:02.1 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin C routed to IRQ 0 Region 0: Memory at 1d205000 (32-bit, non-prefetchable) [disabled] [size=4097] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- 0001:00:02.2 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin D routed to IRQ 0 Region 0: Memory at 1d206000 (32-bit, non-prefetchable) [disabled] [size=4097] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- 0001:00:02.3 USB controller: ULi Electronics Inc. USB 2.0 Controller (rev 01) (prog-if 20 [EHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 0 Region 0: Memory at 1d207000 (32-bit, non-prefetchable) [disabled] [size=257] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Debug port: BAR=1 offset=0090 0001:00:03.0 Ethernet controller: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet (rev 01) Subsystem: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 (16000ns min), Cache Line Size: 128 bytes Interrupt: pin A routed to IRQ 4 Region 0: [virtual] Memory at d080000000 (32-bit, non-prefetchable) [size=16385] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Kernel driver in use: acenic ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case @ 2017-02-13 22:45 ` Bjorn Helgaas 0 siblings, 0 replies; 28+ messages in thread From: Bjorn Helgaas @ 2017-02-13 22:45 UTC (permalink / raw) To: Joshua Kinard Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci It looks like IP30 is using some code that's not upstream yet, so I'll just point out some things that don't look right. I'm ignoring the IP27 stuff here in case that has different issues. Joshua wrote: > On 02/07/2017 13:29, Bjorn Helgaas wrote: >> Is there any chance you can collect complete dmesg logs and >> /proc/iomem contents from IP27 and IP30? Maybe "lspci -vv" output, >> too? I'm not sure where to look to understand the ioremap() behavior. > > ... > A note about IP30, since that's part of an external patch set, the > dmesg output is a little different, as it is using the newer > BRIDGE/Xtalk code I just sent in as patches. ... > > Also, IP30 doesn't use the PCI_PROBE_ONLY thing anymore. This causes > /proc/iomem to be far more detailed than it used to be. > > IP30's /proc/iomem: > 00000000-00003fff : reserved > 1d200000-1d9fffff : Bridge MEM > d080000000-d080003fff : 0001:00:03.0 > 1d204000-1d204fff : 0001:00:02.0 > 1d205000-1d205fff : 0001:00:02.1 > 1d206000-1d206fff : 0001:00:02.2 > 1d207000-1d2070ff : 0001:00:02.3 > 1d207100-1d20717f : 0001:00:01.0 > 1f200000-1f9fffff : Bridge MEM > f080100000-f0801fffff : 0000:00:02.0 > f080010000-f08001ffff : 0000:00:00.0 > f080030000-f08003ffff : 0000:00:01.0 > f080000000-f080000fff : 0000:00:00.0 > f080020000-f080020fff : 0000:00:01.0 > 20004000-209b8fff : reserved > 20004000-206acb13 : Kernel code > 206acb14-208affff : Kernel data > 209b9000-20efffff : System RAM > 20f00000-20ffffff : System RAM > 21000000-9fffffff : System RAM > f080100000-f0801fffff : ioc3 ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case @ 2017-02-13 22:45 ` Bjorn Helgaas 0 siblings, 0 replies; 28+ messages in thread From: Bjorn Helgaas @ 2017-02-13 22:45 UTC (permalink / raw) To: Joshua Kinard Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci It looks like IP30 is using some code that's not upstream yet, so I'll just point out some things that don't look right. I'm ignoring the IP27 stuff here in case that has different issues. Joshua wrote: > On 02/07/2017 13:29, Bjorn Helgaas wrote: >> Is there any chance you can collect complete dmesg logs and >> /proc/iomem contents from IP27 and IP30? Maybe "lspci -vv" output, >> too? I'm not sure where to look to understand the ioremap() behavior. > > ... > A note about IP30, since that's part of an external patch set, the > dmesg output is a little different, as it is using the newer > BRIDGE/Xtalk code I just sent in as patches. ... > > Also, IP30 doesn't use the PCI_PROBE_ONLY thing anymore. This causes > /proc/iomem to be far more detailed than it used to be. > > IP30's /proc/iomem: > 00000000-00003fff : reserved > 1d200000-1d9fffff : Bridge MEM > d080000000-d080003fff : 0001:00:03.0 > 1d204000-1d204fff : 0001:00:02.0 > 1d205000-1d205fff : 0001:00:02.1 > 1d206000-1d206fff : 0001:00:02.2 > 1d207000-1d2070ff : 0001:00:02.3 > 1d207100-1d20717f : 0001:00:01.0 > 1f200000-1f9fffff : Bridge MEM > f080100000-f0801fffff : 0000:00:02.0 > f080010000-f08001ffff : 0000:00:00.0 > f080030000-f08003ffff : 0000:00:01.0 > f080000000-f080000fff : 0000:00:00.0 > f080020000-f080020fff : 0000:00:01.0 > 20004000-209b8fff : reserved > 20004000-206acb13 : Kernel code > 206acb14-208affff : Kernel data > 209b9000-20efffff : System RAM > 20f00000-20ffffff : System RAM > 21000000-9fffffff : System RAM > f080100000-f0801fffff : ioc3 From the dmesg you attached (ip30-dmesg-20170208.txt, PCI parts appended below): pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff] pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff] These are shown correctly in /proc/iomem. It would be nice if the /proc/iomem string identified which bridge was which (x86 uses "PCI Bus 0000:00", "PCI Bus 0001:00", etc.) but that's not essential. pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff] ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit. This is funky. We read 0x1d200000 from the BAR (PCI bus addresses are apparently identical to CPU physical addresses). We claimed that space from the host bridge window, so /proc/iomem would have looked like this at that point: 1d200000-1d9fffff : Bridge MEM 1d200000-1d203fff : 0001:00:03.0 But it looks like whatever this ip30-bridge thing is overwrote the 0001:00:03.0 resource, which makes /proc/iomem wrong. All the resources under the bridge to 0000:00 are wrong, too: 1f200000-1f9fffff : Bridge MEM f080100000-f0801fffff : 0000:00:02.0 f080010000-f08001ffff : 0000:00:00.0 f080030000-f08003ffff : 0000:00:01.0 f080000000-f080000fff : 0000:00:00.0 f080020000-f080020fff : 0000:00:01.0 We read values from the BARs: pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff] pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff] pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref] pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff] pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref] pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff] pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff] These aren't zero, so somebody apparently has assigned them, but they aren't inside the host bridge window. Therefore the PCI core assumes they will not work, and it reassigns space from the window: pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff] pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref] pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref] pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff] pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff] I don't know why we didn't assign space to 0000:00:03.0 BAR 1. Anyway, it looks like they got inserted in /proc/iomem, then overwritten by the ip30-bridge stuff again. From /proc/iomem, I would expect that only the USB devices (0001:00:02 functions 0, 1, 2, 3) would work. But apparently your system does actually work, so there must be corresponding funkiness somewhere else that compensates for these resources. I'm attaching the PCI parts of your ip30-dmesg-20170208.txt and the complete ip30-lspci-20170208.txt below for context. pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff] pci_bus 0000:00: root bus resource [io 0x1fa00000-0x1fbfffff] pci_bus 0000:00: root bus resource [bus 00-ff] pci 0000:00:00.0: [1077:1020] type 00 class 0x010000 pci 0000:00:00.0: reg 0x10: [io 0x200000-0x2000ff] pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff] pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref] pci 0000:00:01.0: [1077:1020] type 00 class 0x010000 pci 0000:00:01.0: reg 0x10: [io 0x400000-0x4000ff] pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff] pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref] pci 0000:00:02.0: [10a9:0003] type 00 class 0xff0000 pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff] pci 0000:00:03.0: [10a9:0005] type 00 class 0x000000 pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff] pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff] pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref] pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref] pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff] pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff] pci 0000:00:00.0: BAR 0: assigned [io 0x1fa00000-0x1fa000ff] pci 0000:00:01.0: BAR 0: assigned [io 0x1fa00400-0x1fa004ff] ip30-bridge: 0000:00:00.0 Bar 0 with size 0x00000100 at bus 0x00000000 vma 0x000000f100000000 is Direct I/O. ip30-bridge: 0000:00:00.0 Bar 1 with size 0x00001000 at bus 0x00000000 vma 0x000000f080000000 is Direct 64-bit. ip30-bridge: 0000:00:00.0 Bar 6 with size 0x00010000 at bus 0x00010000 vma 0x000000f080010000 is Direct 64-bit. ip30-bridge: 0000:00:01.0 Bar 0 with size 0x00000100 at bus 0x00000100 vma 0x000000f100000100 is Direct I/O. ip30-bridge: 0000:00:01.0 Bar 1 with size 0x00001000 at bus 0x00020000 vma 0x000000f080020000 is Direct 64-bit. ip30-bridge: 0000:00:01.0 Bar 6 with size 0x00010000 at bus 0x00030000 vma 0x000000f080030000 is Direct 64-bit. ip30-bridge: 0000:00:02.0 Bar 0 with size 0x00100000 at bus 0x00100000 vma 0x000000f080100000 is Direct 64-bit. PCI host bridge to bus 0001:00 pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff] pci_bus 0001:00: root bus resource [io 0x1da00000-0x1dbfffff] pci_bus 0001:00: root bus resource [bus 01-ff] pci 0001:00:01.0: [11fe:080e] type 00 class 0x078000 pci 0001:00:01.0: reg 0x10: [mem 0x00200000-0x0020007f] pci 0001:00:01.0: reg 0x14: [io 0x200000-0x20007f] pci 0001:00:01.0: reg 0x18: [io 0x204000-0x2040ff] pci 0001:00:02.0: [10b9:5237] type 00 class 0x0c0310 pci 0001:00:02.0: reg 0x10: [mem 0x00300000-0x00300fff] pci 0001:00:02.0: PME# supported from D0 D1 D3hot D3cold pci 0001:00:02.1: [10b9:5237] type 00 class 0x0c0310 pci 0001:00:02.1: reg 0x10: [mem 0x00000000-0x00000fff] pci 0001:00:02.1: PME# supported from D0 D1 D3hot D3cold pci 0001:00:02.2: [10b9:5237] type 00 class 0x0c0310 pci 0001:00:02.2: reg 0x10: [mem 0x00000000-0x00000fff] pci 0001:00:02.2: PME# supported from D0 D1 D3hot D3cold pci 0001:00:02.3: [10b9:5239] type 00 class 0x0c0320 pci 0001:00:02.3: reg 0x10: [mem 0x00000000-0x000000ff] pci 0001:00:02.3: PME# supported from D0 D3hot D3cold pci 0001:00:03.0: [10a9:0009] type 00 class 0x020000 pci 0001:00:03.0: reg 0x10: [mem 0x00400000-0x00403fff] pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff] pci 0001:00:02.0: BAR 0: assigned [mem 0x1d204000-0x1d204fff] pci 0001:00:02.1: BAR 0: assigned [mem 0x1d205000-0x1d205fff] pci 0001:00:02.2: BAR 0: assigned [mem 0x1d206000-0x1d206fff] pci 0001:00:01.0: BAR 2: assigned [io 0x1da00000-0x1da000ff] pci 0001:00:02.3: BAR 0: assigned [mem 0x1d207000-0x1d2070ff] pci 0001:00:01.0: BAR 0: assigned [mem 0x1d207100-0x1d20717f] pci 0001:00:01.0: BAR 1: assigned [io 0x1da00400-0x1da0047f] ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit. Here's the ip30-lspci-20170208.txt attachment from your mail: 0000:00:00.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64, Cache Line Size: 256 bytes Interrupt: pin A routed to IRQ 0 Region 0: I/O ports at f100000000 [size=257] Region 1: [virtual] Memory at f080000000 (32-bit, non-prefetchable) [size=4097] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at f080010000 [disabled] [size=65537] Kernel driver in use: qla1280 lspci: Unable to load libkmod resources: error -12 0000:00:01.0 SCSI storage controller: QLogic Corp. ISP1020 Fast-wide SCSI (rev 05) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64, Cache Line Size: 256 bytes Interrupt: pin A routed to IRQ 1 Region 0: I/O ports at f100000100 [size=257] Region 1: Memory at f080020000 (32-bit, non-prefetchable) [size=4097] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at f080030000 [disabled] [size=65537] Kernel driver in use: qla1280 0000:00:02.0 Unassigned class [ff00]: Silicon Graphics Intl. Corp. IOC3 I/O controller (rev 01) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin A routed to IRQ 2 Region 0: Memory at f080100000 (32-bit, non-prefetchable) [size=1048577] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Kernel driver in use: IOC3 0000:00:03.0 Non-VGA unclassified device: Silicon Graphics Intl. Corp. RAD Audio (rev c0) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=slow >TAbort+ <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 255 Region 0: Memory at 00600000 (32-bit, non-prefetchable) [size=8193] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] 0001:00:01.0 Communication controller: Comtrol Corporation Device 080e (rev 01) Subsystem: Comtrol Corporation Device 080e Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 0 Region 0: Memory at 1d207100 (32-bit, non-prefetchable) [size=129] Region 1: I/O ports at 1da00400 [size=129] Region 2: I/O ports at 1da00000 [size=257] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] 0001:00:02.0 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 16 (20000ns max) Interrupt: pin B routed to IRQ 0 Region 0: Memory at 1d204000 (32-bit, non-prefetchable) [size=4097] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- 0001:00:02.1 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin C routed to IRQ 0 Region 0: Memory at 1d205000 (32-bit, non-prefetchable) [disabled] [size=4097] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- 0001:00:02.2 USB controller: ULi Electronics Inc. USB 1.1 Controller (rev 03) (prog-if 10 [OHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin D routed to IRQ 0 Region 0: Memory at 1d206000 (32-bit, non-prefetchable) [disabled] [size=4097] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- 0001:00:02.3 USB controller: ULi Electronics Inc. USB 2.0 Controller (rev 01) (prog-if 20 [EHCI]) Subsystem: ULi Electronics Inc. ASRock 939Dual-SATA2 Motherboard Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 0 Region 0: Memory at 1d207000 (32-bit, non-prefetchable) [disabled] [size=257] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [disabled] [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Debug port: BAR=1 offset=0090 0001:00:03.0 Ethernet controller: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet (rev 01) Subsystem: Silicon Graphics Intl. Corp. AceNIC Gigabit Ethernet Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 (16000ns min), Cache Line Size: 128 bytes Interrupt: pin A routed to IRQ 4 Region 0: [virtual] Memory at d080000000 (32-bit, non-prefetchable) [size=16385] Region 1: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 2: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 3: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 4: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Region 5: Memory at <unassigned> (32-bit, non-prefetchable) [size=2] Expansion ROM at <unassigned> [disabled] [size=2] Kernel driver in use: acenic ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-13 22:45 ` Bjorn Helgaas (?) (?) @ 2017-02-14 7:39 ` Joshua Kinard 2017-02-14 14:56 ` Bjorn Helgaas -1 siblings, 1 reply; 28+ messages in thread From: Joshua Kinard @ 2017-02-14 7:39 UTC (permalink / raw) To: Bjorn Helgaas Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci On 02/13/2017 17:45, Bjorn Helgaas wrote: > It looks like IP30 is using some code that's not upstream yet, so I'll > just point out some things that don't look right. I'm ignoring the > IP27 stuff here in case that has different issues. Sorry about that. It's an external set of patches I've been maintaining for the last decade (I'm not the original author). Being that I do this for a hobby, sometimes I get waylaid by other priorities. I've spent some time re-organizing my patch collection lately so that I can start sending these things in. This series and an earlier one to the Linux/MIPS list are the beginnings of that, as time permits. However, I've got a ways to go before getting to the IP30 patches. E.g., IP27 comes before IP30, so that's where my focus has been lately. I've put an 4.10-rc7 tree with all patches applied here (including the IOC3, IP27, and IP30 patches): http://dev.gentoo.org/~kumba/mips/ip30/linux-4.10_rc7-20170208.ip30/ The patchset itself, if interested: http://dev.gentoo.org/~kumba/mips/ip30/mips-patches-4.10.0/ You can find the IP30 code in: arch/mips/sgi-ip30/* arch/mips/include/asm/mach-ip30/* IP27's code is in these folders: arch/mips/sgi-ip27/* arch/mips/include/asm/mach-ip27/* arch/mips/include/asm/sn/* PCI code: arch/mips/pci/pci-bridge.c arch/mips/pci/ops-bridge.c arch/mips/include/asm/pci/bridge.h > Joshua wrote: >> On 02/07/2017 13:29, Bjorn Helgaas wrote: >>> Is there any chance you can collect complete dmesg logs and >>> /proc/iomem contents from IP27 and IP30? Maybe "lspci -vv" output, >>> too? I'm not sure where to look to understand the ioremap() behavior. >> >> ... >> A note about IP30, since that's part of an external patch set, the >> dmesg output is a little different, as it is using the newer >> BRIDGE/Xtalk code I just sent in as patches. ... >> >> Also, IP30 doesn't use the PCI_PROBE_ONLY thing anymore. This causes >> /proc/iomem to be far more detailed than it used to be. >> >> IP30's /proc/iomem: >> 00000000-00003fff : reserved >> 1d200000-1d9fffff : Bridge MEM >> d080000000-d080003fff : 0001:00:03.0 >> 1d204000-1d204fff : 0001:00:02.0 >> 1d205000-1d205fff : 0001:00:02.1 >> 1d206000-1d206fff : 0001:00:02.2 >> 1d207000-1d2070ff : 0001:00:02.3 >> 1d207100-1d20717f : 0001:00:01.0 >> 1f200000-1f9fffff : Bridge MEM >> f080100000-f0801fffff : 0000:00:02.0 >> f080010000-f08001ffff : 0000:00:00.0 >> f080030000-f08003ffff : 0000:00:01.0 >> f080000000-f080000fff : 0000:00:00.0 >> f080020000-f080020fff : 0000:00:01.0 >> 20004000-209b8fff : reserved >> 20004000-206acb13 : Kernel code >> 206acb14-208affff : Kernel data >> 209b9000-20efffff : System RAM >> 20f00000-20ffffff : System RAM >> 21000000-9fffffff : System RAM >> f080100000-f0801fffff : ioc3 > >>From the dmesg you attached (ip30-dmesg-20170208.txt, PCI parts > appended below): > > pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff] > pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff] > > These are shown correctly in /proc/iomem. It would be nice if the > /proc/iomem string identified which bridge was which (x86 uses "PCI > Bus 0000:00", "PCI Bus 0001:00", etc.) but that's not essential. > > pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff] > ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit. > > This is funky. We read 0x1d200000 from the BAR (PCI bus addresses are > apparently identical to CPU physical addresses). We claimed that > space from the host bridge window, so /proc/iomem would have looked > like this at that point: > > 1d200000-1d9fffff : Bridge MEM > 1d200000-1d203fff : 0001:00:03.0 > > But it looks like whatever this ip30-bridge thing is overwrote the > 0001:00:03.0 resource, which makes /proc/iomem wrong. > > All the resources under the bridge to 0000:00 are wrong, too: > > 1f200000-1f9fffff : Bridge MEM > f080100000-f0801fffff : 0000:00:02.0 > f080010000-f08001ffff : 0000:00:00.0 > f080030000-f08003ffff : 0000:00:01.0 > f080000000-f080000fff : 0000:00:00.0 > f080020000-f080020fff : 0000:00:01.0 Actually, that's not wrong. It took me a long time to figure this out, too, but once I stumbled upon the BRIDGE ASIC specification on an anonymous FTP site, via Google, by complete accident (hint), it finally made sense. BRIDGE is what SGI calls the chip that interfaces the Crosstalk (Xtalk) bus to a PCI bus. Most known "XIO" (Xtalk I/O) boards contain a BRIDGE chip under a heatsink at the edge of the board, next to a "compression connector". Octane can hold up to four XIO boards, with a fifth slot dedicated to an optional "PCI Shoebox" (literally a metal box that can hold up to three PCI or PCI-X cards). One exception to this are the two graphics boards, Impact/MGRAS and Odyssey/VPro, which are "pure" XIO devices, as far as anyone's figured out. Each BRIDGE chip can address up to eight different PCI devices. Crosstalk on IP30 is managed by the Crossbow (XBOW) ASIC. It implements a crossbar switch that's not totally unlike a networking switch. A single XBOW supports up to 16 XIO widgets, numbered 0x0 to 0xf. Widgets 0x1 to 0x7 are for internal use or unused. XBOW itself is widget 0x0. HEART, Octane's system controller, is widget 0x8. The four standard XIO slots are numbered (I think) as 0x9 to 0xc. I think widget 0xe is unused. The system board has a BRIDGE chip on it to provide the "BaseIO" devices, such as two SCSI (QL1040B), IOC3 Ethernet, KB/Mouse, Serial/Parallel, Audio, and an RTC. This BaseIO BRIDGE is widget 0xf, which is what you're seeing with the first set of addresses starting with 0x1fxxx. The second BRIDGE that you're seeing is the PCI Shoebox I have installed. It's widget 0xd, and has addresses starting with 0x1dxxx (second nibble from left is the widget), with three random PCI cards I found in a storage bin plugged into it. The Xtalk scan happens in reverse order, starting at widget 0xf down to 0x0. Both 0x1fxxx and 0x1dxxx addresses are "small window" addresses. The HEART system controller gives you three "windows" into Xtalk space, which uses 48-bit addressing: 0x0000_1000_0000 - 0x0000_1fff_ffff - Small windows, x16 @ 16MB each 0x0008_0000_0000 - 0x000f_ffff_ffff - Medium windows, x16 @ 2GB each 0x0010_0000_0000 - 0x00ff_ffff_ffff - Big windows, x15 @ 64GB each So a device on widget 0xf mapped into a small window at 0x0000_1fxx_xxxx could also be referred to by a big window at address 0x00f0_8xxx_xxxx. The only difference is the big window gives you a larger address space to play around with. Medium windows are not currently used on the IP30 platform in Linux. Widget 0x0 (XBOW) is only accessible via small or medium windows. As such, this layout in /proc/iomem is oddly correct: > 1f200000-1f9fffff : Bridge MEM > f080100000-f0801fffff : 0000:00:02.0 > f080010000-f08001ffff : 0000:00:00.0 > f080030000-f08003ffff : 0000:00:01.0 > f080000000-f080000fff : 0000:00:00.0 > f080020000-f080020fff : 0000:00:01.0 My bug is probably that I do the initial BRIDGE scan with small windows in the main BRIDGE driver (arch/mips/pci/pci-bridge.c), and then IP30's BRIDGE glue driver (arch/mips/sgi-ip30/ip30-bridge.c) is switching to big windows to probe each device. So, yes, different address ranges, but still looking at the same device. I'll look into fixing that. It's partially tied to how I was kludging the IP27 to also start doing PCI correctly within its own BRIDGE glue driver (ip27-bridge.c), but I just learned over the weekend that there's missing functionality I have to port over from old 2.5.x-era IA64 code. In IP30, big windows are available by default, but on IP27, what it calls "big windows" need to go through some kind of translation table, and I'm trying to figure out how to set that up. There's also an issue with DMA on BRIDGE on IP30 that limits the system's memory to 2GB to avoid kernel panics. OpenBSD's figured it out -- BRIDGE's IOMMU is busted so it can only do 31-bit DMA max. I just haven't figured out how to set that limit in Linux for the entire BRIDGE, not just for individual devices. > We read values from the BARs: > > pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff] > pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff] > pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref] > pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff] > pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref] > pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff] > pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff] > > These aren't zero, so somebody apparently has assigned them, but they > aren't inside the host bridge window. Therefore the PCI core assumes > they will not work, and it reassigns space from the window: > > pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff] > pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref] > pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref] > pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff] > pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff] Hmm, from the PCI device's point of view, 0x00200000-0x00200fff would be correct. The same address beginning with 0x1fxxx is the Crosstalk view of the device. Odds are likely I am not masking something off somewhere, but somehow, it all still finds a way to work. > I don't know why we didn't assign space to 0000:00:03.0 BAR 1. The PCI device sitting at 0000:00:03.0 is the "RAD Audio" chip. There's an issue with the driver that I keep forgetting to try and fix, so I've left it out of my recent kernel builds. That's probably why no space is being assigned to it. >>From /proc/iomem, I would expect that only the USB devices (0001:00:02 > functions 0, 1, 2, 3) would work. But apparently your system does > actually work, so there must be corresponding funkiness somewhere else > that compensates for these resources. USB is...interesting on this platform. Some USB devices will work, but not many have been tested. If my memory recalls, I think UHCI was known to play nice, but OHCI had issues -- might be the other way around. EHCI is dependent on what you plug in and the position of several planets in the sky. xHCI has endianess issues with one card that I tried. It's hard to find PCI or PCI-X xHCI cards, though. Most everything now is PCIe. And due to chassis space restrictions, using PCIe-to-PCI-X adapters is a no-go. PCI-to-PCI bridges and USB hubs are totally busted, because BRIDGE's PCI Type 1 configuration space isn't very well understood. The BRIDGE Specification I think finally cleared that up, but I have not tried to learn how to actually set the space up and find a spare USB Hub to play with. All-in-all, the entire machine is as much a work of art in its inner workings as how it looks on the outside. As with all art, it's up to the eye of the beholder on whether that's a good thing or not </smirk> -- Joshua Kinard Gentoo/MIPS kumba@gentoo.org 6144R/F5C6C943 2015-04-27 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-14 7:39 ` Joshua Kinard @ 2017-02-14 14:56 ` Bjorn Helgaas 2017-02-24 8:50 ` Joshua Kinard 0 siblings, 1 reply; 28+ messages in thread From: Bjorn Helgaas @ 2017-02-14 14:56 UTC (permalink / raw) To: Joshua Kinard Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci On Tue, Feb 14, 2017 at 02:39:45AM -0500, Joshua Kinard wrote: > On 02/13/2017 17:45, Bjorn Helgaas wrote: > > It looks like IP30 is using some code that's not upstream yet, so I'll > > just point out some things that don't look right. I'm ignoring the > > IP27 stuff here in case that has different issues. > > Sorry about that. It's an external set of patches I've been maintaining for > the last decade (I'm not the original author). Being that I do this for a > hobby, sometimes I get waylaid by other priorities. I've spent some time > re-organizing my patch collection lately so that I can start sending these > things in. This series and an earlier one to the Linux/MIPS list are the > beginnings of that, as time permits. However, I've got a ways to go before > getting to the IP30 patches. E.g., IP27 comes before IP30, so that's where my > focus has been lately. > > I've put an 4.10-rc7 tree with all patches applied here (including the IOC3, > IP27, and IP30 patches): > http://dev.gentoo.org/~kumba/mips/ip30/linux-4.10_rc7-20170208.ip30/ > > The patchset itself, if interested: > http://dev.gentoo.org/~kumba/mips/ip30/mips-patches-4.10.0/ Thanks for the pointers. > > All the resources under the bridge to 0000:00 are wrong, too: > > > > 1f200000-1f9fffff : Bridge MEM > > f080100000-f0801fffff : 0000:00:02.0 > > f080010000-f08001ffff : 0000:00:00.0 > > f080030000-f08003ffff : 0000:00:01.0 > > f080000000-f080000fff : 0000:00:00.0 > > f080020000-f080020fff : 0000:00:01.0 > > Actually, that's not wrong. It took me a long time to figure this out, too, > but once I stumbled upon the BRIDGE ASIC specification on an anonymous FTP > site, via Google, by complete accident (hint), it finally made sense. > > BRIDGE is what SGI calls the chip that interfaces the Crosstalk (Xtalk) bus to > a PCI bus. Most known "XIO" (Xtalk I/O) boards contain a BRIDGE chip under a > heatsink at the edge of the board, next to a "compression connector". Octane > can hold up to four XIO boards, with a fifth slot dedicated to an optional "PCI > Shoebox" (literally a metal box that can hold up to three PCI or PCI-X cards). > One exception to this are the two graphics boards, Impact/MGRAS and > Odyssey/VPro, which are "pure" XIO devices, as far as anyone's figured out. > > Each BRIDGE chip can address up to eight different PCI devices. > > Crosstalk on IP30 is managed by the Crossbow (XBOW) ASIC. It implements a > crossbar switch that's not totally unlike a networking switch. A single XBOW > supports up to 16 XIO widgets, numbered 0x0 to 0xf. Widgets 0x1 to 0x7 are for > internal use or unused. XBOW itself is widget 0x0. HEART, Octane's system > controller, is widget 0x8. The four standard XIO slots are numbered (I think) > as 0x9 to 0xc. I think widget 0xe is unused. > > The system board has a BRIDGE chip on it to provide the "BaseIO" devices, such > as two SCSI (QL1040B), IOC3 Ethernet, KB/Mouse, Serial/Parallel, Audio, and an > RTC. This BaseIO BRIDGE is widget 0xf, which is what you're seeing with the > first set of addresses starting with 0x1fxxx. The second BRIDGE that you're > seeing is the PCI Shoebox I have installed. It's widget 0xd, and has addresses > starting with 0x1dxxx (second nibble from left is the widget), with three > random PCI cards I found in a storage bin plugged into it. The Xtalk scan > happens in reverse order, starting at widget 0xf down to 0x0. > > Both 0x1fxxx and 0x1dxxx addresses are "small window" addresses. The HEART > system controller gives you three "windows" into Xtalk space, which uses 48-bit > addressing: > > 0x0000_1000_0000 - 0x0000_1fff_ffff - Small windows, x16 @ 16MB each > 0x0008_0000_0000 - 0x000f_ffff_ffff - Medium windows, x16 @ 2GB each > 0x0010_0000_0000 - 0x00ff_ffff_ffff - Big windows, x15 @ 64GB each > > So a device on widget 0xf mapped into a small window at 0x0000_1fxx_xxxx could > also be referred to by a big window at address 0x00f0_8xxx_xxxx. The only > difference is the big window gives you a larger address space to play around > with. Medium windows are not currently used on the IP30 platform in Linux. > Widget 0x0 (XBOW) is only accessible via small or medium windows. > > As such, this layout in /proc/iomem is oddly correct: > > > 1f200000-1f9fffff : Bridge MEM > > f080100000-f0801fffff : 0000:00:02.0 > > f080010000-f08001ffff : 0000:00:00.0 > > f080030000-f08003ffff : 0000:00:01.0 > > f080000000-f080000fff : 0000:00:00.0 > > f080020000-f080020fff : 0000:00:01.0 > > My bug is probably that I do the initial BRIDGE scan with small windows in the > main BRIDGE driver (arch/mips/pci/pci-bridge.c), and then IP30's BRIDGE glue > driver (arch/mips/sgi-ip30/ip30-bridge.c) is switching to big windows to probe > each device. So, yes, different address ranges, but still looking at the same > device. I'll look into fixing that. It's partially tied to how I was kludging > the IP27 to also start doing PCI correctly within its own BRIDGE glue driver > (ip27-bridge.c), but I just learned over the weekend that there's missing > functionality I have to port over from old 2.5.x-era IA64 code. In IP30, big > windows are available by default, but on IP27, what it calls "big windows" need > to go through some kind of translation table, and I'm trying to figure out how > to set that up. I agree, it sounds like the problem is the switch from small windows to big ones. In order for the PCI core to work correctly, the host bridge window ("1f200000-1f9fffff : Bridge MEM") must enclose the BARs of the devices below the bridge. If you use the big windows for the device BARs, you should use big windows for the host bridge windows. I think we're talking about widget 0xf here, so the 0xf big window would be 0xf0_0000_0000 - 0xff_ffff_ffff. This should all be set up *before* calling pci_scan_root_bus() instead of after, as it seems to be today. The scan would look like this: pci_bus 0000:00: root bus resource [mem 0xf000000000-0xffffffffff] (bus addresses [0x00000000-0xfffffffff]) pci 0000:00:00.0: reg 0x14: [mem 0xf000200000-0xf000200fff] pci 0000:00:00.0: reg 0x30: [mem 0xf000210000-0xf00021ffff pref] pci 0000:00:01.0: reg 0x14: [mem 0xf000400000-0xf000400fff] pci 0000:00:01.0: reg 0x30: [mem 0xf000410000-0xf00041ffff pref] pci 0000:00:02.0: reg 0x10: [mem 0xf000500000-0xf0005fffff] pci 0000:00:03.0: reg 0x10: [mem 0xf000600000-0xf000601fff] This would make /proc/iomem look like this: f000000000-ffffffffff : Bridge MEM f000500000-f0005fffff : 0000:00:02.0 f000210000-f00021ffff : 0000:00:00.0 f000410000-f00041ffff : 0000:00:01.0 f000200000-f000200fff : 0000:00:00.0 f000400000-f000400fff : 0000:00:01.0 This doesn't match the device BARs in your /proc/iomem, so there must be some other transformation going on as well. As long as you tell the PCI core about the host bridge windows you're going to use, along with offsets that include *all* these transformations, the core should just work, and /proc/iomem should also make sense. The details of small/medium/big windows, widgets, etc., are immaterial to the core. Bjorn ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-14 14:56 ` Bjorn Helgaas @ 2017-02-24 8:50 ` Joshua Kinard 2017-02-24 18:38 ` Bjorn Helgaas 0 siblings, 1 reply; 28+ messages in thread From: Joshua Kinard @ 2017-02-24 8:50 UTC (permalink / raw) To: Bjorn Helgaas Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci On 02/14/2017 09:56, Bjorn Helgaas wrote: > On Tue, Feb 14, 2017 at 02:39:45AM -0500, Joshua Kinard wrote: >> On 02/13/2017 17:45, Bjorn Helgaas wrote: >>> It looks like IP30 is using some code that's not upstream yet, so I'll >>> just point out some things that don't look right. I'm ignoring the >>> IP27 stuff here in case that has different issues. >> >> Sorry about that. It's an external set of patches I've been maintaining for >> the last decade (I'm not the original author). Being that I do this for a >> hobby, sometimes I get waylaid by other priorities. I've spent some time >> re-organizing my patch collection lately so that I can start sending these >> things in. This series and an earlier one to the Linux/MIPS list are the >> beginnings of that, as time permits. However, I've got a ways to go before >> getting to the IP30 patches. E.g., IP27 comes before IP30, so that's where my >> focus has been lately. >> >> I've put an 4.10-rc7 tree with all patches applied here (including the IOC3, >> IP27, and IP30 patches): >> http://dev.gentoo.org/~kumba/mips/ip30/linux-4.10_rc7-20170208.ip30/ >> >> The patchset itself, if interested: >> http://dev.gentoo.org/~kumba/mips/ip30/mips-patches-4.10.0/ > > Thanks for the pointers. > >>> All the resources under the bridge to 0000:00 are wrong, too: >>> >>> 1f200000-1f9fffff : Bridge MEM >>> f080100000-f0801fffff : 0000:00:02.0 >>> f080010000-f08001ffff : 0000:00:00.0 >>> f080030000-f08003ffff : 0000:00:01.0 >>> f080000000-f080000fff : 0000:00:00.0 >>> f080020000-f080020fff : 0000:00:01.0 >> >> Actually, that's not wrong. It took me a long time to figure this out, too, >> but once I stumbled upon the BRIDGE ASIC specification on an anonymous FTP >> site, via Google, by complete accident (hint), it finally made sense. >> >> BRIDGE is what SGI calls the chip that interfaces the Crosstalk (Xtalk) bus to >> a PCI bus. Most known "XIO" (Xtalk I/O) boards contain a BRIDGE chip under a >> heatsink at the edge of the board, next to a "compression connector". Octane >> can hold up to four XIO boards, with a fifth slot dedicated to an optional "PCI >> Shoebox" (literally a metal box that can hold up to three PCI or PCI-X cards). >> One exception to this are the two graphics boards, Impact/MGRAS and >> Odyssey/VPro, which are "pure" XIO devices, as far as anyone's figured out. >> >> Each BRIDGE chip can address up to eight different PCI devices. >> >> Crosstalk on IP30 is managed by the Crossbow (XBOW) ASIC. It implements a >> crossbar switch that's not totally unlike a networking switch. A single XBOW >> supports up to 16 XIO widgets, numbered 0x0 to 0xf. Widgets 0x1 to 0x7 are for >> internal use or unused. XBOW itself is widget 0x0. HEART, Octane's system >> controller, is widget 0x8. The four standard XIO slots are numbered (I think) >> as 0x9 to 0xc. I think widget 0xe is unused. >> >> The system board has a BRIDGE chip on it to provide the "BaseIO" devices, such >> as two SCSI (QL1040B), IOC3 Ethernet, KB/Mouse, Serial/Parallel, Audio, and an >> RTC. This BaseIO BRIDGE is widget 0xf, which is what you're seeing with the >> first set of addresses starting with 0x1fxxx. The second BRIDGE that you're >> seeing is the PCI Shoebox I have installed. It's widget 0xd, and has addresses >> starting with 0x1dxxx (second nibble from left is the widget), with three >> random PCI cards I found in a storage bin plugged into it. The Xtalk scan >> happens in reverse order, starting at widget 0xf down to 0x0. >> >> Both 0x1fxxx and 0x1dxxx addresses are "small window" addresses. The HEART >> system controller gives you three "windows" into Xtalk space, which uses 48-bit >> addressing: >> >> 0x0000_1000_0000 - 0x0000_1fff_ffff - Small windows, x16 @ 16MB each >> 0x0008_0000_0000 - 0x000f_ffff_ffff - Medium windows, x16 @ 2GB each >> 0x0010_0000_0000 - 0x00ff_ffff_ffff - Big windows, x15 @ 64GB each >> >> So a device on widget 0xf mapped into a small window at 0x0000_1fxx_xxxx could >> also be referred to by a big window at address 0x00f0_8xxx_xxxx. The only >> difference is the big window gives you a larger address space to play around >> with. Medium windows are not currently used on the IP30 platform in Linux. >> Widget 0x0 (XBOW) is only accessible via small or medium windows. >> >> As such, this layout in /proc/iomem is oddly correct: >> >>> 1f200000-1f9fffff : Bridge MEM >>> f080100000-f0801fffff : 0000:00:02.0 >>> f080010000-f08001ffff : 0000:00:00.0 >>> f080030000-f08003ffff : 0000:00:01.0 >>> f080000000-f080000fff : 0000:00:00.0 >>> f080020000-f080020fff : 0000:00:01.0 >> >> My bug is probably that I do the initial BRIDGE scan with small windows in the >> main BRIDGE driver (arch/mips/pci/pci-bridge.c), and then IP30's BRIDGE glue >> driver (arch/mips/sgi-ip30/ip30-bridge.c) is switching to big windows to probe >> each device. So, yes, different address ranges, but still looking at the same >> device. I'll look into fixing that. It's partially tied to how I was kludging >> the IP27 to also start doing PCI correctly within its own BRIDGE glue driver >> (ip27-bridge.c), but I just learned over the weekend that there's missing >> functionality I have to port over from old 2.5.x-era IA64 code. In IP30, big >> windows are available by default, but on IP27, what it calls "big windows" need >> to go through some kind of translation table, and I'm trying to figure out how >> to set that up. > > I agree, it sounds like the problem is the switch from small windows > to big ones. In order for the PCI core to work correctly, the host > bridge window ("1f200000-1f9fffff : Bridge MEM") must enclose the BARs > of the devices below the bridge. > > If you use the big windows for the device BARs, you should use big > windows for the host bridge windows. I think we're talking about > widget 0xf here, so the 0xf big window would be 0xf0_0000_0000 - > 0xff_ffff_ffff. This should all be set up *before* calling > pci_scan_root_bus() instead of after, as it seems to be today. Okay, so after a week of downtime due to hardware failure on another machine, I started looking at this again, and it looks like it's more than just getting the windows wrong. The existing BRIDGE code (pci-bridge.c) is limiting the PCI window to the first five devices on BaseIO in the small window (my fault), and the IP30 glue code (ip30-bridge.c) is re-writing the PCI BARs to access the BRIDGE PCI Memory and PCI I/O spaces via big windows, without updating the original mapping. We're not teaching the Linux PCI core at all, I think, about how to generically access PCI Memory Space and PCI I/O Space on the BRIDGE. The fact this actually works appears to be pure luck. Quoting from the BRIDGE docs somewhat, from the point of view of generic Crosstalk space and not specific to IP30 or IP27, there are several views into PCI space. Each address is 48-bits, the size of a Crosstalk address. 0000_0000_0000 to 0000_00ff_ffff is "Widget space". It seems the BRIDGE docs use the term "widget" here to refer to the eight possible PCI devices addressable by a single BRIDGE ASIC. This plus the small windows is how the code is currently getting things to work. I think. 0000_4000_0000 to 0000_7fff_ffff is BRIDGE's view into PCI Memory Space for direct-mapped 32-bit devices. It is 1GB in size. My take is this is normally a 4GB space? Can the Linux PCI core be taught that only 1GB is usable? 0000_8000_0000 to 0000_bfff_ffff is an alias view by BRIDGE into PCI Memory space for 64-bit PCI devices, immediately after the 32-bit space. It is also 1GB in size. 0000_c000_0000 to 0000_ffff_ffff is 2GB adjacent to PCI Memory space, but is either unused or used for Crosstalk/BRIDGE DMA operations (the docs aren't very clear on this point or I am simply not understanding). 0001_0000_0000 to 0001_ffff_ffff is BRIDGE's view into PCI I/O Space. It has the entire 4GB range available to use. I suspect the existing BRIDGE driver is mapping through "widget space" first to probe the PCI device slots, then the ip30-bridge.c code is re-writing the PCI BARs to go through the big window on IP30, but it never updates the original BRIDGE setup (if that's even possible). As such, as you pointed out, there's some kind of logic in the generic PCI core that's re-assigning space from the window and we're just getting lucky and everything still works. I dug up some old debugging code given to me by the original author of the IP30 port and built up a set of macros that get me to the PCI Configuration Space on Widget 0xf (BaseIO Bridge), Device #0 (the first QLA1040B SCSI controller). All three addresses access the same PCI device and return the same config space info. Small window: 0x900000001f02xxxx Medium window: 0x9000000f8002xxxx Big window: 0x900000f00002xxxx Each additional device's configuration space is offset 0x1000, which gives me the following addresses (dev's 4 to 7 are special-cased for IRQ trickery, so can't probe them): - Dev #0: scsi0/qla1040b: 0x90xxxxxxxxx20000 - Dev #1: scsi1/qla1040b: 0x90xxxxxxxxx21000 - Dev #2: io/ioc3 : 0x90xxxxxxxxx22000 - Dev #3: audio/rad1 : 0x90xxxxxxxxx23000 If I probe and dump the config space data for each, I get the following: PCI slot 0 information: vendor ID : 0x1077 device ID : 0x1020 command : 0x0006 status : 0x0200 revision : 0x05 prog if : 0x00 class : 0x0100 cache line : 0x40 latency : 0x40 hdr type : 0x00 BIST : 0x00 region 0 : 0x00200001 region 1 : 0x00200000 region 2 : 0x00000000 region 3 : 0x00000000 region 4 : 0x00000000 region 5 : 0x00000000 IRQ line : 0x00 IRQ pin : 0x01 PCI slot 1 information: vendor ID : 0x1077 device ID : 0x1020 command : 0x0006 status : 0x0200 revision : 0x05 prog if : 0x00 class : 0x0100 cache line : 0x40 latency : 0x40 hdr type : 0x00 BIST : 0x00 region 0 : 0x00400001 region 1 : 0x00400000 region 2 : 0x00000000 region 3 : 0x00000000 region 4 : 0x00000000 region 5 : 0x00000000 IRQ line : 0x00 IRQ pin : 0x01 PCI slot 2 information: vendor ID : 0x10a9 device ID : 0x0003 command : 0x0146 status : 0x0280 revision : 0x01 prog if : 0x00 class : 0xff00 cache line : 0x00 latency : 0x28 hdr type : 0x00 BIST : 0x00 region 0 : 0x00500000 region 1 : 0x00000000 region 2 : 0x00000000 region 3 : 0x00500000 region 4 : 0x000310a9 region 5 : 0x02800146 IRQ line : 0x00 IRQ pin : 0x00 PCI slot 3 information: vendor ID : 0x10a9 device ID : 0x0005 command : 0x0006 status : 0x0480 revision : 0xc0 prog if : 0x00 class : 0x0000 cache line : 0x00 latency : 0xff hdr type : 0x00 BIST : 0x00 region 0 : 0x00600000 region 1 : 0x00000000 region 2 : 0x00000000 region 3 : 0x00000000 region 4 : 0x00000000 region 5 : 0x00000000 IRQ line : 0x00 IRQ pin : 0x00 This is where my knowledge runs short -- I'm not fluent in PCI device programming, so I'm not sure what is really supposed to be going on with these different BAR regions. I dug up a copy of the PCI Spec 2.2, but it's 322 pages and I'm not sure where I should start reading from. Are these regions device-specific? E.g., do I need the QLA1040B programming manual to understand why region 0 is 0x00200001 and region 1 is 0x00200000? I am also not yet considering how PCI views Crosstalk space for DMA operations -- BRIDGE has several mechanisms for that, depending on if the PCI device is a 32-bit device or a 64-bit device. It can do direct-mapping for 32-bit or 64-bit, or built-in page-mapping hardware is available (but apparently to be avoided due to numerous hardware quirks/bugs that would make the driver overly-complicated). There's a good write-up in the OpenBSD "xbridge" driver, which handles BRIDGE (IP27/IP30), XBRIDGE (IP35), and PIC (IP35/IA64 Altix): http://bxr.su/OpenBSD/sys/arch/sgi/xbow/xbridge.c It's starting to make some sense to me, but I am still uncertain how to work with the 32-bit and 64-bit 1GB PCI memory spaces on BRIDGE as well as the 4GB PCI I/O space from a Linux point of view. Do they only matter when dealing with DMA? --J > The scan would look like this: > > pci_bus 0000:00: root bus resource [mem 0xf000000000-0xffffffffff] (bus addresses [0x00000000-0xfffffffff]) > pci 0000:00:00.0: reg 0x14: [mem 0xf000200000-0xf000200fff] > pci 0000:00:00.0: reg 0x30: [mem 0xf000210000-0xf00021ffff pref] > pci 0000:00:01.0: reg 0x14: [mem 0xf000400000-0xf000400fff] > pci 0000:00:01.0: reg 0x30: [mem 0xf000410000-0xf00041ffff pref] > pci 0000:00:02.0: reg 0x10: [mem 0xf000500000-0xf0005fffff] > pci 0000:00:03.0: reg 0x10: [mem 0xf000600000-0xf000601fff] > > This would make /proc/iomem look like this: > > f000000000-ffffffffff : Bridge MEM > f000500000-f0005fffff : 0000:00:02.0 > f000210000-f00021ffff : 0000:00:00.0 > f000410000-f00041ffff : 0000:00:01.0 > f000200000-f000200fff : 0000:00:00.0 > f000400000-f000400fff : 0000:00:01.0 > > This doesn't match the device BARs in your /proc/iomem, so there must > be some other transformation going on as well. > > As long as you tell the PCI core about the host bridge windows you're > going to use, along with offsets that include *all* these > transformations, the core should just work, and /proc/iomem should > also make sense. The details of small/medium/big windows, widgets, > etc., are immaterial to the core. > > Bjorn > ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-24 8:50 ` Joshua Kinard @ 2017-02-24 18:38 ` Bjorn Helgaas 2017-02-25 9:34 ` Joshua Kinard 0 siblings, 1 reply; 28+ messages in thread From: Bjorn Helgaas @ 2017-02-24 18:38 UTC (permalink / raw) To: Joshua Kinard Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci On Fri, Feb 24, 2017 at 03:50:26AM -0500, Joshua Kinard wrote: > On 02/14/2017 09:56, Bjorn Helgaas wrote: > > On Tue, Feb 14, 2017 at 02:39:45AM -0500, Joshua Kinard wrote: > >> On 02/13/2017 17:45, Bjorn Helgaas wrote: > >>> It looks like IP30 is using some code that's not upstream yet, so I'll > >>> just point out some things that don't look right. I'm ignoring the > >>> IP27 stuff here in case that has different issues. > >> > >> Sorry about that. It's an external set of patches I've been maintaining for > >> the last decade (I'm not the original author). Being that I do this for a > >> hobby, sometimes I get waylaid by other priorities. I've spent some time > >> re-organizing my patch collection lately so that I can start sending these > >> things in. This series and an earlier one to the Linux/MIPS list are the > >> beginnings of that, as time permits. However, I've got a ways to go before > >> getting to the IP30 patches. E.g., IP27 comes before IP30, so that's where my > >> focus has been lately. > >> > >> I've put an 4.10-rc7 tree with all patches applied here (including the IOC3, > >> IP27, and IP30 patches): > >> http://dev.gentoo.org/~kumba/mips/ip30/linux-4.10_rc7-20170208.ip30/ > >> > >> The patchset itself, if interested: > >> http://dev.gentoo.org/~kumba/mips/ip30/mips-patches-4.10.0/ > > > > Thanks for the pointers. > > > >>> All the resources under the bridge to 0000:00 are wrong, too: > >>> > >>> 1f200000-1f9fffff : Bridge MEM > >>> f080100000-f0801fffff : 0000:00:02.0 > >>> f080010000-f08001ffff : 0000:00:00.0 > >>> f080030000-f08003ffff : 0000:00:01.0 > >>> f080000000-f080000fff : 0000:00:00.0 > >>> f080020000-f080020fff : 0000:00:01.0 > >> > >> Actually, that's not wrong. It took me a long time to figure this out, too, > >> but once I stumbled upon the BRIDGE ASIC specification on an anonymous FTP > >> site, via Google, by complete accident (hint), it finally made sense. > >> > >> BRIDGE is what SGI calls the chip that interfaces the Crosstalk (Xtalk) bus to > >> a PCI bus. Most known "XIO" (Xtalk I/O) boards contain a BRIDGE chip under a > >> heatsink at the edge of the board, next to a "compression connector". Octane > >> can hold up to four XIO boards, with a fifth slot dedicated to an optional "PCI > >> Shoebox" (literally a metal box that can hold up to three PCI or PCI-X cards). > >> One exception to this are the two graphics boards, Impact/MGRAS and > >> Odyssey/VPro, which are "pure" XIO devices, as far as anyone's figured out. > >> > >> Each BRIDGE chip can address up to eight different PCI devices. > >> > >> Crosstalk on IP30 is managed by the Crossbow (XBOW) ASIC. It implements a > >> crossbar switch that's not totally unlike a networking switch. A single XBOW > >> supports up to 16 XIO widgets, numbered 0x0 to 0xf. Widgets 0x1 to 0x7 are for > >> internal use or unused. XBOW itself is widget 0x0. HEART, Octane's system > >> controller, is widget 0x8. The four standard XIO slots are numbered (I think) > >> as 0x9 to 0xc. I think widget 0xe is unused. > >> > >> The system board has a BRIDGE chip on it to provide the "BaseIO" devices, such > >> as two SCSI (QL1040B), IOC3 Ethernet, KB/Mouse, Serial/Parallel, Audio, and an > >> RTC. This BaseIO BRIDGE is widget 0xf, which is what you're seeing with the > >> first set of addresses starting with 0x1fxxx. The second BRIDGE that you're > >> seeing is the PCI Shoebox I have installed. It's widget 0xd, and has addresses > >> starting with 0x1dxxx (second nibble from left is the widget), with three > >> random PCI cards I found in a storage bin plugged into it. The Xtalk scan > >> happens in reverse order, starting at widget 0xf down to 0x0. > >> > >> Both 0x1fxxx and 0x1dxxx addresses are "small window" addresses. The HEART > >> system controller gives you three "windows" into Xtalk space, which uses 48-bit > >> addressing: > >> > >> 0x0000_1000_0000 - 0x0000_1fff_ffff - Small windows, x16 @ 16MB each > >> 0x0008_0000_0000 - 0x000f_ffff_ffff - Medium windows, x16 @ 2GB each > >> 0x0010_0000_0000 - 0x00ff_ffff_ffff - Big windows, x15 @ 64GB each > >> > >> So a device on widget 0xf mapped into a small window at 0x0000_1fxx_xxxx could > >> also be referred to by a big window at address 0x00f0_8xxx_xxxx. The only > >> difference is the big window gives you a larger address space to play around > >> with. Medium windows are not currently used on the IP30 platform in Linux. > >> Widget 0x0 (XBOW) is only accessible via small or medium windows. > >> > >> As such, this layout in /proc/iomem is oddly correct: > >> > >>> 1f200000-1f9fffff : Bridge MEM > >>> f080100000-f0801fffff : 0000:00:02.0 > >>> f080010000-f08001ffff : 0000:00:00.0 > >>> f080030000-f08003ffff : 0000:00:01.0 > >>> f080000000-f080000fff : 0000:00:00.0 > >>> f080020000-f080020fff : 0000:00:01.0 > >> > >> My bug is probably that I do the initial BRIDGE scan with small windows in the > >> main BRIDGE driver (arch/mips/pci/pci-bridge.c), and then IP30's BRIDGE glue > >> driver (arch/mips/sgi-ip30/ip30-bridge.c) is switching to big windows to probe > >> each device. So, yes, different address ranges, but still looking at the same > >> device. I'll look into fixing that. It's partially tied to how I was kludging > >> the IP27 to also start doing PCI correctly within its own BRIDGE glue driver > >> (ip27-bridge.c), but I just learned over the weekend that there's missing > >> functionality I have to port over from old 2.5.x-era IA64 code. In IP30, big > >> windows are available by default, but on IP27, what it calls "big windows" need > >> to go through some kind of translation table, and I'm trying to figure out how > >> to set that up. > > > > I agree, it sounds like the problem is the switch from small windows > > to big ones. In order for the PCI core to work correctly, the host > > bridge window ("1f200000-1f9fffff : Bridge MEM") must enclose the BARs > > of the devices below the bridge. > > > > If you use the big windows for the device BARs, you should use big > > windows for the host bridge windows. I think we're talking about > > widget 0xf here, so the 0xf big window would be 0xf0_0000_0000 - > > 0xff_ffff_ffff. This should all be set up *before* calling > > pci_scan_root_bus() instead of after, as it seems to be today. > > Okay, so after a week of downtime due to hardware failure on another machine, I > started looking at this again, and it looks like it's more than just getting > the windows wrong. The existing BRIDGE code (pci-bridge.c) is limiting the PCI > window to the first five devices on BaseIO in the small window (my fault), and > the IP30 glue code (ip30-bridge.c) is re-writing the PCI BARs to access the > BRIDGE PCI Memory and PCI I/O spaces via big windows, without updating the > original mapping. We're not teaching the Linux PCI core at all, I think, about > how to generically access PCI Memory Space and PCI I/O Space on the BRIDGE. > The fact this actually works appears to be pure luck. I agree about the luck part :) > Quoting from the BRIDGE docs somewhat, from the point of view of generic > Crosstalk space and not specific to IP30 or IP27, there are several views into > PCI space. Each address is 48-bits, the size of a Crosstalk address. > > 0000_0000_0000 to 0000_00ff_ffff is "Widget space". It seems the BRIDGE > docs use the term "widget" here to refer to the eight possible PCI devices > addressable by a single BRIDGE ASIC. This plus the small windows is how > the code is currently getting things to work. I think. > > 0000_4000_0000 to 0000_7fff_ffff is BRIDGE's view into PCI Memory Space > for direct-mapped 32-bit devices. It is 1GB in size. My take is this is > normally a 4GB space? Can the Linux PCI core be taught that only 1GB > is usable? Bridges normally have a window that contains some 32-bit PCI memory space, because many PCI devices have 32-bit BARs that have to be located below 4GB. This window is usually smaller than 4GB (1GB would be typical) because these devices likely can only generate 32-bit DMA as well, and the DMA has to use 32-bit PCI addresses that are outside the host bridge window. The host bridge code, i.e., the pcibios_scanbus() path, tells the core how big the window is. In this case, "hose->mem_resource" contains the CPU physical address range (and size), and "host->mem_offset" contains the offset between the CPU physical address and the PCI bus address. So if "hose->mem_resource" is only 1GB, that's all the PCI core will use. > 0000_8000_0000 to 0000_bfff_ffff is an alias view by BRIDGE into PCI > Memory space for 64-bit PCI devices, immediately after the 32-bit space. > It is also 1GB in size. > > 0000_c000_0000 to 0000_ffff_ffff is 2GB adjacent to PCI Memory space, but > is either unused or used for Crosstalk/BRIDGE DMA operations > (the docs aren't very clear on this point or I am simply not > understanding). > > 0001_0000_0000 to 0001_ffff_ffff is BRIDGE's view into PCI I/O Space. > It has the entire 4GB range available to use. > > > I suspect the existing BRIDGE driver is mapping through "widget space" first to > probe the PCI device slots, then the ip30-bridge.c code is re-writing the PCI > BARs to go through the big window on IP30, but it never updates the original > BRIDGE setup (if that's even possible). As such, as you pointed out, there's > some kind of logic in the generic PCI core that's re-assigning space from the > window and we're just getting lucky and everything still works. > > I dug up some old debugging code given to me by the original author of the IP30 > port and built up a set of macros that get me to the PCI Configuration Space on > Widget 0xf (BaseIO Bridge), Device #0 (the first QLA1040B SCSI controller). > All three addresses access the same PCI device and return the same config space > info. > > Small window: 0x900000001f02xxxx > Medium window: 0x9000000f8002xxxx > Big window: 0x900000f00002xxxx > > > Each additional device's configuration space is offset 0x1000, which gives me > the following addresses (dev's 4 to 7 are special-cased for IRQ trickery, so > can't probe them): > > - Dev #0: scsi0/qla1040b: 0x90xxxxxxxxx20000 > - Dev #1: scsi1/qla1040b: 0x90xxxxxxxxx21000 > - Dev #2: io/ioc3 : 0x90xxxxxxxxx22000 > - Dev #3: audio/rad1 : 0x90xxxxxxxxx23000 PCIe functions have 4K of config space each, so a 0x1000 offset makes sense. Whether a platform can access all of it is a separate question. It looks like the type 0 config accessors (pci_conf0_read_config(), pci_conf0_write_config()) are basically like ECAM, where config space is memory-mapped. Per spec, the ECAM window for a bridge that could have buses 00-ff below it would be 256MB (256 buses * 32 devices/bus * 8 functions/device * 4096 bytes config space/function). But based on b_type0_cfg_dev[], it looks like SGI only made space for 8 devices on the root bus, each with 8 functions. I think that's OK, because they can control how many devices can be on the root bus. The type 1 config accessors (pci_conf1_read_config(), pci_conf1_write_config()) are for devices below a PCI-to-PCI bridge. It looks like there's a single 4K memory-mapped window, and you point it at a specific bus & device with bridge->b_pci_cfg. The Linux type 0 accessors look like they support 4K config space per function, while the type 1 accessors put the function number in bits 8-10, so it looks like they only support 256 bytes per function. The OpenBSD accessors (xbridge_conf_read() and xbridge_conf_write()) look like they only support 256 bytes per function regardless of whether it's type 0 or type 1. Supporting 4K of space for type 0 seems like a potential Linux problem. Also, pci_conf1_write_config() uses b_type0_cfg_dev in one place where it looks like it should be using b_type1_cfg. But that's in the IOC3 path, and I don't know if that even makes sense for non-root bus devices -- I doubt you can put an IOC3 behind a PCI-to-PCI bridge. If the hardware only supports 256 bytes of config space on non-root bus devices, that's not a disaster. We should still be able to enumerate them and use all the conventional PCI features and even basic PCIe features. But the extended config space (offsets 0x100-0xfff) would be inaccessible, and we wouldn't see any PCIe extended capabilities (AER, VC, SR-IOV, etc., see PCI_EXT_CAP_ID_ERR and subsequent definitions) because they live in that space. > If I probe and dump the config space data for each, I get the following: > > PCI slot 0 information: > vendor ID : 0x1077 > device ID : 0x1020 > command : 0x0006 > status : 0x0200 > revision : 0x05 > prog if : 0x00 > class : 0x0100 > cache line : 0x40 > latency : 0x40 > hdr type : 0x00 > BIST : 0x00 > region 0 : 0x00200001 > region 1 : 0x00200000 > region 2 : 0x00000000 > region 3 : 0x00000000 > region 4 : 0x00000000 > region 5 : 0x00000000 > IRQ line : 0x00 > IRQ pin : 0x01 > > PCI slot 1 information: > vendor ID : 0x1077 > device ID : 0x1020 > command : 0x0006 > status : 0x0200 > revision : 0x05 > prog if : 0x00 > class : 0x0100 > cache line : 0x40 > latency : 0x40 > hdr type : 0x00 > BIST : 0x00 > region 0 : 0x00400001 > region 1 : 0x00400000 > region 2 : 0x00000000 > region 3 : 0x00000000 > region 4 : 0x00000000 > region 5 : 0x00000000 > IRQ line : 0x00 > IRQ pin : 0x01 > > PCI slot 2 information: > vendor ID : 0x10a9 > device ID : 0x0003 > command : 0x0146 > status : 0x0280 > revision : 0x01 > prog if : 0x00 > class : 0xff00 > cache line : 0x00 > latency : 0x28 > hdr type : 0x00 > BIST : 0x00 > region 0 : 0x00500000 > region 1 : 0x00000000 > region 2 : 0x00000000 > region 3 : 0x00500000 > region 4 : 0x000310a9 > region 5 : 0x02800146 > IRQ line : 0x00 > IRQ pin : 0x00 > > PCI slot 3 information: > vendor ID : 0x10a9 > device ID : 0x0005 > command : 0x0006 > status : 0x0480 > revision : 0xc0 > prog if : 0x00 > class : 0x0000 > cache line : 0x00 > latency : 0xff > hdr type : 0x00 > BIST : 0x00 > region 0 : 0x00600000 > region 1 : 0x00000000 > region 2 : 0x00000000 > region 3 : 0x00000000 > region 4 : 0x00000000 > region 5 : 0x00000000 > IRQ line : 0x00 > IRQ pin : 0x00 > > This is where my knowledge runs short -- I'm not fluent in PCI device > programming, so I'm not sure what is really supposed to be going on with these > different BAR regions. I dug up a copy of the PCI Spec 2.2, but it's 322 pages > and I'm not sure where I should start reading from. Are these regions > device-specific? E.g., do I need the QLA1040B programming manual to understand > why region 0 is 0x00200001 and region 1 is 0x00200000? The regions (BARs) are definitely device-specific. The low order bit in some of them indicates an I/O BAR. In the PCI r3.0 spec, sec 6.2.5 covers "Base Addresses" and tells you how to interpret the low bits. For most of the BARs above, they're either 0b01, indicating an I/O BAR, or 0b00000, indicating a non-prefetchable 32-bit memory BAR. > I am also not yet considering how PCI views Crosstalk space for DMA operations > -- BRIDGE has several mechanisms for that, depending on if the PCI device is a > 32-bit device or a 64-bit device. It can do direct-mapping for 32-bit or > 64-bit, or built-in page-mapping hardware is available (but apparently to be > avoided due to numerous hardware quirks/bugs that would make the driver > overly-complicated). > > There's a good write-up in the OpenBSD "xbridge" driver, which handles BRIDGE > (IP27/IP30), XBRIDGE (IP35), and PIC (IP35/IA64 Altix): > > http://bxr.su/OpenBSD/sys/arch/sgi/xbow/xbridge.c > > It's starting to make some sense to me, but I am still uncertain how to work > with the 32-bit and 64-bit 1GB PCI memory spaces on BRIDGE as well as the 4GB > PCI I/O space from a Linux point of view. Do they only matter when dealing > with DMA? The host bridge windows (the "root bus resource" lines in dmesg) are only for PIO, i.e., a driver running on the CPU reading or writing registers on the PCI device (it could be actual registers, or a frame buffer, etc., but it resides on the device and its PCI bus address is determined by a BAR). The driver uses ioremap(), pci_iomap(), pcim_iomap_regions(), etc. to map these into the kernel virtual space. DMA is coming the other direction and the windows are irrelevant except that the target PCI bus address must be outside all the windows (if the DMA target address were *inside* a host bridge window, the bridge would assume it is intended for a PCI device, e.g., for peer-to-peer DMA). I think the biggest problem is that you need to set up the BRIDGE *before* calling pcibios_scan_bus(). That way the windows ("hose->mem_resource", "hose->io_resource", etc.) will be correct when the PCI core enumerates the devices. Note that you can have as many windows in the "&resources" list as you need -- the current code there only has one memory and one I/O window, but you can add more. > > The scan would look like this: > > > > pci_bus 0000:00: root bus resource [mem 0xf000000000-0xffffffffff] (bus addresses [0x00000000-0xfffffffff]) > > pci 0000:00:00.0: reg 0x14: [mem 0xf000200000-0xf000200fff] > > pci 0000:00:00.0: reg 0x30: [mem 0xf000210000-0xf00021ffff pref] > > pci 0000:00:01.0: reg 0x14: [mem 0xf000400000-0xf000400fff] > > pci 0000:00:01.0: reg 0x30: [mem 0xf000410000-0xf00041ffff pref] > > pci 0000:00:02.0: reg 0x10: [mem 0xf000500000-0xf0005fffff] > > pci 0000:00:03.0: reg 0x10: [mem 0xf000600000-0xf000601fff] > > > > This would make /proc/iomem look like this: > > > > f000000000-ffffffffff : Bridge MEM > > f000500000-f0005fffff : 0000:00:02.0 > > f000210000-f00021ffff : 0000:00:00.0 > > f000410000-f00041ffff : 0000:00:01.0 > > f000200000-f000200fff : 0000:00:00.0 > > f000400000-f000400fff : 0000:00:01.0 > > > > This doesn't match the device BARs in your /proc/iomem, so there must > > be some other transformation going on as well. > > > > As long as you tell the PCI core about the host bridge windows you're > > going to use, along with offsets that include *all* these > > transformations, the core should just work, and /proc/iomem should > > also make sense. The details of small/medium/big windows, widgets, > > etc., are immaterial to the core. > > > > Bjorn > > > ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-24 18:38 ` Bjorn Helgaas @ 2017-02-25 9:34 ` Joshua Kinard 2017-02-27 16:36 ` Bjorn Helgaas 0 siblings, 1 reply; 28+ messages in thread From: Joshua Kinard @ 2017-02-25 9:34 UTC (permalink / raw) To: Bjorn Helgaas Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci On 02/24/2017 13:38, Bjorn Helgaas wrote: > On Fri, Feb 24, 2017 at 03:50:26AM -0500, Joshua Kinard wrote: >> Quoting from the BRIDGE docs somewhat, from the point of view of generic >> Crosstalk space and not specific to IP30 or IP27, there are several views into >> PCI space. Each address is 48-bits, the size of a Crosstalk address. >> >> 0000_0000_0000 to 0000_00ff_ffff is "Widget space". It seems the BRIDGE >> docs use the term "widget" here to refer to the eight possible PCI devices >> addressable by a single BRIDGE ASIC. This plus the small windows is how >> the code is currently getting things to work. I think. >> >> 0000_4000_0000 to 0000_7fff_ffff is BRIDGE's view into PCI Memory Space >> for direct-mapped 32-bit devices. It is 1GB in size. My take is this is >> normally a 4GB space? Can the Linux PCI core be taught that only 1GB >> is usable? > > Bridges normally have a window that contains some 32-bit PCI memory > space, because many PCI devices have 32-bit BARs that have to be > located below 4GB. > > This window is usually smaller than 4GB (1GB would be typical) because > these devices likely can only generate 32-bit DMA as well, and the DMA > has to use 32-bit PCI addresses that are outside the host bridge > window. Okay, this fits then. I wasn't sure, because the verbiage in the BRIDGE docs kinda-suggests that the 1GB window for 32/64-bit memory space isn't common. Likely, what they're getting at is the fact the lower 1GB for 32-bit is aliased in the next 1GB for use by 64-bit PCI devices. It looks like the remaining 2GB in memory space is available for DMA, and I'll have to try and wrap my head around that at later date. > The host bridge code, i.e., the pcibios_scanbus() path, tells the core > how big the window is. In this case, "hose->mem_resource" contains > the CPU physical address range (and size), and "host->mem_offset" > contains the offset between the CPU physical address and the PCI bus > address. So if "hose->mem_resource" is only 1GB, that's all the PCI > core will use. So basically, this bit of code from the proposed pci-bridge.c needs additional work: bc->mem.name = "Bridge MEM"; bc->mem.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_MEM); bc->mem.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO - 1); bc->mem.flags = IORESOURCE_MEM; bc->io.name = "Bridge IO"; bc->io.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO); bc->io.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_END - 1); bc->io.flags = IORESOURCE_IO; The values I've been using for PCIBR_OFFSET_MEM and PCIBR_OFFSET_IO are obviously wrong. For IP30, they ultimately direct "BRIDGE MEM" to the first five device slots via small windows, and then "Bridge IO" to the last two slots. This is one of those "luck" moments that happened to work on BaseIO, but is probably why devices in the secondary PCI shoebox are hit-or-miss. Per your comment further down that I can tell the PCI core about //multiple// windows, probably what I want to do is hardwire access to the small window spaces in the main BRIDGE driver, with machine-specific offsets passed via platform_data. Small windows, for both IP30 and IP27, are always guaranteed to be available without any additional magic. This window range can be used to do the initial slot probes to query devices for their desired BAR values. Then, in IP30-specific or IP27-specific code, we add in additional memory windows as needed. IP30 would simply pass along the hardwired address ranges for BRIDGE MEM or BRIDGE IO via both medium and big window addresses. IP27 is trickier -- we'll need some logic that attempts to use a small window (16MB) mapping first, but if that is too small, we'll have to setup an entry in the HUB's IOTTE buffer that maps a specific block of physical memory to Crosstalk address space. This is what IP27 calls "big windows", of which there are seven entries maximum, on 512MB boundaries, per HUB chip. However, only six entries are available because one IOTTE is used to handle a known hardware bug on early HUB revisions, leaving only six left. As such, the additional window information will go into the machine-specific BRIDGE glue drivers (ip30-bridge.c or ip27-bridge.c). Sound sane? >> 0000_8000_0000 to 0000_bfff_ffff is an alias view by BRIDGE into PCI >> Memory space for 64-bit PCI devices, immediately after the 32-bit space. >> It is also 1GB in size. >> >> 0000_c000_0000 to 0000_ffff_ffff is 2GB adjacent to PCI Memory space, but >> is either unused or used for Crosstalk/BRIDGE DMA operations >> (the docs aren't very clear on this point or I am simply not >> understanding). >> >> 0001_0000_0000 to 0001_ffff_ffff is BRIDGE's view into PCI I/O Space. >> It has the entire 4GB range available to use. >> >> >> I suspect the existing BRIDGE driver is mapping through "widget space" first to >> probe the PCI device slots, then the ip30-bridge.c code is re-writing the PCI >> BARs to go through the big window on IP30, but it never updates the original >> BRIDGE setup (if that's even possible). As such, as you pointed out, there's >> some kind of logic in the generic PCI core that's re-assigning space from the >> window and we're just getting lucky and everything still works. >> >> I dug up some old debugging code given to me by the original author of the IP30 >> port and built up a set of macros that get me to the PCI Configuration Space on >> Widget 0xf (BaseIO Bridge), Device #0 (the first QLA1040B SCSI controller). >> All three addresses access the same PCI device and return the same config space >> info. >> >> Small window: 0x900000001f02xxxx >> Medium window: 0x9000000f8002xxxx >> Big window: 0x900000f00002xxxx >> >> >> Each additional device's configuration space is offset 0x1000, which gives me >> the following addresses (dev's 4 to 7 are special-cased for IRQ trickery, so >> can't probe them): >> >> - Dev #0: scsi0/qla1040b: 0x90xxxxxxxxx20000 >> - Dev #1: scsi1/qla1040b: 0x90xxxxxxxxx21000 >> - Dev #2: io/ioc3 : 0x90xxxxxxxxx22000 >> - Dev #3: audio/rad1 : 0x90xxxxxxxxx23000 > > PCIe functions have 4K of config space each, so a 0x1000 offset makes > sense. Whether a platform can access all of it is a separate > question. No need to worry about PCIe on this platform. You're talking about 15+-year old hardware that was built to last. I believe SGI only implemented up to PCI 2.1 in the BRIDGE and XBRIDGE ASICs. It's possible the PIC ASIC might do 2.2 or 3.0. But I don't have to worry about PIC for a good while, as that's only found in the IP35-class of hardware (Origin 300/350/3000, Fuel, Tezro, Onyx4, etc). We need fully-working IP27 and IP30 first, because much of the working logic can then be re-purposed for bringing up IP35 hardware. Especially since IA64 did a lot of the work already for the Altix systems, which is just IP35 with an IA64 CPU, but I digress... > It looks like the type 0 config accessors (pci_conf0_read_config(), > pci_conf0_write_config()) are basically like ECAM, where config space > is memory-mapped. > > Per spec, the ECAM window for a bridge that could have buses 00-ff > below it would be 256MB (256 buses * 32 devices/bus * 8 > functions/device * 4096 bytes config space/function). > > But based on b_type0_cfg_dev[], it looks like SGI only made space for > 8 devices on the root bus, each with 8 functions. I think that's OK, > because they can control how many devices can be on the root bus. Not sure what ECAM is, but yes, SGI wired each BRIDGE ASIC for a maximum of 8 PCI devices (or slots). There can be multiple BRIDGEs in an SGI system, though, and for several of their XIO add-in boards, you have one PCI device hardwired into a BRIDGE ASIC, so whatever logic I cook up is going to need to handle BRIDGE's with one device or eight. So much for this being easy... > The type 1 config accessors (pci_conf1_read_config(), > pci_conf1_write_config()) are for devices below a PCI-to-PCI bridge. > It looks like there's a single 4K memory-mapped window, and you point > it at a specific bus & device with bridge->b_pci_cfg. Yeah, PCI-to-PCI bridges are a known-broken item right now. Since there appears to only be a single Type 1 config space, if I have a PCI Shoebox with three PCI-X slots, and I stick three USB 2.0 boards in them, I have to pick which one can handle PCI-to-PCI bridges, like USB hubs, and program that into this Type 1 register, right? > The Linux type 0 accessors look like they support 4K config space per > function, while the type 1 accessors put the function number in bits > 8-10, so it looks like they only support 256 bytes per function. > > The OpenBSD accessors (xbridge_conf_read() and xbridge_conf_write()) > look like they only support 256 bytes per function regardless of > whether it's type 0 or type 1. I'll have to pass this question to the OpenBSD dev who maintains the xbridge driver. He understands this hardware far better than I do, and there might be a reason why they do it that way. > Supporting 4K of space for type 0 seems like a potential Linux > problem. Also, pci_conf1_write_config() uses b_type0_cfg_dev in one > place where it looks like it should be using b_type1_cfg. But that's > in the IOC3 path, and I don't know if that even makes sense for > non-root bus devices -- I doubt you can put an IOC3 behind a > PCI-to-PCI bridge. IOC3 is....special. You'll never see one of those beind a PCI-to-PCI bridge, thankfully. That device is evil enough just sitting exposed on the main PCI bus. Just look at the comments in mainline arch/mips/pci/ops-bridge.c for some rather flavourful language regarding IOC3. Basically, IOC3: - Claims it's a single-function device but is really multi-function. - Implements only half of the PCI configuration space. - Read/write ops to the unimplemented PCI config registers are undefined. Living "behind" the IOC3 device is your ethernet, serial ports (RS232 and RS422), parallel port, keyboard/mouse PS/2 ports, and the RTC chip. The IA64 people have an IOC3 "metadriver" in drivers/sn/ioc3.c that takes care of dealing with that monster, and one of the later patches I've yet to send in moves that driver to drivers/misc/ioc3.c (next to ioc4.c) for use by the SGI MIPS platforms. But that is a ways off, as we need working Xtalk code and working BRIDGE code before the IOC3 metadriver can be worked on. > If the hardware only supports 256 bytes of config space on non-root > bus devices, that's not a disaster. We should still be able to > enumerate them and use all the conventional PCI features and even > basic PCIe features. But the extended config space (offsets > 0x100-0xfff) would be inaccessible, and we wouldn't see any PCIe > extended capabilities (AER, VC, SR-IOV, etc., see PCI_EXT_CAP_ID_ERR > and subsequent definitions) because they live in that space. I do not think that the BRIDGE supports AER, VC, or SR-IOV at all, primarily due to age of the hardware. Even if a PCI device that does support those is plugged into a BRIDGE, I am not 100% certain those features would even be usable if the BRIDGE doesn't know about them. Or is this a case of where the BRIDGE wouldn't care and once it programs the BARs correctly, these features would be available to the Linux drivers? >> If I probe and dump the config space data for each, I get the following: >> >> PCI slot 0 information: >> vendor ID : 0x1077 >> device ID : 0x1020 >> command : 0x0006 >> status : 0x0200 >> revision : 0x05 >> prog if : 0x00 >> class : 0x0100 >> cache line : 0x40 >> latency : 0x40 >> hdr type : 0x00 >> BIST : 0x00 >> region 0 : 0x00200001 >> region 1 : 0x00200000 >> region 2 : 0x00000000 >> region 3 : 0x00000000 >> region 4 : 0x00000000 >> region 5 : 0x00000000 >> IRQ line : 0x00 >> IRQ pin : 0x01 >> >> PCI slot 1 information: >> vendor ID : 0x1077 >> device ID : 0x1020 >> command : 0x0006 >> status : 0x0200 >> revision : 0x05 >> prog if : 0x00 >> class : 0x0100 >> cache line : 0x40 >> latency : 0x40 >> hdr type : 0x00 >> BIST : 0x00 >> region 0 : 0x00400001 >> region 1 : 0x00400000 >> region 2 : 0x00000000 >> region 3 : 0x00000000 >> region 4 : 0x00000000 >> region 5 : 0x00000000 >> IRQ line : 0x00 >> IRQ pin : 0x01 >> >> PCI slot 2 information: >> vendor ID : 0x10a9 >> device ID : 0x0003 >> command : 0x0146 >> status : 0x0280 >> revision : 0x01 >> prog if : 0x00 >> class : 0xff00 >> cache line : 0x00 >> latency : 0x28 >> hdr type : 0x00 >> BIST : 0x00 >> region 0 : 0x00500000 >> region 1 : 0x00000000 >> region 2 : 0x00000000 >> region 3 : 0x00500000 >> region 4 : 0x000310a9 >> region 5 : 0x02800146 >> IRQ line : 0x00 >> IRQ pin : 0x00 >> >> PCI slot 3 information: >> vendor ID : 0x10a9 >> device ID : 0x0005 >> command : 0x0006 >> status : 0x0480 >> revision : 0xc0 >> prog if : 0x00 >> class : 0x0000 >> cache line : 0x00 >> latency : 0xff >> hdr type : 0x00 >> BIST : 0x00 >> region 0 : 0x00600000 >> region 1 : 0x00000000 >> region 2 : 0x00000000 >> region 3 : 0x00000000 >> region 4 : 0x00000000 >> region 5 : 0x00000000 >> IRQ line : 0x00 >> IRQ pin : 0x00 >> >> This is where my knowledge runs short -- I'm not fluent in PCI device >> programming, so I'm not sure what is really supposed to be going on with these >> different BAR regions. I dug up a copy of the PCI Spec 2.2, but it's 322 pages >> and I'm not sure where I should start reading from. Are these regions >> device-specific? E.g., do I need the QLA1040B programming manual to understand >> why region 0 is 0x00200001 and region 1 is 0x00200000? > > The regions (BARs) are definitely device-specific. The low order bit > in some of them indicates an I/O BAR. In the PCI r3.0 spec, sec 6.2.5 > covers "Base Addresses" and tells you how to interpret the low bits. > For most of the BARs above, they're either 0b01, indicating an I/O > BAR, or 0b00000, indicating a non-prefetchable 32-bit memory BAR. Okay, I stumbled across this URL: http://moi.vonos.net/linux/the-pci-bus/ Which gives a good, plain-english breakdown of BARs and whatnot. I remember from the OpenBSD xbridge driver, it states that it currently uses the BAR mappings setup by the ARCS firmware, but the plan is to eventually move away from those mappings and calculate its own. I didn't quite get that, but after reading that site, now I do. ARCS is writing ~0 to the BARs to query the device to figure out how much memory each BAR wants, then programs in some ranges for us. By tallying up the total requested size of all BAR mappings on a given device, we can then make a determination on whether we can use a small, medium, or large window and then program the BAR with the correct address. Right? Could also do my own probes (or maybe Linux already does this?), but for the short-term, using what ARCS has already setup might not be a bad idea to get things working again. >> I am also not yet considering how PCI views Crosstalk space for DMA operations >> -- BRIDGE has several mechanisms for that, depending on if the PCI device is a >> 32-bit device or a 64-bit device. It can do direct-mapping for 32-bit or >> 64-bit, or built-in page-mapping hardware is available (but apparently to be >> avoided due to numerous hardware quirks/bugs that would make the driver >> overly-complicated). >> >> There's a good write-up in the OpenBSD "xbridge" driver, which handles BRIDGE >> (IP27/IP30), XBRIDGE (IP35), and PIC (IP35/IA64 Altix): >> >> http://bxr.su/OpenBSD/sys/arch/sgi/xbow/xbridge.c >> >> It's starting to make some sense to me, but I am still uncertain how to work >> with the 32-bit and 64-bit 1GB PCI memory spaces on BRIDGE as well as the 4GB >> PCI I/O space from a Linux point of view. Do they only matter when dealing >> with DMA? > > The host bridge windows (the "root bus resource" lines in dmesg) are > only for PIO, i.e., a driver running on the CPU reading or writing > registers on the PCI device (it could be actual registers, or a frame > buffer, etc., but it resides on the device and its PCI bus address is > determined by a BAR). The driver uses ioremap(), pci_iomap(), > pcim_iomap_regions(), etc. to map these into the kernel virtual space. So for the MIPS case, knowing that ioremap() and friends is not guaranteed to return a workable virtual address, I need to be careful of what addresses I program into each BAR. E.g., given that on IP30, if a physical address starts with 0x9000000f800xxxxx, it is using medium windows to talk to a device on the BaseIO BRIDGE (Xtalk widget 0xf). As such, knowing MIPS' ioremap() returns, for the R10000 CPU case, the requested address OR'ed with IO_BASE (0x9000000000000000), I want to tell the PCI core to use 0x0000000f800xxxxx so that ioremap() will return back 0x9000000f800xxxxx? And since I can apparently specify multiple window ranges for memory space and I/O space, I probably want to, as stated earlier, specify all three of IP30's known window ranges for each BRIDGE so that the Linux PCI core can walk each resource struct and find a matching window? If that thinking is correct, then I have some idea of how to set that up and then see if things start working again on IP30 and then eventually, IP27. > DMA is coming the other direction and the windows are irrelevant > except that the target PCI bus address must be outside all the windows > (if the DMA target address were *inside* a host bridge window, the > bridge would assume it is intended for a PCI device, e.g., for > peer-to-peer DMA). DMA is the tricky one. OpenBSD's xbridge driver implies that BRIDGE's IOMMU has a bug of some kind in it that restricts our DMA-able address ranges to 31 bits, or 0x00000000 to 0x7fffffff. IP27 doesn't seem to be bothered by this in the current code, and as such, I can use all 8GB of RAM on that platform. But IP30 has its physical memory offset 512MB, so physical memory maps begin at 0x20000000 and run to 0x9fffffff. This means someone playing with Linux on an IP30 platform can only install up to 2GB of RAM right now, if they want the machine to be stable. Though, there are other issues going on that imply that "stable" is a very subjective term... In either event, how do I teach the Linux PCI core that the BRIDGE itself is limited to 31bits of DMA-able address space? There seems to be plenty of documentation on having an individual PCI device set this up by setting its DMA mask, but I can't seem to find any wording on how to do this for an entire PCI bus. Or is this an architecture-specific thing? MIPS has the dma-coherence.h header for defining system-specific DMA quirks, but that code seems heavily biased towards PCI devices, and on IP30 at least, the Impact and Odyssey video cards are NOT PCI devices at all, instead being native XIO devices that support DMA operations. > I think the biggest problem is that you need to set up the BRIDGE > *before* calling pcibios_scan_bus(). That way the windows > ("hose->mem_resource", "hose->io_resource", etc.) will be correct when > the PCI core enumerates the devices. Note that you can have as many > windows in the "&resources" list as you need -- the current code there > only has one memory and one I/O window, but you can add more. Agreed, however, I think this is actually being done to some extent already. We're configuring BRIDGE-specific properties and writing some values to BRIDGE registers and the per-slot registers in bridge_probe() in the generic pci-bridge.c driver, then, at the very end, calling register_pci_controller(), which I believe is what kicks off the PCI bus scan. The IP30 BRIDGE glue code adds a new function hook called "pre_enable" where it does, in ip30-bridge.c, some additional PCI device tweaking, but this code will run after the PCI bus scan has happened. My guess is that I want to reverse this and have the IP30 glue code twiddle the PCI devices on a given BRIDGE before the PCI bus scan, right? That code uses pci_read_config_dword() and pci_write_config_dword() -- are these functions safe to use if we haven't already done a PCI bus scan? Thanks! -- Joshua Kinard Gentoo/MIPS kumba@gentoo.org 6144R/F5C6C943 2015-04-27 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-25 9:34 ` Joshua Kinard @ 2017-02-27 16:36 ` Bjorn Helgaas 2017-02-28 0:25 ` Joshua Kinard 0 siblings, 1 reply; 28+ messages in thread From: Bjorn Helgaas @ 2017-02-27 16:36 UTC (permalink / raw) To: Joshua Kinard Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci On Sat, Feb 25, 2017 at 04:34:12AM -0500, Joshua Kinard wrote: > On 02/24/2017 13:38, Bjorn Helgaas wrote: > > On Fri, Feb 24, 2017 at 03:50:26AM -0500, Joshua Kinard wrote: > > The host bridge code, i.e., the pcibios_scanbus() path, tells the > > core how big the window is. In this case, "hose->mem_resource" > > contains the CPU physical address range (and size), and > > "host->mem_offset" contains the offset between the CPU physical > > address and the PCI bus address. So if "hose->mem_resource" is > > only 1GB, that's all the PCI core will use. > > So basically, this bit of code from the proposed pci-bridge.c needs > additional work: > > bc->mem.name = "Bridge MEM"; > bc->mem.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_MEM); > bc->mem.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO - 1); > bc->mem.flags = IORESOURCE_MEM; > > bc->io.name = "Bridge IO"; > bc->io.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO); > bc->io.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_END - 1); > bc->io.flags = IORESOURCE_IO; > > The values I've been using for PCIBR_OFFSET_MEM and PCIBR_OFFSET_IO > are obviously wrong. For IP30, they ultimately direct "BRIDGE MEM" > to the first five device slots via small windows, and then "Bridge > IO" to the last two slots. This is one of those "luck" moments that > happened to work on BaseIO, but is probably why devices in the > secondary PCI shoebox are hit-or-miss. > > Per your comment further down that I can tell the PCI core about > //multiple// windows, probably what I want to do is hardwire access > to the small window spaces in the main BRIDGE driver, with > machine-specific offsets passed via platform_data. Small windows, > for both IP30 and IP27, are always guaranteed to be available > without any additional magic. This window range can be used to do > the initial slot probes to query devices for their desired BAR > values. > > Then, in IP30-specific or IP27-specific code, we add in additional > memory windows as needed. IP30 would simply pass along the > hardwired address ranges for BRIDGE MEM or BRIDGE IO via both medium > and big window addresses. > > IP27 is trickier -- we'll need some logic that attempts to use a > small window (16MB) mapping first, but if that is too small, we'll > have to setup an entry in the HUB's IOTTE buffer that maps a > specific block of physical memory to Crosstalk address space. pcibios_scanbus() builds the &resources list of host bridge windows, then calls pci_scan_root_bus(). If you're proposing to change the list of host bridge windows *after* calling pci_scan_root_bus(), that sounds a little problematic because we do not have a PCI core interface to do that. > This is what IP27 calls "big windows", of which there are seven > entries maximum, on 512MB boundaries, per HUB chip. However, only > six entries are available because one IOTTE is used to handle a > known hardware bug on early HUB revisions, leaving only six left. > > As such, the additional window information will go into the > machine-specific BRIDGE glue drivers (ip30-bridge.c or > ip27-bridge.c). > > Sound sane? Do you need to allocate the big windows based on what PCI devices you find? If so, that is going to be a hard problem. > > PCIe functions have 4K of config space each, so a 0x1000 offset > > makes sense. Whether a platform can access all of it is a > > separate question. > > No need to worry about PCIe on this platform. You're talking about > 15+-year old hardware that was built to last. I believe SGI only > implemented up to PCI 2.1 in the BRIDGE and XBRIDGE ASICs. It's > possible the PIC ASIC might do 2.2 or 3.0. But I don't have to > worry about PIC for a good while, as that's only found in the > IP35-class of hardware (Origin 300/350/3000, Fuel, Tezro, Onyx4, > etc). We need fully-working IP27 and IP30 first, because much of > the working logic can then be re-purposed for bringing up IP35 > hardware. Especially since IA64 did a lot of the work already for > the Altix systems, which is just IP35 with an IA64 CPU, but I > digress... OK. If you only plug in PCI devices, there should be no problem. If you did plug in PCIe devices (via a PCI-to-PCIe bridge, for example), the core should still enumerate them correctly and they should be functional, though some PCIe-only features like AER won't work. > > The type 1 config accessors (pci_conf1_read_config(), > > pci_conf1_write_config()) are for devices below a PCI-to-PCI > > bridge. It looks like there's a single 4K memory-mapped window, > > and you point it at a specific bus & device with > > bridge->b_pci_cfg. > > Yeah, PCI-to-PCI bridges are a known-broken item right now. Since > there appears to only be a single Type 1 config space, if I have a > PCI Shoebox with three PCI-X slots, and I stick three USB 2.0 boards > in them, I have to pick which one can handle PCI-to-PCI bridges, > like USB hubs, and program that into this Type 1 register, right? I don't think USB hubs are relevant here because they are completely in the USB domain. The USB 2.0 board is a USB host controller with a PCI interface on it. The USB hub is on the USB side and is invisible to the PCI core. The way I read the BSD code, it looks like you should be able to use the Type 1 config interface to generate accesses to any PCI bus, as long as you serialize them, e.g., with a lock around the use of b_type1_cfg. > > The Linux type 0 accessors look like they support 4K config space > > per function, while the type 1 accessors put the function number > > in bits 8-10, so it looks like they only support 256 bytes per > > function. > > > > The OpenBSD accessors (xbridge_conf_read() and > > xbridge_conf_write()) look like they only support 256 bytes per > > function regardless of whether it's type 0 or type 1. > > I'll have to pass this question to the OpenBSD dev who maintains the > xbridge driver. He understands this hardware far better than I do, > and there might be a reason why they do it that way. If you only support PCI devices, 256 bytes of config space is enough. The 4K config space is only supported for PCI-X Mode 2 and PCIe devices. Even those PCI-X Mode 2 and PCIe devices should be functional with only 256 bytes. > > Supporting 4K of space for type 0 seems like a potential Linux > > problem. Also, pci_conf1_write_config() uses b_type0_cfg_dev in > > one place where it looks like it should be using b_type1_cfg. I didn't word this well. I was trying to point out things that look like bugs in pci_conf1_write_config() (the use of b_type0_cfg_dev) and maybe pci_conf0_read_config() (the fact that it allows 4K config space but the hardware probably only supports 256 bytes). > > If the hardware only supports 256 bytes of config space on > > non-root bus devices, that's not a disaster. We should still be > > able to enumerate them and use all the conventional PCI features > > and even basic PCIe features. But the extended config space > > (offsets 0x100-0xfff) would be inaccessible, and we wouldn't see > > any PCIe extended capabilities (AER, VC, SR-IOV, etc., see > > PCI_EXT_CAP_ID_ERR and subsequent definitions) because they live > > in that space. > > I do not think that the BRIDGE supports AER, VC, or SR-IOV at all, > primarily due to age of the hardware. Even if a PCI device that > does support those is plugged into a BRIDGE, I am not 100% certain > those features would even be usable if the BRIDGE doesn't know about > them. Or is this a case of where the BRIDGE wouldn't care and once > it programs the BARs correctly, these features would be available to > the Linux drivers? These are mostly PCIe features and I don't think you should worry about them, at least for now. > I remember from the OpenBSD xbridge driver, it states that it > currently uses the BAR mappings setup by the ARCS firmware, but the > plan is to eventually move away from those mappings and calculate > its own. I didn't quite get that, but after reading that site, now > I do. ARCS is writing ~0 to the BARs to query the device to figure > out how much memory each BAR wants, then programs in some ranges for > us. > > By tallying up the total requested size of all BAR mappings on a > given device, we can then make a determination on whether we can use > a small, medium, or large window and then program the BAR with the > correct address. Right? This is the problem I alluded to above: Linux does not currently support anything like this. We assume the Linux host bridge driver knows the window sizes at the beginning of time (it may learn them from firmware or it may have hard-coded knowledge of the address map). Linux does enumerate the devices and tally up all the BAR sizes (see pci_bus_assign_resources()), but it does not have a way to change the host bridge windows based on how much BAR space we need. The common thing is to use whatever host bridge windows and device BAR values were set up by firmware. If those are all sensible, Linux won't change anything. > > The host bridge windows (the "root bus resource" lines in dmesg) > > are only for PIO, i.e., a driver running on the CPU reading or > > writing registers on the PCI device (it could be actual registers, > > or a frame buffer, etc., but it resides on the device and its PCI > > bus address is determined by a BAR). The driver uses ioremap(), > > pci_iomap(), pcim_iomap_regions(), etc. to map these into the > > kernel virtual space. > > So for the MIPS case, knowing that ioremap() and friends is not > guaranteed to return a workable virtual address, I need to be > careful of what addresses I program into each BAR. E.g., given that > on IP30, if a physical address starts with 0x9000000f800xxxxx, it is > using medium windows to talk to a device on the BaseIO BRIDGE (Xtalk > widget 0xf). As such, knowing MIPS' ioremap() returns, for the > R10000 CPU case, the requested address OR'ed with IO_BASE > (0x9000000000000000), I want to tell the PCI core to use > 0x0000000f800xxxxx so that ioremap() will return back > 0x9000000f800xxxxx? If ioremap() doesn't return a virtual address, or at least something that can be used like a virtual address, I think that's fundamentally broken. All drivers assume they should ioremap() a BAR resource, i.e., the CPU physical address that maps to a PCI BAR value, and use the result as a virtual address. Documentation/DMA-API-HOWTO.txt has a picture that may help clarify the different address spaces. > And since I can apparently specify multiple window ranges for memory > space and I/O space, I probably want to, as stated earlier, specify > all three of IP30's known window ranges for each BRIDGE so that the > Linux PCI core can walk each resource struct and find a matching > window? If all three windows are disjoint, you can specify them all. If a range in the small window is aliased and can also be reached via a medium or large window, you should not specify both. > > DMA is coming the other direction and the windows are irrelevant > > except that the target PCI bus address must be outside all the > > windows (if the DMA target address were *inside* a host bridge > > window, the bridge would assume it is intended for a PCI device, > > e.g., for peer-to-peer DMA). > > DMA is the tricky one. OpenBSD's xbridge driver implies that > BRIDGE's IOMMU has a bug of some kind in it that restricts our > DMA-able address ranges to 31 bits, or 0x00000000 to 0x7fffffff. > IP27 doesn't seem to be bothered by this in the current code, and as > such, I can use all 8GB of RAM on that platform. > > But IP30 has its physical memory offset 512MB, so physical memory > maps begin at 0x20000000 and run to 0x9fffffff. This means someone > playing with Linux on an IP30 platform can only install up to 2GB of > RAM right now, if they want the machine to be stable. Though, there > are other issues going on that imply that "stable" is a very > subjective term... > > In either event, how do I teach the Linux PCI core that the BRIDGE > itself is limited to 31bits of DMA-able address space? There seems > to be plenty of documentation on having an individual PCI device set > this up by setting its DMA mask, but I can't seem to find any > wording on how to do this for an entire PCI bus. > > Or is this an architecture-specific thing? MIPS has the > dma-coherence.h header for defining system-specific DMA quirks, but > that code seems heavily biased towards PCI devices, and on IP30 at > least, the Impact and Odyssey video cards are NOT PCI devices at > all, instead being native XIO devices that support DMA operations. The PCI core isn't involved much with DMA, so I don't know this off the top of my head. > > I think the biggest problem is that you need to set up the BRIDGE > > *before* calling pcibios_scan_bus(). That way the windows > > ("hose->mem_resource", "hose->io_resource", etc.) will be correct > > when the PCI core enumerates the devices. Note that you can have > > as many windows in the "&resources" list as you need -- the > > current code there only has one memory and one I/O window, but you > > can add more. > > Agreed, however, I think this is actually being done to some extent > already. We're configuring BRIDGE-specific properties and writing > some values to BRIDGE registers and the per-slot registers in > bridge_probe() in the generic pci-bridge.c driver, then, at the very > end, calling register_pci_controller(), which I believe is what > kicks off the PCI bus scan. > > The IP30 BRIDGE glue code adds a new function hook called > "pre_enable" where it does, in ip30-bridge.c, some additional PCI > device tweaking, but this code will run after the PCI bus scan has > happened. My guess is that I want to reverse this and have the IP30 > glue code twiddle the PCI devices on a given BRIDGE before the PCI > bus scan, right? Theoretically you should not need to do any PCI device tweaking: PCI devices are basically arch-independent and shouldn't need arch-specific tweaks. All the arch stuff should be encapsulated in the host bridge driver, i.e., the code that sets up the window list and calls pci_scan_root_bus(). In the MIPS case, this code is kind of spread out and doesn't really look like a single "driver". The ideal thing would be if you can set up the bridge to always use large windows. Then the PCI core will enumerate the devices and set up their resources automatically. I assume the small/medium windows exist because there's not enough address space to always use the large windows. I don't have any good suggestions for that -- we don't have support for adjusting the window sizes based on what devices we find. > That code uses pci_read_config_dword() and > pci_write_config_dword() -- are these functions safe to use if we > haven't already done a PCI bus scan? You can't use pci_read_config_dword() before we scan the bus because it requires a "struct pci_dev *", and those are created during the bus scan. Bjorn ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-27 16:36 ` Bjorn Helgaas @ 2017-02-28 0:25 ` Joshua Kinard 2017-03-01 15:39 ` Bjorn Helgaas 0 siblings, 1 reply; 28+ messages in thread From: Joshua Kinard @ 2017-02-28 0:25 UTC (permalink / raw) To: Bjorn Helgaas Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci On 02/27/2017 11:36, Bjorn Helgaas wrote: > On Sat, Feb 25, 2017 at 04:34:12AM -0500, Joshua Kinard wrote: >> On 02/24/2017 13:38, Bjorn Helgaas wrote: >>> On Fri, Feb 24, 2017 at 03:50:26AM -0500, Joshua Kinard wrote: > >>> The host bridge code, i.e., the pcibios_scanbus() path, tells the >>> core how big the window is. In this case, "hose->mem_resource" >>> contains the CPU physical address range (and size), and >>> "host->mem_offset" contains the offset between the CPU physical >>> address and the PCI bus address. So if "hose->mem_resource" is >>> only 1GB, that's all the PCI core will use. >> >> So basically, this bit of code from the proposed pci-bridge.c needs >> additional work: >> >> bc->mem.name = "Bridge MEM"; >> bc->mem.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_MEM); >> bc->mem.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO - 1); >> bc->mem.flags = IORESOURCE_MEM; >> >> bc->io.name = "Bridge IO"; >> bc->io.start = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_IO); >> bc->io.end = (NODE_SWIN_BASE(nasid, widget_id) + PCIBR_OFFSET_END - 1); >> bc->io.flags = IORESOURCE_IO; >> >> The values I've been using for PCIBR_OFFSET_MEM and PCIBR_OFFSET_IO >> are obviously wrong. For IP30, they ultimately direct "BRIDGE MEM" >> to the first five device slots via small windows, and then "Bridge >> IO" to the last two slots. This is one of those "luck" moments that >> happened to work on BaseIO, but is probably why devices in the >> secondary PCI shoebox are hit-or-miss. >> >> Per your comment further down that I can tell the PCI core about >> //multiple// windows, probably what I want to do is hardwire access >> to the small window spaces in the main BRIDGE driver, with >> machine-specific offsets passed via platform_data. Small windows, >> for both IP30 and IP27, are always guaranteed to be available >> without any additional magic. This window range can be used to do >> the initial slot probes to query devices for their desired BAR >> values. >> >> Then, in IP30-specific or IP27-specific code, we add in additional >> memory windows as needed. IP30 would simply pass along the >> hardwired address ranges for BRIDGE MEM or BRIDGE IO via both medium >> and big window addresses. >> >> IP27 is trickier -- we'll need some logic that attempts to use a >> small window (16MB) mapping first, but if that is too small, we'll >> have to setup an entry in the HUB's IOTTE buffer that maps a >> specific block of physical memory to Crosstalk address space. > > pcibios_scanbus() builds the &resources list of host bridge windows, > then calls pci_scan_root_bus(). If you're proposing to change the > list of host bridge windows *after* calling pci_scan_root_bus(), that > sounds a little problematic because we do not have a PCI core > interface to do that. I believe I can determine the size of the windows on BRIDGE before hitting the generic PCI code. The bridge_probe() function calls register_pci_controller() at the very bottom, even in the current mainline file (pci-ip27.c), after it's twiddled a number of BRIDGE register bits. I'll first have to try and understand the 'pre_enable' code from the IP30 patch that changes the window mappings in ip30-bridge.c, then find a way to move that code to execute earlier before any of the generic PCI code, since it contains some of the logic to size out the window mappings. I'll see about getting it to work on IP30 (Octane) first, because all three window spaces are available without any special magic. That way, I'll know what can be done in the generic BRIDGE driver and what needs to happen in platform-specific files. That'll make figuring out the IP27 logic a lot easier....I hope. >> This is what IP27 calls "big windows", of which there are seven >> entries maximum, on 512MB boundaries, per HUB chip. However, only >> six entries are available because one IOTTE is used to handle a >> known hardware bug on early HUB revisions, leaving only six left. >> >> As such, the additional window information will go into the >> machine-specific BRIDGE glue drivers (ip30-bridge.c or >> ip27-bridge.c). >> >> Sound sane? > > Do you need to allocate the big windows based on what PCI devices you > find? If so, that is going to be a hard problem. I believe so. The IOTTE stuff in the HUB ASIC on IP27 appears to behave like an extremely simplified TLB, and once you work out how much space a given PCI device needs, then you can calculate out the matching Crosstalk address, and you write that to an unused IOTTE slot. It doesn't have to be accurate -- each IOTTE entry needs to be aligned on a 512MB boundary, so once you've set up a mapping, if another PCI device (or BAR?) needs a Crosstalk address within an already-mapped boundary, you just point it to the existing IOTTE entry that matches. That's my understanding anyways after looking at really old IA64 code in 2.5.70 that used virtually the same hardware when SGI was bringing up the Altix platform. Specifically, hub_piomap_alloc() on Line 117 in Linux-2.5.70/arch/ia64/sn/io/io.c: https://git.linux-mips.org/cgit/ralf/linux.git/tree/arch/ia64/sn/io/io.c?h=linux-2.5.70 So I just need to make sure all of this happens during the initial bridge_probe() function, so that everything is ready before calling register_pci_controller(). We'll find out! >>> PCIe functions have 4K of config space each, so a 0x1000 offset >>> makes sense. Whether a platform can access all of it is a >>> separate question. >> >> No need to worry about PCIe on this platform. You're talking about >> 15+-year old hardware that was built to last. I believe SGI only >> implemented up to PCI 2.1 in the BRIDGE and XBRIDGE ASICs. It's >> possible the PIC ASIC might do 2.2 or 3.0. But I don't have to >> worry about PIC for a good while, as that's only found in the >> IP35-class of hardware (Origin 300/350/3000, Fuel, Tezro, Onyx4, >> etc). We need fully-working IP27 and IP30 first, because much of >> the working logic can then be re-purposed for bringing up IP35 >> hardware. Especially since IA64 did a lot of the work already for >> the Altix systems, which is just IP35 with an IA64 CPU, but I >> digress... > > OK. If you only plug in PCI devices, there should be no problem. If > you did plug in PCIe devices (via a PCI-to-PCIe bridge, for example), > the core should still enumerate them correctly and they should be > functional, though some PCIe-only features like AER won't work. There might be physical limitations on using any kind of PCIe bridge device. Getting a PCI device into an SGI Octane (IP30) or Origin 2000/Onyx2 (IP27) that isn't either the BaseIO on IP30 or IO6 on IP27 requires an XIO Shoehorn or PCI Shoebox. There's a little bit of leeway on how wide of a PCI card you can stuff into either one, but it's not much. I'll have to measure a shoehorn if I can find one of my spares in a bin someplace. The Origin 200 tower machine might let you get away with it...but that's a corner case in my book and not really worth worrying about it. It's hard to find one of those completely intact these days, let alone with a working power supply that doesn't double as a fireworks display. >>> The type 1 config accessors (pci_conf1_read_config(), >>> pci_conf1_write_config()) are for devices below a PCI-to-PCI >>> bridge. It looks like there's a single 4K memory-mapped window, >>> and you point it at a specific bus & device with >>> bridge->b_pci_cfg. >> >> Yeah, PCI-to-PCI bridges are a known-broken item right now. Since >> there appears to only be a single Type 1 config space, if I have a >> PCI Shoebox with three PCI-X slots, and I stick three USB 2.0 boards >> in them, I have to pick which one can handle PCI-to-PCI bridges, >> like USB hubs, and program that into this Type 1 register, right? > > I don't think USB hubs are relevant here because they are completely > in the USB domain. The USB 2.0 board is a USB host controller with a > PCI interface on it. The USB hub is on the USB side and is invisible > to the PCI core. > > The way I read the BSD code, it looks like you should be able to use > the Type 1 config interface to generate accesses to any PCI bus, as > long as you serialize them, e.g., with a lock around the use of > b_type1_cfg. Huh, well, I've never actually tried a USB Hub device on my Octane. If I can fix PCI up properly and get a 2.0 card to actually work, I'll try it out before looking too deeply into the Type 1 stuff. >>> The Linux type 0 accessors look like they support 4K config space >>> per function, while the type 1 accessors put the function number >>> in bits 8-10, so it looks like they only support 256 bytes per >>> function. >>> >>> The OpenBSD accessors (xbridge_conf_read() and >>> xbridge_conf_write()) look like they only support 256 bytes per >>> function regardless of whether it's type 0 or type 1. >> >> I'll have to pass this question to the OpenBSD dev who maintains the >> xbridge driver. He understands this hardware far better than I do, >> and there might be a reason why they do it that way. > > If you only support PCI devices, 256 bytes of config space is enough. > The 4K config space is only supported for PCI-X Mode 2 and PCIe > devices. Even those PCI-X Mode 2 and PCIe devices should be > functional with only 256 bytes. It's unknown why SGI left that much space between the registers on the BRIDGE. Could've been future-proofing, or maybe there really is a PCI or PCI-X device out there that currently only works under IRIX that needed it all. If I can get things to work better than they are now, I've got a bit of a collection of PCI devices to test things out with. Might expose any wrong assumptions. >>> Supporting 4K of space for type 0 seems like a potential Linux >>> problem. Also, pci_conf1_write_config() uses b_type0_cfg_dev in >>> one place where it looks like it should be using b_type1_cfg. > > I didn't word this well. I was trying to point out things that look > like bugs in pci_conf1_write_config() (the use of b_type0_cfg_dev) and > maybe pci_conf0_read_config() (the fact that it allows 4K config space > but the hardware probably only supports 256 bytes). If you're referring to those uses in ops-bridge.c, that file probably needs some additional TLC. The patchset I've got fixed it up a little, but looking at it made my head hurt (and not just because of the IOC3 voodoo in it), so I kinda passed on spending too much time on it. I'll probably have to double back on it in the future after fixing the other areas, so I'll keep this in mind. >>> If the hardware only supports 256 bytes of config space on >>> non-root bus devices, that's not a disaster. We should still be >>> able to enumerate them and use all the conventional PCI features >>> and even basic PCIe features. But the extended config space >>> (offsets 0x100-0xfff) would be inaccessible, and we wouldn't see >>> any PCIe extended capabilities (AER, VC, SR-IOV, etc., see >>> PCI_EXT_CAP_ID_ERR and subsequent definitions) because they live >>> in that space. >> >> I do not think that the BRIDGE supports AER, VC, or SR-IOV at all, >> primarily due to age of the hardware. Even if a PCI device that >> does support those is plugged into a BRIDGE, I am not 100% certain >> those features would even be usable if the BRIDGE doesn't know about >> them. Or is this a case of where the BRIDGE wouldn't care and once >> it programs the BARs correctly, these features would be available to >> the Linux drivers? > > These are mostly PCIe features and I don't think you should worry > about them, at least for now. Good to know then. Hopefully never :) >> I remember from the OpenBSD xbridge driver, it states that it >> currently uses the BAR mappings setup by the ARCS firmware, but the >> plan is to eventually move away from those mappings and calculate >> its own. I didn't quite get that, but after reading that site, now >> I do. ARCS is writing ~0 to the BARs to query the device to figure >> out how much memory each BAR wants, then programs in some ranges for >> us. >> >> By tallying up the total requested size of all BAR mappings on a >> given device, we can then make a determination on whether we can use >> a small, medium, or large window and then program the BAR with the >> correct address. Right? > > This is the problem I alluded to above: Linux does not currently > support anything like this. We assume the Linux host bridge driver > knows the window sizes at the beginning of time (it may learn them > from firmware or it may have hard-coded knowledge of the address map). > > Linux does enumerate the devices and tally up all the BAR sizes (see > pci_bus_assign_resources()), but it does not have a way to change the > host bridge windows based on how much BAR space we need. > > The common thing is to use whatever host bridge windows and device BAR > values were set up by firmware. If those are all sensible, Linux > won't change anything. Well, as I stated earlier above, it looks like the bridge_probe() function in the new driver will have to do some probing of its own. One of the patches I have in a different series does a minimal probe to read each device slot on BRIDGE to get the vendor ID and device ID to see if the slot is actually populated. This is because not all slots are "populated"; e.g., Octane wires one of the PCI interrupt pins on slot 6 on BaseIO to its power button, for example. So the logic used to read the vendor/dev IDs can probably be extended to poke the BARs via the ~0 trick OR read out the pre-defined mappings from ARCS firmware, then use that info to size out the BRIDGE windows into Crosstalk space and then set up our IORESOURCE_MEM and IORESOURCE_IO structs with the right information to pass into register_pci_controller(). I dunno, I'll probably try reading the ARCS mappings idea first, as that seems easier. I won't have a lot of time to play with things in March, so getting something to work, even if it is sub-optimal, is better than nothing working at all. >>> The host bridge windows (the "root bus resource" lines in dmesg) >>> are only for PIO, i.e., a driver running on the CPU reading or >>> writing registers on the PCI device (it could be actual registers, >>> or a frame buffer, etc., but it resides on the device and its PCI >>> bus address is determined by a BAR). The driver uses ioremap(), >>> pci_iomap(), pcim_iomap_regions(), etc. to map these into the >>> kernel virtual space. >> >> So for the MIPS case, knowing that ioremap() and friends is not >> guaranteed to return a workable virtual address, I need to be >> careful of what addresses I program into each BAR. E.g., given that >> on IP30, if a physical address starts with 0x9000000f800xxxxx, it is >> using medium windows to talk to a device on the BaseIO BRIDGE (Xtalk >> widget 0xf). As such, knowing MIPS' ioremap() returns, for the >> R10000 CPU case, the requested address OR'ed with IO_BASE >> (0x9000000000000000), I want to tell the PCI core to use >> 0x0000000f800xxxxx so that ioremap() will return back >> 0x9000000f800xxxxx? > > If ioremap() doesn't return a virtual address, or at least something > that can be used like a virtual address, I think that's fundamentally > broken. All drivers assume they should ioremap() a BAR resource, > i.e., the CPU physical address that maps to a PCI BAR value, and use > the result as a virtual address. This is an issue that the core Linux/MIPS people will have to work out. Ralf probably knows the precise history on why ioremap*() and related functions aren't guaranteed to be usable as virtual addresses. I suspect it's tied to the other issues with IP27, as that was one of the first MIPS platforms that Linux ran on in the early 2000's. >> And since I can apparently specify multiple window ranges for memory >> space and I/O space, I probably want to, as stated earlier, specify >> all three of IP30's known window ranges for each BRIDGE so that the >> Linux PCI core can walk each resource struct and find a matching >> window? > > If all three windows are disjoint, you can specify them all. If a > range in the small window is aliased and can also be reached via a > medium or large window, you should not specify both. Each window, at least on Octane, lives at distinct addresses in Crosstalk space. A specific address within each window can point to the same device on a subordinate PCI bus, but I think that will be transparent to the Linux PCI core. My thinking is, if I setup IORESOURCE_MEM structs for each window for each BRIDGE widget, then the PCI code will check each struct to see which one contains the address range that the PCI device's BAR wants. E.g., Using widget 0xd as an example for readability, if I know my three windows in Crosstalk space are: Small: 0x0000_1d00_0000 - 0x0000_1dff_ffff Medium: 0x000e_8000_0000 - 0x000e_ffff_ffff Large: 0x00d0_0000_0000 - 0x00df_ffff_ffff And for that specific BRIDGE, if I pass those three ranges as IORESOURCE_MEM structs, then shouldn't the PCI core, if it is told that a specific devices BAR wants range 1d200000 to 1d203fff, select the small window mapping? E,g., is it going to try to find the smallest possible window to fit first? Or will it instead try to match using the large window? I guess I'll find out when I can get some time to actually test the idea out... [snip] >>> I think the biggest problem is that you need to set up the BRIDGE >>> *before* calling pcibios_scan_bus(). That way the windows >>> ("hose->mem_resource", "hose->io_resource", etc.) will be correct >>> when the PCI core enumerates the devices. Note that you can have >>> as many windows in the "&resources" list as you need -- the >>> current code there only has one memory and one I/O window, but you >>> can add more. >> >> Agreed, however, I think this is actually being done to some extent >> already. We're configuring BRIDGE-specific properties and writing >> some values to BRIDGE registers and the per-slot registers in >> bridge_probe() in the generic pci-bridge.c driver, then, at the very >> end, calling register_pci_controller(), which I believe is what >> kicks off the PCI bus scan. >> >> The IP30 BRIDGE glue code adds a new function hook called >> "pre_enable" where it does, in ip30-bridge.c, some additional PCI >> device tweaking, but this code will run after the PCI bus scan has >> happened. My guess is that I want to reverse this and have the IP30 >> glue code twiddle the PCI devices on a given BRIDGE before the PCI >> bus scan, right? > > Theoretically you should not need to do any PCI device tweaking: PCI > devices are basically arch-independent and shouldn't need > arch-specific tweaks. All the arch stuff should be encapsulated in > the host bridge driver, i.e., the code that sets up the window list > and calls pci_scan_root_bus(). In the MIPS case, this code is kind of > spread out and doesn't really look like a single "driver". In an ideal world, this would be correct. However, owning SGI equipment must have an effect on the curvature of local spacetime, and things seem to operate a bit differently. The IOC3 device is a perfect example of this phenomenon. > The ideal thing would be if you can set up the bridge to always use > large windows. Then the PCI core will enumerate the devices and set > up their resources automatically. I'll keep this in mind when I can start testing. I'll initially try feeding three IORESOURCE_MEM structs to the PCI core to describe the three windows in order from smallest to largest and see what it does. Then try reversing the order and see if it still finds the right window or not. And whether that actually leads to correct probing of the PCI devices or not. > I assume the small/medium windows exist because there's not enough > address space to always use the large windows. I don't have any good > suggestions for that -- we don't have support for adjusting the window > sizes based on what devices we find. IMHO, "Use a bigger hammer" seems most appropriate when it comes to this hardware. >> That code uses pci_read_config_dword() and >> pci_write_config_dword() -- are these functions safe to use if we >> haven't already done a PCI bus scan? > > You can't use pci_read_config_dword() before we scan the bus because > it requires a "struct pci_dev *", and those are created during the bus > scan. Okay, that's good to know then. There's other accessors that are MIPS-specific that I think I can use. I don't think I specifically need a 'struct pci_dev *' at that point anyways. -- Joshua Kinard Gentoo/MIPS kumba@gentoo.org 6144R/F5C6C943 2015-04-27 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-28 0:25 ` Joshua Kinard @ 2017-03-01 15:39 ` Bjorn Helgaas 0 siblings, 0 replies; 28+ messages in thread From: Bjorn Helgaas @ 2017-03-01 15:39 UTC (permalink / raw) To: Joshua Kinard Cc: Bjorn Helgaas, Ralf Baechle, James Hogan, Lorenzo Pieralisi, Thomas Bogendoerfer, Linux/MIPS, linux-pci On Mon, Feb 27, 2017 at 07:25:58PM -0500, Joshua Kinard wrote: > On 02/27/2017 11:36, Bjorn Helgaas wrote: > > On Sat, Feb 25, 2017 at 04:34:12AM -0500, Joshua Kinard wrote: > >> On 02/24/2017 13:38, Bjorn Helgaas wrote: > >>> On Fri, Feb 24, 2017 at 03:50:26AM -0500, Joshua Kinard wrote: > So the logic used to read the vendor/dev IDs can probably be > extended to poke the BARs via the ~0 trick OR read out the > pre-defined mappings from ARCS firmware, then use that info to size > out the BRIDGE windows into Crosstalk space and then set up our > IORESOURCE_MEM and IORESOURCE_IO structs with the right information > to pass into register_pci_controller(). As you probably know, you can determine the size of a BAR regardless of whether it has been assigned. So it doesn't matter whether ARCS has already assigned anything. See __pci_read_base(). > I dunno, I'll probably try reading the ARCS mappings idea first, as > that seems easier. I won't have a lot of time to play with things > in March, so getting something to work, even if it is sub-optimal, > is better than nothing working at all. The simplest possible thing is to enable some pre-defined set of windows and accept that this may lead to configuration restrictions, e.g., a device with large BARs may work only in certain slots. > Each window, at least on Octane, lives at distinct addresses in > Crosstalk space. A specific address within each window can point to > the same device on a subordinate PCI bus, but I think that will be > transparent to the Linux PCI core. My thinking is, if I setup > IORESOURCE_MEM structs for each window for each BRIDGE widget, then > the PCI code will check each struct to see which one contains the > address range that the PCI device's BAR wants. > > E.g., Using widget 0xd as an example for readability, if I know my > three windows in Crosstalk space are: > Small: 0x0000_1d00_0000 - 0x0000_1dff_ffff > Medium: 0x000e_8000_0000 - 0x000e_ffff_ffff > Large: 0x00d0_0000_0000 - 0x00df_ffff_ffff > > And for that specific BRIDGE, if I pass those three ranges as > IORESOURCE_MEM structs, then shouldn't the PCI core, if it is told > that a specific devices BAR wants range 1d200000 to 1d203fff, select > the small window mapping? E,g., is it going to try to find the > smallest possible window to fit first? Or will it instead try to > match using the large window? No, when matching a BAR value to a host bridge window, the PCI core will definitely not look for the smallest (or largest) window that contains the BAR. The core reads the BAR value (a bus address), then searches the host bridge windows for one that maps to a region that contains the bus address. This happens in pcibios_bus_to_resource(), which is called by __pci_read_base(). The search order is undefined because the core assumes the windows do not overlap in PCI bus address space. It can't deal with two windows that map to the same PCI space because when we allocate space when assigning a BAR, we search for available CPU address space, not for available PCI bus address space. If window A and window B both map to PCI bus address X, we may assign space from window A to PCI BAR 1 and space from window B to PCI BAR 2. Then we have two BARs at the same PCI bus address, which will cause conflicts. I don't think we currently check in the PCI core for host bridge windows that map to overlapping PCI bus space. Maybe we should, and just reject anything that overlaps. Bjorn ^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case 2017-02-07 18:29 ` Bjorn Helgaas 2017-02-08 9:39 ` Joshua Kinard @ 2017-02-09 18:36 ` Joshua Kinard 1 sibling, 0 replies; 28+ messages in thread From: Joshua Kinard @ 2017-02-09 18:36 UTC (permalink / raw) To: Bjorn Helgaas Cc: Ralf Baechle, Linux/MIPS, Lorenzo Pieralisi, Thomas Bogendoerfer On 02/07/2017 13:29, Bjorn Helgaas wrote: [snip] >> >> However, IP27 is completely different in this regard. Instead of using >> ioremapped addresses for I/O, IP27 has a dedicated address range, >> 0x92xxxxxxxxxxxxxx, that is used for all I/O access. Since this is >> uncached physical address space, the generic MIPS PCI code will not >> probe it correctly and thus, the original behavior of PCI_PROBE_ONLY >> needs to be restored only for the IP27 platform to bypass this logic >> and have working PCI, at least for the IO6/IO6G board that houses the >> base devices, until a better solution is found. > > It sounds like there's something different about how ioremap() works > on these platforms and PCI probing is tripping over that. I'd really > like to understand more about this difference to see if we can > converge that instead of adding back the PCI_PROBE_ONLY usage. > > Drivers shouldn't know whether they're running on IP27 or IP30, and > they should be using ioremap() in both cases. Does ioremap() work > differently on IP27 and IP30? Does this have something to do with > plat_ioremap() or fixup_bigphys_addr()? Okay, I think I have a rough idea on the differences with the I/O addresses on IP27 and IP30. It all boils down to the return value of a macro called NODE_SWIN_BASE, which operates differently on each platform. I don't fuilly understand //why// SGI did it a certain way, but HUB in IP27 gives you two sets of "windows" into Crosstalk space, "Big" (BWIN) and "Little" (LWIN). IP30's HEART chip has three sets of these windows (as I recall now from and appendix in the BRIDGE docs), Big (BWIN), medium (MWIN) and small (SWIN). So on IP27, HUB has seven BWIN spaces at 512MB each, and one LWIN space at 256MB (with a second LWIN that's aliased to the first). The LWIN is broken into sixteen 16MB spaces, which, I think, is what we're calling "Small windows" or SWIN's (Ralf, correct me if wrong). IP30's HEART allocates 64G of space per Xtalk widget for BWIN, 2G/widget for MWIN, and 16MB/widget for SWIN. As such, since HUB doesn't have an MWIN, you only find definitions of NODE_BWIN_BASE and NODE_SWIN_BASE in either IP27 or IP30's sources. Specifically, our focus is really only on NODE_SWIN_BASE, because the BWIN spaces need to be mapped into translation tables somewhere, and Linux lacks that capability. There is also a corresponding RAW_NODE_SWIN_BASE for both platforms that gives back the full address in uncached physical address space. IP30's definition of NODE_SWIN_BASE is: #define NODE_SWIN_BASE(nasid, widget) \ (0x0000000010000000 | (((unsigned long)(widget)) << 24)) And IP30's RAW_NODE_SWIN_BASE: #define RAW_NODE_SWIN_BASE(nasid, widget) \ (0x9000000010000000 | (((unsigned long)(widget)) << 24)) Contrast that to IP27's NODE_SWIN_BASE (for the __ASSEMBLY__ case) in arch/mips/include/asm/sn/sn0/addrs.h: #define NODE_SWIN_BASE(nasid, widget) \ (NODE_IO_BASE(nasid) + (UINT64_CAST(widget) << SWIN_SIZE_BITS)) (IP27's RAW_NODE_SWIN_BASE is the same as above) SWIN_SIZE_BITS is '24', so it matches IP30's. The difference is the value of NODE_IO_BASE that takes a nasid (NUMA address space identifier or node id) and returns the correct address in IP27's memory space to get to a specific node's I/O Base, which is 0x9200000000000000 plus the node ID. Merged with the widget ID shifted by 24 bits, and that's why we're getting the 0x92xx addresses. Basically, IP30 is returning the widget offset //without// the I/O base factored in, and that's why RAW_NODE_SWIN_BASE is provided to get the widget address in uncached physical address space. Whereas IP27's NODE_SWIN_BASE is an alias to its RAW_NODE_SWIN_BASE definition and turning 0x92xxx addresses. --- Then we look in arch/mips/include/asm/io.h at the function __ioremap_mode and see this construct near the top: if (cpu_has_64bit_addresses) { u64 base = UNCAC_BASE; /* * R10000 supports a 2 bit uncached attribute therefore * UNCAC_BASE may not equal IO_BASE. */ if (flags == _CACHE_UNCACHED) base = (u64) IO_BASE; return (void __iomem *) (unsigned long) (base + offset); For IP30, when we set up the BRIDGE parameters, we use NODE_SWIN_BASE to set the BRIDGE's MEM and IO start/end ranges and thus get back an address where the upper bits are all zero's. Which the PCI layer then fiddles with to yield an address like 0x000000f100000000. When that gets run through ioremap() by a driver's probe, we should be getting something like 0x900000f100000000, because IP30 uses the generic definition of IO_BASE defined in MIPS asm-generic/spaces.h. And the devices probe correctly. For IP27, I program BRIDGE the same way, but because IP27's NODE_SWIN_BASE is returning the 0x92xxx addresses in uncached space, my guess is ioremap(), as called by the drivers, is doubling that, yielding addresses like 0x24xxx, which unprotected, should generate an unhandled kernel unaligned access or such. Properly protected, attempts to access such an address probably just fail outright. --- So my thinking, which I'll test out over the weekend, is whether we can change IP27's NODE_SWIN_BASE to behave more like IP30's and just return the widget id shifted by 24 bits, then let ioremap() calls actually do the conversion to the 0x92xxx IO_BASE address space. There's only a handful of places in IP27's code and headers where NODE_SWIN_BASE is actually used, so I'll have to see which ones need to become RAW_NODE_SWIN_BASE (e.g., for direct HUB or XBOW access), and which can stay as NODE_SWIN_BASE. That would then allow the correct operation of ioremap() to return the 0x92xxx address space for drivers and I guess PCI to probe correctly. We'll find out! --J ^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2017-03-01 15:39 UTC | newest]
Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-02-07 6:13 [PATCH 00/12] MIPS: BRIDGE Updates Joshua Kinard
2017-02-07 6:13 ` [PATCH 01/12] MIPS: BRIDGE: Rename pci-ip27.c to pci-bridge.c Joshua Kinard
2017-02-07 6:13 ` [PATCH 02/12] MIPS: IP27: Add pcibr.h header for IP27 Joshua Kinard
2017-02-07 6:13 ` [PATCH 03/12] MIPS: PCI: Minor clean-ups to pci-legacy.c Joshua Kinard
2017-02-07 6:13 ` [PATCH 04/12] MIPS: PCI: Add BRIDGE 'pre_enable' hook Joshua Kinard
2017-02-07 6:13 ` [PATCH 05/12] MIPS: BRIDGE: Clean-up bridge.h header file Joshua Kinard
2017-02-07 6:13 ` [PATCH 06/12] MIPS: BRIDGE: Overhaul " Joshua Kinard
2017-02-07 6:13 ` [PATCH 07/12] MIPS: BRIDGE: Add XBRIDGE revs and SWAP bit Joshua Kinard
2017-02-07 6:13 ` [PATCH 08/12] MIPS: BRIDGE: Update ops-bridge.c Joshua Kinard
2017-02-07 6:13 ` [PATCH 09/12] MIPS: BRIDGE: Use !pci_is_root_bus(bus) to check for bus->number > 0 Joshua Kinard
2017-02-07 6:13 ` [PATCH 10/12] MIPS: BRIDGE: Overhaul pci-bridge.c driver Joshua Kinard
2017-02-07 6:13 ` [PATCH 11/12] MIPS: IP27: Update IRQ code to work with the new BRIDGE code Joshua Kinard
2017-02-07 6:13 ` [PATCH 12/12] MIPS: PCI: Fix IP27 for the PCI_PROBE_ONLY case Joshua Kinard
2017-02-07 18:29 ` Bjorn Helgaas
2017-02-08 9:39 ` Joshua Kinard
[not found] ` <CAErSpo4LsrPCtdZwp6CyT0jKhXLt3j=fGSiFjpRRTPUjFoKHtQ@mail.gmail.com>
2017-02-12 4:09 ` Joshua Kinard
2017-02-13 22:45 ` Bjorn Helgaas
2017-02-13 22:45 ` Bjorn Helgaas
2017-02-13 22:45 ` Bjorn Helgaas
2017-02-14 7:39 ` Joshua Kinard
2017-02-14 14:56 ` Bjorn Helgaas
2017-02-24 8:50 ` Joshua Kinard
2017-02-24 18:38 ` Bjorn Helgaas
2017-02-25 9:34 ` Joshua Kinard
2017-02-27 16:36 ` Bjorn Helgaas
2017-02-28 0:25 ` Joshua Kinard
2017-03-01 15:39 ` Bjorn Helgaas
2017-02-09 18:36 ` Joshua Kinard
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.