From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.kernel.org ([198.145.29.99]:60990 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751234AbeEBUd4 (ORCPT ); Wed, 2 May 2018 16:33:56 -0400 Date: Wed, 2 May 2018 15:33:55 -0500 From: Bjorn Helgaas To: Ivan Kokshaysky Cc: Matt Turner , Yinghai Lu , linux-pci@vger.kernel.org, linux-alpha , Richard Henderson , Jay Estabrook , Bjorn Helgaas Subject: Re: Some Alphas broken by f75b99d5a77d (PCI: Enforce bus address limits in resource allocation) Message-ID: <20180502203355.GA147318@bhelgaas-glaptop.roam.corp.google.com> References: <20180416215044.GE28657@bhelgaas-glaptop.roam.corp.google.com> <20180417194344.GK28657@bhelgaas-glaptop.roam.corp.google.com> <20180418204808.GA2352@mail.rc.ru> <20180423173423.GA9138@mail.rc.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <20180423173423.GA9138@mail.rc.ru> Sender: linux-pci-owner@vger.kernel.org List-ID: On Mon, Apr 23, 2018 at 06:34:23PM +0100, Ivan Kokshaysky wrote: > On Sun, Apr 22, 2018 at 01:07:38PM -0700, Matt Turner wrote: > > On Wed, Apr 18, 2018 at 1:48 PM, Ivan Kokshaysky > > wrote: > > > On Tue, Apr 17, 2018 at 02:43:44PM -0500, Bjorn Helgaas wrote: > > >> On Mon, Apr 16, 2018 at 09:43:42PM -0700, Matt Turner wrote: > > >> > On Mon, Apr 16, 2018 at 2:50 PM, Bjorn Helgaas wrote: > > >> > > Hi Matt, > > >> > > > > >> > > First of all, sorry about breaking Nautilus, and thanks very much for > > >> > > tracking it down to this commit. > > >> > > > >> > It's a particularly weird case, as far as I've been able to discern :) > > >> > > > >> > > On Mon, Apr 16, 2018 at 07:33:57AM -0700, Matt Turner wrote: > > >> > >> Commit f75b99d5a77d63f20e07bd276d5a427808ac8ef6 (PCI: Enforce bus > > >> > >> address limits in resource allocation) broke Alpha systems using > > >> > >> CONFIG_ALPHA_NAUTILUS. Alpha is 64-bit, but Nautilus systems use a > > >> > >> 32-bit AMD 751/761 chipset. arch/alpha/kernel/sys_nautilus.c maps PCI > > >> > >> into the upper addresses just below 4GB. > > >> > >> > > >> > >> I can get a working kernel by ifdef'ing out the code in > > >> > >> drivers/pci/bus.c:pci_bus_alloc_resource. We can't tie > > >> > >> PCI_BUS_ADDR_T_64BIT to ALPHA_NAUTILUS without breaking generic > > >> > >> kernels. > > >> > >> > > >> > >> How can we get Nautilus working again? > > >> > > > > >> > > Can you collect a complete dmesg log, ideally both before and after > > >> > > f75b99d5a77d? I assume the problem is that after f75b99d5a77d? we > > >> > > erroneously assign space for something above 4GB. But if we know the > > >> > > correct host bridge apertures, we shouldn't assign space outside them, > > >> > > regardless of the PCI bus address size. > > >> > > > >> > I made a mistake in my initial report. Commit f75b99d5a77d is actually > > >> > the last *working* commit. My apologies. The next commit is > > >> > d56dbf5bab8c (PCI: Allocate 64-bit BARs above 4G when possible) and it > > >> > breaks Nautilus I've confirmed. > > >> > > > >> > Please find attached dmesgs from those two commits, from the commit > > >> > immediately before them, and another from 4.17-rc1 with my hack of #if > > >> > 0'ing out the pci_bus_alloc_from_region(..., &pci_high) code. > > >> > > > >> > Thanks for having a look! > > >> > > >> We're telling the PCI core that the host bridge MMIO aperture is the > > >> entire 64-bit address space, so when we assign BARs, some of them end > > >> up above 4GB: > > >> > > >> pci_bus 0000:00: root bus resource [mem 0x00000000-0xffffffffffffffff] > > >> pci 0000:00:09.0: BAR 0: assigned [mem 0x100000000-0x10000ffff 64bit] > > >> > > >> But it sounds like the MMIO aperture really ends at 0xffffffff, so > > >> that's not going to work. > > > > > > Correct... This would do as a quick fix, I think: > > > > > > diff --git a/arch/alpha/kernel/sys_nautilus.c b/arch/alpha/kernel/sys_nautilus.c > > > index ff4f54b..477ba65 100644 > > > --- a/arch/alpha/kernel/sys_nautilus.c > > > +++ b/arch/alpha/kernel/sys_nautilus.c > > > @@ -193,6 +193,8 @@ static struct resource irongate_io = { > > > }; > > > static struct resource irongate_mem = { > > > .name = "Irongate PCI MEM", > > > + .start = 0, > > > + .end = 0xffffffff, > > > .flags = IORESOURCE_MEM, > > > }; > > > static struct resource busn_resource = { > > > @@ -218,7 +220,7 @@ nautilus_init_pci(void) > > > return; > > > > > > pci_add_resource(&bridge->windows, &ioport_resource); > > > - pci_add_resource(&bridge->windows, &iomem_resource); > > > + pci_add_resource(&bridge->windows, &irongate_mem); > > > pci_add_resource(&bridge->windows, &busn_resource); > > > bridge->dev.parent = NULL; > > > bridge->sysdata = hose; > > > > Thanks. But with that I get > > > > PCI host bridge to bus 0000:00 > > pci_bus 0000:00: root bus resource [io 0x0000-0xffff] > > pci_bus 0000:00: root bus resource [mem 0x00000000-0xffffffff] > > pci_bus 0000:00: root bus resource [bus 00-ff] > > pci 0000:00:10.0: [Firmware Bug]: reg 0x10: invalid BAR (can't size) > > pci 0000:00:10.0: [Firmware Bug]: reg 0x14: invalid BAR (can't size) > > pci 0000:00:10.0: [Firmware Bug]: reg 0x18: invalid BAR (can't size) > > pci 0000:00:10.0: [Firmware Bug]: reg 0x1c: invalid BAR (can't size) > > pci 0000:00:10.0: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] > > pci 0000:00:10.0: legacy IDE quirk: reg 0x14: [io 0x03f6] > > pci 0000:00:10.0: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] > > pci 0000:00:10.0: legacy IDE quirk: reg 0x1c: [io 0x0376] > > pci 0000:00:11.0: quirk: [io 0x4000-0x403f] claimed by ali7101 ACPI > > pci 0000:00:11.0: quirk: [io 0x5000-0x501f] claimed by ali7101 SMB > > pci 0000:00:01.0: BAR 9: assigned [mem 0xc0000000-0xc2ffffff pref] > > pci 0000:00:01.0: BAR 8: assigned [mem 0xc3000000-0xc3bfffff] > > pci 0000:00:0b.0: BAR 6: assigned [mem 0xc3c00000-0xc3c3ffff pref] > > pci 0000:00:08.0: BAR 6: assigned [mem 0xc3c40000-0xc3c5ffff pref] > > pci 0000:00:09.0: BAR 0: assigned [mem 0xc3c60000-0xc3c6ffff 64bit] > > pci 0000:00:03.0: BAR 0: assigned [mem 0xc3c70000-0xc3c70fff] > > pci 0000:00:06.0: BAR 1: assigned [mem 0xc3c71000-0xc3c71fff] > > pci 0000:00:08.0: BAR 1: assigned [mem 0xc3c72000-0xc3c72fff 64bit] > > pci 0000:00:14.0: BAR 0: assigned [mem 0xc3c73000-0xc3c73fff] > > pci 0000:00:0b.0: BAR 1: assigned [mem 0xc3c74000-0xc3c743ff] > > pci 0000:00:03.0: BAR 1: assigned [io 0x8000-0x80ff] > > pci 0000:00:06.0: BAR 0: assigned [io 0x8400-0x84ff] > > pci 0000:00:08.0: BAR 0: assigned [io 0x8800-0x88ff] > > pci 0000:00:0b.0: BAR 0: assigned [io 0x8c00-0x8c7f] > > pci 0000:00:10.0: BAR 4: assigned [io 0x8c80-0x8c8f] > > pci 0000:01:05.0: BAR 0: assigned [mem 0xc0000000-0xc1ffffff pref] > > pci 0000:01:05.0: BAR 2: assigned [mem 0xc3000000-0xc37fffff] > > pci 0000:01:05.0: BAR 6: assigned [mem 0xc2000000-0xc200ffff pref] > > pci 0000:01:05.0: BAR 1: assigned [mem 0xc3800000-0xc3803fff] > > pci 0000:00:01.0: PCI bridge to [bus 01] > > pci 0000:00:01.0: bridge window [mem 0xc3000000-0xc3bfffff] > > pci 0000:00:01.0: bridge window [mem 0xc0000000-0xc2ffffff pref] > > > > Looks like the bridge window is overlapping with some previously assigned BARs? > > No, there are no overlaps. This is AGP bridge window with AGP card BARs > inside it. > > > The result is the SCSI card failing to work: > > > > scsi host0: Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0 > > > > aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs > > > > scsi0: ahc_intr - referenced scb not valid during SELTO scb(0, 255) > > >>>>>>>>>>>>>>>>>> Dump Card State Begins <<<<<<<<<<<<<<<<< > > scsi0: Dumping Card State while idle, at SEQADDR 0x18 > > [snip endless spew] > > It looks like Irongate PCI memory window setting is completely wrong, > perhaps it starts at zero, so that PCI DMA doesn't work at all. > > Please try the patch below. It tries to fix things properly, but if > PCI bus sizing fails for some reason, it falls back to a "safe" PCI > memory window placement at 3GB. Ping, Matt, any feedback on this patch? > diff --git a/arch/alpha/kernel/sys_nautilus.c b/arch/alpha/kernel/sys_nautilus.c > index ff4f54b..b1fb9fd 100644 > --- a/arch/alpha/kernel/sys_nautilus.c > +++ b/arch/alpha/kernel/sys_nautilus.c > @@ -187,10 +187,6 @@ nautilus_machine_check(unsigned long vector, unsigned long la_ptr) > > extern void pcibios_claim_one_bus(struct pci_bus *); > > -static struct resource irongate_io = { > - .name = "Irongate PCI IO", > - .flags = IORESOURCE_IO, > -}; > static struct resource irongate_mem = { > .name = "Irongate PCI MEM", > .flags = IORESOURCE_MEM, > @@ -218,7 +214,7 @@ nautilus_init_pci(void) > return; > > pci_add_resource(&bridge->windows, &ioport_resource); > - pci_add_resource(&bridge->windows, &iomem_resource); > + pci_add_resource(&bridge->windows, &irongate_mem); > pci_add_resource(&bridge->windows, &busn_resource); > bridge->dev.parent = NULL; > bridge->sysdata = hose; > @@ -238,20 +234,25 @@ nautilus_init_pci(void) > pcibios_claim_one_bus(bus); > > irongate = pci_get_domain_bus_and_slot(pci_domain_nr(bus), 0, 0); > + /* Pretend that it's not a root bus to allow sizing. */ > + bus->parent = bus; > bus->self = irongate; > - bus->resource[0] = &irongate_io; > - bus->resource[1] = &irongate_mem; > > pci_bus_size_bridges(bus); > - > - /* IO port range. */ > - bus->resource[0]->start = 0; > - bus->resource[0]->end = 0xffff; > + bus->parent = NULL; > > /* Set up PCI memory range - limit is hardwired to 0xffffffff, > base must be at aligned to 16Mb. */ > - bus_align = bus->resource[1]->start; > - bus_size = bus->resource[1]->end + 1 - bus_align; > + if (bus->resource[1]->end) { > + bus_align = bus->resource[1]->start; > + bus_size = bus->resource[1]->end + 1 - bus_align; > + printk(KERN_INFO "PCI MEM size 0x%08lx, align 0x%08lx\n", > + bus_size, bus_align); > + } else { > + bus_size = 0x100000000UL - memtop; > + bus_align = bus_size; > + printk(KERN_ERR "PCI MEM sizing failed, using 3GB limit\n"); > + } > if (bus_align < 0x1000000UL) > bus_align = 0x1000000UL; >