PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization

PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization

[Stéphane Graber]

Hello,

I've recently been given access to a set of 4 APM X-Gene2 Merlin
boards (old-ish development platform).
Running them on Ubuntu 20.04's stock 5.4 kernel worked fine but trying
to run anything else would fail to boot due to a NVME initialization
timeout preventing the main drive from showing up at all.

Tracking this issue, I first moved to clean mainline kernels and then
isolated the issue to be somewhere between 5.4.0 and 5.5.0-rc1, which
sadly meant the merge window (so much for a quick bisect...). I've
then bisected between those two points and came up with:

  6dce5aa59e0bf2430733d7a8b11c205ec10f408e (refs/bisect/bad) PCI:
xgene: Use inbound resources for setup

I finally switched to the latest 5.15.2 tree, reverted that one
commit, built a new kernel and confirmed that those boards now work
flawlessly.

Unfortunately that's about the extent of my abilities with kernel
debugging and I won't pretend to understand what that commit does or
how it may be breaking PCIe initialization on those systems.

I'm not technically blocked on this, I can manually build my own
kernels by reverting that one commit every time, but that's obviously
not ideal and I'd much rather have this fixed upstream :)

== Good boot on 5.15.2 (commit reverted) ==
Full log at: https://gist.github.com/stgraber/e489b7e55dd7ffaac9f77dd8634ca2ff

root@entak:~# dmesg | grep -Ei "nvme|pci"
[    0.094146] PCI: CLS 0 bytes, default 64
[    0.130573] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    0.131324] xgene-pcie 1f2b0000.pcie: host bridge /soc/pcie@1f2b0000 ranges:
[    0.131344] xgene-pcie 1f2b0000.pcie:   No bus range found for
/soc/pcie@1f2b0000, using [bus 00-ff]
[    0.131365] xgene-pcie 1f2b0000.pcie:       IO
0xc010000000..0xc01000ffff -> 0x0000000000
[    0.131388] xgene-pcie 1f2b0000.pcie:      MEM
0xc120000000..0xc13fffffff -> 0x0020000000
[    0.131401] xgene-pcie 1f2b0000.pcie:      MEM
0xe000000000..0xffffffffff -> 0xe000000000
[    0.131416] xgene-pcie 1f2b0000.pcie:   IB MEM
0x8000000000..0x807fffffff -> 0x8000000000
[    0.131427] xgene-pcie 1f2b0000.pcie:   IB MEM
0x0000000000..0x7fffffffff -> 0x0000000000
[    0.131510] xgene-pcie 1f2b0000.pcie: (rc) x4 gen-3 link up
[    0.131600] xgene-pcie 1f2b0000.pcie: PCI host bridge to bus 0000:00
[    0.131612] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.131619] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.131629] pci_bus 0000:00: root bus resource [mem
0xc120000000-0xc13fffffff] (bus address [0x20000000-0x3fffffff])
[    0.131637] pci_bus 0000:00: root bus resource [mem
0xe000000000-0xffffffffff pref]
[    0.131671] pci 0000:00:00.0: [10e8:e004] type 01 class 0x060400
[    0.131682] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131693] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131705] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131715] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131725] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131733] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131742] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131753] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131781] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x3e may corrupt adjacent RW1C bits
[    0.131832] pci 0000:00:00.0: supports D1 D2
[    0.132373] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x3e may corrupt adjacent RW1C bits
[    0.132482] pci 0000:01:00.0: [144d:a80a] type 00 class 0x010802
[    0.132518] pci 0000:01:00.0: reg 0x10: [mem 0x40000000-0x40003fff 64bit]
[    0.132778] pci 0000:01:00.0: 31.504 Gb/s available PCIe bandwidth,
limited by 8.0 GT/s PCIe x4 link at 0000:00:00.0 (capable of 63.012
Gb/s with 16.0 GT/s PCIe x4 link)
[    0.143064] pci 0000:00:00.0: BAR 14: assigned [mem
0xc120000000-0xc1200fffff]
[    0.143086] pci 0000:01:00.0: BAR 0: assigned [mem
0xc120000000-0xc120003fff 64bit]
[    0.143105] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.143114] pci 0000:00:00.0:   bridge window [mem 0xc120000000-0xc1200fffff]
[    0.143315] pcieport 0000:00:00.0: PME: Signaling with IRQ 59
[    0.143518] pcieport 0000:00:00.0: AER: enabled with IRQ 59
[    1.596986] ehci-pci: EHCI PCI platform driver
[    1.611674] ohci-pci: OHCI PCI platform driver
[    3.347499] nvme nvme0: pci function 0000:01:00.0
[    3.347531] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[    3.350353] nvme nvme0: Shutdown timeout set to 10 seconds
[    3.535444] nvme nvme0: 8/0/0 default/read/poll queues
[    3.551454]  nvme0n1: p1 p2 p3 p4
[    6.963428] EXT4-fs (nvme0n1p2): mounted filesystem with ordered
data mode. Opts: (null). Quota mode: none.
[    8.415778] EXT4-fs (nvme0n1p2): re-mounted. Opts: (null). Quota mode: none.

== Bad boot on 5.15.2 (clean build, nothing reverted) ==
Full log at: https://gist.github.com/stgraber/605e8e852d8de35c6bbe64fab0f83815

root@entak:~# cat /boot/efi/dmesg | grep -Ei "nvme|pci"
[    0.094130] PCI: CLS 0 bytes, default 64
[    0.130822] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    0.131556] xgene-pcie 1f2b0000.pcie: host bridge /soc/pcie@1f2b0000 ranges:
[    0.131576] xgene-pcie 1f2b0000.pcie:   No bus range found for
/soc/pcie@1f2b0000, using [bus 00-ff]
[    0.131596] xgene-pcie 1f2b0000.pcie:       IO
0xc010000000..0xc01000ffff -> 0x0000000000
[    0.131618] xgene-pcie 1f2b0000.pcie:      MEM
0xc120000000..0xc13fffffff -> 0x0020000000
[    0.131630] xgene-pcie 1f2b0000.pcie:      MEM
0xe000000000..0xffffffffff -> 0xe000000000
[    0.131646] xgene-pcie 1f2b0000.pcie:   IB MEM
0x8000000000..0x807fffffff -> 0x8000000000
[    0.131659] xgene-pcie 1f2b0000.pcie:   IB MEM
0x0000000000..0x7fffffffff -> 0x0000000000
[    0.131729] xgene-pcie 1f2b0000.pcie: (rc) x4 gen-3 link up
[    0.131816] xgene-pcie 1f2b0000.pcie: PCI host bridge to bus 0000:00
[    0.131827] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.131834] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.131844] pci_bus 0000:00: root bus resource [mem
0xc120000000-0xc13fffffff] (bus address [0x20000000-0x3fffffff])
[    0.131852] pci_bus 0000:00: root bus resource [mem
0xe000000000-0xffffffffff pref]
[    0.131886] pci 0000:00:00.0: [10e8:e004] type 01 class 0x060400
[    0.131897] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131908] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131919] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131929] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131938] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131946] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131955] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131966] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x4 may corrupt adjacent RW1C bits
[    0.131994] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x3e may corrupt adjacent RW1C bits
[    0.132044] pci 0000:00:00.0: supports D1 D2
[    0.132590] pci_bus 0000:00: 2-byte config write to 0000:00:00.0
offset 0x3e may corrupt adjacent RW1C bits
[    0.132700] pci 0000:01:00.0: [144d:a80a] type 00 class 0x010802
[    0.132735] pci 0000:01:00.0: reg 0x10: [mem 0x40000000-0x40003fff 64bit]
[    0.132996] pci 0000:01:00.0: 31.504 Gb/s available PCIe bandwidth,
limited by 8.0 GT/s PCIe x4 link at 0000:00:00.0 (capable of 63.012
Gb/s with 16.0 GT/s PCIe x4 link)
[    0.143038] pci 0000:00:00.0: BAR 14: assigned [mem
0xc120000000-0xc1200fffff]
[    0.143059] pci 0000:01:00.0: BAR 0: assigned [mem
0xc120000000-0xc120003fff 64bit]
[    0.143079] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.143087] pci 0000:00:00.0:   bridge window [mem 0xc120000000-0xc1200fffff]
[    0.143286] pcieport 0000:00:00.0: PME: Signaling with IRQ 59
[    0.143474] pcieport 0000:00:00.0: AER: enabled with IRQ 59
[    1.598863] ehci-pci: EHCI PCI platform driver
[    1.613544] ohci-pci: OHCI PCI platform driver
[    3.280872] nvme nvme0: pci function 0000:01:00.0
[    3.280929] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[    7.393328] pcieport 0000:00:00.0: AER: Corrected error received:
0000:01:00.0
[    7.400550] nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected,
type=Physical Layer, (Receiver ID)
[    7.409733] nvme 0000:01:00.0:   device [144d:a80a] error
status/mask=00000001/0000e000
[    7.417703] nvme 0000:01:00.0:    [ 0] RxErr
[    7.423434] pci_generic_config_write32: 28 callbacks suppressed
[    7.423439] pci_bus 0000:01: 2-byte config write to 0000:01:00.0
offset 0x7a may corrupt adjacent RW1C bits
[   11.524622] pcieport 0000:00:00.0: AER: Corrected error received:
0000:01:00.0
[   11.531828] nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected,
type=Physical Layer, (Receiver ID)
[   11.541008] nvme 0000:01:00.0:   device [144d:a80a] error
status/mask=00000001/0000e000
[   11.548978] nvme 0000:01:00.0:    [ 0] RxErr
[   11.554707] pci_bus 0000:01: 2-byte config write to 0000:01:00.0
offset 0x7a may corrupt adjacent RW1C bits
[   64.046090] pcieport 0000:00:00.0: AER: Corrected error received:
0000:01:00.0
[   64.053295] nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected,
type=Physical Layer, (Receiver ID)
[   64.062475] nvme 0000:01:00.0:   device [144d:a80a] error
status/mask=00000001/0000e000
[   64.070446] nvme 0000:01:00.0:    [ 0] RxErr
[   64.076175] pci_bus 0000:01: 2-byte config write to 0000:01:00.0
offset 0x7a may corrupt adjacent RW1C bits
[   64.478625] nvme nvme0: I/O 16 QID 0 timeout, disable controller
[   64.590606] nvme nvme0: Device shutdown incomplete; abort shutdown
[   64.610619] pci_bus 0000:01: 2-byte config write to 0000:01:00.0
offset 0xb2 may corrupt adjacent RW1C bits
[   64.620324] pci_bus 0000:01: 2-byte config write to 0000:01:00.0
offset 0x4 may corrupt adjacent RW1C bits
[   64.629984] pci_bus 0000:01: 2-byte config write to 0000:01:00.0
offset 0x78 may corrupt adjacent RW1C bits
[   64.639694] pci_bus 0000:01: 2-byte config write to 0000:01:00.0
offset 0x4 may corrupt adjacent RW1C bits
[   64.649330] nvme nvme0: Identify Controller failed (-4)
[   64.654541] nvme nvme0: Removing after probe failure status: -5

Thanks!

Stéphane

Re: PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization

[Rob Herring]

On Thu, Nov 18, 2021 at 12:10 PM Stéphane Graber <stgraber@ubuntu.com> wrote:
>
> Hello,
>
> I've recently been given access to a set of 4 APM X-Gene2 Merlin
> boards (old-ish development platform).
> Running them on Ubuntu 20.04's stock 5.4 kernel worked fine but trying
> to run anything else would fail to boot due to a NVME initialization
> timeout preventing the main drive from showing up at all.
>
> Tracking this issue, I first moved to clean mainline kernels and then
> isolated the issue to be somewhere between 5.4.0 and 5.5.0-rc1, which
> sadly meant the merge window (so much for a quick bisect...). I've
> then bisected between those two points and came up with:
>
>   6dce5aa59e0bf2430733d7a8b11c205ec10f408e (refs/bisect/bad) PCI:
> xgene: Use inbound resources for setup
>
> I finally switched to the latest 5.15.2 tree, reverted that one
> commit, built a new kernel and confirmed that those boards now work
> flawlessly.
>
> Unfortunately that's about the extent of my abilities with kernel
> debugging and I won't pretend to understand what that commit does or
> how it may be breaking PCIe initialization on those systems.
>
> I'm not technically blocked on this, I can manually build my own
> kernels by reverting that one commit every time, but that's obviously
> not ideal and I'd much rather have this fixed upstream :)

Doesn't this platform have ACPI f/w you can use? From the log, it
looks like ACPI tables are passed to the kernel, but since a full DT
is passed it is used by default. Does 'acpi=on' not work?

Given no one noticed the breakage for 2 years, I'd really like to
remove these dts and binding files otherwise someone needs to convert
bindings to schema and fix warnings. Current stats look like this:
Processing apm:
warnings: 240
undocumented compat: 114

For example, I noticed that dma-ranges declares the entries are 32-bit
(0x42000000 is 32-bit prefetch), yet the PCI bus address and sizes are
>32-bit. AFAICT, that isn't part of the problem here.

> == Good boot on 5.15.2 (commit reverted) ==
> Full log at: https://gist.github.com/stgraber/e489b7e55dd7ffaac9f77dd8634ca2ff
>
> root@entak:~# dmesg | grep -Ei "nvme|pci"
> [    0.094146] PCI: CLS 0 bytes, default 64
> [    0.130573] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
> [    0.131324] xgene-pcie 1f2b0000.pcie: host bridge /soc/pcie@1f2b0000 ranges:
> [    0.131344] xgene-pcie 1f2b0000.pcie:   No bus range found for
> /soc/pcie@1f2b0000, using [bus 00-ff]
> [    0.131365] xgene-pcie 1f2b0000.pcie:       IO
> 0xc010000000..0xc01000ffff -> 0x0000000000
> [    0.131388] xgene-pcie 1f2b0000.pcie:      MEM
> 0xc120000000..0xc13fffffff -> 0x0020000000
> [    0.131401] xgene-pcie 1f2b0000.pcie:      MEM
> 0xe000000000..0xffffffffff -> 0xe000000000
> [    0.131416] xgene-pcie 1f2b0000.pcie:   IB MEM
> 0x8000000000..0x807fffffff -> 0x8000000000
> [    0.131427] xgene-pcie 1f2b0000.pcie:   IB MEM
> 0x0000000000..0x7fffffffff -> 0x0000000000

My best guess is while the above is the parsed order of 'IB MEM'
regions, we sort the entries by address now and that changes which
inbound registers get used for each region. And one doesn't handle >
32-bit addresses. Can you try out this change? It's not what I want
for a final change because the code is just as fragile:

diff --git a/drivers/pci/controller/pci-xgene.c
b/drivers/pci/controller/pci-xgene.c
index 56d0d50338c8..18f05b65439e 100644
--- a/drivers/pci/controller/pci-xgene.c
+++ b/drivers/pci/controller/pci-xgene.c
@@ -465,16 +465,16 @@ static int xgene_pcie_select_ib_reg(u8
*ib_reg_mask, u64 size)
                return 1;
        }

-       if ((size > SZ_1K) && (size < SZ_1T) && !(*ib_reg_mask & (1 << 0))) {
-               *ib_reg_mask |= (1 << 0);
-               return 0;
-       }
-
        if ((size > SZ_1M) && (size < SZ_1T) && !(*ib_reg_mask & (1 << 2))) {
                *ib_reg_mask |= (1 << 2);
                return 2;
        }

+       if ((size > SZ_1K) && (size < SZ_1T) && !(*ib_reg_mask & (1 << 0))) {
+               *ib_reg_mask |= (1 << 0);
+               return 0;
+       }
+
        return -EINVAL;
 }

Re: PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization

[Rob Herring]

On Thu, Nov 18, 2021 at 3:20 PM Rob Herring <robh@kernel.org> wrote:
>
> On Thu, Nov 18, 2021 at 12:10 PM Stéphane Graber <stgraber@ubuntu.com> wrote:
> >
> > Hello,
> >
> > I've recently been given access to a set of 4 APM X-Gene2 Merlin
> > boards (old-ish development platform).
> > Running them on Ubuntu 20.04's stock 5.4 kernel worked fine but trying
> > to run anything else would fail to boot due to a NVME initialization
> > timeout preventing the main drive from showing up at all.
> >
> > Tracking this issue, I first moved to clean mainline kernels and then
> > isolated the issue to be somewhere between 5.4.0 and 5.5.0-rc1, which
> > sadly meant the merge window (so much for a quick bisect...). I've
> > then bisected between those two points and came up with:
> >
> >   6dce5aa59e0bf2430733d7a8b11c205ec10f408e (refs/bisect/bad) PCI:
> > xgene: Use inbound resources for setup
> >
> > I finally switched to the latest 5.15.2 tree, reverted that one
> > commit, built a new kernel and confirmed that those boards now work
> > flawlessly.
> >
> > Unfortunately that's about the extent of my abilities with kernel
> > debugging and I won't pretend to understand what that commit does or
> > how it may be breaking PCIe initialization on those systems.
> >
> > I'm not technically blocked on this, I can manually build my own
> > kernels by reverting that one commit every time, but that's obviously
> > not ideal and I'd much rather have this fixed upstream :)
>
> Doesn't this platform have ACPI f/w you can use? From the log, it
> looks like ACPI tables are passed to the kernel, but since a full DT
> is passed it is used by default. Does 'acpi=on' not work?
>
> Given no one noticed the breakage for 2 years, I'd really like to
> remove these dts and binding files otherwise someone needs to convert
> bindings to schema and fix warnings. Current stats look like this:
> Processing apm:
> warnings: 240
> undocumented compat: 114
>
> For example, I noticed that dma-ranges declares the entries are 32-bit
> (0x42000000 is 32-bit prefetch), yet the PCI bus address and sizes are
> >32-bit. AFAICT, that isn't part of the problem here.
>
> > == Good boot on 5.15.2 (commit reverted) ==
> > Full log at: https://gist.github.com/stgraber/e489b7e55dd7ffaac9f77dd8634ca2ff
> >
> > root@entak:~# dmesg | grep -Ei "nvme|pci"
> > [    0.094146] PCI: CLS 0 bytes, default 64
> > [    0.130573] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
> > [    0.131324] xgene-pcie 1f2b0000.pcie: host bridge /soc/pcie@1f2b0000 ranges:
> > [    0.131344] xgene-pcie 1f2b0000.pcie:   No bus range found for
> > /soc/pcie@1f2b0000, using [bus 00-ff]
> > [    0.131365] xgene-pcie 1f2b0000.pcie:       IO
> > 0xc010000000..0xc01000ffff -> 0x0000000000
> > [    0.131388] xgene-pcie 1f2b0000.pcie:      MEM
> > 0xc120000000..0xc13fffffff -> 0x0020000000
> > [    0.131401] xgene-pcie 1f2b0000.pcie:      MEM
> > 0xe000000000..0xffffffffff -> 0xe000000000
> > [    0.131416] xgene-pcie 1f2b0000.pcie:   IB MEM
> > 0x8000000000..0x807fffffff -> 0x8000000000
> > [    0.131427] xgene-pcie 1f2b0000.pcie:   IB MEM
> > 0x0000000000..0x7fffffffff -> 0x0000000000
>
> My best guess is while the above is the parsed order of 'IB MEM'
> regions, we sort the entries by address now and that changes which
> inbound registers get used for each region. And one doesn't handle >
> 32-bit addresses. Can you try out this change? It's not what I want
> for a final change because the code is just as fragile:

Actually, a better change is this:

diff --git a/drivers/pci/controller/pci-xgene.c
b/drivers/pci/controller/pci-xgene.c
index 56d0d50338c8..d83dbd977418 100644
--- a/drivers/pci/controller/pci-xgene.c
+++ b/drivers/pci/controller/pci-xgene.c
@@ -465,7 +465,7 @@ static int xgene_pcie_select_ib_reg(u8
*ib_reg_mask, u64 size)
                return 1;
        }

-       if ((size > SZ_1K) && (size < SZ_1T) && !(*ib_reg_mask & (1 << 0))) {
+       if ((size > SZ_1K) && (size < SZ_4G) && !(*ib_reg_mask & (1 << 0))) {
                *ib_reg_mask |= (1 << 0);
                return 0;
        }

Re: PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization

[Stéphane Graber]

On Thu, Nov 18, 2021 at 5:03 PM Rob Herring <robh@kernel.org> wrote:
>
> On Thu, Nov 18, 2021 at 3:20 PM Rob Herring <robh@kernel.org> wrote:
> >
> > On Thu, Nov 18, 2021 at 12:10 PM Stéphane Graber <stgraber@ubuntu.com> wrote:
> > >
> > > Hello,
> > >
> > > I've recently been given access to a set of 4 APM X-Gene2 Merlin
> > > boards (old-ish development platform).
> > > Running them on Ubuntu 20.04's stock 5.4 kernel worked fine but trying
> > > to run anything else would fail to boot due to a NVME initialization
> > > timeout preventing the main drive from showing up at all.
> > >
> > > Tracking this issue, I first moved to clean mainline kernels and then
> > > isolated the issue to be somewhere between 5.4.0 and 5.5.0-rc1, which
> > > sadly meant the merge window (so much for a quick bisect...). I've
> > > then bisected between those two points and came up with:
> > >
> > >   6dce5aa59e0bf2430733d7a8b11c205ec10f408e (refs/bisect/bad) PCI:
> > > xgene: Use inbound resources for setup
> > >
> > > I finally switched to the latest 5.15.2 tree, reverted that one
> > > commit, built a new kernel and confirmed that those boards now work
> > > flawlessly.
> > >
> > > Unfortunately that's about the extent of my abilities with kernel
> > > debugging and I won't pretend to understand what that commit does or
> > > how it may be breaking PCIe initialization on those systems.
> > >
> > > I'm not technically blocked on this, I can manually build my own
> > > kernels by reverting that one commit every time, but that's obviously
> > > not ideal and I'd much rather have this fixed upstream :)
> >
> > Doesn't this platform have ACPI f/w you can use? From the log, it
> > looks like ACPI tables are passed to the kernel, but since a full DT
> > is passed it is used by default. Does 'acpi=on' not work?

Gave that a try with a clean 5.15.2 and unfortunately it's not booting
at all, all I get is:

Loading Linux 5.15.2 ...
Loading initial ramdisk ...
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
EFI stub: ERROR: FIRMWARE BUG: Image BSS overlaps adjacent EFI memory region
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
L3C: 8MB

> > Given no one noticed the breakage for 2 years, I'd really like to
> > remove these dts and binding files otherwise someone needs to convert
> > bindings to schema and fix warnings. Current stats look like this:
> > Processing apm:
> > warnings: 240
> > undocumented compat: 114
> >
> > For example, I noticed that dma-ranges declares the entries are 32-bit
> > (0x42000000 is 32-bit prefetch), yet the PCI bus address and sizes are
> > >32-bit. AFAICT, that isn't part of the problem here.
> >
> > > == Good boot on 5.15.2 (commit reverted) ==
> > > Full log at: https://gist.github.com/stgraber/e489b7e55dd7ffaac9f77dd8634ca2ff
> > >
> > > root@entak:~# dmesg | grep -Ei "nvme|pci"
> > > [    0.094146] PCI: CLS 0 bytes, default 64
> > > [    0.130573] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
> > > [    0.131324] xgene-pcie 1f2b0000.pcie: host bridge /soc/pcie@1f2b0000 ranges:
> > > [    0.131344] xgene-pcie 1f2b0000.pcie:   No bus range found for
> > > /soc/pcie@1f2b0000, using [bus 00-ff]
> > > [    0.131365] xgene-pcie 1f2b0000.pcie:       IO
> > > 0xc010000000..0xc01000ffff -> 0x0000000000
> > > [    0.131388] xgene-pcie 1f2b0000.pcie:      MEM
> > > 0xc120000000..0xc13fffffff -> 0x0020000000
> > > [    0.131401] xgene-pcie 1f2b0000.pcie:      MEM
> > > 0xe000000000..0xffffffffff -> 0xe000000000
> > > [    0.131416] xgene-pcie 1f2b0000.pcie:   IB MEM
> > > 0x8000000000..0x807fffffff -> 0x8000000000
> > > [    0.131427] xgene-pcie 1f2b0000.pcie:   IB MEM
> > > 0x0000000000..0x7fffffffff -> 0x0000000000
> >
> > My best guess is while the above is the parsed order of 'IB MEM'
> > regions, we sort the entries by address now and that changes which
> > inbound registers get used for each region. And one doesn't handle >
> > 32-bit addresses. Can you try out this change? It's not what I want
> > for a final change because the code is just as fragile:
>
> Actually, a better change is this:
>
> diff --git a/drivers/pci/controller/pci-xgene.c
> b/drivers/pci/controller/pci-xgene.c
> index 56d0d50338c8..d83dbd977418 100644
> --- a/drivers/pci/controller/pci-xgene.c
> +++ b/drivers/pci/controller/pci-xgene.c
> @@ -465,7 +465,7 @@ static int xgene_pcie_select_ib_reg(u8
> *ib_reg_mask, u64 size)
>                 return 1;
>         }
>
> -       if ((size > SZ_1K) && (size < SZ_1T) && !(*ib_reg_mask & (1 << 0))) {
> +       if ((size > SZ_1K) && (size < SZ_4G) && !(*ib_reg_mask & (1 << 0))) {
>                 *ib_reg_mask |= (1 << 0);
>                 return 0;
>         }

Just tested it, and it booted just fine!

Full boot log: https://gist.github.com/stgraber/41b2419ef88611ab7a2b4dceb028b4f7

Stéphane

Re: PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization

[Thorsten Leemhuis]

Hi, this is your Linux kernel regression tracker speaking.

CC in the regressions list.

On 18.11.21 19:10, Stéphane Graber wrote:
>
> I've recently been given access to a set of 4 APM X-Gene2 Merlin
> boards (old-ish development platform).
> Running them on Ubuntu 20.04's stock 5.4 kernel worked fine but trying
> to run anything else would fail to boot due to a NVME initialization
> timeout preventing the main drive from showing up at all.
> 
> Tracking this issue, I first moved to clean mainline kernels and then
> isolated the issue to be somewhere between 5.4.0 and 5.5.0-rc1, which
> sadly meant the merge window (so much for a quick bisect...). I've
> then bisected between those two points and came up with:
> 
>   6dce5aa59e0bf2430733d7a8b11c205ec10f408e (refs/bisect/bad) PCI:
> xgene: Use inbound resources for setup

TWIMC: To be sure this issue doesn't fall through the cracks unnoticed,
I'm adding it to regzbot, my Linux kernel regression tracking bot:

#regzbot ^introduced 6dce5aa59e0bf2430733d7a8b11c205ec10f408e
#regzbot ignore-activity

Ciao, Thorsten, your Linux kernel regression tracker.


P.S.: If you want to know more about regzbot, check out its
web-interface, the getting start guide, and/or the references documentation:

https://linux-regtracking.leemhuis.info/regzbot/
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md

The last two documents will explain how you can interact with regzbot
yourself if your want to.

Hint for the reporter: when reporting a regression it's in your interest
to tell #regzbot about it in the report, as that will ensure the
regression gets on the radar of regzbot and the regression tracker.
That's in your interest, as they will make sure the report won't fall
through the cracks unnoticed.

Hint for developers: you normally don't need to care about regzbot, just
fix the issue as you normally would. Just remember to include a 'Link:'
tag to the report in the commit message, as explained in
Documentation/process/submitting-patches.rst, which recently was made
more explicit in commit 1f57bd42b77c:
https://git.kernel.org/linus/1f57bd42b77c