Re: [Bug 216094] New: pci-mvebu: SATA HDDs via 88SE6121 AHCI fail with Marvell 88F6281 PCIe

Re: [Bug 216094] New: pci-mvebu: SATA HDDs via 88SE6121 AHCI fail with Marvell 88F6281 PCIe

[Bjorn Helgaas]

On Tue, Jun 07, 2022 at 07:29:03AM +0000, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=216094
> 
>            Summary: pci-mvebu: SATA HDDs via 88SE6121 AHCI fail with
>                     Marvell 88F6281 PCIe
>     Kernel Version: 3.16 ... 5.10
>           Reporter: hajo-linux-bugzilla@noerenberg.de
>                 CC: pali@kernel.org
>         Regression: No
> 
> I would like to continue the SATA-related topic started with Pali
> Rohár at the U-Boot mailing list [1]. I have analysed the issue
> further and come the following conclusion that it is related to the
> PCIe subsystem:
> 
> SATA-2 and SATA-3 hard disks connected to a 88SE6121 (AHCI)
> controller, wired via PCIe to the 88F6281 SoC fail to operate
> ("failed to IDENTIFY" ... "qc timeout") when the pci-mvebu driver
> (Kernel 3.16 .. 5.10 Debian) is in use.

Please attach the complete dmesg logs showing this issue.

From your lspci output with v3.2, the SATA controller is at 00:01.0:

  00:01.0 IDE [0101]: Marvell 88SE6121 SATA II

The v3.16 (with DTB) lspci output is essentially the same except the
controller is at 01:00.0:

  01:00.0 IDE [0101]: Marvell 88SE6121 SATA II

The ahci driver is bound in both cases.  The PCI address and I/O port
assignment differences are of no consequence unless some mvebu driver
defect keeps them from working.

I think what we need is a complete dmesg log and DTB from the newest
possible kernel that fails, plus the same logs from the newest kernel
that works.

> More details:
> 
> - The problem does not exist in 2.6 and 3.16 kernels. With the old
> mach-kirkwood/pcie.c driver all SATA-2/3 hard disks work correctly.
> Especially with a 3.16 kernel it is possible to have identical
> ATA/AHCI drivers but try both PCIe drivers: without DTB ->
> mach-kirkwood -> SATA-2/3 HDDs work; with DTB -> mach-mvebu -> HDDs
> fail.
> 
> - The problem is specific to SATA-2/3 HDDs. Very old SATA-1-only
> HDDs work without problems. This might be related to the available
> data lanes, DMA or other bandwidth-related things -- I can only
> guess. Interestingly it does not help to limit SATA speed
> (libata.force=1.5G ...) with SATA-2/3 HDDs, only 'pure' SATA-1 HDDs
> work with pci-mvebu.
> 
> - The problem was identified with the Seagate Blackarmor NAS440
> hardware. Forum posts show that other users experience similar
> problems with the (very similar) Iomega ix4-200d NAS [2].
> 
> - Within patched U-Boot [3] all (Sata-1/2/3) HDDs always work. Same
> for the 88F6281 SoC onboard SATA ports (sata_mv - not connected via
> PCIe).
> 
> - The mach-kirkwood driver operates the 6281 as class "Host bridge
> [0600]" with Cap "Express (v1) Root Port", the mach-mvebu driver as
> class "PCI bridge [0604]" with "Express (v2) Root Port"
> [4][5][6][7]. Notably the v1/v2, cache line size 32/64 or the
> missing interrupt route might be a key difference.
> 
> From the sources I see that all PCI drivers (mach-kirkwood,
> mach-mvebu and U-Boot) do various unconventional 'magic' things
> (rewriting PCI class of the root complex, changing capabilitys, host
> emulation and so on). This is the point where I currently get lost
> and ask for your help.

> [1] https://lists.denx.de/pipermail/u-boot/2022-March/479197.html
> [2] https://forum.doozan.com/read.php?2,94079,95519#msg-95519
> [3] https://lists.denx.de/pipermail/u-boot/2022-March/479227.html
> [4] lspci Linux version 3.2.0-4-kirkwood mach-kirkwood/pci.c: HDDs ok
> [5] lspci Linux version 3.16.0-0.bpo.4-kirkwood - with DTB -> mvebu-pci: HDDs
> fail
> [6] lspci Linux version 3.16.0-0.bpo.4-kirkwood - without DTB ->
> mach-kirkwood/pci.c: HDDs ok
> [7] lspci Linux version 5.10.0-11-marvell (Debian bullseye) mvebu-pci: HDDs
> fail

Re: [Bug 216094] New: pci-mvebu: SATA HDDs via 88SE6121 AHCI fail with Marvell 88F6281 PCIe

[Pali Rohár]

On Tuesday 07 June 2022 18:49:37 Bjorn Helgaas wrote:
> On Tue, Jun 07, 2022 at 07:29:03AM +0000, bugzilla-daemon@kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=216094
> > 
> >            Summary: pci-mvebu: SATA HDDs via 88SE6121 AHCI fail with
> >                     Marvell 88F6281 PCIe
> >     Kernel Version: 3.16 ... 5.10
> >           Reporter: hajo-linux-bugzilla@noerenberg.de
> >                 CC: pali@kernel.org
> >         Regression: No
> > 
> > I would like to continue the SATA-related topic started with Pali
> > Rohár at the U-Boot mailing list [1]. I have analysed the issue
> > further and come the following conclusion that it is related to the
> > PCIe subsystem:
> > 
> > SATA-2 and SATA-3 hard disks connected to a 88SE6121 (AHCI)
> > controller, wired via PCIe to the 88F6281 SoC fail to operate
> > ("failed to IDENTIFY" ... "qc timeout") when the pci-mvebu driver
> > (Kernel 3.16 .. 5.10 Debian) is in use.
> 
> Please attach the complete dmesg logs showing this issue.
> 
> From your lspci output with v3.2, the SATA controller is at 00:01.0:
> 
>   00:01.0 IDE [0101]: Marvell 88SE6121 SATA II
> 
> The v3.16 (with DTB) lspci output is essentially the same except the
> controller is at 01:00.0:

Old kernel (incorrectly) reports Root Port and Endpoint device from the
other end of the link to the same bus zero. Hence the difference in bus
number between old kernel and new kernel.

>   01:00.0 IDE [0101]: Marvell 88SE6121 SATA II
> 
> The ahci driver is bound in both cases.  The PCI address and I/O port
> assignment differences are of no consequence unless some mvebu driver
> defect keeps them from working.
> 
> I think what we need is a complete dmesg log and DTB from the newest
> possible kernel that fails, plus the same logs from the newest kernel
> that works.

+1

> > More details:
> > 
> > - The problem does not exist in 2.6 and 3.16 kernels. With the old
> > mach-kirkwood/pcie.c driver all SATA-2/3 hard disks work correctly.
> > Especially with a 3.16 kernel it is possible to have identical
> > ATA/AHCI drivers but try both PCIe drivers: without DTB ->
> > mach-kirkwood -> SATA-2/3 HDDs work; with DTB -> mach-mvebu -> HDDs
> > fail.

So it looks like that issue is with DT based setup.

Just by a chance, could you try to boot kernel with pcie_aspm=off or
pci=nomsi cmdline options? There are some issues in driver/controller
related to link retraining and MSI interrupts which ASPM kernel code can
trigger.

> > - The problem is specific to SATA-2/3 HDDs. Very old SATA-1-only
> > HDDs work without problems. This might be related to the available
> > data lanes, DMA or other bandwidth-related things -- I can only
> > guess. Interestingly it does not help to limit SATA speed
> > (libata.force=1.5G ...) with SATA-2/3 HDDs, only 'pure' SATA-1 HDDs
> > work with pci-mvebu.
> > 
> > - The problem was identified with the Seagate Blackarmor NAS440
> > hardware. Forum posts show that other users experience similar
> > problems with the (very similar) Iomega ix4-200d NAS [2].
> > 
> > - Within patched U-Boot [3] all (Sata-1/2/3) HDDs always work. Same
> > for the 88F6281 SoC onboard SATA ports (sata_mv - not connected via
> > PCIe).
> > 
> > - The mach-kirkwood driver operates the 6281 as class "Host bridge
> > [0600]" with Cap "Express (v1) Root Port", the mach-mvebu driver as
> > class "PCI bridge [0604]" with "Express (v2) Root Port"
> > [4][5][6][7]. Notably the v1/v2, cache line size 32/64 or the
> > missing interrupt route might be a key difference.
> > 
> > From the sources I see that all PCI drivers (mach-kirkwood,
> > mach-mvebu and U-Boot) do various unconventional 'magic' things
> > (rewriting PCI class of the root complex, changing capabilitys, host
> > emulation and so on). This is the point where I currently get lost
> > and ask for your help.

This 'magic' is there because of broken PCIe controller and its PCIe
Root Port in all 32-bit Marvell SoCs.

> > [1] https://lists.denx.de/pipermail/u-boot/2022-March/479197.html
> > [2] https://forum.doozan.com/read.php?2,94079,95519#msg-95519
> > [3] https://lists.denx.de/pipermail/u-boot/2022-March/479227.html
> > [4] lspci Linux version 3.2.0-4-kirkwood mach-kirkwood/pci.c: HDDs ok
> > [5] lspci Linux version 3.16.0-0.bpo.4-kirkwood - with DTB -> mvebu-pci: HDDs
> > fail
> > [6] lspci Linux version 3.16.0-0.bpo.4-kirkwood - without DTB ->
> > mach-kirkwood/pci.c: HDDs ok
> > [7] lspci Linux version 5.10.0-11-marvell (Debian bullseye) mvebu-pci: HDDs
> > fail