Re: [EXT] Re: pci mvebu issue (memory controller)

[Bjorn Helgaas <helgaas@kernel.org>]

On Wed, Jun 02, 2021 at 01:07:03PM +0200, Pali Rohár wrote:

> In configuration with *bad* suffix is used U-Boot which does not ignore
> PCIe device Memory controller and configure it when U-Boot initialize.
> In this configuration loaded kernel is unable to initialize wifi cards.
> 
> In configuration with *ok* suffix is U-Boot explicitly patched to
> ignores PCIe device Memory controller and loaded kernel can use wifi
> cards without any issue.
> 
> In both configurations is used same kernel version. As I wrote in
> previous emails kernel already ignores and hides Memory controller PCIe
> device, so lspci does not see it.
> 
> In attachment I'm sending dmesg and lspci outputs from Linux and pci
> output from U-Boot.
> 
> What is suspicious for me is that this Memory controller device is at
> the same bus as wifi card. PCIe is "point to point", so at the other end
> of link should be only one device... Therefore I'm not sure if kernel
> can handle such thing like "two PCIe devices" at other end of PCIe link.
> 
> Could you look at attached logs if you see something suspicious here? Or
> if you need other logs (either from U-Boot or kernel) please let me
> know.
> 
> Note that U-Boot does not see PCIe Bridge as it is emulated only by
> kernel. So U-Boot enumerates buses from zero and kernel from one (as
> kernel's bus zero is for emulated PCIe Bridges).

I've lost track of what the problem is or what patch we're evaluating.

Here's what I see from dmesg/lspci/uboot:

  # dmesg (both bad/ok) and lspci:
  00:01.0 [11ab:6820] Root Port to [bus 01]
  00:02.0 [11ab:6820] Root Port to [bus 02]
  00:03.0 [11ab:6820] Root Port to [bus 03]
  01:00.0 [168c:002e] Atheros AR9287 NIC
  02:00.0 [168c:0046] Atheros QCA9984 NIC
  03:00.0 [168c:003c] Atheros QCA986x/988x NIC

The above looks perfectly reasonable.

  # uboot (bad):
  00.00.00 [11ab:6820] memory controller
  00.01.00 [168c:002e] NIC
  01.00.00 [11ab:6820] memory controller
  01.01.00 [168c:0046] NIC
  02.00.00 [11ab:6820] memory controller
  02.01.00 [168c:003c] NIC

The above looks dubious at best.  Bus 00 clearly must be a root bus
because bus 00 can never be a bridge's secondary bus.

Either buses 01 and 02 need to also be root buses (e.g., if we had
three host bridges, one leading to bus 00, another to bus 01, and
another to bus 02), OR there must be Root Ports that act as bridges
leading from bus 00 to bus 01 and bus 02.  The "memory controllers"
are vendor/device ID [11ab:6820], which Linux thinks are Root Ports,
so I assume they are really Root Ports (or some emulation of them).

It's *possible* to have both a Root Port and a NIC on bus 0, as shown
here.  However, the NIC would have to be a Root Complex integrated
Endpoint, and this NIC ([168c:002e]) is not one of those.  It's a
garden-variety PCIe legacy endpoint connected by a link.  So this NIC
cannot actually be on bus 00.

All these NICs are PCIe legacy endpoints with links, so they all must
have a Root Port leading to them.  So this topology is not really
possible.

  # uboot (ok):
  00.00.00 [168c:002e] NIC
  01.00.00 [168c:0046] NIC
  02.00.00 [168c:003c] NIC

This topology is impossible from a PCI perspective because there's no
way to get from bus 00 to bus 01 or 02.