Re: [EXT] Re: pci mvebu issue (memory controller)

["Pali Rohár" <pali@kernel.org>]

On Wednesday 10 February 2021 13:59:41 Stefan Chulski wrote:
> > > (sending this e-mail again because previously I sent it to Thomas' old
> > > e-mail address at free-electrons)
> > 
> > Thanks. Turns out I still receive e-mail sent to @free-electrons.com, so I had
> > seen your previous e-mail but didn't had the chance to reply.
> > 
> > > we have enountered an issue with pci-mvebu driver and would like your
> > > opinion, since you are the author of commit
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pu
> > > b_scm_linux_kernel_git_torvalds_linux.git_commit_-3Fid-
> > 3Df4ac99011e542
> > >
> > d06ea2bda10063502583c6d7991&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&
> > r=DDQ3dK
> > > wkTIxKAl6_Bs7GMx4zhJArrXKN2mDMOXGh7lg&m=lENmudbu2hlK44mVm-
> > e8bgdi9Rm2AC
> > > DXN8QY0frgcuY&s=7109I-
> > xvpx1wW532pxvk1W8s_XeG77VQf2iP7QzhEao&e=
> > >
> > > After upgrading to new version of U-Boot on a Armada XP / 38x device,
> > > some WiFi cards stopped working in kernel. Ath10k driver, for example,
> > > could not load firmware into the card.
> > >
> > > We discovered that the issue is caused by U-Boot:
> > > - when U-Boot's pci_mvebu driver was converted to driver model API,
> > >   U-Boot started to configure PCIe registers not only for the newtork
> > >   adapter, but also for the Marvell Memory Controller (that you are
> > >   mentioning in your commit).
> > > - Since pci-mvebu driver in Linux is ignoring the Marvell Memory
> > >   Controller device, and U-Boot configures its registers (BARs and what
> > >   not), after kernel boots, the registers of this device are
> > >   incompatible with kernel, or something, and this causes problems for
> > >   the real PCIe device.
> > > - Stefan Roese has temporarily solved this issue with U-Boot commit
> > >   https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__gitlab.denx.de_u-2Dboot_custodians_u-2Dboot-2Dmarvell_-
> > 2D_commit_6a2fa284aee2981be2c7661b3757ce112de8d528&d=DwIFaQ&c=n
> > KjWec2b6R0mOyPaz7xtfQ&r=DDQ3dKwkTIxKAl6_Bs7GMx4zhJArrXKN2mDM
> > OXGh7lg&m=lENmudbu2hlK44mVm-
> > e8bgdi9Rm2ACDXN8QY0frgcuY&s=B0eKBkblEygPGYvKDdMuwzzYhDg5Jlh_Q4
> > eXHlIL-oc&e=
> > >   which basically just masks the Memory Controller's existence.
> > >
> > > - in Linux commit f4ac99011e54 ("pci: mvebu: no longer fake the slot
> > >   location of downstream devices") you mention that:
> > >
> > >    * On slot 0, a "Marvell Memory controller", identical on all PCIe
> > >      interfaces, and which isn't useful when the Marvell SoC is the PCIe
> > >      root complex (i.e, the normal case when we run Linux on the Marvell
> > >      SoC).
> > >
> > > What we are wondering is:
> > > - what does the Marvell Memory controller really do? Can it be used to
> > >   configure something? It clearly does something, because if it is
> > >   configured in U-Boot somehow but not in kernel, problems can occur.
> > > - is the best solution really just to ignore this device?
> > > - should U-Boot also start doing what commit f4ac99011e54 does? I.e.
> > >   to make sure that the real device is in slot 0, and Marvell Memory
> > >   Controller in slot 1.
> > > - why is Linux ignoring this device? It isn't even listed in lspci
> > >   output.
> > 
> > To be honest, I don't have much details about what this device does, and my
> > memory is unclear on whether I really ever had any details. I vaguely
> > remember that this is a device that made sense when the Marvell PCIe
> > controller is used as an endpoint, and in such a situation this device also the
> > root complex to "see" the physical memory of the Marvell SoC. And
> > therefore in a situation where the Marvell PCIe controller is the root
> > complex, seeing this device didn't make sense.
> > 
> > In addition, I /think/ it was causing problems with the MBus windows
> > allocation. Indeed, if this device is visible, then we will try to allocate MBus
> > windows for its different BARs, and those windows are in limited number.
> > 
> > I know this isn't a very helpful answer, but the documentation on this is
> > pretty much nonexistent, and I don't remember ever having very solid and
> > convincing answers.
> > 
> > I've added in Cc Stefan Chulski, from Marvell, who has recently posted
> > patches on the PPv2 driver. I don't know if he will have details about PCIe,
> > but perhaps he will be able to ask internally at Marvell.
> > 
> > Best regards,
> 
> I not familiar with Armada XP PCIe. But I can check internally at Marvell.
> 
> Best Regards,
> Stefan.
> 

Stefan: If you get any information internally in Marvell, please let us know!

Bjorn: What do you think, should Linux kernel completely hide some PCIe
devices from /sys hierarchy and also from 'lspci' output? Or should
kernel preserve even non-functional / unknown PCIe devices visible in
'lspci' output?