pci mvebu issue (memory controller)

pci mvebu issue (memory controller)

[Marek Behún]

Hello Thomas,

(sending this e-mail again because previously I sent it to Thomas' old
e-mail address at free-electrons)

we have enountered an issue with pci-mvebu driver and would like your
opinion, since you are the author of commit
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f4ac99011e542d06ea2bda10063502583c6d7991

After upgrading to new version of U-Boot on a Armada XP / 38x device,
some WiFi cards stopped working in kernel. Ath10k driver, for example,
could not load firmware into the card.

We discovered that the issue is caused by U-Boot:
- when U-Boot's pci_mvebu driver was converted to driver model API,
  U-Boot started to configure PCIe registers not only for the newtork
  adapter, but also for the Marvell Memory Controller (that you are
  mentioning in your commit).
- Since pci-mvebu driver in Linux is ignoring the Marvell Memory
  Controller device, and U-Boot configures its registers (BARs and what
  not), after kernel boots, the registers of this device are
  incompatible with kernel, or something, and this causes problems for
  the real PCIe device.
- Stefan Roese has temporarily solved this issue with U-Boot commit
  https://gitlab.denx.de/u-boot/custodians/u-boot-marvell/-/commit/6a2fa284aee2981be2c7661b3757ce112de8d528
  which basically just masks the Memory Controller's existence.

- in Linux commit f4ac99011e54 ("pci: mvebu: no longer fake the slot
  location of downstream devices") you mention that:

   * On slot 0, a "Marvell Memory controller", identical on all PCIe
     interfaces, and which isn't useful when the Marvell SoC is the PCIe
     root complex (i.e, the normal case when we run Linux on the Marvell
     SoC).

What we are wondering is:
- what does the Marvell Memory controller really do? Can it be used to
  configure something? It clearly does something, because if it is
  configured in U-Boot somehow but not in kernel, problems can occur.
- is the best solution really just to ignore this device?
- should U-Boot also start doing what commit f4ac99011e54 does? I.e.
  to make sure that the real device is in slot 0, and Marvell Memory
  Controller in slot 1.
- why is Linux ignoring this device? It isn't even listed in lspci
  output.

Thanks,

Marek

Re: pci mvebu issue (memory controller)

[Thomas Petazzoni]

Hello Marek,

On Tue, 9 Feb 2021 14:17:59 +0100
Marek Behún <kabel@kernel.org> wrote:

> (sending this e-mail again because previously I sent it to Thomas' old
> e-mail address at free-electrons)

Thanks. Turns out I still receive e-mail sent to @free-electrons.com,
so I had seen your previous e-mail but didn't had the chance to reply.

> we have enountered an issue with pci-mvebu driver and would like your
> opinion, since you are the author of commit
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f4ac99011e542d06ea2bda10063502583c6d7991
> 
> After upgrading to new version of U-Boot on a Armada XP / 38x device,
> some WiFi cards stopped working in kernel. Ath10k driver, for example,
> could not load firmware into the card.
> 
> We discovered that the issue is caused by U-Boot:
> - when U-Boot's pci_mvebu driver was converted to driver model API,
>   U-Boot started to configure PCIe registers not only for the newtork
>   adapter, but also for the Marvell Memory Controller (that you are
>   mentioning in your commit).
> - Since pci-mvebu driver in Linux is ignoring the Marvell Memory
>   Controller device, and U-Boot configures its registers (BARs and what
>   not), after kernel boots, the registers of this device are
>   incompatible with kernel, or something, and this causes problems for
>   the real PCIe device.
> - Stefan Roese has temporarily solved this issue with U-Boot commit
>   https://gitlab.denx.de/u-boot/custodians/u-boot-marvell/-/commit/6a2fa284aee2981be2c7661b3757ce112de8d528
>   which basically just masks the Memory Controller's existence.
> 
> - in Linux commit f4ac99011e54 ("pci: mvebu: no longer fake the slot
>   location of downstream devices") you mention that:
> 
>    * On slot 0, a "Marvell Memory controller", identical on all PCIe
>      interfaces, and which isn't useful when the Marvell SoC is the PCIe
>      root complex (i.e, the normal case when we run Linux on the Marvell
>      SoC).
> 
> What we are wondering is:
> - what does the Marvell Memory controller really do? Can it be used to
>   configure something? It clearly does something, because if it is
>   configured in U-Boot somehow but not in kernel, problems can occur.
> - is the best solution really just to ignore this device?
> - should U-Boot also start doing what commit f4ac99011e54 does? I.e.
>   to make sure that the real device is in slot 0, and Marvell Memory
>   Controller in slot 1.
> - why is Linux ignoring this device? It isn't even listed in lspci
>   output.

To be honest, I don't have much details about what this device does,
and my memory is unclear on whether I really ever had any details. I
vaguely remember that this is a device that made sense when the Marvell
PCIe controller is used as an endpoint, and in such a situation this
device also the root complex to "see" the physical memory of the
Marvell SoC. And therefore in a situation where the Marvell PCIe
controller is the root complex, seeing this device didn't make sense.

In addition, I /think/ it was causing problems with the MBus windows
allocation. Indeed, if this device is visible, then we will try to
allocate MBus windows for its different BARs, and those windows are in
limited number.

I know this isn't a very helpful answer, but the documentation on this
is pretty much nonexistent, and I don't remember ever having very
solid and convincing answers.

I've added in Cc Stefan Chulski, from Marvell, who has recently posted
patches on the PPv2 driver. I don't know if he will have details about
PCIe, but perhaps he will be able to ask internally at Marvell.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

RE: [EXT] Re: pci mvebu issue (memory controller)

[Stefan Chulski]

> > (sending this e-mail again because previously I sent it to Thomas' old
> > e-mail address at free-electrons)
> 
> Thanks. Turns out I still receive e-mail sent to @free-electrons.com, so I had
> seen your previous e-mail but didn't had the chance to reply.
> 
> > we have enountered an issue with pci-mvebu driver and would like your
> > opinion, since you are the author of commit
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pu
> > b_scm_linux_kernel_git_torvalds_linux.git_commit_-3Fid-
> 3Df4ac99011e542
> >
> d06ea2bda10063502583c6d7991&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&
> r=DDQ3dK
> > wkTIxKAl6_Bs7GMx4zhJArrXKN2mDMOXGh7lg&m=lENmudbu2hlK44mVm-
> e8bgdi9Rm2AC
> > DXN8QY0frgcuY&s=7109I-
> xvpx1wW532pxvk1W8s_XeG77VQf2iP7QzhEao&e=
> >
> > After upgrading to new version of U-Boot on a Armada XP / 38x device,
> > some WiFi cards stopped working in kernel. Ath10k driver, for example,
> > could not load firmware into the card.
> >
> > We discovered that the issue is caused by U-Boot:
> > - when U-Boot's pci_mvebu driver was converted to driver model API,
> >   U-Boot started to configure PCIe registers not only for the newtork
> >   adapter, but also for the Marvell Memory Controller (that you are
> >   mentioning in your commit).
> > - Since pci-mvebu driver in Linux is ignoring the Marvell Memory
> >   Controller device, and U-Boot configures its registers (BARs and what
> >   not), after kernel boots, the registers of this device are
> >   incompatible with kernel, or something, and this causes problems for
> >   the real PCIe device.
> > - Stefan Roese has temporarily solved this issue with U-Boot commit
> >   https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__gitlab.denx.de_u-2Dboot_custodians_u-2Dboot-2Dmarvell_-
> 2D_commit_6a2fa284aee2981be2c7661b3757ce112de8d528&d=DwIFaQ&c=n
> KjWec2b6R0mOyPaz7xtfQ&r=DDQ3dKwkTIxKAl6_Bs7GMx4zhJArrXKN2mDM
> OXGh7lg&m=lENmudbu2hlK44mVm-
> e8bgdi9Rm2ACDXN8QY0frgcuY&s=B0eKBkblEygPGYvKDdMuwzzYhDg5Jlh_Q4
> eXHlIL-oc&e=
> >   which basically just masks the Memory Controller's existence.
> >
> > - in Linux commit f4ac99011e54 ("pci: mvebu: no longer fake the slot
> >   location of downstream devices") you mention that:
> >
> >    * On slot 0, a "Marvell Memory controller", identical on all PCIe
> >      interfaces, and which isn't useful when the Marvell SoC is the PCIe
> >      root complex (i.e, the normal case when we run Linux on the Marvell
> >      SoC).
> >
> > What we are wondering is:
> > - what does the Marvell Memory controller really do? Can it be used to
> >   configure something? It clearly does something, because if it is
> >   configured in U-Boot somehow but not in kernel, problems can occur.
> > - is the best solution really just to ignore this device?
> > - should U-Boot also start doing what commit f4ac99011e54 does? I.e.
> >   to make sure that the real device is in slot 0, and Marvell Memory
> >   Controller in slot 1.
> > - why is Linux ignoring this device? It isn't even listed in lspci
> >   output.
> 
> To be honest, I don't have much details about what this device does, and my
> memory is unclear on whether I really ever had any details. I vaguely
> remember that this is a device that made sense when the Marvell PCIe
> controller is used as an endpoint, and in such a situation this device also the
> root complex to "see" the physical memory of the Marvell SoC. And
> therefore in a situation where the Marvell PCIe controller is the root
> complex, seeing this device didn't make sense.
> 
> In addition, I /think/ it was causing problems with the MBus windows
> allocation. Indeed, if this device is visible, then we will try to allocate MBus
> windows for its different BARs, and those windows are in limited number.
> 
> I know this isn't a very helpful answer, but the documentation on this is
> pretty much nonexistent, and I don't remember ever having very solid and
> convincing answers.
> 
> I've added in Cc Stefan Chulski, from Marvell, who has recently posted
> patches on the PPv2 driver. I don't know if he will have details about PCIe,
> but perhaps he will be able to ask internally at Marvell.
> 
> Best regards,

I not familiar with Armada XP PCIe. But I can check internally at Marvell.

Best Regards,
Stefan.

Re: [EXT] Re: pci mvebu issue (memory controller)

[Pali Rohár]

On Wednesday 10 February 2021 13:59:41 Stefan Chulski wrote:
> > > (sending this e-mail again because previously I sent it to Thomas' old
> > > e-mail address at free-electrons)
> > 
> > Thanks. Turns out I still receive e-mail sent to @free-electrons.com, so I had
> > seen your previous e-mail but didn't had the chance to reply.
> > 
> > > we have enountered an issue with pci-mvebu driver and would like your
> > > opinion, since you are the author of commit
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pu
> > > b_scm_linux_kernel_git_torvalds_linux.git_commit_-3Fid-
> > 3Df4ac99011e542
> > >
> > d06ea2bda10063502583c6d7991&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&
> > r=DDQ3dK
> > > wkTIxKAl6_Bs7GMx4zhJArrXKN2mDMOXGh7lg&m=lENmudbu2hlK44mVm-
> > e8bgdi9Rm2AC
> > > DXN8QY0frgcuY&s=7109I-
> > xvpx1wW532pxvk1W8s_XeG77VQf2iP7QzhEao&e=
> > >
> > > After upgrading to new version of U-Boot on a Armada XP / 38x device,
> > > some WiFi cards stopped working in kernel. Ath10k driver, for example,
> > > could not load firmware into the card.
> > >
> > > We discovered that the issue is caused by U-Boot:
> > > - when U-Boot's pci_mvebu driver was converted to driver model API,
> > >   U-Boot started to configure PCIe registers not only for the newtork
> > >   adapter, but also for the Marvell Memory Controller (that you are
> > >   mentioning in your commit).
> > > - Since pci-mvebu driver in Linux is ignoring the Marvell Memory
> > >   Controller device, and U-Boot configures its registers (BARs and what
> > >   not), after kernel boots, the registers of this device are
> > >   incompatible with kernel, or something, and this causes problems for
> > >   the real PCIe device.
> > > - Stefan Roese has temporarily solved this issue with U-Boot commit
> > >   https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__gitlab.denx.de_u-2Dboot_custodians_u-2Dboot-2Dmarvell_-
> > 2D_commit_6a2fa284aee2981be2c7661b3757ce112de8d528&d=DwIFaQ&c=n
> > KjWec2b6R0mOyPaz7xtfQ&r=DDQ3dKwkTIxKAl6_Bs7GMx4zhJArrXKN2mDM
> > OXGh7lg&m=lENmudbu2hlK44mVm-
> > e8bgdi9Rm2ACDXN8QY0frgcuY&s=B0eKBkblEygPGYvKDdMuwzzYhDg5Jlh_Q4
> > eXHlIL-oc&e=
> > >   which basically just masks the Memory Controller's existence.
> > >
> > > - in Linux commit f4ac99011e54 ("pci: mvebu: no longer fake the slot
> > >   location of downstream devices") you mention that:
> > >
> > >    * On slot 0, a "Marvell Memory controller", identical on all PCIe
> > >      interfaces, and which isn't useful when the Marvell SoC is the PCIe
> > >      root complex (i.e, the normal case when we run Linux on the Marvell
> > >      SoC).
> > >
> > > What we are wondering is:
> > > - what does the Marvell Memory controller really do? Can it be used to
> > >   configure something? It clearly does something, because if it is
> > >   configured in U-Boot somehow but not in kernel, problems can occur.
> > > - is the best solution really just to ignore this device?
> > > - should U-Boot also start doing what commit f4ac99011e54 does? I.e.
> > >   to make sure that the real device is in slot 0, and Marvell Memory
> > >   Controller in slot 1.
> > > - why is Linux ignoring this device? It isn't even listed in lspci
> > >   output.
> > 
> > To be honest, I don't have much details about what this device does, and my
> > memory is unclear on whether I really ever had any details. I vaguely
> > remember that this is a device that made sense when the Marvell PCIe
> > controller is used as an endpoint, and in such a situation this device also the
> > root complex to "see" the physical memory of the Marvell SoC. And
> > therefore in a situation where the Marvell PCIe controller is the root
> > complex, seeing this device didn't make sense.
> > 
> > In addition, I /think/ it was causing problems with the MBus windows
> > allocation. Indeed, if this device is visible, then we will try to allocate MBus
> > windows for its different BARs, and those windows are in limited number.
> > 
> > I know this isn't a very helpful answer, but the documentation on this is
> > pretty much nonexistent, and I don't remember ever having very solid and
> > convincing answers.
> > 
> > I've added in Cc Stefan Chulski, from Marvell, who has recently posted
> > patches on the PPv2 driver. I don't know if he will have details about PCIe,
> > but perhaps he will be able to ask internally at Marvell.
> > 
> > Best regards,
> 
> I not familiar with Armada XP PCIe. But I can check internally at Marvell.
> 
> Best Regards,
> Stefan.
> 

Stefan: If you get any information internally in Marvell, please let us know!

Bjorn: What do you think, should Linux kernel completely hide some PCIe
devices from /sys hierarchy and also from 'lspci' output? Or should
kernel preserve even non-functional / unknown PCIe devices visible in
'lspci' output?

Re: [EXT] Re: pci mvebu issue (memory controller)

[Bjorn Helgaas]

On Fri, Feb 19, 2021 at 06:44:06PM +0100, Pali Rohár wrote:
> On Wednesday 10 February 2021 13:59:41 Stefan Chulski wrote:
> > > > (sending this e-mail again because previously I sent it to Thomas' old
> > > > e-mail address at free-electrons)
> > > 
> > > Thanks. Turns out I still receive e-mail sent to @free-electrons.com, so I had
> > > seen your previous e-mail but didn't had the chance to reply.
> > > 
> > > > we have enountered an issue with pci-mvebu driver and would like your
> > > > opinion, since you are the author of commit
> > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__git.kernel.org_pu
> > > > b_scm_linux_kernel_git_torvalds_linux.git_commit_-3Fid-
> > > 3Df4ac99011e542
> > > >
> > > d06ea2bda10063502583c6d7991&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&
> > > r=DDQ3dK
> > > > wkTIxKAl6_Bs7GMx4zhJArrXKN2mDMOXGh7lg&m=lENmudbu2hlK44mVm-
> > > e8bgdi9Rm2AC
> > > > DXN8QY0frgcuY&s=7109I-
> > > xvpx1wW532pxvk1W8s_XeG77VQf2iP7QzhEao&e=
> > > >
> > > > After upgrading to new version of U-Boot on a Armada XP / 38x device,
> > > > some WiFi cards stopped working in kernel. Ath10k driver, for example,
> > > > could not load firmware into the card.
> > > >
> > > > We discovered that the issue is caused by U-Boot:
> > > > - when U-Boot's pci_mvebu driver was converted to driver model API,
> > > >   U-Boot started to configure PCIe registers not only for the newtork
> > > >   adapter, but also for the Marvell Memory Controller (that you are
> > > >   mentioning in your commit).
> > > > - Since pci-mvebu driver in Linux is ignoring the Marvell Memory
> > > >   Controller device, and U-Boot configures its registers (BARs and what
> > > >   not), after kernel boots, the registers of this device are
> > > >   incompatible with kernel, or something, and this causes problems for
> > > >   the real PCIe device.
> > > > - Stefan Roese has temporarily solved this issue with U-Boot commit
> > > >   https://urldefense.proofpoint.com/v2/url?u=https-
> > > 3A__gitlab.denx.de_u-2Dboot_custodians_u-2Dboot-2Dmarvell_-
> > > 2D_commit_6a2fa284aee2981be2c7661b3757ce112de8d528&d=DwIFaQ&c=n
> > > KjWec2b6R0mOyPaz7xtfQ&r=DDQ3dKwkTIxKAl6_Bs7GMx4zhJArrXKN2mDM
> > > OXGh7lg&m=lENmudbu2hlK44mVm-
> > > e8bgdi9Rm2ACDXN8QY0frgcuY&s=B0eKBkblEygPGYvKDdMuwzzYhDg5Jlh_Q4
> > > eXHlIL-oc&e=
> > > >   which basically just masks the Memory Controller's existence.
> > > >
> > > > - in Linux commit f4ac99011e54 ("pci: mvebu: no longer fake the slot
> > > >   location of downstream devices") you mention that:
> > > >
> > > >    * On slot 0, a "Marvell Memory controller", identical on all PCIe
> > > >      interfaces, and which isn't useful when the Marvell SoC is the PCIe
> > > >      root complex (i.e, the normal case when we run Linux on the Marvell
> > > >      SoC).
> > > >
> > > > What we are wondering is:
> > > > - what does the Marvell Memory controller really do? Can it be used to
> > > >   configure something? It clearly does something, because if it is
> > > >   configured in U-Boot somehow but not in kernel, problems can occur.
> > > > - is the best solution really just to ignore this device?
> > > > - should U-Boot also start doing what commit f4ac99011e54 does? I.e.
> > > >   to make sure that the real device is in slot 0, and Marvell Memory
> > > >   Controller in slot 1.
> > > > - why is Linux ignoring this device? It isn't even listed in lspci
> > > >   output.
> > > 
> > > To be honest, I don't have much details about what this device does, and my
> > > memory is unclear on whether I really ever had any details. I vaguely
> > > remember that this is a device that made sense when the Marvell PCIe
> > > controller is used as an endpoint, and in such a situation this device also the
> > > root complex to "see" the physical memory of the Marvell SoC. And
> > > therefore in a situation where the Marvell PCIe controller is the root
> > > complex, seeing this device didn't make sense.
> > > 
> > > In addition, I /think/ it was causing problems with the MBus windows
> > > allocation. Indeed, if this device is visible, then we will try to allocate MBus
> > > windows for its different BARs, and those windows are in limited number.
> > > 
> > > I know this isn't a very helpful answer, but the documentation on this is
> > > pretty much nonexistent, and I don't remember ever having very solid and
> > > convincing answers.
> > > 
> > > I've added in Cc Stefan Chulski, from Marvell, who has recently posted
> > > patches on the PPv2 driver. I don't know if he will have details about PCIe,
> > > but perhaps he will be able to ask internally at Marvell.
> > > 
> > > Best regards,
> > 
> > I not familiar with Armada XP PCIe. But I can check internally at Marvell.
> > 
> > Best Regards,
> > Stefan.
> > 
> 
> Stefan: If you get any information internally in Marvell, please let us know!
> 
> Bjorn: What do you think, should Linux kernel completely hide some PCIe
> devices from /sys hierarchy and also from 'lspci' output? Or should
> kernel preserve even non-functional / unknown PCIe devices visible in
> 'lspci' output?

In general I don't think the kernel should hide PCI devices.  The PCI
core has no way of knowing whether devices are non-functional, and
"unknown" doesn't really mean anything because a driver could be
loaded later.

But if a device is in use by firmware, or if exposing it causes some
problem, it might make sense to hide it.

In your case, the problem description is "... the registers of this
device are incompatible with kernel, or something, and this causes
problems for the real PCIe device ..."

That's not much to go on.  Someone with more knowledge of the actual
problem would have to weigh in on whether hiding a device is the best
approach.

With more details we might see what the conflict between the devices
is.  E.g., maybe we assign the same resources to both, or maybe we
don't assign a bridge window to reach the WiFi card.

Bjorn

Re: pci mvebu issue (memory controller)

[Pali Rohár]

Hello! See explanation below.

On Tuesday 09 February 2021 14:17:59 Marek Behún wrote:
> Hello Thomas,
> 
> (sending this e-mail again because previously I sent it to Thomas' old
> e-mail address at free-electrons)
> 
> we have enountered an issue with pci-mvebu driver and would like your
> opinion, since you are the author of commit
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f4ac99011e542d06ea2bda10063502583c6d7991
> 
> After upgrading to new version of U-Boot on a Armada XP / 38x device,
> some WiFi cards stopped working in kernel. Ath10k driver, for example,
> could not load firmware into the card.
> 
> We discovered that the issue is caused by U-Boot:
> - when U-Boot's pci_mvebu driver was converted to driver model API,
>   U-Boot started to configure PCIe registers not only for the newtork
>   adapter, but also for the Marvell Memory Controller (that you are
>   mentioning in your commit).
> - Since pci-mvebu driver in Linux is ignoring the Marvell Memory
>   Controller device, and U-Boot configures its registers (BARs and what
>   not), after kernel boots, the registers of this device are
>   incompatible with kernel, or something, and this causes problems for
>   the real PCIe device.
> - Stefan Roese has temporarily solved this issue with U-Boot commit
>   https://gitlab.denx.de/u-boot/custodians/u-boot-marvell/-/commit/6a2fa284aee2981be2c7661b3757ce112de8d528
>   which basically just masks the Memory Controller's existence.
> 
> - in Linux commit f4ac99011e54 ("pci: mvebu: no longer fake the slot
>   location of downstream devices") you mention that:
> 
>    * On slot 0, a "Marvell Memory controller", identical on all PCIe
>      interfaces, and which isn't useful when the Marvell SoC is the PCIe
>      root complex (i.e, the normal case when we run Linux on the Marvell
>      SoC).
> 
> What we are wondering is:
> - what does the Marvell Memory controller really do? Can it be used to
>   configure something? It clearly does something, because if it is
>   configured in U-Boot somehow but not in kernel, problems can occur.
> - is the best solution really just to ignore this device?
> - should U-Boot also start doing what commit f4ac99011e54 does? I.e.
>   to make sure that the real device is in slot 0, and Marvell Memory
>   Controller in slot 1.
> - why is Linux ignoring this device? It isn't even listed in lspci
>   output.
> 
> Thanks,
> 
> Marek

tl;dr

- Mysterious Marvell Memory Controller is PCIe Root Port (it can be
  verified by e.g. doing config space dump from U-Boot and then parse
  it via lspci)
- Config space of this PCIe device is mapped directly to the address
  space of PCIe controller (to offset zero)
- It has config space with Header Type 0 and Class Code 0x5080
- BARs configure PCIe controller itself, BAR0 must point to beginning of
  the SoC registers, other BARs to DDR memory address space
- Both U-Boot and Kernel pci mvebu drivers set Secondary Bus num to zero
- Patch which was fixing this issue disappeared from kernel

I think this explains all mentioned issues in previous email. Controller
driver configures registers for SoC and DDR and then PCI core/pnp
reconfigured them via config space to different values = no PCIe device
is working as PCIe controller it not able to access SoC registers and
DDR memory correctly anymore.

Bjorn, it is normal that PCIe Root Port device has Type 0 config space
and Class Code 0x5080 (Memory controller)? Because I thought that PCIe
Root Port device must have Class Code 0x6004 (PCI Bridge) with Type 1
config space.

And what should happen according to PCIe standards when both primary and
secondary bus numbers are configured to zeros? Or to other same numbers?

On primary bus is that Memory Controller == Root Port and on secondary
bus is endpoint card. Marvell has additional register for specifying
device number at which Root Port appears. And it looks like that if
primary and secondary bus numbers are same, then on this bus at Root
Port device address appears Root Port and on all other device addresses
appears endpoint card (which looks crazy if endpoint card is at all
possible BDF addresses where B=primary=secondary and D!=root_port).
But I have no idea what happens for other buses.

Seems that due to these issues pci-mvebu.c kernel driver filters access
to this PCIe Root Port device and uses pci-bridge-emul.c for providing
emulated PCIe Root Port device. It sets Root Port device address to 1
and allow access only to device address 0 (at which is endpoint card).

This issues appears in all Marvell SoCs. Here are just few lspci output
sent by different people in past. And All have one common thing: device
with "Root Port" and "Memory controller: Marvell" marking:

https://lore.kernel.org/linux-arm-kernel/alpine.DEB.2.02.1210261857100.20029@mirri/
https://lore.kernel.org/linux-arm-kernel/ad9478410910120746g2ce82af1t71a84ea02e9eecb7@mail.gmail.com/
https://lore.kernel.org/ath9k-devel/4FF0EBCE.3020308@allnet.de/

And that is not all. It looks like that this issue with Root Port /
Memory Controller was known also for kernel developers. In past, about
10 years ago, into kernel was merged following commit which explained it
and fixed class code from Memory Controller to PCI Bridge:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1dc831bf53fddcc6443f74a39e72db5bcea4f15d

Apparently this patch completely disappeared as I'm not able find any
code with this comment or fixup in mainline kernel anymore.
Bjorn, Lorenzo, do you have any idea what happened?

Re: [EXT] Re: pci mvebu issue (memory controller)

[Jason Gunthorpe]

On Thu, Mar 04, 2021 at 12:29:08PM -0600, Bjorn Helgaas wrote:

> That's not much to go on.  Someone with more knowledge of the actual
> problem would have to weigh in on whether hiding a device is the best
> approach.

Since Pali asked..

The issue with this HW is the IP designers took an end port PCIe core
and glued it up to act as a root port without changing anything. This
is why it doesn't present a bridge config space. It is a *completely*
non compliant design.

The pci-mvebu host bridge driver is designged to fix this. It provides
a compliant PCI register view for a root port device using SW to
inspect config space operations and remaps the config space accesses
to their non-compliant positions within the SOC.

Hoping that the PCI core can directly drive this PCI device as a root
port without the above driver is just an endless sea of hacks.

Jason

pci mvebu issue (memory controller)

[Marek Behún]

Hello Thomas,

we have enountered an issue with pci-mvebu driver and would like your
opinion, since you are the author of commit
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f4ac99011e542d06ea2bda10063502583c6d7991

After upgrading to new version of U-Boot on a Armada XP / 38x device,
some WiFi cards stopped working in kernel. Ath10k driver, for example,
could not load firmware into the card.

We discovered that the issue is caused by U-Boot:
- when U-Boot's pci_mvebu driver was converted to driver model API,
  U-Boot started to configure PCIe registers not only for the newtork
  adapter, but also for the Marvell Memory Controller (that you are
  mentioning in your commit).
- Since pci-mvebu driver in Linux is ignoring the Marvell Memory
  Controller device, and U-Boot configures its registers (BARs and what
  not), after kernel boots, the registers of this device are
  incompatible with kernel, or something, and this causes problems for
  the real PCIe device.
- Stefan Roese has temporarily solved this issue with U-Boot commit
  https://gitlab.denx.de/u-boot/custodians/u-boot-marvell/-/commit/6a2fa284aee2981be2c7661b3757ce112de8d528
  which basically just masks the Memory Controller's existence.

- in Linux commit f4ac99011e54 ("pci: mvebu: no longer fake the slot
  location of downstream devices") you mention that:

   * On slot 0, a "Marvell Memory controller", identical on all PCIe
     interfaces, and which isn't useful when the Marvell SoC is the PCIe
     root complex (i.e, the normal case when we run Linux on the Marvell
     SoC).

What we are wondering is:
- what does the Marvell Memory controller really do? Can it be used to
  configure something? It clearly does something, because if it is
  configured in U-Boot somehow but not in kernel, problems can occur.
- is the best solution really just to ignore this device?
- should U-Boot also start doing what commit f4ac99011e54 does? I.e.
  to make sure that the real device is in slot 0, and Marvell Memory
  Controller in slot 1.
- why is Linux ignoring this device? It isn't even listed in lspci
  output.

Thanks,

Marek