Re: PCIe AER generates no interrupts on host (ZynqMP)

["Pali Rohár" <pali@kernel.org>]

Hello! You asked me in another email for comments to this email, so I'm
replying directly to this email...

On Tuesday 04 January 2022 10:02:18 Stefan Roese wrote:
> Hi,
> 
> I'm trying to get the Kernel PCIe AER infrastructure to work on my
> ZynqMP based system. E.g. handle the events (correctable, uncorrectable
> etc). In my current tests, no AER interrupt is generated though. I'm
> currently using the "surprise down error status" in the uncorrectable
> error status register of the connected PCIe switch (PLX / Broadcom
> PEX8718). Here the bit is correctly logged in the PEX switch
> uncorrectable error status register but no interrupt is generated
> to the root-port / system. And hence no AER message(s) reported.
> 
> Does any one of you have some ideas on what might be missing? Why are
> these events not reported to the PCIe rootport driver via IRQ? Might
> this be a problem of the missing MSI-X support of the ZynqMP? The AER
> interrupt is connected as legacy IRQ:
> 
> cat /proc/interrupts | grep -i aer
>  58:          0          0          0          0  nwl_pcie:legacy   0 Level
> PCIe PME, aerdrv

Error events (correctable, non-fatal and fatal) are reported by PCIe
devices to the Root Complex via PCIe error messages (Message code of TLP
is set to Error Message) and not via interrupts. Root Port is then
responsible to "convert" these PCIe error messages to MSI(X) interrupt
and report it to the system. According to PCIe spec, AER is supported
only via MSI(X) interrupts, not legacy INTx.

Via Bridge Control register (SERR# enable bit) on the Root Port it is
possible to enable / disable reporting of these errors from PCIe devices
on the other end of PCIe link to the system. Then via Command register
(SERR# enable bit) and Device Control register it is possible to enable
/ disable reporting of all errors (from Root Port and also devices on
other end of the link). And via AER registers on the Root Port it is
also possible to disable generating MSI(X) interrupts when error is
reported. And IIRC via PCIe Downstream Port Containment there is also
way how to "mask" reporting of error events. But I do not have PCIe
devices with DPC support, so I have not played with it yet. So there are
many places where error event can be stopped. But important is that
kernel AER driver should correctly enable all required bits to start
receiving MSI(X) interrupts for error events.

On other devices I'm seeing following issues... Root Ports are not
compliant to PCIe spec and do not implement error reporting at all. Or
they do not implement those enable/disable bits correctly. Or they do
not implement proper support for extended PCIe config space for Root
Port (AER is in extended space). Or they report error events via custom
proprietary interrupts and not via MSI(X) as required by PCIe spec. This
is the case for (all?) Marvell PCIe controllers and I saw here on
linux-pci list that it applies also for PCIe controllers from some other
vendors. Also drivers for Marvell PCIe controllers requires additional
code to access extended PCIe config space of Root Port (accessing config
space of PCIe devices on the other end of PCIe link is working fine).

So the first suspicious thing is why kernel AER driver is using legacy
shared INTx interrupt as in most cases Root Port would not report any
error event via INTx. And the second thing, try to look into
documentation for used PCIe controller, just in case if vendor
"invented" some proprietary and non-compliant way how to report error /
AER events to OS...

I saw more issues with PCIe controllers as with PCIe switches so in my
opinion issue would be either in controller driver or controller hw
itself. And if you see event status logged in PCIe switch register I
would expect that switch correctly sent PCIe Error message to Root
Complex.

> BTW: This was tested on v5.10 and recent v5.16-rc6.
> 
> Thanks,
> Stefan