Re: PCIe AER generates no interrupts on host (ZynqMP)

[Stefan Roese <sr@denx.de>]

On 1/7/22 11:04, Pali Rohár wrote:
> Hello! You asked me in another email for comments to this email, so I'm
> replying directly to this email...

Thanks a lot for you input here. Please find some comments below...

> On Tuesday 04 January 2022 10:02:18 Stefan Roese wrote:
>> Hi,
>>
>> I'm trying to get the Kernel PCIe AER infrastructure to work on my
>> ZynqMP based system. E.g. handle the events (correctable, uncorrectable
>> etc). In my current tests, no AER interrupt is generated though. I'm
>> currently using the "surprise down error status" in the uncorrectable
>> error status register of the connected PCIe switch (PLX / Broadcom
>> PEX8718). Here the bit is correctly logged in the PEX switch
>> uncorrectable error status register but no interrupt is generated
>> to the root-port / system. And hence no AER message(s) reported.
>>
>> Does any one of you have some ideas on what might be missing? Why are
>> these events not reported to the PCIe rootport driver via IRQ? Might
>> this be a problem of the missing MSI-X support of the ZynqMP? The AER
>> interrupt is connected as legacy IRQ:
>>
>> cat /proc/interrupts | grep -i aer
>>   58:          0          0          0          0  nwl_pcie:legacy   0 Level
>> PCIe PME, aerdrv
> 
> Error events (correctable, non-fatal and fatal) are reported by PCIe
> devices to the Root Complex via PCIe error messages (Message code of TLP
> is set to Error Message) and not via interrupts. Root Port is then
> responsible to "convert" these PCIe error messages to MSI(X) interrupt
> and report it to the system. According to PCIe spec, AER is supported
> only via MSI(X) interrupts, not legacy INTx.

This seems not correct. From the ML link reported by Bjorn, there seems
to be a platform specific interrupt on ZynqMP, to report the AER events.
At least this is how I interpret the patch from Bharat from that time.
In the meantime Bharat from Xilinx has sent me a link to a newer,
updated patch series to use this "misc" interrupts for AER instead. I'll
get into more details on this in another reply.

> Via Bridge Control register (SERR# enable bit) on the Root Port it is
> possible to enable / disable reporting of these errors from PCIe devices
> on the other end of PCIe link to the system.

Here the BridgeCtl of the Root Port:
   BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-

> Then via Command register
> (SERR# enable bit) and Device Control register it is possible to enable
> / disable reporting of all errors (from Root Port and also devices on
> other end of the link).

The command registers have SERR disabled on all PCIe devices:
   Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-

Not sure if this is a problem. I would expect the Kernel PCI subsystem
and the AER driver to enable the necessary bits. So is 'SERR-' here
a problem?

Device Control has the error reporting enabled:
   DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+

> And via AER registers on the Root Port it is
> also possible to disable generating MSI(X) interrupts when error is
> reported.

Here is what the Root Port reports:
         Capabilities: [140 v1] Advanced Error Reporting
                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
AdvNonFatalErr-
                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
AdvNonFatalErr+
                 AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- 
ECRCChkCap+ ECRCChkEn-
                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                 HeaderLog: 00000000 00000000 00000000 00000000
                 RootCmd: CERptEn+ NFERptEn+ FERptEn+
                 RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
                          FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0

Mask (..Msk) is set to "-" which is correct for not masking this error
I assume.

> And IIRC via PCIe Downstream Port Containment there is also
> way how to "mask" reporting of error events. But I do not have PCIe
> devices with DPC support, so I have not played with it yet.

Me neither.

> So there are
> many places where error event can be stopped. But important is that
> kernel AER driver should correctly enable all required bits to start
> receiving MSI(X) interrupts for error events.

Agreed.

> On other devices I'm seeing following issues... Root Ports are not
> compliant to PCIe spec and do not implement error reporting at all. Or
> they do not implement those enable/disable bits correctly. Or they do
> not implement proper support for extended PCIe config space for Root
> Port (AER is in extended space). Or they report error events via custom
> proprietary interrupts and not via MSI(X) as required by PCIe spec. This
> is the case for (all?) Marvell PCIe controllers and I saw here on
> linux-pci list that it applies also for PCIe controllers from some other
> vendors. Also drivers for Marvell PCIe controllers requires additional
> code to access extended PCIe config space of Root Port (accessing config
> space of PCIe devices on the other end of PCIe link is working fine).
> 
> So the first suspicious thing is why kernel AER driver is using legacy
> shared INTx interrupt as in most cases Root Port would not report any
> error event via INTx. And the second thing, try to look into
> documentation for used PCIe controller, just in case if vendor
> "invented" some proprietary and non-compliant way how to report error /
> AER events to OS...

At least this seems to be the case. Even though I'm still not receiving
the surprise down error status with additional change. More to this in
the other mail.

> I saw more issues with PCIe controllers as with PCIe switches so in my
> opinion issue would be either in controller driver or controller hw
> itself. And if you see event status logged in PCIe switch register I
> would expect that switch correctly sent PCIe Error message to Root
> Complex.

That is my understanding and best guess right now as well.

Thanks,
Stefan