linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Recovering from AER: Uncorrected (Fatal) error
@ 2020-12-04 12:52 Hinko Kocevar
  2020-12-04 22:38 ` Bjorn Helgaas
  0 siblings, 1 reply; 12+ messages in thread
From: Hinko Kocevar @ 2020-12-04 12:52 UTC (permalink / raw)
  To: linux-pci

Hi,

I'm trying to figure out how to recover from Uncorrected (Fatal) error that is seen by the PCI root on a CPU PCIe controller:

Dec  1 02:16:37 bd-cpu18 kernel: pcieport 0000:00:01.0: AER: Uncorrected (Fatal) error received: id=0008
Dec  1 02:16:37 bd-cpu18 kernel: pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, id=0008(Requester ID)
Dec  1 02:16:37 bd-cpu18 kernel: pcieport 0000:00:01.0:   device [8086:1901] error status/mask=00004000/00000000
Dec  1 02:16:37 bd-cpu18 kernel: pcieport 0000:00:01.0:    [14] Completion Timeout     (First)

This is the complete PCI device tree that I'm working with:

$ sudo /usr/local/bin/pcicrawler -t
00:01.0 root_port, "J6B2", slot 1, device present, speed 8GT/s, width x8
 ├─01:00.0 upstream_port, PLX Technology, Inc. (10b5), device 8725
 │  ├─02:01.0 downstream_port, slot 1, device present, power: Off, speed 8GT/s, width x4
 │  │  └─03:00.0 upstream_port, PLX Technology, Inc. (10b5) PEX 8748 48-Lane, 12-Port PCI Express Gen 3 (8 GT/s) Switch, 27 x 27mm FCBGA (8748)
 │  │     ├─04:00.0 downstream_port, slot 10, power: Off
 │  │     ├─04:01.0 downstream_port, slot 4, device present, power: Off, speed 8GT/s, width x4
 │  │     │  └─06:00.0 endpoint, Research Centre Juelich (1796), device 0024
 │  │     ├─04:02.0 downstream_port, slot 9, power: Off
 │  │     ├─04:03.0 downstream_port, slot 3, device present, power: Off, speed 8GT/s, width x4
 │  │     │  └─08:00.0 endpoint, Research Centre Juelich (1796), device 0024
 │  │     ├─04:08.0 downstream_port, slot 5, device present, power: Off, speed 8GT/s, width x4
 │  │     │  └─09:00.0 endpoint, Research Centre Juelich (1796), device 0024
 │  │     ├─04:09.0 downstream_port, slot 11, power: Off
 │  │     ├─04:0a.0 downstream_port, slot 6, device present, power: Off, speed 8GT/s, width x4
 │  │     │  └─0b:00.0 endpoint, Research Centre Juelich (1796), device 0024
 │  │     ├─04:0b.0 downstream_port, slot 12, power: Off
 │  │     ├─04:10.0 downstream_port, slot 8, power: Off
 │  │     ├─04:11.0 downstream_port, slot 2, device present, power: Off, speed 2.5GT/s, width x1
 │  │     │  └─0e:00.0 endpoint, Xilinx Corporation (10ee), device 7011
 │  │     └─04:12.0 downstream_port, slot 7, power: Off
 │  ├─02:02.0 downstream_port, slot 2
 │  ├─02:08.0 downstream_port, slot 8
 │  ├─02:09.0 downstream_port, slot 9, power: Off
 │  └─02:0a.0 downstream_port, slot 10
 ├─01:00.1 endpoint, PLX Technology, Inc. (10b5), device 87d0
 ├─01:00.2 endpoint, PLX Technology, Inc. (10b5), device 87d0
 ├─01:00.3 endpoint, PLX Technology, Inc. (10b5), device 87d0
 └─01:00.4 endpoint, PLX Technology, Inc. (10b5), device 87d0

00:01.0 is on a CPU board, The 03:00.0 and everything below that is not on a CPU board (working with a micro TCA system here). I'm working with FPGA based devices seen as endpoints here.
After the error all the endpoints in the above list are not able to talk to CPU anymore; register reads return 0xFFFFFFFF. At the same time PCI config space looks sane and accessible for those devices.

How can I debug this further? I'm getting the "I/O to channel is blocked" (pci_channel_io_frozen) state reported to the the endpoint devices I provide driver for.
Is there any way of telling if the PCI switch devices between 00:01.0 ... 06:00.0 have all recovered ; links up and running and similar? Is this information provided by the Linux kernel somehow?

For reference, I've experimented with AER inject and my tests showed that if the same type of error is injected in any other PCI device in the path 01:00.0 ... 06:00.0, IOW not into 00:01.0, the link is recovered successfully, and I can continue working with the endpoint devices. In those cases the "I/O channel is in normal state" (pci_channel_io_normal) state was reported; only error injection into 00:01.0 reports pci_channel_io_frozen state. Recovery code in the endpoint drivers I maintain is just calling the pci_cleanup_aer_uncorrect_error_status() in error handler resume() callback.

FYI, this is on 3.10.0-1160.6.1.el7.x86_64.debug CentOS7 kernel which I believe is quite old. At the moment I can not use newer kernel, but would be prepared to take that step if told that it would help.

Thank you in advance,

Hinko

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2020-12-15 18:57 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-04 12:52 Recovering from AER: Uncorrected (Fatal) error Hinko Kocevar
2020-12-04 22:38 ` Bjorn Helgaas
2020-12-09 10:02   ` Hinko Kocevar
2020-12-09 17:40     ` Bjorn Helgaas
2020-12-09 20:31       ` Hinko Kocevar
2020-12-09 20:50       ` Hinko Kocevar
2020-12-09 21:32         ` Bjorn Helgaas
2020-12-09 22:55           ` Hinko Kocevar
2020-12-10 12:56             ` Hinko Kocevar
2020-12-14 21:23             ` Keith Busch
2020-12-15 12:56               ` Hinko Kocevar
2020-12-15 18:56                 ` Keith Busch

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).