Hotswap with an ATCA PCI Express carrier containing a PCIe Switch

Hotswap with an ATCA PCI Express carrier containing a PCIe Switch

[Paulo Fortuna Carvalho]

Hello,
Im trying to remove and insert a carrier PCI Express card that
contains an endpoint card from an ATCA system. The PCI express card
that contains the PCIe Swittch (PEX 8696 from PLX) is attached to an
OSS (One Stop System) card that contains the Root Complex (PEX 8733
from Plx) via a PCI Express cable. The OSS card is a x16 PCI Express
card placed in the PCIE motherboard slot from a computer running
Redhat 6.3 with real time kernel version 3.0.9.rt.
When I hotplug, remove or insert, the endpoint connected to the
carrier card everything goes well with the removal and insertion
procedures (also the device drivers are unloaded and loaded from the
system when I run "remove" and "rescan" commands through an hotplug
developed service for this purpose).

Problems arrive when I try to hotplug (hotswap to be precise) the
carrier card containing PEX 8696 (PCIe Switch). I can perform the
operation once by removing and inserting the PCI Express cable
connected between carrier and OSS cards (and consequently unloading
and loading the corresponding carrier device drivers). When performing
the same operation for the second time I can remove the PCI express
cable and unload carrier device drivers again but the insert procedure
(and device drivers load operation) no longer occur because PCI
Express upstream (from PEX 8733, the root complex point of view) link
does not seems to happen (signalled by a LED indicator placed in the
carrier card).

There was a test in which I could manage to perform the above
operation 3 times and in the 4th insertion the same happened (but in
this test I opened the Plx editor software between each operation of
removal and insertion to capture a snapshot of the PEX 8733 upstream
registers to track and isolate the problem). So I suppose that somehow
when the link negotiation procedure fails the hotswap procedure no
longer occurs.

Also when I run "lspci" command PEX 8696 no longer appears (of course)
and the branch of the entire PCI express tree (path until the FPGA
endpoint) reaches only until the PEX 8733 downstream port.

Can anybody explain what is happening and what can I do to restore the
PCI express link each time I perform the hotswap procedure?
Also, from the registers snapshot I could see that somehow when this
happens there is always TLP packet that is waiting to be delivered
(pending I suppose) but I don't know if this is an hardware or
software issue.

Regards,
Paulo.

Re: Hotswap with an ATCA PCI Express carrier containing a PCIe Switch

[Bjorn Helgaas]

On Wed, May 7, 2014 at 7:09 AM, Paulo Fortuna Carvalho
<pricardofc@ipfn.ist.utl.pt> wrote:
> Hello,
> Im trying to remove and insert a carrier PCI Express card that
> contains an endpoint card from an ATCA system. ...

> Also when I run "lspci" command PEX 8696 no longer appears (of course)
> and the branch of the entire PCI express tree (path until the FPGA
> endpoint) reaches only until the PEX 8733 downstream port.
>
> Can anybody explain what is happening and what can I do to restore the
> PCI express link each time I perform the hotswap procedure?
> Also, from the registers snapshot I could see that somehow when this
> happens there is always TLP packet that is waiting to be delivered
> (pending I suppose) but I don't know if this is an hardware or
> software issue.

Please include complete "lspci -vv" output (with carrier installed)
and a complete dmesg log that includes removal and insertion events.

Bjorn