Advice on delaying de-asserting PCIe reset

Advice on delaying de-asserting PCIe reset

[sainath grandhi]

Hello,
We are potentially facing a scenario where de-asserting the PCIe
PERST# should wait until an endpoint in the PCI hierarchy is ready.
Since the endpoint of interest is an FPGA, it takes "some" time to
come out of reset, boot etc. to be ready and participate in Link
training followed by config space requests from Linux.

So we are checking for options on how we can delay de-asserting PERST#
in the Linux PCIe controller driver, if possible in a standard way.

A simple approach would be to add some time delay or wait for a signal
(via some pin) from the endpoint in the PCIe controller driver before
de-asserting PERST#.
But that would make the change specific to our use-case in an
otherwise generic board controller driver. And maintaining that logic
can become cumbersome.

How does Linux in general support such PCI endpoints to work fine?
Any advice on how to approach this scenario is appreciated.

Thanks

Re: Advice on delaying de-asserting PCIe reset

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

On Tue, May 11, 2021 at 12:50:34PM -0700, sainath grandhi wrote:
> We are potentially facing a scenario where de-asserting the PCIe
> PERST# should wait until an endpoint in the PCI hierarchy is ready.
> Since the endpoint of interest is an FPGA, it takes "some" time to
> come out of reset, boot etc. to be ready and participate in Link
> training followed by config space requests from Linux.

I've worked on devices like this but they are a violation of the PCIe
spec.  You have something like 10ms or 100ms by PCIe standards for your
device to come out of reset.

The cases where I've dealt with this we hacked the BIOS to just delay
after the PERST# but before probing.

> So we are checking for options on how we can delay de-asserting PERST#
> in the Linux PCIe controller driver, if possible in a standard way.
> 
> A simple approach would be to add some time delay or wait for a signal
> (via some pin) from the endpoint in the PCIe controller driver before
> de-asserting PERST#.
> But that would make the change specific to our use-case in an
> otherwise generic board controller driver. And maintaining that logic
> can become cumbersome.
> 
> How does Linux in general support such PCI endpoints to work fine?
> Any advice on how to approach this scenario is appreciated.

Are you asking about Linux on the BMC or Linux on the managed host?
I'm trying to figure out how your questions are related to OpenBMC.

One possibility, if you're talking about a PCIe device attached to
the managed host, would be to separate the power sequencing of the PCIe
card from the host processors.  You can bring up power to the PCIe card
independent from the host processors to give it time to come up and
be ready to listen to PERST#.  That is another option if you can't
modify the BIOS.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Advice on delaying de-asserting PCIe reset

[sainath grandhi]

On Tue, May 11, 2021 at 2:46 PM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> On Tue, May 11, 2021 at 12:50:34PM -0700, sainath grandhi wrote:
> > We are potentially facing a scenario where de-asserting the PCIe
> > PERST# should wait until an endpoint in the PCI hierarchy is ready.
> > Since the endpoint of interest is an FPGA, it takes "some" time to
> > come out of reset, boot etc. to be ready and participate in Link
> > training followed by config space requests from Linux.
>
> I've worked on devices like this but they are a violation of the PCIe
> spec.  You have something like 10ms or 100ms by PCIe standards for your
> device to come out of reset.
>
> The cases where I've dealt with this we hacked the BIOS to just delay
> after the PERST# but before probing.
Got it.
When you say BIOS, do you refer to u-boot?
>
> > So we are checking for options on how we can delay de-asserting PERST#
> > in the Linux PCIe controller driver, if possible in a standard way.
> >
> > A simple approach would be to add some time delay or wait for a signal
> > (via some pin) from the endpoint in the PCIe controller driver before
> > de-asserting PERST#.
> > But that would make the change specific to our use-case in an
> > otherwise generic board controller driver. And maintaining that logic
> > can become cumbersome.
> >
> > How does Linux in general support such PCI endpoints to work fine?
> > Any advice on how to approach this scenario is appreciated.
>
> Are you asking about Linux on the BMC or Linux on the managed host?
> I'm trying to figure out how your questions are related to OpenBMC.
>
I am asking about Linux on BMC.

> One possibility, if you're talking about a PCIe device attached to
> the managed host, would be to separate the power sequencing of the PCIe
> card from the host processors.  You can bring up power to the PCIe card
> independent from the host processors to give it time to come up and
> be ready to listen to PERST#.  That is another option if you can't
> modify the BIOS.
>
> --
> Patrick Williams

Re: Advice on delaying de-asserting PCIe reset

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2703 bytes --]

On Wed, May 12, 2021 at 06:36:28AM -0700, sainath grandhi wrote:
> On Tue, May 11, 2021 at 2:46 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> >
> > On Tue, May 11, 2021 at 12:50:34PM -0700, sainath grandhi wrote:
> > > We are potentially facing a scenario where de-asserting the PCIe
> > > PERST# should wait until an endpoint in the PCI hierarchy is ready.
> > > Since the endpoint of interest is an FPGA, it takes "some" time to
> > > come out of reset, boot etc. to be ready and participate in Link
> > > training followed by config space requests from Linux.
> >
> > I've worked on devices like this but they are a violation of the PCIe
> > spec.  You have something like 10ms or 100ms by PCIe standards for your
> > device to come out of reset.
> >
> > The cases where I've dealt with this we hacked the BIOS to just delay
> > after the PERST# but before probing.
> Got it.
> When you say BIOS, do you refer to u-boot?

No, this is just my confusion and invalid assumption on what you're
trying to do.

I assumed you were trying to deal with a custom PCIe device attached to
your host processors and trying to sequence the power and PERST# via the
BMC, not that you were attaching this device directly to the BMC.

> >
> > > So we are checking for options on how we can delay de-asserting PERST#
> > > in the Linux PCIe controller driver, if possible in a standard way.
> > >
> > > A simple approach would be to add some time delay or wait for a signal
> > > (via some pin) from the endpoint in the PCIe controller driver before
> > > de-asserting PERST#.
> > > But that would make the change specific to our use-case in an
> > > otherwise generic board controller driver. And maintaining that logic
> > > can become cumbersome.
> > >
> > > How does Linux in general support such PCI endpoints to work fine?
> > > Any advice on how to approach this scenario is appreciated.
> >
> > Are you asking about Linux on the BMC or Linux on the managed host?
> > I'm trying to figure out how your questions are related to OpenBMC.
> >
> I am asking about Linux on BMC.

I don't think many people have PCIe devices attached to the BMC and
especially not custom PCIe devices like what you're describing.

> > One possibility, if you're talking about a PCIe device attached to
> > the managed host, would be to separate the power sequencing of the PCIe
> > card from the host processors.  You can bring up power to the PCIe card
> > independent from the host processors to give it time to come up and
> > be ready to listen to PERST#.  That is another option if you can't
> > modify the BIOS.
> >
> > --
> > Patrick Williams

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]