how to get pci config space

how to get pci config space

[Brad Bishop]

Is anyone already sending information from the PCI config space down to the BMC?  Does anyone think exposing that kind of information via the BMC would be an interesting OpenBMC feature?  How would you do it?  PLDM?

thanks!

-brad

RE: [EXTERNAL] how to get pci config space

[Neeraj Ladkani]

Is there any broader use case for this?  I am not sure on Power but on x86 it really depends on how the CPU PECI is router to BMC. If its wired to ME then BMC->ME->CPU( via PECI over DMI or serial PECI) but if its wired to CPU then BMC->CPU( via serial PECI). 

Neeraj

-----Original Message-----
From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> On Behalf Of Brad Bishop
Sent: Tuesday, December 17, 2019 8:38 AM
To: OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: [EXTERNAL] how to get pci config space

Is anyone already sending information from the PCI config space down to the BMC?  Does anyone think exposing that kind of information via the BMC would be an interesting OpenBMC feature?  How would you do it?  PLDM?

thanks!

-brad

Re: [EXTERNAL] how to get pci config space

[Brad Bishop]

Thanks for the reply Neeraj!

> On Dec 17, 2019, at 4:15 PM, Neeraj Ladkani <neladk@microsoft.com> wrote:
> 
> Is there any broader use case for this?

I want to report mfg/model/pn/sn for pcie devices via the pciedevice Redfish schema.  I assumed the config space is where this information would originate.  Good assumption?

I’m also aware of a need to dynamically tune fan control parameters when cards that are known to run hot are installed.  Are there better ways to do something like this?

> I am not sure on Power

Similar to PECI we have something called FSI.

> but on x86 it really depends on how the CPU PECI is router to BMC. If its wired to ME then BMC->ME->CPU( via PECI over DMI or serial PECI) but if its wired to CPU then BMC->CPU( via serial PECI). 

This helps, thanks.  If we assume the second case for a moment - is the entire path in hardware? - e.g. does PECI give the BMC direct access to the address space and the BMC just reads it?  Or is there a message passing interface?

thx - brad

RE: [EXTERNAL] how to get pci config space

[Neeraj Ladkani]

IMO, we only need DeviceID and VendorID fields from PCIe Config space and we can let system firmware send this information during boot or preprogram the information to BMC using EntityManager. 

Regarding BMC-CPU(via PECI), BMC needs to send PECI command to CPU. CPU should support RdPCICfg as supported PECI command and thus respond with data.

Neeraj


-----Original Message-----
From: Brad Bishop <bradleyb@fuzziesquirrel.com> 
Sent: Tuesday, December 17, 2019 2:51 PM
To: Neeraj Ladkani <neladk@microsoft.com>
Cc: OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: Re: [EXTERNAL] how to get pci config space

Thanks for the reply Neeraj!

> On Dec 17, 2019, at 4:15 PM, Neeraj Ladkani <neladk@microsoft.com> wrote:
> 
> Is there any broader use case for this?

I want to report mfg/model/pn/sn for pcie devices via the pciedevice Redfish schema.  I assumed the config space is where this information would originate.  Good assumption?

I’m also aware of a need to dynamically tune fan control parameters when cards that are known to run hot are installed.  Are there better ways to do something like this?

> I am not sure on Power

Similar to PECI we have something called FSI.

> but on x86 it really depends on how the CPU PECI is router to BMC. If its wired to ME then BMC->ME->CPU( via PECI over DMI or serial PECI) but if its wired to CPU then BMC->CPU( via serial PECI). 

This helps, thanks.  If we assume the second case for a moment - is the entire path in hardware? - e.g. does PECI give the BMC direct access to the address space and the BMC just reads it?  Or is there a message passing interface?

thx - brad

Re: [EXTERNAL] how to get pci config space

[Brad Bishop]

Thanks Neeraj

> On Dec 18, 2019, at 2:18 AM, Neeraj Ladkani <neladk@microsoft.com> wrote:
> 
> IMO, we only need DeviceID and VendorID fields from PCIe Config space

This would probably meet my need to dynamically tune fan control parameters.  Is it possible to populate instances of the pciedevice schema based on devid and vendorid alone?

> and we can let system firmware send this information during boot

This is how it works on Power systems before OpenBMC, but we have a custom protocol with a proprietary implementation.  The purpose of my note was to find out if there are typical implementations or even standards out there for doing this.

> or preprogram the information to BMC using EntityManager. 

Can you elaborate on how this would work?  Given the number of pcie devices out there this seems like it would be hard to do it this way without a huge database of some kind on the bmc?

> Regarding BMC-CPU(via PECI), BMC needs to send PECI command to CPU. CPU should support RdPCICfg as supported PECI command and thus respond with data.

Ok - that sounds like its all in hardware.  But above it sounded like you suggest we skip RdPCICfg and instead let system firmware push this information down to the BMC - do I have it right?  If so why do you prefer that mechanism?

thx!

-brad

RE: [EXTERNAL] how to get pci config space

[Neeraj Ladkani]

It depends on requirements like in our use case, our PCIe devices are fixed so we can preprogram a array in JSON file to include all PCI functions from a device but for someone else for example OEM who is selling the servers, it not possible to predict all PCI device can be connected on PCIe slot so we can let system firmware send this information or do RdPCIRd via PECI. 

I am not aware of any standards on "Implementation". I have seen typical implementations where system firmware sends post PCIe data ( exact schema) to BMC using redfish and BMC produces this data over redfish ( just act like passthrough). 

Neeraj

-----Original Message-----
From: Brad Bishop <bradleyb@fuzziesquirrel.com> 
Sent: Wednesday, December 18, 2019 4:35 AM
To: Neeraj Ladkani <neladk@microsoft.com>
Cc: OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: Re: [EXTERNAL] how to get pci config space

Thanks Neeraj

> On Dec 18, 2019, at 2:18 AM, Neeraj Ladkani <neladk@microsoft.com> wrote:
> 
> IMO, we only need DeviceID and VendorID fields from PCIe Config space

This would probably meet my need to dynamically tune fan control parameters.  Is it possible to populate instances of the pciedevice schema based on devid and vendorid alone?

> and we can let system firmware send this information during boot

This is how it works on Power systems before OpenBMC, but we have a custom protocol with a proprietary implementation.  The purpose of my note was to find out if there are typical implementations or even standards out there for doing this.

> or preprogram the information to BMC using EntityManager. 

Can you elaborate on how this would work?  Given the number of pcie devices out there this seems like it would be hard to do it this way without a huge database of some kind on the bmc?

> Regarding BMC-CPU(via PECI), BMC needs to send PECI command to CPU. CPU should support RdPCICfg as supported PECI command and thus respond with data.

Ok - that sounds like its all in hardware.  But above it sounded like you suggest we skip RdPCICfg and instead let system firmware push this information down to the BMC - do I have it right?  If so why do you prefer that mechanism?

thx!

-brad

Re: [EXTERNAL] how to get pci config space

[Bills, Jason M]

On 12/19/2019 12:45 AM, Neeraj Ladkani wrote:
> It depends on requirements like in our use case, our PCIe devices are fixed so we can preprogram a array in JSON file to include all PCI functions from a device but for someone else for example OEM who is selling the servers, it not possible to predict all PCI device can be connected on PCIe slot so we can let system firmware send this information or do RdPCIRd via PECI.

Sorry for not replying earlier.  I had some legal questions that I was 
waiting to be resolved.

Intel has a downstream solution that uses PECI to get the PCIe 
information onto D-Bus which is then published to Redfish.  I can now 
share what we have upstream if there is interest.

If so, I guess I'd need a new 'peci-pcie' repo to check into?

Thanks,
-Jason>
> I am not aware of any standards on "Implementation". I have seen typical implementations where system firmware sends post PCIe data ( exact schema) to BMC using redfish and BMC produces this data over redfish ( just act like passthrough).
> 
> Neeraj
> 
> -----Original Message-----
> From: Brad Bishop <bradleyb@fuzziesquirrel.com>
> Sent: Wednesday, December 18, 2019 4:35 AM
> To: Neeraj Ladkani <neladk@microsoft.com>
> Cc: OpenBMC Maillist <openbmc@lists.ozlabs.org>
> Subject: Re: [EXTERNAL] how to get pci config space
> 
> Thanks Neeraj
> 
>> On Dec 18, 2019, at 2:18 AM, Neeraj Ladkani <neladk@microsoft.com> wrote:
>>
>> IMO, we only need DeviceID and VendorID fields from PCIe Config space
> 
> This would probably meet my need to dynamically tune fan control parameters.  Is it possible to populate instances of the pciedevice schema based on devid and vendorid alone?
> 
>> and we can let system firmware send this information during boot
> 
> This is how it works on Power systems before OpenBMC, but we have a custom protocol with a proprietary implementation.  The purpose of my note was to find out if there are typical implementations or even standards out there for doing this.
> 
>> or preprogram the information to BMC using EntityManager.
> 
> Can you elaborate on how this would work?  Given the number of pcie devices out there this seems like it would be hard to do it this way without a huge database of some kind on the bmc?
> 
>> Regarding BMC-CPU(via PECI), BMC needs to send PECI command to CPU. CPU should support RdPCICfg as supported PECI command and thus respond with data.
> 
> Ok - that sounds like its all in hardware.  But above it sounded like you suggest we skip RdPCICfg and instead let system firmware push this information down to the BMC - do I have it right?  If so why do you prefer that mechanism?
> 
> thx!
> 
> -brad
> 

Re: [EXTERNAL] how to get pci config space

[Bills, Jason M]

Hi Brad,

On 4/30/2020 2:14 PM, Bills, Jason M wrote:
> 
> 
> On 12/19/2019 12:45 AM, Neeraj Ladkani wrote:
>> It depends on requirements like in our use case, our PCIe devices are 
>> fixed so we can preprogram a array in JSON file to include all PCI 
>> functions from a device but for someone else for example OEM who is 
>> selling the servers, it not possible to predict all PCI device can be 
>> connected on PCIe slot so we can let system firmware send this 
>> information or do RdPCIRd via PECI.
> 
> Sorry for not replying earlier.  I had some legal questions that I was 
> waiting to be resolved.
> 
> Intel has a downstream solution that uses PECI to get the PCIe 
> information onto D-Bus which is then published to Redfish.  I can now 
> share what we have upstream if there is interest.
> 
> If so, I guess I'd need a new 'peci-pcie' repo to check into?
Not sure if anyone saw this or if there is just no interest. :)

Could you please create a peci-pcie repo for this application?

Thanks!
-Jason

> 
> Thanks,
> -Jason>
>> I am not aware of any standards on "Implementation". I have seen 
>> typical implementations where system firmware sends post PCIe data ( 
>> exact schema) to BMC using redfish and BMC produces this data over 
>> redfish ( just act like passthrough).
>>
>> Neeraj
>>
>> -----Original Message-----
>> From: Brad Bishop <bradleyb@fuzziesquirrel.com>
>> Sent: Wednesday, December 18, 2019 4:35 AM
>> To: Neeraj Ladkani <neladk@microsoft.com>
>> Cc: OpenBMC Maillist <openbmc@lists.ozlabs.org>
>> Subject: Re: [EXTERNAL] how to get pci config space
>>
>> Thanks Neeraj
>>
>>> On Dec 18, 2019, at 2:18 AM, Neeraj Ladkani <neladk@microsoft.com> 
>>> wrote:
>>>
>>> IMO, we only need DeviceID and VendorID fields from PCIe Config space
>>
>> This would probably meet my need to dynamically tune fan control 
>> parameters.  Is it possible to populate instances of the pciedevice 
>> schema based on devid and vendorid alone?
>>
>>> and we can let system firmware send this information during boot
>>
>> This is how it works on Power systems before OpenBMC, but we have a 
>> custom protocol with a proprietary implementation.  The purpose of my 
>> note was to find out if there are typical implementations or even 
>> standards out there for doing this.
>>
>>> or preprogram the information to BMC using EntityManager.
>>
>> Can you elaborate on how this would work?  Given the number of pcie 
>> devices out there this seems like it would be hard to do it this way 
>> without a huge database of some kind on the bmc?
>>
>>> Regarding BMC-CPU(via PECI), BMC needs to send PECI command to CPU. 
>>> CPU should support RdPCICfg as supported PECI command and thus 
>>> respond with data.
>>
>> Ok - that sounds like its all in hardware.  But above it sounded like 
>> you suggest we skip RdPCICfg and instead let system firmware push this 
>> information down to the BMC - do I have it right?  If so why do you 
>> prefer that mechanism?
>>
>> thx!
>>
>> -brad
>>

Re: [EXTERNAL] how to get pci config space

[Richard Hanley]

[-- Attachment #1: Type: text/plain, Size: 3550 bytes --]

Hi Jason,

I guess I missed this conversation when it first started. We have some
interest in getting PCIe metadata forwarded to the BMC. I kind of assumed
that the only way was to use a side-band channel, but I getting it through
PECI would be very interesting.

Nancy should have a better idea what we'd be looking for.

- Richard

On Wed, Jun 17, 2020 at 4:38 PM Bills, Jason M <
jason.m.bills@linux.intel.com> wrote:

> Hi Brad,
>
> On 4/30/2020 2:14 PM, Bills, Jason M wrote:
> >
> >
> > On 12/19/2019 12:45 AM, Neeraj Ladkani wrote:
> >> It depends on requirements like in our use case, our PCIe devices are
> >> fixed so we can preprogram a array in JSON file to include all PCI
> >> functions from a device but for someone else for example OEM who is
> >> selling the servers, it not possible to predict all PCI device can be
> >> connected on PCIe slot so we can let system firmware send this
> >> information or do RdPCIRd via PECI.
> >
> > Sorry for not replying earlier.  I had some legal questions that I was
> > waiting to be resolved.
> >
> > Intel has a downstream solution that uses PECI to get the PCIe
> > information onto D-Bus which is then published to Redfish.  I can now
> > share what we have upstream if there is interest.
> >
> > If so, I guess I'd need a new 'peci-pcie' repo to check into?
> Not sure if anyone saw this or if there is just no interest. :)
>
> Could you please create a peci-pcie repo for this application?
>
> Thanks!
> -Jason
>
> >
> > Thanks,
> > -Jason>
> >> I am not aware of any standards on "Implementation". I have seen
> >> typical implementations where system firmware sends post PCIe data (
> >> exact schema) to BMC using redfish and BMC produces this data over
> >> redfish ( just act like passthrough).
> >>
> >> Neeraj
> >>
> >> -----Original Message-----
> >> From: Brad Bishop <bradleyb@fuzziesquirrel.com>
> >> Sent: Wednesday, December 18, 2019 4:35 AM
> >> To: Neeraj Ladkani <neladk@microsoft.com>
> >> Cc: OpenBMC Maillist <openbmc@lists.ozlabs.org>
> >> Subject: Re: [EXTERNAL] how to get pci config space
> >>
> >> Thanks Neeraj
> >>
> >>> On Dec 18, 2019, at 2:18 AM, Neeraj Ladkani <neladk@microsoft.com>
> >>> wrote:
> >>>
> >>> IMO, we only need DeviceID and VendorID fields from PCIe Config space
> >>
> >> This would probably meet my need to dynamically tune fan control
> >> parameters.  Is it possible to populate instances of the pciedevice
> >> schema based on devid and vendorid alone?
> >>
> >>> and we can let system firmware send this information during boot
> >>
> >> This is how it works on Power systems before OpenBMC, but we have a
> >> custom protocol with a proprietary implementation.  The purpose of my
> >> note was to find out if there are typical implementations or even
> >> standards out there for doing this.
> >>
> >>> or preprogram the information to BMC using EntityManager.
> >>
> >> Can you elaborate on how this would work?  Given the number of pcie
> >> devices out there this seems like it would be hard to do it this way
> >> without a huge database of some kind on the bmc?
> >>
> >>> Regarding BMC-CPU(via PECI), BMC needs to send PECI command to CPU.
> >>> CPU should support RdPCICfg as supported PECI command and thus
> >>> respond with data.
> >>
> >> Ok - that sounds like its all in hardware.  But above it sounded like
> >> you suggest we skip RdPCICfg and instead let system firmware push this
> >> information down to the BMC - do I have it right?  If so why do you
> >> prefer that mechanism?
> >>
> >> thx!
> >>
> >> -brad
> >>
>

[-- Attachment #2: Type: text/html, Size: 4908 bytes --]

Re: [EXTERNAL] how to get pci config space

[Brad Bishop]

On Wed, 2020-06-17 at 16:37 -0700, Bills, Jason M wrote:
> Hi Brad,
> 
> Not sure if anyone saw this or if there is just no interest. :)
> 
> Could you please create a peci-pcie repo for this application?

Hi Jason.  peci-pcie created!

thx - brad

Re: [EXTERNAL] how to get pci config space

[Nancy Yuen]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

Oops, I followed this thread a while back, but I lost track of it.

Jason, I have some use cases for getting PCIe information from the BMC.
Some systems where the PCIe configuration isn't static on a given system,
they will have different combinations of cards.  If the BMC can detect
which cards are plugged in, then the BMC can take card-specific action.
Thank you for sharing your code!

----------
Nancy


On Mon, Jun 22, 2020 at 8:48 AM Brad Bishop <bradleyb@fuzziesquirrel.com>
wrote:

> On Wed, 2020-06-17 at 16:37 -0700, Bills, Jason M wrote:
> > Hi Brad,
> >
> > Not sure if anyone saw this or if there is just no interest. :)
> >
> > Could you please create a peci-pcie repo for this application?
>
> Hi Jason.  peci-pcie created!
>
> thx - brad
>
>

[-- Attachment #2: Type: text/html, Size: 1248 bytes --]