Propose a new application for reading DIMM SPD directly

Propose a new application for reading DIMM SPD directly

[Michael Shen]

Hi Openbmc,

We would like to propose an application that directly reads the DIMM
SPD over HW interface(mostly I2C/I3C).

If I understand correctly, the main method for obtaining the SPD
information in openbmc is from SMBIOS which is prepared by BIOS. And
We are exploring another way that excludes the involvement of BIOS.

The architecture of this application will be similar to the
openbmc/smbios-mdr (the dimm part). The main difference will be the
data source (changed from SMBIOS to SPD).

Currently the code is still being implemented, and we plan to support
DDR5 SPD first, then expand to other DDR generation(if needed).

If this proposal looks good to you, please help to create a repo for
this application.

Best,
Michael

Re: Propose a new application for reading DIMM SPD directly

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1987 bytes --]

On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
> Hi Openbmc,
> 
> We would like to propose an application that directly reads the DIMM
> SPD over HW interface(mostly I2C/I3C).

Who is "we"?

> If I understand correctly, the main method for obtaining the SPD
> information in openbmc is from SMBIOS which is prepared by BIOS. And
> We are exploring another way that excludes the involvement of BIOS.

Unless you're proposing that the BIOS itself comes to the BMC to get the SPD
data, I'm somewhat surprised you could come up with a hardware design to make
this work.  Due to the number of DIMM channels and the limited number of CS pins
on JEDEC DIMMs, you're going to have muxes on the bus somewhere.  Mixing muxes
and multi-master access is pretty problematic.  Either the BIOS and BMC are
fighting over the mux, which is going to mess with the mux driver's view of the
world, or you've got one mux for each in which case you're switching masters
onto a bus, which violates a few i2c design rules.

> The architecture of this application will be similar to the
> openbmc/smbios-mdr (the dimm part). The main difference will be the
> data source (changed from SMBIOS to SPD).
> 
> Currently the code is still being implemented, and we plan to support
> DDR5 SPD first, then expand to other DDR generation(if needed).

Hopefully you're leveraging the existing kernel drivers for reading SPD EEPROMs.
This creates you a sysfs file containing the whole of the EEPROM content.  You'd
just need to write a parser for JEDEC SPD format data.

You should take a look at what is already existing in fru-device (part of
entity-manager repository).  This is already doing this for IPMI-format EEPROM
data.  We should be able to replicate/enhance this code, in the same repository,
to handle SPD format.

> If this proposal looks good to you, please help to create a repo for
> this application.
> 
> Best,
> Michael

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Propose a new application for reading DIMM SPD directly

[Michael Shen]

On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
> > Hi Openbmc,
> >
> > We would like to propose an application that directly reads the DIMM
> > SPD over HW interface(mostly I2C/I3C).
>
> Who is "we"?

Google BMC team. Benjamin Fair is our techlead for this project. Just CC him.

>
> > If I understand correctly, the main method for obtaining the SPD
> > information in openbmc is from SMBIOS which is prepared by BIOS. And
> > We are exploring another way that excludes the involvement of BIOS.
>
> Unless you're proposing that the BIOS itself comes to the BMC to get the SPD
> data, I'm somewhat surprised you could come up with a hardware design to make
> this work.  Due to the number of DIMM channels and the limited number of CS pins
> on JEDEC DIMMs, you're going to have muxes on the bus somewhere.  Mixing muxes
> and multi-master access is pretty problematic.

Yes, we need an additional MUX for switching the master(between BIOS or BMC).
However, compared to the single-master(BIOS only) design, this MUX is
the only hardware difference.

> Either the BIOS and BMC are
> fighting over the mux, which is going to mess with the mux driver's view of the
> world, or you've got one mux for each in which case you're switching masters
> onto a bus, which violates a few i2c design rules.

BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
From my understanding, BIOS only needs to read SPD during the POST stage.
For the rest of time, BIOS will hand over the SPD bus to BMC.

>
> > The architecture of this application will be similar to the
> > openbmc/smbios-mdr (the dimm part). The main difference will be the
> > data source (changed from SMBIOS to SPD).
> >
> > Currently the code is still being implemented, and we plan to support
> > DDR5 SPD first, then expand to other DDR generation(if needed).
>
> Hopefully you're leveraging the existing kernel drivers for reading SPD EEPROMs.
> This creates you a sysfs file containing the whole of the EEPROM content.  You'd
> just need to write a parser for JEDEC SPD format data.

Yes, the data is coming from the sysfs file. And the SPD parser is one
of the main parts.
The second main part is a D-Dus service that exposes data from SPD to
D-Bus interface.

>
> You should take a look at what is already existing in fru-device (part of
> entity-manager repository).  This is already doing this for IPMI-format EEPROM
> data.  We should be able to replicate/enhance this code, in the same repository,
> to handle SPD format.

I am not sure if it's a good idea to put it into the entity-manager
repo. As you said EM
is designed for IPMI-format EEPROM. Adding another parser into that
repo may violate
EM's design.

Re: Propose a new application for reading DIMM SPD directly

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
> On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> > On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
> > > If I understand correctly, the main method for obtaining the SPD
> > > information in openbmc is from SMBIOS which is prepared by BIOS. And
> > > We are exploring another way that excludes the involvement of BIOS.
> >
> > Unless you're proposing that the BIOS itself comes to the BMC to get the SPD
> > data, I'm somewhat surprised you could come up with a hardware design to make
> > this work.  Due to the number of DIMM channels and the limited number of CS pins
> > on JEDEC DIMMs, you're going to have muxes on the bus somewhere.  Mixing muxes
> > and multi-master access is pretty problematic.
> 
> Yes, we need an additional MUX for switching the master(between BIOS or BMC).
> However, compared to the single-master(BIOS only) design, this MUX is
> the only hardware difference.
> 
> > Either the BIOS and BMC are
> > fighting over the mux, which is going to mess with the mux driver's view of the
> > world, or you've got one mux for each in which case you're switching masters
> > onto a bus, which violates a few i2c design rules.
> 
> BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
> From my understanding, BIOS only needs to read SPD during the POST stage.
> For the rest of time, BIOS will hand over the SPD bus to BMC.

That seems like it might work.  You'll have to deal with the time when the BIOS
has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
be fed to the BMC as an input GPIO so that you can differentiate between "we
don't own the mux" and "all the devices are NAKing us".

> > You should take a look at what is already existing in fru-device (part of
> > entity-manager repository).  This is already doing this for IPMI-format EEPROM
> > data.  We should be able to replicate/enhance this code, in the same repository,
> > to handle SPD format.
> 
> I am not sure if it's a good idea to put it into the entity-manager
> repo. As you said EM
> is designed for IPMI-format EEPROM. Adding another parser into that
> repo may violate
> EM's design.

I'm not sure why it would be an issue.  Hopefully one of the maintainers of that
repo can weigh in.  I wouldn't expect "parsing only IPMI-format EEPROMs" is a
design but just the current state of implementation.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Propose a new application for reading DIMM SPD directly

[Ed Tanous]

On Wed, Feb 9, 2022 at 11:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
> > On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> > > On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
> > > > If I understand correctly, the main method for obtaining the SPD
> > > > information in openbmc is from SMBIOS which is prepared by BIOS. And
> > > > We are exploring another way that excludes the involvement of BIOS.
> > >
> > > Unless you're proposing that the BIOS itself comes to the BMC to get the SPD
> > > data, I'm somewhat surprised you could come up with a hardware design to make
> > > this work.  Due to the number of DIMM channels and the limited number of CS pins
> > > on JEDEC DIMMs, you're going to have muxes on the bus somewhere.  Mixing muxes
> > > and multi-master access is pretty problematic.
> >
> > Yes, we need an additional MUX for switching the master(between BIOS or BMC).
> > However, compared to the single-master(BIOS only) design, this MUX is
> > the only hardware difference.

The other thing to recognize is that in these systems, the bmc
controls the CPU init state machine, so in practice, the BMC can
generally read the JEDEC eeproms long before the host would even want
to access them, and can hold off the host until the read is complete,
which mostly renders this race condition moot (daemon crashes might
still have problems.)

> >
> > > Either the BIOS and BMC are
> > > fighting over the mux, which is going to mess with the mux driver's view of the
> > > world, or you've got one mux for each in which case you're switching masters
> > > onto a bus, which violates a few i2c design rules.
> >
> > BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
> > From my understanding, BIOS only needs to read SPD during the POST stage.
> > For the rest of time, BIOS will hand over the SPD bus to BMC.
>
> That seems like it might work.  You'll have to deal with the time when the BIOS
> has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
> be fed to the BMC as an input GPIO so that you can differentiate between "we
> don't own the mux" and "all the devices are NAKing us".

This seems like a nitty gritty design detail that's best handled in
code when we review it.  I think the important bit here is that there
are paths where this could work without a significant design issue.

>
> > > You should take a look at what is already existing in fru-device (part of
> > > entity-manager repository).  This is already doing this for IPMI-format EEPROM
> > > data.  We should be able to replicate/enhance this code, in the same repository,
> > > to handle SPD format.
> >
> > I am not sure if it's a good idea to put it into the entity-manager
> > repo. As you said EM
> > is designed for IPMI-format EEPROM. Adding another parser into that
> > repo may violate
> > EM's design.
>
> I'm not sure why it would be an issue.  Hopefully one of the maintainers of that
> repo can weigh in.  I wouldn't expect "parsing only IPMI-format EEPROMs" is a
> design but just the current state of implementation.

So long as it can function properly in its current design, i have no
problem with FruDevice adding more parsing types.  In fact, there's
already patchsets out to add Linkedins proprietary fru type to
FruDevice, so in terms of design, Patricks request seems reasonable.



>
> --
> Patrick Williams

Re: Propose a new application for reading DIMM SPD directly

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1544 bytes --]

On Wed, Feb 09, 2022 at 12:20:00PM -0800, Ed Tanous wrote:
> On Wed, Feb 9, 2022 at 11:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> > On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
> > > On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> > > > On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:

> > > BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
> > > From my understanding, BIOS only needs to read SPD during the POST stage.
> > > For the rest of time, BIOS will hand over the SPD bus to BMC.
> >
> > That seems like it might work.  You'll have to deal with the time when the BIOS
> > has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
> > be fed to the BMC as an input GPIO so that you can differentiate between "we
> > don't own the mux" and "all the devices are NAKing us".
> 
> This seems like a nitty gritty design detail that's best handled in
> code when we review it.  I think the important bit here is that there
> are paths where this could work without a significant design issue.

Just one subtlety.  I wouldn't expect this, necessarily, to be in _our_ design
and/or code, except that we'd want to document the GPIO line like we do all
others.  I was trying to hint that "if I were involved in this hardware design,
I'd ask for...".  If you leave it out, I'm sure it'll work _most_ of the time
just fine and it'll be your problem to debug it when it doesn't.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Propose a new application for reading DIMM SPD directly

[Ed Tanous]

On Wed, Feb 9, 2022 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> On Wed, Feb 09, 2022 at 12:20:00PM -0800, Ed Tanous wrote:
> > On Wed, Feb 9, 2022 at 11:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> > > On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
> > > > On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> > > > > On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
>
> > > > BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
> > > > From my understanding, BIOS only needs to read SPD during the POST stage.
> > > > For the rest of time, BIOS will hand over the SPD bus to BMC.
> > >
> > > That seems like it might work.  You'll have to deal with the time when the BIOS
> > > has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
> > > be fed to the BMC as an input GPIO so that you can differentiate between "we
> > > don't own the mux" and "all the devices are NAKing us".
> >
> > This seems like a nitty gritty design detail that's best handled in
> > code when we review it.  I think the important bit here is that there
> > are paths where this could work without a significant design issue.
>
> Just one subtlety.  I wouldn't expect this, necessarily, to be in _our_ design
> and/or code, except that we'd want to document the GPIO line like we do all
> others.  I was trying to hint that "if I were involved in this hardware design,
> I'd ask for...".  If you leave it out, I'm sure it'll work _most_ of the time
> just fine and it'll be your problem to debug it when it doesn't.

Understood.

>
> --
> Patrick Williams

Re: Propose a new application for reading DIMM SPD directly

[Michael Shen]

On Thu, Feb 10, 2022 at 6:45 AM Ed Tanous <edtanous@google.com> wrote:
>
> On Wed, Feb 9, 2022 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> >
> > On Wed, Feb 09, 2022 at 12:20:00PM -0800, Ed Tanous wrote:
> > > On Wed, Feb 9, 2022 at 11:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> > > > On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
> > > > > On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> > > > > > On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
> >
> > > > > BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
> > > > > From my understanding, BIOS only needs to read SPD during the POST stage.
> > > > > For the rest of time, BIOS will hand over the SPD bus to BMC.
> > > >
> > > > That seems like it might work.  You'll have to deal with the time when the BIOS
> > > > has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
> > > > be fed to the BMC as an input GPIO so that you can differentiate between "we
> > > > don't own the mux" and "all the devices are NAKing us".
> > >
> > > This seems like a nitty gritty design detail that's best handled in
> > > code when we review it.  I think the important bit here is that there
> > > are paths where this could work without a significant design issue.
> >
> > Just one subtlety.  I wouldn't expect this, necessarily, to be in _our_ design
> > and/or code, except that we'd want to document the GPIO line like we do all
> > others.  I was trying to hint that "if I were involved in this hardware design,
> > I'd ask for...".  If you leave it out, I'm sure it'll work _most_ of the time
> > just fine and it'll be your problem to debug it when it doesn't.
>
> Understood.

Thanks for all your suggestions. I will keep them in mind during implementation.

> > > > You should take a look at what is already existing in fru-device (part of
> > > > entity-manager repository).  This is already doing this for IPMI-format EEPROM
> > > > data.  We should be able to replicate/enhance this code, in the same repository,
> > > > to handle SPD format.
> > >
> > > I am not sure if it's a good idea to put it into the entity-manager
> > > repo. As you said EM
> > > is designed for IPMI-format EEPROM. Adding another parser into that
> > > repo may violate
> > > EM's design.
> >
> > I'm not sure why it would be an issue.  Hopefully one of the maintainers of that
> > repo can weigh in.  I wouldn't expect "parsing only IPMI-format EEPROMs" is a
> > design but just the current state of implementation.
>
> So long as it can function properly in its current design, i have no
> problem with FruDevice adding more parsing types.  In fact, there's
> already patchsets out to add Linkedins proprietary fru type to
> FruDevice, so in terms of design, Patricks request seems reasonable.

Got it. Then I will push the code to EM.

Re: Propose a new application for reading DIMM SPD directly

[Zbigniew, Lukwinski]

On 2/11/2022 1:40 AM, Michael Shen wrote:
> On Thu, Feb 10, 2022 at 6:45 AM Ed Tanous <edtanous@google.com> wrote:
>> On Wed, Feb 9, 2022 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> wrote:
>>> On Wed, Feb 09, 2022 at 12:20:00PM -0800, Ed Tanous wrote:
>>>> On Wed, Feb 9, 2022 at 11:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>>>>> On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
>>>>>> On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
>>>>>>> On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
>>>>>> BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
>>>>>>  From my understanding, BIOS only needs to read SPD during the POST stage.
>>>>>> For the rest of time, BIOS will hand over the SPD bus to BMC.
>>>>> That seems like it might work.  You'll have to deal with the time when the BIOS
>>>>> has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
>>>>> be fed to the BMC as an input GPIO so that you can differentiate between "we
>>>>> don't own the mux" and "all the devices are NAKing us".
>>>> This seems like a nitty gritty design detail that's best handled in
>>>> code when we review it.  I think the important bit here is that there
>>>> are paths where this could work without a significant design issue.
>>> Just one subtlety.  I wouldn't expect this, necessarily, to be in _our_ design
>>> and/or code, except that we'd want to document the GPIO line like we do all
>>> others.  I was trying to hint that "if I were involved in this hardware design,
>>> I'd ask for...".  If you leave it out, I'm sure it'll work _most_ of the time
>>> just fine and it'll be your problem to debug it when it doesn't.
>> Understood.
> Thanks for all your suggestions. I will keep them in mind during implementation.

What about CLTT? Switching MUX to the BMC makes CPU not able to get DIMM 
temperature. Are you assuming here this feature is enabled in BMC FW?

Are you going to support DCPMM as well? If so, there is another problem 
since switching MUX to BMC you brakes NVDIMM related FW/SW running on 
Host OS.

>>>>> You should take a look at what is already existing in fru-device (part of
>>>>> entity-manager repository).  This is already doing this for IPMI-format EEPROM
>>>>> data.  We should be able to replicate/enhance this code, in the same repository,
>>>>> to handle SPD format.
>>>> I am not sure if it's a good idea to put it into the entity-manager
>>>> repo. As you said EM
>>>> is designed for IPMI-format EEPROM. Adding another parser into that
>>>> repo may violate
>>>> EM's design.
>>> I'm not sure why it would be an issue.  Hopefully one of the maintainers of that
>>> repo can weigh in.  I wouldn't expect "parsing only IPMI-format EEPROMs" is a
>>> design but just the current state of implementation.
>> So long as it can function properly in its current design, i have no
>> problem with FruDevice adding more parsing types.  In fact, there's
>> already patchsets out to add Linkedins proprietary fru type to
>> FruDevice, so in terms of design, Patricks request seems reasonable.
> Got it. Then I will push the code to EM.

Re: Propose a new application for reading DIMM SPD directly

[Benjamin Fair]

On Fri, 11 Feb 2022 at 13:21, Zbigniew, Lukwinski
<zbigniew.lukwinski@linux.intel.com> wrote:
>
> On 2/11/2022 1:40 AM, Michael Shen wrote:
> > On Thu, Feb 10, 2022 at 6:45 AM Ed Tanous <edtanous@google.com> wrote:
> >> On Wed, Feb 9, 2022 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> >>> On Wed, Feb 09, 2022 at 12:20:00PM -0800, Ed Tanous wrote:
> >>>> On Wed, Feb 9, 2022 at 11:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> >>>>> On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
> >>>>>> On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> >>>>>>> On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
> >>>>>> BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
> >>>>>>  From my understanding, BIOS only needs to read SPD during the POST stage.
> >>>>>> For the rest of time, BIOS will hand over the SPD bus to BMC.
> >>>>> That seems like it might work.  You'll have to deal with the time when the BIOS
> >>>>> has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
> >>>>> be fed to the BMC as an input GPIO so that you can differentiate between "we
> >>>>> don't own the mux" and "all the devices are NAKing us".
> >>>> This seems like a nitty gritty design detail that's best handled in
> >>>> code when we review it.  I think the important bit here is that there
> >>>> are paths where this could work without a significant design issue.
> >>> Just one subtlety.  I wouldn't expect this, necessarily, to be in _our_ design
> >>> and/or code, except that we'd want to document the GPIO line like we do all
> >>> others.  I was trying to hint that "if I were involved in this hardware design,
> >>> I'd ask for...".  If you leave it out, I'm sure it'll work _most_ of the time
> >>> just fine and it'll be your problem to debug it when it doesn't.
> >> Understood.
> > Thanks for all your suggestions. I will keep them in mind during implementation.
>
> What about CLTT? Switching MUX to the BMC makes CPU not able to get DIMM
> temperature. Are you assuming here this feature is enabled in BMC FW?

BMC could assist with CLTT, but since this is CPU-specific it would
belong in a separate daemon. That daemon could get the relevant
temperatures over D-Bus using the standard sensor interface, so I
don't think it should affect the design for this component.

> Are you going to support DCPMM as well? If so, there is another problem
> since switching MUX to BMC you brakes NVDIMM related FW/SW running on
> Host OS.

There are no plans currently for supporting NVDIMMs, just DDR5 at
first as Michael mentioned, and possibly other DDR versions in the
future.

> >>>>> You should take a look at what is already existing in fru-device (part of
> >>>>> entity-manager repository).  This is already doing this for IPMI-format EEPROM
> >>>>> data.  We should be able to replicate/enhance this code, in the same repository,
> >>>>> to handle SPD format.
> >>>> I am not sure if it's a good idea to put it into the entity-manager
> >>>> repo. As you said EM
> >>>> is designed for IPMI-format EEPROM. Adding another parser into that
> >>>> repo may violate
> >>>> EM's design.
> >>> I'm not sure why it would be an issue.  Hopefully one of the maintainers of that
> >>> repo can weigh in.  I wouldn't expect "parsing only IPMI-format EEPROMs" is a
> >>> design but just the current state of implementation.
> >> So long as it can function properly in its current design, i have no
> >> problem with FruDevice adding more parsing types.  In fact, there's
> >> already patchsets out to add Linkedins proprietary fru type to
> >> FruDevice, so in terms of design, Patricks request seems reasonable.
> > Got it. Then I will push the code to EM.

Re: Propose a new application for reading DIMM SPD directly

[Michael Shen]

> What about CLTT? Switching MUX to the BMC makes CPU not able to get DIMM
> temperature. Are you assuming here this feature is enabled in BMC FW?
CPU can still monitor the DIMM temp by reading the MR4 register and
trigger the DIMM throttling if needed. So I think the CLTT will not be
affected.
If CPU needs something more than MR4 register provided, BMC can assist
it in another separate daemon like Benjamin mentioned.

On Tue, Feb 15, 2022 at 6:17 AM Benjamin Fair <benjaminfair@google.com> wrote:
>
> On Fri, 11 Feb 2022 at 13:21, Zbigniew, Lukwinski
> <zbigniew.lukwinski@linux.intel.com> wrote:
> >
> > On 2/11/2022 1:40 AM, Michael Shen wrote:
> > > On Thu, Feb 10, 2022 at 6:45 AM Ed Tanous <edtanous@google.com> wrote:
> > >> On Wed, Feb 9, 2022 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> > >>> On Wed, Feb 09, 2022 at 12:20:00PM -0800, Ed Tanous wrote:
> > >>>> On Wed, Feb 9, 2022 at 11:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> > >>>>> On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
> > >>>>>> On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> > >>>>>>> On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
> > >>>>>> BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
> > >>>>>>  From my understanding, BIOS only needs to read SPD during the POST stage.
> > >>>>>> For the rest of time, BIOS will hand over the SPD bus to BMC.
> > >>>>> That seems like it might work.  You'll have to deal with the time when the BIOS
> > >>>>> has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
> > >>>>> be fed to the BMC as an input GPIO so that you can differentiate between "we
> > >>>>> don't own the mux" and "all the devices are NAKing us".
> > >>>> This seems like a nitty gritty design detail that's best handled in
> > >>>> code when we review it.  I think the important bit here is that there
> > >>>> are paths where this could work without a significant design issue.
> > >>> Just one subtlety.  I wouldn't expect this, necessarily, to be in _our_ design
> > >>> and/or code, except that we'd want to document the GPIO line like we do all
> > >>> others.  I was trying to hint that "if I were involved in this hardware design,
> > >>> I'd ask for...".  If you leave it out, I'm sure it'll work _most_ of the time
> > >>> just fine and it'll be your problem to debug it when it doesn't.
> > >> Understood.
> > > Thanks for all your suggestions. I will keep them in mind during implementation.
> >
> > What about CLTT? Switching MUX to the BMC makes CPU not able to get DIMM
> > temperature. Are you assuming here this feature is enabled in BMC FW?
>
> BMC could assist with CLTT, but since this is CPU-specific it would
> belong in a separate daemon. That daemon could get the relevant
> temperatures over D-Bus using the standard sensor interface, so I
> don't think it should affect the design for this component.
>
> > Are you going to support DCPMM as well? If so, there is another problem
> > since switching MUX to BMC you brakes NVDIMM related FW/SW running on
> > Host OS.
>
> There are no plans currently for supporting NVDIMMs, just DDR5 at
> first as Michael mentioned, and possibly other DDR versions in the
> future.
>
> > >>>>> You should take a look at what is already existing in fru-device (part of
> > >>>>> entity-manager repository).  This is already doing this for IPMI-format EEPROM
> > >>>>> data.  We should be able to replicate/enhance this code, in the same repository,
> > >>>>> to handle SPD format.
> > >>>> I am not sure if it's a good idea to put it into the entity-manager
> > >>>> repo. As you said EM
> > >>>> is designed for IPMI-format EEPROM. Adding another parser into that
> > >>>> repo may violate
> > >>>> EM's design.
> > >>> I'm not sure why it would be an issue.  Hopefully one of the maintainers of that
> > >>> repo can weigh in.  I wouldn't expect "parsing only IPMI-format EEPROMs" is a
> > >>> design but just the current state of implementation.
> > >> So long as it can function properly in its current design, i have no
> > >> problem with FruDevice adding more parsing types.  In fact, there's
> > >> already patchsets out to add Linkedins proprietary fru type to
> > >> FruDevice, so in terms of design, Patricks request seems reasonable.
> > > Got it. Then I will push the code to EM.

Re: Propose a new application for reading DIMM SPD directly

[Zbigniew, Lukwinski]

On 2/15/2022 2:50 AM, Michael Shen wrote:
>> What about CLTT? Switching MUX to the BMC makes CPU not able to get DIMM
>> temperature. Are you assuming here this feature is enabled in BMC FW?
> CPU can still monitor the DIMM temp by reading the MR4 register and
> trigger the DIMM throttling if needed. So I think the CLTT will not be
> affected.
> If CPU needs something more than MR4 register provided, BMC can assist
> it in another separate daemon like Benjamin mentioned.

Got it.

> On Tue, Feb 15, 2022 at 6:17 AM Benjamin Fair <benjaminfair@google.com> wrote:
>> On Fri, 11 Feb 2022 at 13:21, Zbigniew, Lukwinski
>> <zbigniew.lukwinski@linux.intel.com> wrote:
>>> On 2/11/2022 1:40 AM, Michael Shen wrote:
>>>> On Thu, Feb 10, 2022 at 6:45 AM Ed Tanous <edtanous@google.com> wrote:
>>>>> On Wed, Feb 9, 2022 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> wrote:
>>>>>> On Wed, Feb 09, 2022 at 12:20:00PM -0800, Ed Tanous wrote:
>>>>>>> On Wed, Feb 9, 2022 at 11:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>>>>>>>> On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
>>>>>>>>> On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
>>>>>>>>>> On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
>>>>>>>>> BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
>>>>>>>>>   From my understanding, BIOS only needs to read SPD during the POST stage.
>>>>>>>>> For the rest of time, BIOS will hand over the SPD bus to BMC.
>>>>>>>> That seems like it might work.  You'll have to deal with the time when the BIOS
>>>>>>>> has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
>>>>>>>> be fed to the BMC as an input GPIO so that you can differentiate between "we
>>>>>>>> don't own the mux" and "all the devices are NAKing us".
>>>>>>> This seems like a nitty gritty design detail that's best handled in
>>>>>>> code when we review it.  I think the important bit here is that there
>>>>>>> are paths where this could work without a significant design issue.
>>>>>> Just one subtlety.  I wouldn't expect this, necessarily, to be in _our_ design
>>>>>> and/or code, except that we'd want to document the GPIO line like we do all
>>>>>> others.  I was trying to hint that "if I were involved in this hardware design,
>>>>>> I'd ask for...".  If you leave it out, I'm sure it'll work _most_ of the time
>>>>>> just fine and it'll be your problem to debug it when it doesn't.
>>>>> Understood.
>>>> Thanks for all your suggestions. I will keep them in mind during implementation.
>>> What about CLTT? Switching MUX to the BMC makes CPU not able to get DIMM
>>> temperature. Are you assuming here this feature is enabled in BMC FW?
>> BMC could assist with CLTT, but since this is CPU-specific it would
>> belong in a separate daemon. That daemon could get the relevant
>> temperatures over D-Bus using the standard sensor interface, so I
>> don't think it should affect the design for this component.
>>
>>> Are you going to support DCPMM as well? If so, there is another problem
>>> since switching MUX to BMC you brakes NVDIMM related FW/SW running on
>>> Host OS.
>> There are no plans currently for supporting NVDIMMs, just DDR5 at
>> first as Michael mentioned, and possibly other DDR versions in the
>> future.

I see. So the app will just read SPD non-volatile content and provide it 
for user, e.g. over DBus, right?

Are you going to access DIMM periodically? It seems that it shall be 
enough to access DIMMs once per ac cycle/dc cycle. And just return SPD 
ownership to the CPU for the rest of time.

>>>>>>>> You should take a look at what is already existing in fru-device (part of
>>>>>>>> entity-manager repository).  This is already doing this for IPMI-format EEPROM
>>>>>>>> data.  We should be able to replicate/enhance this code, in the same repository,
>>>>>>>> to handle SPD format.
>>>>>>> I am not sure if it's a good idea to put it into the entity-manager
>>>>>>> repo. As you said EM
>>>>>>> is designed for IPMI-format EEPROM. Adding another parser into that
>>>>>>> repo may violate
>>>>>>> EM's design.
>>>>>> I'm not sure why it would be an issue.  Hopefully one of the maintainers of that
>>>>>> repo can weigh in.  I wouldn't expect "parsing only IPMI-format EEPROMs" is a
>>>>>> design but just the current state of implementation.
>>>>> So long as it can function properly in its current design, i have no
>>>>> problem with FruDevice adding more parsing types.  In fact, there's
>>>>> already patchsets out to add Linkedins proprietary fru type to
>>>>> FruDevice, so in terms of design, Patricks request seems reasonable.
>>>> Got it. Then I will push the code to EM.

Re: Propose a new application for reading DIMM SPD directly

[Michael Shen]

> I see. So the app will just read SPD non-volatile content and provide it
> for user, e.g. over DBus, right?

Yes, the plan is over DBus.

> Are you going to access DIMM periodically? It seems that it shall be
> enough to access DIMMs once per ac cycle/dc cycle. And just return SPD
> ownership to the CPU for the rest of time.

I think reading the SPD once per ac/dc cycle is enough like you said.
But if we want BMC to monitor the DIMM temp instead of CPU, then BMC
can't return the SPD ownership since the temp needs to be read
periodically.

On Wed, Feb 16, 2022 at 4:39 AM Zbigniew, Lukwinski

<zbigniew.lukwinski@linux.intel.com> wrote:
>
> On 2/15/2022 2:50 AM, Michael Shen wrote:
> >> What about CLTT? Switching MUX to the BMC makes CPU not able to get DIMM
> >> temperature. Are you assuming here this feature is enabled in BMC FW?
> > CPU can still monitor the DIMM temp by reading the MR4 register and
> > trigger the DIMM throttling if needed. So I think the CLTT will not be
> > affected.
> > If CPU needs something more than MR4 register provided, BMC can assist
> > it in another separate daemon like Benjamin mentioned.
>
> Got it.
>
> > On Tue, Feb 15, 2022 at 6:17 AM Benjamin Fair <benjaminfair@google.com> wrote:
> >> On Fri, 11 Feb 2022 at 13:21, Zbigniew, Lukwinski
> >> <zbigniew.lukwinski@linux.intel.com> wrote:
> >>> On 2/11/2022 1:40 AM, Michael Shen wrote:
> >>>> On Thu, Feb 10, 2022 at 6:45 AM Ed Tanous <edtanous@google.com> wrote:
> >>>>> On Wed, Feb 9, 2022 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> >>>>>> On Wed, Feb 09, 2022 at 12:20:00PM -0800, Ed Tanous wrote:
> >>>>>>> On Wed, Feb 9, 2022 at 11:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> >>>>>>>> On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
> >>>>>>>>> On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
> >>>>>>>>>> On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
> >>>>>>>>> BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
> >>>>>>>>>   From my understanding, BIOS only needs to read SPD during the POST stage.
> >>>>>>>>> For the rest of time, BIOS will hand over the SPD bus to BMC.
> >>>>>>>> That seems like it might work.  You'll have to deal with the time when the BIOS
> >>>>>>>> has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
> >>>>>>>> be fed to the BMC as an input GPIO so that you can differentiate between "we
> >>>>>>>> don't own the mux" and "all the devices are NAKing us".
> >>>>>>> This seems like a nitty gritty design detail that's best handled in
> >>>>>>> code when we review it.  I think the important bit here is that there
> >>>>>>> are paths where this could work without a significant design issue.
> >>>>>> Just one subtlety.  I wouldn't expect this, necessarily, to be in _our_ design
> >>>>>> and/or code, except that we'd want to document the GPIO line like we do all
> >>>>>> others.  I was trying to hint that "if I were involved in this hardware design,
> >>>>>> I'd ask for...".  If you leave it out, I'm sure it'll work _most_ of the time
> >>>>>> just fine and it'll be your problem to debug it when it doesn't.
> >>>>> Understood.
> >>>> Thanks for all your suggestions. I will keep them in mind during implementation.
> >>> What about CLTT? Switching MUX to the BMC makes CPU not able to get DIMM
> >>> temperature. Are you assuming here this feature is enabled in BMC FW?
> >> BMC could assist with CLTT, but since this is CPU-specific it would
> >> belong in a separate daemon. That daemon could get the relevant
> >> temperatures over D-Bus using the standard sensor interface, so I
> >> don't think it should affect the design for this component.
> >>
> >>> Are you going to support DCPMM as well? If so, there is another problem
> >>> since switching MUX to BMC you brakes NVDIMM related FW/SW running on
> >>> Host OS.
> >> There are no plans currently for supporting NVDIMMs, just DDR5 at
> >> first as Michael mentioned, and possibly other DDR versions in the
> >> future.
>
> I see. So the app will just read SPD non-volatile content and provide it
> for user, e.g. over DBus, right?
>
> Are you going to access DIMM periodically? It seems that it shall be
> enough to access DIMMs once per ac cycle/dc cycle. And just return SPD
> ownership to the CPU for the rest of time.
>
> >>>>>>>> You should take a look at what is already existing in fru-device (part of
> >>>>>>>> entity-manager repository).  This is already doing this for IPMI-format EEPROM
> >>>>>>>> data.  We should be able to replicate/enhance this code, in the same repository,
> >>>>>>>> to handle SPD format.
> >>>>>>> I am not sure if it's a good idea to put it into the entity-manager
> >>>>>>> repo. As you said EM
> >>>>>>> is designed for IPMI-format EEPROM. Adding another parser into that
> >>>>>>> repo may violate
> >>>>>>> EM's design.
> >>>>>> I'm not sure why it would be an issue.  Hopefully one of the maintainers of that
> >>>>>> repo can weigh in.  I wouldn't expect "parsing only IPMI-format EEPROMs" is a
> >>>>>> design but just the current state of implementation.
> >>>>> So long as it can function properly in its current design, i have no
> >>>>> problem with FruDevice adding more parsing types.  In fact, there's
> >>>>> already patchsets out to add Linkedins proprietary fru type to
> >>>>> FruDevice, so in terms of design, Patricks request seems reasonable.
> >>>> Got it. Then I will push the code to EM.

Re: Propose a new application for reading DIMM SPD directly

[Zbigniew, Lukwinski]

On 2/17/2022 4:59 AM, Michael Shen wrote:
>> I see. So the app will just read SPD non-volatile content and provide it
>> for user, e.g. over DBus, right?
> Yes, the plan is over DBus.
Ok. Thanks.
>> Are you going to access DIMM periodically? It seems that it shall be
>> enough to access DIMMs once per ac cycle/dc cycle. And just return SPD
>> ownership to the CPU for the rest of time.
> I think reading the SPD once per ac/dc cycle is enough like you said.
> But if we want BMC to monitor the DIMM temp instead of CPU, then BMC
> can't return the SPD ownership since the temp needs to be read
> periodically.

Got it!

>
> On Wed, Feb 16, 2022 at 4:39 AM Zbigniew, Lukwinski
>
> <zbigniew.lukwinski@linux.intel.com> wrote:
>> On 2/15/2022 2:50 AM, Michael Shen wrote:
>>>> What about CLTT? Switching MUX to the BMC makes CPU not able to get DIMM
>>>> temperature. Are you assuming here this feature is enabled in BMC FW?
>>> CPU can still monitor the DIMM temp by reading the MR4 register and
>>> trigger the DIMM throttling if needed. So I think the CLTT will not be
>>> affected.
>>> If CPU needs something more than MR4 register provided, BMC can assist
>>> it in another separate daemon like Benjamin mentioned.
>> Got it.
>>
>>> On Tue, Feb 15, 2022 at 6:17 AM Benjamin Fair <benjaminfair@google.com> wrote:
>>>> On Fri, 11 Feb 2022 at 13:21, Zbigniew, Lukwinski
>>>> <zbigniew.lukwinski@linux.intel.com> wrote:
>>>>> On 2/11/2022 1:40 AM, Michael Shen wrote:
>>>>>> On Thu, Feb 10, 2022 at 6:45 AM Ed Tanous <edtanous@google.com> wrote:
>>>>>>> On Wed, Feb 9, 2022 at 1:14 PM Patrick Williams <patrick@stwcx.xyz> wrote:
>>>>>>>> On Wed, Feb 09, 2022 at 12:20:00PM -0800, Ed Tanous wrote:
>>>>>>>>> On Wed, Feb 9, 2022 at 11:56 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>>>>>>>>>> On Tue, Feb 08, 2022 at 04:23:12PM +0800, Michael Shen wrote:
>>>>>>>>>>> On Tue, Feb 8, 2022 at 3:11 PM Patrick Williams <patrick@stwcx.xyz> wrote:
>>>>>>>>>>>> On Tue, Feb 08, 2022 at 01:10:37PM +0800, Michael Shen wrote:
>>>>>>>>>>> BIOS owns the MUX select pin and it can decide who owns the SPD(I2C/I3C) bus.
>>>>>>>>>>>    From my understanding, BIOS only needs to read SPD during the POST stage.
>>>>>>>>>>> For the rest of time, BIOS will hand over the SPD bus to BMC.
>>>>>>>>>> That seems like it might work.  You'll have to deal with the time when the BIOS
>>>>>>>>>> has the mux in the BMC code somehow.  Ideally I'd ask for the mux select to also
>>>>>>>>>> be fed to the BMC as an input GPIO so that you can differentiate between "we
>>>>>>>>>> don't own the mux" and "all the devices are NAKing us".
>>>>>>>>> This seems like a nitty gritty design detail that's best handled in
>>>>>>>>> code when we review it.  I think the important bit here is that there
>>>>>>>>> are paths where this could work without a significant design issue.
>>>>>>>> Just one subtlety.  I wouldn't expect this, necessarily, to be in _our_ design
>>>>>>>> and/or code, except that we'd want to document the GPIO line like we do all
>>>>>>>> others.  I was trying to hint that "if I were involved in this hardware design,
>>>>>>>> I'd ask for...".  If you leave it out, I'm sure it'll work _most_ of the time
>>>>>>>> just fine and it'll be your problem to debug it when it doesn't.
>>>>>>> Understood.
>>>>>> Thanks for all your suggestions. I will keep them in mind during implementation.
>>>>> What about CLTT? Switching MUX to the BMC makes CPU not able to get DIMM
>>>>> temperature. Are you assuming here this feature is enabled in BMC FW?
>>>> BMC could assist with CLTT, but since this is CPU-specific it would
>>>> belong in a separate daemon. That daemon could get the relevant
>>>> temperatures over D-Bus using the standard sensor interface, so I
>>>> don't think it should affect the design for this component.
>>>>
>>>>> Are you going to support DCPMM as well? If so, there is another problem
>>>>> since switching MUX to BMC you brakes NVDIMM related FW/SW running on
>>>>> Host OS.
>>>> There are no plans currently for supporting NVDIMMs, just DDR5 at
>>>> first as Michael mentioned, and possibly other DDR versions in the
>>>> future.
>> I see. So the app will just read SPD non-volatile content and provide it
>> for user, e.g. over DBus, right?
>>
>> Are you going to access DIMM periodically? It seems that it shall be
>> enough to access DIMMs once per ac cycle/dc cycle. And just return SPD
>> ownership to the CPU for the rest of time.
>>
>>>>>>>>>> You should take a look at what is already existing in fru-device (part of
>>>>>>>>>> entity-manager repository).  This is already doing this for IPMI-format EEPROM
>>>>>>>>>> data.  We should be able to replicate/enhance this code, in the same repository,
>>>>>>>>>> to handle SPD format.
>>>>>>>>> I am not sure if it's a good idea to put it into the entity-manager
>>>>>>>>> repo. As you said EM
>>>>>>>>> is designed for IPMI-format EEPROM. Adding another parser into that
>>>>>>>>> repo may violate
>>>>>>>>> EM's design.
>>>>>>>> I'm not sure why it would be an issue.  Hopefully one of the maintainers of that
>>>>>>>> repo can weigh in.  I wouldn't expect "parsing only IPMI-format EEPROMs" is a
>>>>>>>> design but just the current state of implementation.
>>>>>>> So long as it can function properly in its current design, i have no
>>>>>>> problem with FruDevice adding more parsing types.  In fact, there's
>>>>>>> already patchsets out to add Linkedins proprietary fru type to
>>>>>>> FruDevice, so in terms of design, Patricks request seems reasonable.
>>>>>> Got it. Then I will push the code to EM.