Re: apply for a new repo "openbmc/node-data-sync"

Re: apply for a new repo "openbmc/node-data-sync"

[yugang.chen]

thanks, Brad.

+ openbmc@lists.ozlabs.org .

Dear All,

I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes.

1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control.

2. Some sensor are in secondary nodes, they are needed to show them on primary node.

So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks.

Best Regards

Daniel(Yugang)


On 4/10/2021 10:55 PM, Brad Bishop wrote:
> Hi Daniel
>
> Can you please send this to the mailing list so everyone can be aware of the work you are doing?
>
> thanks - brad
>
>> On Apr 7, 2021, at 1:31 AM, yugang.chen <yugang.chen@linux.intel.com> wrote:
>>
>> Hi Brad,
>>
>> I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes.
>>
>> 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control.
>>
>> 2. Some sensor are in secondary nodes, they are needed to show them on primary node.
>>
>> So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks.
>>
>>
>> Best Regards
>>
>> Daniel(Yugang)
>>

Re: apply for a new repo "openbmc/node-data-sync"

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1637 bytes --]

On Mon, Apr 12, 2021 at 02:38:38PM +0800, yugang.chen wrote:
> thanks, Brad.
> 
> + openbmc@lists.ozlabs.org .
> 
> Dear All,
> 
> I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes.
> 
> 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control.
> 
> 2. Some sensor are in secondary nodes, they are needed to show them on primary node.
> 
> So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks.

Do you have a design on how this would be accomplished?  I'm curious
what your proposal ends up looking like at a dbus-level.  We should make
sure this is documented at a high level and also aligns with how we are
handling multi-host scenarios already in our dbus layout.

How are the two BMC devices connected?  How do you ensure you can safely
/ securely accept the data between the two BMCs?  If done poorly this
could open up a large security attack surface.

I suspect your software design is somewhat driven by a particular hardware
design as well, so we need to make sure it is clear the scoping of your
solution and should probably even communicate that in the name.  For
instance, if your data transport is IPMB over i2c and you don't plan on
this being a general daemon that could be expanded to a network
connection, we shouldn't name it so generally.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: apply for a new repo "openbmc/node-data-sync"

[Ed Tanous]

On Sun, Apr 11, 2021 at 11:40 PM yugang.chen
<yugang.chen@linux.intel.com> wrote:
>
> thanks, Brad.
>
> + openbmc@lists.ozlabs.org .
>
> Dear All,
>
> I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes.
>
> 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control.
>
> 2. Some sensor are in secondary nodes, they are needed to show them on primary node.
>
> So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks.
>

+1 to wanting a design document for this.  In absence of that, I'm
going to make the possibly wrong assumption that you're planning on
using IPMB.  In dbus-sensors, we already have IPMB Sensor syncing, and
patches are in progress to add IPMB version remoting.  You might want
to look into those to see if they meet your needs, or can be amended
to meet your needs before trying to push a new project.

> Best Regards
>
> Daniel(Yugang)
>
>
> On 4/10/2021 10:55 PM, Brad Bishop wrote:
> > Hi Daniel
> >
> > Can you please send this to the mailing list so everyone can be aware of the work you are doing?
> >
> > thanks - brad
> >
> >> On Apr 7, 2021, at 1:31 AM, yugang.chen <yugang.chen@linux.intel.com> wrote:
> >>
> >> Hi Brad,
> >>
> >> I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes.
> >>
> >> 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control.
> >>
> >> 2. Some sensor are in secondary nodes, they are needed to show them on primary node.
> >>
> >> So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks.
> >>
> >>
> >> Best Regards
> >>
> >> Daniel(Yugang)
> >>

Re: apply for a new repo "openbmc/node-data-sync"

[yugang.chen]

[-- Attachment #1: Type: text/plain, Size: 2289 bytes --]

attach the design document, please take a look.

Best Regards
Daniel(Yugang)

On 4/13/2021 3:26 AM, Ed Tanous wrote:
> On Sun, Apr 11, 2021 at 11:40 PM yugang.chen
> <yugang.chen@linux.intel.com> wrote:
>> thanks, Brad.
>>
>> + openbmc@lists.ozlabs.org .
>>
>> Dear All,
>>
>> I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes.
>>
>> 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control.
>>
>> 2. Some sensor are in secondary nodes, they are needed to show them on primary node.
>>
>> So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks.
>>
> +1 to wanting a design document for this.  In absence of that, I'm
> going to make the possibly wrong assumption that you're planning on
> using IPMB.  In dbus-sensors, we already have IPMB Sensor syncing, and
> patches are in progress to add IPMB version remoting.  You might want
> to look into those to see if they meet your needs, or can be amended
> to meet your needs before trying to push a new project.
>
>> Best Regards
>>
>> Daniel(Yugang)
>>
>>
>> On 4/10/2021 10:55 PM, Brad Bishop wrote:
>>> Hi Daniel
>>>
>>> Can you please send this to the mailing list so everyone can be aware of the work you are doing?
>>>
>>> thanks - brad
>>>
>>>> On Apr 7, 2021, at 1:31 AM, yugang.chen <yugang.chen@linux.intel.com> wrote:
>>>>
>>>> Hi Brad,
>>>>
>>>> I'm Intel BMC engineer, we have a module is used to sync sensor data between BMC nodes in one Chassis system. In our Modular system, there are 2 nodes or 4 nodes, one node works as Primary node, the other nodes work as secondary nodes.
>>>>
>>>> 1. Some sensors are only visible for primary BMC, but secondary nodes need the sensors for Fan control.
>>>>
>>>> 2. Some sensor are in secondary nodes, they are needed to show them on primary node.
>>>>
>>>> So, we need a new repo to sync the sensor data between primary node and secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks.
>>>>
>>>>
>>>> Best Regards
>>>>
>>>> Daniel(Yugang)
>>>>

[-- Attachment #2: node-data-sync.md --]
[-- Type: text/plain, Size: 3573 bytes --]

# Modular System Support - PCH IO Expander

Author: Chen, Yugang

## Problem Description
When multi modular 2 socket reference platforms (RP) are connected together,
the BMCs on each RP shall support the ability to manage the system as either
single 4S/8S system or standalone as 2S systems.

## Background and References
In 4S/8S mode, only one PCH works at DMI (Direct Media Interface) mode,
this PCH is called legacy PCH (PCH.L), the other PCHs are called PCH IO Expander
(PCH.IO). The BMC attached to the PCH.L works as a primary and the other BMCs
work as secondaries.

## Requirements
Check a modular system as either 4S/8S or standalone mode by GPIOs on BMC each
time system AC on.
Each BMC works independently in standalone mode.
In 4S/8S mode, only primary BMC is responsible for management of system level
resources, secondary BMCs need to collect its local management resources and report
them to primary BMC.

## Proposed Design
Each BMC will have a property to reveal its work mode. Users can know if BMC is in standalone mode or 4S/8S mode.
Each BMC has another property to show if the BMC mode is PCH Legacy BMC or
Non-Legacy BMC. PCH Legacy BMC is working as primary and Non-Legacy BMC working
as secondary.

Each time a BMC reboot, The BMC needs to check 3 GPIO Pins: FM_STANDALONE_MONDE_N / FM_4S_8S_N_MODE / FM_NODE_ID to get working mode as standalone mode or primary /secondary(4S/8S) role according to the GPIO values.
After confirming the mode and BMCs' role, BMCs should set properties according
to the correct configuration.
In 4S/8S mode only node id 0 will be primary BMC because only this node will be the PCH.L. Node id 1,2,3 will be the secondary nodes.

Once a BMC gets mode is in 4S/8S, node roles are configured by node
ID (GPIO Pins) and keep consistent once AC on. Once node role check is done,

In 4S/8S mode, Primary BMC needs to broadcast its role to make sure there is only
one primary BMC in the system.

Need a new feature to make sure secondary BMCs send local redfish events to primary
BMC. And primary BMC needs to add a tag to those events coming from secondary BMC so
that user can know the event logs happened on which node.

Even in 4S/8S mode, each BMC will collect its local management resources,
including sensors, fans and do FSC according to the values of local sensors.
PSU and Fans on each node will not be connected together. Configuration
settings of each secondary node will remain the same, and won't be synced across
the nodes.

In 4S/8S mode, PECI will only be connected to primary node. Primary BMC needs to
monitor all CPU and DIMM sensors and deliver the sensor values of the CPUs/DIMMs
on secondary nodes to secondary BMCs. So that secondary BMCs can use the sensor values to control their own FSCs. Primary BMC also needs to have a way to find how many
CPUs are in the whole system include Primary and Secondary nodes.

## Alternatives Considered
Primary node monitors all the IPMI sensors in secondary nodes and creates redfish log
by itself.

Instead of BMC reboot, only AC cycle will make BMCs check GPIO pins and set
Legacy BMC or Non-Legacy BMC mode.

Only primary BMC broadcast its role and secondary only waiting for the broadcast
from primary.

## Impacts
Only on the motherboard where legacy PCH is located, POST code/Front Panel/KCS
port/UART will work, while these interfaces on board with non-legacy PCH will
not work due to BIOS and HW design. And this will cause non-functioning of
SOL/KVM/Virtual media on secondary BMCs.

Re: apply for a new repo "openbmc/node-data-sync"

[Brad Bishop]

Hi Daniel

Typically email attachments are not opened.  To ensure your mail gets 
read, avoid attachments.  Resending without the attachment...

On Mon, Apr 19, 2021 at 12:41:00PM +0800, yugang.chen wrote:
>attach the design document, please take a look.
>
>Best Regards
>Daniel(Yugang)
>

># Modular System Support - PCH IO Expander
>
>Author: Chen, Yugang
>
>## Problem Description
>When multi modular 2 socket reference platforms (RP) are connected together,
>the BMCs on each RP shall support the ability to manage the system as either
>single 4S/8S system or standalone as 2S systems.
>
>## Background and References
>In 4S/8S mode, only one PCH works at DMI (Direct Media Interface) mode,
>this PCH is called legacy PCH (PCH.L), the other PCHs are called PCH IO Expander
>(PCH.IO). The BMC attached to the PCH.L works as a primary and the other BMCs
>work as secondaries.
>
>## Requirements
>Check a modular system as either 4S/8S or standalone mode by GPIOs on BMC each
>time system AC on.
>Each BMC works independently in standalone mode.
>In 4S/8S mode, only primary BMC is responsible for management of system level
>resources, secondary BMCs need to collect its local management resources and report
>them to primary BMC.
>
>## Proposed Design
>Each BMC will have a property to reveal its work mode. Users can know if BMC is in standalone mode or 4S/8S mode.
>Each BMC has another property to show if the BMC mode is PCH Legacy BMC or
>Non-Legacy BMC. PCH Legacy BMC is working as primary and Non-Legacy BMC working
>as secondary.
>
>Each time a BMC reboot, The BMC needs to check 3 GPIO Pins: FM_STANDALONE_MONDE_N / FM_4S_8S_N_MODE / FM_NODE_ID to get working mode as standalone mode or primary /secondary(4S/8S) role according to the GPIO values.
>After confirming the mode and BMCs' role, BMCs should set properties according
>to the correct configuration.
>In 4S/8S mode only node id 0 will be primary BMC because only this node will be the PCH.L. Node id 1,2,3 will be the secondary nodes.
>
>Once a BMC gets mode is in 4S/8S, node roles are configured by node
>ID (GPIO Pins) and keep consistent once AC on. Once node role check is done,
>
>In 4S/8S mode, Primary BMC needs to broadcast its role to make sure there is only
>one primary BMC in the system.
>
>Need a new feature to make sure secondary BMCs send local redfish events to primary
>BMC. And primary BMC needs to add a tag to those events coming from secondary BMC so
>that user can know the event logs happened on which node.
>
>Even in 4S/8S mode, each BMC will collect its local management resources,
>including sensors, fans and do FSC according to the values of local sensors.
>PSU and Fans on each node will not be connected together. Configuration
>settings of each secondary node will remain the same, and won't be synced across
>the nodes.
>
>In 4S/8S mode, PECI will only be connected to primary node. Primary BMC needs to
>monitor all CPU and DIMM sensors and deliver the sensor values of the CPUs/DIMMs
>on secondary nodes to secondary BMCs. So that secondary BMCs can use the sensor values to control their own FSCs. Primary BMC also needs to have a way to find how many
>CPUs are in the whole system include Primary and Secondary nodes.
>
>## Alternatives Considered
>Primary node monitors all the IPMI sensors in secondary nodes and creates redfish log
>by itself.
>
>Instead of BMC reboot, only AC cycle will make BMCs check GPIO pins and set
>Legacy BMC or Non-Legacy BMC mode.
>
>Only primary BMC broadcast its role and secondary only waiting for the broadcast
>from primary.
>
>## Impacts
>Only on the motherboard where legacy PCH is located, POST code/Front Panel/KCS
>port/UART will work, while these interfaces on board with non-legacy PCH will
>not work due to BIOS and HW design. And this will cause non-functioning of
>SOL/KVM/Virtual media on secondary BMCs.

Re: apply for a new repo "openbmc/node-data-sync"

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 7438 bytes --]

Hello Daniel,

I've read through this and feel like most of my original questions are
still unanswered.  I've pasted them here again for reference:

---
> So, we need a new repo to sync the sensor data between primary node and
+secondary nodes, could you create a repo "openbmc/node-data-sync"? thanks.

Do you have a design on how this would be accomplished?  I'm curious
what your proposal ends up looking like at a dbus-level.  We should make
sure this is documented at a high level and also aligns with how we are
handling multi-host scenarios already in our dbus layout.

How are the two BMC devices connected?  How do you ensure you can safely
/ securely accept the data between the two BMCs?  If done poorly this
could open up a large security attack surface.

I suspect your software design is somewhat driven by a particular hardware
design as well, so we need to make sure it is clear the scoping of your
solution and should probably even communicate that in the name.  For
instance, if your data transport is IPMB over i2c and you don't plan on
this being a general daemon that could be expanded to a network
connection, we shouldn't name it so generally.
---


On Mon, Apr 19, 2021 at 10:26:31AM -0400, Brad Bishop wrote:
> Hi Daniel
> 
> Typically email attachments are not opened.  To ensure your mail gets 
> read, avoid attachments.  Resending without the attachment...
> 
> On Mon, Apr 19, 2021 at 12:41:00PM +0800, yugang.chen wrote:
> >attach the design document, please take a look.
> >
> >Best Regards
> >Daniel(Yugang)
> >
> 
> >## Requirements
> >resources, secondary BMCs need to collect its local management resources and report
> >them to primary BMC.

Why is this a requirement?  What is driving it?

You could alternatively have DC-level software gather data directly from
both BMCs.  Please put that in the alternative and give rationale on why
this extra software is a better solution to the problem.

> >## Proposed Design
> >Each time a BMC reboot, The BMC needs to check 3 GPIO Pins: FM_STANDALONE_MONDE_N / FM_4S_8S_N_MODE / FM_NODE_ID to get working mode as standalone mode or primary /secondary(4S/8S) role according to the GPIO values.
> >After confirming the mode and BMCs' role, BMCs should set properties according
> >to the correct configuration.
> >In 4S/8S mode only node id 0 will be primary BMC because only this node will be the PCH.L. Node id 1,2,3 will be the secondary nodes.

Some of this is very specific to your hardware design / architecture
being used (ex. "PCH" is an Intel term).  We may want to put some of
this into a background or architecture-specific section.

I get two high-level design points from reading this in my mind:
    1. You are not handling dynamic primary/secondary BMC selection; it
       is all assigned by physical position.  This is not unreasonable
       but it isn't sufficient for some other designs, so we should
       spell this out.
    2. Assignment of role will be done based on GPIOs for your
       particular implementation, but could be handled in a different
       manner for other systems.

Based on #2, I would suggest the role-selection code is separated from
any dbus/Redfish propagation code.  The role should likely be a dbus
property.

Also, these GPIO names seem like they are schematic names.  You might
want to look at docs:designs/device-tree-gpio-naming.md.  We typically
want to see a logical name from a userspace application point of view.

> >
> >Once a BMC gets mode is in 4S/8S, node roles are configured by node
> >ID (GPIO Pins) and keep consistent once AC on. 

I think I know what you meant by this, but the wording is very difficult for
BMCs to accomplish.  OCP systems don't even have "AC power" since the
server is supplied DC 12 or 48v from the rack.  The BMC has almost no
way to know if it was reset in certain ways vs had standby power cycled.

I would suggest at a hardware level that the node roles are kept static
during the entire standby power cycle.  Likely this is what you meant, but
the wording here could be clearer.

> >Once node role check is done,
> >
> >In 4S/8S mode, Primary BMC needs to broadcast its role to make sure there is only
> >one primary BMC in the system.

And what if not?

> >Need a new feature to make sure secondary BMCs send local redfish events to primary
> >BMC. And primary BMC needs to add a tag to those events coming from secondary BMC so
> >that user can know the event logs happened on which node.

Why Redfish and not dbus?  I think this is the first time that Redfish
is mentioned.

Does the primary really just need "events" or does it really need all of
the data necessary to create a Redfish model of the resources modeled by
the secondary?  I assume the latter, because if it was just events then
something still needs to go to the secondary to get the rest of the
model and to make changes to it.

> >Even in 4S/8S mode, each BMC will collect its local management resources,
> >including sensors, fans and do FSC according to the values of local sensors.
> >PSU and Fans on each node will not be connected together. Configuration
> >settings of each secondary node will remain the same, and won't be synced across
> >the nodes.

FSC?  Fan Speed Control?  Please don't use new and previously unused acronyms,
unless they are a well-known / industry standard protocol.

> >In 4S/8S mode, PECI will only be connected to primary node. Primary BMC needs to
> >monitor all CPU and DIMM sensors and deliver the sensor values of the CPUs/DIMMs
> >on secondary nodes to secondary BMCs. So that secondary BMCs can use the sensor values to control their own FSCs. Primary BMC also needs to have a way to find how many
> >CPUs are in the whole system include Primary and Secondary nodes.

This paragraph implies to me that the primary is also *pushing* sensor
values to the secondary so that the secondary can make local decisions
about fan speeds from information obtained over PECI on the primary?

It really seems like Redfish "events" is not sufficient for what you're
trying to accomplish, but you want some generic "sync dbus state between
BMCs" design.

> >## Alternatives Considered

All of these alternatives need an explaination of why this alternative
was not chosen.

> >Primary node monitors all the IPMI sensors in secondary nodes and creates redfish log
> >by itself.

Is "IPMI sensors" really what you wanted to say here?

> >Instead of BMC reboot, only AC cycle will make BMCs check GPIO pins and set
> >Legacy BMC or Non-Legacy BMC mode.

I didn't get the impression above that BMC reboot was when GPIO pins
were checked (and part of the reason why I pointed that out above).

> >Only primary BMC broadcast its role and secondary only waiting for the broadcast
> >from primary.

I'm not sure what this means...

> >## Impacts
> >Only on the motherboard where legacy PCH is located, POST code/Front Panel/KCS
> >port/UART will work, while these interfaces on board with non-legacy PCH will
> >not work due to BIOS and HW design. And this will cause non-functioning of
> >SOL/KVM/Virtual media on secondary BMCs.

We are interested in *software* impacts here.  What code needs to
change?  How does this new design affect the way we think about existing
software components?

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: apply for a new repo "openbmc/node-data-sync"

[Milton Miller II]

[-- Attachment #1.1: Type: text/plain, Size: 204 bytes --]


yugang.chen wrote:

> attach the design document, please take a look.

Please create a gerrit review placing this in an appropriate directory in the OpenBMC docs repository.

Thanks,
Milton



[-- Attachment #1.2: Type: text/html, Size: 228 bytes --]