Multiple BMCs in a system: IPMB? Redfish? MCTP?

Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Nancy Yuen]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

I've talked with some people a while back (long while back) about multiple
BMCs in a system.  Either for redundancy or managing separate parts of a
system.   I'm wondering what other people are thinking in this area if at
all.

We are considering similar designs and I'm looking into options for BMC-BMC
communications.  Some BMCs may not be externally accessible. Here are some
options that we've looked at:

   1. i2c/IPMB
   2. usbnet/Redfish
   3. i2c/MCTP/PLDM or something else?
   4. internal network via switch chip/Redfish or MCTP

 I'd like to reduce our use of IPMI so I want to avoid (1).

----------
Nancy

[-- Attachment #2: Type: text/html, Size: 1006 bytes --]

RE: [EXTERNAL] Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Neeraj Ladkani]

[-- Attachment #1: Type: text/plain, Size: 1086 bytes --]

Nancy,

Just trying to understand, are there any specific scenarios that need aggregation at Head Node using BMC-BMC communication? Have you thought about aggregation at cloud service?

Neeraj

From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> On Behalf Of Nancy Yuen
Sent: Wednesday, April 29, 2020 3:52 PM
To: OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: [EXTERNAL] Multiple BMCs in a system: IPMB? Redfish? MCTP?

I've talked with some people a while back (long while back) about multiple BMCs in a system.  Either for redundancy or managing separate parts of a system.   I'm wondering what other people are thinking in this area if at all.

We are considering similar designs and I'm looking into options for BMC-BMC communications.  Some BMCs may not be externally accessible. Here are some options that we've looked at:

  1.  i2c/IPMB
  2.  usbnet/Redfish
  3.  i2c/MCTP/PLDM or something else?
  4.  internal network via switch chip/Redfish or MCTP
 I'd like to reduce our use of IPMI so I want to avoid (1).

----------
Nancy

[-- Attachment #2: Type: text/html, Size: 3978 bytes --]

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Vijay Khemka]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

Hi Nancy,
We are currently using (1) in our current multi host design. Option (3) also looks good.

Regards
-Vijay

From: openbmc <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org> on behalf of Nancy Yuen <yuenn@google.com>
Date: Wednesday, April 29, 2020 at 3:53 PM
To: OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: Multiple BMCs in a system: IPMB? Redfish? MCTP?

I've talked with some people a while back (long while back) about multiple BMCs in a system.  Either for redundancy or managing separate parts of a system.   I'm wondering what other people are thinking in this area if at all.

We are considering similar designs and I'm looking into options for BMC-BMC communications.  Some BMCs may not be externally accessible. Here are some options that we've looked at:

  1.  i2c/IPMB
  2.  usbnet/Redfish
  3.  i2c/MCTP/PLDM or something else?
  4.  internal network via switch chip/Redfish or MCTP
 I'd like to reduce our use of IPMI so I want to avoid (1).

----------
Nancy

[-- Attachment #2: Type: text/html, Size: 3740 bytes --]

RE: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Neeraj Ladkani]

[-- Attachment #1: Type: text/plain, Size: 1741 bytes --]

Redfish defines computer system to be multiple chassis or multiple blades within a chassis.  It would be good to clarify if you are referring to “System” as:


  *   Multiple chassis with their own BMCs ( Storage, Compute, GPU etc )
  *   Multiple Blades with their own BMC within a chassis.

Neeraj

From: openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> On Behalf Of Vijay Khemka
Sent: Wednesday, April 29, 2020 5:16 PM
To: Nancy Yuen <yuenn@google.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: [EXTERNAL] Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

Hi Nancy,
We are currently using (1) in our current multi host design. Option (3) also looks good.

Regards
-Vijay

From: openbmc <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org<mailto:openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org>> on behalf of Nancy Yuen <yuenn@google.com<mailto:yuenn@google.com>>
Date: Wednesday, April 29, 2020 at 3:53 PM
To: OpenBMC Maillist <openbmc@lists.ozlabs.org<mailto:openbmc@lists.ozlabs.org>>
Subject: Multiple BMCs in a system: IPMB? Redfish? MCTP?

I've talked with some people a while back (long while back) about multiple BMCs in a system.  Either for redundancy or managing separate parts of a system.   I'm wondering what other people are thinking in this area if at all.

We are considering similar designs and I'm looking into options for BMC-BMC communications.  Some BMCs may not be externally accessible. Here are some options that we've looked at:

  1.  i2c/IPMB
  2.  usbnet/Redfish
  3.  i2c/MCTP/PLDM or something else?
  4.  internal network via switch chip/Redfish or MCTP
 I'd like to reduce our use of IPMI so I want to avoid (1).

----------
Nancy

[-- Attachment #2: Type: text/html, Size: 8815 bytes --]

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Nancy Yuen]

[-- Attachment #1: Type: text/plain, Size: 1501 bytes --]

Neeraj, I was not considering aggregation in this case, just having the
intermediate BMC "route".

Vijay, thanks.  I was wondering what your applications for IPMB were for.
What's the rationale for using IPMB vs something else?  In your multihost
system, one BMC supports multiple host CPUs?  Are there also multiple BMCs?

----------
Nancy


On Wed, Apr 29, 2020 at 5:15 PM Vijay Khemka <vijaykhemka@fb.com> wrote:

> Hi Nancy,
>
> We are currently using (1) in our current multi host design. Option (3)
> also looks good.
>
>
>
> Regards
>
> -Vijay
>
>
>
> *From: *openbmc <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org> on
> behalf of Nancy Yuen <yuenn@google.com>
> *Date: *Wednesday, April 29, 2020 at 3:53 PM
> *To: *OpenBMC Maillist <openbmc@lists.ozlabs.org>
> *Subject: *Multiple BMCs in a system: IPMB? Redfish? MCTP?
>
>
>
> I've talked with some people a while back (long while back) about multiple
> BMCs in a system.  Either for redundancy or managing separate parts of a
> system.   I'm wondering what other people are thinking in this area if at
> all.
>
>
>
> We are considering similar designs and I'm looking into options for
> BMC-BMC communications.  Some BMCs may not be externally accessible. Here
> are some options that we've looked at:
>
>    1. i2c/IPMB
>    2. usbnet/Redfish
>    3. i2c/MCTP/PLDM or something else?
>    4. internal network via switch chip/Redfish or MCTP
>
>  I'd like to reduce our use of IPMI so I want to avoid (1).
>
>
>
> ----------
> Nancy
>

[-- Attachment #2: Type: text/html, Size: 3282 bytes --]

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Nancy Yuen]

[-- Attachment #1: Type: text/plain, Size: 2090 bytes --]

Neeraj, that's a great question.  I'm not sure we have a good answer in
some cases.  I think the closest analog in general (for us) is multiple
blades with their own BMCs in a chassis.

----------
Nancy


On Wed, Apr 29, 2020 at 5:23 PM Neeraj Ladkani <neladk@microsoft.com> wrote:

> Redfish defines computer system to be multiple chassis or multiple blades
> within a chassis.  It would be good to clarify if you are referring to
> “System” as:
>
>
>
>    - Multiple chassis with their own BMCs ( Storage, Compute, GPU etc )
>    - Multiple Blades with their own BMC within a chassis.
>
>
>
> Neeraj
>
>
>
> *From:* openbmc <openbmc-bounces+neladk=microsoft.com@lists.ozlabs.org> *On
> Behalf Of *Vijay Khemka
> *Sent:* Wednesday, April 29, 2020 5:16 PM
> *To:* Nancy Yuen <yuenn@google.com>; OpenBMC Maillist <
> openbmc@lists.ozlabs.org>
> *Subject:* [EXTERNAL] Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?
>
>
>
> Hi Nancy,
>
> We are currently using (1) in our current multi host design. Option (3)
> also looks good.
>
>
>
> Regards
>
> -Vijay
>
>
>
> *From: *openbmc <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org> on
> behalf of Nancy Yuen <yuenn@google.com>
> *Date: *Wednesday, April 29, 2020 at 3:53 PM
> *To: *OpenBMC Maillist <openbmc@lists.ozlabs.org>
> *Subject: *Multiple BMCs in a system: IPMB? Redfish? MCTP?
>
>
>
> I've talked with some people a while back (long while back) about multiple
> BMCs in a system.  Either for redundancy or managing separate parts of a
> system.   I'm wondering what other people are thinking in this area if at
> all.
>
>
>
> We are considering similar designs and I'm looking into options for
> BMC-BMC communications.  Some BMCs may not be externally accessible. Here
> are some options that we've looked at:
>
>    1. i2c/IPMB
>    2. usbnet/Redfish
>    3. i2c/MCTP/PLDM or something else?
>    4. internal network via switch chip/Redfish or MCTP
>
>  I'd like to reduce our use of IPMI so I want to avoid (1).
>
>
>
> ----------
> Nancy
>

[-- Attachment #2: Type: text/html, Size: 4722 bytes --]

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Vijay Khemka]

[-- Attachment #1: Type: text/plain, Size: 2012 bytes --]

Nancy,
We have multiple hosts which has a small controller called BIC (Bridge IC). This BIC on each host connected via i2c to main BMC and communicates through ipmi messages to BMC over IPMB/i2c.

Regards
-Vijay

From: Nancy Yuen <yuenn@google.com>
Date: Wednesday, April 29, 2020 at 5:25 PM
To: Vijay Khemka <vijaykhemka@fb.com>
Cc: OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

Neeraj, I was not considering aggregation in this case, just having the intermediate BMC "route".

Vijay, thanks.  I was wondering what your applications for IPMB were for.  What's the rationale for using IPMB vs something else?  In your multihost system, one BMC supports multiple host CPUs?  Are there also multiple BMCs?

----------
Nancy


On Wed, Apr 29, 2020 at 5:15 PM Vijay Khemka <vijaykhemka@fb.com<mailto:vijaykhemka@fb.com>> wrote:
Hi Nancy,
We are currently using (1) in our current multi host design. Option (3) also looks good.

Regards
-Vijay

From: openbmc <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org<mailto:fb.com@lists.ozlabs.org>> on behalf of Nancy Yuen <yuenn@google.com<mailto:yuenn@google.com>>
Date: Wednesday, April 29, 2020 at 3:53 PM
To: OpenBMC Maillist <openbmc@lists.ozlabs.org<mailto:openbmc@lists.ozlabs.org>>
Subject: Multiple BMCs in a system: IPMB? Redfish? MCTP?

I've talked with some people a while back (long while back) about multiple BMCs in a system.  Either for redundancy or managing separate parts of a system.   I'm wondering what other people are thinking in this area if at all.

We are considering similar designs and I'm looking into options for BMC-BMC communications.  Some BMCs may not be externally accessible. Here are some options that we've looked at:

  1.  i2c/IPMB
  2.  usbnet/Redfish
  3.  i2c/MCTP/PLDM or something else?
  4.  internal network via switch chip/Redfish or MCTP
 I'd like to reduce our use of IPMI so I want to avoid (1).

----------
Nancy

[-- Attachment #2: Type: text/html, Size: 7020 bytes --]

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Nancy Yuen]

[-- Attachment #1: Type: text/plain, Size: 2229 bytes --]

Vijay, I wonder if IPMB would still be used in later generations or would
something like MCTP be considered?
----------
Nancy


On Wed, Apr 29, 2020 at 5:42 PM Vijay Khemka <vijaykhemka@fb.com> wrote:

> Nancy,
>
> We have multiple hosts which has a small controller called BIC (Bridge
> IC). This BIC on each host connected via i2c to main BMC and communicates
> through ipmi messages to BMC over IPMB/i2c.
>
>
>
> Regards
>
> -Vijay
>
>
>
> *From: *Nancy Yuen <yuenn@google.com>
> *Date: *Wednesday, April 29, 2020 at 5:25 PM
> *To: *Vijay Khemka <vijaykhemka@fb.com>
> *Cc: *OpenBMC Maillist <openbmc@lists.ozlabs.org>
> *Subject: *Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?
>
>
>
> Neeraj, I was not considering aggregation in this case, just having the
> intermediate BMC "route".
>
>
>
> Vijay, thanks.  I was wondering what your applications for IPMB were for.
> What's the rationale for using IPMB vs something else?  In your multihost
> system, one BMC supports multiple host CPUs?  Are there also multiple BMCs?
>
>
> ----------
> Nancy
>
>
>
>
>
> On Wed, Apr 29, 2020 at 5:15 PM Vijay Khemka <vijaykhemka@fb.com> wrote:
>
> Hi Nancy,
>
> We are currently using (1) in our current multi host design. Option (3)
> also looks good.
>
>
>
> Regards
>
> -Vijay
>
>
>
> *From: *openbmc <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org> on
> behalf of Nancy Yuen <yuenn@google.com>
> *Date: *Wednesday, April 29, 2020 at 3:53 PM
> *To: *OpenBMC Maillist <openbmc@lists.ozlabs.org>
> *Subject: *Multiple BMCs in a system: IPMB? Redfish? MCTP?
>
>
>
> I've talked with some people a while back (long while back) about multiple
> BMCs in a system.  Either for redundancy or managing separate parts of a
> system.   I'm wondering what other people are thinking in this area if at
> all.
>
>
>
> We are considering similar designs and I'm looking into options for
> BMC-BMC communications.  Some BMCs may not be externally accessible. Here
> are some options that we've looked at:
>
>    1. i2c/IPMB
>    2. usbnet/Redfish
>    3. i2c/MCTP/PLDM or something else?
>    4. internal network via switch chip/Redfish or MCTP
>
>  I'd like to reduce our use of IPMI so I want to avoid (1).
>
>
>
> ----------
> Nancy
>
>

[-- Attachment #2: Type: text/html, Size: 5518 bytes --]

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Vijay Khemka]

[-- Attachment #1: Type: text/plain, Size: 2820 bytes --]

Nancy,
As IPMI is still a legacy so ipmb will still be used I guess 😊. Once MCTP starts getting supported widely and they become more stable and generic then it can be used. But currently it is still a challenge to use across all devices.

-Vijay

From: Nancy Yuen <yuenn@google.com>
Date: Wednesday, April 29, 2020 at 6:32 PM
To: Vijay Khemka <vijaykhemka@fb.com>
Cc: OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

Vijay, I wonder if IPMB would still be used in later generations or would something like MCTP be considered?
----------
Nancy


On Wed, Apr 29, 2020 at 5:42 PM Vijay Khemka <vijaykhemka@fb.com<mailto:vijaykhemka@fb.com>> wrote:
Nancy,
We have multiple hosts which has a small controller called BIC (Bridge IC). This BIC on each host connected via i2c to main BMC and communicates through ipmi messages to BMC over IPMB/i2c.

Regards
-Vijay

From: Nancy Yuen <yuenn@google.com<mailto:yuenn@google.com>>
Date: Wednesday, April 29, 2020 at 5:25 PM
To: Vijay Khemka <vijaykhemka@fb.com<mailto:vijaykhemka@fb.com>>
Cc: OpenBMC Maillist <openbmc@lists.ozlabs.org<mailto:openbmc@lists.ozlabs.org>>
Subject: Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

Neeraj, I was not considering aggregation in this case, just having the intermediate BMC "route".

Vijay, thanks.  I was wondering what your applications for IPMB were for.  What's the rationale for using IPMB vs something else?  In your multihost system, one BMC supports multiple host CPUs?  Are there also multiple BMCs?

----------
Nancy


On Wed, Apr 29, 2020 at 5:15 PM Vijay Khemka <vijaykhemka@fb.com<mailto:vijaykhemka@fb.com>> wrote:
Hi Nancy,
We are currently using (1) in our current multi host design. Option (3) also looks good.

Regards
-Vijay

From: openbmc <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org<mailto:fb.com@lists.ozlabs.org>> on behalf of Nancy Yuen <yuenn@google.com<mailto:yuenn@google.com>>
Date: Wednesday, April 29, 2020 at 3:53 PM
To: OpenBMC Maillist <openbmc@lists.ozlabs.org<mailto:openbmc@lists.ozlabs.org>>
Subject: Multiple BMCs in a system: IPMB? Redfish? MCTP?

I've talked with some people a while back (long while back) about multiple BMCs in a system.  Either for redundancy or managing separate parts of a system.   I'm wondering what other people are thinking in this area if at all.

We are considering similar designs and I'm looking into options for BMC-BMC communications.  Some BMCs may not be externally accessible. Here are some options that we've looked at:

  1.  i2c/IPMB
  2.  usbnet/Redfish
  3.  i2c/MCTP/PLDM or something else?
  4.  internal network via switch chip/Redfish or MCTP
 I'd like to reduce our use of IPMI so I want to avoid (1).

----------
Nancy

[-- Attachment #2: Type: text/html, Size: 9995 bytes --]

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Deepak Kodihalli]

On 30/04/20 4:21 am, Nancy Yuen wrote:
> I've talked with some people a while back (long while back) about 
> multiple BMCs in a system.  Either for redundancy or managing separate 
> parts of a system.   I'm wondering what other people are thinking in 
> this area if at all.
> 
> We are considering similar designs and I'm looking into options for 
> BMC-BMC communications.  Some BMCs may not be externally 
> accessible. Here are some options that we've looked at:
> 
>  1. i2c/IPMB
>  2. usbnet/Redfish
>  3. i2c/MCTP/PLDM or something else?
>  4. internal network via switch chip/Redfish or MCTP
> 
>   I'd like to reduce our use of IPMI so I want to avoid (1).
> 
> ----------
> Nancy

Hi Nancy,

I think it depends on whether the BMCs need to talk to each other for 
platform management, or if they manage their own hosts and we need one 
of the BMCs to broadcast out of band requests and aggregate responses.

For the former I think PLDM over MCTP could be a good fit. This is more 
of an "inband" use-case in my opinion so I'm not sure if Redfish is well 
suited. For the latter, a Redfish based aggregation is a good option.

Regards,
Deepak

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Andrew Jeffery]

On Thu, 30 Apr 2020, at 17:08, Deepak Kodihalli wrote:
> On 30/04/20 4:21 am, Nancy Yuen wrote:
> > I've talked with some people a while back (long while back) about 
> > multiple BMCs in a system.  Either for redundancy or managing separate 
> > parts of a system.   I'm wondering what other people are thinking in 
> > this area if at all.
> > 
> > We are considering similar designs and I'm looking into options for 
> > BMC-BMC communications.  Some BMCs may not be externally 
> > accessible. Here are some options that we've looked at:
> > 
> >  1. i2c/IPMB
> >  2. usbnet/Redfish
> >  3. i2c/MCTP/PLDM or something else?
> >  4. internal network via switch chip/Redfish or MCTP
> > 
> >   I'd like to reduce our use of IPMI so I want to avoid (1).
> > 
> > ----------
> > Nancy
> 
> Hi Nancy,
> 
> I think it depends on whether the BMCs need to talk to each other for 
> platform management, or if they manage their own hosts and we need one 
> of the BMCs to broadcast out of band requests and aggregate responses.
> 
> For the former I think PLDM over MCTP could be a good fit. This is more 
> of an "inband" use-case in my opinion so I'm not sure if Redfish is well 
> suited. For the latter, a Redfish based aggregation is a good option.

I think this looks like the right approach. Certainly some thought would be
required in designing the MCTP networks given the (intentionally) limited
number of endpoint IDs.

Andrew

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Nancy Yuen]

[-- Attachment #1: Type: text/plain, Size: 2203 bytes --]

On Mon, May 4, 2020 at 3:31 AM Andrew Jeffery <andrew@aj.id.au> wrote:

>
>
> On Thu, 30 Apr 2020, at 17:08, Deepak Kodihalli wrote:
> > On 30/04/20 4:21 am, Nancy Yuen wrote:
> > > I've talked with some people a while back (long while back) about
> > > multiple BMCs in a system.  Either for redundancy or managing separate
> > > parts of a system.   I'm wondering what other people are thinking in
> > > this area if at all.
> > >
> > > We are considering similar designs and I'm looking into options for
> > > BMC-BMC communications.  Some BMCs may not be externally
> > > accessible. Here are some options that we've looked at:
> > >
> > >  1. i2c/IPMB
> > >  2. usbnet/Redfish
> > >  3. i2c/MCTP/PLDM or something else?
> > >  4. internal network via switch chip/Redfish or MCTP
> > >
> > >   I'd like to reduce our use of IPMI so I want to avoid (1).
> > >
> > > ----------
> > > Nancy
> >
> > Hi Nancy,
> >
> > I think it depends on whether the BMCs need to talk to each other for
> > platform management, or if they manage their own hosts and we need one
> > of the BMCs to broadcast out of band requests and aggregate responses.
> >
> > For the former I think PLDM over MCTP could be a good fit. This is more
> > of an "inband" use-case in my opinion so I'm not sure if Redfish is well
> > suited. For the latter, a Redfish based aggregation is a good option.
>
> I think this looks like the right approach. Certainly some thought would be
> required in designing the MCTP networks given the (intentionally) limited
> number of endpoint IDs.
>
> Andrew
>

Hi Andrew, Deepak,

Sorry for the delayed response!

For the purposes of this discussion I have BMCs managing their own host:
BMC3<--->BMC2<--->BMC1/controller<--> external.  I wrote BMC1 but there
maybe some other controller there that isn't strictly a BMC.  BMC2 would be
"passing through" information between BMC1 and BMC3.  But I don't
necessarily want BMC2 to do any form of aggregation from information from
BMC3.  For BMC1/Controller and BMC 2 we are thinking Redfish.

In my case I'm interested in what ways people are thinking BMC2 and BMC3
should be connected and speaking to each other. Is MCTP/PLDM where everyone
is leaning?

[-- Attachment #2: Type: text/html, Size: 3006 bytes --]

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Andrew Jeffery]

On Fri, 8 May 2020, at 05:14, Nancy Yuen wrote:
> 
> 
> On Mon, May 4, 2020 at 3:31 AM Andrew Jeffery <andrew@aj.id.au> wrote:
> > 
> > 
> >  On Thu, 30 Apr 2020, at 17:08, Deepak Kodihalli wrote:
> >  > On 30/04/20 4:21 am, Nancy Yuen wrote:
> >  > > I've talked with some people a while back (long while back) about 
> >  > > multiple BMCs in a system. Either for redundancy or managing separate 
> >  > > parts of a system. I'm wondering what other people are thinking in 
> >  > > this area if at all.
> >  > > 
> >  > > We are considering similar designs and I'm looking into options for 
> >  > > BMC-BMC communications. Some BMCs may not be externally 
> >  > > accessible. Here are some options that we've looked at:
> >  > > 
> >  > > 1. i2c/IPMB
> >  > > 2. usbnet/Redfish
> >  > > 3. i2c/MCTP/PLDM or something else?
> >  > > 4. internal network via switch chip/Redfish or MCTP
> >  > > 
> >  > > I'd like to reduce our use of IPMI so I want to avoid (1).
> >  > > 
> >  > > ----------
> >  > > Nancy
> >  > 
> >  > Hi Nancy,
> >  > 
> >  > I think it depends on whether the BMCs need to talk to each other for 
> >  > platform management, or if they manage their own hosts and we need one 
> >  > of the BMCs to broadcast out of band requests and aggregate responses.
> >  > 
> >  > For the former I think PLDM over MCTP could be a good fit. This is more 
> >  > of an "inband" use-case in my opinion so I'm not sure if Redfish is well 
> >  > suited. For the latter, a Redfish based aggregation is a good option.
> > 
> >  I think this looks like the right approach. Certainly some thought would be
> >  required in designing the MCTP networks given the (intentionally) limited
> >  number of endpoint IDs.
> > 
> >  Andrew
> 
> Hi Andrew, Deepak,
> 
> Sorry for the delayed response!
> 
> For the purposes of this discussion I have BMCs managing their own 
> host: BMC3<--->BMC2<--->BMC1/controller<--> external. I wrote BMC1 but 
> there maybe some other controller there that isn't strictly a BMC. BMC2 
> would be "passing through" information between BMC1 and BMC3. But I 
> don't necessarily want BMC2 to do any form of aggregation from 
> information from BMC3. For BMC1/Controller and BMC 2 we are thinking 
> Redfish. 

So to restate, you're looking to bridge data from BMC1/controller to BMC3
via BMC2?

MCTP caters to this but it probably depends on what transport medium
you were thinking of using. Redfish for BMC1-to-BMC2 suggests networking,
which starts to go outside the use-cases for which MCTP is intended.

Others might be able to comment on whether Redfish is a good fit for a data
bridge.

Andrew

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Vijay Khemka]

[-- Attachment #1: Type: text/plain, Size: 2752 bytes --]



From: openbmc <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org> on behalf of Nancy Yuen <yuenn@google.com>
Date: Thursday, May 7, 2020 at 12:47 PM
To: Andrew Jeffery <andrew@aj.id.au>
Cc: OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?


On Mon, May 4, 2020 at 3:31 AM Andrew Jeffery <andrew@aj.id.au<mailto:andrew@aj.id.au>> wrote:


On Thu, 30 Apr 2020, at 17:08, Deepak Kodihalli wrote:
> On 30/04/20 4:21 am, Nancy Yuen wrote:
> > I've talked with some people a while back (long while back) about
> > multiple BMCs in a system.  Either for redundancy or managing separate
> > parts of a system.   I'm wondering what other people are thinking in
> > this area if at all.
> >
> > We are considering similar designs and I'm looking into options for
> > BMC-BMC communications.  Some BMCs may not be externally
> > accessible. Here are some options that we've looked at:
> >
> >  1. i2c/IPMB
> >  2. usbnet/Redfish
> >  3. i2c/MCTP/PLDM or something else?
> >  4. internal network via switch chip/Redfish or MCTP
> >
> >   I'd like to reduce our use of IPMI so I want to avoid (1).
> >
> > ----------
> > Nancy
>
> Hi Nancy,
>
> I think it depends on whether the BMCs need to talk to each other for
> platform management, or if they manage their own hosts and we need one
> of the BMCs to broadcast out of band requests and aggregate responses.
>
> For the former I think PLDM over MCTP could be a good fit. This is more
> of an "inband" use-case in my opinion so I'm not sure if Redfish is well
> suited. For the latter, a Redfish based aggregation is a good option.

I think this looks like the right approach. Certainly some thought would be
required in designing the MCTP networks given the (intentionally) limited
number of endpoint IDs.

Andrew

Hi Andrew, Deepak,

Sorry for the delayed response!

For the purposes of this discussion I have BMCs managing their own host: BMC3<--->BMC2<--->BMC1/controller<--> external.  I wrote BMC1 but there maybe some other controller there that isn't strictly a BMC.  BMC2 would be "passing through" information between BMC1 and BMC3.  But I don't necessarily want BMC2 to do any form of aggregation from information from BMC3.  For BMC1/Controller and BMC 2 we are thinking Redfish.

In my case I'm interested in what ways people are thinking BMC2 and BMC3 should be connected and speaking to each other. Is MCTP/PLDM where everyone is leaning?

Nancy,
What is connectivity between BMC1, BMC2 and BMC3. Are these exposing any out of band connections. For security reasons, aggregator should use inband connectivity between BMCs and as Deepak said MCTP is good option for inband.




[-- Attachment #2: Type: text/html, Size: 5886 bytes --]

Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Thomaiyar, Richard Marian]

[-- Attachment #1: Type: text/plain, Size: 4197 bytes --]

PICMG has defined certain specifications in this angle (HPM), but those 
basically uses IPMI (PICMG group extension commands)

1. Physical interface (i2c/IPMB) - IPMI

2. LAN Channel - IPMI RMCP+ other Direct stuff for say console access etc.

PICMG started working with DMTF redfish for the purpose of IPMC to IPMC 
communication few years before (but not sure where it stands now)

No one solution will fit for all, and need to consider based on 1. 
Features required (ssh / serial console access / SOL / KVM etc.) and 2. 
Physical interface between the controller (I2C/ LAN / others?) . Nancy / 
Vijay??

Say, with i2c- anything related to SSH / KVM forwarding etc is going to 
be a problem, but with LAN these can be addressed.

With i2c-> IPMB / PLDM  (MCTP over smbus) (with IPMI moving out of 
picture PLDM will be better alternative).

With LAN -> PLDM / Redfish. (Using Redfish has advantage over PLDM for 
RDE, and still other features can be supported)

Apart from this, need to define fail-over management(PICMG defined some) 
/ discovery(MCTP offers some) etc.

Regards,

Richard


On 5/8/2020 10:48 PM, Vijay Khemka wrote:
>
> *From: *openbmc <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org> 
> on behalf of Nancy Yuen <yuenn@google.com>
> *Date: *Thursday, May 7, 2020 at 12:47 PM
> *To: *Andrew Jeffery <andrew@aj.id.au>
> *Cc: *OpenBMC Maillist <openbmc@lists.ozlabs.org>
> *Subject: *Re: Multiple BMCs in a system: IPMB? Redfish? MCTP?
>
> On Mon, May 4, 2020 at 3:31 AM Andrew Jeffery <andrew@aj.id.au 
> <mailto:andrew@aj.id.au>> wrote:
>
>
>
>     On Thu, 30 Apr 2020, at 17:08, Deepak Kodihalli wrote:
>     > On 30/04/20 4:21 am, Nancy Yuen wrote:
>     > > I've talked with some people a while back (long while back) about
>     > > multiple BMCs in a system.  Either for redundancy or managing
>     separate
>     > > parts of a system.   I'm wondering what other people are
>     thinking in
>     > > this area if at all.
>     > >
>     > > We are considering similar designs and I'm looking into
>     options for
>     > > BMC-BMC communications.  Some BMCs may not be externally
>     > > accessible. Here are some options that we've looked at:
>     > >
>     > >  1. i2c/IPMB
>     > >  2. usbnet/Redfish
>     > >  3. i2c/MCTP/PLDM or something else?
>     > >  4. internal network via switch chip/Redfish or MCTP
>     > >
>     > >   I'd like to reduce our use of IPMI so I want to avoid (1).
>     > >
>     > > ----------
>     > > Nancy
>     >
>     > Hi Nancy,
>     >
>     > I think it depends on whether the BMCs need to talk to each
>     other for
>     > platform management, or if they manage their own hosts and we
>     need one
>     > of the BMCs to broadcast out of band requests and aggregate
>     responses.
>     >
>     > For the former I think PLDM over MCTP could be a good fit. This
>     is more
>     > of an "inband" use-case in my opinion so I'm not sure if Redfish
>     is well
>     > suited. For the latter, a Redfish based aggregation is a good
>     option.
>
>     I think this looks like the right approach. Certainly some thought
>     would be
>     required in designing the MCTP networks given the (intentionally)
>     limited
>     number of endpoint IDs.
>
>     Andrew
>
> Hi Andrew, Deepak,
>
> Sorry for the delayed response!
>
> For the purposes of this discussion I have BMCs managing their own 
> host: BMC3<--->BMC2<--->BMC1/controller<--> external.  I wrote BMC1 
> but there maybe some other controller there that isn't strictly a 
> BMC.  BMC2 would be "passing through" information between BMC1 and 
> BMC3. But I don't necessarily want BMC2 to do any form of aggregation 
> from information from BMC3.  For BMC1/Controller and BMC 2 we are 
> thinking Redfish.
>
> In my case I'm interested in what ways people are thinking BMC2 and 
> BMC3 should be connected and speaking to each other. Is MCTP/PLDM 
> where everyone is leaning?
>
> Nancy,
>
> What is connectivity between BMC1, BMC2 and BMC3. Are these exposing 
> any out of band connections. For security reasons, aggregator should 
> use inband connectivity between BMCs and as Deepak said MCTP is good 
> option for inband.
>

[-- Attachment #2: Type: text/html, Size: 9342 bytes --]