reference platforms?

reference platforms?

[Brad Bishop]

I think in the not-too-distant-future we are going to need a concept of reference
platforms.

openbmc/openbmc is essentially the same thing as Poky.  A super-repo
of other upstream repos (bitbake, oe-core, etc…).  Our upstream repos
just happen to be different things like poky, meta-virtualization, meta-phosphor,
meta-aspeed, etc.

But the similarity ends when you look at the platforms supported.  Poky
has a set of reference platforms.  Any other platforms or distros using
Poky are not the Yocto projects concern - they are completely maintained
by the project that uses Poky (like us!).

Thus far we have not really put any filter on what machines you get
when you clone openbmc/openbmc.  I think that needs to change to a model
like the one the Yocto project uses.  The clear benefits I see to that
are:

- It addresses a scaling problem in the openbmc/openbmc repository.  We
are on track to repeat the mistakes of the openembedded project that led
to the formation of Poky in the first place.  That is, we cannot add new
platform and BSP layers indefinitely.  At some point it will become
unmanageable.  Part of me wonders if we are already there.

- Testing.  Obviously a developer cannot test a patch on 100 platforms or
10 different SOCs.  So what is the developer expected to test on?  A reference
platform.  Then who tests the other systems?  The maintainers of those
systems.  Adopting the same mode of thinking as Poky makes this distinction
clear.

So what does everyone think of the idea of reference platforms?  Can
they help the OpenBMC project?

So the obvious next question is…what are the metrics for defining reference
platforms?  I predict finding the answer to that will be a challenge :-).

Go ahead and throw crazy ideas out there.  Pay money to the project?  Most
days without a compile failure?  There are about a million things we could
do here - but what does everyone find fair?

thx - brad

Re: reference platforms?

[Sai Dasari]

On 4/12/18, 10:41 AM, "openbmc on behalf of Brad Bishop" <openbmc-bounces+sdasari=fb.com@lists.ozlabs.org on behalf of bradleyb@fuzziesquirrel.com> wrote:

    I think in the not-too-distant-future we are going to need a concept of reference
    platforms.
    
    openbmc/openbmc is essentially the same thing as Poky.  A super-repo
    of other upstream repos (bitbake, oe-core, etc…).  Our upstream repos
    just happen to be different things like poky, meta-virtualization, meta-phosphor,
    meta-aspeed, etc.
    
    But the similarity ends when you look at the platforms supported.  Poky
    has a set of reference platforms.  Any other platforms or distros using
    Poky are not the Yocto projects concern - they are completely maintained
    by the project that uses Poky (like us!).
    
    Thus far we have not really put any filter on what machines you get
    when you clone openbmc/openbmc.  I think that needs to change to a model
    like the one the Yocto project uses.  The clear benefits I see to that
    are:
    
    - It addresses a scaling problem in the openbmc/openbmc repository.  We
    are on track to repeat the mistakes of the openembedded project that led
    to the formation of Poky in the first place.  That is, we cannot add new
    platform and BSP layers indefinitely.  At some point it will become
    unmanageable.  Part of me wonders if we are already there.
>>To understand more, is the proposal to remove the bsp/machines layers i.e. meta-openbmc-bsp and meta-openbmc-machines? I wonder how to find the real BSP/machines that are supported by OpenBMC. Does each openbmc adopter encouraged to maintains independent github repo with recipes needed to bring in the openbmc code and build for their supported machines? 

    - Testing.  Obviously a developer cannot test a patch on 100 platforms or
    10 different SOCs.  So what is the developer expected to test on?  A reference
    platform.  Then who tests the other systems?  The maintainers of those
    systems.  Adopting the same mode of thinking as Poky makes this distinction
    clear.
    
    So what does everyone think of the idea of reference platforms?  Can
    they help the OpenBMC project?
    
    So the obvious next question is…what are the metrics for defining reference
    platforms?  I predict finding the answer to that will be a challenge :-).
    
    Go ahead and throw crazy ideas out there.  Pay money to the project?  Most
    days without a compile failure?  There are about a million things we could
    do here - but what does everyone find fair?
>> With the nature of OpenBMC supporting features needed for divergent/heterogeneous hardware, it might be hard to define the reference platform(s). One (crazy) idea would be to come up with a minimum acceptance test suite (which can be automated) that validates all the core features and use this for accepting new platforms as reference platforms from community.
    thx - brad

Re: reference platforms?

[Brad Bishop]

> On Apr 12, 2018, at 5:31 PM, Sai Dasari <sdasari@fb.com> wrote:
>> On 4/12/18, 10:41 AM, "Brad Bishop" <bradleyb@fuzziesquirrel.com> wrote:
>>    I think in the not-too-distant-future we are going to need a concept of reference
>>    platforms.
> To understand more, is the proposal to remove the bsp/machines
> layers i.e. meta-openbmc-bsp and meta-openbmc-machines?

Yes and no.  I’m focused on what you get when you clone openbmc/openbmc
and what is automatically regression tested by the project CI when it is
changed.  What is specifically added/removed depends on the implementation.
Here is one possibility:

github.com/openbmc/openbmc:
    meta
    bitbake
    meta-oe
    meta-virtualization
    meta-phosphor
    meta-openbmc-machines/meta-<reference-machine>
    meta-openbmc-bsp/meta-<ref-bsp>

github.com/openbmc/meta-phosphor
github.com/meta-openbmc-machines/meta-<reference-machine>
github.com/openbmc/meta-<nonreference-machine1>
github.com/openbmc/meta-<nonreference-machine2>
github.com/openbmc/meta-<nonreference-bsp>

I’m trying to show that when you clone openbmc/openbmc you get a collection
of other upstreams.  Where those upstreams are hosted doesn’t matter (to me
anyway).  I’m also trying to show that we can host things on the openbmc Github
organization that aren’t included when you clone openbmc/openbmc.

>  I wonder how to find the real BSP/machines that are supported by OpenBMC.

IMHO for the project to scale we can’t think of it in this way.  I would flip
the question around as:

How to find the machines/BSPs that work with OpenBMC?

In my mind OpenBMC doesn’t elevate or depend on some machines.  Its the other
way around… machines are elevated by their dependency on OpenBMC.  I hope that
makes sense…having trouble articulating what is in my head here.

What does that mean exactly?  Something like you can do this and it works:
  git clone github.com/openbmc/openbmc
  cd openbmc
  git clone github.com/<awesome-company>/my-obmc-machine-layer
  bitbake obmc-phosphor-image

And if you ask the question in this way, I’m not sure what the answer is.
How do you find all the layers that work with Poky?  Can you elaborate on
why the answer might be important to someone?  It is a fair question but I
can tell you that it isn't one that IBM for example needs answered in order
for the project meet its product development needs.

Put another way, we _could_ build the answer to the question into the
core of the project, but why (and at what cost to the project’s ability
to scale)?  Someone could also just host a list somewhere, or maybe a
layer-index like OE has?

> Does each openbmc adopter encouraged to maintains independent github repo with recipes needed to bring in the openbmc code and build for their supported machines? 

I think so.  The specifics of what the adopter does/maintains are up to
the adopter.  What are some pros/cons of a model like that?  This is what
most everyone is doing already.  They fork some part of the project (or
all of it), build on it (mostly in a machine layer?  hopefully?)  and then
maybe/maybe not contribute something of what they did back to the upstream
project.  It is my guess that this is the usual way in which product
development teams use FLOSS.

>>    Go ahead and throw crazy ideas out there.  Pay money to the project?  Most
>>    days without a compile failure?  There are about a million things we could
>>    do here - but what does everyone find fair?
> With the nature of OpenBMC supporting features needed for divergent/heterogeneous hardware, it might be hard to define the reference platform(s).


Agreed it will be tough.  Although for the heterogeneous aspect of it
could it be as simple as we just have some references per class of
thing-being-managed-by-a-bmc?

> One (crazy) idea would be to come up with a minimum acceptance test suite (which can be automated) that validates all the core features and use this for accepting new platforms as reference platforms from community.

This is the opposite of crazy Sai :-)

I like it, with a tweak/clarification.  Remember that one of the points of
a reference platform is to enable regression testing for the project.  So
when someone changes the common code in phosphor-hwmon for instance, we need
a way to test that it isn’t going to break as many OpenBMC users as we can,
without needing to test all their platforms for them.

So my clarification is that the acceptance test would not measure how well
a machine meets end-user requirements, but rather how well a machine makes
use of the project’s common code.  Maybe a metric measured by the test could
be something like least number of deviations from the defaults?

Re: reference platforms?

[Brad Bishop]

> On Apr 12, 2018, at 1:40 PM, Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:
> 
> Go ahead and throw crazy ideas out there.  Pay money to the project?  Most
> days without a compile failure?  There are about a million things we could
> do here - but what does everyone find fair?

Another idea - put it to a community vote?  Who gets a vote?  Developers might(should?)
know which platforms test their code best?

Re: reference platforms?

[Sai Dasari]

    
    
    What does that mean exactly?  Something like you can do this and it works:
      git clone github.com/openbmc/openbmc
      cd openbmc
      git clone github.com/<awesome-company>/my-obmc-machine-layer
      bitbake obmc-phosphor-image
    
    And if you ask the question in this way, I’m not sure what the answer is.
    How do you find all the layers that work with Poky?  Can you elaborate on
    why the answer might be important to someone?  It is a fair question but I
    can tell you that it isn't one that IBM for example needs answered in order
    for the project meet its product development needs.
Your proposed changes are good from scaling the development. My original question was from the end-user point of view who acquires bunch of OCP machines and tries to find a way to build OpenBMC for those machines. I was wondering if we can point those end-users to one unified repo @ "github.com/openbmc" and allow them to build OpenBMC binary for that platform. The other option would be for them to let them figure out the repo for that machine e.g. "github.com/<awesome-company>". I think any of these methods will work in practice. 
    
    Put another way, we _could_ build the answer to the question into the
    core of the project, but why (and at what cost to the project’s ability
    to scale)?  Someone could also just host a list somewhere, or maybe a
    layer-index like OE has?
    
    > Does each openbmc adopter encouraged to maintains independent github repo with recipes needed to bring in the openbmc code and build for their supported machines? 
    
    I think so.  The specifics of what the adopter does/maintains are up to
    the adopter.  What are some pros/cons of a model like that?  This is what
    most everyone is doing already.  They fork some part of the project (or
    all of it), build on it (mostly in a machine layer?  hopefully?)  and then
    maybe/maybe not contribute something of what they did back to the upstream
    project.  It is my guess that this is the usual way in which product
    development teams use FLOSS.
I am sure FLOSS allows forks, but for the benefit of community, it would be good to encourage contribution to the upstream openbmc repo. While it helps to scale to separate machine layer repo outside of upstream, I have concern that the adopter's focus would shift away from upstream to that external repo. This might have side-effect of duplicate work in independent repos (might have good intention of upstreaming later), as opposed to ideal work flow (in my mind) of {'upstream-first'-> cherry pick to external repo}. 
    
    > One (crazy) idea would be to come up with a minimum acceptance test suite (which can be automated) that validates all the core features and use this for accepting new platforms as reference platforms from community.
    
    This is the opposite of crazy Sai :-)
    
    I like it, with a tweak/clarification.  Remember that one of the points of
    a reference platform is to enable regression testing for the project.  So
    when someone changes the common code in phosphor-hwmon for instance, we need
    a way to test that it isn’t going to break as many OpenBMC users as we can,
    without needing to test all their platforms for them.
    
    So my clarification is that the acceptance test would not measure how well
    a machine meets end-user requirements, but rather how well a machine makes
    use of the project’s common code.  Maybe a metric measured by the test could
    be something like least number of deviations from the defaults?
Looks good metric to me for various application level service/daemons. Still not sure how the code changes specific to hardware can be regression tested e.g. KCS vs. BT, IPMB, NC-SI etc.
    
    
    

Re: reference platforms?

[Avi Fishman]

I am sending this on behalf of Uri Trichter, Nuvoton BMC project manager:

We suggest to consider a "manageability connector" on the Reference
Board such that every BMC SOC can test it's compatibility vs the
reference board functionality.
We understand, this doesn’t resolve the different HOST systems, but,
the focus of this group is BMC and therefore we see value in the
proposal.

In recent OCP summit,  a concept project called RunBMC presented this idea.
We analyzed RunBMC proposal and think that their functional signal
coverage is good, but the connector selected is less practical.
So, we are looking to define an SO-DIMM based edge connector that
contain similar++  functional signals in a usable form factor.

If OpenBMC form, will decide to go with a reference platform +
manageability connector on MB, we will make this specification
available for comments.

On Thu, Apr 12, 2018 at 8:40 PM, Brad Bishop
<bradleyb@fuzziesquirrel.com> wrote:
> I think in the not-too-distant-future we are going to need a concept of reference
> platforms.
>
> openbmc/openbmc is essentially the same thing as Poky.  A super-repo
> of other upstream repos (bitbake, oe-core, etc…).  Our upstream repos
> just happen to be different things like poky, meta-virtualization, meta-phosphor,
> meta-aspeed, etc.
>
> But the similarity ends when you look at the platforms supported.  Poky
> has a set of reference platforms.  Any other platforms or distros using
> Poky are not the Yocto projects concern - they are completely maintained
> by the project that uses Poky (like us!).
>
> Thus far we have not really put any filter on what machines you get
> when you clone openbmc/openbmc.  I think that needs to change to a model
> like the one the Yocto project uses.  The clear benefits I see to that
> are:
>
> - It addresses a scaling problem in the openbmc/openbmc repository.  We
> are on track to repeat the mistakes of the openembedded project that led
> to the formation of Poky in the first place.  That is, we cannot add new
> platform and BSP layers indefinitely.  At some point it will become
> unmanageable.  Part of me wonders if we are already there.
>
> - Testing.  Obviously a developer cannot test a patch on 100 platforms or
> 10 different SOCs.  So what is the developer expected to test on?  A reference
> platform.  Then who tests the other systems?  The maintainers of those
> systems.  Adopting the same mode of thinking as Poky makes this distinction
> clear.
>
> So what does everyone think of the idea of reference platforms?  Can
> they help the OpenBMC project?
>
> So the obvious next question is…what are the metrics for defining reference
> platforms?  I predict finding the answer to that will be a challenge :-).
>
> Go ahead and throw crazy ideas out there.  Pay money to the project?  Most
> days without a compile failure?  There are about a million things we could
> do here - but what does everyone find fair?
>
> thx - brad

Re: reference platforms?

[Brad Bishop]

Hey Sai

Sorry it has taken me so long to respond.  I hope others will weigh in
with their ideas on this topic.

> On Apr 13, 2018, at 7:55 PM, Sai Dasari <sdasari@fb.com> wrote:
> 
> Your proposed changes are good from scaling the development. My original question was from the end-user point of view who acquires bunch of OCP machines and tries to find a way to build OpenBMC for those machines. I was wondering if we can point those end-users to one unified repo @ "github.com/openbmc" and allow them to build OpenBMC binary for that platform. The other option would be for them to let them figure out the repo for that machine e.g. "github.com/<awesome-company>". I think any of these methods will work in practice. 

Yep, I understand the goal.  I don’t have any great ideas off the top
of my head but I don’t think it should be too hard to come up with a
solution that meets both needs - a scalable development process _and_
a one-stop-shop for OCP platform builders (or others).

> 
>> Put another way, we _could_ build the answer to the question into the
>> core of the project, but why (and at what cost to the project’s ability
>> to scale)?  Someone could also just host a list somewhere, or maybe a
>> layer-index like OE has?
>> 
>>> Does each openbmc adopter encouraged to maintains independent github repo with recipes needed to bring in the openbmc code and build for their supported machines? 
>> 
>>    I think so.  The specifics of what the adopter does/maintains are up to
>>    the adopter.  What are some pros/cons of a model like that?  This is what
>>    most everyone is doing already.  They fork some part of the project (or
>>    all of it), build on it (mostly in a machine layer?  hopefully?)  and then
>>    maybe/maybe not contribute something of what they did back to the upstream
>>    project.  It is my guess that this is the usual way in which product
>>    development teams use FLOSS.
> I am sure FLOSS allows forks, but for the benefit of community, it would be good to encourage contribution to the upstream openbmc repo. While it helps to scale to separate machine layer repo outside of upstream, I have concern that the adopter's focus would shift away from upstream to that external repo.  This might have side-effect of duplicate work in independent repos (might have good intention of upstreaming later), as opposed to ideal work flow (in my mind) of {'upstream-first'-> cherry pick to external repo}. 

I think this valid concern could mostly be mitigated by careful selection of
maintainers and how we structure content amongst repositories.  I will try
and elaborate on how I think it could work with some pictures in the near
future.

> 
>>> One (crazy) idea would be to come up with a minimum acceptance test suite (which can be automated) that validates all the core features and use this for accepting new platforms as reference platforms from community.
>> 
>>    So my clarification is that the acceptance test would not measure how well
>>    a machine meets end-user requirements, but rather how well a machine makes
>>    use of the project’s common code.  Maybe a metric measured by the test could
>>    be something like least number of deviations from the defaults?
> Looks good metric to me for various application level service/daemons. Still not sure how the code changes specific to hardware can be regression tested e.g. KCS vs. BT, IPMB, NC-SI etc.

Re: reference platforms?

[krtaylor]

On 4/30/18 2:34 PM, Brad Bishop wrote:
> Hey Sai
>
> Sorry it has taken me so long to respond.  I hope others will weigh in
> with their ideas on this topic.
>
>> On Apr 13, 2018, at 7:55 PM, Sai Dasari <sdasari@fb.com> wrote:
>>
>> Your proposed changes are good from scaling the development. My original question was from the end-user point of view who acquires bunch of OCP machines and tries to find a way to build OpenBMC for those machines. I was wondering if we can point those end-users to one unified repo @ "github.com/openbmc" and allow them to build OpenBMC binary for that platform. The other option would be for them to let them figure out the repo for that machine e.g. "github.com/<awesome-company>". I think any of these methods will work in practice.
> Yep, I understand the goal.  I don’t have any great ideas off the top
> of my head but I don’t think it should be too hard to come up with a
> solution that meets both needs - a scalable development process _and_
> a one-stop-shop for OCP platform builders (or others).
It has been said before, but I like the layered approach where obmc is 
the base and then the machine specifics layer on top of that. So the 
build process would be clone obmc, then <awesome-company> repo and 
build. This will require good third-party CI by the <awesome-company>, 
as well as clean modular boundaries for machine support.
>
>>> Put another way, we _could_ build the answer to the question into the
>>> core of the project, but why (and at what cost to the project’s ability
>>> to scale)?  Someone could also just host a list somewhere, or maybe a
>>> layer-index like OE has?
>>>
>>>> Does each openbmc adopter encouraged to maintains independent github repo with recipes needed to bring in the openbmc code and build for their supported machines?
>>>     I think so.  The specifics of what the adopter does/maintains are up to
>>>     the adopter.  What are some pros/cons of a model like that?  This is what
>>>     most everyone is doing already.  They fork some part of the project (or
>>>     all of it), build on it (mostly in a machine layer?  hopefully?)  and then
>>>     maybe/maybe not contribute something of what they did back to the upstream
>>>     project.  It is my guess that this is the usual way in which product
>>>     development teams use FLOSS.
>> I am sure FLOSS allows forks, but for the benefit of community, it would be good to encourage contribution to the upstream openbmc repo. While it helps to scale to separate machine layer repo outside of upstream, I have concern that the adopter's focus would shift away from upstream to that external repo.  This might have side-effect of duplicate work in independent repos (might have good intention of upstreaming later), as opposed to ideal work flow (in my mind) of {'upstream-first'-> cherry pick to external repo}.
> I think this valid concern could mostly be mitigated by careful selection of
> maintainers and how we structure content amongst repositories.  I will try
> and elaborate on how I think it could work with some pictures in the near
> future.
>
>>>> One (crazy) idea would be to come up with a minimum acceptance test suite (which can be automated) that validates all the core features and use this for accepting new platforms as reference platforms from community.
>>>     So my clarification is that the acceptance test would not measure how well
>>>     a machine meets end-user requirements, but rather how well a machine makes
>>>     use of the project’s common code.  Maybe a metric measured by the test could
>>>     be something like least number of deviations from the defaults?
Yes, a good test suite for the base, as well as third-party testing for 
company machine support, testing both sides and reporting back. Sounds 
like the basis and requirements for a Test Working Group.
>> Looks good metric to me for various application level service/daemons. Still not sure how the code changes specific to hardware can be regression tested e.g. KCS vs. BT, IPMB, NC-SI etc.
Kurt Taylor (krtaylor)