Redfish OEM commands in OpenBMC

Redfish OEM commands in OpenBMC

[Andrew Geissler]

Greetings,

As more and more of us are implementing Redfish within OpenBMC we are starting
to run into situations where the function we need to implement is not available
in the current Redfish schemas.

We've been joining DMTF workgroups and starting to get familiar with the process
but, if OpenBMC upstreaming is considered slow, it's looking like upstreaming
to the DMTF could be even slower.

So my question is, are we taking a no OEM policy within OpenBMC? Or are we going
to take a more flexible approach. Similar to openbmc/linux, as long as you're
working upstream with the DMTF on your changes, OEM would be acceptable?

From what I've seen so far, it looks like a lot of companies have gone the OEM
route initially. They then use that experience to guide the discussion with the
DMTF workgroups.

There could be some gray areas. For example, I want to add a Priority
property to the UpdateService [1]. Maybe this would be ok to carry as an OEM
for a bit? Adriana wants to propose a whole new backup and restore design [2].
Maybe that needs a bit more traction upstream to ensure it's going in the right
direction for something DMTF would approve first? If we start limiting ourselves
as a community on what DMTF has approved, I think we're going to struggle to
deliver certain new functions.

Andrew

[1] https://github.com/DMTF/Redfish/issues/3357 (can only see with DMTF access)
[2] https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/18163

Re: Redfish OEM commands in OpenBMC - OpenBMC extensions to the Redfish standard

[Joseph Reynolds]

On 2019-04-16 14:02, Andrew Geissler wrote:
> Greetings,
> 
> As more and more of us are implementing Redfish within OpenBMC we are 
> starting
> to run into situations where the function we need to implement is not 
> available
> in the current Redfish schemas.
> 
> We've been joining DMTF workgroups and starting to get familiar with 
> the process
> but, if OpenBMC upstreaming is considered slow, it's looking like 
> upstreaming
> to the DMTF could be even slower.
> 
> So my question is, are we taking a no OEM policy within OpenBMC? Or are 
> we going
> to take a more flexible approach. Similar to openbmc/linux, as long as 
> you're
> working upstream with the DMTF on your changes, OEM would be 
> acceptable?

Good questions!

A third approach: At the OpenBMC project level, we document, discuss, 
and agree on "OpenBMC extensions to the Redfish standard".  Once 
approved, we can simultaneously (a) implement the extensions and (b) 
attempt to upstream them to DMTF.  ==> If the extensions are accepted, 
celebrate!  If not, ... we'll have more work.

- Joseph

> From what I've seen so far, it looks like a lot of companies have gone 
> the OEM
> route initially. They then use that experience to guide the discussion 
> with the
> DMTF workgroups.
> 
> There could be some gray areas. For example, I want to add a Priority
> property to the UpdateService [1]. Maybe this would be ok to carry as 
> an OEM
> for a bit? Adriana wants to propose a whole new backup and restore 
> design [2].
> Maybe that needs a bit more traction upstream to ensure it's going in 
> the right
> direction for something DMTF would approve first? If we start limiting 
> ourselves
> as a community on what DMTF has approved, I think we're going to 
> struggle to
> deliver certain new functions.
> 
> Andrew
> 
> [1] https://github.com/DMTF/Redfish/issues/3357 (can only see with DMTF 
> access)
> [2] https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/18163

Re: Redfish OEM commands in OpenBMC

[Andrew Jeffery]

On Wed, 17 Apr 2019, at 04:33, Andrew Geissler wrote:
> Greetings,
> 
> As more and more of us are implementing Redfish within OpenBMC we are starting
> to run into situations where the function we need to implement is not available
> in the current Redfish schemas.
> 
> We've been joining DMTF workgroups and starting to get familiar with the process
> but, if OpenBMC upstreaming is considered slow, it's looking like upstreaming
> to the DMTF could be even slower.
> 
> So my question is, are we taking a no OEM policy within OpenBMC? Or are we going
> to take a more flexible approach. Similar to openbmc/linux, as long as you're
> working upstream with the DMTF on your changes, OEM would be acceptable?

I'd be more cautious here - breaking the OpenBMC kernel ABI due to carrying
patches that are not yet upstreamed isn't great, but we can adapt userspace
without breaking the external interface to OpenBMC. Breaking the external
interface to OpenBMC is much worse from a user perspective. Essentially, you
need to be prepared to support the OEM extensions well beyond standardisation
of the non-OEM equivalent.

RFC6648[1] seems informative here:

Deprecating the "X-" Prefix and Similar Constructs in Application Protocols

At least, that's my $0.02.

Andrew

[1] https://tools.ietf.org/html/rfc6648

> 
> From what I've seen so far, it looks like a lot of companies have gone the OEM
> route initially. They then use that experience to guide the discussion with the
> DMTF workgroups.
> 
> There could be some gray areas. For example, I want to add a Priority
> property to the UpdateService [1]. Maybe this would be ok to carry as an OEM
> for a bit? Adriana wants to propose a whole new backup and restore design [2].
> Maybe that needs a bit more traction upstream to ensure it's going in the right
> direction for something DMTF would approve first? If we start limiting ourselves
> as a community on what DMTF has approved, I think we're going to struggle to
> deliver certain new functions.
> 
> Andrew
> 
> [1] https://github.com/DMTF/Redfish/issues/3357 (can only see with DMTF access)
> [2] https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/18163
>

Re: Redfish OEM commands in OpenBMC

[Ed Tanous]

First of all, thank you for asking this question.  It's been a long time
coming.  I'm for one surprised we've avoided it for so long.  With that
said, lets dig in.

On 4/16/19 12:02 PM, Andrew Geissler wrote:
> Greetings,
> 
> As more and more of us are implementing Redfish within OpenBMC we are starting
> to run into situations where the function we need to implement is not available
> in the current Redfish schemas.
> 
> We've been joining DMTF workgroups and starting to get familiar with the process
> but, if OpenBMC upstreaming is considered slow, it's looking like upstreaming
> to the DMTF could be even slower.

It's slow because most of the interfaces being added are required to:
1. Prove utility to multiple members of the Redfish community that have
implementations.
2. Provide adequate, and unambiguous documentation on what the property
is for, and how it's intended to be used.
3. Get the associated metadata files in order, to enable the validators
to work.
4. Prototype, and make sure it meets the need.
5. Wait for a merge/release window.

If I look at some of the recent initiatives, certificates, composable
systems, and update, I would argue they're no slower than OpenBMC, they
just require a level of documentation and discussion that's a bit of a
higher bar than a "normal" OpenBMC feature.  As OpenBMC progresses, we
seem to be ratcheting up the rules around required documentation and
discussion, so I wonder if we'll get to that same level at some point as
well.

> 
> So my question is, are we taking a no OEM policy within OpenBMC? Or are we going
> to take a more flexible approach. Similar to openbmc/linux, as long as you're
> working upstream with the DMTF on your changes, OEM would be acceptable?

My overall goal when we first discussed this in the Redfish working
group was that the OpenBMC OEM schemas are common among all OpenBMC
implementations.  Said another way, if we're implementing an Oem schema,
it should be under the "OpenBMC" namespace, and we as a community should
have enough agreement to implement it in a way that works for the people
that need it.  The one exception to this is if we're implementing to
someone elses specification, which might call out specific namespaces
for the specification.

We have 1 OEM schema today for phosphor-pid-control, and it already
causes heartache for people implementing against it.  I'm tempted to
remove it before it actually makes it into an official release, so we
don't have to support it long term.  This is a discussion for another
thread.

> 
> From what I've seen so far, it looks like a lot of companies have gone the OEM
> route initially. They then use that experience to guide the discussion with the
> DMTF workgroups.

Some of the issue with this is that user-facing applications get written
against the OEM implementation, and now that OEM implementation has to
live forever, and support the same API.  This is great for companies
hoping to get lock-in on their products, but I really suspect it's not
what we want for OpenBMC.

The other thing to remember:  This is the public facing API that OpenBMC
exposes to users.  It is forever.  Think of it like the kernel userspace
API, and the guarantees that provides.  With that constraint, the
approach of "going the OEM route initially" doesn't really feel in line
with what OpenBMCs goals are.  I would much rather a standardized
solution that's 12 months after the spec releases, than an OEM solution
12 months early that we have to break often.

> 
> There could be some gray areas. For example, I want to add a Priority
> property to the UpdateService [1]. Maybe this would be ok to carry as an OEM
> for a bit?
Before I give +1 or -1 to the proposal, I've been meaning to really
understand what we think we're solving with priority, and why the
existing redfish mechanisms don't meet the need.  If I understand the
use correctly (which it's very likely I don't) posting to the target URI
property could cause the system to flip image priority.  This is
slightly less granular than an integer priority, but the use case you
gave only had 2 images, so it's very possible I'm missing something.
See above, there's no such thing as "carry as an OEM".

> Adriana wants to propose a whole new backup and restore design [2].
I believe there are redfish mechanisms for arbitrary "settings" type
interfaces that would probably be the recommended way.  To my
understanding, this is how systems currently implement BIOS settings,
which I think has roughly the same use case as BMC settings.  With that
said, this is better discussed in the Redfish forum, as there are more
experts present.


> Maybe that needs a bit more traction upstream to ensure it's going in the right
> direction for something DMTF would approve first? If we start limiting ourselves
> as a community on what DMTF has approved, I think we're going to struggle to
> deliver certain new functions.

From my perspective, there are a lot of development options that would
let us build code that doesn't necessarily get turned on.  Configure
options for implementing examples of new DMTF schemas could certainly be
done which would let people develop out ideas, but not affect OpenBMC
mainline until it's been included in part of the specification.

One thing to remember here: APIs are forever.  I would rather merge a
long-lived API "late" than fight with changing API behavior every release.

> 
> Andrew
> 
> [1] https://github.com/DMTF/Redfish/issues/3357 (can only see with DMTF access)
> [2] https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/18163
> 

I'll answer one of the questions I saw raised on the call: Should we
implement pluggable OEM handlers.

No. (full stop)

In my mind, this defeats most of the of the purpose of having an OpenBMC
community in the first place.  When I ask most of the "consumers" of
OpenBMC why they want to use it, usually the first or second answer is
around "standardization".  They want to be able to jump between hardware
vendors without all their public facing APIs breaking.  If I ask them
why Redfish doesn't meet the need, they talk specifically about OEM
parameters and schemas, and how implementations aren't standard enough
for their needs.

Re: Redfish OEM commands in OpenBMC

[Ed Tanous]

On 4/17/19 10:26 PM, Andrew Jeffery wrote:
> 
> 
> On Wed, 17 Apr 2019, at 04:33, Andrew Geissler wrote:
> 
> I'd be more cautious here - breaking the OpenBMC kernel ABI due to carrying
> patches that are not yet upstreamed isn't great, but we can adapt userspace
> without breaking the external interface to OpenBMC. Breaking the external
> interface to OpenBMC is much worse from a user perspective. Essentially, you
> need to be prepared to support the OEM extensions well beyond standardisation
> of the non-OEM equivalent.

YES! exactly this.

> 
> RFC6648[1] seems informative here:
> 
> Deprecating the "X-" Prefix and Similar Constructs in Application Protocols
> 

I never thought of that as similar, but that's almost exactly the same
idea.  I had no idea that document even existed.

Re: Redfish OEM commands in OpenBMC

[Brad Bishop]

On Mon, Apr 22, 2019 at 09:19:37AM -0700, Ed Tanous wrote:
>I'll answer one of the questions I saw raised on the call: Should we
>implement pluggable OEM handlers.
>
>No. (full stop)

If this is the case, how am I supposed to implement our custom logging
format, PEL?

Re: Redfish OEM commands in OpenBMC

[Ed Tanous]

On 4/23/19 6:51 AM, Brad Bishop wrote:
> On Mon, Apr 22, 2019 at 09:19:37AM -0700, Ed Tanous wrote:
>> I'll answer one of the questions I saw raised on the call: Should we
>> implement pluggable OEM handlers.
>>
>> No. (full stop)
> 
> If this is the case, how am I supposed to implement our custom logging
> format, PEL?

Lots of options are available, while staying within the bounds I gave
above.  Which you choose is likely constrained by what you're trying to
solve by shoehorning in an older logging format ontop of a specification
that already defines a logging format.

1. Attend the DMTF spec meetings (I believe they still have a logging
working group), and bespell how fantastic PEL logs are, and why they
should be the industry standard.  Get industry adoption for a "PelEntry"
and "PelCollection" schema definitions, then implement them in OpenBMC.
If this is done, and it becomes a matter of "they haven't published the
spec yet, but it's final" while waiting for an official DMTF drop, I'm
happy to discuss OpenBMC rollout strategies for bringing the
implementation in ahead of the spec.

2. Use LogEntry and LogService to log your PEL entries like the other
3-4 examples of logging do today.  Given that schema is very broad it
allows you to log essentially whatever you want into the message field.
This would also give the easiest time for people to port, and give you a
"standard Redfish" way to download your PEL logs.

3. (Really not preferred) Use one of the other open source Redfish
servers that supports runtime pluggable endpoints.  The point here is
mostly that all the servers that defined endpoints that were "pluggable"
the way it was described ended up becoming overly complicated, and IMHO
lost a lot of battles because of said complexity.  This is not to say
bmcweb is "simple" by any means.

4. Implement it as an odata endpoint, not necessarily in the Redfish
tree.  If your goal isn't industry compliance and acceptance, you might
consider moving a level up, and simply define your endpoints as odata
endpoints under a different URL tree.  This would depend on your end
goals, that I don't fully understand at this point.

5. Get a percentage OpenBMC community to agree that PEL logging is the
future, and that's the direction we should start going, then define an
OEM endpoint for it under the OpenBMC namespace that we can all find
ways to implement against the backend.

6. Write a maintained, versioned, multi-system implementation spec for
"PEL over Redfish", that defines a PEL logging definition that could be
implemented by any BMC (not just OpenBMC) and requires OEM endpoints.  A
decent example of this is the RSD specification:
https://www.intel.com/content/dam/www/public/us/en/documents/guides/podm-api-spec-v2-4-guide.pdf



Have you asked the same question on Redfish forum?  There are
significantly more experts there, and they might have even better ideas
than I do to improve your standards compliance on what you're trying to
implement.

Re: Redfish OEM commands in OpenBMC

[Brad Bishop]

On Tue, Apr 23, 2019 at 09:27:36AM -0700, Ed Tanous wrote:
>On 4/23/19 6:51 AM, Brad Bishop wrote:
>> On Mon, Apr 22, 2019 at 09:19:37AM -0700, Ed Tanous wrote:
>>> I'll answer one of the questions I saw raised on the call: Should we
>>> implement pluggable OEM handlers.
>>>
>>> No. (full stop)
>>
>> If this is the case, how am I supposed to implement our custom logging
>> format, PEL?
>
>Lots of options are available, while staying within the bounds I gave
>above.  Which you choose is likely constrained by what you're trying to
>solve by shoehorning in an older logging format ontop of a specification
>that already defines a logging format.

In a nutshell, the problem we are trying to solve is enablement of our
support structure.  IBM provides its customers with a deep level of
support.

To do this, IBM service personel require detailed failure information.
A lot of it.  For that we have PEL.

This isn't a new problem - we already solved it long ago.  PEL has long
been the "glue" between our systems and our support infrastructure.  We
are just trying to enable that for ourselves, off to the side,  within
the OpenBMC framework.

>
>1. Attend the DMTF spec meetings (I believe they still have a logging
>working group), and bespell how fantastic PEL logs are, and why they
>should be the industry standard.  Get industry adoption for a "PelEntry"
>and "PelCollection" schema definitions, then implement them in OpenBMC.
>If this is done, and it becomes a matter of "they haven't published the
>spec yet, but it's final" while waiting for an official DMTF drop, I'm
>happy to discuss OpenBMC rollout strategies for bringing the
>implementation in ahead of the spec.

But PEL is highly specific to IBM processes.  It doesn't make any sense
to try to push for standardization of something like that.  I'm not here
to try and standardize the mechanisms that enable server vendors provide
support and service to their customers.

>
>2. Use LogEntry and LogService to log your PEL entries like the other
>3-4 examples of logging do today.  Given that schema is very broad it
>allows you to log essentially whatever you want into the message field.
>This would also give the easiest time for people to port, and give you a
>"standard Redfish" way to download your PEL logs.

This might be feasible.  I just don't know enough about the logging
schema.  I'll have to do some reading.

>
>3. (Really not preferred) Use one of the other open source Redfish
>servers that supports runtime pluggable endpoints.  

I guess option 3 prime is we fork bmcweb, and implement pluggable OEM
callbacks there :wink:

>The point here is mostly that all the servers that defined endpoints
>that were "pluggable" the way it was described ended up becoming overly
>complicated, and IMHO lost a lot of battles because of said complexity.
>This is not to say bmcweb is "simple" by any means.
>
>4. Implement it as an odata endpoint, not necessarily in the Redfish
>tree.  If your goal isn't industry compliance and acceptance, you might
>consider moving a level up, and simply define your endpoints as odata
>endpoints under a different URL tree.  This would depend on your end
>goals, that I don't fully understand at this point.

This is the closest thing to what I had in mind.  But how is this not a
a pluggable OEM handler?  Are you saying pluggable OEM handlers in
bmcweb are OK as long as the route being handled is outside the /redfish
namespace?

>
>5. Get a percentage OpenBMC community to agree that PEL logging is the
>future, and that's the direction we should start going, then define an
>OEM endpoint for it under the OpenBMC namespace that we can all find
>ways to implement against the backend.

I'm not here to try to convince anyone to use PEL.  I have to have it on
my systems and I just want to understand what the path for getting it
there is.

>
>6. Write a maintained, versioned, multi-system implementation spec for
>"PEL over Redfish", that defines a PEL logging definition that could be
>implemented by any BMC (not just OpenBMC) and requires OEM endpoints.  A
>decent example of this is the RSD specification:
>https://www.intel.com/content/dam/www/public/us/en/documents/guides/podm-api-spec-v2-4-guide.pdf

Confused by this one.  Are you saying if we write a spec, we can
implement whatever we want in a pluggable handler for that spec?

>
>
>
>Have you asked the same question on Redfish forum?  There are
>significantly more experts there, and they might have even better ideas
>than I do to improve your standards compliance on what you're trying to
>implement.

No.  I hope it is clear now that I really am not motivated to try and
standardize PEL, or on the flip side, to change all of our internal
business processes around to make use of something else.

Re: Redfish OEM commands in OpenBMC

[Ed Tanous]

On 4/23/19 10:52 AM, Brad Bishop wrote:
> On Tue, Apr 23, 2019 at 09:27:36AM -0700, Ed Tanous wrote:
>> On 4/23/19 6:51 AM, Brad Bishop wrote:
>>> On Mon, Apr 22, 2019 at 09:19:37AM -0700, Ed Tanous wrote:
>>>> I'll answer one of the questions I saw raised on the call: Should we
>>>> implement pluggable OEM handlers.
>>>>
>>>> No. (full stop)
>>>
>>> If this is the case, how am I supposed to implement our custom logging
>>> format, PEL?
>>
>> Lots of options are available, while staying within the bounds I gave
>> above.  Which you choose is likely constrained by what you're trying to
>> solve by shoehorning in an older logging format ontop of a specification
>> that already defines a logging format.
> 
> In a nutshell, the problem we are trying to solve is enablement of our
> support structure.  IBM provides its customers with a deep level of
> support.
> 
> To do this, IBM service personel require detailed failure information.
> A lot of it.  For that we have PEL.
> 
> This isn't a new problem - we already solved it long ago.  PEL has long
> been the "glue" between our systems and our support infrastructure.  We
> are just trying to enable that for ourselves, off to the side,  within
> the OpenBMC framework.

Some of what I'm trying to understand here is what you touched on.  If
this has already been solved long ago, why are you trying to add it to
Redfish?  That seems like opening a whole other can of problems,
including standardizing PEL over Redfish across all of IBMs systems.

> 
>>
>> 1. Attend the DMTF spec meetings (I believe they still have a logging
>> working group), and bespell how fantastic PEL logs are, and why they
>> should be the industry standard.  Get industry adoption for a "PelEntry"
>> and "PelCollection" schema definitions, then implement them in OpenBMC.
>> If this is done, and it becomes a matter of "they haven't published the
>> spec yet, but it's final" while waiting for an official DMTF drop, I'm
>> happy to discuss OpenBMC rollout strategies for bringing the
>> implementation in ahead of the spec.
> 
> But PEL is highly specific to IBM processes.  It doesn't make any sense
> to try to push for standardization of something like that.  I'm not here
> to try and standardize the mechanisms that enable server vendors provide
> support and service to their customers.
> 

So I think this is the part I don't quite follow.  If it's highly
specific, and important that it be standardized inside IBM, and unlikely
to be changed or standardized, why is a Redfish version important?
Wouldn't you rather just have unification with your other systems?

>>
>> 2. Use LogEntry and LogService to log your PEL entries like the other
>> 3-4 examples of logging do today.  Given that schema is very broad it
>> allows you to log essentially whatever you want into the message field.
>> This would also give the easiest time for people to port, and give you a
>> "standard Redfish" way to download your PEL logs.
> 
> This might be feasible.  I just don't know enough about the logging
> schema.  I'll have to do some reading.
> 
>>
>> 3. (Really not preferred) Use one of the other open source Redfish
>> servers that supports runtime pluggable endpoints.  
> 
> I guess option 3 prime is we fork bmcweb, and implement pluggable OEM
> callbacks there :wink:

This was originally listed as option seven, but I realized if you wanted
to do this, you would've already, and likely wouldn't be bringing it up
on the mailing list.  I'm happy to be proven wrong here that pluggable
callbacks could be done in a non-complicated easy to follow way, but
every option I've seen makes some really unfortunate tradeoffs in
readability, complexity, or performance.

> 
>> The point here is mostly that all the servers that defined endpoints
>> that were "pluggable" the way it was described ended up becoming overly
>> complicated, and IMHO lost a lot of battles because of said complexity.
>> This is not to say bmcweb is "simple" by any means.
>>
>> 4. Implement it as an odata endpoint, not necessarily in the Redfish
>> tree.  If your goal isn't industry compliance and acceptance, you might
>> consider moving a level up, and simply define your endpoints as odata
>> endpoints under a different URL tree.  This would depend on your end
>> goals, that I don't fully understand at this point.
> 
> This is the closest thing to what I had in mind.  But how is this not a
> a pluggable OEM handler?  Are you saying pluggable OEM handlers in
> bmcweb are OK as long as the route being handled is outside the /redfish
> namespace?

AH, I see the confusion.  When I hear "pluggable" I assume that means
runtime-pluggable dynamic resolution of plugins and interfaces.  If I
understand you correctly, that's not what the "plugin" discussion was.
We can certainly enable/disable features using the existing configure
mechanisms at compile time.  There are already a dozen or so examples of
enabling/disabling features at compile time, including the existing
logging implementations.
https://github.com/openbmc/bmcweb/blob/424c4176bbbc55c0cfd08f273b7e4500c977a138/CMakeLists.txt#L11

We could certainly add another custom URL handler for PEL logs if that's
the route you want to go down, although I'd personally still prefer
option 2, as we can at least keep it somewhat in the standard.

> 
>>
>> 5. Get a percentage OpenBMC community to agree that PEL logging is the
>> future, and that's the direction we should start going, then define an
>> OEM endpoint for it under the OpenBMC namespace that we can all find
>> ways to implement against the backend.
> 
> I'm not here to try to convince anyone to use PEL.  I have to have it on
> my systems and I just want to understand what the path for getting it
> there is.

Your existing systems have PEL over Redfish?  What do the metadata files
look like?  I think that would help the discussion a lot.

> 
>>
>> 6. Write a maintained, versioned, multi-system implementation spec for
>> "PEL over Redfish", that defines a PEL logging definition that could be
>> implemented by any BMC (not just OpenBMC) and requires OEM endpoints.  A
>> decent example of this is the RSD specification:
>> https://www.intel.com/content/dam/www/public/us/en/documents/guides/podm-api-spec-v2-4-guide.pdf
>>
> 
> Confused by this one.  Are you saying if we write a spec, we can
> implement whatever we want in a pluggable handler for that spec?

See note about plugins above.  "Configurable feature" != "plugin" in my
mind.  The "if there's a spec" statement is a bit of a cop out on the
OEM properties, because there would be no way to implement said spec
compliance without defining OEM properties.  With that said, it's an
open spec that any BMC can implement against, so it is easier to claim
widespread usage and standards, which I think is one of the goals of
this project, and Redfish in general.

To be clear, I don't have any intention of implementing the POD manager
spec above on OpenBMC, it's just an example of a standardized OEM-based
interface that multiple implementations implement against.

> 
>>
>>
>>
>> Have you asked the same question on Redfish forum?  There are
>> significantly more experts there, and they might have even better ideas
>> than I do to improve your standards compliance on what you're trying to
>> implement.
> 
> No.  I hope it is clear now that I really am not motivated to try and
> standardize PEL, or on the flip side, to change all of our internal
> business processes around to make use of something else.

I'm sure there's a way to work through it as a community, and I will
fully admit that I have similar things of my own that have the same need.

Any feature is going to be hard going if the position starts with the
premise: "we have this feature on non openbmc systems, with a closed
spec, we don't want to change it, and we want it implemented in openbmc
master because we need backward compatibility".  With that said, I'm
sure there's a way to get it in if it's something you need.

I did just find the "IBM enterprise error logging requirements" review.
I didn't realize that was related to PEL.  I need to read through it and
inform myself a little better.

Re: Redfish OEM commands in OpenBMC

[Brad Bishop]

On Tue, Apr 23, 2019 at 03:12:55PM -0700, Ed Tanous wrote:
>On 4/23/19 10:52 AM, Brad Bishop wrote:
>> On Tue, Apr 23, 2019 at 09:27:36AM -0700, Ed Tanous wrote:
>>> On 4/23/19 6:51 AM, Brad Bishop wrote:
>>>> On Mon, Apr 22, 2019 at 09:19:37AM -0700, Ed Tanous wrote:

>>> 2. Use LogEntry and LogService to log your PEL entries like the other
>>> 3-4 examples of logging do today.  Given that schema is very broad it
>>> allows you to log essentially whatever you want into the message field.
>>> This would also give the easiest time for people to port, and give you a
>>> "standard Redfish" way to download your PEL logs.
>>
>> This might be feasible.  I just don't know enough about the logging
>> schema.  I'll have to do some reading.

I asked around here and this is actually what we intend to do.  Create
"Event" type instances of LogEntry, and put our PEL stuff in the OEM
property.  It sounds like this is ok with you.  Can you confirm?

>>> 3. (Really not preferred) Use one of the other open source Redfish
>>> servers that supports runtime pluggable endpoints. 
>>
>> I guess option 3 prime is we fork bmcweb, and implement pluggable OEM
>> callbacks there :wink:
>
>This was originally listed as option seven, but I realized if you wanted
>to do this, you would've already, and likely wouldn't be bringing it up
>on the mailing list.  I'm happy to be proven wrong here that pluggable
>callbacks could be done in a non-complicated easy to follow way, but
>every option I've seen makes some really unfortunate tradeoffs in
>readability, complexity, or performance.

I don't understand what is so hard here.  Vernon was able to pull this
off with IPMI...why does this have to be any different (and in the same
vein, why is the flexibility offered and required in our IPMI core but
not our Redfish core?)

>>> The point here is mostly that all the servers that defined endpoints
>>> that were "pluggable" the way it was described ended up becoming overly
>>> complicated, and IMHO lost a lot of battles because of said complexity.
>>> This is not to say bmcweb is "simple" by any means.
>>>
>>> 4. Implement it as an odata endpoint, not necessarily in the Redfish
>>> tree.  If your goal isn't industry compliance and acceptance, you might
>>> consider moving a level up, and simply define your endpoints as odata
>>> endpoints under a different URL tree.  This would depend on your end
>>> goals, that I don't fully understand at this point.
>>
>> This is the closest thing to what I had in mind.  But how is this not a
>> a pluggable OEM handler?  Are you saying pluggable OEM handlers in
>> bmcweb are OK as long as the route being handled is outside the /redfish
>> namespace?
>
>AH, I see the confusion.  When I hear "pluggable" I assume that means
>runtime-pluggable dynamic resolution of plugins and interfaces.  If I
>understand you correctly, that's not what the "plugin" discussion was.
>We can certainly enable/disable features using the existing configure
>mechanisms at compile time.  There are already a dozen or so examples of
>enabling/disabling features at compile time, including the existing
>logging implementations.
>https://github.com/openbmc/bmcweb/blob/424c4176bbbc55c0cfd08f273b7e4500c977a138/CMakeLists.txt#L11
>
>We could certainly add another custom URL handler for PEL logs if that's
>the route you want to go down, although I'd personally still prefer
>option 2, as we can at least keep it somewhat in the standard.

Agreed on option #2.  However I still think you should consider runtime
pluggable dynamic resolution.

It sounds like you want everyone to put their implementations of OEM
properties right next to each other in bmcweb and surround them with
ifdefs.  Do I have that right?  Shouldn't we allow the OEM to maintain
their own implementation?  Also, when you (the bmcweb maintainer) look
at the core bmcweb/redfish code, do you want to be distracted by the
twenty implementations of an OEM property?  Does it make sense for you
to be the maintainer of code you have zero investment in?

>>> Have you asked the same question on Redfish forum?  There are
>>> significantly more experts there, and they might have even better ideas
>>> than I do to improve your standards compliance on what you're trying to
>>> implement.
>>
>> No.  I hope it is clear now that I really am not motivated to try and
>> standardize PEL, or on the flip side, to change all of our internal
>> business processes around to make use of something else.
>
>I'm sure there's a way to work through it as a community, and I will
>fully admit that I have similar things of my own that have the same need.
>
>Any feature is going to be hard going if the position starts with the
>premise: "we have this feature on non openbmc systems, with a closed
>spec, we don't want to change it, and we want it implemented in openbmc
>master because we need backward compatibility".  With that said, I'm
>sure there's a way to get it in if it's something you need.

This is just the reality of the world we live in...  It is precisely why
we need robust (yes, sometimes complicated, sometimes performance
impacting, sometimes harder to read) frameworks and abstractions that
allow us to share and collaborate where it makes sense and to move
quickly where it does not.

Re: Redfish OEM commands in OpenBMC

[Ed Tanous]

On 4/24/19 11:18 AM, Brad Bishop wrote:
> On Tue, Apr 23, 2019 at 03:12:55PM -0700, Ed Tanous wrote:
>> On 4/23/19 10:52 AM, Brad Bishop wrote:
>>> On Tue, Apr 23, 2019 at 09:27:36AM -0700, Ed Tanous wrote:
>>>> On 4/23/19 6:51 AM, Brad Bishop wrote:
>>>>> On Mon, Apr 22, 2019 at 09:19:37AM -0700, Ed Tanous wrote:
> 
>>>> 2. Use LogEntry and LogService to log your PEL entries like the other
>>>> 3-4 examples of logging do today.  Given that schema is very broad it
>>>> allows you to log essentially whatever you want into the message field.
>>>> This would also give the easiest time for people to port, and give
>>>> you a
>>>> "standard Redfish" way to download your PEL logs.
>>>
>>> This might be feasible.  I just don't know enough about the logging
>>> schema.  I'll have to do some reading.
> 
> I asked around here and this is actually what we intend to do.  Create
> "Event" type instances of LogEntry, and put our PEL stuff in the OEM
> property.  It sounds like this is ok with you.  Can you confirm?
> 

On the nose, it sounds ok, but it would be good to see a proposal that's
a little more detailed.  Also, like we talked about this morning, we
need to figure out how to make this work in the long run.

>>>> 3. (Really not preferred) Use one of the other open source Redfish
>>>> servers that supports runtime pluggable endpoints. 
>>>
>>> I guess option 3 prime is we fork bmcweb, and implement pluggable OEM
>>> callbacks there :wink:
>>
>> This was originally listed as option seven, but I realized if you wanted
>> to do this, you would've already, and likely wouldn't be bringing it up
>> on the mailing list.  I'm happy to be proven wrong here that pluggable
>> callbacks could be done in a non-complicated easy to follow way, but
>> every option I've seen makes some really unfortunate tradeoffs in
>> readability, complexity, or performance.
> 
> I don't understand what is so hard here.  Vernon was able to pull this
> off with IPMI...why does this have to be any different (and in the same
> vein, why is the flexibility offered and required in our IPMI core but
> not our Redfish core?)
> 

Flexibility is required in the IPMI core because of its pedigree and
interfaces.  Today, OpenBMC doesn't own or maintain code for our
respective host systems pre-boot, which means that to actually boot a
useful system you need to be able to publish a specific interface
matching what the "old" system did;  Some of that includes OEM commands,
and everyone has implemented them differently.
Redfish isn't encumbered by that baggage (yet).  In all the examples
I've heard about in an OpenBMC context, we're creating something "new",
and don't yet have backward compatibility concerns, which is what makes
it different.  Redfish is also introspectable, which gives clients a way
to understand what features/commands/properties are supported by a given
implementation at runtime, which reduces (but doesn't completely remove)
the need for that level of flexibility.
If someone came in with an incumbent set of Redfish schemas, and they
needed the interfaces to match, that's a very different discussion than
I think we're having now, and it would (likely) have my full support.
In theory that's what I was trying to get at with the part about if it's
in an existing specification.

On the note about Vernon: ask him how difficult it was (and still is) to
be able to pull this interface change off with IPMI.  There were quite a
few gymnastics required, and at this point, we haven't "fixed" the
interface so much as gave a more modern option.  The process of
deprecating the old interface is still in process and will be for a long
time.

>>>> The point here is mostly that all the servers that defined endpoints
>>>> that were "pluggable" the way it was described ended up becoming overly
>>>> complicated, and IMHO lost a lot of battles because of said complexity.
>>>> This is not to say bmcweb is "simple" by any means.
>>>>
>>>> 4. Implement it as an odata endpoint, not necessarily in the Redfish
>>>> tree.  If your goal isn't industry compliance and acceptance, you might
>>>> consider moving a level up, and simply define your endpoints as odata
>>>> endpoints under a different URL tree.  This would depend on your end
>>>> goals, that I don't fully understand at this point.
>>>
>>> This is the closest thing to what I had in mind.  But how is this not a
>>> a pluggable OEM handler?  Are you saying pluggable OEM handlers in
>>> bmcweb are OK as long as the route being handled is outside the /redfish
>>> namespace?
>>
>> AH, I see the confusion.  When I hear "pluggable" I assume that means
>> runtime-pluggable dynamic resolution of plugins and interfaces.  If I
>> understand you correctly, that's not what the "plugin" discussion was.
>> We can certainly enable/disable features using the existing configure
>> mechanisms at compile time.  There are already a dozen or so examples of
>> enabling/disabling features at compile time, including the existing
>> logging implementations.
>> https://github.com/openbmc/bmcweb/blob/424c4176bbbc55c0cfd08f273b7e4500c977a138/CMakeLists.txt#L11
>>
>>
>> We could certainly add another custom URL handler for PEL logs if that's
>> the route you want to go down, although I'd personally still prefer
>> option 2, as we can at least keep it somewhat in the standard.
> 
> Agreed on option #2.  However I still think you should consider runtime
> pluggable dynamic resolution.

None of the features I've heard so far necessitates the use of runtime
discovery, or even fall outside what configurability is available in
bmcweb today.  "runtime pluggable" on an embedded system is a strange
concept, given that all uses I know of today are really just an
extension of compile time plugability, using the rootfs as the medium
for "discovery".  With that said, I'm imagining you're thinking
something like what IPMI has today, which I might be misinterpreting
based on our discussion this morning?

It should be noted, we also have DBus "plugin" capability with the dbus
interface already: anyone can host logs on dbus, and redfish will
populate them.  That option doesn't really get to the core of the OEM
issue though, but is certainly an option in the PEL log case that I
didn't think of right away.

> 
> It sounds like you want everyone to put their implementations of OEM
> properties right next to each other in bmcweb and surround them with
> ifdefs.  Do I have that rightYes, I believe you have that right.

> Shouldn't we allow the OEM to maintain
> their own implementation?  Also, when you (the bmcweb maintainer) look
> at the core bmcweb/redfish code, do you want to be distracted by the
> twenty implementations of an OEM property?

If my goal is to make my changes without breaking any of the other
twenty implementations at the same time, absolutely, I want to be
"distracted" by them.  Also, if I see similarities between interfaces, I
want to be able to reduce/reuse/recycle that code without having to go
across a library boundary.  If they become too distracting to be in the
same file, they can certainly be put in a different file, or organized
in a way that makes sense.

>  Does it make sense for you
> to be the maintainer of code you have zero investment in?

We can definitely both agree that me personally maintaining all Redfish
code is unmaintainable in the long term.  I don't want to be the
maintainer of code that I have no investment in, but I'm not sure how
you came to the "Ed is the only maintainer of bmcweb for all time"
conclusion.  The current maintainer files have provisions for layering
just like the Kernel does.  Long-term, we can split maintainership on
whatever lines are appropriate.

> 
> This is just the reality of the world we live in...  It is precisely why
> we need robust (yes, sometimes complicated, sometimes performance
> impacting, sometimes harder to read) frameworks and abstractions that
> allow us to share and collaborate where it makes sense and to move
> quickly where it does not.

I would argue that the kernel driver interface is a great example of how
this should be done, and handles scales that we could only hope to get
to get to.  The kernel doesn't publish a "stable" driver interface
internally, but does publish a concrete user-facing API.  That's
somewhat what bmcweb has attempted to model in its design;  User facing
code should be able to be concrete, while keeping the internals flexible
enough to make changes as better patterns emerge.

Re: Redfish OEM commands in OpenBMC

[Brad Bishop]

On Mon, Apr 29, 2019 at 10:32:36AM -0700, Ed Tanous wrote:

>On the nose, it sounds ok, but it would be good to see a proposal
>that's a little more detailed.  
I put a proposal here:
https://lists.ozlabs.org/pipermail/openbmc/2019-April/016126.html

>>Agreed on option #2.  However I still think you should consider
>>runtime pluggable dynamic resolution.
>
>None of the features I've heard so far necessitates the use of runtime
>discovery, or even fall outside what configurability is available in
>bmcweb today.  "runtime pluggable" on an embedded system is a strange
>concept, given that all uses I know of today are really just an
>extension of compile time plugability, using the rootfs as the medium
>for "discovery".  With that said, I'm imagining you're thinking
>something like what IPMI has today, which I might be misinterpreting
>based on our discussion this morning?
Yes, this is what I was thinking.  I agree with you that we don't have
features that require the use of runtime discovery.  My reasoning is
rooted in concerns around code maintanence and adoption.

maintanence:  I don't think code with a bunch of #ifdefs is desired or
readable.  It might not be too bad starting out.

adoption:  Do you recall how one of Michael Brown's issues with bmcweb
was that there wasn't "any way to extend or replace functions without
forking the codebase?"  I believe this is exactly what he was talking
about.  I suspect he didn't even consider compile time plugins because
in my experience maintaining codebases structured that way is painful.

>It should be noted, we also have DBus "plugin" capability with the dbus
>interface already: anyone can host logs on dbus, and redfish will
>populate them.  That option doesn't really get to the core of the OEM
>issue though, but is certainly an option in the PEL log case that I
>didn't think of right away.
This is an interesting thought.  We could put OEM things on DBus and
maybe bmcweb looks at those to fill out OEM properties?

>>It sounds like you want everyone to put their implementations of OEM
>>properties right next to each other in bmcweb and surround them with
>>ifdefs.  Do I have that right
> Yes, I believe you have that right.

>>Shouldn't we allow the OEM to maintain their own implementation? 
>>Also, when you (the bmcweb maintainer) look at the core bmcweb/redfish
>>code, do you want to be distracted by the twenty implementations of an
>>OEM property?
>
>If my goal is to make my changes without breaking any of the other
>twenty implementations at the same time, absolutely, I want to be
>"distracted" by them.  
Right.  This is the fallout of choosing to not have a framework/have an
unstable API.  This is also what makes this approach not scale very well
from a maintanence POV, IMO.

>>Does it make sense for you to be the maintainer of code you have
>>zero investment in?
>We can definitely both agree that me personally maintaining all Redfish
>code is unmaintainable in the long term.  I don't want to be the
>maintainer of code that I have no investment in, but I'm not sure how
>you came to the "Ed is the only maintainer of bmcweb for all time"
>conclusion.  The current maintainer files have provisions for layering
>just like the Kernel does.  Long-term, we can split maintainership on
>whatever lines are appropriate.
Sounds good!

>>This is just the reality of the world we live in...  It is precisely
>>why we need robust (yes, sometimes complicated, sometimes performance
>>impacting, sometimes harder to read) frameworks and abstractions that
>>allow us to share and collaborate where it makes sense and to move
>>quickly where it does not.
>I would argue that the kernel driver interface is a great example of
>how this should be done, and handles scales that we could only hope to
>get to get to.  The kernel doesn't publish a "stable" driver interface
>internally, but does publish a concrete user-facing API.  
It may not be a stable interface but there _is_ an interface to which
out of tree modules for instance can plug into.  And those modules could
implement a custom, OEM ABI via sysfs or ioctls.

>somewhat what bmcweb has attempted to model in its design;  User facing
>code should be able to be concrete, while keeping the internals
>flexible enough to make changes as better patterns emerge.

This thread raises an important design point or "community norm" being
established for OpenBMC.  IMO it has the potential to impact
participation rates and cost of maintanence in the future.  It would be
nice to get some opinions from more than just two people.  Anyone care
to chime in?

-brad

Re: Redfish OEM commands in OpenBMC

[Brad Bishop]

On Wed, May 01, 2019 at 02:05:47PM -0400, Brad Bishop wrote:
>It would be nice to get some opinions from more than just two people.

Sorry Andrew and Joseph - you spoke up early on, so make that four
people.

Re: Redfish OEM commands in OpenBMC

[Ed Tanous]

On 5/1/19 11:05 AM, Brad Bishop wrote:
> On Mon, Apr 29, 2019 at 10:32:36AM -0700, Ed Tanous wrote:
> 
>> On the nose, it sounds ok, but it would be good to see a proposal
>> that's a little more detailed.  
> I put a proposal here:
> https://lists.ozlabs.org/pipermail/openbmc/2019-April/016126.html
> 
>>> Agreed on option #2.  However I still think you should consider
>>> runtime pluggable dynamic resolution.
>>
>> None of the features I've heard so far necessitates the use of runtime
>> discovery, or even fall outside what configurability is available in
>> bmcweb today.  "runtime pluggable" on an embedded system is a strange
>> concept, given that all uses I know of today are really just an
>> extension of compile time plugability, using the rootfs as the medium
>> for "discovery".  With that said, I'm imagining you're thinking
>> something like what IPMI has today, which I might be misinterpreting
>> based on our discussion this morning?
> Yes, this is what I was thinking.  I agree with you that we don't have
> features that require the use of runtime discovery.  My reasoning is
> rooted in concerns around code maintanence and adoption.
> 
> maintanence:  I don't think code with a bunch of #ifdefs is desired or
> readable.  It might not be too bad starting out.

I think the above statement would be a lot more productive if you had a
patchset that removed some of the existing ifdefs, so we can see what
you're thinking.  I personally don't find them unreadable or
undesirable, and significantly less confusing than the runtime plugin
system in ipmi.  Given that several people, from several companies have
been able to extend them without a lot of documentation or handholding
makes me think that they're meeting the intended purpose, although to a
much lower scale than IPMI.

With that said, the ifdef feature model works at much higher scales in
the Kernel, so there was an assumption it could scale to whatever we need.

Of the half-dozen or so implementations of Redfish that started out,
there were several that didn't use ifdefs for configuration.

> 
> adoption:  Do you recall how one of Michael Brown's issues with bmcweb
> was that there wasn't "any way to extend or replace functions without
> forking the codebase?"  I believe this is exactly what he was talking
> about.  I suspect he didn't even consider compile time plugins because
> in my experience maintaining codebases structured that way is painful.

I do recall, and I recall his attempted solution to it with go-redfish
was difficult to understand and modify, and had some CPU usage penalties
due to its caching and plugin requirements.  If plugins met the exact
intent of what the framework intended them to do, then it functioned
just fine, but the minute it reached past what the interface was able to
define, things got very complicated very quickly IMHO.
With respect to that, the primary goal of that plugin interface was an
issue that I don't think OpenBMC cares about: the ability to take
OpenBMC components (in this case redfish) and integrate them into
private, non-OpenBMC codebases.  If you're integrating with a private
codebase, you're forking either way.

> 
>> It should be noted, we also have DBus "plugin" capability with the dbus
>> interface already: anyone can host logs on dbus, and redfish will
>> populate them.  That option doesn't really get to the core of the OEM
>> issue though, but is certainly an option in the PEL log case that I
>> didn't think of right away.
> This is an interesting thought.  We could put OEM things on DBus and
> maybe bmcweb looks at those to fill out OEM properties?

Isn't this already how phosphor-logging was defined?  (on second
thought, this discussion is probably better had in your other thread
about logging)

> 
>>> It sounds like you want everyone to put their implementations of OEM
>>> properties right next to each other in bmcweb and surround them with
>>> ifdefs.  Do I have that right
>> Yes, I believe you have that right.
> 
>>> Shouldn't we allow the OEM to maintain their own implementation? 
>>> Also, when you (the bmcweb maintainer) look at the core bmcweb/redfish
>>> code, do you want to be distracted by the twenty implementations of an
>>> OEM property?
>>
>> If my goal is to make my changes without breaking any of the other
>> twenty implementations at the same time, absolutely, I want to be
>> "distracted" by them.  
> Right.  This is the fallout of choosing to not have a framework/have an
> unstable API.  This is also what makes this approach not scale very well
> from a maintenance POV, IMO.

At the end of the day IMO, the API that matters is the one the system
user sees out the HTTP port.  All else is secondary.  Given the wide
range of things HTTP is capable of, I didn't see a way to define a
concrete API that was extensible in the features that we knew the
specifications supported and we knew were coming, while keeping the
performance and size guarantees required to run on the system.
The other thing to remember is that the #ifdef features apply to more
than just redfish, so unless you define a full concrete HTTP api, not
just redfish, you end up back in the same boat where you started when
non-redfish protocol needs come up.
Also, so far as I know, the Redfish "Node" class API hasn't been broken
since it was first committed by Lukasz, so to call it an "unstable" API
is a bit of a misnomer.  "Unspecified" API, I could accept.

> 
>>> Does it make sense for you to be the maintainer of code you have
>>> zero investment in?
>> We can definitely both agree that me personally maintaining all Redfish
>> code is unmaintainable in the long term.  I don't want to be the
>> maintainer of code that I have no investment in, but I'm not sure how
>> you came to the "Ed is the only maintainer of bmcweb for all time"
>> conclusion.  The current maintainer files have provisions for layering
>> just like the Kernel does.  Long-term, we can split maintainership on
>> whatever lines are appropriate.
> Sounds good!
> 
>>> This is just the reality of the world we live in...  It is precisely
>>> why we need robust (yes, sometimes complicated, sometimes performance
>>> impacting, sometimes harder to read) frameworks and abstractions that
>>> allow us to share and collaborate where it makes sense and to move
>>> quickly where it does not.
>> I would argue that the kernel driver interface is a great example of
>> how this should be done, and handles scales that we could only hope to
>> get to get to.  The kernel doesn't publish a "stable" driver interface
>> internally, but does publish a concrete user-facing API.  
> It may not be a stable interface but there _is_ an interface to which
> out of tree modules for instance can plug into.  And those modules could
> implement a custom, OEM ABI via sysfs or ioctls.
> 
>> somewhat what bmcweb has attempted to model in its design;  User facing
>> code should be able to be concrete, while keeping the internals
>> flexible enough to make changes as better patterns emerge.
> 
> This thread raises an important design point or "community norm" being
> established for OpenBMC.  IMO it has the potential to impact
> participation rates and cost of maintanence in the future.  It would be
> nice to get some opinions from more than just two people.  Anyone care
> to chime in?
> 
Agreed, would love to hear more opinions on this.

> -brad

Re: Redfish OEM commands in OpenBMC

[Brad Bishop]

On Thu, May 02, 2019 at 09:09:08AM -0700, Ed Tanous wrote:
>On 5/1/19 11:05 AM, Brad Bishop wrote:
>> On Mon, Apr 29, 2019 at 10:32:36AM -0700, Ed Tanous wrote:
>>
>>> On the nose, it sounds ok, but it would be good to see a proposal
>>> that's a little more detailed. 
>> I put a proposal here:
>> https://lists.ozlabs.org/pipermail/openbmc/2019-April/016126.html
>>
>>>> Agreed on option #2.  However I still think you should consider
>>>> runtime pluggable dynamic resolution.
>>>
>>> None of the features I've heard so far necessitates the use of runtime
>>> discovery, or even fall outside what configurability is available in
>>> bmcweb today.  "runtime pluggable" on an embedded system is a strange
>>> concept, given that all uses I know of today are really just an
>>> extension of compile time plugability, using the rootfs as the medium
>>> for "discovery".  With that said, I'm imagining you're thinking
>>> something like what IPMI has today, which I might be misinterpreting
>>> based on our discussion this morning?
>> Yes, this is what I was thinking.  I agree with you that we don't have
>> features that require the use of runtime discovery.  My reasoning is
>> rooted in concerns around code maintanence and adoption.
>>
>> maintanence:  I don't think code with a bunch of #ifdefs is desired or
>> readable.  It might not be too bad starting out.
>
>I think the above statement would be a lot more productive if you had a
>patchset that removed some of the existing ifdefs, so we can see what
>you're thinking.  
Haha now you are asking me to do real work, so its time for me to drop
the point.

>> adoption:  Do you recall how one of Michael Brown's issues with bmcweb
>> was that there wasn't "any way to extend or replace functions without
>> forking the codebase?"  I believe this is exactly what he was talking
>> about.  I suspect he didn't even consider compile time plugins because
>> in my experience maintaining codebases structured that way is painful.
>
>I do recall, and I recall his attempted solution to it with go-redfish
>was difficult to understand and modify, and had some CPU usage penalties
>due to its caching and plugin requirements.  If plugins met the exact
>intent of what the framework intended them to do, then it functioned
>just fine, but the minute it reached past what the interface was able to
>define, things got very complicated very quickly IMHO.
>With respect to that, the primary goal of that plugin interface was an
>issue that I don't think OpenBMC cares about: the ability to take
>OpenBMC components (in this case redfish) and integrate them into
>private, non-OpenBMC codebases.  If you're integrating with a private
>codebase, you're forking either way.
I don't understand "non-OpenBMC codebase."  How many lines of code does
one have to change before a fork is no longer an OpenBMC codebase?

I also don't understand how "OpenBMC" can care about something.  OpenBMC
is me, you, IBM, Intel, and whole host of others that all care about
lots of different, conflicting even, things.

If we structure the code in such a way that it enables product teams
with "non-OpenBMC codebases" to make use of and therefore have a
motivation to contribute back to and share in the development and
maintanence burden of the shared pieces of code then that is something
that I very much care about.

>> This thread raises an important design point or "community norm" being
>> established for OpenBMC.  IMO it has the potential to impact
>> participation rates and cost of maintanence in the future.  It would be
>> nice to get some opinions from more than just two people.  Anyone care
>> to chime in?
>>
>Agreed, would love to hear more opinions on this.

Re: Redfish OEM commands in OpenBMC

[Deepak Kodihalli]

On 01/05/19 11:35 PM, Brad Bishop wrote:
> On Mon, Apr 29, 2019 at 10:32:36AM -0700, Ed Tanous wrote:
> 
>> On the nose, it sounds ok, but it would be good to see a proposal
>> that's a little more detailed. 
> I put a proposal here:
> https://lists.ozlabs.org/pipermail/openbmc/2019-April/016126.html
> 
>>> Agreed on option #2.  However I still think you should consider
>>> runtime pluggable dynamic resolution.
>>
>> None of the features I've heard so far necessitates the use of runtime
>> discovery, or even fall outside what configurability is available in
>> bmcweb today.  "runtime pluggable" on an embedded system is a strange
>> concept, given that all uses I know of today are really just an
>> extension of compile time plugability, using the rootfs as the medium
>> for "discovery".  With that said, I'm imagining you're thinking
>> something like what IPMI has today, which I might be misinterpreting
>> based on our discussion this morning?
> Yes, this is what I was thinking.  I agree with you that we don't have
> features that require the use of runtime discovery.  My reasoning is
> rooted in concerns around code maintanence and adoption.
> 
> maintanence:  I don't think code with a bunch of #ifdefs is desired or
> readable.  It might not be too bad starting out.
> 
> adoption:  Do you recall how one of Michael Brown's issues with bmcweb
> was that there wasn't "any way to extend or replace functions without
> forking the codebase?"  I believe this is exactly what he was talking
> about.  I suspect he didn't even consider compile time plugins because
> in my experience maintaining codebases structured that way is painful.
> 
>> It should be noted, we also have DBus "plugin" capability with the dbus
>> interface already: anyone can host logs on dbus, and redfish will
>> populate them.  That option doesn't really get to the core of the OEM
>> issue though, but is certainly an option in the PEL log case that I
>> didn't think of right away.
> This is an interesting thought.  We could put OEM things on DBus and
> maybe bmcweb looks at those to fill out OEM properties?
> 
>>> It sounds like you want everyone to put their implementations of OEM
>>> properties right next to each other in bmcweb and surround them with
>>> ifdefs.  Do I have that right
>> Yes, I believe you have that right.
> 
>>> Shouldn't we allow the OEM to maintain their own implementation?
>>> Also, when you (the bmcweb maintainer) look at the core bmcweb/redfish
>>> code, do you want to be distracted by the twenty implementations of an
>>> OEM property?
>>
>> If my goal is to make my changes without breaking any of the other
>> twenty implementations at the same time, absolutely, I want to be
>> "distracted" by them. 
> Right.  This is the fallout of choosing to not have a framework/have an
> unstable API.  This is also what makes this approach not scale very well
> from a maintanence POV, IMO.
> 
>>> Does it make sense for you to be the maintainer of code you have
>>> zero investment in?
>> We can definitely both agree that me personally maintaining all Redfish
>> code is unmaintainable in the long term.  I don't want to be the
>> maintainer of code that I have no investment in, but I'm not sure how
>> you came to the "Ed is the only maintainer of bmcweb for all time"
>> conclusion.  The current maintainer files have provisions for layering
>> just like the Kernel does.  Long-term, we can split maintainership on
>> whatever lines are appropriate.
> Sounds good!
> 
>>> This is just the reality of the world we live in...  It is precisely
>>> why we need robust (yes, sometimes complicated, sometimes performance
>>> impacting, sometimes harder to read) frameworks and abstractions that
>>> allow us to share and collaborate where it makes sense and to move
>>> quickly where it does not.
>> I would argue that the kernel driver interface is a great example of
>> how this should be done, and handles scales that we could only hope to
>> get to get to.  The kernel doesn't publish a "stable" driver interface
>> internally, but does publish a concrete user-facing API. 
> It may not be a stable interface but there _is_ an interface to which
> out of tree modules for instance can plug into.  And those modules could
> implement a custom, OEM ABI via sysfs or ioctls.
> 
>> somewhat what bmcweb has attempted to model in its design;  User facing
>> code should be able to be concrete, while keeping the internals
>> flexible enough to make changes as better patterns emerge.
> 
> This thread raises an important design point or "community norm" being
> established for OpenBMC.  IMO it has the potential to impact
> participation rates and cost of maintanence in the future.  It would be
> nice to get some opinions from more than just two people.  Anyone care
> to chime in?

We have a similar problem to solve with PLDM OEM commands. I was 
initially considering dynamically loaded libraries implementing OEM, but 
I am interested to know the pitfalls of the same from Vernon or others, 
as opposed to compile-time plugging-in of these libraries (not sure if 
this was already discussed in the last community call, which I couldn't 
attend).

Do we want to disallow someone who uses OpenBMC from writing proprietary 
PLDM OEM implementations - libraries which can be dynamically loaded at 
runtime as opposed to compile time linking in of those implementations 
to the main PLDM code base, thereby requiring the OEM implementations to 
exist in the main PLDM code base?

Thanks,
Deepak

Re: Redfish OEM commands in OpenBMC

[Andrew Jeffery]

On Mon, 6 May 2019, at 18:21, Deepak Kodihalli wrote:
> On 01/05/19 11:35 PM, Brad Bishop wrote:
> > On Mon, Apr 29, 2019 at 10:32:36AM -0700, Ed Tanous wrote:
> > 
> >> On the nose, it sounds ok, but it would be good to see a proposal
> >> that's a little more detailed. 
> > I put a proposal here:
> > https://lists.ozlabs.org/pipermail/openbmc/2019-April/016126.html
> > 
> >>> Agreed on option #2.  However I still think you should consider
> >>> runtime pluggable dynamic resolution.
> >>
> >> None of the features I've heard so far necessitates the use of runtime
> >> discovery, or even fall outside what configurability is available in
> >> bmcweb today.  "runtime pluggable" on an embedded system is a strange
> >> concept, given that all uses I know of today are really just an
> >> extension of compile time plugability, using the rootfs as the medium
> >> for "discovery".  With that said, I'm imagining you're thinking
> >> something like what IPMI has today, which I might be misinterpreting
> >> based on our discussion this morning?
> > Yes, this is what I was thinking.  I agree with you that we don't have
> > features that require the use of runtime discovery.  My reasoning is
> > rooted in concerns around code maintanence and adoption.
> > 
> > maintanence:  I don't think code with a bunch of #ifdefs is desired or
> > readable.  It might not be too bad starting out.
> > 
> > adoption:  Do you recall how one of Michael Brown's issues with bmcweb
> > was that there wasn't "any way to extend or replace functions without
> > forking the codebase?"  I believe this is exactly what he was talking
> > about.  I suspect he didn't even consider compile time plugins because
> > in my experience maintaining codebases structured that way is painful.
> > 
> >> It should be noted, we also have DBus "plugin" capability with the dbus
> >> interface already: anyone can host logs on dbus, and redfish will
> >> populate them.  That option doesn't really get to the core of the OEM
> >> issue though, but is certainly an option in the PEL log case that I
> >> didn't think of right away.
> > This is an interesting thought.  We could put OEM things on DBus and
> > maybe bmcweb looks at those to fill out OEM properties?
> > 
> >>> It sounds like you want everyone to put their implementations of OEM
> >>> properties right next to each other in bmcweb and surround them with
> >>> ifdefs.  Do I have that right
> >> Yes, I believe you have that right.
> > 
> >>> Shouldn't we allow the OEM to maintain their own implementation?
> >>> Also, when you (the bmcweb maintainer) look at the core bmcweb/redfish
> >>> code, do you want to be distracted by the twenty implementations of an
> >>> OEM property?
> >>
> >> If my goal is to make my changes without breaking any of the other
> >> twenty implementations at the same time, absolutely, I want to be
> >> "distracted" by them. 
> > Right.  This is the fallout of choosing to not have a framework/have an
> > unstable API.  This is also what makes this approach not scale very well
> > from a maintanence POV, IMO.
> > 
> >>> Does it make sense for you to be the maintainer of code you have
> >>> zero investment in?
> >> We can definitely both agree that me personally maintaining all Redfish
> >> code is unmaintainable in the long term.  I don't want to be the
> >> maintainer of code that I have no investment in, but I'm not sure how
> >> you came to the "Ed is the only maintainer of bmcweb for all time"
> >> conclusion.  The current maintainer files have provisions for layering
> >> just like the Kernel does.  Long-term, we can split maintainership on
> >> whatever lines are appropriate.
> > Sounds good!
> > 
> >>> This is just the reality of the world we live in...  It is precisely
> >>> why we need robust (yes, sometimes complicated, sometimes performance
> >>> impacting, sometimes harder to read) frameworks and abstractions that
> >>> allow us to share and collaborate where it makes sense and to move
> >>> quickly where it does not.
> >> I would argue that the kernel driver interface is a great example of
> >> how this should be done, and handles scales that we could only hope to
> >> get to get to.  The kernel doesn't publish a "stable" driver interface
> >> internally, but does publish a concrete user-facing API. 
> > It may not be a stable interface but there _is_ an interface to which
> > out of tree modules for instance can plug into.  And those modules could
> > implement a custom, OEM ABI via sysfs or ioctls.
> > 
> >> somewhat what bmcweb has attempted to model in its design;  User facing
> >> code should be able to be concrete, while keeping the internals
> >> flexible enough to make changes as better patterns emerge.
> > 
> > This thread raises an important design point or "community norm" being
> > established for OpenBMC.  IMO it has the potential to impact
> > participation rates and cost of maintanence in the future.  It would be
> > nice to get some opinions from more than just two people.  Anyone care
> > to chime in?
> 
> We have a similar problem to solve with PLDM OEM commands. I was 
> initially considering dynamically loaded libraries implementing OEM, but 
> I am interested to know the pitfalls of the same from Vernon or others, 
> as opposed to compile-time plugging-in of these libraries (not sure if 
> this was already discussed in the last community call, which I couldn't 
> attend).
> 
> Do we want to disallow someone who uses OpenBMC from writing proprietary 
> PLDM OEM implementations - libraries which can be dynamically loaded at 
> runtime as opposed to compile time linking in of those implementations 
> to the main PLDM code base, thereby requiring the OEM implementations to 
> exist in the main PLDM code base?

If people are determined to write proprietary OEM implementations they can
fork the codebase. It's Apache 2.0, not GPL, so there's no requirement they
redistribute the associated source. I'd be surprised if we as system designers
backed ourselves into a corner where we were forced to load some third-party
proprietary PLDM plugin, given that we're working on OpenBMC.

Not that I'm advocating anyone forks, but it's not like they don't have any
options. That's my $0.02.

Andrew

> 
> Thanks,
> Deepak
> 
> 
> 
>