Message registries continuation

Message registries continuation

[Matt Spinler]

Hi James,

Picking up the discussion again from 
https://lists.ozlabs.org/pipermail/openbmc/2020-February/020620.html
about reading  in message registries...

When this was left off, I believe we were leaning toward being able to 
copy message registry JSON files
into some target directory on the BMC during the build where bmcweb 
would load them on startup
and leave them in their JSON objects, and they would be pulled from 
there when LogService requests
were made.

This was to be able to support multiple languages, and in general just 
to support other registries besides
the existing OpenBMC one that is hardcoded in a header file.  (We're 
going to have an IBM registry we
use together with our D-Bus logs based LogService.)

An open issue we still had was if these registries had to be validated, 
or if that was left to whoever
made them.  A stake in the ground could be that we leave the OpenBMC 
registry as it is in a header
file, which negates validation, or put it in JSON too and validate just 
that one during the build.
Or if there are any other ideas here...

As far as the directory used, I think that /usr/share/bmcweb/ would be 
appropriate, or maybe
/usr/share/bmcweb/registries/ if either of those are OK with you.

Also, it may be overkill to need to  read in the same registry for every 
language, as there could
be dozens and realistically they will never all be used on a single 
system, but if the desire is only
to load them at startup before the current language is known I don't 
really see a way around it.

Thanks!
Matt

Re: Message registries continuation

[James Feist]

On 6/16/2020 12:47 PM, Matt Spinler wrote:
> Hi James,
> 
> Picking up the discussion again from 
> https://lists.ozlabs.org/pipermail/openbmc/2020-February/020620.html
> about reading  in message registries...
> 
> When this was left off, I believe we were leaning toward being able to 
> copy message registry JSON files
> into some target directory on the BMC during the build where bmcweb 
> would load them on startup
> and leave them in their JSON objects, and they would be pulled from 
> there when LogService requests
> were made.
> 
> This was to be able to support multiple languages, and in general just 
> to support other registries besides
> the existing OpenBMC one that is hardcoded in a header file.  (We're 
> going to have an IBM registry we
> use together with our D-Bus logs based LogService.)

There's 3 registries, fwiw: 
https://github.com/openbmc/bmcweb/tree/master/redfish-core/include/registries

> 
> An open issue we still had was if these registries had to be validated, 
> or if that was left to whoever
> made them.  A stake in the ground could be that we leave the OpenBMC 
> registry as it is in a header
> file, which negates validation, or put it in JSON too and validate just 
> that one during the build.
> Or if there are any other ideas here...

Entity-manager uses valijson, in that way you could validate them 
against a schema: https://github.com/tristanpenman/valijson. It plays 
nicely with nlohmann-json. Although if these are compiled in json files, 
I'm not sure it's a large issue. We could just create a compile-time 
script to validate.

> 
> As far as the directory used, I think that /usr/share/bmcweb/ would be 
> appropriate, or maybe
> /usr/share/bmcweb/registries/ if either of those are OK with you.

I think the directory is a good idea so it can load any json file from 
that directory. Maybe /usr/share/bmcweb/registries/<language> even to 
make it easier to switch between languages?

> 
> Also, it may be overkill to need to  read in the same registry for every 
> language, as there could
> be dozens and realistically they will never all be used on a single 
> system, but if the desire is only
> to load them at startup before the current language is known I don't 
> really see a way around it.

I think this would require a default language and a bmcweb reload if the 
language changed? Its probably ok to change languages after startup, as 
long as the default language is loaded immediately to lower the chances 
of run-time issues. As changing languages is probably a very infrequent 
operation.

> 
> Thanks!
> Matt

Re: Message registries continuation

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

On Tue, Jun 16, 2020 at 01:39:18PM -0700, James Feist wrote:
> On 6/16/2020 12:47 PM, Matt Spinler wrote:
> > Also, it may be overkill to need to  read in the same registry for every 
> > language, as there could
> > be dozens and realistically they will never all be used on a single 
> > system, but if the desire is only
> > to load them at startup before the current language is known I don't 
> > really see a way around it.
> 
> I think this would require a default language and a bmcweb reload if the 
> language changed? Its probably ok to change languages after startup, as 
> long as the default language is loaded immediately to lower the chances 
> of run-time issues. As changing languages is probably a very infrequent 
> operation.

Couldn't two users of the same machine request / require different
languages?  I know that any machine used by the Australian developers on
this project I'd need to change the settings on so that I get the proper
spelling of color.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Message registries continuation

[James Feist]

On 6/16/2020 1:50 PM, Patrick Williams wrote:
> On Tue, Jun 16, 2020 at 01:39:18PM -0700, James Feist wrote:
>> On 6/16/2020 12:47 PM, Matt Spinler wrote:
>>> Also, it may be overkill to need to  read in the same registry for every
>>> language, as there could
>>> be dozens and realistically they will never all be used on a single
>>> system, but if the desire is only
>>> to load them at startup before the current language is known I don't
>>> really see a way around it.
>>
>> I think this would require a default language and a bmcweb reload if the
>> language changed? Its probably ok to change languages after startup, as
>> long as the default language is loaded immediately to lower the chances
>> of run-time issues. As changing languages is probably a very infrequent
>> operation.
> 
> Couldn't two users of the same machine request / require different
> languages?  I know that any machine used by the Australian developers on
> this project I'd need to change the settings on so that I get the proper
> spelling of color.

:)

> 

Yes that's a good point, if you wanted to add language as a user 
setting, you could iterate through users and populate the needed 
languages that way too.

Re: Message registries continuation

[Matt Spinler]

On 6/16/2020 3:39 PM, James Feist wrote:
> On 6/16/2020 12:47 PM, Matt Spinler wrote:
>> Hi James,
>>
>> Picking up the discussion again from 
>> https://lists.ozlabs.org/pipermail/openbmc/2020-February/020620.html
>> about reading  in message registries...
>>
>> When this was left off, I believe we were leaning toward being able 
>> to copy message registry JSON files
>> into some target directory on the BMC during the build where bmcweb 
>> would load them on startup
>> and leave them in their JSON objects, and they would be pulled from 
>> there when LogService requests
>> were made.
>>
>> This was to be able to support multiple languages, and in general 
>> just to support other registries besides
>> the existing OpenBMC one that is hardcoded in a header file. (We're 
>> going to have an IBM registry we
>> use together with our D-Bus logs based LogService.)
>
> There's 3 registries, fwiw: 
> https://github.com/openbmc/bmcweb/tree/master/redfish-core/include/registries
>

Do you want these 3 to remain as header files, or be read from the JSON 
as well?  The base one I can get
as JSON already, and I guess I could write a script to convert the other 
2 if necessary.

>>
>> An open issue we still had was if these registries had to be 
>> validated, or if that was left to whoever
>> made them.  A stake in the ground could be that we leave the OpenBMC 
>> registry as it is in a header
>> file, which negates validation, or put it in JSON too and validate 
>> just that one during the build.
>> Or if there are any other ideas here...
>
> Entity-manager uses valijson, in that way you could validate them 
> against a schema: https://github.com/tristanpenman/valijson. It plays 
> nicely with nlohmann-json. Although if these are compiled in json 
> files, I'm not sure it's a large issue. We could just create a 
> compile-time script to validate.
>
>>
>> As far as the directory used, I think that /usr/share/bmcweb/ would 
>> be appropriate, or maybe
>> /usr/share/bmcweb/registries/ if either of those are OK with you.
>
> I think the directory is a good idea so it can load any json file from 
> that directory. Maybe /usr/share/bmcweb/registries/<language> even to 
> make it easier to switch between languages?
>
>>
>> Also, it may be overkill to need to  read in the same registry for 
>> every language, as there could
>> be dozens and realistically they will never all be used on a single 
>> system, but if the desire is only
>> to load them at startup before the current language is known I don't 
>> really see a way around it.
>
> I think this would require a default language and a bmcweb reload if 
> the language changed? Its probably ok to change languages after 
> startup, as long as the default language is loaded immediately to 
> lower the chances of run-time issues. As changing languages is 
> probably a very infrequent operation.
>
>>
>> Thanks!
>> Matt

Re: Message registries continuation

[James Feist]

On 6/16/2020 2:30 PM, Matt Spinler wrote:
> 
> 
> On 6/16/2020 3:39 PM, James Feist wrote:
>> On 6/16/2020 12:47 PM, Matt Spinler wrote:
>> There's 3 registries, fwiw: 
>> https://github.com/openbmc/bmcweb/tree/master/redfish-core/include/registries 
>>
>>
> 
> Do you want these 3 to remain as header files, or be read from the JSON 
> as well?  The base one I can get
> as JSON already, and I guess I could write a script to convert the other 
> 2 if necessary.

Consistency is always better, but I'd put it under nice to have if you 
have limited time. Task you can also get in JSON from 
https://redfish.dmtf.org/registries/. Personally I'd think having 
multiple examples would help in the development more than anything.

Re: Message registries continuation

[Brad Bishop]

On Tue, 2020-06-16 at 13:39 -0700, James Feist wrote:
> 
> Entity-manager uses valijson, in that way you could validate them 
> against a schema: https://github.com/tristanpenman/valijson. It plays 
> nicely with nlohmann-json. Although if these are compiled in json
> files, I'm not sure it's a large issue. We could just create a
> compile-time script to validate.

FYI for EM, I've actually done this work.  I just need to submit it for
review.

Re: Message registries continuation

[Matt Spinler]

Hi James,

Something I forgot below - when building up our event logs, I have about 
a dozen fields (mostly OEM)
that I have to get from the OpenBMC event log's corresponding PEL (IBM's 
enterprise log format).

PELs aren't on D-Bus for a few reasons, such as they can be several KB 
in size and consist of several
dozen discrete fields, so that rules out bmcweb getting them that way.

I do have a shared library that has the PEL APIs I need (PELs themselves 
are in files). Is it OK if I
just link in that library as needed when a USE_PELs or whatever option 
is set?
Alternatively, I could also dlopen it I suppose.

Just trying to avoid a surprise during review.

Thanks

On 6/16/2020 3:39 PM, James Feist wrote:
> On 6/16/2020 12:47 PM, Matt Spinler wrote:
>> Hi James,
>>
>> Picking up the discussion again from 
>> https://lists.ozlabs.org/pipermail/openbmc/2020-February/020620.html
>> about reading  in message registries...
>>
>> When this was left off, I believe we were leaning toward being able 
>> to copy message registry JSON files
>> into some target directory on the BMC during the build where bmcweb 
>> would load them on startup
>> and leave them in their JSON objects, and they would be pulled from 
>> there when LogService requests
>> were made.
>>
>> This was to be able to support multiple languages, and in general 
>> just to support other registries besides
>> the existing OpenBMC one that is hardcoded in a header file. (We're 
>> going to have an IBM registry we
>> use together with our D-Bus logs based LogService.)
>
> There's 3 registries, fwiw: 
> https://github.com/openbmc/bmcweb/tree/master/redfish-core/include/registries
>
>>
>> An open issue we still had was if these registries had to be 
>> validated, or if that was left to whoever
>> made them.  A stake in the ground could be that we leave the OpenBMC 
>> registry as it is in a header
>> file, which negates validation, or put it in JSON too and validate 
>> just that one during the build.
>> Or if there are any other ideas here...
>
> Entity-manager uses valijson, in that way you could validate them 
> against a schema: https://github.com/tristanpenman/valijson. It plays 
> nicely with nlohmann-json. Although if these are compiled in json 
> files, I'm not sure it's a large issue. We could just create a 
> compile-time script to validate.
>
>>
>> As far as the directory used, I think that /usr/share/bmcweb/ would 
>> be appropriate, or maybe
>> /usr/share/bmcweb/registries/ if either of those are OK with you.
>
> I think the directory is a good idea so it can load any json file from 
> that directory. Maybe /usr/share/bmcweb/registries/<language> even to 
> make it easier to switch between languages?
>
>>
>> Also, it may be overkill to need to  read in the same registry for 
>> every language, as there could
>> be dozens and realistically they will never all be used on a single 
>> system, but if the desire is only
>> to load them at startup before the current language is known I don't 
>> really see a way around it.
>
> I think this would require a default language and a bmcweb reload if 
> the language changed? Its probably ok to change languages after 
> startup, as long as the default language is loaded immediately to 
> lower the chances of run-time issues. As changing languages is 
> probably a very infrequent operation.
>
>>
>> Thanks!
>> Matt

Re: Message registries continuation

[James Feist]

On 6/22/2020 1:46 PM, Matt Spinler wrote:
> Hi James,
> 
> Something I forgot below - when building up our event logs, I have about 
> a dozen fields (mostly OEM)
> that I have to get from the OpenBMC event log's corresponding PEL (IBM's 
> enterprise log format).
> 
> PELs aren't on D-Bus for a few reasons, such as they can be several KB 
> in size and consist of several
> dozen discrete fields, so that rules out bmcweb getting them that way.

Would doing something like having the fields in the journal with a link 
to a file work? See this design for more info: 
https://github.com/openbmc/docs/blob/master/architecture/redfish-logging-in-bmcweb.md

> 
> I do have a shared library that has the PEL APIs I need (PELs themselves 
> are in files). Is it OK if I
> just link in that library as needed when a USE_PELs or whatever option 
> is set?
> Alternatively, I could also dlopen it I suppose.

There's another thread over here 
https://lists.ozlabs.org/pipermail/openbmc/2020-June/022082.html 
happening right now discussing types of logging. As we already have 2 
forms of logging supported, I'm a little hesitant to the idea of a third 
without at least some formal direction of what we want logging to look 
like as a project. More so as we add advanced features on top of 
logging, it makes it more difficult to support different methods.

> 
> Just trying to avoid a surprise during review.
> 
> Thanks
> 

Re: Message registries continuation

[Matt Spinler]

On 6/22/2020 4:16 PM, James Feist wrote:
> On 6/22/2020 1:46 PM, Matt Spinler wrote:
>> Hi James,
>>
>> Something I forgot below - when building up our event logs, I have 
>> about a dozen fields (mostly OEM)
>> that I have to get from the OpenBMC event log's corresponding PEL 
>> (IBM's enterprise log format).
>>
>> PELs aren't on D-Bus for a few reasons, such as they can be several 
>> KB in size and consist of several
>> dozen discrete fields, so that rules out bmcweb getting them that way.
>
> Would doing something like having the fields in the journal with a 
> link to a file work? See this design for more info: 
> https://github.com/openbmc/docs/blob/master/architecture/redfish-logging-in-bmcweb.md
>

I forgot to mention that this would use the existing D-Bus based event 
log code, and just add new fields
to those responses.  At  this point, I don't think I can use the journal 
because I need to ensure the lifetime
of the errors matches that of the event logs on D-Bus and the PELs 
themselves.

>>
>> I do have a shared library that has the PEL APIs I need (PELs 
>> themselves are in files). Is it OK if I
>> just link in that library as needed when a USE_PELs or whatever 
>> option is set?
>> Alternatively, I could also dlopen it I suppose.
>
> There's another thread over here 
> https://lists.ozlabs.org/pipermail/openbmc/2020-June/022082.html 
> happening right now discussing types of logging. As we already have 2 
> forms of logging supported, I'm a little hesitant to the idea of a 
> third without at least some formal direction of what we want logging 
> to look like as a project. More so as we add advanced features on top 
> of logging, it makes it more difficult to support different methods.
>

This would just be an enhancement to the 2nd form that exists today, the 
D-Bus based one, but yea, I can
wait and see how that thread shakes out.


>>
>> Just trying to avoid a surprise during review.
>>
>> Thanks
>>
>