Redfish message registries, plus translation

Redfish message registries, plus translation

[Matt Spinler]

Hi James,

It's looking like in our future we will have to provide:

1) Redfish logs using multiple message registries that probably can't be
    hardcoded into hpp files and checked in.
    - For example, a registry for our host firmware and hypervisor errors,
      as our BMC handles displaying logs for them, that we would pick up
      during our build process.
and

2) Message registry text in multiple languages

Those being my goals, I have a few questions:

In general, is the multi-language strategy to just provide a standalone
registry array for each language which the code then selects based on the
Language property?

To support both of the above, would you be open to having functionality 
added
to read in registries from data files, including based on language? Or, 
would you
have any other ideas for how to support these items?

Thanks!

Re: Redfish message registries, plus translation

[Andrew Jeffery]

On Sat, 15 Feb 2020, at 06:27, Matt Spinler wrote:
> Hi James,
> 
> It's looking like in our future we will have to provide:
> 
> 1) Redfish logs using multiple message registries that probably can't be
>     hardcoded into hpp files and checked in.
>     - For example, a registry for our host firmware and hypervisor errors,
>       as our BMC handles displaying logs for them, that we would pick up
>       during our build process.
> and
> 
> 2) Message registry text in multiple languages
> 
> Those being my goals, I have a few questions:
> 
> In general, is the multi-language strategy to just provide a standalone
> registry array for each language which the code then selects based on the
> Language property?
> 
> To support both of the above, would you be open to having functionality 
> added
> to read in registries from data files, including based on language? Or, 
> would you
> have any other ideas for how to support these items?

Bit of a drive-by question, but shouldn't we be using GNU gettext[1] to
handle translation?

[1] https://www.gnu.org/software/gettext/

Andrew

Re: Redfish message registries, plus translation

[Joseph Reynolds]

On 2/16/20 5:37 PM, Andrew Jeffery wrote:
>
> On Sat, 15 Feb 2020, at 06:27, Matt Spinler wrote:
>> Hi James,
>>
>> It's looking like in our future we will have to provide:
>>
>> 1) Redfish logs using multiple message registries that probably can't be
>>      hardcoded into hpp files and checked in.
>>      - For example, a registry for our host firmware and hypervisor errors,
>>        as our BMC handles displaying logs for them, that we would pick up
>>        during our build process.
>> and
>>
>> 2) Message registry text in multiple languages
>>
>> Those being my goals, I have a few questions:
>>
>> In general, is the multi-language strategy to just provide a standalone
>> registry array for each language which the code then selects based on the
>> Language property?
>>
>> To support both of the above, would you be open to having functionality
>> added
>> to read in registries from data files, including based on language? Or,
>> would you
>> have any other ideas for how to support these items?
> Bit of a drive-by question, but shouldn't we be using GNU gettext[1] to
> handle translation?

Matt's proposal is for BMCWeb to have two languages:

1. The implementation language.  When BMCWeb writes internal log 
entries, these will be in English.  The BMCWeb developers will use these 
to understand what is going on,  and the BMC admin may be able to read 
the, but they are not for BMC users.
We can certainly use gettext to internationalize this. But it is not 
what Matt is asking about.

2. The languages used for Redfish messages.  These message are added to 
the HTTP response body.  These are meant to be read by any BMC REST API 
user.  For example, messages from the Redfish "Base" message registry 
Base.1.7.0.json within DSP8011.  BMCWeb has a hard-coded version of a 
subset of these messages, for example, the AccountModified message in 
https://github.com/openbmc/bmcweb/blob/master/redfish-core/src/error_messages.cpp
These messages are defined in English and meant to be translated. That's 
what Matt is asking about.


Here's my drive-by response.

The current implementation is locked into English-only.  BMCWeb code 
sends messages like this:
1. #include 
https://github.com/openbmc/bmcweb/blob/master/redfish-core/include/error_messages.hpp
2. Write code like: propertyValueModified(res, "myProperty", 
myModifiedValue);
This calls the function in error_messages.cpp which is hard-coded English.

Having hard-coded function calls and hard-coded implementations helps 
with BMCWeb's goal for performance and startup times.
https://github.com/openbmc/bmcweb/blob/master/DEVELOPING.md

However, IMHO, the development process to add new Redfish messages to 
BMCWeb is cumbersome.  It should be as easy as copying Redfish's new 
JSON file.

IMHO, the easiest way to to support multiple languages (including 
English) is to keep the prototypes in error_messages.hpp so all the 
existing uses will continue to work.  Then change the implementation to 
select a message from a JSON file based on the hard-coded message name 
(example: AccountNotModified) and language (en-us, zh-cn, or whatever).

All of the usual issues come up with approach:
- Will BMCWeb pre-load or load-on-demand the JSON file would be 
pre-loaded.  It certainly will not be loaded for every message.
- Fallbacks.  For example, if the ja-jp message files does not have the 
AccountNotModified message, format the en-us message instead.
- Need a default message.  For example, if BMCWeb tries to sent the 
message "NotARealMessage" which is not defined in any message file, it 
might be useful to send the raw data instead.
- We use the Redfish "Base" message registry, and I think we also use an 
OpenBMC Oem message registry as well.  So BMCWeb will have to load 
multiple message registries per language.

More:
- Will a initial set of supported languages be baked into the firmware 
image (on the BMC's file system) in English or some other languages?
- Can the BMC admin add and remove supported languages (by adding or 
deleting JSON message registry files)?
- Will there be REST APIs for that?
- Supported by the Redfish spec?

So what the the requirements?

- Joseph

You can see one open source project's work here: 
http://userguide.icu-project.org/formatparse/messages

>
> [1] https://www.gnu.org/software/gettext/
>
> Andrew

Re: Redfish message registries, plus translation

[Bills, Jason M]

On 2/17/2020 5:53 PM, Joseph Reynolds wrote:
> On 2/16/20 5:37 PM, Andrew Jeffery wrote:
>>
>> On Sat, 15 Feb 2020, at 06:27, Matt Spinler wrote:
>>> Hi James,
>>>
>>> It's looking like in our future we will have to provide:
>>>
>>> 1) Redfish logs using multiple message registries that probably can't be
>>>      hardcoded into hpp files and checked in.
>>>      - For example, a registry for our host firmware and hypervisor 
>>> errors,
>>>        as our BMC handles displaying logs for them, that we would 
>>> pick up
>>>        during our build process.
>>> and
>>>
>>> 2) Message registry text in multiple languages
>>>
>>> Those being my goals, I have a few questions:
>>>
>>> In general, is the multi-language strategy to just provide a standalone
>>> registry array for each language which the code then selects based on 
>>> the
>>> Language property?
>>>
>>> To support both of the above, would you be open to having functionality
>>> added
>>> to read in registries from data files, including based on language? Or,
>>> would you
>>> have any other ideas for how to support these items?
>> Bit of a drive-by question, but shouldn't we be using GNU gettext[1] to
>> handle translation?
> 
> Matt's proposal is for BMCWeb to have two languages:
> 
> 1. The implementation language.  When BMCWeb writes internal log 
> entries, these will be in English.  The BMCWeb developers will use these 
> to understand what is going on,  and the BMC admin may be able to read 
> the, but they are not for BMC users.
> We can certainly use gettext to internationalize this. But it is not 
> what Matt is asking about.
> 
> 2. The languages used for Redfish messages.  These message are added to 
> the HTTP response body.  These are meant to be read by any BMC REST API 
> user.  For example, messages from the Redfish "Base" message registry 
> Base.1.7.0.json within DSP8011.  BMCWeb has a hard-coded version of a 
> subset of these messages, for example, the AccountModified message in 
> https://github.com/openbmc/bmcweb/blob/master/redfish-core/src/error_messages.cpp 
> 
> These messages are defined in English and meant to be translated. That's 
> what Matt is asking about.
> 
> 
> Here's my drive-by response.
> 
> The current implementation is locked into English-only.  BMCWeb code 
> sends messages like this:
> 1. #include 
> https://github.com/openbmc/bmcweb/blob/master/redfish-core/include/error_messages.hpp 
> 
> 2. Write code like: propertyValueModified(res, "myProperty", 
> myModifiedValue);
> This calls the function in error_messages.cpp which is hard-coded English.
> 
> Having hard-coded function calls and hard-coded implementations helps 
> with BMCWeb's goal for performance and startup times.
> https://github.com/openbmc/bmcweb/blob/master/DEVELOPING.md
> 
> However, IMHO, the development process to add new Redfish messages to 
> BMCWeb is cumbersome.  It should be as easy as copying Redfish's new 
> JSON file.
To ease this, there is a script in bmcweb that is designed to parse the 
JSON message registries into the header file.  I just ran it and it has 
a couple minor bugs that were introduced when we changed the format to 
fix build warnings.  I'll push a fix.

One possible solution would be to adapt this script to run automatically 
at build time.  Then any JSON message registry files that get added 
either to bmcweb or as an append could be transparently converted to the 
header file.

> 
> IMHO, the easiest way to to support multiple languages (including 
> English) is to keep the prototypes in error_messages.hpp so all the 
> existing uses will continue to work.  Then change the implementation to 
> select a message from a JSON file based on the hard-coded message name 
> (example: AccountNotModified) and language (en-us, zh-cn, or whatever).
> 
> All of the usual issues come up with approach:
> - Will BMCWeb pre-load or load-on-demand the JSON file would be 
> pre-loaded.  It certainly will not be loaded for every message.
> - Fallbacks.  For example, if the ja-jp message files does not have the 
> AccountNotModified message, format the en-us message instead.
> - Need a default message.  For example, if BMCWeb tries to sent the 
> message "NotARealMessage" which is not defined in any message file, it 
> might be useful to send the raw data instead.
bmcweb currently provides the given MessageId with a blank Message field 
if it cannot be found in a registry.

> - We use the Redfish "Base" message registry, and I think we also use an 
> OpenBMC Oem message registry as well.  So BMCWeb will have to load 
> multiple message registries per language.
> 
> More:
> - Will a initial set of supported languages be baked into the firmware 
> image (on the BMC's file system) in English or some other languages?
> - Can the BMC admin add and remove supported languages (by adding or 
> deleting JSON message registry files)?
> - Will there be REST APIs for that?
> - Supported by the Redfish spec?
> 
> So what the the requirements?
> 
> - Joseph
> 
> You can see one open source project's work here: 
> http://userguide.icu-project.org/formatparse/messages
> 
>>
>> [1] https://www.gnu.org/software/gettext/
>>
>> Andrew
> 

Re: Redfish message registries, plus translation

[Brad Bishop]

> On Feb 18, 2020, at 2:25 PM, Bills, Jason M <jason.m.bills@linux.intel.com> wrote:
> 
> 
> 
> On 2/17/2020 5:53 PM, Joseph Reynolds wrote:
>> On 2/16/20 5:37 PM, Andrew Jeffery wrote:
>>> 
>>> On Sat, 15 Feb 2020, at 06:27, Matt Spinler wrote:
>>>> Hi James,
>>>> 
>>>> It's looking like in our future we will have to provide:
>>>> 
>>>> 1) Redfish logs using multiple message registries that probably can't be
>>>>      hardcoded into hpp files and checked in.
>>>>      - For example, a registry for our host firmware and hypervisor errors,
>>>>        as our BMC handles displaying logs for them, that we would pick up
>>>>        during our build process.
>>>> and
>>>> 
>>>> 2) Message registry text in multiple languages
>>>> 
>>>> Those being my goals, I have a few questions:
>>>> 
>>>> In general, is the multi-language strategy to just provide a standalone
>>>> registry array for each language which the code then selects based on the
>>>> Language property?
>>>> 
>>>> To support both of the above, would you be open to having functionality
>>>> added
>>>> to read in registries from data files, including based on language? Or,
>>>> would you
>>>> have any other ideas for how to support these items?
>>> Bit of a drive-by question, but shouldn't we be using GNU gettext[1] to
>>> handle translation?
>> Matt's proposal is for BMCWeb to have two languages:
>> 1. The implementation language.  When BMCWeb writes internal log entries, these will be in English.  The BMCWeb developers will use these to understand what is going on,  and the BMC admin may be able to read the, but they are not for BMC users.
>> We can certainly use gettext to internationalize this. But it is not what Matt is asking about.
>> 2. The languages used for Redfish messages.  These message are added to the HTTP response body.  These are meant to be read by any BMC REST API user.  For example, messages from the Redfish "Base" message registry Base.1.7.0.json within DSP8011.  BMCWeb has a hard-coded version of a subset of these messages, for example, the AccountModified message in https://github.com/openbmc/bmcweb/blob/master/redfish-core/src/error_messages.cpp These messages are defined in English and meant to be translated. That's what Matt is asking about.
>> Here's my drive-by response.
>> The current implementation is locked into English-only.  BMCWeb code sends messages like this:
>> 1. #include https://github.com/openbmc/bmcweb/blob/master/redfish-core/include/error_messages.hpp 2. Write code like: propertyValueModified(res, "myProperty", myModifiedValue);
>> This calls the function in error_messages.cpp which is hard-coded English.
>> Having hard-coded function calls and hard-coded implementations helps with BMCWeb's goal for performance and startup times.
>> https://github.com/openbmc/bmcweb/blob/master/DEVELOPING.md
>> However, IMHO, the development process to add new Redfish messages to BMCWeb is cumbersome.  It should be as easy as copying Redfish's new JSON file.
> To ease this, there is a script in bmcweb that is designed to parse the JSON message registries into the header file.  

So I’m a total bmcweb noob - but any reason why the runtime can’t just parse the json?  Why does it have to get converted to a header?

Re: Redfish message registries, plus translation

[Joseph Reynolds]

On 2/18/20 1:37 PM, Brad Bishop wrote:
>> On Feb 18, 2020, at 2:25 PM, Bills, Jason M <jason.m.bills@linux.intel.com> wrote:
>>
>>
>> On 2/17/2020 5:53 PM, Joseph Reynolds wrote:
>>> On 2/16/20 5:37 PM, Andrew Jeffery wrote:
>>>> On Sat, 15 Feb 2020, at 06:27, Matt Spinler wrote:
>>>>> Hi James,
>>>>>
>>>>> It's looking like in our future we will have to provide:
>>>>>
>>>>> 1) Redfish logs using multiple message registries that probably can't be
>>>>>       hardcoded into hpp files and checked in.
>>>>>       - For example, a registry for our host firmware and hypervisor errors,
>>>>>         as our BMC handles displaying logs for them, that we would pick up
>>>>>         during our build process.
>>>>> and
>>>>>
>>>>> 2) Message registry text in multiple languages
>>>>>
>>>>> Those being my goals, I have a few questions:
>>>>>
>>>>> In general, is the multi-language strategy to just provide a standalone
>>>>> registry array for each language which the code then selects based on the
>>>>> Language property?
>>>>>
>>>>> To support both of the above, would you be open to having functionality
>>>>> added
>>>>> to read in registries from data files, including based on language? Or,
>>>>> would you
>>>>> have any other ideas for how to support these items?
>>>> Bit of a drive-by question, but shouldn't we be using GNU gettext[1] to
>>>> handle translation?
>>> Matt's proposal is for BMCWeb to have two languages:
>>> 1. The implementation language.  When BMCWeb writes internal log entries, these will be in English.  The BMCWeb developers will use these to understand what is going on,  and the BMC admin may be able to read the, but they are not for BMC users.
>>> We can certainly use gettext to internationalize this. But it is not what Matt is asking about.
>>> 2. The languages used for Redfish messages.  These message are added to the HTTP response body.  These are meant to be read by any BMC REST API user.  For example, messages from the Redfish "Base" message registry Base.1.7.0.json within DSP8011.  BMCWeb has a hard-coded version of a subset of these messages, for example, the AccountModified message in https://github.com/openbmc/bmcweb/blob/master/redfish-core/src/error_messages.cpp These messages are defined in English and meant to be translated. That's what Matt is asking about.
>>> Here's my drive-by response.
>>> The current implementation is locked into English-only.  BMCWeb code sends messages like this:
>>> 1. #include https://github.com/openbmc/bmcweb/blob/master/redfish-core/include/error_messages.hpp 2. Write code like: propertyValueModified(res, "myProperty", myModifiedValue);
>>> This calls the function in error_messages.cpp which is hard-coded English.
>>> Having hard-coded function calls and hard-coded implementations helps with BMCWeb's goal for performance and startup times.
>>> https://github.com/openbmc/bmcweb/blob/master/DEVELOPING.md
>>> However, IMHO, the development process to add new Redfish messages to BMCWeb is cumbersome.  It should be as easy as copying Redfish's new JSON file.
>> To ease this, there is a script in bmcweb that is designed to parse the JSON message registries into the header file.
> So I’m a total bmcweb noob - but any reason why the runtime can’t just parse the json?  Why does it have to get converted to a header?

It can.  A few thoughts:

1. As I mentioned above: <quote> Having hard-coded function calls and 
hard-coded implementations helps with BMCWeb's goal for performance and 
startup times. 
https://github.com/openbmc/bmcweb/blob/master/DEVELOPING.md </quote>
Having to read message registries during startup impacts performance.  I 
guess bmcweb startup times would be a tiny bit slower.  Redfish message 
registries are currently 40Kb ... but will grow fatter over time.

2. Having to read message registries during startup time is another 
thing that can go wrong.  An error at build-time, on the BMC's file 
system, or when bmcweb reads the files may prevent BMCWeb from issuing 
meaningful Redfish messages.

3. Presumably one of the hard-coded messages registries would be the 
ultimate fallback message registry, that is, it will be used when the 
required message cannot be found in the message registry for the 
requested language.  Not having a hard-coded message registry removes 
this option.

4. If you are okay with the items above then we can consider a 
build-time BMCWeb config option to build without any built-in message 
registry and rely only on reading message registries from the BMC's file 
system at BMCWeb startup time.  Here are some thoughts:
- The BMCWeb build process could just copy the Redfish message registry 
into a directory BMCWeb reads during startup.
- We would need new code in BMCWeb to read and parse the message 
registries.  The JSON parser is already available within bmcweb.
- We would need new code in BMCWeb to issue messages from the message 
registries.


IMHO, the prospect of removing support for the built-in Redfish "Base" 
message registry seems do-able.  The potential problems and 
disadvantages outlined above seem small.  But I don't have a strong 
opinion either way.

- Joseph

Re: Redfish message registries, plus translation

[James Feist]

On 2/18/20 11:25 AM, Bills, Jason M wrote:
> 
> 
> On 2/17/2020 5:53 PM, Joseph Reynolds wrote:
>> On 2/16/20 5:37 PM, Andrew Jeffery wrote:
>>>
>>> On Sat, 15 Feb 2020, at 06:27, Matt Spinler wrote:
>>>> Hi James,
>>>>
>>>> It's looking like in our future we will have to provide:
>>>>
>>>> 1) Redfish logs using multiple message registries that probably 
>>>> can't be
>>>>      hardcoded into hpp files and checked in.
>>>>      - For example, a registry for our host firmware and hypervisor 
>>>> errors,
>>>>        as our BMC handles displaying logs for them, that we would 
>>>> pick up
>>>>        during our build process.
>>>> and
>>>>
>>>> 2) Message registry text in multiple languages
>>>>
>>>> Those being my goals, I have a few questions:
>>>>
>>>> In general, is the multi-language strategy to just provide a standalone
>>>> registry array for each language which the code then selects based 
>>>> on the
>>>> Language property?
>>>>
>>>> To support both of the above, would you be open to having functionality
>>>> added
>>>> to read in registries from data files, including based on language? Or,
>>>> would you
>>>> have any other ideas for how to support these items?
>>> Bit of a drive-by question, but shouldn't we be using GNU gettext[1] to
>>> handle translation?
>>
>> Matt's proposal is for BMCWeb to have two languages:
>>
>> 1. The implementation language.  When BMCWeb writes internal log 
>> entries, these will be in English.  The BMCWeb developers will use 
>> these to understand what is going on,  and the BMC admin may be able 
>> to read the, but they are not for BMC users.
>> We can certainly use gettext to internationalize this. But it is not 
>> what Matt is asking about.
>>
>> 2. The languages used for Redfish messages.  These message are added 
>> to the HTTP response body.  These are meant to be read by any BMC REST 
>> API user.  For example, messages from the Redfish "Base" message 
>> registry Base.1.7.0.json within DSP8011.  BMCWeb has a hard-coded 
>> version of a subset of these messages, for example, the 
>> AccountModified message in 
>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/src/error_messages.cpp 
>>
>> These messages are defined in English and meant to be translated. 
>> That's what Matt is asking about.
>>
>>
>> Here's my drive-by response.
>>
>> The current implementation is locked into English-only.  BMCWeb code 
>> sends messages like this:
>> 1. #include 
>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/include/error_messages.hpp 
>>
>> 2. Write code like: propertyValueModified(res, "myProperty", 
>> myModifiedValue);
>> This calls the function in error_messages.cpp which is hard-coded 
>> English.
>>
>> Having hard-coded function calls and hard-coded implementations helps 
>> with BMCWeb's goal for performance and startup times.
>> https://github.com/openbmc/bmcweb/blob/master/DEVELOPING.md
>>
>> However, IMHO, the development process to add new Redfish messages to 
>> BMCWeb is cumbersome.  It should be as easy as copying Redfish's new 
>> JSON file.
> To ease this, there is a script in bmcweb that is designed to parse the 
> JSON message registries into the header file.  I just ran it and it has 
> a couple minor bugs that were introduced when we changed the format to 
> fix build warnings.  I'll push a fix.
> 
> One possible solution would be to adapt this script to run automatically 
> at build time.  Then any JSON message registry files that get added 
> either to bmcweb or as an append could be transparently converted to the 
> header file.

An added benefit of this is that it could verify JSON is in legal format 
before it makes it to the BMC.

> 
>>
>> IMHO, the easiest way to to support multiple languages (including 
>> English) is to keep the prototypes in error_messages.hpp so all the 
>> existing uses will continue to work.  Then change the implementation 
>> to select a message from a JSON file based on the hard-coded message 
>> name (example: AccountNotModified) and language (en-us, zh-cn, or 
>> whatever).


Is there any reason to have this update-able at runtime? Regardless to 
limit the possibility of errors I'd like to avoid constant pulling from 
JSON. I'd rather see the JSON being pulled at application start, so we 
take any impact (timing or bugs/crashes) early. However I'm not sure I 
know of a use-case to have this be done at runtime.


>>
>> All of the usual issues come up with approach:
>> - Will BMCWeb pre-load or load-on-demand the JSON file would be 
>> pre-loaded.  It certainly will not be loaded for every message.
>> - Fallbacks.  For example, if the ja-jp message files does not have 
>> the AccountNotModified message, format the en-us message instead.
>> - Need a default message.  For example, if BMCWeb tries to sent the 
>> message "NotARealMessage" which is not defined in any message file, it 
>> might be useful to send the raw data instead.
> bmcweb currently provides the given MessageId with a blank Message field 
> if it cannot be found in a registry.
> 
>> - We use the Redfish "Base" message registry, and I think we also use 
>> an OpenBMC Oem message registry as well.  So BMCWeb will have to load 
>> multiple message registries per language.
>>
>> More:
>> - Will a initial set of supported languages be baked into the firmware 
>> image (on the BMC's file system) in English or some other languages?
>> - Can the BMC admin add and remove supported languages (by adding or 
>> deleting JSON message registry files)?
>> - Will there be REST APIs for that?
>> - Supported by the Redfish spec?
>>
>> So what the the requirements?
>>
>> - Joseph
>>
>> You can see one open source project's work here: 
>> http://userguide.icu-project.org/formatparse/messages
>>
>>>
>>> [1] https://www.gnu.org/software/gettext/
>>>
>>> Andrew
>>

Re: Redfish message registries, plus translation

[Joseph Reynolds]

On 2/18/20 4:02 PM, James Feist wrote:
> On 2/18/20 11:25 AM, Bills, Jason M wrote:
>>
>>
>> On 2/17/2020 5:53 PM, Joseph Reynolds wrote:
>>> On 2/16/20 5:37 PM, Andrew Jeffery wrote:
>>>>
>>>> On Sat, 15 Feb 2020, at 06:27, Matt Spinler wrote:
>>>>> Hi James,
>>>>>
>>>>> It's looking like in our future we will have to provide:
>>>>>
>>>>> 1) Redfish logs using multiple message registries that probably 
>>>>> can't be
>>>>>      hardcoded into hpp files and checked in.
>>>>>      - For example, a registry for our host firmware and 
>>>>> hypervisor errors,
>>>>>        as our BMC handles displaying logs for them, that we would 
>>>>> pick up
>>>>>        during our build process.
>>>>> and
>>>>>
>>>>> 2) Message registry text in multiple languages
>>>>>
>>>>> Those being my goals, I have a few questions:
>>>>>
>>>>> In general, is the multi-language strategy to just provide a 
>>>>> standalone
>>>>> registry array for each language which the code then selects based 
>>>>> on the
>>>>> Language property?
>>>>>
>>>>> To support both of the above, would you be open to having 
>>>>> functionality
>>>>> added
>>>>> to read in registries from data files, including based on 
>>>>> language? Or,
>>>>> would you
>>>>> have any other ideas for how to support these items?
>>>> Bit of a drive-by question, but shouldn't we be using GNU 
>>>> gettext[1] to
>>>> handle translation?
>>>
>>> Matt's proposal is for BMCWeb to have two languages:
>>>
>>> 1. The implementation language.  When BMCWeb writes internal log 
>>> entries, these will be in English.  The BMCWeb developers will use 
>>> these to understand what is going on,  and the BMC admin may be able 
>>> to read the, but they are not for BMC users.
>>> We can certainly use gettext to internationalize this. But it is not 
>>> what Matt is asking about.
>>>
>>> 2. The languages used for Redfish messages.  These message are added 
>>> to the HTTP response body.  These are meant to be read by any BMC 
>>> REST API user.  For example, messages from the Redfish "Base" 
>>> message registry Base.1.7.0.json within DSP8011.  BMCWeb has a 
>>> hard-coded version of a subset of these messages, for example, the 
>>> AccountModified message in 
>>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/src/error_messages.cpp 
>>>
>>> These messages are defined in English and meant to be translated. 
>>> That's what Matt is asking about.
>>>
>>>
>>> Here's my drive-by response.
>>>
>>> The current implementation is locked into English-only. BMCWeb code 
>>> sends messages like this:
>>> 1. #include 
>>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/include/error_messages.hpp 
>>>
>>> 2. Write code like: propertyValueModified(res, "myProperty", 
>>> myModifiedValue);
>>> This calls the function in error_messages.cpp which is hard-coded 
>>> English.
>>>
>>> Having hard-coded function calls and hard-coded implementations 
>>> helps with BMCWeb's goal for performance and startup times.
>>> https://github.com/openbmc/bmcweb/blob/master/DEVELOPING.md
>>>
>>> However, IMHO, the development process to add new Redfish messages 
>>> to BMCWeb is cumbersome.  It should be as easy as copying Redfish's 
>>> new JSON file.
>> To ease this, there is a script in bmcweb that is designed to parse 
>> the JSON message registries into the header file.  I just ran it and 
>> it has a couple minor bugs that were introduced when we changed the 
>> format to fix build warnings.  I'll push a fix.
>>
>> One possible solution would be to adapt this script to run 
>> automatically at build time.  Then any JSON message registry files 
>> that get added either to bmcweb or as an append could be 
>> transparently converted to the header file.
>
> An added benefit of this is that it could verify JSON is in legal 
> format before it makes it to the BMC.

Agreed.  As an alternative, we could enhance the BMCWeb build to 
validate the JSON message registry files that are destined for use by 
BMCWeb.  That would catch most errors, but be weaker than converting 
them to header files: it leaves a window where those files could be 
modified after they are packaged into the image and before they are read 
into bmcweb.

>
>>
>>>
>>> IMHO, the easiest way to to support multiple languages (including 
>>> English) is to keep the prototypes in error_messages.hpp so all the 
>>> existing uses will continue to work.  Then change the implementation 
>>> to select a message from a JSON file based on the hard-coded message 
>>> name (example: AccountNotModified) and language (en-us, zh-cn, or 
>>> whatever).
>
>
> Is there any reason to have this update-able at runtime? Regardless to 
> limit the possibility of errors I'd like to avoid constant pulling 
> from JSON. I'd rather see the JSON being pulled at application start, 
> so we take any impact (timing or bugs/crashes) early. However I'm not 
> sure I know of a use-case to have this be done at runtime.

I did not mean for the JSON files to be modifiable at runtime; I see no 
use case for that.  I imagined that BMCWeb would read any given JSON 
file exactly once and stay in bmcweb heap storage until the server 
terminates.  The simplest design is to read all of the JSON files from 
its message registry directory: all messages files (Base, Oem.Redfish, 
etc.) and all language versions (en-us, cn-zh, etc.). If the registry 
gets large because of many messages multiplied by many translated 
versions and memory consumption is a concern, then we can consider 
just-in-time reading JSON files for that language. (But let's not go 
there now.)

To be clear: I am also not advocating for the JSON message translation 
files to be updatable by any BMC interface.  As far as I am concerned, 
they can be baked into the image.  Doing it this way, the messages can 
be updated by a firmware update.

- Joseph

>
>>>
>>> All of the usual issues come up with approach:
>>> - Will BMCWeb pre-load or load-on-demand the JSON file would be 
>>> pre-loaded.  It certainly will not be loaded for every message.
>>> - Fallbacks.  For example, if the ja-jp message files does not have 
>>> the AccountNotModified message, format the en-us message instead.
>>> - Need a default message.  For example, if BMCWeb tries to sent the 
>>> message "NotARealMessage" which is not defined in any message file, 
>>> it might be useful to send the raw data instead.
>> bmcweb currently provides the given MessageId with a blank Message 
>> field if it cannot be found in a registry.
>>
>>> - We use the Redfish "Base" message registry, and I think we also 
>>> use an OpenBMC Oem message registry as well.  So BMCWeb will have to 
>>> load multiple message registries per language.
>>>
>>> More:
>>> - Will a initial set of supported languages be baked into the 
>>> firmware image (on the BMC's file system) in English or some other 
>>> languages?
>>> - Can the BMC admin add and remove supported languages (by adding or 
>>> deleting JSON message registry files)?
>>> - Will there be REST APIs for that?
>>> - Supported by the Redfish spec?
>>>
>>> So what the the requirements?
>>>
>>> - Joseph
>>>
>>> You can see one open source project's work here: 
>>> http://userguide.icu-project.org/formatparse/messages
>>>
>>>>
>>>> [1] https://www.gnu.org/software/gettext/
>>>>
>>>> Andrew
>>>

Re: Redfish message registries, plus translation

[James Feist]

On 2/19/20 9:07 AM, Joseph Reynolds wrote:
> On 2/18/20 4:02 PM, James Feist wrote:
>> On 2/18/20 11:25 AM, Bills, Jason M wrote:
>>>
>>>
>>> On 2/17/2020 5:53 PM, Joseph Reynolds wrote:
>>>> On 2/16/20 5:37 PM, Andrew Jeffery wrote:
>>>>>
>>>>> On Sat, 15 Feb 2020, at 06:27, Matt Spinler wrote:
>>>>>> Hi James,
>>>>>>
>>>>>> It's looking like in our future we will have to provide:
>>>>>>
>>>>>> 1) Redfish logs using multiple message registries that probably 
>>>>>> can't be
>>>>>>      hardcoded into hpp files and checked in.
>>>>>>      - For example, a registry for our host firmware and 
>>>>>> hypervisor errors,
>>>>>>        as our BMC handles displaying logs for them, that we would 
>>>>>> pick up
>>>>>>        during our build process.
>>>>>> and
>>>>>>
>>>>>> 2) Message registry text in multiple languages
>>>>>>
>>>>>> Those being my goals, I have a few questions:
>>>>>>
>>>>>> In general, is the multi-language strategy to just provide a 
>>>>>> standalone
>>>>>> registry array for each language which the code then selects based 
>>>>>> on the
>>>>>> Language property?
>>>>>>
>>>>>> To support both of the above, would you be open to having 
>>>>>> functionality
>>>>>> added
>>>>>> to read in registries from data files, including based on 
>>>>>> language? Or,
>>>>>> would you
>>>>>> have any other ideas for how to support these items?
>>>>> Bit of a drive-by question, but shouldn't we be using GNU 
>>>>> gettext[1] to
>>>>> handle translation?
>>>>
>>>> Matt's proposal is for BMCWeb to have two languages:
>>>>
>>>> 1. The implementation language.  When BMCWeb writes internal log 
>>>> entries, these will be in English.  The BMCWeb developers will use 
>>>> these to understand what is going on,  and the BMC admin may be able 
>>>> to read the, but they are not for BMC users.
>>>> We can certainly use gettext to internationalize this. But it is not 
>>>> what Matt is asking about.
>>>>
>>>> 2. The languages used for Redfish messages.  These message are added 
>>>> to the HTTP response body.  These are meant to be read by any BMC 
>>>> REST API user.  For example, messages from the Redfish "Base" 
>>>> message registry Base.1.7.0.json within DSP8011.  BMCWeb has a 
>>>> hard-coded version of a subset of these messages, for example, the 
>>>> AccountModified message in 
>>>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/src/error_messages.cpp 
>>>>
>>>> These messages are defined in English and meant to be translated. 
>>>> That's what Matt is asking about.
>>>>
>>>>
>>>> Here's my drive-by response.
>>>>
>>>> The current implementation is locked into English-only. BMCWeb code 
>>>> sends messages like this:
>>>> 1. #include 
>>>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/include/error_messages.hpp 
>>>>
>>>> 2. Write code like: propertyValueModified(res, "myProperty", 
>>>> myModifiedValue);
>>>> This calls the function in error_messages.cpp which is hard-coded 
>>>> English.
>>>>
>>>> Having hard-coded function calls and hard-coded implementations 
>>>> helps with BMCWeb's goal for performance and startup times.
>>>> https://github.com/openbmc/bmcweb/blob/master/DEVELOPING.md
>>>>
>>>> However, IMHO, the development process to add new Redfish messages 
>>>> to BMCWeb is cumbersome.  It should be as easy as copying Redfish's 
>>>> new JSON file.
>>> To ease this, there is a script in bmcweb that is designed to parse 
>>> the JSON message registries into the header file.  I just ran it and 
>>> it has a couple minor bugs that were introduced when we changed the 
>>> format to fix build warnings.  I'll push a fix.
>>>
>>> One possible solution would be to adapt this script to run 
>>> automatically at build time.  Then any JSON message registry files 
>>> that get added either to bmcweb or as an append could be 
>>> transparently converted to the header file.
>>
>> An added benefit of this is that it could verify JSON is in legal 
>> format before it makes it to the BMC.
> 
> Agreed.  As an alternative, we could enhance the BMCWeb build to 
> validate the JSON message registry files that are destined for use by 
> BMCWeb.  That would catch most errors, but be weaker than converting 
> them to header files: it leaves a window where those files could be 
> modified after they are packaged into the image and before they are read 
> into bmcweb.

Also if someone uses a bbappend and patches in their own files after the 
build completes (or using a different recipe entirely), they won't be 
validated.

> 
>>
>>>
>>>>
>>>> IMHO, the easiest way to to support multiple languages (including 
>>>> English) is to keep the prototypes in error_messages.hpp so all the 
>>>> existing uses will continue to work.  Then change the implementation 
>>>> to select a message from a JSON file based on the hard-coded message 
>>>> name (example: AccountNotModified) and language (en-us, zh-cn, or 
>>>> whatever).
>>
>>
>> Is there any reason to have this update-able at runtime? Regardless to 
>> limit the possibility of errors I'd like to avoid constant pulling 
>> from JSON. I'd rather see the JSON being pulled at application start, 
>> so we take any impact (timing or bugs/crashes) early. However I'm not 
>> sure I know of a use-case to have this be done at runtime.
> 
> I did not mean for the JSON files to be modifiable at runtime; I see no 
> use case for that.  I imagined that BMCWeb would read any given JSON 
> file exactly once and stay in bmcweb heap storage until the server 
> terminates.  The simplest design is to read all of the JSON files from 
> its message registry directory: all messages files (Base, Oem.Redfish, 
> etc.) and all language versions (en-us, cn-zh, etc.). If the registry 
> gets large because of many messages multiplied by many translated 
> versions and memory consumption is a concern, then we can consider 
> just-in-time reading JSON files for that language. (But let's not go 
> there now.)
> 
> To be clear: I am also not advocating for the JSON message translation 
> files to be updatable by any BMC interface.  As far as I am concerned, 
> they can be baked into the image.  Doing it this way, the messages can 
> be updated by a firmware update.

Great, I was just wondering if there was some use case I missed.

> 
> - Joseph
> 
>>
>>>>
>>>> All of the usual issues come up with approach:
>>>> - Will BMCWeb pre-load or load-on-demand the JSON file would be 
>>>> pre-loaded.  It certainly will not be loaded for every message.
>>>> - Fallbacks.  For example, if the ja-jp message files does not have 
>>>> the AccountNotModified message, format the en-us message instead.
>>>> - Need a default message.  For example, if BMCWeb tries to sent the 
>>>> message "NotARealMessage" which is not defined in any message file, 
>>>> it might be useful to send the raw data instead.
>>> bmcweb currently provides the given MessageId with a blank Message 
>>> field if it cannot be found in a registry.
>>>
>>>> - We use the Redfish "Base" message registry, and I think we also 
>>>> use an OpenBMC Oem message registry as well.  So BMCWeb will have to 
>>>> load multiple message registries per language.
>>>>
>>>> More:
>>>> - Will a initial set of supported languages be baked into the 
>>>> firmware image (on the BMC's file system) in English or some other 
>>>> languages?
>>>> - Can the BMC admin add and remove supported languages (by adding or 
>>>> deleting JSON message registry files)?
>>>> - Will there be REST APIs for that?
>>>> - Supported by the Redfish spec?
>>>>
>>>> So what the the requirements?
>>>>
>>>> - Joseph
>>>>
>>>> You can see one open source project's work here: 
>>>> http://userguide.icu-project.org/formatparse/messages
>>>>
>>>>>
>>>>> [1] https://www.gnu.org/software/gettext/
>>>>>
>>>>> Andrew
>>>>
> 

Re: Redfish message registries, plus translation

[Matt Spinler]

On 2/19/2020 12:03 PM, James Feist wrote:
> On 2/19/20 9:07 AM, Joseph Reynolds wrote:
>> On 2/18/20 4:02 PM, James Feist wrote:
>>> On 2/18/20 11:25 AM, Bills, Jason M wrote:
>>>>
>>>>
>>>> On 2/17/2020 5:53 PM, Joseph Reynolds wrote:
>>>>> On 2/16/20 5:37 PM, Andrew Jeffery wrote:
>>>>>>
>>>>>> On Sat, 15 Feb 2020, at 06:27, Matt Spinler wrote:
>>>>>>> Hi James,
>>>>>>>
>>>>>>> It's looking like in our future we will have to provide:
>>>>>>>
>>>>>>> 1) Redfish logs using multiple message registries that probably 
>>>>>>> can't be
>>>>>>>      hardcoded into hpp files and checked in.
>>>>>>>      - For example, a registry for our host firmware and 
>>>>>>> hypervisor errors,
>>>>>>>        as our BMC handles displaying logs for them, that we 
>>>>>>> would pick up
>>>>>>>        during our build process.
>>>>>>> and
>>>>>>>
>>>>>>> 2) Message registry text in multiple languages
>>>>>>>
>>>>>>> Those being my goals, I have a few questions:
>>>>>>>
>>>>>>> In general, is the multi-language strategy to just provide a 
>>>>>>> standalone
>>>>>>> registry array for each language which the code then selects 
>>>>>>> based on the
>>>>>>> Language property?
>>>>>>>
>>>>>>> To support both of the above, would you be open to having 
>>>>>>> functionality
>>>>>>> added
>>>>>>> to read in registries from data files, including based on 
>>>>>>> language? Or,
>>>>>>> would you
>>>>>>> have any other ideas for how to support these items?
>>>>>> Bit of a drive-by question, but shouldn't we be using GNU 
>>>>>> gettext[1] to
>>>>>> handle translation?
>>>>>
>>>>> Matt's proposal is for BMCWeb to have two languages:
>>>>>
>>>>> 1. The implementation language.  When BMCWeb writes internal log 
>>>>> entries, these will be in English.  The BMCWeb developers will use 
>>>>> these to understand what is going on,  and the BMC admin may be 
>>>>> able to read the, but they are not for BMC users.
>>>>> We can certainly use gettext to internationalize this. But it is 
>>>>> not what Matt is asking about.
>>>>>
>>>>> 2. The languages used for Redfish messages.  These message are 
>>>>> added to the HTTP response body.  These are meant to be read by 
>>>>> any BMC REST API user.  For example, messages from the Redfish 
>>>>> "Base" message registry Base.1.7.0.json within DSP8011.  BMCWeb 
>>>>> has a hard-coded version of a subset of these messages, for 
>>>>> example, the AccountModified message in 
>>>>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/src/error_messages.cpp 
>>>>>
>>>>> These messages are defined in English and meant to be translated. 
>>>>> That's what Matt is asking about.
>>>>>
>>>>>
>>>>> Here's my drive-by response.
>>>>>
>>>>> The current implementation is locked into English-only. BMCWeb 
>>>>> code sends messages like this:
>>>>> 1. #include 
>>>>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/include/error_messages.hpp 
>>>>>
>>>>> 2. Write code like: propertyValueModified(res, "myProperty", 
>>>>> myModifiedValue);
>>>>> This calls the function in error_messages.cpp which is hard-coded 
>>>>> English.
>>>>>
>>>>> Having hard-coded function calls and hard-coded implementations 
>>>>> helps with BMCWeb's goal for performance and startup times.
>>>>> https://github.com/openbmc/bmcweb/blob/master/DEVELOPING.md
>>>>>
>>>>> However, IMHO, the development process to add new Redfish messages 
>>>>> to BMCWeb is cumbersome.  It should be as easy as copying 
>>>>> Redfish's new JSON file.
>>>> To ease this, there is a script in bmcweb that is designed to parse 
>>>> the JSON message registries into the header file. I just ran it and 
>>>> it has a couple minor bugs that were introduced when we changed the 
>>>> format to fix build warnings.  I'll push a fix.
>>>>
>>>> One possible solution would be to adapt this script to run 
>>>> automatically at build time.  Then any JSON message registry files 
>>>> that get added either to bmcweb or as an append could be 
>>>> transparently converted to the header file.
>>>
>>> An added benefit of this is that it could verify JSON is in legal 
>>> format before it makes it to the BMC.
>>
>> Agreed.  As an alternative, we could enhance the BMCWeb build to 
>> validate the JSON message registry files that are destined for use by 
>> BMCWeb.  That would catch most errors, but be weaker than converting 
>> them to header files: it leaves a window where those files could be 
>> modified after they are packaged into the image and before they are 
>> read into bmcweb.
>
> Also if someone uses a bbappend and patches in their own files after 
> the build completes (or using a different recipe entirely), they won't 
> be validated.
>
>>
>>>
>>>>
>>>>>
>>>>> IMHO, the easiest way to to support multiple languages (including 
>>>>> English) is to keep the prototypes in error_messages.hpp so all 
>>>>> the existing uses will continue to work.  Then change the 
>>>>> implementation to select a message from a JSON file based on the 
>>>>> hard-coded message name (example: AccountNotModified) and language 
>>>>> (en-us, zh-cn, or whatever).
>>>
>>>
>>> Is there any reason to have this update-able at runtime? Regardless 
>>> to limit the possibility of errors I'd like to avoid constant 
>>> pulling from JSON. I'd rather see the JSON being pulled at 
>>> application start, so we take any impact (timing or bugs/crashes) 
>>> early. However I'm not sure I know of a use-case to have this be 
>>> done at runtime.
>>
>> I did not mean for the JSON files to be modifiable at runtime; I see 
>> no use case for that.  I imagined that BMCWeb would read any given 
>> JSON file exactly once and stay in bmcweb heap storage until the 
>> server terminates.  The simplest design is to read all of the JSON 
>> files from its message registry directory: all messages files (Base, 
>> Oem.Redfish, etc.) and all language versions (en-us, cn-zh, etc.). If 
>> the registry gets large because of many messages multiplied by many 
>> translated versions and memory consumption is a concern, then we can 
>> consider just-in-time reading JSON files for that language. (But 
>> let's not go there now.)
>>
>> To be clear: I am also not advocating for the JSON message 
>> translation files to be updatable by any BMC interface.  As far as I 
>> am concerned, they can be baked into the image.  Doing it this way, 
>> the messages can be updated by a firmware update.
>
> Great, I was just wondering if there was some use case I missed.

Thanks for all of the replies.

To address Andrew's earlier suggestion of gettext, that seems to only 
apply to hardcoded
strings in code, which wouldn't help if we were to pull in other message 
registry data files.

I think that for the sake of extensibility, being able to read in the 
message registries from JSON
files is a big plus, but is that a worse translation scheme?

Does anyone with translation experience have any thoughts on the 2 
different methods?

>
>>
>> - Joseph
>>
>>>
>>>>>
>>>>> All of the usual issues come up with approach:
>>>>> - Will BMCWeb pre-load or load-on-demand the JSON file would be 
>>>>> pre-loaded.  It certainly will not be loaded for every message.
>>>>> - Fallbacks.  For example, if the ja-jp message files does not 
>>>>> have the AccountNotModified message, format the en-us message 
>>>>> instead.
>>>>> - Need a default message.  For example, if BMCWeb tries to sent 
>>>>> the message "NotARealMessage" which is not defined in any message 
>>>>> file, it might be useful to send the raw data instead.
>>>> bmcweb currently provides the given MessageId with a blank Message 
>>>> field if it cannot be found in a registry.
>>>>
>>>>> - We use the Redfish "Base" message registry, and I think we also 
>>>>> use an OpenBMC Oem message registry as well.  So BMCWeb will have 
>>>>> to load multiple message registries per language.
>>>>>
>>>>> More:
>>>>> - Will a initial set of supported languages be baked into the 
>>>>> firmware image (on the BMC's file system) in English or some other 
>>>>> languages?
>>>>> - Can the BMC admin add and remove supported languages (by adding 
>>>>> or deleting JSON message registry files)?
>>>>> - Will there be REST APIs for that?
>>>>> - Supported by the Redfish spec?
>>>>>
>>>>> So what the the requirements?
>>>>>
>>>>> - Joseph
>>>>>
>>>>> You can see one open source project's work here: 
>>>>> http://userguide.icu-project.org/formatparse/messages
>>>>>
>>>>>>
>>>>>> [1] https://www.gnu.org/software/gettext/
>>>>>>
>>>>>> Andrew
>>>>>
>>