(no subject)

[Bills, Jason M]

I just added a comment to a github discussion about the IPMI SEL and thought I should share it here as well:

I have been working on a proof-of-concept to move the IPMI SEL entries out of D-Bus into journald instead.

Since journald allows custom metadata for log entries, I've thought of having the SEL message logged to the journal and using metadata to store the necessary IPMI info associated with the entry. Here is an example of logging a type 0x02 system event entry to journald:

sd_journal_send("MESSAGE=%s", message.c_str(),
                            "PRIORITY=%i", selPriority,
                            "MESSAGE_ID=%s", selMessageId,
                            "IPMI_SEL_RECORD_ID=%d", recordId,
                            "IPMI_SEL_RECORD_TYPE=%x", selSystemType,
                            "IPMI_SEL_GENERATOR_ID=%x", genId,
                            "IPMI_SEL_SENSOR_PATH=%s", path.c_str(),
                            "IPMI_SEL_EVENT_DIR=%x", assert,
                            "IPMI_SEL_DATA=%s", selDataStr,
                            NULL);
Using journald should allow for scaling to more SEL entries which should also enable us to support more generic IPMI behavior such as the Add SEL command.

-Jason

Re:

[Brad Bishop]

> On Aug 28, 2018, at 1:34 PM, Bills, Jason M <jason.m.bills@intel.com> wrote:
> 
> I just added a comment to a github discussion about the IPMI SEL and thought I should share it here as well:

Can you send a link to the issue?

> 
> I have been working on a proof-of-concept to move the IPMI SEL entries out of D-Bus into journald instead.
> 
> Since journald allows custom metadata for log entries, I've thought of having the SEL message logged to the journal and using metadata to store the necessary IPMI info associated with the entry. Here is an example of logging a type 0x02 system event entry to journald:
> 
> sd_journal_send("MESSAGE=%s", message.c_str(),
>                            "PRIORITY=%i", selPriority,
>                            "MESSAGE_ID=%s", selMessageId,
>                            "IPMI_SEL_RECORD_ID=%d", recordId,
>                            "IPMI_SEL_RECORD_TYPE=%x", selSystemType,
>                            "IPMI_SEL_GENERATOR_ID=%x", genId,
>                            "IPMI_SEL_SENSOR_PATH=%s", path.c_str(),
>                            "IPMI_SEL_EVENT_DIR=%x", assert,
>                            "IPMI_SEL_DATA=%s", selDataStr,
>                            NULL);
> Using journald should allow for scaling to more SEL entries which should also enable us to support more generic IPMI behavior such as the Add SEL command.

A design point of OpenBMC from day one was to not design it around IPMI.
At a glance this feels counter to that goal.

I’m not immediately opposed to moving our error logs out of DBus, but can you provide
an extendible abstraction?  Not everyone uses SEL, or IPMI even.  At a minimum please
drop the letters ‘ipmi’ and ‘sel’ :-) from the base design, and save those for something
that translates to IPMI-speak.

As some background, our systems tend towards fewer ‘error logs’ with much more data per
log (4-16k), and yes I admit the current design is biased towards that and does not scale
when we approach 1000s of small SEL entries.

thx - brad

> 
> -Jason

RE:

[Bills, Jason M]

Here is a link to the issue: https://github.com/openbmc/openbmc/issues/3283#issuecomment-414361325.

The main things that started this proof-of-concept are that we have requirements to be fully IPMI compliant and to support 4000+ SEL entries.  Our attempts to scale the D-Bus logs to that level were not successful, so we started considering directly accessing journald as an alternative.

So far, I've been focused only on IPMI SEL, so I hadn't considered extending the change to non-IPMI error logs; however, these IPMI SEL entries should still fit in well as a subset of all other error logs which could also be moved to the journal.

My goal is to align with the OpenBMC design and keep anything IPMI-related isolated only to things that care about IPMI.  My thinking was that the metadata is a bit like background info, so it is a good place to hide data that only matters to the minority, such as the IPMI-specific data.  With this, the IPMI SEL logs can be included among all the existing error logs but still have the metadata for additional IPMI stuff that doesn't matter for anyone else.

So, for writing logs:
A. non-IPMI error logs can be written as normal
B. IPMI SEL entries are written with the IPMI-specific metadata populated

For reading logs:
A. non-IPMI readers see IPMI SEL entries as normal text logs
B. IPMI readers dump just the IPMI SEL entries and get the associated IPMI-specific info from the metadata

Thanks,
-Jason

-----Original Message-----
From: Brad Bishop <bradleyb@fuzziesquirrel.com> 
Sent: Tuesday, August 28, 2018 10:59 AM
To: Bills, Jason M <jason.m.bills@intel.com>
Cc: openbmc@lists.ozlabs.org
Subject: Re: 



> On Aug 28, 2018, at 1:34 PM, Bills, Jason M <jason.m.bills@intel.com> wrote:
> 
> I just added a comment to a github discussion about the IPMI SEL and thought I should share it here as well:

Can you send a link to the issue?

> 
> I have been working on a proof-of-concept to move the IPMI SEL entries out of D-Bus into journald instead.
> 
> Since journald allows custom metadata for log entries, I've thought of having the SEL message logged to the journal and using metadata to store the necessary IPMI info associated with the entry. Here is an example of logging a type 0x02 system event entry to journald:
> 
> sd_journal_send("MESSAGE=%s", message.c_str(),
>                            "PRIORITY=%i", selPriority,
>                            "MESSAGE_ID=%s", selMessageId,
>                            "IPMI_SEL_RECORD_ID=%d", recordId,
>                            "IPMI_SEL_RECORD_TYPE=%x", selSystemType,
>                            "IPMI_SEL_GENERATOR_ID=%x", genId,
>                            "IPMI_SEL_SENSOR_PATH=%s", path.c_str(),
>                            "IPMI_SEL_EVENT_DIR=%x", assert,
>                            "IPMI_SEL_DATA=%s", selDataStr,
>                            NULL);
> Using journald should allow for scaling to more SEL entries which should also enable us to support more generic IPMI behavior such as the Add SEL command.

A design point of OpenBMC from day one was to not design it around IPMI.
At a glance this feels counter to that goal.

I’m not immediately opposed to moving our error logs out of DBus, but can you provide an extendible abstraction?  Not everyone uses SEL, or IPMI even.  At a minimum please drop the letters ‘ipmi’ and ‘sel’ :-) from the base design, and save those for something that translates to IPMI-speak.

As some background, our systems tend towards fewer ‘error logs’ with much more data per log (4-16k), and yes I admit the current design is biased towards that and does not scale when we approach 1000s of small SEL entries.

thx - brad

> 
> -Jason

Re:

[Brad Bishop]

> On Aug 28, 2018, at 7:26 PM, Bills, Jason M <jason.m.bills@intel.com> wrote:
> 
> Here is a link to the issue: https://github.com/openbmc/openbmc/issues/3283#issuecomment-414361325.
> 
> The main things that started this proof-of-concept are that we have requirements to be fully IPMI compliant and to support 4000+ SEL entries.  Our attempts to scale the D-Bus logs to that level were not successful, so we started considering directly accessing journald as an alternative.
> 
> So far, I've been focused only on IPMI SEL, so I hadn't considered extending the change to non-IPMI error logs; however, these IPMI SEL entries should still fit in well as a subset of all other error logs which could also be moved to the journal.
> 
> My goal is to align with the OpenBMC design and keep anything IPMI-related isolated only to things that care about IPMI.  

But it seems like you are proposing that every application that wants to make
a log needs to have the logic to translate its internal data model to IPMI speak,
so it can make a journal call with all the IPMI metadata populated.  Am I
understanding correctly?  That doesn’t seem aligned with keeping IPMI isolated.

A concrete example - phosphor-hwmon.  How do you intend to figure out something
like IPMI_SEL_SENSOR_PATH in the phosphor-hwmon application?  Actually it would
help quite a bit to understand how each of the fields in your sample below would
be determined by an arbitrary dbus application (like phosphor-hwmon).

Further, if you expand this approach to further log formats other than SEL,
won’t the applications become a mess of translation logic from the applications
data mode <-> log format in use?

> My thinking was that the metadata is a bit like background info, so it is a good place to hide data that only matters to the minority, such as the IPMI-specific data.  With this, the IPMI SEL logs can be included among all the existing error logs but still have the metadata for additional IPMI stuff that doesn't matter for anyone else.
> 
> So, for writing logs:
> A. non-IPMI error logs can be written as normal
> B. IPMI SEL entries are written with the IPMI-specific metadata populated
> 
> For reading logs:
> A. non-IPMI readers see IPMI SEL entries as normal text logs
> B. IPMI readers dump just the IPMI SEL entries and get the associated IPMI-specific info from the metadata

I’d rather have a single approach that works for everyone; although, I’m
not sure how that would look.

> 
> Thanks,
> -Jason

This is called top posting, please try to avoid when using the mail-list.
It makes threaded conversation hard to follow and respond to.  thx.

> 
> -----Original Message-----
> From: Brad Bishop <bradleyb@fuzziesquirrel.com> 
> Sent: Tuesday, August 28, 2018 10:59 AM
> To: Bills, Jason M <jason.m.bills@intel.com>
> Cc: openbmc@lists.ozlabs.org
> Subject: Re: 
> 
> 
> 
>> On Aug 28, 2018, at 1:34 PM, Bills, Jason M <jason.m.bills@intel.com> wrote:
>> 
>> I just added a comment to a github discussion about the IPMI SEL and thought I should share it here as well:
> 
> Can you send a link to the issue?
> 
>> 
>> I have been working on a proof-of-concept to move the IPMI SEL entries out of D-Bus into journald instead.
>> 
>> Since journald allows custom metadata for log entries, I've thought of having the SEL message logged to the journal and using metadata to store the necessary IPMI info associated with the entry. Here is an example of logging a type 0x02 system event entry to journald:
>> 
>> sd_journal_send("MESSAGE=%s", message.c_str(),
>>                           "PRIORITY=%i", selPriority,
>>                           "MESSAGE_ID=%s", selMessageId,
>>                           "IPMI_SEL_RECORD_ID=%d", recordId,
>>                           "IPMI_SEL_RECORD_TYPE=%x", selSystemType,
>>                           "IPMI_SEL_GENERATOR_ID=%x", genId,
>>                           "IPMI_SEL_SENSOR_PATH=%s", path.c_str(),
>>                           "IPMI_SEL_EVENT_DIR=%x", assert,
>>                           "IPMI_SEL_DATA=%s", selDataStr,
>>                           NULL);
>> Using journald should allow for scaling to more SEL entries which should also enable us to support more generic IPMI behavior such as the Add SEL command.
> 
> A design point of OpenBMC from day one was to not design it around IPMI.
> At a glance this feels counter to that goal.
> 
> I’m not immediately opposed to moving our error logs out of DBus, but can you provide an extendible abstraction?  Not everyone uses SEL, or IPMI even.  At a minimum please drop the letters ‘ipmi’ and ‘sel’ :-) from the base design, and save those for something that translates to IPMI-speak.
> 
> As some background, our systems tend towards fewer ‘error logs’ with much more data per log (4-16k), and yes I admit the current design is biased towards that and does not scale when we approach 1000s of small SEL entries.
> 
> thx - brad
> 
>> 
>> -Jason

Re: Re:

[Ed Tanous]

On 09/04/2018 01:46 PM, Brad Bishop wrote:
> 
> But it seems like you are proposing that every application that wants to make
> a log needs to have the logic to translate its internal data model to IPMI speak,
> so it can make a journal call with all the IPMI metadata populated.  Am I
> understanding correctly?  That doesn’t seem aligned with keeping IPMI isolated.
> 

I think a key here is that not all logs will be implicitly converted to 
IPMI logs.  Having them be identical was the design that we started 
with, and abandoned because IPMI has some requirements that don't 
cleanly map from a standard syslog/text style to IPMI.


> A concrete example - phosphor-hwmon.  How do you intend to figure out something
> like IPMI_SEL_SENSOR_PATH in the phosphor-hwmon application?  Actually it would
> help quite a bit to understand how each of the fields in your sample below would
> be determined by an arbitrary dbus application (like phosphor-hwmon).

I'm not really understanding the root of the question.  If 
phosphor-hwmon is generating a threshold crossing log that stemmed from 
the /xyz/openbmc_project/sensors/my_super_awesome_temp_sensor, then it 
would simply fill that path into the IPMI_SEL_SENSOR_PATH field.  This 
is the same kind of mapping that the associations produce today, but 
captured in journald instead of the mapper.

Our thinking was that we could build either a static library, or a dbus 
daemon to simplify producing IPMI logs.  Because of the IPMI 
requirements around unique record ids, right now we're leaning toward 
the dbus interface with a single daemon responsible for IPMI SEL creation.
While technically it could be a part of phosphor-logging, we really want 
it to be easily removable for future platforms that have no need for 
IPMI, so the thought at this time it to keep it separate.

> 
> Further, if you expand this approach to further log formats other than SEL,
> won’t the applications become a mess of translation logic from the applications
> data mode <-> log format in use?
> 

I'm not really following this question.  Are there other binary log 
formats that we expect to come in the future that aren't text based, and 
could just be a journald translation?  So far as I know, IPMI SEL is the 
only one on my road map that has weird requirements, and needs some 
translation.  I don't expect it to be a mess, and I'm running under the 
assumption that _most_ daemons won't care about or support IPMI given 
its limitations.
You're right, this isn't intended to be a general solution for all 
binary logging formats, it's intended to be a short term hack while the 
industry transitions away from IPMI and toward something easier to 
generate arbitrarily.

> 
> I’d rather have a single approach that works for everyone; although, I’m
> not sure how that would look.
> 
The single approach is where we started, and weren't able to come up 
with anything that even came close to working in a production sense.  If 
you have ideas here on how this could be built that are cleaner than 
what we're proposing, we're very much interested.

> 
> This is called top posting, please try to avoid when using the mail-list.
> It makes threaded conversation hard to follow and respond to.  thx.
> 

(Ed beats Jason with very big stick)

Re:

[Brad Bishop]

> On Sep 4, 2018, at 5:28 PM, Ed Tanous <ed.tanous@intel.com> wrote:
> 
> On 09/04/2018 01:46 PM, Brad Bishop wrote:
>> But it seems like you are proposing that every application that wants to make
>> a log needs to have the logic to translate its internal data model to IPMI speak,
>> so it can make a journal call with all the IPMI metadata populated.  Am I
>> understanding correctly?  That doesn’t seem aligned with keeping IPMI isolated.
> 
> I think a key here is that not all logs will be implicitly converted to IPMI logs.  Having them be identical was the design that we started with, and abandoned because IPMI has some requirements that don't cleanly map from a standard syslog/text style to IPMI.
> 
> 
>> A concrete example - phosphor-hwmon.  How do you intend to figure out something
>> like IPMI_SEL_SENSOR_PATH in the phosphor-hwmon application?  Actually it would
>> help quite a bit to understand how each of the fields in your sample below would
>> be determined by an arbitrary dbus application (like phosphor-hwmon).
> 
> I'm not really understanding the root of the question.  If phosphor-hwmon is generating a threshold crossing log that stemmed from the /xyz/openbmc_project/sensors/my_super_awesome_temp_sensor, then it would simply fill that path into the IPMI_SEL_SENSOR_PATH field.  

ok, then this is my ignorance of IPMI showing.  I thought IPMI_SEL_SENSOR_PATH was
an IPMI construct...

If this is the case then why not just call it SENSOR_PATH?  Then other logging formats
could use that metadata key without it being weird that it has ‘ipmi_sel’ in the name
of it.  And can we apply the same logic to the other keys or do some of the other keys
have more complicated translation logic (than none at all as in the case of the sensor
path) ?

> This is the same kind of mapping that the associations produce today, but captured in journald instead of the mapper.
> 
> Our thinking was that we could build either a static library, or a dbus daemon to simplify producing IPMI logs.  Because of the IPMI requirements around unique record ids, right now we're leaning toward the dbus interface with a single daemon responsible for IPMI SEL creation.

Thats great!  This is similar to how the phosphor-logging daemon creates dbus error
objects today.

Would you mind elaborating on this daemon and its dbus API?  I’m guessing it would probably
clear up any concerns I have.

> While technically it could be a part of phosphor-logging,

That isn’t what I was going for.  If you plan to implement a (separate) daemon that acts on
the journald metadata I think that is right approach too.

> we really want it to be easily removable for future platforms that have no need for IPMI, so the thought at this time it to keep it separate.

Agreed.

> 
>> Further, if you expand this approach to further log formats other than SEL,
>> won’t the applications become a mess of translation logic from the applications
>> data mode <-> log format in use?
> 
> I'm not really following this question.  Are there other binary log formats that we expect to come in the future that aren't text based, and could just be a journald translation?

Yes.  We have a binary format called PEL.  I doubt anyone would be interested in using
it but we need a foundation in OpenBMC that enables us to use it...

>  So far as I know, IPMI SEL is the only one on my road map that has weird requirements, and needs some translation.

Where is the translation happening?  In the new ipmi-sel daemon?  Or somewhere else?

>  I don't expect it to be a mess, and I'm running under the assumption that _most_ daemons won't care about or support IPMI given its limitations.

Well _all_ daemons already support IPMI SEL today.  The problem is just that the
implementation doesn’t scale.  I’m confused by _most_ daemons wouldn’t support
IPMI?

> You're right, this isn't intended to be a general solution for all binary logging formats, it's intended to be a short term hack while the industry transitions away from IPMI and toward something easier to generate arbitrarily.
> 
>> I’d rather have a single approach that works for everyone; although, I’m
>> not sure how that would look.
> The single approach is where we started, and weren't able to come up with anything that even came close to working in a production sense.  If you have ideas here on how this could be built that are cleaner than what we're proposing, we're very much interested.

I’m still trying to understand what is being proposed.

> 
>> This is called top posting, please try to avoid when using the mail-list.
>> It makes threaded conversation hard to follow and respond to.  thx.
> 
> (Ed beats Jason with very big stick)

Re: Re:

[Ed Tanous]

On 09/04/2018 03:34 PM, Brad Bishop wrote:
> 
> ok, then this is my ignorance of IPMI showing.  I thought IPMI_SEL_SENSOR_PATH was
> an IPMI construct...
> 
> If this is the case then why not just call it SENSOR_PATH?  Then other logging formats
> could use that metadata key without it being weird that it has ‘ipmi_sel’ in the name
> of it.  And can we apply the same logic to the other keys or do some of the other keys
> have more complicated translation logic (than none at all as in the case of the sensor
> path) ?

The thinking was that we would namespace all the parameters using 
IPMI_SEL to make it clear that was the only place they were used, and to 
avoid someone else using it inadvertently.  With that said, I could 
understand how it could be confusing.  Jason, any objections to 
un-namespacing the parameters?

> 
> Thats great!  This is similar to how the phosphor-logging daemon creates dbus error
> objects today.
> 
> Would you mind elaborating on this daemon and its dbus API?  I’m guessing it would probably
> clear up any concerns I have.
> 

Patches to phosphor-dbus-interfaces for a suggested interface are being 
put together as we speak.  Hopefully that will clarify it a little bit.

>> While technically it could be a part of phosphor-logging,
> 
> That isn’t what I was going for.  If you plan to implement a (separate) daemon that acts on
> the journald metadata I think that is right approach too.
> 
Agreed.

>> we really want it to be easily removable for future platforms that have no need for IPMI, so the thought at this time it to keep it separate.
> 
> Agreed.
> 
>>
>>> Further, if you expand this approach to further log formats other than SEL,
>>> won’t the applications become a mess of translation logic from the applications
>>> data mode <-> log format in use?
>>
>> I'm not really following this question.  Are there other binary log formats that we expect to come in the future that aren't text based, and could just be a journald translation?
> 
> Yes.  We have a binary format called PEL.  I doubt anyone would be interested in using
> it but we need a foundation in OpenBMC that enables us to use it...
> 

That makes more sense now.  A quick google on PEL makes it look like it 
could follow a similar model to what we're doing with IPMI by adding the 
extra metadata to journald when needed, while still producing the string 
versions for the basic cases.  By foundation do you mean shared code?  A 
quick skim of the implementation makes me suspect that there isn't going 
to be a lot of shared code, although they could share a similar design 
with a different implementation.

>>   So far as I know, IPMI SEL is the only one on my road map that has weird requirements, and needs some translation.
> 
> Where is the translation happening?  In the new ipmi-sel daemon?  Or somewhere else?

The translation would happen on the "addSel" IPMI command that gets used 
in-band by most BIOS implementations.  The ipmi-sel daemon will 
translate the raw bytes to a string, to be used in most modern loggers, 
along with the IPMI metadata, to be used in IPMI to source the various 
"get" SEL entry commands.

> 
>>   I don't expect it to be a mess, and I'm running under the assumption that _most_ daemons won't care about or support IPMI given its limitations.
> 
> Well _all_ daemons already support IPMI SEL today.  The problem is just that the
> implementation doesn’t scale.  I’m confused by _most_ daemons wouldn’t support
> IPMI?
> 

That should've clarified that most daemons won't care about IPMI _SEL_, 
given the extra calls and metadata that needs to be provided to 
implement it correctly.  My teams intention was to support the minimum 
subset of SEL that we can for backward compatibility, while providing 
the advanced logging (journald/syslog/redfish LogService) for a greater 
level of detail and capability.
If this assumption turns out to not be true, and we end up adding IPMI 
SEL logging to all the daemons, so be it, I think it will still scale, 
but I really hope that's not the path we go down.


>> You're right, this isn't intended to be a general solution for all binary logging formats, it's intended to be a short term hack while the industry transitions away from IPMI and toward something easier to generate arbitrarily.
>>
>>> I’d rather have a single approach that works for everyone; although, I’m
>>> not sure how that would look.
>> The single approach is where we started, and weren't able to come up with anything that even came close to working in a production sense.  If you have ideas here on how this could be built that are cleaner than what we're proposing, we're very much interested.
> 
> I’m still trying to understand what is being proposed.
> 
>>
>>> This is called top posting, please try to avoid when using the mail-list.
>>> It makes threaded conversation hard to follow and respond to.  thx.
>>
>> (Ed beats Jason with very big stick)

Re:

[Brad Bishop]

> On Sep 4, 2018, at 7:18 PM, Ed Tanous <ed.tanous@intel.com> wrote:
> 
> On 09/04/2018 03:34 PM, Brad Bishop wrote:
>> ok, then this is my ignorance of IPMI showing.  I thought IPMI_SEL_SENSOR_PATH was
>> an IPMI construct...
>> If this is the case then why not just call it SENSOR_PATH?  Then other logging formats
>> could use that metadata key without it being weird that it has ‘ipmi_sel’ in the name
>> of it.  And can we apply the same logic to the other keys or do some of the other keys
>> have more complicated translation logic (than none at all as in the case of the sensor
>> path) ?
> 
> The thinking was that we would namespace all the parameters using IPMI_SEL to make it clear that was the only place they were used, and to avoid someone else using it inadvertently.  
> With that said, I could understand how it could be confusing.  Jason, any objections to un-namespacing the parameters?

Thanks for being flexible on this but lets wait until we are on the same page before
changing anything.  Why would you want to discourage it from being used in another
context?

> 
>> Thats great!  This is similar to how the phosphor-logging daemon creates dbus error
>> objects today.
>> Would you mind elaborating on this daemon and its dbus API?  I’m guessing it would probably
>> clear up any concerns I have.
> 
> Patches to phosphor-dbus-interfaces for a suggested interface are being put together as we speak.  Hopefully that will clarify it a little bit.

Great, thank you.

> 
>>> While technically it could be a part of phosphor-logging,
>> That isn’t what I was going for.  If you plan to implement a (separate) daemon that acts on
>> the journald metadata I think that is right approach too.
> Agreed.
> 
>>> we really want it to be easily removable for future platforms that have no need for IPMI, so the thought at this time it to keep it separate.
>> Agreed.
>>> 
>>>> Further, if you expand this approach to further log formats other than SEL,
>>>> won’t the applications become a mess of translation logic from the applications
>>>> data mode <-> log format in use?
>>> 
>>> I'm not really following this question.  Are there other binary log formats that we expect to come in the future that aren't text based, and could just be a journald translation?
>> Yes.  We have a binary format called PEL.  I doubt anyone would be interested in using
>> it but we need a foundation in OpenBMC that enables us to use it...
> 
> That makes more sense now.  A quick google on PEL makes it look like it could follow a similar model to what we're doing with IPMI by adding the extra metadata to journald when needed, while still producing the string versions for the basic cases.  By foundation do you mean shared code?  A quick skim of the implementation makes me suspect that there isn't going to be a lot of shared code, although they could share a similar design with a different implementation.

By foundation I simply mean we need a way to support multiple logging formats that doesn’t
require every OpenBMC application to know how to translate from its internal data model
(usually dbus) to N logging formats.

> 
>>>  So far as I know, IPMI SEL is the only one on my road map that has weird requirements, and needs some translation.
>> Where is the translation happening?  In the new ipmi-sel daemon?  Or somewhere else?
> 
> The translation would happen on the "addSel" IPMI command that gets used in-band by most BIOS implementations.  The ipmi-sel daemon will translate the raw bytes to a string, to be used in most modern loggers, along with the IPMI metadata, to be used in IPMI to source the various "get" SEL entry commands.

That all sounds fine.  But what about applications on the BMC creating SELs for their
own errors?  Do you want to do that?  How will that work?

> 
>>>  I don't expect it to be a mess, and I'm running under the assumption that _most_ daemons won't care about or support IPMI given its limitations.
>> Well _all_ daemons already support IPMI SEL today.  The problem is just that the
>> implementation doesn’t scale.  I’m confused by _most_ daemons wouldn’t support
>> IPMI?
> 
> That should've clarified that most daemons won't care about IPMI _SEL_, given the extra calls and metadata that needs to be provided to implement it correctly.  My teams intention was to support the minimum subset of SEL that we can for backward compatibility, while providing the advanced logging (journald/syslog/redfish LogService) for a greater level of detail and capability.
> If this assumption turns out to not be true, and we end up adding IPMI SEL logging to all the daemons, so be it, I think it will still scale, but I really hope that's not the path we go down.

Oh.  Does this mean you intend for code like Jason originally proposed to _only_ appear in
the ipmi-sel daemon?  And not in applications like phosphor-hwmon?

Re:

[Bills, Jason M]

>>> ok, then this is my ignorance of IPMI showing.  I thought IPMI_SEL_SENSOR_PATH was
>>> an IPMI construct...
>>> If this is the case then why not just call it SENSOR_PATH?  Then other logging formats
>>> could use that metadata key without it being weird that it has ‘ipmi_sel’ in the name
>>> of it.  And can we apply the same logic to the other keys or do some of the other keys
>>> have more complicated translation logic (than none at all as in the case of the sensor
>>> path) ?
>>
>> The thinking was that we would namespace all the parameters using IPMI_SEL to make it clear that was the only place they were used, and to avoid someone else using it inadvertently.
>> With that said, I could understand how it could be confusing.  Jason, any objections to un-namespacing the parameters?
> 
> Thanks for being flexible on this but lets wait until we are on the same page before
> changing anything.  Why would you want to discourage it from being used in another
> context?
> 
Except for the sensor path, all of the proposed IPMI metadata is 
specific for IPMI:
"IPMI_SEL_RECORD_ID" = Two byte unique ID number for each SEL entry
"IPMI_SEL_RECORD_TYPE" = The type of SEL entry (system or OEM) which 
determines the definition of the remaining bytes
"IPMI_SEL_GENERATOR_ID" = The IPMI Generator ID (usually the IPMB Slave 
Address) of the SEL entry requester
"IPMI_SEL_SENSOR_PATH" = Path of the sensor used to find IPMI data (such 
as sensor number) for the sensor
"IPMI_SEL_EVENT_DIR" = Whether the sensor is asserting or de-asserting
"IPMI_SEL_DATA" = Raw binary data included in the SEL entry

I named them all as IPMI_SEL as a group so they would be clearly 
separate and easy to remove in the future.  However, if any of the 
metadata would be useful elsewhere, the names can be more generic.

>>
>>> Thats great!  This is similar to how the phosphor-logging daemon creates dbus error
>>> objects today.
>>> Would you mind elaborating on this daemon and its dbus API?  I’m guessing it would probably
>>> clear up any concerns I have.
>>
>> Patches to phosphor-dbus-interfaces for a suggested interface are being put together as we speak.  Hopefully that will clarify it a little bit.
> 
> Great, thank you.
> 
I have pushed a suggestion for the interface here: 
https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-dbus-interfaces/+/12494/

>>
>>>> While technically it could be a part of phosphor-logging,
>>> That isn’t what I was going for.  If you plan to implement a (separate) daemon that acts on
>>> the journald metadata I think that is right approach too.
>> Agreed.
>>
>>>> we really want it to be easily removable for future platforms that have no need for IPMI, so the thought at this time it to keep it separate.
>>> Agreed.
>>>>
>>>>> Further, if you expand this approach to further log formats other than SEL,
>>>>> won’t the applications become a mess of translation logic from the applications
>>>>> data mode <-> log format in use?
>>>>
>>>> I'm not really following this question.  Are there other binary log formats that we expect to come in the future that aren't text based, and could just be a journald translation?
>>> Yes.  We have a binary format called PEL.  I doubt anyone would be interested in using
>>> it but we need a foundation in OpenBMC that enables us to use it...
>>
>> That makes more sense now.  A quick google on PEL makes it look like it could follow a similar model to what we're doing with IPMI by adding the extra metadata to journald when needed, while still producing the string versions for the basic cases.  By foundation do you mean shared code?  A quick skim of the implementation makes me suspect that there isn't going to be a lot of shared code, although they could share a similar design with a different implementation.
> 
> By foundation I simply mean we need a way to support multiple logging formats that doesn’t
> require every OpenBMC application to know how to translate from its internal data model
> (usually dbus) to N logging formats.
> 
>>
>>>>   So far as I know, IPMI SEL is the only one on my road map that has weird requirements, and needs some translation.
>>> Where is the translation happening?  In the new ipmi-sel daemon?  Or somewhere else?
>>
>> The translation would happen on the "addSel" IPMI command that gets used in-band by most BIOS implementations.  The ipmi-sel daemon will translate the raw bytes to a string, to be used in most modern loggers, along with the IPMI metadata, to be used in IPMI to source the various "get" SEL entry commands.
> 
> That all sounds fine.  But what about applications on the BMC creating SELs for their
> own errors?  Do you want to do that?  How will that work?
> 

An application on the BMC that needs to create a SEL can call the 
IpmiSelAdd method to request a new SEL entry in the journal.
>>
>>>>   I don't expect it to be a mess, and I'm running under the assumption that _most_ daemons won't care about or support IPMI given its limitations.
>>> Well _all_ daemons already support IPMI SEL today.  The problem is just that the
>>> implementation doesn’t scale.  I’m confused by _most_ daemons wouldn’t support
>>> IPMI?
>>
>> That should've clarified that most daemons won't care about IPMI _SEL_, given the extra calls and metadata that needs to be provided to implement it correctly.  My teams intention was to support the minimum subset of SEL that we can for backward compatibility, while providing the advanced logging (journald/syslog/redfish LogService) for a greater level of detail and capability.
>> If this assumption turns out to not be true, and we end up adding IPMI SEL logging to all the daemons, so be it, I think it will still scale, but I really hope that's not the path we go down.
> 
> Oh.  Does this mean you intend for code like Jason originally proposed to _only_ appear in
> the ipmi-sel daemon?  And not in applications like phosphor-hwmon?
> 

Yes, the original proposed code:
             sd_journal_send("MESSAGE=%s", message.c_str(),
                             "PRIORITY=%i", selPriority,
                             "MESSAGE_ID=%s", selMessageId,
                             "IPMI_SEL_RECORD_ID=%d", recordId,
                             "IPMI_SEL_RECORD_TYPE=%x", selSystemType,
                             "IPMI_SEL_GENERATOR_ID=%x", genId,
                             "IPMI_SEL_SENSOR_PATH=%s", path.c_str(),
                             "IPMI_SEL_EVENT_DIR=%x", assert,
                             "IPMI_SEL_DATA=%s", selDataStr,
                             NULL);
will only be in the ipmi-sel daemon.  Applications like phosphor-hwmon 
would use the IpmiSelAdd method to request SEL entries.