Host console logs service

Host console logs service

[Artem Senichev]

Hi All,

As you know, OpenBMC contains obmc-console project that intended to
work with host's console. It's very useful that omc-console-server
stores last logs inside its ring buffer, but it would be better to
make the log collector more smarter.

On our VESNIN server, we have an own extension of obmc-console - it is
a standalone service that reads log messages from obmc-console-server,
marks each message with a timestamp and save it to an internal memory
buffer. We have a custom policy that determines how large a buffer can
be (limit by time or message numbers), in which cases the buffer
should be flushed to a file, and how many previous log files can be
saved on a flash drive (some kind of rotation).
By default the buffer has limit of 3000 messages and it flushed to a
file on every significant host state change (host on/off, OS boot
complete). This gives us the opportunity to store not only the latest
messages, but also the logs of the last boot process separately
(hostboot, skiboot, etc).

So, if you find this service useful for the OpenBMC project, we can
upload the source code to the community.

Sources:
https://github.com/YADRO-KNS/phosphor-hostlogger

Small demo:
# ls -lA /var/lib/obmc/hostlogs
-rw-r--r--    1 root     root          6336 Oct 24 09:26
host_20181024_092655.log.gz
-rw-r--r--    1 root     root          9589 Oct 24 11:43
host_20181024_114342.log.gz
-rw-r--r--    1 root     root          6547 Oct 24 12:01
host_20181024_120119.log.gz
# zcat /var/lib/obmc/hostlogs/host_20181024_120119.log.gz
[ 11:55:02 ]: >>> Log collection started at 24.10.2018 11:55:02
[ 11:55:02 ]: --== Welcome to Hostboot v2.1-199-g38a9f30-dirty/hbicore.bin/ ==--
[ 11:55:06 ]:   5.66868|ISTEP  6. 3 - host_init_fsi
[ 11:55:06 ]:   6.27102|ISTEP  6. 4 - host_set_ipl_parms
[ 11:55:06 ]:   6.27234|ISTEP  6. 5 - host_discover_targets
...

Best regards,
Artem Senichev
Software Engineer, YADRO.

Re: Host console logs service

[Sai Dasari]

On 10/24/18, 8:56 AM, "openbmc on behalf of Artem Senichev" <openbmc-bounces+sdasari=fb.com@lists.ozlabs.org on behalf of artemsen@gmail.com> wrote:

    Hi All,
    
    As you know, OpenBMC contains obmc-console project that intended to
    work with host's console. It's very useful that omc-console-server
    stores last logs inside its ring buffer, but it would be better to
    make the log collector more smarter.
    
    On our VESNIN server, we have an own extension of obmc-console - it is
    a standalone service that reads log messages from obmc-console-server,
    marks each message with a timestamp and save it to an internal memory
    buffer. We have a custom policy that determines how large a buffer can
    be (limit by time or message numbers), in which cases the buffer
    should be flushed to a file, and how many previous log files can be
    saved on a flash drive (some kind of rotation).
    By default the buffer has limit of 3000 messages and it flushed to a
    file on every significant host state change (host on/off, OS boot
    complete). This gives us the opportunity to store not only the latest
    messages, but also the logs of the last boot process separately
    (hostboot, skiboot, etc).
    
    So, if you find this service useful for the OpenBMC project, we can
    upload the source code to the community.
Thanks Artem! Based on the description above, it looks like very useful feature. Few questions related to design are: Is this stand-alone service designed as a D-Bus service? How does this change the existing user interface with obmc-console i.e. any new CLI options change or behavior change? Can this be used from REST-API or Redfish? How to configure the custom policy to configure? Etc. It would be great if you could use the Design template @ https://github.com/openbmc/docs/tree/master/designs to explain the design details for helping in understanding and providing feedback.
   
    Sources:
    https://github.com/YADRO-KNS/phosphor-hostlogger
    
    Small demo:
    # ls -lA /var/lib/obmc/hostlogs
    -rw-r--r--    1 root     root          6336 Oct 24 09:26
    host_20181024_092655.log.gz
    -rw-r--r--    1 root     root          9589 Oct 24 11:43
    host_20181024_114342.log.gz
    -rw-r--r--    1 root     root          6547 Oct 24 12:01
    host_20181024_120119.log.gz
    # zcat /var/lib/obmc/hostlogs/host_20181024_120119.log.gz
    [ 11:55:02 ]: >>> Log collection started at 24.10.2018 11:55:02
    [ 11:55:02 ]: --== Welcome to Hostboot v2.1-199-g38a9f30-dirty/hbicore.bin/ ==--
    [ 11:55:06 ]:   5.66868|ISTEP  6. 3 - host_init_fsi
    [ 11:55:06 ]:   6.27102|ISTEP  6. 4 - host_set_ipl_parms
    [ 11:55:06 ]:   6.27234|ISTEP  6. 5 - host_discover_targets
    ...
    
    Best regards,
    Artem Senichev
    Software Engineer, YADRO.
    

Re: Host console logs service

[Artem Senichev]

I have described the architecture and service capabilities:
https://github.com/YADRO-KNS/phosphor-hostlogger/blob/master/README.md

Please, look through and let me know what you think.

Best regards,
Artem Senichev
Software Engineer, YADRO.

On Wed, Oct 24, 2018 at 8:38 PM Sai Dasari <sdasari@fb.com> wrote:
> On 10/24/18, 8:56 AM, "openbmc on behalf of Artem Senichev" <openbmc-bounces+sdasari=fb.com@lists.ozlabs.org on behalf of artemsen@gmail.com> wrote:
>
>     Hi All,
>
>     As you know, OpenBMC contains obmc-console project that intended to
>     work with host's console. It's very useful that omc-console-server
>     stores last logs inside its ring buffer, but it would be better to
>     make the log collector more smarter.
>
>     On our VESNIN server, we have an own extension of obmc-console - it is
>     a standalone service that reads log messages from obmc-console-server,
>     marks each message with a timestamp and save it to an internal memory
>     buffer. We have a custom policy that determines how large a buffer can
>     be (limit by time or message numbers), in which cases the buffer
>     should be flushed to a file, and how many previous log files can be
>     saved on a flash drive (some kind of rotation).
>     By default the buffer has limit of 3000 messages and it flushed to a
>     file on every significant host state change (host on/off, OS boot
>     complete). This gives us the opportunity to store not only the latest
>     messages, but also the logs of the last boot process separately
>     (hostboot, skiboot, etc).
>
>     So, if you find this service useful for the OpenBMC project, we can
>     upload the source code to the community.
> Thanks Artem! Based on the description above, it looks like very useful feature. Few questions related to design are: Is this stand-alone service designed as a D-Bus service? How does this change the existing user interface with obmc-console i.e. any new CLI options change or behavior change? Can this be used from REST-API or Redfish? How to configure the custom policy to configure? Etc. It would be great if you could use the Design template @ https://github.com/openbmc/docs/tree/master/designs to explain the design details for helping in understanding and providing feedback.

Re: Host console logs service

[Sai Dasari]

On 10/26/18, 7:21 AM, "Artem Senichev" <artemsen@gmail.com> wrote:

    I have described the architecture and service capabilities:
    https://github.com/YADRO-KNS/phosphor-hostlogger/blob/master/README.md
    
    Please, look through and let me know what you think.
The writeup/description looks good to me. It would be great to submit this as gerrit request for wider community review and helps in commenting inline for the document and have design conversation. An example for ongoing review @ https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/14712/
    Best regards,
    Artem Senichev
    Software Engineer, YADRO.
    
 

Re: Host console logs service

[Artem Senichev]

How can I create a gerrit request if there is no git repository for
this project inside OpenBMC?

Best regards,
Artem Senichev

On Fri, Oct 26, 2018 at 7:18 PM Sai Dasari <sdasari@fb.com> wrote:
>
>
>
> On 10/26/18, 7:21 AM, "Artem Senichev" <artemsen@gmail.com> wrote:
>
>     I have described the architecture and service capabilities:
>     https://github.com/YADRO-KNS/phosphor-hostlogger/blob/master/README.md
>
>     Please, look through and let me know what you think.
> The writeup/description looks good to me. It would be great to submit this as gerrit request for wider community review and helps in commenting inline for the document and have design conversation. An example for ongoing review @ https://gerrit.openbmc-project.xyz/#/c/openbmc/docs/+/14712/
>     Best regards,
>     Artem Senichev
>     Software Engineer, YADRO.
>
>
>

Re: Host console logs service

[Lei YU]

On Sat, Oct 27, 2018 at 1:59 AM Artem Senichev <artemsen@gmail.com> wrote:
>
> How can I create a gerrit request if there is no git repository for
> this project inside OpenBMC?

This looks good and could be very useful for end users.
You could ask Brad to create a repo in OpenBMC, or possible post this design
doc into openbmc/docs for preview.

Thanks!

Re: Host console logs service

[Artem Senichev]

Hi Brad,

Could you create phosphor-hostlogger repository inside the OpenBMC project?
Or even clone existing from https://github.com/YADRO-KNS/phosphor-hostlogger.

Best regards,
Artem Senichev

On Mon, Oct 29, 2018 at 5:01 AM Lei YU <mine260309@gmail.com> wrote:
>
> On Sat, Oct 27, 2018 at 1:59 AM Artem Senichev <artemsen@gmail.com> wrote:
> >
> > How can I create a gerrit request if there is no git repository for
> > this project inside OpenBMC?
>
> This looks good and could be very useful for end users.
> You could ask Brad to create a repo in OpenBMC, or possible post this design
> doc into openbmc/docs for preview.
>
> Thanks!

Re: Host console logs service

[Sai Dasari]

On 10/28/18, 7:02 PM, "Lei YU" <mine260309@gmail.com> wrote:

    On Sat, Oct 27, 2018 at 1:59 AM Artem Senichev <artemsen@gmail.com> wrote:
    >
    > How can I create a gerrit request if there is no git repository for
    > this project inside OpenBMC?
    
    This looks good and could be very useful for end users.
    You could ask Brad to create a repo in OpenBMC, or possible post this design
    doc into openbmc/docs for preview.
Creating a new repo for design review seems to be overhead. Would propose to submit the new design docs for review @ https://github.com/openbmc/docs/tree/master/designs. This will allow community to review the new designs and propose either implementing as extensions to existing repo (e.g. enhancements to existing feature set) or create a new repo (e.g. design is something new or alternative). Suggestions?
    
    Thanks!
    

Re: Host console logs service

[Brad Bishop]

> On Oct 29, 2018, at 3:22 AM, Artem Senichev <artemsen@gmail.com> wrote:
> 
> Hi Brad,
> 
> Could you create phosphor-hostlogger repository inside the OpenBMC project?
> Or even clone existing from https://github.com/YADRO-KNS/phosphor-hostlogger.

Sure thing.  As usual, I’ll wait a couple days for anyone to raise any objections
to this.

> 
> Best regards,
> Artem Senichev
> 
> On Mon, Oct 29, 2018 at 5:01 AM Lei YU <mine260309@gmail.com> wrote:
>> 
>> On Sat, Oct 27, 2018 at 1:59 AM Artem Senichev <artemsen@gmail.com> wrote:
>>> 
>>> How can I create a gerrit request if there is no git repository for
>>> this project inside OpenBMC?
>> 
>> This looks good and could be very useful for end users.
>> You could ask Brad to create a repo in OpenBMC, or possible post this design
>> doc into openbmc/docs for preview.
>> 
>> Thanks!

Re: Host console logs service

[Brad Bishop]

> On Oct 29, 2018, at 3:22 AM, Artem Senichev <artemsen@gmail.com> wrote:
> 
> Hi Brad,
> 
> Could you create phosphor-hostlogger repository inside the OpenBMC project?
> Or even clone existing from https://github.com/YADRO-KNS/phosphor-hostlogger.


https://github.com/openbmc/phosphor-hostlogger

and

https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-hostlogger/+/15176/

Thanks Artem!