Re: Link phosphor-hostlogger and bmcweb

[Nan Zhou <nanzhou@google.com>]

[-- Attachment #1: Type: text/plain, Size: 3601 bytes --]

>
> > > > 3. zlib_file.xpp, zlib_exception.xpp:
> > > > will be removed or slightly changed; we can potentially use the linux
> > > > logrotate which has built-in compression and file rotation (in this
> case
> > > > these compression utilities will be removed).
> > > > The latest log file isn't compressed any more. History log files are
> > > > still compressed.
> > > Just curious, how are you going to remove the oldest messages from the
> > > latest file in runtime? You are not going to rewrite the entire file on
> > > every input character, are you?
> >
> > The following is my current idea: we will rename the latest file to
> > something else and notify the writer (hostlogger) to close its old file
> > descriptor and open a new one (should be doable via linux logrotate and
> > inotify or some signal handlers, as logrotate is able to send some
> signals
> > to hostlogger if a rotation is performed). The writer keeps appending
> logs
> > most of the time using the same fd unless the latest file is rotated.
> This
> > should be better than truncating the file where the reader (BMCWeb) won't
> > have race conditions (it might read old snapshots but it is not a big
> deal
> > in our case).
> Currently we can keep the last N lines of the host's output, the oldest
> messages are removed. It is easy to implement with a buffer in memory.
> But how are you going to get rid of the old lines if you write data
> directly
> to the log file?
> Rotation will not help you with that (we actually don't need to store such
> old
> logs).

We plan to implement something similar to rotate count
<https://linux.die.net/man/8/logrotate> in linux logrotate. It is basically
like a ring buffer in the file system. We keep N log files. The latest log
file is in plain text and the writer keeps appending data to it. The rest
N-1 files are compressed.

On Wed, May 26, 2021 at 1:56 AM Artem Senichev <artemsen@gmail.com> wrote:

> On Tue, May 25, 2021 at 11:51:44PM -0700, Nan Zhou wrote:
> > > > 3. zlib_file.xpp, zlib_exception.xpp:
> > > > will be removed or slightly changed; we can potentially use the linux
> > > > logrotate which has built-in compression and file rotation (in this
> case
> > > > these compression utilities will be removed).
> > > > The latest log file isn't compressed any more. History log files are
> > > > still compressed.
> > > Just curious, how are you going to remove the oldest messages from the
> > > latest file in runtime? You are not going to rewrite the entire file on
> > > every input character, are you?
> >
> > The following is my current idea: we will rename the latest file to
> > something else and notify the writer (hostlogger) to close its old file
> > descriptor and open a new one (should be doable via linux logrotate and
> > inotify or some signal handlers, as logrotate is able to send some
> signals
> > to hostlogger if a rotation is performed). The writer keeps appending
> logs
> > most of the time using the same fd unless the latest file is rotated.
> This
> > should be better than truncating the file where the reader (BMCWeb) won't
> > have race conditions (it might read old snapshots but it is not a big
> deal
> > in our case).
>
> Currently we can keep the last N lines of the host's output, the oldest
> messages are removed. It is easy to implement with a buffer in memory.
> But how are you going to get rid of the old lines if you write data
> directly
> to the log file?
> Rotation will not help you with that (we actually don't need to store such
> old
> logs).
>
> --
> Regards,
> Artem Senichev
> Software Engineer, YADRO.
>

[-- Attachment #2: Type: text/html, Size: 5500 bytes --]