>
> > We plan to implement something similar to rotate count
> > <https://linux.die.net/man/8/logrotate> in linux logrotate. It is
> basically
> > like a ring buffer in the file system. We keep N log files. The latest
> log
> > file is in plain text and the writer keeps appending data to it. The rest
> > N-1 files are compressed.
> In this case, you will keep full logs without gaps:
> ```
> Host start <- log is empty, start logging
> |
> [...] <- write file, compress and rotate file
> |
> Host reboot or shut down
> ```
> If there are too many logs, logrotate removes the oldest one and we lose
> the
> boot log (form host start).
> This is the default Hostlogger mode:
> ```
> Host start <- log is empty, start logging
> |
> [line 3000] <- flush 3000 lines to the persistent file
> |
> [...] <- these logs are skipped (the last 3000 lines are in memory)
> |
> Host reboot or shut down <- flush last 3000 lines to the file


Thanks for your explanation, but I didn't get it. Are you arguing that one
can keep more logs in memory rather than on disk? If there are too many
logs in a boot cycle, won't the current hostlogger lose some earlier logs
(boot logs) as well? Or did me missing something?

Also, we already talked about it: there's a problem that if BMC loses the
power before it sends out a signal to hostlogger, data in memory won't be
persisted.

On Wed, May 26, 2021 at 9:08 AM Artem Senichev <artemsen@gmail.com> wrote:

> On Wed, May 26, 2021 at 08:17:11AM -0700, Nan Zhou wrote:
> > >
> > > > > > 3. zlib_file.xpp, zlib_exception.xpp:
> > > > > > will be removed or slightly changed; we can potentially use the
> linux
> > > > > > logrotate which has built-in compression and file rotation (in
> this
> > > case
> > > > > > these compression utilities will be removed).
> > > > > > The latest log file isn't compressed any more. History log files
> are
> > > > > > still compressed.
> > > > > Just curious, how are you going to remove the oldest messages from
> the
> > > > > latest file in runtime? You are not going to rewrite the entire
> file on
> > > > > every input character, are you?
> > > >
> > > > The following is my current idea: we will rename the latest file to
> > > > something else and notify the writer (hostlogger) to close its old
> file
> > > > descriptor and open a new one (should be doable via linux logrotate
> and
> > > > inotify or some signal handlers, as logrotate is able to send some
> > > signals
> > > > to hostlogger if a rotation is performed). The writer keeps appending
> > > logs
> > > > most of the time using the same fd unless the latest file is rotated.
> > > This
> > > > should be better than truncating the file where the reader (BMCWeb)
> won't
> > > > have race conditions (it might read old snapshots but it is not a big
> > > deal
> > > > in our case).
> > > Currently we can keep the last N lines of the host's output, the oldest
> > > messages are removed. It is easy to implement with a buffer in memory.
> > > But how are you going to get rid of the old lines if you write data
> > > directly
> > > to the log file?
> > > Rotation will not help you with that (we actually don't need to store
> such
> > > old
> > > logs).
> >
> > We plan to implement something similar to rotate count
> > <https://linux.die.net/man/8/logrotate> in linux logrotate. It is
> basically
> > like a ring buffer in the file system. We keep N log files. The latest
> log
> > file is in plain text and the writer keeps appending data to it. The rest
> > N-1 files are compressed.
>
> In this case, you will keep full logs without gaps:
> ```
> Host start <- log is empty, start logging
> |
> [...] <- write file, compress and rotate file
> |
> Host reboot or shut down
> ```
>
> If there are too many logs, logrotate removes the oldest one and we lose
> the
> boot log (form host start).
>
> This is the default Hostlogger mode:
> ```
> Host start <- log is empty, start logging
> |
> [line 3000] <- flush 3000 lines to the persistent file
> |
> [...] <- these logs are skipped (the last 3000 lines are in memory)
> |
> Host reboot or shut down <- flush last 3000 lines to the file
> ```
>
> --
> Regards,
> Artem Senichev
> Software Engineer, YADRO.
>