Host Serial Console Logs via Redfish

* Host Serial Console Logs via Redfish
@ 2021-03-08 21:45 Nan Zhou
  2021-03-09 14:29 ` Brad Bishop
                   ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: Nan Zhou @ 2021-03-08 21:45 UTC (permalink / raw)
  To: openbmc
  Cc: Spencer Ku, Litzung Chen, Ofer Yehielli, Ed Tanous,
	Richard Hanley, Justin Chen, Zhenfei Tai

[-- Attachment #1: Type: text/plain, Size: 3020 bytes --]

Hi All,

We are designing and implementing a new logging service in Redfish to
expose host serial console logs. The goal is that clients can talk to bmc
via Redfish and get a real-time console (just like a read-only serial
console). It will improve the debuggability of BMCs.

We divide the work into two phases. Phase 1 is to use the pull model. That
is, clients do periodical pull against the Redfish server. In Phase 2, we
will consider the post model via Redfish Events and subscriptions.

Implementation for Phase 1 is in
https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/39093. It is
based on obmc-console, phosphor-hostlogger, and bmcweb. The basic idea is
that phosphor-hostlogger collects host serial console logs via obmc-console
and generates tarballs in a rotation manner. These tarballs are then
consumed and exposed by a new node in bmcweb log service.

We found there are some improvements as listed below,

   - Logs are not exposed to Redfish until they reach BUF_MAXSIZE or
   BUF_MAXTIME (defined in https://github.com/openbmc/phosphor-hostlogger),
   but we want to achieve a stream-like console. We could set BUF_MAXSIZE to 1
   or BUF_MAXTIME to a very short interval, but it will amplify the overhead
   of compression and decompression.
   - Persistence isn’t optional. phosphor-hostlogger doesn’t expose any IPC
   interface. bmcweb can only talk to phosphor-hostlogger via zip files, which
   makes persistence of logs a necessary condition.

We propose the following methods to improve it.

   - Method 1: *D-Bus Signal*; phosphor-hostlogger implements an interface
   which contains a signal. The payload of the signal should contain
   timestamps and log messages.  BmcWeb registers as a listener and once it
   receives a signal, it populates a new LogEntry. BmcWeb should implement its
   own configurable ring buffer to store log entries received from D-Bus.
   - Method 2: *File Watcher*; add file watchers in BmcWeb to monitor the
   log files produced by phosphor-hostlogger. This method is similar to method
   1. But persistence is still a necessary condition.
   - Method 3: *obmc-console + bmcweb*: install the console collection and
   ring buffer parts of phosphor-hostlogger as a library. Use the library
   directly in BmcWeb to collect console logs.
   - Method 4: *phosphor-hostlogger + journal + rsyslog + bmcweb*: this
   architecture is very similar to what the current OpenBMC uses for
   redfish-event
   <https://github.com/openbmc/docs/blob/master/architecture/redfish-logging-in-bmcweb.md>.
   Add a new schema for log entries. Publish journal logs in
   phosphor-hostlogger. Add file watchers in BmcWeb to monitor the log files
   produced by rsyslog. rsyslog should have log rotation enabled. Persistence
   is still a necessary condition.

Before we move forward, we would like to see what your preference is. We
are willing to see other suggestions and alternatives as well. Thanks!

Sincerely,
Nan

[-- Attachment #2: Type: text/html, Size: 3637 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread