Streaming logs via rsyslog

Streaming logs via rsyslog

[Deepak Kodihalli]

Hi,

We have a requirement to forward BMC logs (the systemd journal) to an 
external logging server.

We're proposing doing this by making use of rsyslog. Here are notes 
based on some initial investigation.

Forwarding the journal logs to a syslog daemon :
Journald has this feature of being able to write journal logs to a 
socket which then can be read by a syslog daemon. I see that (on 
Witherspoon at least) the syslog daemon running by default (busybox 
syslogd) doesn't support reading logs forwarded by the journald to due 
to this commit : 
https://git.busybox.net/busybox/commit/?id=accd9eeb719916da974584b33b1aeced5f3bb346. 
However, I could get the forwarding working with the rsyslog daemon. I 
found an existing rsyslog recipe in the meta-oe layer.

rsyslog configuration :
rsyslog is configurable in various ways. For example, I can turn off 
local logging altogether via a config file, if I'm interested only in 
using rsyslog for streaming journal logs out. Various optional features 
can be turned off at build time to save flash space.

rsyslog server provision :
The BMC would need to be made aware of the external rsyslog server. This 
can just be an admin updating the rsyslog config file on the BMC, and 
there could also be a REST API if need be. I think a small app can 
implement the 
https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/Client.interface.yaml 
interface to update the rsyslog config.

streaming :
rsyslog can be configured to make use of TCP, or UDP, and optionally 
TLS. I think TCP is the default.


Thoughts on this proposal? If this sounds okay, I'm planning to get some 
commits up in Gerrit for review. I can pull rsyslog in a phosphor layer 
(I mean as the default choice of a syslog daemon, replacing busybox 
syslog) if there's interest, as opposed to making it only a 
meta-witherspoon package. Specific rsyslog package config and config 
file overrides are possible in system specific layers.

Regards,
Deepak

Re: Streaming logs via rsyslog

[Andrew Geissler]

On Thu, Aug 16, 2018 at 5:21 AM Deepak Kodihalli
<dkodihal@linux.vnet.ibm.com> wrote:
>
> Hi,
>
> We have a requirement to forward BMC logs (the systemd journal) to an
> external logging server.
>
> We're proposing doing this by making use of rsyslog. Here are notes
> based on some initial investigation.
>
> Forwarding the journal logs to a syslog daemon :
> Journald has this feature of being able to write journal logs to a
> socket which then can be read by a syslog daemon. I see that (on
> Witherspoon at least) the syslog daemon running by default (busybox
> syslogd) doesn't support reading logs forwarded by the journald to due
> to this commit :
> https://git.busybox.net/busybox/commit/?id=accd9eeb719916da974584b33b1aeced5f3bb346.
> However, I could get the forwarding working with the rsyslog daemon. I
> found an existing rsyslog recipe in the meta-oe layer.
>
> rsyslog configuration :
> rsyslog is configurable in various ways. For example, I can turn off
> local logging altogether via a config file, if I'm interested only in
> using rsyslog for streaming journal logs out. Various optional features
> can be turned off at build time to save flash space.
>
> rsyslog server provision :
> The BMC would need to be made aware of the external rsyslog server. This
> can just be an admin updating the rsyslog config file on the BMC, and
> there could also be a REST API if need be. I think a small app can
> implement the
> https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/Client.interface.yaml
> interface to update the rsyslog config.

Yeah, I'd tend to vote for a REST api, that would enable GUI support
and cleaner scripting (vs. an scp command).

>
> streaming :
> rsyslog can be configured to make use of TCP, or UDP, and optionally
> TLS. I think TCP is the default.
>
>
> Thoughts on this proposal? If this sounds okay, I'm planning to get some
> commits up in Gerrit for review. I can pull rsyslog in a phosphor layer
> (I mean as the default choice of a syslog daemon, replacing busybox
> syslog) if there's interest, as opposed to making it only a
> meta-witherspoon package. Specific rsyslog package config and config
> file overrides are possible in system specific layers.

Seems reasonable to me.  Do you have a rough feel for what the bare
bones rsyslog package will take up in flash space?  That would be my
only concern at this point, we're getting a bit pressed for flash
space.

>
> Regards,
> Deepak
>

Re: Streaming logs via rsyslog

[Deepak Kodihalli]

On 16/08/18 6:14 PM, Andrew Geissler wrote:

> Seems reasonable to me.  Do you have a rough feel for what the bare
> bones rsyslog package will take up in flash space?  That would be my
> only concern at this point, we're getting a bit pressed for flash
> space.

Roughly 300K. Plus, we might have some savings by removing the busybox 
syslog package (wouldn't need both this and rsyslog).

Regards,
Deepak

Re: Streaming logs via rsyslog

[Ed Tanous]

On 08/16/2018 03:21 AM, Deepak Kodihalli wrote:
> Hi,
> 
> We have a requirement to forward BMC logs (the systemd journal) to an 
> external logging server.
> 
> We're proposing doing this by making use of rsyslog. Here are notes 
> based on some initial investigation.
> 

 From the rsyslog github page

"""
Rsyslog's main sponsor Adiscon tries to fund rsyslog by selling custom 
development and support contracts. Adiscon does NOT license rsyslog 
under a commercial license (this is simply impossible for anyone due to 
rsyslog's license structure).
"""

I think this license makes rsyslog a non-starter for most of the people 
in this project, but I'm not a license expert.  We've looked at it in 
the past, and while nice to use in practice, we couldn't overcome the 
license conflicts and legal implications.

Oddly enough, my team did some prototype work in this area a couple 
weeks ago that I've pushed to gerrit for comment.

As part of an experiment we implemented a prototype Redfish endpoint 
that exposed the journald log entries.  It seems to have worked 
relatively well, and it's considerably faster than we thought it would 
be.  1000 entries can be grabbed in about a second including all the TLS 
overhead.

An example from my system is here:
https://imgur.com/a/vtlFvHB

The thought was that at some point we would implement server-sent 
events, which is part of the Redfish 1.5.0 specification, and would 
allow the logs to be streamed off the system using a pretty well defined 
protocol that's not specific to OpenBMC, as well as implement pretty 
decent security.  It's not directly syslog, and I can still see the use 
case for having a syslog compatible mechanism for streaming.

https://gerrit.openbmc-project.xyz/#/c/openbmc/bmcweb/+/12008/

Re: Streaming logs via rsyslog

[Deepak Kodihalli]

On 16/08/18 11:37 PM, Ed Tanous wrote:
> On 08/16/2018 03:21 AM, Deepak Kodihalli wrote:
>> Hi,
>>
>> We have a requirement to forward BMC logs (the systemd journal) to an 
>> external logging server.
>>
>> We're proposing doing this by making use of rsyslog. Here are notes 
>> based on some initial investigation.
>>
> 
>  From the rsyslog github page
> 
> """
> Rsyslog's main sponsor Adiscon tries to fund rsyslog by selling custom 
> development and support contracts. Adiscon does NOT license rsyslog 
> under a commercial license (this is simply impossible for anyone due to 
> rsyslog's license structure).
> """
> 
> I think this license makes rsyslog a non-starter for most of the people 
> in this project, but I'm not a license expert.  We've looked at it in 
> the past, and while nice to use in practice, we couldn't overcome the 
> license conflicts and legal implications.

I'm not sure if that's the case, but I'm no license expert either. It 
looks like rsyslog's license is GPLv3. Is it that GPLv3 packages can't 
be part of a project that has an Apache license? I'll try to find out 
more about this from license experts.

Thanks,
Deepak

Re: Streaming logs via rsyslog

[Deepak Kodihalli]

On 17/08/18 12:00 PM, Deepak Kodihalli wrote:
> On 16/08/18 11:37 PM, Ed Tanous wrote:
>> On 08/16/2018 03:21 AM, Deepak Kodihalli wrote:
>>> Hi,
>>>
>>> We have a requirement to forward BMC logs (the systemd journal) to an 
>>> external logging server.
>>>
>>> We're proposing doing this by making use of rsyslog. Here are notes 
>>> based on some initial investigation.
>>>
>>
>>  From the rsyslog github page
>>
>> """
>> Rsyslog's main sponsor Adiscon tries to fund rsyslog by selling custom 
>> development and support contracts. Adiscon does NOT license rsyslog 
>> under a commercial license (this is simply impossible for anyone due 
>> to rsyslog's license structure).
>> """
>>
>> I think this license makes rsyslog a non-starter for most of the 
>> people in this project, but I'm not a license expert.  We've looked at 
>> it in the past, and while nice to use in practice, we couldn't 
>> overcome the license conflicts and legal implications.
> 
> I'm not sure if that's the case, but I'm no license expert either. It 
> looks like rsyslog's license is GPLv3. Is it that GPLv3 packages can't 
> be part of a project that has an Apache license? I'll try to find out 
> more about this from license experts.

I was just pointed out that Busybox itself is GPLv2, so rsyslog being 
GPLv3 seems to be usable without any concern?

Regards,
Deepak

RE: Streaming logs via rsyslog

[Tanous, Ed]

> 
> I was just pointed out that Busybox itself is GPLv2, so rsyslog being
> GPLv3 seems to be usable without any concern?
> 

GPLv2 and GPLv3 are very different licenses in some respects.  For a different product, we just went through this discussion about using rsyslog specifically, and the conclusion was that we couldn't use it given the license restrictions, but I don't know the specific details.  If you guys find a way around those restrictions, it would be good to know.

The team that did the analysis is out today, but I can try to find more next week.

-Ed

Re: Streaming logs via rsyslog

[Amithash Prasad]

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]

Again, I am not a licensing expert, nor a lawyer so take my opinion with a grain of salt. That out of the way, shouldn't we worry about license restrictions only if we are statically or dynamically linking these to other projects with a different license structure which might potentially be incompatible with GPLv3?


We have been using rsyslog's remote logging at FB for a bit now and it brings with it a lot of flexibility since we can reuse a lot of the infrastructure already in place to aggregate the logs coming from the servers.

[-- Attachment #2: Type: text/html, Size: 1230 bytes --]

Re: Streaming logs via rsyslog

[Brad Bishop]

> On Aug 16, 2018, at 6:21 AM, Deepak Kodihalli <dkodihal@linux.vnet.ibm.com> wrote:
> 
> Hi,
> 
> We have a requirement to forward BMC logs (the systemd journal) to an external logging server.
> 
> We're proposing doing this by making use of rsyslog. Here are notes based on some initial investigation.
> 
> Forwarding the journal logs to a syslog daemon :
> Journald has this feature of being able to write journal logs to a socket which then can be read by a syslog daemon. I see that (on Witherspoon at least) the syslog daemon running by default (busybox syslogd) doesn't support reading logs forwarded by the journald to due to this commit : https://git.busybox.net/busybox/commit/?id=accd9eeb719916da974584b33b1aeced5f3bb346. However, I could get the forwarding working with the rsyslog daemon. I found an existing rsyslog recipe in the meta-oe layer.
> 
> rsyslog configuration :
> rsyslog is configurable in various ways. For example, I can turn off local logging altogether via a config file, if I'm interested only in using rsyslog for streaming journal logs out. Various optional features can be turned off at build time to save flash space.
> 
> rsyslog server provision :
> The BMC would need to be made aware of the external rsyslog server. This can just be an admin updating the rsyslog config file on the BMC, and there could also be a REST API if need be. I think a small app can implement the https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/Client.interface.yaml interface to update the rsyslog config.
> 
> streaming :
> rsyslog can be configured to make use of TCP, or UDP, and optionally TLS. I think TCP is the default.
> 
> 
> Thoughts on this proposal? If this sounds okay, I'm planning to get some commits up in Gerrit for review. I can pull rsyslog in a phosphor layer (I mean as the default choice of a syslog daemon, replacing busybox syslog) if there's interest, as opposed to making it only a meta-witherspoon package. Specific rsyslog package config and config file overrides are possible in system specific layers.
> 
> Regards,
> Deepak

Hi Deepak

Plain-old syslogd can forward log entries over the network to another
syslogd instance.  Did we evaluate that?

-brad

Re: Streaming logs via rsyslog

[Deepak Kodihalli]

On 21/08/18 5:48 PM, Brad Bishop wrote:
> 
>> On Aug 16, 2018, at 6:21 AM, Deepak Kodihalli <dkodihal@linux.vnet.ibm.com> wrote:
>>
>> Hi,
>>
>> We have a requirement to forward BMC logs (the systemd journal) to an external logging server.
>>
>> We're proposing doing this by making use of rsyslog. Here are notes based on some initial investigation.
>>
>> Forwarding the journal logs to a syslog daemon :
>> Journald has this feature of being able to write journal logs to a socket which then can be read by a syslog daemon. I see that (on Witherspoon at least) the syslog daemon running by default (busybox syslogd) doesn't support reading logs forwarded by the journald to due to this commit : https://git.busybox.net/busybox/commit/?id=accd9eeb719916da974584b33b1aeced5f3bb346. However, I could get the forwarding working with the rsyslog daemon. I found an existing rsyslog recipe in the meta-oe layer.
>>
>> rsyslog configuration :
>> rsyslog is configurable in various ways. For example, I can turn off local logging altogether via a config file, if I'm interested only in using rsyslog for streaming journal logs out. Various optional features can be turned off at build time to save flash space.
>>
>> rsyslog server provision :
>> The BMC would need to be made aware of the external rsyslog server. This can just be an admin updating the rsyslog config file on the BMC, and there could also be a REST API if need be. I think a small app can implement the https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/Client.interface.yaml interface to update the rsyslog config.
>>
>> streaming :
>> rsyslog can be configured to make use of TCP, or UDP, and optionally TLS. I think TCP is the default.
>>
>>
>> Thoughts on this proposal? If this sounds okay, I'm planning to get some commits up in Gerrit for review. I can pull rsyslog in a phosphor layer (I mean as the default choice of a syslog daemon, replacing busybox syslog) if there's interest, as opposed to making it only a meta-witherspoon package. Specific rsyslog package config and config file overrides are possible in system specific layers.
>>
>> Regards,
>> Deepak
> 
> Hi Deepak
> 
> Plain-old syslogd can forward log entries over the network to another
> syslogd instance.  Did we evaluate that?
> 
> -brad

Hi Brad,

The problem with syslogd was not being able to read logs forwarded by 
journald (due to the commit I noted in my original email). Besides that, 
rsyslog's filtering capabilities are much more impressive.

Regards,
Deepak