Integrate collectd with OpenBMC

Integrate collectd with OpenBMC

[Kun Yi]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

Hello there,

In the context of reporting BMC performance metrics, my intern Gabriel
(cc'ed here) and I have started looking at integrating collectd as a
metrics collection tool on OpenBMC. We have got it running, which is
trivial, but the next question is how to report the data.

We have thought about it and thinks implementing a D-Bus interface to be
the most flexible approach. At first, we could implement a snapshot
(instantaneos read) interface. It would then be fairly straightforward to
add them as Redfish/IPMI sensors.

There are two ways to do this:
1. Implement as a collectd "D-Bus" plugin [1]. Collectd supports writing
custom plugins which are C files calling the internal plugin APIs. Could
probably use sdbus to implement.

+ could potentially be upstreamed to collectd
- the code probably will live in a downstream fork first, and if it doesn't
end up upstream, maintaining could become an issue since collectd plugin
API is not guaranteed stable
- C

2. Implement as an interposer daemon that translates between one of the
formats that collectd supports (unix socket, plaintext, RRDTool..) to D-Bus

+ project could be purely OpenBMC
+ can use sdbusplus
- another daemon

Any advice on this? Currently we are leaning towards the first approach,
but do you agree the D-Bus plugin is general enough to be of interest to
the upstream collectd community?

I can definitely reach out to the collectd group but just want to ask here
first :)

[1] Collectd plugins:
https://collectd.org/wiki/index.php/Plugin_architecture
-- 
Regards,
Kun

[-- Attachment #2: Type: text/html, Size: 2061 bytes --]

Re: Integrate collectd with OpenBMC

[vishwa]

[-- Attachment #1: Type: text/plain, Size: 2029 bytes --]

Hi Kun,

I support approach:1 as it's more collectd standard. IMO, using "C" 
should be fine if that makes more sense.

For example: https://github.com/openbmc/pldm/tree/master/libpldm is in 
C, so it can be used elsewhere.

btw, for your approach:2, how is this daemon first getting the data from 
collectd before it can translate.

!! Vishwa !!

On 6/28/19 11:21 AM, Kun Yi wrote:
> Hello there,
>
> In the context of reporting BMC performance metrics, my intern Gabriel 
> (cc'ed here) and I have started looking at integrating collectd as a 
> metrics collection tool on OpenBMC. We have got it running, which is 
> trivial, but the next question is how to report the data.
>
> We have thought about it and thinks implementing a D-Bus interface to 
> be the most flexible approach. At first, we could implement a snapshot 
> (instantaneos read) interface. It would then be fairly straightforward 
> to add them as Redfish/IPMI sensors.
>
> There are two ways to do this:
> 1. Implement as a collectd "D-Bus" plugin [1]. Collectd supports 
> writing custom plugins which are C files calling the internal plugin 
> APIs. Could probably use sdbus to implement.
>
> + could potentially be upstreamed to collectd
> - the code probably will live in a downstream fork first, and if it 
> doesn't end up upstream, maintaining could become an issue since 
> collectd plugin API is not guaranteed stable
> - C
>
> 2. Implement as an interposer daemon that translates between one of 
> the formats that collectd supports (unix socket, plaintext, RRDTool..) 
> to D-Bus
>
> + project could be purely OpenBMC
> + can use sdbusplus
> - another daemon
>
> Any advice on this? Currently we are leaning towards the first 
> approach, but do you agree the D-Bus plugin is general enough to be of 
> interest to the upstream collectd community?
>
> I can definitely reach out to the collectd group but just want to ask 
> here first :)
>
> [1] Collectd plugins: 
> https://collectd.org/wiki/index.php/Plugin_architecture
> -- 
> Regards,
> Kun

[-- Attachment #2: Type: text/html, Size: 3609 bytes --]

Re: Integrate collectd with OpenBMC

[Brad Bishop]

On Thu, Jun 27, 2019 at 10:51:03PM -0700, Kun Yi wrote:
>Hello there,
>
>In the context of reporting BMC performance metrics, my intern Gabriel
>(cc'ed here) and I have started looking at integrating collectd as a
>metrics collection tool on OpenBMC. We have got it running, which is
>trivial, but the next question is how to report the data.
>
>We have thought about it and thinks implementing a D-Bus interface to be
>the most flexible approach. At first, we could implement a snapshot
>(instantaneos read) interface. It would then be fairly straightforward to
>add them as Redfish/IPMI sensors.

Just curious in what situations DBus was required?  In the design I 
thought you had applications going right to librrd - where did that fall 
down?

>
>There are two ways to do this:
>1. Implement as a collectd "D-Bus" plugin [1]. Collectd supports writing
>custom plugins which are C files calling the internal plugin APIs. Could
>probably use sdbus to implement.
>
>+ could potentially be upstreamed to collectd
>- the code probably will live in a downstream fork first, and if it doesn't
>end up upstream, maintaining could become an issue since collectd plugin
>API is not guaranteed stable
>- C
>
>2. Implement as an interposer daemon that translates between one of the
>formats that collectd supports (unix socket, plaintext, RRDTool..) to D-Bus
>
>+ project could be purely OpenBMC
>+ can use sdbusplus
>- another daemon
>
>Any advice on this? Currently we are leaning towards the first approach,
>but do you agree the D-Bus plugin is general enough to be of interest to
>the upstream collectd community?
>
>I can definitely reach out to the collectd group but just want to ask here
>first :)
>
>[1] Collectd plugins:
>https://collectd.org/wiki/index.php/Plugin_architecture
>-- 
>Regards,
>Kun

Re: Integrate collectd with OpenBMC

[vishwa]

Hi,


On 7/22/19 10:00 PM, Brad Bishop wrote:
> On Thu, Jun 27, 2019 at 10:51:03PM -0700, Kun Yi wrote:
>> Hello there,
>>
>> In the context of reporting BMC performance metrics, my intern Gabriel
>> (cc'ed here) and I have started looking at integrating collectd as a
>> metrics collection tool on OpenBMC. We have got it running, which is
>> trivial, but the next question is how to report the data.
>>
>> We have thought about it and thinks implementing a D-Bus interface to be
>> the most flexible approach. At first, we could implement a snapshot
>> (instantaneos read) interface. It would then be fairly 
>> straightforward to
>> add them as Redfish/IPMI sensors.
>
> Just curious in what situations DBus was required?  In the design I 
> thought you had applications going right to librrd - where did that 
> fall down?
>

 From what I saw in the design template, librrd to be used to enable any 
daemon to make their data available to collectd.

The proposal here is to externally report the data collected. I 
understood it as; we going to have per plugin D-Bus APIs to report it. 
But looking at it again, I am not sure how is that achieved.

Kun, Do you have a flow schematic ?


>>
>> There are two ways to do this:
>> 1. Implement as a collectd "D-Bus" plugin [1]. Collectd supports writing
>> custom plugins which are C files calling the internal plugin APIs. Could
>> probably use sdbus to implement.
>>
>> + could potentially be upstreamed to collectd
>> - the code probably will live in a downstream fork first, and if it 
>> doesn't
>> end up upstream, maintaining could become an issue since collectd plugin
>> API is not guaranteed stable
>> - C
>>
>> 2. Implement as an interposer daemon that translates between one of the
>> formats that collectd supports (unix socket, plaintext, RRDTool..) to 
>> D-Bus
>>
>> + project could be purely OpenBMC
>> + can use sdbusplus
>> - another daemon
>>
>> Any advice on this? Currently we are leaning towards the first approach,
>> but do you agree the D-Bus plugin is general enough to be of interest to
>> the upstream collectd community?
>>
>> I can definitely reach out to the collectd group but just want to ask 
>> here
>> first :)
>>
>> [1] Collectd plugins:
>> https://collectd.org/wiki/index.php/Plugin_architecture
>> -- 
>> Regards,
>> Kun
>

Re: Integrate collectd with OpenBMC

[Kun Yi]

hi Brad,

We are making some progress toward two things that we'd like to push upstream:
1. An C++ wrapper for RRD "-- librrdplus"
2. An IPMI OEM handler -- "rrd-ipmi-blobs"

Would you please help create these two repositories?

On Mon, Jul 22, 2019 at 9:30 AM Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:
>
> On Thu, Jun 27, 2019 at 10:51:03PM -0700, Kun Yi wrote:
> >Hello there,
> >
> >In the context of reporting BMC performance metrics, my intern Gabriel
> >(cc'ed here) and I have started looking at integrating collectd as a
> >metrics collection tool on OpenBMC. We have got it running, which is
> >trivial, but the next question is how to report the data.
> >
> >We have thought about it and thinks implementing a D-Bus interface to be
> >the most flexible approach. At first, we could implement a snapshot
> >(instantaneos read) interface. It would then be fairly straightforward to
> >add them as Redfish/IPMI sensors.
>
> Just curious in what situations DBus was required?  In the design I
> thought you had applications going right to librrd - where did that fall
> down?

Originally we did envision a D-Bus based data pipeline, but after
researching we think a
file based approach is going to be as flexible and more efficient.
Currently we don't see
D-Bus as strictly needed in the collection path. It *might* be useful
to have a few tunable
knobs exposed through D-Bus, but again it can just be a build-time configuration
for now.

>
> >
> >There are two ways to do this:
> >1. Implement as a collectd "D-Bus" plugin [1]. Collectd supports writing
> >custom plugins which are C files calling the internal plugin APIs. Could
> >probably use sdbus to implement.
> >
> >+ could potentially be upstreamed to collectd
> >- the code probably will live in a downstream fork first, and if it doesn't
> >end up upstream, maintaining could become an issue since collectd plugin
> >API is not guaranteed stable
> >- C
> >
> >2. Implement as an interposer daemon that translates between one of the
> >formats that collectd supports (unix socket, plaintext, RRDTool..) to D-Bus
> >
> >+ project could be purely OpenBMC
> >+ can use sdbusplus
> >- another daemon
> >
> >Any advice on this? Currently we are leaning towards the first approach,
> >but do you agree the D-Bus plugin is general enough to be of interest to
> >the upstream collectd community?
> >
> >I can definitely reach out to the collectd group but just want to ask here
> >first :)
> >
> >[1] Collectd plugins:
> >https://collectd.org/wiki/index.php/Plugin_architecture
> >--
> >Regards,
> >Kun



-- 
Regards,
Kun

Re: Integrate collectd with OpenBMC

[Brad Bishop]

at 12:30 PM, Kun Yi <kunyi@google.com> wrote:

> hi Brad,
>
> We are making some progress toward two things that we'd like to push  
> upstream:
> 1. An C++ wrapper for RRD "-- librrdplus”
> 2. An IPMI OEM handler -- "rrd-ipmi-blobs”
>
> Would you please help create these two repositories?

done!

Sorry about the delay Kun.  I added this to my todo list and then forgot  
about it :-(

-brad

Re: Integrate collectd with OpenBMC

[Kun Yi]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

No problem, thanks!
Brad, could you also add Gabriel to the list of OpenBMC contributors? His
email is gabomatute@gmail.com
Does he need to sign the CLA individually?

On Tue, Aug 20, 2019 at 9:19 AM Brad Bishop <bradleyb@fuzziesquirrel.com>
wrote:

> at 12:30 PM, Kun Yi <kunyi@google.com> wrote:
>
> > hi Brad,
> >
> > We are making some progress toward two things that we'd like to push
> > upstream:
> > 1. An C++ wrapper for RRD "-- librrdplus”
> > 2. An IPMI OEM handler -- "rrd-ipmi-blobs”
> >
> > Would you please help create these two repositories?
>
> done!
>
> Sorry about the delay Kun.  I added this to my todo list and then forgot
> about it :-(
>
> -brad
>


-- 
Regards,
Kun

[-- Attachment #2: Type: text/html, Size: 1308 bytes --]

Re: Integrate collectd with OpenBMC

[Brad Bishop]

at 4:46 PM, Kun Yi <kunyi@google.com> wrote:

> No problem, thanks!
> Brad, could you also add Gabriel to the list of OpenBMC contributors? His  
> email is gabomatute@gmail.com
> Does he need to sign the CLA individually?

Hi Kun

No the corporate CLA has the concept of a CLA manager who can add and  
remove people from the company CLA.

Nancy can you add Gabriel to Googles schedule A and to the  
google/ci-authorized gerrit group?

thx - brad

Re: Integrate collectd with OpenBMC

[Nancy Yuen]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

Will do.
----------
Nancy


On Wed, Aug 21, 2019 at 5:14 AM Brad Bishop <bradleyb@fuzziesquirrel.com>
wrote:

> at 4:46 PM, Kun Yi <kunyi@google.com> wrote:
>
> > No problem, thanks!
> > Brad, could you also add Gabriel to the list of OpenBMC contributors?
> His
> > email is gabomatute@gmail.com
> > Does he need to sign the CLA individually?
>
> Hi Kun
>
> No the corporate CLA has the concept of a CLA manager who can add and
> remove people from the company CLA.
>
> Nancy can you add Gabriel to Googles schedule A and to the
> google/ci-authorized gerrit group?
>
> thx - brad
>

[-- Attachment #2: Type: text/html, Size: 1169 bytes --]