Historical Sensor Information

Historical Sensor Information

[Wilfred Smith]

I presume most vendors desire the ability to query historical sensor information from the BMC.
	Has this feature been implemented already? If so, please direct me.
	If not, has someone already begun development?
	Is there an existing specification or write-up?
	Any ‘druthers or preferences on how I might proceed such that my effort benefits the wider community?

Wilfred

Re: Historical Sensor Information

[Emily Shaffer]

On Wed, Aug 14, 2019 at 4:05 PM Wilfred Smith <wilfredsmith@fb.com> wrote:
>
> I presume most vendors desire the ability to query historical sensor information from the BMC.
>         Has this feature been implemented already? If so, please direct me.
>         If not, has someone already begun development?
>         Is there an existing specification or write-up?
>         Any ‘druthers or preferences on how I might proceed such that my effort benefits the wider community?
>
> Wilfred

I think that the space constriction on many BMCs has left folks to
instead query over IPMI/Redfish and compile historical information
elsewhere. Can you tell a little more about the use case and indicate
why you would rather save history on the BMC than off the BMC?

A related topic which - as I recall - was discussed and never
implemented is the topic of metrics reporting. It's possible that the
community has moved further on these topics than I remember, though,
as I've been fairly out of the loop lately.
 - Emily

Re: Historical Sensor Information

[Wilfred Smith]

Emily,

I’ll need to check with my compatriots here at Facebook for our specific use cases.

Can you point me to the discussion on metrics reporting?

Wilfred
 

> On Aug 14, 2019, at 7:51 PM, Emily Shaffer <emilyshaffer@google.com> wrote:
> 
> On Wed, Aug 14, 2019 at 4:05 PM Wilfred Smith <wilfredsmith@fb.com> wrote:
>> 
>> I presume most vendors desire the ability to query historical sensor information from the BMC.
>>        Has this feature been implemented already? If so, please direct me.
>>        If not, has someone already begun development?
>>        Is there an existing specification or write-up?
>>        Any ‘druthers or preferences on how I might proceed such that my effort benefits the wider community?
>> 
>> Wilfred
> 
> I think that the space constriction on many BMCs has left folks to
> instead query over IPMI/Redfish and compile historical information
> elsewhere. Can you tell a little more about the use case and indicate
> why you would rather save history on the BMC than off the BMC?
> 
> A related topic which - as I recall - was discussed and never
> implemented is the topic of metrics reporting. It's possible that the
> community has moved further on these topics than I remember, though,
> as I've been fairly out of the loop lately.
> - Emily

RE: Historical Sensor Information

[Milton Miller II]

On August 15, 2019, Wilfred Smith wrote:
>I’ll need to check with my compatriots here at Facebook for our
>specific use cases.
>
>Can you point me to the discussion on metrics reporting?
>

Probably this thread here, there seems to be a working group with a meeting schedule:

https://lists.ozlabs.org/pipermail/openbmc/2019-August/017412.html

Platform telemetry and health monitoring - PST AM 

https://github.com/openbmc/openbmc/wiki/Platform-telemetry-and-health-monitoring-Work-Group

>
>> On Aug 14, 2019, at 7:51 PM, Emily Shaffer
><emilyshaffer@google.com> wrote:
>> 
>> On Wed, Aug 14, 2019 at 4:05 PM Wilfred Smith <wilfredsmith@fb.com>
>wrote:
>>> 
>>> I presume most vendors desire the ability to query historical
>sensor information from the BMC.
>>>        Has this feature been implemented already? If so, please
>direct me.
>>>        If not, has someone already begun development?
>>>        Is there an existing specification or write-up?
>>>        Any ‘druthers or preferences on how I might proceed such
>that my effort benefits the wider community?
>>> 
>>> Wilfred
>> 
>> I think that the space constriction on many BMCs has left folks to
>> instead query over IPMI/Redfish and compile historical information
>> elsewhere. Can you tell a little more about the use case and
>indicate
>> why you would rather save history on the BMC than off the BMC?
>> 
>> A related topic which - as I recall - was discussed and never
>> implemented is the topic of metrics reporting. It's possible that
>the
>> community has moved further on these topics than I remember,
>though,
>> as I've been fairly out of the loop lately.
>> - Emily
>
>

Re: Historical Sensor Information

[Kun Yi]

[-- Attachment #1: Type: text/plain, Size: 2580 bytes --]

On Thu, Aug 15, 2019 at 9:41 AM Milton Miller II <miltonm@us.ibm.com> wrote:

> On August 15, 2019, Wilfred Smith wrote:
> >I’ll need to check with my compatriots here at Facebook for our
> >specific use cases.
> >
> >Can you point me to the discussion on metrics reporting?
> >
>
> Probably this thread here, there seems to be a working group with a
> meeting schedule:
>
> https://lists.ozlabs.org/pipermail/openbmc/2019-August/017412.html
>
> Platform telemetry and health monitoring - PST AM
>
>
> https://github.com/openbmc/openbmc/wiki/Platform-telemetry-and-health-monitoring-Work-Group
>
> We are currently looking into using collectd to store metrics. You can
look into the design doc proposal here:
https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/22257
The design still needs to be fleshed out in a few places, but we are
working towards implementing a prototype.

How far back and how often you need to collect sensor information is also
important. If you are interested in discussion, could you fill out this
form to specify your FR? thanks

 https://docs.google.com/spreadsheets/d/12gMMXB9r_WfWDf5wz-Z_zXsz6RNheC6p2LKp7HePAEE/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/12gMMXB9r_WfWDf5wz-Z_zXsz6RNheC6p2LKp7HePAEE/edit?usp=sharing>

>
> >> On Aug 14, 2019, at 7:51 PM, Emily Shaffer
> ><emilyshaffer@google.com> wrote:
> >>
> >> On Wed, Aug 14, 2019 at 4:05 PM Wilfred Smith <wilfredsmith@fb.com>
> >wrote:
> >>>
> >>> I presume most vendors desire the ability to query historical
> >sensor information from the BMC.
> >>>        Has this feature been implemented already? If so, please
> >direct me.
> >>>        If not, has someone already begun development?
> >>>        Is there an existing specification or write-up?
> >>>        Any ‘druthers or preferences on how I might proceed such
> >that my effort benefits the wider community?
> >>>
> >>> Wilfred
> >>
> >> I think that the space constriction on many BMCs has left folks to
> >> instead query over IPMI/Redfish and compile historical information
> >> elsewhere. Can you tell a little more about the use case and
> >indicate
> >> why you would rather save history on the BMC than off the BMC?
> >>
> >> A related topic which - as I recall - was discussed and never
> >> implemented is the topic of metrics reporting. It's possible that
> >the
> >> community has moved further on these topics than I remember,
> >though,
> >> as I've been fairly out of the loop lately.
> >> - Emily
> >
> >
>
>

-- 
Regards,
Kun

[-- Attachment #2: Type: text/html, Size: 4014 bytes --]

Re: Historical Sensor Information

[Wilfred Smith]

[-- Attachment #1: Type: text/plain, Size: 4368 bytes --]

Many thanks to Emily, Milton and Kun Yi for their quick responses and pointers.

Among the reasons for local historical data collection are independent auditing, disaster recovery, debugging and increasing availability during periods of intermittent network connectivity.

Vijay is already participating in the Telemetry workgroup, so I’ll try to get a download from him to see what can be leveraged.

The requirement at Facebook includes the ability to retrieve historical sensor information for a user-defined period and interval on the BMC console (through sensor-util, and in the same format). Based on my cursory review of collectd, its lossy multicast network protocol wouldn’t allow this information to be re-synthesized with fidelity on the client side, but it would be a huge win if we can get a few concessions. I intend to study the Telemetry Workgroup’s progress carefully.

Wilfred

On Aug 15, 2019, at 9:57 AM, Kun Yi <kunyi@google.com<mailto:kunyi@google.com>> wrote:


On Thu, Aug 15, 2019 at 9:41 AM Milton Miller II <miltonm@us.ibm.com<mailto:miltonm@us.ibm.com>> wrote:
On August 15, 2019, Wilfred Smith wrote:
>I’ll need to check with my compatriots here at Facebook for our
>specific use cases.
>
>Can you point me to the discussion on metrics reporting?
>

Probably this thread here, there seems to be a working group with a meeting schedule:

https://lists.ozlabs.org/pipermail/openbmc/2019-August/017412.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ozlabs.org_pipermail_openbmc_2019-2DAugust_017412.html&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=-ektT-tD9zf2rfUisE63RqiDagGyhGey2hbEGa-47kc&m=h9DyV6CCrv1XxjDfs7mqBtORSNDPxHcMgyqSCKMBzUk&s=e8YQZXFiubzEHkXPHBtAOpIY5c1PeWSsYpIVhp_ISho&e=>

Platform telemetry and health monitoring - PST AM

https://github.com/openbmc/openbmc/wiki/Platform-telemetry-and-health-monitoring-Work-Group

We are currently looking into using collectd to store metrics. You can look into the design doc proposal here:
https://gerrit.openbmc-project.xyz/c/openbmc/docs/+/22257<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.openbmc-2Dproject.xyz_c_openbmc_docs_-2B_22257&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=-ektT-tD9zf2rfUisE63RqiDagGyhGey2hbEGa-47kc&m=h9DyV6CCrv1XxjDfs7mqBtORSNDPxHcMgyqSCKMBzUk&s=i2mO2GcFSRK7u4Qz-5Aig3Y3869N3g79h54_d68e-HE&e=>
The design still needs to be fleshed out in a few places, but we are working towards implementing a prototype.

How far back and how often you need to collect sensor information is also important. If you are interested in discussion, could you fill out this form to specify your FR? thanks

 https://docs.google.com/spreadsheets/d/12gMMXB9r_WfWDf5wz-Z_zXsz6RNheC6p2LKp7HePAEE/edit?usp=sharing<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_spreadsheets_d_12gMMXB9r-5FWfWDf5wz-2DZ-5FzXsz6RNheC6p2LKp7HePAEE_edit-3Fusp-3Dsharing&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=-ektT-tD9zf2rfUisE63RqiDagGyhGey2hbEGa-47kc&m=h9DyV6CCrv1XxjDfs7mqBtORSNDPxHcMgyqSCKMBzUk&s=nGuMcmHWAz6JOxHQMvHHPtZMq_xKjt1w3yIfmYDVkc8&e=>

>
>> On Aug 14, 2019, at 7:51 PM, Emily Shaffer
><emilyshaffer@google.com<mailto:emilyshaffer@google.com>> wrote:
>>
>> On Wed, Aug 14, 2019 at 4:05 PM Wilfred Smith <wilfredsmith@fb.com<mailto:wilfredsmith@fb.com>>
>wrote:
>>>
>>> I presume most vendors desire the ability to query historical
>sensor information from the BMC.
>>>        Has this feature been implemented already? If so, please
>direct me.
>>>        If not, has someone already begun development?
>>>        Is there an existing specification or write-up?
>>>        Any ‘druthers or preferences on how I might proceed such
>that my effort benefits the wider community?
>>>
>>> Wilfred
>>
>> I think that the space constriction on many BMCs has left folks to
>> instead query over IPMI/Redfish and compile historical information
>> elsewhere. Can you tell a little more about the use case and
>indicate
>> why you would rather save history on the BMC than off the BMC?
>>
>> A related topic which - as I recall - was discussed and never
>> implemented is the topic of metrics reporting. It's possible that
>the
>> community has moved further on these topics than I remember,
>though,
>> as I've been fairly out of the loop lately.
>> - Emily
>
>



--
Regards,
Kun


[-- Attachment #2: Type: text/html, Size: 7284 bytes --]

Re: Historical Sensor Information

[Joseph Reynolds]

On 8/15/19 5:50 PM, Wilfred Smith wrote:
> Many thanks to Emily, Milton and Kun Yi for their quick responses and 
> pointers.
>
> Among the reasons for local historical data collection are independent 
> auditing, disaster recovery, debugging and increasing availability 
> during periods of intermittent network connectivity.
>
> Vijay is already participating in the Telemetry workgroup, so I’ll try 
> to get a download from him to see what can be leveraged.
>
> The requirement at Facebook includes the ability to retrieve 
> historical sensor information for a user-defined period and interval 
> on the BMC console (through sensor-util, and in the same format). 
> Based on my cursory review of collectd, its lossy multicast network 
> protocol wouldn’t allow this information to be re-synthesized with 
> fidelity on the

I share this concern and would prefer to have a more reliable way to get 
sensor data off the BMC.  This data may be valuable to help detect when 
the BMC is being attacked.

- Joseph

> client side, but it would be a huge win if we can get a few 
> concessions. I intend to study the Telemetry Workgroup’s progress 
> carefully.
>
> Wilfred
>
...snip...

Re: Historical Sensor Information

[Brad Bishop]

at 1:02 PM, Joseph Reynolds <jrey@linux.ibm.com> wrote:

>
>
> On 8/15/19 5:50 PM, Wilfred Smith wrote:
>> Many thanks to Emily, Milton and Kun Yi for their quick responses and  
>> pointers.
>>
>> Among the reasons for local historical data collection are independent  
>> auditing, disaster recovery, debugging and increasing availability  
>> during periods of intermittent network connectivity.
>>
>> Vijay is already participating in the Telemetry workgroup, so I’ll try  
>> to get a download from him to see what can be leveraged.
>>
>> The requirement at Facebook includes the ability to retrieve historical  
>> sensor information for a user-defined period and interval on the BMC  
>> console (through sensor-util, and in the same format). Based on my  
>> cursory review of collectd, its lossy multicast network protocol  
>> wouldn’t allow this information to be re-synthesized with fidelity on the
>
> I share this concern and would prefer to have a more reliable way to get  
> sensor data off the BMC.  This data may be valuable to help detect when  
> the BMC is being attacked.

I don’t think anyone intended to use the collectd network plugin to publish  
the sensor data (although you could).  IIUC the intent was to use collectd  
only in the capacity of collecting the sensor data into an rrd database on  
the BMC flash.  From there you can get the data out of the BMC however you  
like.  IPMI<->rrd.  Redfish<->rrd.  ssh+cli<->rrd.  Do I have it right Kun?

Re: Historical Sensor Information

[Brad Bishop]

at 10:51 PM, Emily Shaffer <emilyshaffer@google.com> wrote:

> On Wed, Aug 14, 2019 at 4:05 PM Wilfred Smith <wilfredsmith@fb.com> wrote:
>> I presume most vendors desire the ability to query historical sensor  
>> information from the BMC.
>>         Has this feature been implemented already? If so, please direct me.
>>         If not, has someone already begun development?
>>         Is there an existing specification or write-up?
>>         Any ‘druthers or preferences on how I might proceed such that my effort benefits the wider community?
>>
>> Wilfred
>
> I think that the space constriction on many BMCs has left folks to
> instead query over IPMI/Redfish and compile historical information
> elsewhere. Can you tell a little more about the use case and indicate
> why you would rather save history on the BMC than off the BMC?

Hi Emily.  FWIW, some of IBMs customers also have the desire to get  
historical sensor information from the BMC.  In our existing server  
products this is something we’ve offered them and will continue to offer  
going forward.

-brad

Re: Historical Sensor Information

[Brad Bishop]

at 7:04 PM, Wilfred Smith <wilfredsmith@fb.com> wrote:

> 	Any ‘druthers or preferences on how I might proceed such that my effort benefits the wider community?

Hi Wilfred

Since you asked it in this way :-) if I had my ‘druthers on how you’d  
proceed to benefit me it would be prototyping collectd with the sensors  
input plugin and the rrd output plugin.

Kun, Neeraj do you agree?

thx - brad