Telemetry Reports dynamic update

Telemetry Reports dynamic update

[Justin Thaler]

Hi All,
	I've been sort of silently reviewing the designs to telemetry reports 
design along with subscriptions. As I was reviewing it, I've had a 
couple of different design points come up and an interesting use case 
I've wanted to ask about. So here's my scenario:

I've set up a telemetry report to the system dynamically by querying the 
sensors and request the values are updated every second. The system was 
in a state where it was powered off, so not many sensors were available. 
After the system is powered on, more sensors are added as different 
devices are powered up. At this point I'd want to add the additional 
sensors to my report. I query the system again for all the available 
sensors and see thermal and power readings for the CPUs. I want updates 
on these every second just like my initial report.

This leads me to the following question:
Can I push a new report definition to the TelemetryService and have the 
event service start providing that new definition without tearing down 
the subscription first, which would leave a gap in the data?

Thanks,
Justin Thaler

RE: [EXTERNAL] Telemetry Reports dynamic update

[Neeraj Ladkani]

We don’t have telemetryservice in BMC currently so are you currently doing it using webui? Or you have written a app using redfish APIs?  I asked similar question in last week work group meeting. I think it will be useful to have ability to add new report definitions.

-----Original Message-----
From: Justin Thaler <thalerj@linux.vnet.ibm.com> 
Sent: Tuesday, December 3, 2019 3:57 PM
To: openbmc@lists.ozlabs.org; Matuszczak, Piotr <piotr.matuszczak@intel.com>; vishwa <vishwa@linux.vnet.ibm.com>; Neeraj Ladkani <neladk@microsoft.com>
Subject: [EXTERNAL] Telemetry Reports dynamic update

Hi All,
	I've been sort of silently reviewing the designs to telemetry reports design along with subscriptions. As I was reviewing it, I've had a couple of different design points come up and an interesting use case I've wanted to ask about. So here's my scenario:

I've set up a telemetry report to the system dynamically by querying the sensors and request the values are updated every second. The system was in a state where it was powered off, so not many sensors were available. 
After the system is powered on, more sensors are added as different devices are powered up. At this point I'd want to add the additional sensors to my report. I query the system again for all the available sensors and see thermal and power readings for the CPUs. I want updates on these every second just like my initial report.

This leads me to the following question:
Can I push a new report definition to the TelemetryService and have the event service start providing that new definition without tearing down the subscription first, which would leave a gap in the data?

Thanks,
Justin Thaler

Re: [EXTERNAL] Telemetry Reports dynamic update

[Justin Thaler]

On 12/4/19 12:43 AM, Neeraj Ladkani wrote:
> We don’t have telemetryservice in BMC currently so are you currently doing it using webui? Or you have written a app using redfish APIs?  I asked similar question in last week work group meeting. I think it will be useful to have ability to add new report definitions.
I have an app that uses the current websocket implementation on openBMC 
and have been reviewing the needs for transitioning the app to redfish 
when the time comes. One limitation of those is not being able to update 
the active subscription without tearing it down and starting it again, 
resulting in a few seconds of missed readings.

Being able to update the content of the subscription on the fly would be 
a huge value add I think.
> 
> -----Original Message-----
> From: Justin Thaler <thalerj@linux.vnet.ibm.com>
> Sent: Tuesday, December 3, 2019 3:57 PM
> To: openbmc@lists.ozlabs.org; Matuszczak, Piotr <piotr.matuszczak@intel.com>; vishwa <vishwa@linux.vnet.ibm.com>; Neeraj Ladkani <neladk@microsoft.com>
> Subject: [EXTERNAL] Telemetry Reports dynamic update
> 
> Hi All,
> 	I've been sort of silently reviewing the designs to telemetry reports design along with subscriptions. As I was reviewing it, I've had a couple of different design points come up and an interesting use case I've wanted to ask about. So here's my scenario:
> 
> I've set up a telemetry report to the system dynamically by querying the sensors and request the values are updated every second. The system was in a state where it was powered off, so not many sensors were available.
> After the system is powered on, more sensors are added as different devices are powered up. At this point I'd want to add the additional sensors to my report. I query the system again for all the available sensors and see thermal and power readings for the CPUs. I want updates on these every second just like my initial report.
> 
> This leads me to the following question:
> Can I push a new report definition to the TelemetryService and have the event service start providing that new definition without tearing down the subscription first, which would leave a gap in the data?
> 
> Thanks,
> Justin Thaler
> 

RE: [EXTERNAL] Telemetry Reports dynamic update

[Matuszczak, Piotr]

Hi Justin, 

You've raised an interesting topic. In current implementation, when OpenBMC boots when platform is in power off state, not all sensors are available, thus they won't appear in Redfish tree and you won't be able to configure metric report if you want to use these sensors. If BMC boots during the platform boot, all available sensors will be discovered and exposed in the Redfish tree. After this, when platform is powered off, the sensors, that are not available in this platform state shall be still available in the Redfish tree, but they won't provide valid readings. Unfortunately, such behavior is not standardized, thus if we want to be able to define metric reports when platform is powered off, we will have to make the required sensors behavior part of the design. 

Regards
Piotr


-----Original Message-----
From: Justin Thaler <thalerj@linux.vnet.ibm.com> 
Sent: Wednesday, December 4, 2019 2:28 PM
To: Neeraj Ladkani <neladk@microsoft.com>; openbmc@lists.ozlabs.org; Matuszczak, Piotr <piotr.matuszczak@intel.com>; vishwa <vishwa@linux.vnet.ibm.com>
Subject: Re: [EXTERNAL] Telemetry Reports dynamic update



On 12/4/19 12:43 AM, Neeraj Ladkani wrote:
> We don’t have telemetryservice in BMC currently so are you currently doing it using webui? Or you have written a app using redfish APIs?  I asked similar question in last week work group meeting. I think it will be useful to have ability to add new report definitions.
I have an app that uses the current websocket implementation on openBMC and have been reviewing the needs for transitioning the app to redfish when the time comes. One limitation of those is not being able to update the active subscription without tearing it down and starting it again, resulting in a few seconds of missed readings.

Being able to update the content of the subscription on the fly would be a huge value add I think.
> 
> -----Original Message-----
> From: Justin Thaler <thalerj@linux.vnet.ibm.com>
> Sent: Tuesday, December 3, 2019 3:57 PM
> To: openbmc@lists.ozlabs.org; Matuszczak, Piotr 
> <piotr.matuszczak@intel.com>; vishwa <vishwa@linux.vnet.ibm.com>; 
> Neeraj Ladkani <neladk@microsoft.com>
> Subject: [EXTERNAL] Telemetry Reports dynamic update
> 
> Hi All,
> 	I've been sort of silently reviewing the designs to telemetry reports design along with subscriptions. As I was reviewing it, I've had a couple of different design points come up and an interesting use case I've wanted to ask about. So here's my scenario:
> 
> I've set up a telemetry report to the system dynamically by querying the sensors and request the values are updated every second. The system was in a state where it was powered off, so not many sensors were available.
> After the system is powered on, more sensors are added as different devices are powered up. At this point I'd want to add the additional sensors to my report. I query the system again for all the available sensors and see thermal and power readings for the CPUs. I want updates on these every second just like my initial report.
> 
> This leads me to the following question:
> Can I push a new report definition to the TelemetryService and have the event service start providing that new definition without tearing down the subscription first, which would leave a gap in the data?
> 
> Thanks,
> Justin Thaler
> 

Re: [EXTERNAL] Telemetry Reports dynamic update

[Justin Thaler]

Hi Piotr,
	Agreed, this type of thing, OOB Telemetry, in the industry is 
relatively new and we've seen a lot of interest in it, with some pretty 
wide varying use cases. I brought up this particular case as we had the 
same behavior on the legacy rest API where the sensors were dynamically 
added and removed, so any listening software had to actively track that 
list. It really boiled down to knowing when the sensors list would be 
updated, and keying a rescan at that point in time. The trouble I ran 
into is it required a resubscribe to change the sensor list and would 
lead into data gaps during the boot cycle. Workaround of course can be 
made for this but do result in some extra BMC loads.

I think if we had a way to get the event service of redfish to see a 
telemetry report was changed, and begin pushing the new results of that 
report, it would go a long way to solving this issue.This option is 
however limited by the external application having to monitor for those 
state changes, then querying for all the sensors, and pushing a new 
report. This could result in a delay causing some initial readings to be 
missed.

The other alternative you hinted at for the sensors would be to have an 
"all" option on the TelemetryReport, that then causes the 
TelemetryService to track the new sensors getting added to the redfish 
API. The report appears to be updated dynamically, with the new sensor 
adds, with no work needed by an external application. This may be the 
more favorable and hardware agnostic approach, but is definitely more 
work from a BMC perspective.

Thanks,
Justin Thaler

On 12/6/19 10:04 AM, Matuszczak, Piotr wrote:
> Hi Justin,
> 
> You've raised an interesting topic. In current implementation, when OpenBMC boots when platform is in power off state, not all sensors are available, thus they won't appear in Redfish tree and you won't be able to configure metric report if you want to use these sensors. If BMC boots during the platform boot, all available sensors will be discovered and exposed in the Redfish tree. After this, when platform is powered off, the sensors, that are not available in this platform state shall be still available in the Redfish tree, but they won't provide valid readings. Unfortunately, such behavior is not standardized, thus if we want to be able to define metric reports when platform is powered off, we will have to make the required sensors behavior part of the design.
> 
> Regards
> Piotr
> 
> 
> -----Original Message-----
> From: Justin Thaler <thalerj@linux.vnet.ibm.com>
> Sent: Wednesday, December 4, 2019 2:28 PM
> To: Neeraj Ladkani <neladk@microsoft.com>; openbmc@lists.ozlabs.org; Matuszczak, Piotr <piotr.matuszczak@intel.com>; vishwa <vishwa@linux.vnet.ibm.com>
> Subject: Re: [EXTERNAL] Telemetry Reports dynamic update
> 
> 
> 
> On 12/4/19 12:43 AM, Neeraj Ladkani wrote:
>> We don’t have telemetryservice in BMC currently so are you currently doing it using webui? Or you have written a app using redfish APIs?  I asked similar question in last week work group meeting. I think it will be useful to have ability to add new report definitions.
> I have an app that uses the current websocket implementation on openBMC and have been reviewing the needs for transitioning the app to redfish when the time comes. One limitation of those is not being able to update the active subscription without tearing it down and starting it again, resulting in a few seconds of missed readings.
> 
> Being able to update the content of the subscription on the fly would be a huge value add I think.
>>
>> -----Original Message-----
>> From: Justin Thaler <thalerj@linux.vnet.ibm.com>
>> Sent: Tuesday, December 3, 2019 3:57 PM
>> To: openbmc@lists.ozlabs.org; Matuszczak, Piotr
>> <piotr.matuszczak@intel.com>; vishwa <vishwa@linux.vnet.ibm.com>;
>> Neeraj Ladkani <neladk@microsoft.com>
>> Subject: [EXTERNAL] Telemetry Reports dynamic update
>>
>> Hi All,
>> 	I've been sort of silently reviewing the designs to telemetry reports design along with subscriptions. As I was reviewing it, I've had a couple of different design points come up and an interesting use case I've wanted to ask about. So here's my scenario:
>>
>> I've set up a telemetry report to the system dynamically by querying the sensors and request the values are updated every second. The system was in a state where it was powered off, so not many sensors were available.
>> After the system is powered on, more sensors are added as different devices are powered up. At this point I'd want to add the additional sensors to my report. I query the system again for all the available sensors and see thermal and power readings for the CPUs. I want updates on these every second just like my initial report.
>>
>> This leads me to the following question:
>> Can I push a new report definition to the TelemetryService and have the event service start providing that new definition without tearing down the subscription first, which would leave a gap in the data?
>>
>> Thanks,
>> Justin Thaler
>>