ObjectMapper - quantity limitations?

ObjectMapper - quantity limitations?

[Ambrozewicz, Adrian]

Hello,

I'm doing some performance measurements of OpenBMC telemetry subsystem. 
I'm using my custom app, which spawns valid D-Bus Sensors, I configure 
TelemetryService to monitor them and EventService to push MetricReports 
to external server.

I observe certain limitation on my system. Each sensor is mapped as two 
objects in ObjectMapper hierarchy. It seems that I am able to correctly 
create up to 1500 sensors. When I go above this limit part of the 
sensors are not represented in ObjectMapper tree.

Our use-case would most likely require creating even more sensors than 
the limit I've reached now. I'm just starting to investigate the issue 
and I would be happy if you could give me pointers or ideas what could 
be the culprit here.

Regards,
Adrian

Re: ObjectMapper - quantity limitations?

[Ed Tanous]

On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian
<adrian.ambrozewicz@linux.intel.com> wrote:
>
> Hello,
>
> I'm doing some performance measurements of OpenBMC telemetry subsystem.
> I'm using my custom app, which spawns valid D-Bus Sensors, I configure
> TelemetryService to monitor them and EventService to push MetricReports
> to external server.

Which sensor code are you using?

>
> I observe certain limitation on my system. Each sensor is mapped as two
> objects in ObjectMapper hierarchy. It seems that I am able to correctly
> create up to 1500 sensors. When I go above this limit part of the
> sensors are not represented in ObjectMapper tree.

When I wrote it originally, there were no arbitrary limits on how many
objects the mapper could cache or return, but considering how big your
responses will be, I'm guessing you're hitting the dbus per-message
limit.  You don't mention if you're seeing any errors in the
journalctl log from either the broker or object manager.  That might
give you more clues.

>
> Our use-case would most likely require creating even more sensors than
> the limit I've reached now. I'm just starting to investigate the issue
> and I would be happy if you could give me pointers or ideas what could
> be the culprit here.
>
> Regards,
> Adrian

Re: ObjectMapper - quantity limitations?

[Ambrozewicz, Adrian]

W dniu 1/27/2021 o 18:14, Ed Tanous pisze:
> On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian
> <adrian.ambrozewicz@linux.intel.com> wrote:
>>
>> Hello,
>>
>> I'm doing some performance measurements of OpenBMC telemetry subsystem.
>> I'm using my custom app, which spawns valid D-Bus Sensors, I configure
>> TelemetryService to monitor them and EventService to push MetricReports
>> to external server.
> 
> Which sensor code are you using?
> 
My homebrew app, which I run like so fake-sensors -c 
"/xyz/openbmc_project/inventory/system/board/AC_Baseboard" --loglevel 1 
--sensors {sensors_cnt} --interval {interval_ms} :)
It's just for tests.

>>
>> I observe certain limitation on my system. Each sensor is mapped as two
>> objects in ObjectMapper hierarchy. It seems that I am able to correctly
>> create up to 1500 sensors. When I go above this limit part of the
>> sensors are not represented in ObjectMapper tree.
> 
> When I wrote it originally, there were no arbitrary limits on how many
> objects the mapper could cache or return, but considering how big your
> responses will be, I'm guessing you're hitting the dbus per-message
> limit.  You don't mention if you're seeing any errors in the
> journalctl log from either the broker or object manager.  That might
> give you more clues.
> 

I've come back to the topic and I observe two kind of behaviors:
- mapperx crash (pretty rare)
- error "Error getting associations from ...".

The former still requires debugging, but for the latter I have simple 
explanation - only one application spawns thousands of sensors and is 
bombarding ObjectMapper with InterfaceAdded signals, while receiving 
'Get' on Associations interface (and possibly other calls). I've 
introduced a workaround - up to 3 retries of do_associations() on 
ETIMEDOUT, which allowed mapperx to correctly consume 2000 sensors.

However, performance could be better. On AST2600 listing of 2000 sensors 
by ObjectMapper takes 130s, 60s for 1000. Furthermore - it's just local 
workaround for bigger problem - D-Bus scalability in embedded system.

It looks like sooner or later we need to figure out more lightweight IPC 
for telemetry.

Regards,
Adrian

Re: ObjectMapper - quantity limitations?

[Andrew Jeffery]

On Wed, 10 Feb 2021, at 02:06, Ambrozewicz, Adrian wrote:
> W dniu 1/27/2021 o 18:14, Ed Tanous pisze:
> > On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian
> > <adrian.ambrozewicz@linux.intel.com> wrote:
> >>
> >> Hello,
> >>
> >> I'm doing some performance measurements of OpenBMC telemetry subsystem.
> >> I'm using my custom app, which spawns valid D-Bus Sensors, I configure
> >> TelemetryService to monitor them and EventService to push MetricReports
> >> to external server.
> > 
> > Which sensor code are you using?
> > 
> My homebrew app, which I run like so fake-sensors -c 
> "/xyz/openbmc_project/inventory/system/board/AC_Baseboard" --loglevel 1 
> --sensors {sensors_cnt} --interval {interval_ms} :)
> It's just for tests.
> 
> >>
> >> I observe certain limitation on my system. Each sensor is mapped as two
> >> objects in ObjectMapper hierarchy. It seems that I am able to correctly
> >> create up to 1500 sensors. When I go above this limit part of the
> >> sensors are not represented in ObjectMapper tree.
> > 
> > When I wrote it originally, there were no arbitrary limits on how many
> > objects the mapper could cache or return, but considering how big your
> > responses will be, I'm guessing you're hitting the dbus per-message
> > limit.  You don't mention if you're seeing any errors in the
> > journalctl log from either the broker or object manager.  That might
> > give you more clues.
> > 
> 
> I've come back to the topic and I observe two kind of behaviors:
> - mapperx crash (pretty rare)
> - error "Error getting associations from ...".
> 
> The former still requires debugging, but for the latter I have simple 
> explanation - only one application spawns thousands of sensors and is 
> bombarding ObjectMapper with InterfaceAdded signals, while receiving 
> 'Get' on Associations interface (and possibly other calls). I've 
> introduced a workaround - up to 3 retries of do_associations() on 
> ETIMEDOUT, which allowed mapperx to correctly consume 2000 sensors.
> 
> However, performance could be better. On AST2600 listing of 2000 sensors 
> by ObjectMapper takes 130s, 60s for 1000. Furthermore - it's just local 
> workaround for bigger problem - D-Bus scalability in embedded system.
> 
> It looks like sooner or later we need to figure out more lightweight IPC 
> for telemetry.

I posted a bit about this on another performance-related thread[1]: I've put together a process-shared-memory shared library that could make these exchanges quite light-weight with less of the usual awkwardness of shared memory. It's still young and needs further work, but might be interesting all the same:

https://github.com/amboar/shmapper#libshmap

[1] https://lore.kernel.org/openbmc/09f8bed7-e19e-4ce9-850f-ae870ae89937@www.fastmail.com/

Andrew