* ObjectMapper - quantity limitations?
@ 2021-01-27 17:03 Ambrozewicz, Adrian
2021-01-27 17:14 ` Ed Tanous
0 siblings, 1 reply; 4+ messages in thread
From: Ambrozewicz, Adrian @ 2021-01-27 17:03 UTC (permalink / raw)
To: openbmc
Hello,
I'm doing some performance measurements of OpenBMC telemetry subsystem.
I'm using my custom app, which spawns valid D-Bus Sensors, I configure
TelemetryService to monitor them and EventService to push MetricReports
to external server.
I observe certain limitation on my system. Each sensor is mapped as two
objects in ObjectMapper hierarchy. It seems that I am able to correctly
create up to 1500 sensors. When I go above this limit part of the
sensors are not represented in ObjectMapper tree.
Our use-case would most likely require creating even more sensors than
the limit I've reached now. I'm just starting to investigate the issue
and I would be happy if you could give me pointers or ideas what could
be the culprit here.
Regards,
Adrian
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: ObjectMapper - quantity limitations?
2021-01-27 17:03 ObjectMapper - quantity limitations? Ambrozewicz, Adrian
@ 2021-01-27 17:14 ` Ed Tanous
2021-02-09 15:36 ` Ambrozewicz, Adrian
0 siblings, 1 reply; 4+ messages in thread
From: Ed Tanous @ 2021-01-27 17:14 UTC (permalink / raw)
To: Ambrozewicz, Adrian; +Cc: openbmc
On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian
<adrian.ambrozewicz@linux.intel.com> wrote:
>
> Hello,
>
> I'm doing some performance measurements of OpenBMC telemetry subsystem.
> I'm using my custom app, which spawns valid D-Bus Sensors, I configure
> TelemetryService to monitor them and EventService to push MetricReports
> to external server.
Which sensor code are you using?
>
> I observe certain limitation on my system. Each sensor is mapped as two
> objects in ObjectMapper hierarchy. It seems that I am able to correctly
> create up to 1500 sensors. When I go above this limit part of the
> sensors are not represented in ObjectMapper tree.
When I wrote it originally, there were no arbitrary limits on how many
objects the mapper could cache or return, but considering how big your
responses will be, I'm guessing you're hitting the dbus per-message
limit. You don't mention if you're seeing any errors in the
journalctl log from either the broker or object manager. That might
give you more clues.
>
> Our use-case would most likely require creating even more sensors than
> the limit I've reached now. I'm just starting to investigate the issue
> and I would be happy if you could give me pointers or ideas what could
> be the culprit here.
>
> Regards,
> Adrian
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: ObjectMapper - quantity limitations?
2021-01-27 17:14 ` Ed Tanous
@ 2021-02-09 15:36 ` Ambrozewicz, Adrian
2021-02-10 12:11 ` Andrew Jeffery
0 siblings, 1 reply; 4+ messages in thread
From: Ambrozewicz, Adrian @ 2021-02-09 15:36 UTC (permalink / raw)
To: Ed Tanous; +Cc: openbmc
W dniu 1/27/2021 o 18:14, Ed Tanous pisze:
> On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian
> <adrian.ambrozewicz@linux.intel.com> wrote:
>>
>> Hello,
>>
>> I'm doing some performance measurements of OpenBMC telemetry subsystem.
>> I'm using my custom app, which spawns valid D-Bus Sensors, I configure
>> TelemetryService to monitor them and EventService to push MetricReports
>> to external server.
>
> Which sensor code are you using?
>
My homebrew app, which I run like so fake-sensors -c
"/xyz/openbmc_project/inventory/system/board/AC_Baseboard" --loglevel 1
--sensors {sensors_cnt} --interval {interval_ms} :)
It's just for tests.
>>
>> I observe certain limitation on my system. Each sensor is mapped as two
>> objects in ObjectMapper hierarchy. It seems that I am able to correctly
>> create up to 1500 sensors. When I go above this limit part of the
>> sensors are not represented in ObjectMapper tree.
>
> When I wrote it originally, there were no arbitrary limits on how many
> objects the mapper could cache or return, but considering how big your
> responses will be, I'm guessing you're hitting the dbus per-message
> limit. You don't mention if you're seeing any errors in the
> journalctl log from either the broker or object manager. That might
> give you more clues.
>
I've come back to the topic and I observe two kind of behaviors:
- mapperx crash (pretty rare)
- error "Error getting associations from ...".
The former still requires debugging, but for the latter I have simple
explanation - only one application spawns thousands of sensors and is
bombarding ObjectMapper with InterfaceAdded signals, while receiving
'Get' on Associations interface (and possibly other calls). I've
introduced a workaround - up to 3 retries of do_associations() on
ETIMEDOUT, which allowed mapperx to correctly consume 2000 sensors.
However, performance could be better. On AST2600 listing of 2000 sensors
by ObjectMapper takes 130s, 60s for 1000. Furthermore - it's just local
workaround for bigger problem - D-Bus scalability in embedded system.
It looks like sooner or later we need to figure out more lightweight IPC
for telemetry.
Regards,
Adrian
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: ObjectMapper - quantity limitations?
2021-02-09 15:36 ` Ambrozewicz, Adrian
@ 2021-02-10 12:11 ` Andrew Jeffery
0 siblings, 0 replies; 4+ messages in thread
From: Andrew Jeffery @ 2021-02-10 12:11 UTC (permalink / raw)
To: Ambrozewicz, Adrian, Ed Tanous; +Cc: openbmc
On Wed, 10 Feb 2021, at 02:06, Ambrozewicz, Adrian wrote:
> W dniu 1/27/2021 o 18:14, Ed Tanous pisze:
> > On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian
> > <adrian.ambrozewicz@linux.intel.com> wrote:
> >>
> >> Hello,
> >>
> >> I'm doing some performance measurements of OpenBMC telemetry subsystem.
> >> I'm using my custom app, which spawns valid D-Bus Sensors, I configure
> >> TelemetryService to monitor them and EventService to push MetricReports
> >> to external server.
> >
> > Which sensor code are you using?
> >
> My homebrew app, which I run like so fake-sensors -c
> "/xyz/openbmc_project/inventory/system/board/AC_Baseboard" --loglevel 1
> --sensors {sensors_cnt} --interval {interval_ms} :)
> It's just for tests.
>
> >>
> >> I observe certain limitation on my system. Each sensor is mapped as two
> >> objects in ObjectMapper hierarchy. It seems that I am able to correctly
> >> create up to 1500 sensors. When I go above this limit part of the
> >> sensors are not represented in ObjectMapper tree.
> >
> > When I wrote it originally, there were no arbitrary limits on how many
> > objects the mapper could cache or return, but considering how big your
> > responses will be, I'm guessing you're hitting the dbus per-message
> > limit. You don't mention if you're seeing any errors in the
> > journalctl log from either the broker or object manager. That might
> > give you more clues.
> >
>
> I've come back to the topic and I observe two kind of behaviors:
> - mapperx crash (pretty rare)
> - error "Error getting associations from ...".
>
> The former still requires debugging, but for the latter I have simple
> explanation - only one application spawns thousands of sensors and is
> bombarding ObjectMapper with InterfaceAdded signals, while receiving
> 'Get' on Associations interface (and possibly other calls). I've
> introduced a workaround - up to 3 retries of do_associations() on
> ETIMEDOUT, which allowed mapperx to correctly consume 2000 sensors.
>
> However, performance could be better. On AST2600 listing of 2000 sensors
> by ObjectMapper takes 130s, 60s for 1000. Furthermore - it's just local
> workaround for bigger problem - D-Bus scalability in embedded system.
>
> It looks like sooner or later we need to figure out more lightweight IPC
> for telemetry.
I posted a bit about this on another performance-related thread[1]: I've put together a process-shared-memory shared library that could make these exchanges quite light-weight with less of the usual awkwardness of shared memory. It's still young and needs further work, but might be interesting all the same:
https://github.com/amboar/shmapper#libshmap
[1] https://lore.kernel.org/openbmc/09f8bed7-e19e-4ce9-850f-ae870ae89937@www.fastmail.com/
Andrew
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-02-10 12:19 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-27 17:03 ObjectMapper - quantity limitations? Ambrozewicz, Adrian
2021-01-27 17:14 ` Ed Tanous
2021-02-09 15:36 ` Ambrozewicz, Adrian
2021-02-10 12:11 ` Andrew Jeffery
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.