* ObjectMapper - quantity limitations? @ 2021-01-27 17:03 Ambrozewicz, Adrian 2021-01-27 17:14 ` Ed Tanous 0 siblings, 1 reply; 4+ messages in thread From: Ambrozewicz, Adrian @ 2021-01-27 17:03 UTC (permalink / raw) To: openbmc Hello, I'm doing some performance measurements of OpenBMC telemetry subsystem. I'm using my custom app, which spawns valid D-Bus Sensors, I configure TelemetryService to monitor them and EventService to push MetricReports to external server. I observe certain limitation on my system. Each sensor is mapped as two objects in ObjectMapper hierarchy. It seems that I am able to correctly create up to 1500 sensors. When I go above this limit part of the sensors are not represented in ObjectMapper tree. Our use-case would most likely require creating even more sensors than the limit I've reached now. I'm just starting to investigate the issue and I would be happy if you could give me pointers or ideas what could be the culprit here. Regards, Adrian ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: ObjectMapper - quantity limitations? 2021-01-27 17:03 ObjectMapper - quantity limitations? Ambrozewicz, Adrian @ 2021-01-27 17:14 ` Ed Tanous 2021-02-09 15:36 ` Ambrozewicz, Adrian 0 siblings, 1 reply; 4+ messages in thread From: Ed Tanous @ 2021-01-27 17:14 UTC (permalink / raw) To: Ambrozewicz, Adrian; +Cc: openbmc On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian <adrian.ambrozewicz@linux.intel.com> wrote: > > Hello, > > I'm doing some performance measurements of OpenBMC telemetry subsystem. > I'm using my custom app, which spawns valid D-Bus Sensors, I configure > TelemetryService to monitor them and EventService to push MetricReports > to external server. Which sensor code are you using? > > I observe certain limitation on my system. Each sensor is mapped as two > objects in ObjectMapper hierarchy. It seems that I am able to correctly > create up to 1500 sensors. When I go above this limit part of the > sensors are not represented in ObjectMapper tree. When I wrote it originally, there were no arbitrary limits on how many objects the mapper could cache or return, but considering how big your responses will be, I'm guessing you're hitting the dbus per-message limit. You don't mention if you're seeing any errors in the journalctl log from either the broker or object manager. That might give you more clues. > > Our use-case would most likely require creating even more sensors than > the limit I've reached now. I'm just starting to investigate the issue > and I would be happy if you could give me pointers or ideas what could > be the culprit here. > > Regards, > Adrian ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: ObjectMapper - quantity limitations? 2021-01-27 17:14 ` Ed Tanous @ 2021-02-09 15:36 ` Ambrozewicz, Adrian 2021-02-10 12:11 ` Andrew Jeffery 0 siblings, 1 reply; 4+ messages in thread From: Ambrozewicz, Adrian @ 2021-02-09 15:36 UTC (permalink / raw) To: Ed Tanous; +Cc: openbmc W dniu 1/27/2021 o 18:14, Ed Tanous pisze: > On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian > <adrian.ambrozewicz@linux.intel.com> wrote: >> >> Hello, >> >> I'm doing some performance measurements of OpenBMC telemetry subsystem. >> I'm using my custom app, which spawns valid D-Bus Sensors, I configure >> TelemetryService to monitor them and EventService to push MetricReports >> to external server. > > Which sensor code are you using? > My homebrew app, which I run like so fake-sensors -c "/xyz/openbmc_project/inventory/system/board/AC_Baseboard" --loglevel 1 --sensors {sensors_cnt} --interval {interval_ms} :) It's just for tests. >> >> I observe certain limitation on my system. Each sensor is mapped as two >> objects in ObjectMapper hierarchy. It seems that I am able to correctly >> create up to 1500 sensors. When I go above this limit part of the >> sensors are not represented in ObjectMapper tree. > > When I wrote it originally, there were no arbitrary limits on how many > objects the mapper could cache or return, but considering how big your > responses will be, I'm guessing you're hitting the dbus per-message > limit. You don't mention if you're seeing any errors in the > journalctl log from either the broker or object manager. That might > give you more clues. > I've come back to the topic and I observe two kind of behaviors: - mapperx crash (pretty rare) - error "Error getting associations from ...". The former still requires debugging, but for the latter I have simple explanation - only one application spawns thousands of sensors and is bombarding ObjectMapper with InterfaceAdded signals, while receiving 'Get' on Associations interface (and possibly other calls). I've introduced a workaround - up to 3 retries of do_associations() on ETIMEDOUT, which allowed mapperx to correctly consume 2000 sensors. However, performance could be better. On AST2600 listing of 2000 sensors by ObjectMapper takes 130s, 60s for 1000. Furthermore - it's just local workaround for bigger problem - D-Bus scalability in embedded system. It looks like sooner or later we need to figure out more lightweight IPC for telemetry. Regards, Adrian ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: ObjectMapper - quantity limitations? 2021-02-09 15:36 ` Ambrozewicz, Adrian @ 2021-02-10 12:11 ` Andrew Jeffery 0 siblings, 0 replies; 4+ messages in thread From: Andrew Jeffery @ 2021-02-10 12:11 UTC (permalink / raw) To: Ambrozewicz, Adrian, Ed Tanous; +Cc: openbmc On Wed, 10 Feb 2021, at 02:06, Ambrozewicz, Adrian wrote: > W dniu 1/27/2021 o 18:14, Ed Tanous pisze: > > On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian > > <adrian.ambrozewicz@linux.intel.com> wrote: > >> > >> Hello, > >> > >> I'm doing some performance measurements of OpenBMC telemetry subsystem. > >> I'm using my custom app, which spawns valid D-Bus Sensors, I configure > >> TelemetryService to monitor them and EventService to push MetricReports > >> to external server. > > > > Which sensor code are you using? > > > My homebrew app, which I run like so fake-sensors -c > "/xyz/openbmc_project/inventory/system/board/AC_Baseboard" --loglevel 1 > --sensors {sensors_cnt} --interval {interval_ms} :) > It's just for tests. > > >> > >> I observe certain limitation on my system. Each sensor is mapped as two > >> objects in ObjectMapper hierarchy. It seems that I am able to correctly > >> create up to 1500 sensors. When I go above this limit part of the > >> sensors are not represented in ObjectMapper tree. > > > > When I wrote it originally, there were no arbitrary limits on how many > > objects the mapper could cache or return, but considering how big your > > responses will be, I'm guessing you're hitting the dbus per-message > > limit. You don't mention if you're seeing any errors in the > > journalctl log from either the broker or object manager. That might > > give you more clues. > > > > I've come back to the topic and I observe two kind of behaviors: > - mapperx crash (pretty rare) > - error "Error getting associations from ...". > > The former still requires debugging, but for the latter I have simple > explanation - only one application spawns thousands of sensors and is > bombarding ObjectMapper with InterfaceAdded signals, while receiving > 'Get' on Associations interface (and possibly other calls). I've > introduced a workaround - up to 3 retries of do_associations() on > ETIMEDOUT, which allowed mapperx to correctly consume 2000 sensors. > > However, performance could be better. On AST2600 listing of 2000 sensors > by ObjectMapper takes 130s, 60s for 1000. Furthermore - it's just local > workaround for bigger problem - D-Bus scalability in embedded system. > > It looks like sooner or later we need to figure out more lightweight IPC > for telemetry. I posted a bit about this on another performance-related thread[1]: I've put together a process-shared-memory shared library that could make these exchanges quite light-weight with less of the usual awkwardness of shared memory. It's still young and needs further work, but might be interesting all the same: https://github.com/amboar/shmapper#libshmap [1] https://lore.kernel.org/openbmc/09f8bed7-e19e-4ce9-850f-ae870ae89937@www.fastmail.com/ Andrew ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-02-10 12:19 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-27 17:03 ObjectMapper - quantity limitations? Ambrozewicz, Adrian 2021-01-27 17:14 ` Ed Tanous 2021-02-09 15:36 ` Ambrozewicz, Adrian 2021-02-10 12:11 ` Andrew Jeffery
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).