From: "Ambrozewicz, Adrian" <adrian.ambrozewicz@linux.intel.com>
To: Ed Tanous <edtanous@google.com>
Cc: "openbmc@lists.ozlabs.org" <openbmc@lists.ozlabs.org>
Subject: Re: ObjectMapper - quantity limitations?
Date: Tue, 9 Feb 2021 16:36:21 +0100 [thread overview]
Message-ID: <d531c46e-033f-6cd3-83e2-d331d87c7310@linux.intel.com> (raw)
In-Reply-To: <CAH2-KxDCorEjGSneVsSDWx7AXFvQZ8fJ1zaqf1zOJfndbW9eNA@mail.gmail.com>
W dniu 1/27/2021 o 18:14, Ed Tanous pisze:
> On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian
> <adrian.ambrozewicz@linux.intel.com> wrote:
>>
>> Hello,
>>
>> I'm doing some performance measurements of OpenBMC telemetry subsystem.
>> I'm using my custom app, which spawns valid D-Bus Sensors, I configure
>> TelemetryService to monitor them and EventService to push MetricReports
>> to external server.
>
> Which sensor code are you using?
>
My homebrew app, which I run like so fake-sensors -c
"/xyz/openbmc_project/inventory/system/board/AC_Baseboard" --loglevel 1
--sensors {sensors_cnt} --interval {interval_ms} :)
It's just for tests.
>>
>> I observe certain limitation on my system. Each sensor is mapped as two
>> objects in ObjectMapper hierarchy. It seems that I am able to correctly
>> create up to 1500 sensors. When I go above this limit part of the
>> sensors are not represented in ObjectMapper tree.
>
> When I wrote it originally, there were no arbitrary limits on how many
> objects the mapper could cache or return, but considering how big your
> responses will be, I'm guessing you're hitting the dbus per-message
> limit. You don't mention if you're seeing any errors in the
> journalctl log from either the broker or object manager. That might
> give you more clues.
>
I've come back to the topic and I observe two kind of behaviors:
- mapperx crash (pretty rare)
- error "Error getting associations from ...".
The former still requires debugging, but for the latter I have simple
explanation - only one application spawns thousands of sensors and is
bombarding ObjectMapper with InterfaceAdded signals, while receiving
'Get' on Associations interface (and possibly other calls). I've
introduced a workaround - up to 3 retries of do_associations() on
ETIMEDOUT, which allowed mapperx to correctly consume 2000 sensors.
However, performance could be better. On AST2600 listing of 2000 sensors
by ObjectMapper takes 130s, 60s for 1000. Furthermore - it's just local
workaround for bigger problem - D-Bus scalability in embedded system.
It looks like sooner or later we need to figure out more lightweight IPC
for telemetry.
Regards,
Adrian
next prev parent reply other threads:[~2021-02-09 15:38 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-27 17:03 ObjectMapper - quantity limitations? Ambrozewicz, Adrian
2021-01-27 17:14 ` Ed Tanous
2021-02-09 15:36 ` Ambrozewicz, Adrian [this message]
2021-02-10 12:11 ` Andrew Jeffery
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d531c46e-033f-6cd3-83e2-d331d87c7310@linux.intel.com \
--to=adrian.ambrozewicz@linux.intel.com \
--cc=edtanous@google.com \
--cc=openbmc@lists.ozlabs.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).