From: Sui Chen <suichen6@gmail.com>
To: "Fran Hsu (徐誌謙)" <Fran.Hsu@quantatw.com>
Cc: "openbmc@lists.ozlabs.org" <openbmc@lists.ozlabs.org>
Subject: Re: dbus-broker caused the system OOM issue.
Date: Wed, 3 Feb 2021 09:24:46 -0800 [thread overview]
Message-ID: <CAFaEeaG3Bt2DJnRHbKVQrYK94EfxhWjOmBfNrWvswj4fN8xW6A@mail.gmail.com> (raw)
In-Reply-To: <TYZPR04MB43666E803D310720362F0CD4F3B59@TYZPR04MB4366.apcprd04.prod.outlook.com>
On Mon, Feb 1, 2021 at 5:40 PM Fran Hsu (徐誌謙) <Fran.Hsu@quantatw.com> wrote:
>
> Hello,
> Sometimes my BMC will run out of memory so that I try to find out which process caused the OOM issue.
> From the htop information, it looks like the dbus-broker uses huge memory space.
> What I'm doing is to send the ipmi sdr list command from Host OS thru the kcs interface every 2 seconds.
> In the begging, I could see the dbus-broker uses up to 400MB and then reduced to less than 100MB.
> But after some hours I could see the memory usage keep increasing and the dbus-broker caused system OOM eventually.
>
> 1 [|||||||||||||||||||||||||||||||| 67.1%] Tasks: 104, 20 thr; 2 running
> 2 [|||||||||||||||||||||||||||||| 60.4%] Load average: 3.12 3.34 3.54
> Mem[|||||||||||||||||||||||||||||||||||||969M/1003M] Uptime: 12:59:12
> Swp[ 0K/0K]
>
> PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
> 281 messagebu 20 0 870M 868M 376 S 0.6 86.6 40:54.32 dbus-broker --log 4 --controller 9 --machine-id 932c
> 265 messagebu 20 0 259M 284 4 S 0.0 0.0 0:00.34 /usr/bin/dbus-broker-launch --scope system --audit
>
> Version:
> dbus-broker :25-r0
>
> I'd appreciate any comments or feedback.
>
> Thanks,
> Fran Hsu
> E-Mail : Fran.Hsu@QuantaTW.com
> Tel: +886-3-327-2345 Ext: 16935
>
>
Hello Fran,
I would do a DBus message capture and use it to examine: 1) what
method calls, signals and errors happened on the system, and 2)
whether there are method calls that take too long to complete, or
never return due to some errors.
The capture may be done using a command like "busctl capture >
dbus_capture.pcap"; the command monitors and dumps the DBus messages
to the output file similarly to Wireshark's capture mode. After
transferring the output file from the BMC to the host, dbus-pcap
(https://github.com/openbmc/openbmc-tools/tree/master/dbus-pcap) or a
work-in-progress visual tool
(https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-tools/+/34263)
may be used to examine it to see if there are method calls that take a
very long time to complete or not return at all.
Hope this helps and thanks,
Sui
prev parent reply other threads:[~2021-02-03 17:26 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-02 1:38 dbus-broker caused the system OOM issue Fran Hsu (徐誌謙)
2021-02-03 17:24 ` Sui Chen [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAFaEeaG3Bt2DJnRHbKVQrYK94EfxhWjOmBfNrWvswj4fN8xW6A@mail.gmail.com \
--to=suichen6@gmail.com \
--cc=Fran.Hsu@quantatw.com \
--cc=openbmc@lists.ozlabs.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).