An IPMI Request Visualization Tool

An IPMI Request Visualization Tool

[Sui Chen]

[-- Attachment #1.1: Type: text/plain, Size: 1183 bytes --]

Hello, OpenBMC Mailing List,

We're working on a GUI tool that allows the user to capture and visualize
BMC IPMI requests.

The tool accepts 2 kinds of inputs:
 1) Load a DBus traffic dump file generated by dbus-monitor on the BMC
 2) Capture IPMI requests on the BMC -- the tool connects to the BMC
console by launching a program of the the user's choice (example: ssh or
telnet), then starts "dbus-monitor" on the BMC and checks its outputs for DBus
calls on the IPMI daemon and then draws the IPMI requests on the canvas
accordingly.

In both ways, this tool operates on data that originate on the BMC (so it's
different from host-side IPMI capture tools.)

The tool makes minimal use of electron.js and should be able to run on
multiple operating systems. We hope this tool might be useful for those who
work on OpenBMC.

What I'm not sure is: do I need to have this accessible on a repo managed
by my Company first so OpenBMC can review it, or can I request to create a
repo for this tool under openbmc-tools now --- provided that OpenBMC is
okay with adding this tool?

A screenshot is attached to illustrate what it does:
[image: scrnshot_compressed.png]


Thanks a lot!

Sui

[-- Attachment #1.2: Type: text/html, Size: 2028 bytes --]

[-- Attachment #2: scrnshot_compressed.png --]
[-- Type: image/png, Size: 36637 bytes --]

Re: An IPMI Request Visualization Tool

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On Wed, May 13, 2020 at 03:38:47PM -0700, Sui Chen wrote:
> We're working on a GUI tool that allows the user to capture and visualize
> BMC IPMI requests.
> 
> The tool accepts 2 kinds of inputs:
>  1) Load a DBus traffic dump file generated by dbus-monitor on the BMC
>  2) Capture IPMI requests on the BMC -- the tool connects to the BMC
> console by launching a program of the the user's choice (example: ssh or
> telnet), then starts "dbus-monitor" on the BMC and checks its outputs for DBus
> calls on the IPMI daemon and then draws the IPMI requests on the canvas
> accordingly.
 
> A screenshot is attached to illustrate what it does:
> [image: scrnshot_compressed.png]

Neat.  I wonder if we can use this to analyse dbus messages with high
amounts of latency (especially once that are infrequently latent) in
order to better target which processes we should move towards async
calls.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: An IPMI Request Visualization Tool

[Andrew Jeffery]

On Thu, 14 May 2020, at 23:27, Patrick Williams wrote:
> On Wed, May 13, 2020 at 03:38:47PM -0700, Sui Chen wrote:
> > We're working on a GUI tool that allows the user to capture and visualize
> > BMC IPMI requests.
> > 
> > The tool accepts 2 kinds of inputs:
> >  1) Load a DBus traffic dump file generated by dbus-monitor on the BMC
> >  2) Capture IPMI requests on the BMC -- the tool connects to the BMC
> > console by launching a program of the the user's choice (example: ssh or
> > telnet), then starts "dbus-monitor" on the BMC and checks its outputs for DBus
> > calls on the IPMI daemon and then draws the IPMI requests on the canvas
> > accordingly.
>  
> > A screenshot is attached to illustrate what it does:
> > [image: scrnshot_compressed.png]
> 
> Neat.  I wonder if we can use this to analyse dbus messages with high
> amounts of latency (especially once that are infrequently latent) in
> order to better target which processes we should move towards async
> calls.

For what it's worth I have a script for parsing D-Bus pcaps:

https://github.com/openbmc/openbmc-tools/tree/master/amboar/obmc-scripts/dbus-pcap

It's helped me tracked down unexpected latencies and other weird
behaviours in the past.

I've just pushed a change adding a README to introduce the capabilities
and provide examples of usage and output:

https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-tools/+/32583

Cheers,

Andrew

Re: An IPMI Request Visualization Tool

[Sui Chen]

Hello Andrew and Patrick,

Thanks for your interest! I'm also using dbus-pcap to track certain
issues on the BMC recently, and would like to add support for all DBus
messages to the visualization tool, making it somewhat resemble a GUI
version of dbus-pcap.
The goal would be to be able to use this tool to investigate both DBus
and IPMI. The way I plan to use it would be more similar to how I use
GPUView (full-system timeline rather than inspecting individual
packets)

If you ask how this user interface might differ from the already
existing dbus visualizers such as bustle, my answer would be: it will
present information in a way that's more relevant to the BMC, putting
a bit more focus on BMC-specific DBus messages, such as HWMon and
RedFish DBus messages, to present information in a high signal-noise
ratio way.

If this makes sense to you, can I know what might be a better way to proceed:
1) Create a change in openbmc-tools? or,
2) Create a new repository under openbmc? or
3) Create a personal/organization-managed repository and create a pull request?

Thanks,
Sui


On Mon, May 18, 2020 at 6:38 PM Andrew Jeffery <andrew@aj.id.au> wrote:
>
>
>
> On Thu, 14 May 2020, at 23:27, Patrick Williams wrote:
> > On Wed, May 13, 2020 at 03:38:47PM -0700, Sui Chen wrote:
> > > We're working on a GUI tool that allows the user to capture and visualize
> > > BMC IPMI requests.
> > >
> > > The tool accepts 2 kinds of inputs:
> > >  1) Load a DBus traffic dump file generated by dbus-monitor on the BMC
> > >  2) Capture IPMI requests on the BMC -- the tool connects to the BMC
> > > console by launching a program of the the user's choice (example: ssh or
> > > telnet), then starts "dbus-monitor" on the BMC and checks its outputs for DBus
> > > calls on the IPMI daemon and then draws the IPMI requests on the canvas
> > > accordingly.
> >
> > > A screenshot is attached to illustrate what it does:
> > > [image: scrnshot_compressed.png]
> >
> > Neat.  I wonder if we can use this to analyse dbus messages with high
> > amounts of latency (especially once that are infrequently latent) in
> > order to better target which processes we should move towards async
> > calls.
>
> For what it's worth I have a script for parsing D-Bus pcaps:
>
> https://github.com/openbmc/openbmc-tools/tree/master/amboar/obmc-scripts/dbus-pcap
>
> It's helped me tracked down unexpected latencies and other weird
> behaviours in the past.
>
> I've just pushed a change adding a README to introduce the capabilities
> and provide examples of usage and output:
>
> https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-tools/+/32583
>
> Cheers,
>
> Andrew

Re: An IPMI Request Visualization Tool

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]


Sui Chen <suichen6@gmail.com> wrote:
    > Thanks for your interest! I'm also using dbus-pcap to track certain
    > issues on the BMC recently, and would like to add support for all DBus
    > messages to the visualization tool, making it somewhat resemble a GUI
    > version of dbus-pcap.  The goal would be to be able to use this tool to
    > investigate both DBus and IPMI. The way I plan to use it would be more
    > similar to how I use GPUView (full-system timeline rather than
    > inspecting individual packets)

Interestingly, I was not that clearly aware of dbus-pcap :-)
I ought to know more, as the lead libpcap maintainer.
Is this visualization tool part of openbmc, or is it a generic dbus
visualization tool?

We recently brought rpcapd into the mix; it can be invoked via ssh.
I wonder if that might help you as a debug tool?

    > If you ask how this user interface might differ from the already
    > existing dbus visualizers such as bustle, my answer would be: it will
    > present information in a way that's more relevant to the BMC, putting a
    > bit more focus on BMC-specific DBus messages, such as HWMon and RedFish
    > DBus messages, to present information in a high signal-noise ratio way.

Would this need to run on the BMC itself?

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: An IPMI Request Visualization Tool

[Brandon Wyman]

On 2020-05-21 17:39, Michael Richardson wrote:
> Sui Chen <suichen6@gmail.com> wrote:
>      > Thanks for your interest! I'm also using dbus-pcap to track certain
>      > issues on the BMC recently, and would like to add support for all DBus
>      > messages to the visualization tool, making it somewhat resemble a GUI
>      > version of dbus-pcap.  The goal would be to be able to use this tool to
>      > investigate both DBus and IPMI. The way I plan to use it would be more
>      > similar to how I use GPUView (full-system timeline rather than
>      > inspecting individual packets)
>
> Interestingly, I was not that clearly aware of dbus-pcap :-)
> I ought to know more, as the lead libpcap maintainer.
> Is this visualization tool part of openbmc, or is it a generic dbus
> visualization tool?
https://github.com/openbmc/openbmc-tools/tree/master/amboar/obmc-scripts/dbus-pcap
>
> We recently brought rpcapd into the mix; it can be invoked via ssh.
> I wonder if that might help you as a debug tool?
>
>      > If you ask how this user interface might differ from the already
>      > existing dbus visualizers such as bustle, my answer would be: it will
>      > present information in a way that's more relevant to the BMC, putting a
>      > bit more focus on BMC-specific DBus messages, such as HWMon and RedFish
>      > DBus messages, to present information in a high signal-noise ratio way.
>
> Would this need to run on the BMC itself?
>
> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        | network architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [
>

Re: An IPMI Request Visualization Tool

[Vijay Khemka]

On 5/21/20, 9:45 AM, "openbmc on behalf of Sui Chen" <openbmc-bounces+vijaykhemka=fb.com@lists.ozlabs.org on behalf of suichen6@gmail.com> wrote:

    Hello Andrew and Patrick,
    
    Thanks for your interest! I'm also using dbus-pcap to track certain
    issues on the BMC recently, and would like to add support for all DBus
    messages to the visualization tool, making it somewhat resemble a GUI
    version of dbus-pcap.
    The goal would be to be able to use this tool to investigate both DBus
    and IPMI. The way I plan to use it would be more similar to how I use
    GPUView (full-system timeline rather than inspecting individual
    packets)
    
    If you ask how this user interface might differ from the already
    existing dbus visualizers such as bustle, my answer would be: it will
    present information in a way that's more relevant to the BMC, putting
    a bit more focus on BMC-specific DBus messages, such as HWMon and
    RedFish DBus messages, to present information in a high signal-noise
    ratio way.
    
    If this makes sense to you, can I know what might be a better way to proceed:
    1) Create a change in openbmc-tools? or,
    2) Create a new repository under openbmc? or
    3) Create a personal/organization-managed repository and create a pull request?

You should be able to add it under openbmc-tools directly rather than creating 
a new repo or creating personal directory.
    
    Thanks,
    Sui
    
    
    On Mon, May 18, 2020 at 6:38 PM Andrew Jeffery <andrew@aj.id.au> wrote:
    >
    >
    >
    > On Thu, 14 May 2020, at 23:27, Patrick Williams wrote:
    > > On Wed, May 13, 2020 at 03:38:47PM -0700, Sui Chen wrote:
    > > > We're working on a GUI tool that allows the user to capture and visualize
    > > > BMC IPMI requests.
    > > >
    > > > The tool accepts 2 kinds of inputs:
    > > >  1) Load a DBus traffic dump file generated by dbus-monitor on the BMC
    > > >  2) Capture IPMI requests on the BMC -- the tool connects to the BMC
    > > > console by launching a program of the the user's choice (example: ssh or
    > > > telnet), then starts "dbus-monitor" on the BMC and checks its outputs for DBus
    > > > calls on the IPMI daemon and then draws the IPMI requests on the canvas
    > > > accordingly.
    > >
    > > > A screenshot is attached to illustrate what it does:
    > > > [image: scrnshot_compressed.png]
    > >
    > > Neat.  I wonder if we can use this to analyse dbus messages with high
    > > amounts of latency (especially once that are infrequently latent) in
    > > order to better target which processes we should move towards async
    > > calls.
    >
    > For what it's worth I have a script for parsing D-Bus pcaps:
    >
    > https://github.com/openbmc/openbmc-tools/tree/master/amboar/obmc-scripts/dbus-pcap
    >
    > It's helped me tracked down unexpected latencies and other weird
    > behaviours in the past.
    >
    > I've just pushed a change adding a README to introduce the capabilities
    > and provide examples of usage and output:
    >
    > https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.openbmc-2Dproject.xyz_c_openbmc_openbmc-2Dtools_-2B_32583&d=DwIBaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=v9MU0Ki9pWnTXCWwjHPVgpnCR80vXkkcrIaqU7USl5g&m=yvJlaLs3OjdhZWtgqOz5j-3oLG6izLybS-48PNfv-Bo&s=LIuidUUZaTV-K4Uuu79r2C5a4xffwdEd-4Zj-oJjiAo&e= 
    >
    > Cheers,
    >
    > Andrew
    

Re: An IPMI Request Visualization Tool

[Andrew Jeffery]

On Fri, 22 May 2020, at 08:09, Michael Richardson wrote:
> 
> Sui Chen <suichen6@gmail.com> wrote:
>     > Thanks for your interest! I'm also using dbus-pcap to track certain
>     > issues on the BMC recently, and would like to add support for all DBus
>     > messages to the visualization tool, making it somewhat resemble a GUI
>     > version of dbus-pcap.  The goal would be to be able to use this tool to
>     > investigate both DBus and IPMI. The way I plan to use it would be more
>     > similar to how I use GPUView (full-system timeline rather than
>     > inspecting individual packets)
> 
> Interestingly, I was not that clearly aware of dbus-pcap :-)
> I ought to know more, as the lead libpcap maintainer.
> Is this visualization tool part of openbmc, or is it a generic dbus
> visualization tool?

It's not really a visualisation tool so much as a script that will interpret the D-Bus-specifics of a D-Bus pcap. It's a commandline script that provides filtering based on D-Bus match specifications. You can capture all traffic on the system bus in any systemd-based system with `busctl capture > /tmp/my.pcap` and then run `dbus-pcap my.pcap` to dump the packet contents.

> 
> We recently brought rpcapd into the mix; it can be invoked via ssh.
> I wonder if that might help you as a debug tool?

I'm not sure, I'm not familiar with rpcapd. I'll have a google.

> 
>     > If you ask how this user interface might differ from the already
>     > existing dbus visualizers such as bustle, my answer would be: it will
>     > present information in a way that's more relevant to the BMC, putting a
>     > bit more focus on BMC-specific DBus messages, such as HWMon and RedFish
>     > DBus messages, to present information in a high signal-noise ratio way.
> 
> Would this need to run on the BMC itself?

Hopefully not given `busctl capture` ?

Andrew

Re: An IPMI Request Visualization Tool

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 1803 bytes --]


Andrew Jeffery <andrew@aj.id.au> wrote:
    >> Interestingly, I was not that clearly aware of dbus-pcap :-) I ought
    >> to know more, as the lead libpcap maintainer.  Is this visualization
    >> tool part of openbmc, or is it a generic dbus visualization tool?

    > It's not really a visualisation tool so much as a script that will
    > interpret the D-Bus-specifics of a D-Bus pcap. It's a commandline
    > script that provides filtering based on D-Bus match specifications. You
    > can capture all traffic on the system bus in any systemd-based system
    > with `busctl capture > /tmp/my.pcap` and then run `dbus-pcap my.pcap`
    > to dump the packet contents.

Is this something that you'd like to be able to invoke remotely in order to
get diagnostics/debugging info?

    >> We recently brought rpcapd into the mix; it can be invoked via ssh.  I
    >> wonder if that might help you as a debug tool?

    > I'm not sure, I'm not familiar with rpcapd. I'll have a google.

https://www.ca.tcpdump.org/manpages/rpcapd.8.html

    >> > If you ask how this user interface might differ from the already >
    >> existing dbus visualizers such as bustle, my answer would be: it will
    >> > present information in a way that's more relevant to the BMC,
    >> putting a > bit more focus on BMC-specific DBus messages, such as
    >> HWMon and RedFish > DBus messages, to present information in a high
    >> signal-noise ratio way.
    >>
    >> Would this need to run on the BMC itself?

    > Hopefully not given `busctl capture` ?

Understood.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: An IPMI Request Visualization Tool

[Andrew Jeffery]

On Mon, 8 Jun 2020, at 12:37, Michael Richardson wrote:
> 
> Andrew Jeffery <andrew@aj.id.au> wrote:
>     >> Interestingly, I was not that clearly aware of dbus-pcap :-) I ought
>     >> to know more, as the lead libpcap maintainer.  Is this visualization
>     >> tool part of openbmc, or is it a generic dbus visualization tool?
> 
>     > It's not really a visualisation tool so much as a script that will
>     > interpret the D-Bus-specifics of a D-Bus pcap. It's a commandline
>     > script that provides filtering based on D-Bus match specifications. You
>     > can capture all traffic on the system bus in any systemd-based system
>     > with `busctl capture > /tmp/my.pcap` and then run `dbus-pcap my.pcap`
>     > to dump the packet contents.
> 
> Is this something that you'd like to be able to invoke remotely in order to
> get diagnostics/debugging info?

I haven't needed to, but others might.

> 
>     >> We recently brought rpcapd into the mix; it can be invoked via ssh.  I
>     >> wonder if that might help you as a debug tool?
> 
>     > I'm not sure, I'm not familiar with rpcapd. I'll have a google.
> 
> https://www.ca.tcpdump.org/manpages/rpcapd.8.html

Yeah, I found it :) Thanks.

Andrew