Re: Thoughts on performance profiling and tools for OpenBMC

[Andrew Geissler <geissonator@gmail.com>]

> On Mar 22, 2021, at 5:05 PM, Sui Chen <suichen@google.com> wrote:
> 
<snip>
> 
> [ Proposed Design ]
> 
> 1. Continue the previous effort [7] on a sensor-reading performance
> benchmark for the BMC. This will naturally lead to investigation into
> the lower levels such as I2C and async processing.
> 
> 2. Try the community’s ideas on performance optimization in benchmarks
> and measure performance difference. If an optimization generates
> performance gain, attempt to land it in OpenBMC code.
> 
> 3. Distill ideas and observations into performance tools. For example,
> enhance or expand the existing DBus visualizer tool [8].
> 
> 4. Repeat the process in other areas of BMC performance, such as web
> request processing.

I had to workaround a lot of performance issues in our first AST2500 
based systems. A lot of the issues were early in the boot of the BMC
when systemd was starting up all of the different services in parallel
and things like mapper were introspecting all new D-Bus objects 
showing up on the bus.

Moving from python to c++ applications helped a lot. Changing 
application nice levels was not helpful (too many d-bus commands
between apps so if one had a higher priority like mapper it would
timeout waiting for lower priority apps).

AndrewJ and I tried to track some of the issues and tools out on
this wiki:
https://github.com/openbmc/openbmc/wiki/Performance-Profiling-in-OpenBMC

We’ve gotten a bit of a reprieve with our move to the AST2600 but
it’s only a matter of time :)

I’m always a fan on trying to improve existing tools vs. rolling our
own but recognize that’s not always an option.

I’m all for anything and everything we can do in this area! Thanks
for taking the initiative Sui.  

Andrew