Re: Thoughts on performance profiling and tools for OpenBMC

[Joseph Reynolds <jrey@linux.ibm.com>]

On 3/22/21 5:05 PM, Sui Chen wrote:
> Hello OpenBMC Mailing List,
>
> This email is to discuss some thoughts and work in progress regarding
> the performance of BMC. We are aware performance has been brought up a
> few times in the past, so this document covers and keeps track of some
> recent work. The following is written according to the design doc
> format, but might still have some way to go before becoming a more
> concrete "set of benchmarks for OpenBMC". As such, any feedback is
> appreciated. Thanks for reading this!

Sui,

I believe there are tie-ins between performance and security.  For 
example, if a BMC user can cause very bad performance or cause the BMC 
to crash, the BMC will not be able to perform its primary function.  
That constitutes a [denial of service][], a security issue.  So I am 
interested in the outcome of BMC performance profiling (but don't have 
resources to contribute).

More specifically, I believe there are tie-ins between the performance 
profiling work and the threat modeling work.  Threat modeling needs an 
architectural model of the interfaces within the BMC, for example, the 
D-Bus and daemon layers.  The [security working group][] has started 
modeling these interfaces, but making progress is hard (search for 
"model" or "threat model").  I understand performance work also needs 
similar models.  I am interested to see any architectural work you have 
in this area.

Thank you!

Joseph

[denial of service]: https://en.wikipedia.org/wiki/Denial-of-service_attack
[security working group]: 
https://github.com/openbmc/openbmc/wiki/Security-working-group


> [ Problem Description ]
>
> Writing benchmarks and studying profiling results is not only good for
> learning the basic APIs and constructs, but also sometimes useful for
> debugging complicated interactions between multiple moving parts of
> the system.
>
> When developers worked on devices with similar specs as BMCs, such as
> smartphones from a few years back, they got performance profiling
> support from developer tools.
>
> BMCs have many interesting aspects involving kernel drivers, hardware
> interfaces, multi-threading, modern programming language features,
> open-source development all packed together into very tight hardware
> and software constraints and a build workflow that compiles code from
> scratch. During debugging, many steps may be needed to recreate the
> scene where performance-related problems arise. Having benchmarks in
> this scenario makes the process easier.
>
> As BMC becomes more versatile and runs more workloads, performance
> issues may become more imminent.
>
> [ Background and References]
>
> 1. BMC performance problems are asked and encountered, and they may be
> helped by benchmarks and tools. Related posts
>     - “ObjectMapper - quantity limitations?” [1]
>     - “dbus-broker caused the system OOM issue” [2]
>     - “Issue about (polling every second) makes Entity Manager get stuck” [3]
>     - “Performance implication of Sensor Value PropertiesChanged Events” [4]
>
> 2. People have started to find solutions for existing and potential
> problems. Examples are:
>     - io_uring vs epoll [5]
>     - shmapper [6]
>
> 3. BMC workloads have their own characteristics, namely, the extensive
> use of DBus, and the numerous I/O buses, among many others. Some of
> these may not have been captured by existing benchmarks on Linux.
> These reasons might justify spending efforts on making a BMC-specific
> set of benchmarks.
>
> 4. There have been proposals for adding performance testing to the CI
> [9]. A baseline, a way to measure performance are needed. This
> document tries to partially discuss the measurement question.
>
> [ Requirements ]
>
> The benchmarks and tools should report basic metrics such as latency
> and throughput. The performance profiling overhead should not distort
> performance results.
>
> The contents of the benchmark can evolve quickly to keep itself
> up-to-date with the rest of the BMC ecosystem, which also evolves
> quickly. This may be comparable to unit tests that are aimed at
> getting code coverage for incremental additions to the code base. This
> may also be comparable to hardware manufacturers updating their
> drivers with performance tuning parameters for newly released
> software.
>
> Benchmarks and results should be easy to learn and use, help newcomers
> learn the basics, and aid seasoned developers where needed.
>
>
> [ Proposed Design ]
>
> 1. Continue the previous effort [7] on a sensor-reading performance
> benchmark for the BMC. This will naturally lead to investigation into
> the lower levels such as I2C and async processing.
>
> 2. Try the community’s ideas on performance optimization in benchmarks
> and measure performance difference. If an optimization generates
> performance gain, attempt to land it in OpenBMC code.
>
> 3. Distill ideas and observations into performance tools. For example,
> enhance or expand the existing DBus visualizer tool [8].
>
> 4. Repeat the process in other areas of BMC performance, such as web
> request processing.
>
> [ Alternatives Considered ]
>
> Rather than benchmarking real hardware, it might be possible to
> directly measure a cycle-accurate full-system timing simulator (such
> as GEM5). This approach might be subject to relatively slow simulation
> speed compared to running on real hardware. Also, device support may
> also affect the feasibility of certain experiments. As such, writing
> benchmarks and running them on real hardware might be more feasible in
> the short term.
>
> [ References ]
>
> [1] https://lists.ozlabs.org/pipermail/openbmc/2021-February/024978.html
> [2] https://lists.ozlabs.org/pipermail/openbmc/2021-February/024895.html
> [3] https://lists.ozlabs.org/pipermail/openbmc/2021-February/024914.html
> [4] https://lists.ozlabs.org/pipermail/openbmc/2021-February/024889.html
> [5] https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.6-IO-uring-Tests
> [6] https://lists.ozlabs.org/pipermail/openbmc/2021-February/024908.html
> [7] https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-tools/+/35387
> [8] https://github.com/openbmc/webui-vue/issues/41
> [9] https://github.com/ibm-openbmc/dev/issues/73