Re: [PATCH v2 bpf-next 2/3] selftest/bpf: fmod_ret prog and implement test_overhead as part of bench

[Yonghong Song <yhs@fb.com>]

On 5/11/20 9:22 PM, Andrii Nakryiko wrote:
> On Sat, May 9, 2020 at 10:24 AM Yonghong Song <yhs@fb.com> wrote:
>>
>>
>>
>> On 5/8/20 4:20 PM, Andrii Nakryiko wrote:
>>> Add fmod_ret BPF program to existing test_overhead selftest. Also re-implement
>>> user-space benchmarking part into benchmark runner to compare results.  Results
>>> with ./bench are consistently somewhat lower than test_overhead's, but relative
>>> performance of various types of BPF programs stay consisten (e.g., kretprobe is
>>> noticeably slower).
>>>
>>> run_bench_rename.sh script (in benchs/ directory) was used to produce the
>>> following numbers:
>>>
>>>     base      :    3.975 ± 0.065M/s
>>>     kprobe    :    3.268 ± 0.095M/s
>>>     kretprobe :    2.496 ± 0.040M/s
>>>     rawtp     :    3.899 ± 0.078M/s
>>>     fentry    :    3.836 ± 0.049M/s
>>>     fexit     :    3.660 ± 0.082M/s
>>>     fmodret   :    3.776 ± 0.033M/s
>>>
>>> While running test_overhead gives:
>>>
>>>     task_rename base        4457K events per sec
>>>     task_rename kprobe      3849K events per sec
>>>     task_rename kretprobe   2729K events per sec
>>>     task_rename raw_tp      4506K events per sec
>>>     task_rename fentry      4381K events per sec
>>>     task_rename fexit       4349K events per sec
>>>     task_rename fmod_ret    4130K events per sec
>>
>> Do you where the overhead is and how we could provide options in
>> bench to reduce the overhead so we can achieve similar numbers?
>> For benchmarking, sometimes you really want to see "true"
>> potential of a particular implementation.
> 
> Alright, let's make it an official bench-off... :) And the reason for
> this discrepancy, turns out to be... not atomics at all! But rather a
> single-threaded vs multi-threaded process (well, at least task_rename
> happening from non-main thread, I didn't narrow it down further).

It would be good to find out why and have a scheme (e.g. some kind
of affinity binding) to close the gap.

> Atomics actually make very little difference, which gives me a good
> peace of mind :)
> 
> So, I've built and ran test_overhead (selftest) and bench both as
> multi-threaded and single-threaded apps. Corresponding results match
> almost perfectly. And that's while test_overhead doesn't use atomics
> at all, while bench still does. Then I also ran test_overhead with
> added generics to match bench implementation. There are barely any
> differences, see two last sets of results.
> 
> BTW, selftest results seems bit lower from the ones in original
> commit, probably because I made it run more iterations (like 40 times
> more) to have more stable results.
> 
> So here are the results:
> 
> Single-threaded implementations
> ===============================
> 
> /* bench: single-threaded, atomics */
> base      :    4.622 ± 0.049M/s
> kprobe    :    3.673 ± 0.052M/s
> kretprobe :    2.625 ± 0.052M/s
> rawtp     :    4.369 ± 0.089M/s
> fentry    :    4.201 ± 0.558M/s
> fexit     :    4.309 ± 0.148M/s
> fmodret   :    4.314 ± 0.203M/s
> 
> /* selftest: single-threaded, no atomics */
> task_rename base        4555K events per sec
> task_rename kprobe      3643K events per sec
> task_rename kretprobe   2506K events per sec
> task_rename raw_tp      4303K events per sec
> task_rename fentry      4307K events per sec
> task_rename fexit       4010K events per sec
> task_rename fmod_ret    3984K events per sec
> 
> 
> Multi-threaded implementations
> ==============================
> 
> /* bench: multi-threaded w/ atomics */
> base      :    3.910 ± 0.023M/s
> kprobe    :    3.048 ± 0.037M/s
> kretprobe :    2.300 ± 0.015M/s
> rawtp     :    3.687 ± 0.034M/s
> fentry    :    3.740 ± 0.087M/s
> fexit     :    3.510 ± 0.009M/s
> fmodret   :    3.485 ± 0.050M/s
> 
> /* selftest: multi-threaded w/ atomics */
> task_rename base        3872K events per sec
> task_rename kprobe      3068K events per sec
> task_rename kretprobe   2350K events per sec
> task_rename raw_tp      3731K events per sec
> task_rename fentry      3639K events per sec
> task_rename fexit       3558K events per sec
> task_rename fmod_ret    3511K events per sec
> 
> /* selftest: multi-threaded, no atomics */
> task_rename base        3945K events per sec
> task_rename kprobe      3298K events per sec
> task_rename kretprobe   2451K events per sec
> task_rename raw_tp      3718K events per sec
> task_rename fentry      3782K events per sec
> task_rename fexit       3543K events per sec
> task_rename fmod_ret    3526K events per sec
> 
> 
[...]