Re: Proposal for a regular upstream performance testing

["Lukáš Doktor" <ldoktor@redhat.com>]

[-- Attachment #1.1.1: Type: text/plain, Size: 3551 bytes --]

Dne 28. 03. 22 v 11:57 Stefan Hajnoczi napsal(a):
> On Mon, Mar 28, 2022 at 08:18:43AM +0200, Lukáš Doktor wrote:
>> Hello Stefan, folks,
>>
>> I seem to have another hit, an improvement actually and it seems to be bisected all the way to you, Stefan. Let me use this as another example of how such process could look like and we can use this to hammer-out the details like via what means to submit the request, whom to notify and how to proceed further.
>>
>> ---
>>
>> Last week I noticed an improvement in TunedLibvirt/fio-rot-Aj-8i/0000:./write-4KiB/throughput/iops_sec.mean (<driver name="qemu" type="raw" io="native" cache="none"/>, fio, rotationary disk, raw file on host xfs partition, jobs=#cpus, iodepth=8, 4k writes) check and bisected it to:
>>
>> commit fc8796465c6cd4091efe6a2f8b353f07324f49c7
>> Author: Stefan Hajnoczi <stefanha@redhat.com>
>> Date:   Wed Feb 23 15:57:03 2022 +0000
>>
>>     aio-posix: fix spurious ->poll_ready() callbacks in main loop
>>
>> Could you please confirm that it does make sense and that it is expected? (looks like it from the description).
>>
>> Note that this commit was pin pointed using 2 out of 3 commits result, there were actually some little differences between commits fc8 and cc5. The fc8 and 202 results scored similarly to both, good and bad commits with 2 being closer to the bad one. Since cc5 they seem to stabilize further scoring slightly lower than the median fc8 result. Anyway I don't have enough data to declare anything. I can bisect it further if needed.
> 
> Yes, I can imagine that commit fc8796465c6c might improve non-IOThread
> performance!
> 

Hello Stefan and thank you for this confirmation as well as questions. Let me explain that a bit and perhaps you can help me to improve the report as it was designed for CI team and can be I can see where the confusion might come from.

> I don't know how to read the report:
> - What is the difference between "Group stats" and "Failures"?
> - Why are there 3 different means in "Group stats"?

The group stats combine (average) multiple individual failures with some common aspect and stricter thresholds are then used based on the number of matches. The group names contain asterisk (*) character which works the same way linux globs work and all matching tests are included. In this report it doesn't really makes sense as only single test variant was performed, but when various tests/scenarios are being used it can indicate change across all fio-write jobs, or all tests on various profiles...

> - Why are there 3 "fc8" columns in "Failures"?

To avoid bisect wandering away I'm usually using 3-5 samples per run. Still this wasn't reliable enough in some cases and bisect ended up on random commits so I added a 2 out of 3 feature. Every commit is checked twice and when they don't match it runs the third one. In the end all of the attempts are included in the report to better visualize how stable the results are.

> 
> I don't feel confident searching git-log(1) output with 3-character
> commit IDs. git itself uses 12 characters for short commit IDs with a
> reasonably low chance of collisions.
> 

I chose 3 letters only as usually the bisection window is quite small and there is the bisection log with all the full commits attached. Using longer hashes results in very wide table, which I wanted to avoid. Perhaps I can modify the links which currently point to the jenkins results to point to the qemu sha instead, what do you think?

Regards,
Lukáš

> Stefan

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 12153 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]