Re: [REPORT] Nightly Performance Tests - Sunday, September 13, 2020

[Aleksandar Markovic <aleksandar.m.mail@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6144 bytes --]

On Mon, Sep 14, 2020 at 12:52 PM Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
wrote:

> On Mon, Sep 14, 2020 at 8:46 AM Philippe Mathieu-Daudé <f4bug@amsat.org>
> wrote:
> >
> > Hi Ahmed,
> >
> > On 9/14/20 12:07 AM, Ahmed Karaman wrote:
> > > Host CPU         : Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
> > > Host Memory      : 15.49 GB
> > >
> > > Start Time (UTC) : 2020-09-13 21:35:01
> > > End Time (UTC)   : 2020-09-13 22:07:44
> > > Execution Time   : 0:32:42.230467
> > >
> > > Status           : SUCCESS
> > >
> > > Note:
> > > Changes denoted by '-----' are less than 0.01%.
> > >
> > > --------------------------------------------------------
> > >             SUMMARY REPORT - COMMIT f00f57f3
> > > --------------------------------------------------------
> >
> > (Maybe this was already commented earlier but I missed it).
> >
> > What change had a so significant impact on the m68k target?
> > At a glance I only see mostly changes in softfloat:
> >
> > $ git log --oneline v5.1.0..f00f57f3 tcg target/m68k fpu
> > fe4b0b5bfa9 tcg: Implement 256-bit dup for tcg_gen_gvec_dup_mem
> > 6a17646176e tcg: Eliminate one store for in-place 128-bit dup_mem
> > e7e8f33fb60 tcg: Fix tcg gen for vectorized absolute value
> > 5ebf5f4be66 softfloat: Define misc operations for bfloat16
> > 34f0c0a98a5 softfloat: Define convert operations for bfloat16
> > 8282310d853 softfloat: Define operations for bfloat16
> > 0d93d8ec632 softfloat: Add fp16 and uint8/int8 conversion functions
> > fbcc38e4cb1 softfloat: add xtensa specialization for pickNaNMulAdd
> > 913602e3ffe softfloat: pass float_status pointer to pickNaN
> > cc43c692511 softfloat: make NO_SIGNALING_NANS runtime property
> > 73ebe95e8e5 target/ppc: add vmulld to INDEX_op_mul_vec case
> >
> > > --------------------------------------------------------
> > > --------------------------------------------------------
> > > Test Program: matmult_double
> > > --------------------------------------------------------
> > > Target              Instructions      Latest      v5.1.0
> > > ----------  --------------------  ----------  ----------
> > > aarch64            1 412 412 599       -----     +0.311%
> > > alpha              3 233 957 639       -----     +7.472%
> > > arm                8 545 302 995       -----      +1.09%
> > > hppa               3 483 527 330       -----     +4.466%
> > > m68k               3 919 110 506       -----    +18.433%
> > > mips               2 344 641 840       -----     +4.085%
> > > mipsel             3 329 912 425       -----     +5.177%
> > > mips64             2 359 024 910       -----     +4.075%
> > > mips64el           3 343 650 686       -----     +5.166%
> > > ppc                3 209 505 701       -----     +3.248%
> > > ppc64              3 287 495 266       -----     +3.173%
> > > ppc64le            3 287 135 580       -----     +3.171%
> > > riscv64            1 221 617 903       -----     +0.278%
> > > s390x              2 874 160 417       -----     +5.826%
> > > sh4                3 544 094 841       -----      +6.42%
> > > sparc64            3 426 094 848       -----     +7.138%
> > > x86_64             1 249 076 697       -----     +0.335%
> > > --------------------------------------------------------
> > ...
> > > --------------------------------------------------------
> > > Test Program: qsort_double
> > > --------------------------------------------------------
> > > Target              Instructions      Latest      v5.1.0
> > > ----------  --------------------  ----------  ----------
> > > aarch64            2 709 839 947       -----     +2.423%
> > > alpha              1 969 432 086       -----     +3.679%
> > > arm                8 323 168 267       -----     +2.589%
> > > hppa               3 188 316 726       -----       +2.9%
> > > m68k               4 953 947 225       -----    +15.153%
> > > mips               2 123 789 120       -----     +3.049%
> > > mipsel             2 124 235 492       -----     +3.049%
> > > mips64             1 999 025 951       -----     +3.404%
> > > mips64el           1 996 433 190       -----     +3.409%
> > > ppc                2 819 299 843       -----     +5.436%
> > > ppc64              2 768 177 037       -----     +5.512%
> > > ppc64le            2 724 766 044       -----     +5.602%
> > > riscv64            1 638 324 190       -----     +4.021%
> > > s390x              2 519 117 806       -----     +3.364%
> > > sh4                2 595 696 102       -----       +3.0%
> > > sparc64            3 988 892 763       -----     +2.744%
> > > x86_64             2 033 624 062       -----     +3.242%
> > > --------------------------------------------------------
>
> Hi Mr. Philippe,
> The performance degradation from v5.1.0 of all targets, and especially
> m68k, was introduced between the two nightly tests below:
>
> [REPORT] Nightly Performance Tests - Thursday, August 20, 2020:
> https://lists.nongnu.org/archive/html/qemu-devel/2020-08/msg04923.html
>
> [REPORT] Nightly Performance Tests - Saturday, August 22, 2020
> https://lists.nongnu.org/archive/html/qemu-devel/2020-08/msg05537.html
>
> It looks like the new build system is the culprit.
>
> The "bisect.py" script introduced during the "TCG Continuous
> Benchmarking" GSoC project can be very handy in these cases. I wrote
> about the tool and how to use it in the report below:
>
> https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/Finding-Commits-Affecting-QEMU-Performance/
>
>
Hi, Ahmed.

I think the bisect.py script will work only if both "start" and "end"
commits are before build system change, or if both of them are after build
system change.

In other words, the script is unlikely to work if "start" is before, and
"end" is after build system change.

This means that, most probably, one should resort to manual analysis of
origins of performance degradation on Aug 22nd.

One area that definitely might be the culprit is the difference in CFLAGS
before and after.

Yours,
Aleksandar


> Best regards,
> Ahmed Karaman
>
>

[-- Attachment #2: Type: text/html, Size: 8449 bytes --]