* Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
@ 2023-05-25 10:18 Lukáš Doktor
2023-05-25 15:21 ` Stefan Hajnoczi
0 siblings, 1 reply; 8+ messages in thread
From: Lukáš Doktor @ 2023-05-25 10:18 UTC (permalink / raw)
To: Stefan Hajnoczi, qemu-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 1909 bytes --]
Hello Stefan, folks,
the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
Based on the commit message I guess it's expected so take this just as a record of an improvement.
Bisect log:
git bisect start
# good: [aa222a8e4f975284b3f8f131653a4114b3d333b3] Merge tag 'for_upstream' of https://git.kernel.org/pub/scm/virt/kvm/mst/qemu into staging
git bisect good aa222a8e4f975284b3f8f131653a4114b3d333b3
# bad: [ad3387396a71416cacc0b394e5e440dd6e9ba19a] Merge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging
git bisect bad ad3387396a71416cacc0b394e5e440dd6e9ba19a
# good: [71438d8dac07f28c01cf6d90fce14efe04c77824] graph-lock: Honour read locks even in the main thread
git bisect good 71438d8dac07f28c01cf6d90fce14efe04c77824
# good: [4b424c757188f7a47630a4d8edcf4ad9f19255bc] scripts: make sure scripts are invoked via $(PYTHON)
git bisect good 4b424c757188f7a47630a4d8edcf4ad9f19255bc
# bad: [80fc5d260002432628710f8b0c7cfc7d9b97bb9d] graph-lock: Disable locking for now
git bisect bad 80fc5d260002432628710f8b0c7cfc7d9b97bb9d
# bad: [6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee] aio-posix: do not nest poll handlers
git bisect bad 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
# good: [78935fcd88ec2d26d50e45043b262f0326e6d410] iotests/245: Check if 'compress' driver is available
git bisect good 78935fcd88ec2d26d50e45043b262f0326e6d410
# first bad commit: [6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee] aio-posix: do not nest poll handlers
Regards,
Lukáš
PS: The list of perf-ci issues is available here: https://docs.google.com/spreadsheets/d/1HEXI5wDsNgAIgXl5MIhGond898Vz5A1Hrkl0lZmWEbg/edit#gid=0
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 12925 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
2023-05-25 10:18 Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee Lukáš Doktor
@ 2023-05-25 15:21 ` Stefan Hajnoczi
2023-05-26 8:06 ` Lukáš Doktor
0 siblings, 1 reply; 8+ messages in thread
From: Stefan Hajnoczi @ 2023-05-25 15:21 UTC (permalink / raw)
To: Lukáš Doktor; +Cc: Stefan Hajnoczi, qemu-devel
On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
>
> https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
>
> Based on the commit message I guess it's expected so take this just as a record of an improvement.
The commit was not intended to change performance and I'm not sure why
it happens!
Thanks for letting me know.
Stefan
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
2023-05-25 15:21 ` Stefan Hajnoczi
@ 2023-05-26 8:06 ` Lukáš Doktor
2023-05-26 10:56 ` Stefan Hajnoczi
0 siblings, 1 reply; 8+ messages in thread
From: Lukáš Doktor @ 2023-05-26 8:06 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: Stefan Hajnoczi, qemu-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 1304 bytes --]
Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
>>
>> https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
>>
>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
>
> The commit was not intended to change performance and I'm not sure why
> it happens!
>
It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
Regards,
Lukáš
> Thanks for letting me know.
>
> Stefan
>
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 12925 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
2023-05-26 8:06 ` Lukáš Doktor
@ 2023-05-26 10:56 ` Stefan Hajnoczi
2023-05-31 16:50 ` Lukáš Doktor
0 siblings, 1 reply; 8+ messages in thread
From: Stefan Hajnoczi @ 2023-05-26 10:56 UTC (permalink / raw)
To: Lukáš Doktor; +Cc: Stefan Hajnoczi, qemu-devel
On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
>
> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
> > On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
> >>
> >> https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
> >>
> >> Based on the commit message I guess it's expected so take this just as a record of an improvement.
> >
> > The commit was not intended to change performance and I'm not sure why
> > it happens!
> >
>
> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
>
> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
Thanks!
Stefan
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
2023-05-26 10:56 ` Stefan Hajnoczi
@ 2023-05-31 16:50 ` Lukáš Doktor
2023-05-31 17:14 ` Stefan Hajnoczi
0 siblings, 1 reply; 8+ messages in thread
From: Lukáš Doktor @ 2023-05-31 16:50 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: Stefan Hajnoczi, qemu-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 1814 bytes --]
Dne 26. 05. 23 v 12:56 Stefan Hajnoczi napsal(a):
> On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
>>
>> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
>>> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
>>>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
>>>>
>>>> https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
>>>>
>>>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
>>>
>>> The commit was not intended to change performance and I'm not sure why
>>> it happens!
>>>
>>
>> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
>>
>> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
>
Hello Stefan, folks,
the regression proved to be there and stably reproducible. With NVMe 4k writes with jobs=10 and iodepth=4 I can see a 50% regression on my machine:
https://ldoktor.github.io/tmp/RedHat-virtlab722/v8.0.0/150-regression.html
The rest of the cases doesn't show any change at all. I can provide more data if someone is interested.
Regards,
Lukáš
> Thanks!
>
> Stefan
>
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 12925 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
2023-05-31 16:50 ` Lukáš Doktor
@ 2023-05-31 17:14 ` Stefan Hajnoczi
2023-06-01 4:23 ` Performance improvement and regression " Lukáš Doktor
0 siblings, 1 reply; 8+ messages in thread
From: Stefan Hajnoczi @ 2023-05-31 17:14 UTC (permalink / raw)
To: Lukáš Doktor; +Cc: Stefan Hajnoczi, qemu-devel
On Wed, 31 May 2023 at 12:50, Lukáš Doktor <ldoktor@redhat.com> wrote:
>
> Dne 26. 05. 23 v 12:56 Stefan Hajnoczi napsal(a):
> > On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>
> >> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
> >>> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
> >>>>
> >>>> https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
> >>>>
> >>>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
> >>>
> >>> The commit was not intended to change performance and I'm not sure why
> >>> it happens!
> >>>
> >>
> >> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
> >>
> >> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
> >
>
> Hello Stefan, folks,
>
> the regression proved to be there and stably reproducible. With NVMe 4k writes with jobs=10 and iodepth=4 I can see a 50% regression on my machine:
>
> https://ldoktor.github.io/tmp/RedHat-virtlab722/v8.0.0/150-regression.html
>
> The rest of the cases doesn't show any change at all. I can provide more data if someone is interested.
Which commit caused the regression?
Stefan
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Performance improvement and regression with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
2023-05-31 17:14 ` Stefan Hajnoczi
@ 2023-06-01 4:23 ` Lukáš Doktor
2023-06-01 13:17 ` Stefan Hajnoczi
0 siblings, 1 reply; 8+ messages in thread
From: Lukáš Doktor @ 2023-06-01 4:23 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: Stefan Hajnoczi, qemu-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 2148 bytes --]
Dne 31. 05. 23 v 19:14 Stefan Hajnoczi napsal(a):
> On Wed, 31 May 2023 at 12:50, Lukáš Doktor <ldoktor@redhat.com> wrote:
>>
>> Dne 26. 05. 23 v 12:56 Stefan Hajnoczi napsal(a):
>>> On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
>>>>
>>>> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
>>>>> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
>>>>>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
>>>>>>
>>>>>> https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
>>>>>>
>>>>>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
>>>>>
>>>>> The commit was not intended to change performance and I'm not sure why
>>>>> it happens!
>>>>>
>>>>
>>>> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
>>>>
>>>> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
>>>
>>
>> Hello Stefan, folks,
>>
>> the regression proved to be there and stably reproducible. With NVMe 4k writes with jobs=10 and iodepth=4 I can see a 50% regression on my machine:
>>
>> https://ldoktor.github.io/tmp/RedHat-virtlab722/v8.0.0/150-regression.html
>>
>> The rest of the cases doesn't show any change at all. I can provide more data if someone is interested.
>
> Which commit caused the regression?
Hello Stefan,
the same one that caused the improvement on rotational disks: 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
Lukáš
>
> Stefan
>
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 12925 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Performance improvement and regression with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
2023-06-01 4:23 ` Performance improvement and regression " Lukáš Doktor
@ 2023-06-01 13:17 ` Stefan Hajnoczi
0 siblings, 0 replies; 8+ messages in thread
From: Stefan Hajnoczi @ 2023-06-01 13:17 UTC (permalink / raw)
To: Lukáš Doktor; +Cc: Stefan Hajnoczi, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 2406 bytes --]
On Thu, Jun 01, 2023 at 06:23:21AM +0200, Lukáš Doktor wrote:
> Dne 31. 05. 23 v 19:14 Stefan Hajnoczi napsal(a):
> > On Wed, 31 May 2023 at 12:50, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>
> >> Dne 26. 05. 23 v 12:56 Stefan Hajnoczi napsal(a):
> >>> On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>>>
> >>>> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
> >>>>> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>>>>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
> >>>>>>
> >>>>>> https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
> >>>>>>
> >>>>>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
> >>>>>
> >>>>> The commit was not intended to change performance and I'm not sure why
> >>>>> it happens!
> >>>>>
> >>>>
> >>>> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
> >>>>
> >>>> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
> >>>
> >>
> >> Hello Stefan, folks,
> >>
> >> the regression proved to be there and stably reproducible. With NVMe 4k writes with jobs=10 and iodepth=4 I can see a 50% regression on my machine:
> >>
> >> https://ldoktor.github.io/tmp/RedHat-virtlab722/v8.0.0/150-regression.html
> >>
> >> The rest of the cases doesn't show any change at all. I can provide more data if someone is interested.
> >
> > Which commit caused the regression?
>
> Hello Stefan,
>
> the same one that caused the improvement on rotational disks: 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
Thanks. This commit definitely needs more investigation. I didn't expect
it to have a noticeable effect.
I will take a look in July.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2023-06-01 13:18 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-25 10:18 Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee Lukáš Doktor
2023-05-25 15:21 ` Stefan Hajnoczi
2023-05-26 8:06 ` Lukáš Doktor
2023-05-26 10:56 ` Stefan Hajnoczi
2023-05-31 16:50 ` Lukáš Doktor
2023-05-31 17:14 ` Stefan Hajnoczi
2023-06-01 4:23 ` Performance improvement and regression " Lukáš Doktor
2023-06-01 13:17 ` Stefan Hajnoczi
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.