* [PATCH] dm-verity: Remove WQ_UNBOUND.
@ 2023-02-02 1:23 ` Nathan Huckleberry
0 siblings, 0 replies; 10+ messages in thread
From: Nathan Huckleberry @ 2023-02-02 1:23 UTC (permalink / raw)
Cc: Nathan Huckleberry, Sami Tolvanen, Eric Biggers, Alasdair Kergon,
Mike Snitzer, dm-devel, linux-kernel
Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely
due to the asymmetric architecture of ARM64 processors.
I've been unable to reproduce the results that claim WQ_UNBOUND gives a
performance boost on x86-64.
This flag is causing performance issues for multiple subsystems within
Android. Notably, the same slowdown exists for decompression with
EROFS.
| open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND |
|-----------------------|------------|---------------|
| verity wait time (us) | 11746 | 119 (-98%) |
| erofs wait time (us) | 357805 | 174205 (-51%) |
| sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND |
|----------------------------|-----------=---|-------------|
| arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s |
| arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s |
| x86_64 (generic) | bw=233MiB/s | bw=230MiB/s |
Cc: Sami Tolvanen <samitolvanen@google.com>
Cc: Eric Biggers <ebiggers@kernel.org>
Signed-off-by: Nathan Huckleberry <nhuck@google.com>
---
drivers/md/dm-verity-target.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
index ccf5b852fbf7..020fd2341025 100644
--- a/drivers/md/dm-verity-target.c
+++ b/drivers/md/dm-verity-target.c
@@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
goto bad;
}
- /* WQ_UNBOUND greatly improves performance when running on ramdisk */
- wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
+ wq_flags = WQ_MEM_RECLAIM;
+
/*
* Using WQ_HIGHPRI improves throughput and completion latency by
* reducing wait times when reading from a dm-verity device.
--
2.39.1.456.gfc5497dd1b-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [dm-devel] [PATCH] dm-verity: Remove WQ_UNBOUND.
@ 2023-02-02 1:23 ` Nathan Huckleberry
0 siblings, 0 replies; 10+ messages in thread
From: Nathan Huckleberry @ 2023-02-02 1:23 UTC (permalink / raw)
Cc: Mike Snitzer, linux-kernel, Nathan Huckleberry, Eric Biggers,
dm-devel, Sami Tolvanen, Alasdair Kergon
Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely
due to the asymmetric architecture of ARM64 processors.
I've been unable to reproduce the results that claim WQ_UNBOUND gives a
performance boost on x86-64.
This flag is causing performance issues for multiple subsystems within
Android. Notably, the same slowdown exists for decompression with
EROFS.
| open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND |
|-----------------------|------------|---------------|
| verity wait time (us) | 11746 | 119 (-98%) |
| erofs wait time (us) | 357805 | 174205 (-51%) |
| sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND |
|----------------------------|-----------=---|-------------|
| arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s |
| arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s |
| x86_64 (generic) | bw=233MiB/s | bw=230MiB/s |
Cc: Sami Tolvanen <samitolvanen@google.com>
Cc: Eric Biggers <ebiggers@kernel.org>
Signed-off-by: Nathan Huckleberry <nhuck@google.com>
---
drivers/md/dm-verity-target.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
index ccf5b852fbf7..020fd2341025 100644
--- a/drivers/md/dm-verity-target.c
+++ b/drivers/md/dm-verity-target.c
@@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
goto bad;
}
- /* WQ_UNBOUND greatly improves performance when running on ramdisk */
- wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
+ wq_flags = WQ_MEM_RECLAIM;
+
/*
* Using WQ_HIGHPRI improves throughput and completion latency by
* reducing wait times when reading from a dm-verity device.
--
2.39.1.456.gfc5497dd1b-goog
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [dm-devel] [PATCH] dm-verity: Remove WQ_UNBOUND.
2023-02-02 1:23 ` [dm-devel] " Nathan Huckleberry
@ 2023-02-02 16:04 ` Mikulas Patocka
-1 siblings, 0 replies; 10+ messages in thread
From: Mikulas Patocka @ 2023-02-02 16:04 UTC (permalink / raw)
To: Nathan Huckleberry
Cc: Mike Snitzer, linux-kernel, Eric Biggers, dm-devel,
Sami Tolvanen, Alasdair Kergon
On Wed, 1 Feb 2023, Nathan Huckleberry wrote:
> Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely
> due to the asymmetric architecture of ARM64 processors.
>
> I've been unable to reproduce the results that claim WQ_UNBOUND gives a
> performance boost on x86-64.
>
> This flag is causing performance issues for multiple subsystems within
> Android. Notably, the same slowdown exists for decompression with
> EROFS.
>
> | open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND |
> |-----------------------|------------|---------------|
> | verity wait time (us) | 11746 | 119 (-98%) |
> | erofs wait time (us) | 357805 | 174205 (-51%) |
>
> | sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND |
> |----------------------------|-----------=---|-------------|
> | arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s |
> | arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s |
> | x86_64 (generic) | bw=233MiB/s | bw=230MiB/s |
>
> Cc: Sami Tolvanen <samitolvanen@google.com>
> Cc: Eric Biggers <ebiggers@kernel.org>
> Signed-off-by: Nathan Huckleberry <nhuck@google.com>
> ---
> drivers/md/dm-verity-target.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
> index ccf5b852fbf7..020fd2341025 100644
> --- a/drivers/md/dm-verity-target.c
> +++ b/drivers/md/dm-verity-target.c
> @@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
> goto bad;
> }
>
> - /* WQ_UNBOUND greatly improves performance when running on ramdisk */
> - wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
> + wq_flags = WQ_MEM_RECLAIM;
> +
> /*
> * Using WQ_HIGHPRI improves throughput and completion latency by
> * reducing wait times when reading from a dm-verity device.
Hi
If you remove WQ_UNBOUND, you should also change the last argument of
alloc_workqueue from num_online_cpus() to either 0 or 1. Try both 0 and 1
and tell us which performs better.
Mikulas
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dm-devel] [PATCH] dm-verity: Remove WQ_UNBOUND.
@ 2023-02-02 16:04 ` Mikulas Patocka
0 siblings, 0 replies; 10+ messages in thread
From: Mikulas Patocka @ 2023-02-02 16:04 UTC (permalink / raw)
To: Nathan Huckleberry
Cc: Mike Snitzer, linux-kernel, Eric Biggers, dm-devel,
Sami Tolvanen, Alasdair Kergon
On Wed, 1 Feb 2023, Nathan Huckleberry wrote:
> Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely
> due to the asymmetric architecture of ARM64 processors.
>
> I've been unable to reproduce the results that claim WQ_UNBOUND gives a
> performance boost on x86-64.
>
> This flag is causing performance issues for multiple subsystems within
> Android. Notably, the same slowdown exists for decompression with
> EROFS.
>
> | open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND |
> |-----------------------|------------|---------------|
> | verity wait time (us) | 11746 | 119 (-98%) |
> | erofs wait time (us) | 357805 | 174205 (-51%) |
>
> | sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND |
> |----------------------------|-----------=---|-------------|
> | arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s |
> | arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s |
> | x86_64 (generic) | bw=233MiB/s | bw=230MiB/s |
>
> Cc: Sami Tolvanen <samitolvanen@google.com>
> Cc: Eric Biggers <ebiggers@kernel.org>
> Signed-off-by: Nathan Huckleberry <nhuck@google.com>
> ---
> drivers/md/dm-verity-target.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
> index ccf5b852fbf7..020fd2341025 100644
> --- a/drivers/md/dm-verity-target.c
> +++ b/drivers/md/dm-verity-target.c
> @@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
> goto bad;
> }
>
> - /* WQ_UNBOUND greatly improves performance when running on ramdisk */
> - wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
> + wq_flags = WQ_MEM_RECLAIM;
> +
> /*
> * Using WQ_HIGHPRI improves throughput and completion latency by
> * reducing wait times when reading from a dm-verity device.
Hi
If you remove WQ_UNBOUND, you should also change the last argument of
alloc_workqueue from num_online_cpus() to either 0 or 1. Try both 0 and 1
and tell us which performs better.
Mikulas
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: dm-verity: Remove WQ_UNBOUND.
2023-02-02 1:23 ` [dm-devel] " Nathan Huckleberry
@ 2023-02-02 16:41 ` Mike Snitzer
-1 siblings, 0 replies; 10+ messages in thread
From: Mike Snitzer @ 2023-02-02 16:41 UTC (permalink / raw)
To: Nathan Huckleberry, mpatocka
Cc: linux-kernel, Eric Biggers, dm-devel, Sami Tolvanen, Alasdair Kergon
On Wed, Feb 01 2023 at 8:23P -0500,
Nathan Huckleberry <nhuck@google.com> wrote:
> Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely
> due to the asymmetric architecture of ARM64 processors.
>
> I've been unable to reproduce the results that claim WQ_UNBOUND gives a
> performance boost on x86-64.
>
> This flag is causing performance issues for multiple subsystems within
> Android. Notably, the same slowdown exists for decompression with
> EROFS.
>
> | open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND |
> |-----------------------|------------|---------------|
> | verity wait time (us) | 11746 | 119 (-98%) |
> | erofs wait time (us) | 357805 | 174205 (-51%) |
>
> | sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND |
> |----------------------------|-----------=---|-------------|
> | arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s |
> | arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s |
> | x86_64 (generic) | bw=233MiB/s | bw=230MiB/s |
>
> Cc: Sami Tolvanen <samitolvanen@google.com>
> Cc: Eric Biggers <ebiggers@kernel.org>
> Signed-off-by: Nathan Huckleberry <nhuck@google.com>
> ---
> drivers/md/dm-verity-target.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
> index ccf5b852fbf7..020fd2341025 100644
> --- a/drivers/md/dm-verity-target.c
> +++ b/drivers/md/dm-verity-target.c
> @@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
> goto bad;
> }
>
> - /* WQ_UNBOUND greatly improves performance when running on ramdisk */
> - wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
> + wq_flags = WQ_MEM_RECLAIM;
> +
> /*
> * Using WQ_HIGHPRI improves throughput and completion latency by
> * reducing wait times when reading from a dm-verity device.
> --
> 2.39.1.456.gfc5497dd1b-goog
Hi,
I've discussed with Mikulas, tweaked your patch slightly but accepted
your change, please see:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-6.3&id=6f30cc248507ee96c22ff4c3cbc86099ff12b7a9
Thanks,
Mike
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dm-devel] dm-verity: Remove WQ_UNBOUND.
@ 2023-02-02 16:41 ` Mike Snitzer
0 siblings, 0 replies; 10+ messages in thread
From: Mike Snitzer @ 2023-02-02 16:41 UTC (permalink / raw)
To: Nathan Huckleberry, mpatocka
Cc: Eric Biggers, dm-devel, Alasdair Kergon, linux-kernel, Sami Tolvanen
On Wed, Feb 01 2023 at 8:23P -0500,
Nathan Huckleberry <nhuck@google.com> wrote:
> Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely
> due to the asymmetric architecture of ARM64 processors.
>
> I've been unable to reproduce the results that claim WQ_UNBOUND gives a
> performance boost on x86-64.
>
> This flag is causing performance issues for multiple subsystems within
> Android. Notably, the same slowdown exists for decompression with
> EROFS.
>
> | open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND |
> |-----------------------|------------|---------------|
> | verity wait time (us) | 11746 | 119 (-98%) |
> | erofs wait time (us) | 357805 | 174205 (-51%) |
>
> | sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND |
> |----------------------------|-----------=---|-------------|
> | arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s |
> | arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s |
> | x86_64 (generic) | bw=233MiB/s | bw=230MiB/s |
>
> Cc: Sami Tolvanen <samitolvanen@google.com>
> Cc: Eric Biggers <ebiggers@kernel.org>
> Signed-off-by: Nathan Huckleberry <nhuck@google.com>
> ---
> drivers/md/dm-verity-target.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
> index ccf5b852fbf7..020fd2341025 100644
> --- a/drivers/md/dm-verity-target.c
> +++ b/drivers/md/dm-verity-target.c
> @@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
> goto bad;
> }
>
> - /* WQ_UNBOUND greatly improves performance when running on ramdisk */
> - wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
> + wq_flags = WQ_MEM_RECLAIM;
> +
> /*
> * Using WQ_HIGHPRI improves throughput and completion latency by
> * reducing wait times when reading from a dm-verity device.
> --
> 2.39.1.456.gfc5497dd1b-goog
Hi,
I've discussed with Mikulas, tweaked your patch slightly but accepted
your change, please see:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-6.3&id=6f30cc248507ee96c22ff4c3cbc86099ff12b7a9
Thanks,
Mike
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: dm-verity: Remove WQ_UNBOUND.
2023-02-02 16:41 ` [dm-devel] " Mike Snitzer
@ 2023-02-02 19:11 ` Nathan Huckleberry
-1 siblings, 0 replies; 10+ messages in thread
From: Nathan Huckleberry @ 2023-02-02 19:11 UTC (permalink / raw)
To: Mike Snitzer
Cc: mpatocka, linux-kernel, Eric Biggers, dm-devel, Sami Tolvanen,
Alasdair Kergon
Looks good. Thanks.
On Thu, Feb 2, 2023 at 8:41 AM Mike Snitzer <snitzer@kernel.org> wrote:
>
> On Wed, Feb 01 2023 at 8:23P -0500,
> Nathan Huckleberry <nhuck@google.com> wrote:
>
> > Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely
> > due to the asymmetric architecture of ARM64 processors.
> >
> > I've been unable to reproduce the results that claim WQ_UNBOUND gives a
> > performance boost on x86-64.
> >
> > This flag is causing performance issues for multiple subsystems within
> > Android. Notably, the same slowdown exists for decompression with
> > EROFS.
> >
> > | open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND |
> > |-----------------------|------------|---------------|
> > | verity wait time (us) | 11746 | 119 (-98%) |
> > | erofs wait time (us) | 357805 | 174205 (-51%) |
> >
> > | sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND |
> > |----------------------------|-----------=---|-------------|
> > | arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s |
> > | arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s |
> > | x86_64 (generic) | bw=233MiB/s | bw=230MiB/s |
> >
> > Cc: Sami Tolvanen <samitolvanen@google.com>
> > Cc: Eric Biggers <ebiggers@kernel.org>
> > Signed-off-by: Nathan Huckleberry <nhuck@google.com>
> > ---
> > drivers/md/dm-verity-target.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
> > index ccf5b852fbf7..020fd2341025 100644
> > --- a/drivers/md/dm-verity-target.c
> > +++ b/drivers/md/dm-verity-target.c
> > @@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
> > goto bad;
> > }
> >
> > - /* WQ_UNBOUND greatly improves performance when running on ramdisk */
> > - wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
> > + wq_flags = WQ_MEM_RECLAIM;
> > +
> > /*
> > * Using WQ_HIGHPRI improves throughput and completion latency by
> > * reducing wait times when reading from a dm-verity device.
> > --
> > 2.39.1.456.gfc5497dd1b-goog
>
> Hi,
>
> I've discussed with Mikulas, tweaked your patch slightly but accepted
> your change, please see:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-6.3&id=6f30cc248507ee96c22ff4c3cbc86099ff12b7a9
>
> Thanks,
> Mike
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dm-devel] dm-verity: Remove WQ_UNBOUND.
@ 2023-02-02 19:11 ` Nathan Huckleberry
0 siblings, 0 replies; 10+ messages in thread
From: Nathan Huckleberry @ 2023-02-02 19:11 UTC (permalink / raw)
To: Mike Snitzer
Cc: linux-kernel, Eric Biggers, dm-devel, mpatocka, Sami Tolvanen,
Alasdair Kergon
Looks good. Thanks.
On Thu, Feb 2, 2023 at 8:41 AM Mike Snitzer <snitzer@kernel.org> wrote:
>
> On Wed, Feb 01 2023 at 8:23P -0500,
> Nathan Huckleberry <nhuck@google.com> wrote:
>
> > Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely
> > due to the asymmetric architecture of ARM64 processors.
> >
> > I've been unable to reproduce the results that claim WQ_UNBOUND gives a
> > performance boost on x86-64.
> >
> > This flag is causing performance issues for multiple subsystems within
> > Android. Notably, the same slowdown exists for decompression with
> > EROFS.
> >
> > | open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND |
> > |-----------------------|------------|---------------|
> > | verity wait time (us) | 11746 | 119 (-98%) |
> > | erofs wait time (us) | 357805 | 174205 (-51%) |
> >
> > | sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND |
> > |----------------------------|-----------=---|-------------|
> > | arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s |
> > | arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s |
> > | x86_64 (generic) | bw=233MiB/s | bw=230MiB/s |
> >
> > Cc: Sami Tolvanen <samitolvanen@google.com>
> > Cc: Eric Biggers <ebiggers@kernel.org>
> > Signed-off-by: Nathan Huckleberry <nhuck@google.com>
> > ---
> > drivers/md/dm-verity-target.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
> > index ccf5b852fbf7..020fd2341025 100644
> > --- a/drivers/md/dm-verity-target.c
> > +++ b/drivers/md/dm-verity-target.c
> > @@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
> > goto bad;
> > }
> >
> > - /* WQ_UNBOUND greatly improves performance when running on ramdisk */
> > - wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
> > + wq_flags = WQ_MEM_RECLAIM;
> > +
> > /*
> > * Using WQ_HIGHPRI improves throughput and completion latency by
> > * reducing wait times when reading from a dm-verity device.
> > --
> > 2.39.1.456.gfc5497dd1b-goog
>
> Hi,
>
> I've discussed with Mikulas, tweaked your patch slightly but accepted
> your change, please see:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-6.3&id=6f30cc248507ee96c22ff4c3cbc86099ff12b7a9
>
> Thanks,
> Mike
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dm-devel] [PATCH] dm-verity: Remove WQ_UNBOUND.
2023-02-02 16:04 ` Mikulas Patocka
@ 2023-02-07 22:54 ` Nathan Huckleberry
-1 siblings, 0 replies; 10+ messages in thread
From: Nathan Huckleberry @ 2023-02-07 22:54 UTC (permalink / raw)
To: Mikulas Patocka
Cc: Mike Snitzer, linux-kernel, Eric Biggers, dm-devel,
Sami Tolvanen, Alasdair Kergon
On Tue, Feb 7, 2023 at 12:16 AM Mikulas Patocka <mpatocka@redhat.com> wrote:
>
>
>
> On Wed, 1 Feb 2023, Nathan Huckleberry wrote:
>
> > Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely
> > due to the asymmetric architecture of ARM64 processors.
> >
> > I've been unable to reproduce the results that claim WQ_UNBOUND gives a
> > performance boost on x86-64.
> >
> > This flag is causing performance issues for multiple subsystems within
> > Android. Notably, the same slowdown exists for decompression with
> > EROFS.
> >
> > | open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND |
> > |-----------------------|------------|---------------|
> > | verity wait time (us) | 11746 | 119 (-98%) |
> > | erofs wait time (us) | 357805 | 174205 (-51%) |
> >
> > | sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND |
> > |----------------------------|-----------=---|-------------|
> > | arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s |
> > | arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s |
> > | x86_64 (generic) | bw=233MiB/s | bw=230MiB/s |
> >
> > Cc: Sami Tolvanen <samitolvanen@google.com>
> > Cc: Eric Biggers <ebiggers@kernel.org>
> > Signed-off-by: Nathan Huckleberry <nhuck@google.com>
> > ---
> > drivers/md/dm-verity-target.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
> > index ccf5b852fbf7..020fd2341025 100644
> > --- a/drivers/md/dm-verity-target.c
> > +++ b/drivers/md/dm-verity-target.c
> > @@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
> > goto bad;
> > }
> >
> > - /* WQ_UNBOUND greatly improves performance when running on ramdisk */
> > - wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
> > + wq_flags = WQ_MEM_RECLAIM;
> > +
> > /*
> > * Using WQ_HIGHPRI improves throughput and completion latency by
> > * reducing wait times when reading from a dm-verity device.
>
> Hi
>
> If you remove WQ_UNBOUND, you should also change the last argument of
> alloc_workqueue from num_online_cpus() to either 0 or 1. Try both 0 and 1
> and tell us which performs better.
They look roughly the same on ARM64. There's a slight advantage for
using 1, but they're too close to conclusively say one is better.
Thanks,
Huck
>
> Mikulas
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dm-devel] [PATCH] dm-verity: Remove WQ_UNBOUND.
@ 2023-02-07 22:54 ` Nathan Huckleberry
0 siblings, 0 replies; 10+ messages in thread
From: Nathan Huckleberry @ 2023-02-07 22:54 UTC (permalink / raw)
To: Mikulas Patocka
Cc: Mike Snitzer, linux-kernel, Eric Biggers, dm-devel,
Sami Tolvanen, Alasdair Kergon
On Tue, Feb 7, 2023 at 12:16 AM Mikulas Patocka <mpatocka@redhat.com> wrote:
>
>
>
> On Wed, 1 Feb 2023, Nathan Huckleberry wrote:
>
> > Setting WQ_UNBOUND increases scheduler latency on ARM64. This is likely
> > due to the asymmetric architecture of ARM64 processors.
> >
> > I've been unable to reproduce the results that claim WQ_UNBOUND gives a
> > performance boost on x86-64.
> >
> > This flag is causing performance issues for multiple subsystems within
> > Android. Notably, the same slowdown exists for decompression with
> > EROFS.
> >
> > | open-prebuilt-camera | WQ_UNBOUND | ~WQ_UNBOUND |
> > |-----------------------|------------|---------------|
> > | verity wait time (us) | 11746 | 119 (-98%) |
> > | erofs wait time (us) | 357805 | 174205 (-51%) |
> >
> > | sha256 ramdisk random read | WQ_UNBOUND | ~WQ_UNBOUND |
> > |----------------------------|-----------=---|-------------|
> > | arm64 (accelerated) | bw=42.4MiB/s | bw=212MiB/s |
> > | arm64 (generic) | bw=16.5MiB/s | bw=48MiB/s |
> > | x86_64 (generic) | bw=233MiB/s | bw=230MiB/s |
> >
> > Cc: Sami Tolvanen <samitolvanen@google.com>
> > Cc: Eric Biggers <ebiggers@kernel.org>
> > Signed-off-by: Nathan Huckleberry <nhuck@google.com>
> > ---
> > drivers/md/dm-verity-target.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
> > index ccf5b852fbf7..020fd2341025 100644
> > --- a/drivers/md/dm-verity-target.c
> > +++ b/drivers/md/dm-verity-target.c
> > @@ -1399,8 +1399,8 @@ static int verity_ctr(struct dm_target *ti, unsigned argc, char **argv)
> > goto bad;
> > }
> >
> > - /* WQ_UNBOUND greatly improves performance when running on ramdisk */
> > - wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
> > + wq_flags = WQ_MEM_RECLAIM;
> > +
> > /*
> > * Using WQ_HIGHPRI improves throughput and completion latency by
> > * reducing wait times when reading from a dm-verity device.
>
> Hi
>
> If you remove WQ_UNBOUND, you should also change the last argument of
> alloc_workqueue from num_online_cpus() to either 0 or 1. Try both 0 and 1
> and tell us which performs better.
They look roughly the same on ARM64. There's a slight advantage for
using 1, but they're too close to conclusively say one is better.
Thanks,
Huck
>
> Mikulas
>
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2023-02-07 22:54 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-02-02 1:23 [PATCH] dm-verity: Remove WQ_UNBOUND Nathan Huckleberry
2023-02-02 1:23 ` [dm-devel] " Nathan Huckleberry
2023-02-02 16:04 ` Mikulas Patocka
2023-02-02 16:04 ` Mikulas Patocka
2023-02-07 22:54 ` Nathan Huckleberry
2023-02-07 22:54 ` Nathan Huckleberry
2023-02-02 16:41 ` Mike Snitzer
2023-02-02 16:41 ` [dm-devel] " Mike Snitzer
2023-02-02 19:11 ` Nathan Huckleberry
2023-02-02 19:11 ` [dm-devel] " Nathan Huckleberry
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.