* [PATCH yocto-autobuilder-helper] config.json: set ZSTD_THREADS like XZ_THREADS
@ 2021-11-23 15:03 Ross Burton
2021-11-23 17:11 ` [yocto] " Alexander Kanavin
0 siblings, 1 reply; 6+ messages in thread
From: Ross Burton @ 2021-11-23 15:03 UTC (permalink / raw)
To: yocto
Signed-off-by: Ross Burton <ross.burton@arm.com>
---
config.json | 1 +
1 file changed, 1 insertion(+)
diff --git a/config.json b/config.json
index 7203783..c0543d9 100644
--- a/config.json
+++ b/config.json
@@ -47,6 +47,7 @@
"PARALLEL_MAKE = '-j 16 -l 52'",
"XZ_MEMLIMIT = '5%'",
"XZ_THREADS = '8'",
+ "ZSTD_THREADS = '8'",
"BB_TASK_NICE_LEVEL = '5'",
"BB_TASK_NICE_LEVEL:task-testimage = '0'",
"BB_TASK_IONICE_LEVEL = '2.7'",
--
2.25.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [yocto] [PATCH yocto-autobuilder-helper] config.json: set ZSTD_THREADS like XZ_THREADS
2021-11-23 15:03 [PATCH yocto-autobuilder-helper] config.json: set ZSTD_THREADS like XZ_THREADS Ross Burton
@ 2021-11-23 17:11 ` Alexander Kanavin
2021-11-23 19:53 ` Ross Burton
0 siblings, 1 reply; 6+ messages in thread
From: Alexander Kanavin @ 2021-11-23 17:11 UTC (permalink / raw)
To: Ross Burton; +Cc: Yocto-mailing-list
[-- Attachment #1: Type: text/plain, Size: 1081 bytes --]
Is there a particular reason?
Alex
On Tue, 23 Nov 2021 at 16:03, Ross Burton <ross@burtonini.com> wrote:
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
> config.json | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/config.json b/config.json
> index 7203783..c0543d9 100644
> --- a/config.json
> +++ b/config.json
> @@ -47,6 +47,7 @@
> "PARALLEL_MAKE = '-j 16 -l 52'",
> "XZ_MEMLIMIT = '5%'",
> "XZ_THREADS = '8'",
> + "ZSTD_THREADS = '8'",
> "BB_TASK_NICE_LEVEL = '5'",
> "BB_TASK_NICE_LEVEL:task-testimage = '0'",
> "BB_TASK_IONICE_LEVEL = '2.7'",
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#55396):
> https://lists.yoctoproject.org/g/yocto/message/55396
> Mute This Topic: https://lists.yoctoproject.org/mt/87260143/1686489
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
[-- Attachment #2: Type: text/html, Size: 2146 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [yocto] [PATCH yocto-autobuilder-helper] config.json: set ZSTD_THREADS like XZ_THREADS
2021-11-23 17:11 ` [yocto] " Alexander Kanavin
@ 2021-11-23 19:53 ` Ross Burton
2021-11-24 7:27 ` Ernst Sjöstrand
2021-11-24 8:00 ` [yocto] " Alexander Kanavin
0 siblings, 2 replies; 6+ messages in thread
From: Ross Burton @ 2021-11-23 19:53 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: Yocto-mailing-list
My personal build machine has been crashing and dying due to overload
with the default ZSTD_THREADS, and the AB has had problems with xz due
to overload so I was being proactive and ensuring zstd is limited on
the AB the same way xz is.
Ross
On Tue, 23 Nov 2021 at 17:11, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> Is there a particular reason?
>
> Alex
>
> On Tue, 23 Nov 2021 at 16:03, Ross Burton <ross@burtonini.com> wrote:
>>
>> Signed-off-by: Ross Burton <ross.burton@arm.com>
>> ---
>> config.json | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/config.json b/config.json
>> index 7203783..c0543d9 100644
>> --- a/config.json
>> +++ b/config.json
>> @@ -47,6 +47,7 @@
>> "PARALLEL_MAKE = '-j 16 -l 52'",
>> "XZ_MEMLIMIT = '5%'",
>> "XZ_THREADS = '8'",
>> + "ZSTD_THREADS = '8'",
>> "BB_TASK_NICE_LEVEL = '5'",
>> "BB_TASK_NICE_LEVEL:task-testimage = '0'",
>> "BB_TASK_IONICE_LEVEL = '2.7'",
>> --
>> 2.25.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#55396): https://lists.yoctoproject.org/g/yocto/message/55396
>> Mute This Topic: https://lists.yoctoproject.org/mt/87260143/1686489
>> Group Owner: yocto+owner@lists.yoctoproject.org
>> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH yocto-autobuilder-helper] config.json: set ZSTD_THREADS like XZ_THREADS
2021-11-23 19:53 ` Ross Burton
@ 2021-11-24 7:27 ` Ernst Sjöstrand
2021-11-24 8:00 ` [yocto] " Alexander Kanavin
1 sibling, 0 replies; 6+ messages in thread
From: Ernst Sjöstrand @ 2021-11-24 7:27 UTC (permalink / raw)
To: yocto
[-- Attachment #1: Type: text/plain, Size: 135 bytes --]
There was a long thread about this on oe-core recently:
https://lists.openembedded.org/g/openembedded-core/message/157999
//Ernst
[-- Attachment #2: Type: text/html, Size: 147 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [yocto] [PATCH yocto-autobuilder-helper] config.json: set ZSTD_THREADS like XZ_THREADS
2021-11-23 19:53 ` Ross Burton
2021-11-24 7:27 ` Ernst Sjöstrand
@ 2021-11-24 8:00 ` Alexander Kanavin
[not found] ` <473510c439b0d732d956ab327a425221f6019cc7.camel@linuxfoundation.org>
1 sibling, 1 reply; 6+ messages in thread
From: Alexander Kanavin @ 2021-11-24 8:00 UTC (permalink / raw)
To: Ross Burton; +Cc: Yocto-mailing-list
[-- Attachment #1: Type: text/plain, Size: 1737 bytes --]
But the AB has not been exhibiting any problems with zstd, and this will
degrade performance. Let's only fix what is broken.
Alex
On Tue, 23 Nov 2021 at 20:53, Ross Burton <ross@burtonini.com> wrote:
> My personal build machine has been crashing and dying due to overload
> with the default ZSTD_THREADS, and the AB has had problems with xz due
> to overload so I was being proactive and ensuring zstd is limited on
> the AB the same way xz is.
>
> Ross
>
> On Tue, 23 Nov 2021 at 17:11, Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
> >
> > Is there a particular reason?
> >
> > Alex
> >
> > On Tue, 23 Nov 2021 at 16:03, Ross Burton <ross@burtonini.com> wrote:
> >>
> >> Signed-off-by: Ross Burton <ross.burton@arm.com>
> >> ---
> >> config.json | 1 +
> >> 1 file changed, 1 insertion(+)
> >>
> >> diff --git a/config.json b/config.json
> >> index 7203783..c0543d9 100644
> >> --- a/config.json
> >> +++ b/config.json
> >> @@ -47,6 +47,7 @@
> >> "PARALLEL_MAKE = '-j 16 -l 52'",
> >> "XZ_MEMLIMIT = '5%'",
> >> "XZ_THREADS = '8'",
> >> + "ZSTD_THREADS = '8'",
> >> "BB_TASK_NICE_LEVEL = '5'",
> >> "BB_TASK_NICE_LEVEL:task-testimage = '0'",
> >> "BB_TASK_IONICE_LEVEL = '2.7'",
> >> --
> >> 2.25.1
> >>
> >>
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >> Links: You receive all messages sent to this group.
> >> View/Reply Online (#55396):
> https://lists.yoctoproject.org/g/yocto/message/55396
> >> Mute This Topic: https://lists.yoctoproject.org/mt/87260143/1686489
> >> Group Owner: yocto+owner@lists.yoctoproject.org
> >> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> alex.kanavin@gmail.com]
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >>
>
[-- Attachment #2: Type: text/html, Size: 3172 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [yocto] [PATCH yocto-autobuilder-helper] config.json: set ZSTD_THREADS like XZ_THREADS
[not found] ` <473510c439b0d732d956ab327a425221f6019cc7.camel@linuxfoundation.org>
@ 2021-11-24 18:21 ` Khem Raj
0 siblings, 0 replies; 6+ messages in thread
From: Khem Raj @ 2021-11-24 18:21 UTC (permalink / raw)
To: Richard Purdie; +Cc: Alexander Kanavin, Ross Burton, Yocto-mailing-list
On Wed, Nov 24, 2021 at 2:18 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2021-11-24 at 09:00 +0100, Alexander Kanavin wrote:
> > But the AB has not been exhibiting any problems with zstd, and this will
> > degrade performance. Let's only fix what is broken.
>
> I'm not sure I agree with that.
>
> We have 60+ "intermittent" bugs and some of us are in weekly meetings trying to
> do something about working out why these are failing. It feels like we're not
> really getting too far with some subset of them and it is using up a lot of the
> SWAT and bug triage time.
>
> We've made a few changes to try and reduce the load spikes on the systems and
> this fits with the other changes we've made.
>
From a different data point, we have clipped the parallelism for XZ
and ZSTD internally to very low ( 2 and 4 )
and it has in fact reduced unexpected failures and seen no impact on
build performance. Since parallelism settings
are myopic for these tools, they greedily take every CPU which is not
best for overall build thats doing many
things in parallel, so its best to curtail them to a conservative value.
> Cheers,
>
> Richard
>
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> You automatically follow any topics you start or reply to.
> View/Reply Online (#55404): https://lists.yoctoproject.org/g/yocto/message/55404
> Mute This Topic: https://lists.yoctoproject.org/mt/87260143/1997914
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-11-24 18:21 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-23 15:03 [PATCH yocto-autobuilder-helper] config.json: set ZSTD_THREADS like XZ_THREADS Ross Burton
2021-11-23 17:11 ` [yocto] " Alexander Kanavin
2021-11-23 19:53 ` Ross Burton
2021-11-24 7:27 ` Ernst Sjöstrand
2021-11-24 8:00 ` [yocto] " Alexander Kanavin
[not found] ` <473510c439b0d732d956ab327a425221f6019cc7.camel@linuxfoundation.org>
2021-11-24 18:21 ` Khem Raj
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.