* [PATCH 0/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number @ 2021-11-09 8:48 Robert Yang 2021-11-09 8:48 ` [PATCH 1/1] " Robert Yang 0 siblings, 1 reply; 12+ messages in thread From: Robert Yang @ 2021-11-09 8:48 UTC (permalink / raw) To: openembedded-core The following changes since commit afad2f09ebbfe395f8fafce8218e26056479fe39: meson: improve SDK's wrapper to make Wraps work (2021-11-08 22:01:53 +0000) are available in the Git repository at: git://git.openembedded.org/openembedded-core-contrib rbt/zstd http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/zstd Robert Yang (1): bitbake.conf: Set ZSTD_THREADS to half of cpu number meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.17.1 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 8:48 [PATCH 0/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number Robert Yang @ 2021-11-09 8:48 ` Robert Yang 2021-11-09 8:21 ` [OE-core] " Alexander Kanavin ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: Robert Yang @ 2021-11-09 8:48 UTC (permalink / raw) To: openembedded-core The original value is very easy to cause do_packge error when cpu number is larger, for example, 128 cores and 512G mem: error: create archive failed: cpio: write failed - Cannot allocate memory" Set the ZSTD_THREADS to half of the CPU number can avoid the error in my testing. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 71c1e52ad6..46ebf5113f 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" # Default parallelism for zstd -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}" ZSTD_THREADS[vardepvalue] = "1" # Limit the number of threads that OpenMP libraries will use. Otherwise they -- 2.17.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 8:48 ` [PATCH 1/1] " Robert Yang @ 2021-11-09 8:21 ` Alexander Kanavin 2021-11-09 9:45 ` Robert Yang 2021-11-09 8:41 ` Konrad Weihmann 2021-11-09 8:45 ` Jose Quaresma 2 siblings, 1 reply; 12+ messages in thread From: Alexander Kanavin @ 2021-11-09 8:21 UTC (permalink / raw) To: Robert Yang; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 1682 bytes --] But does this mean we need to globally restrict this and make things twice slower for everyone? I'm not sure. Alex On Tue, 9 Nov 2021 at 09:12, Robert Yang <liezhi.yang@windriver.com> wrote: > The original value is very easy to cause do_packge error when cpu number is > larger, for example, 128 cores and 512G mem: > > error: create archive failed: cpio: write failed - Cannot allocate memory" > > Set the ZSTD_THREADS to half of the CPU number can avoid the error in my > testing. > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > --- > meta/conf/bitbake.conf | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 71c1e52ad6..46ebf5113f 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} > --threads=${XZ_THREADS}" > XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > > # Default parallelism for zstd > -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}" > ZSTD_THREADS[vardepvalue] = "1" > > # Limit the number of threads that OpenMP libraries will use. Otherwise > they > -- > 2.17.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#157998): > https://lists.openembedded.org/g/openembedded-core/message/157998 > Mute This Topic: https://lists.openembedded.org/mt/86926962/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > [-- Attachment #2: Type: text/html, Size: 2728 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 8:21 ` [OE-core] " Alexander Kanavin @ 2021-11-09 9:45 ` Robert Yang 0 siblings, 0 replies; 12+ messages in thread From: Robert Yang @ 2021-11-09 9:45 UTC (permalink / raw) To: Alexander Kanavin; +Cc: OE-core Hi Alexander, On 11/9/21 4:21 PM, Alexander Kanavin wrote: > But does this mean we need to globally restrict this and make things twice > slower for everyone? I'm not sure. Please see my reply to Konrad, it doesn't cause any slower, but faster in my testing. BTW, the XZ_MEMLIMIT is 50% for command xz, but there isn't a similar option for zstd, so I have to limit ZSTD_THREADS. // Robert > > > Alex > > On Tue, 9 Nov 2021 at 09:12, Robert Yang <liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com>> wrote: > > The original value is very easy to cause do_packge error when cpu number is > larger, for example, 128 cores and 512G mem: > > error: create archive failed: cpio: write failed - Cannot allocate memory" > > Set the ZSTD_THREADS to half of the CPU number can avoid the error in my > testing. > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com>> > --- > meta/conf/bitbake.conf | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 71c1e52ad6..46ebf5113f 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} > --threads=${XZ_THREADS}" > XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > > # Default parallelism for zstd > -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}" > ZSTD_THREADS[vardepvalue] = "1" > > # Limit the number of threads that OpenMP libraries will use. Otherwise they > -- > 2.17.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#157998): > https://lists.openembedded.org/g/openembedded-core/message/157998 > <https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/message/157998__;!!AjveYdw8EvQ!JYlw1H6vGmlFiU7pGKOBAwXZBD-wGB5MiKIahgkPJl3ojXBUC4Rkt3vdHS9iN8a9RUvg$> > Mute This Topic: https://lists.openembedded.org/mt/86926962/1686489 > <https://urldefense.com/v3/__https://lists.openembedded.org/mt/86926962/1686489__;!!AjveYdw8EvQ!JYlw1H6vGmlFiU7pGKOBAwXZBD-wGB5MiKIahgkPJl3ojXBUC4Rkt3vdHS9iN7BIPv2n$> > Group Owner: openembedded-core+owner@lists.openembedded.org > <mailto:openembedded-core%2Bowner@lists.openembedded.org> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub > <https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/unsub__;!!AjveYdw8EvQ!JYlw1H6vGmlFiU7pGKOBAwXZBD-wGB5MiKIahgkPJl3ojXBUC4Rkt3vdHS9iN_MzaRNb$> > [alex.kanavin@gmail.com <mailto:alex.kanavin@gmail.com>] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 8:48 ` [PATCH 1/1] " Robert Yang 2021-11-09 8:21 ` [OE-core] " Alexander Kanavin @ 2021-11-09 8:41 ` Konrad Weihmann 2021-11-09 9:40 ` Robert Yang 2021-11-09 8:45 ` Jose Quaresma 2 siblings, 1 reply; 12+ messages in thread From: Konrad Weihmann @ 2021-11-09 8:41 UTC (permalink / raw) To: Robert Yang, openembedded-core On 09.11.21 09:48, Robert Yang wrote: > The original value is very easy to cause do_packge error when cpu number is > larger, for example, 128 cores and 512G mem: > > error: create archive failed: cpio: write failed - Cannot allocate memory" > > Set the ZSTD_THREADS to half of the CPU number can avoid the error in my > testing. > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > --- > meta/conf/bitbake.conf | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 71c1e52ad6..46ebf5113f 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" > XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > > # Default parallelism for zstd > -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}" Then why not just limit it for the large setups you are referring to in the example, for instance like ZSTD_THREADS ?= "${@min(int(oe.utils.cpu_count(at_least=4)), <insert a sane value of your choice>)}" BTW this can be also done in your local.conf - as Alex already said, there is simply no reason to make it slower for everyone, while also half the threads of a 128 core machine could be a trouble some setup. Last time I had this issue (with XZ back then) I used "${@min(int(oe.utils.cpu_count(at_least=4)), 20}" > ZSTD_THREADS[vardepvalue] = "1" > > # Limit the number of threads that OpenMP libraries will use. Otherwise they > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#157998): https://lists.openembedded.org/g/openembedded-core/message/157998 > Mute This Topic: https://lists.openembedded.org/mt/86926962/3647476 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [kweihmann@outlook.com] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 8:41 ` Konrad Weihmann @ 2021-11-09 9:40 ` Robert Yang 2021-11-09 9:44 ` Alexander Kanavin 0 siblings, 1 reply; 12+ messages in thread From: Robert Yang @ 2021-11-09 9:40 UTC (permalink / raw) To: Konrad Weihmann, openembedded-core On 11/9/21 4:41 PM, Konrad Weihmann wrote: > > > On 09.11.21 09:48, Robert Yang wrote: >> The original value is very easy to cause do_packge error when cpu number is >> larger, for example, 128 cores and 512G mem: >> >> error: create archive failed: cpio: write failed - Cannot allocate memory" >> >> Set the ZSTD_THREADS to half of the CPU number can avoid the error in my >> testing. >> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >> --- >> meta/conf/bitbake.conf | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf >> index 71c1e52ad6..46ebf5113f 100644 >> --- a/meta/conf/bitbake.conf >> +++ b/meta/conf/bitbake.conf >> @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} >> --threads=${XZ_THREADS}" >> XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" >> # Default parallelism for zstd >> -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" >> +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}" > > Then why not just limit it for the large setups you are referring to in the > example, for instance like > > ZSTD_THREADS ?= "${@min(int(oe.utils.cpu_count(at_least=4)), <insert a sane > value of your choice>)}" This sounds like a good choice. > > BTW this can be also done in your local.conf - as Alex already said, there is Set it in my local.conf can make it work, but it isn't good for oe-core's OOBE since the issue still exists. > simply no reason to make it slower for everyone, while also half the threads of The whole build contains a lot of tasks, for lower cpu cores, the total resources are fixed, so when zstd uses less mem, other tasks can use more. Limit zstd's thread doesn't make the build slower, but faster in my testing, it would be great if you can help test it. The simple testing commands are: # Make everything ready: $ bitbake linux-yocto # Before the patch, set ZSTD_THREADS to 128 and run 6 times: $ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f > before_$i.log; done Note, the time will be printed on the screen, not the before.log, the before.log is for strip the logs and then we can read the time easier. What I get is: real 2m12.079s real 2m0.177s real 1m52.426s real 2m3.396s real 2m16.018s real 1m58.595s Drop the first build time since it *may* contain parsing time, and for the last five builds: Total: 609 seconds Average: 609 / 5.0 = 121.8 # After the patch, set ZSTD_THREADS to 64, and run 6 times: $ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f > after_$i.log; done What I get is: real 1m50.017s real 1m50.400s real 1m53.174s real 2m4.817s real 1m53.476s real 1m56.794s Drop the first build time since it *may* contain parsing time, and for the last five builds: Total: 576 seconds Average: 576 / 5.0 = 115.2 So the smaller number is faster than the larger one. // Robert > a 128 core machine could be a trouble some setup. > > Last time I had this issue (with XZ back then) I used > "${@min(int(oe.utils.cpu_count(at_least=4)), 20}" > >> ZSTD_THREADS[vardepvalue] = "1" >> # Limit the number of threads that OpenMP libraries will use. Otherwise they >> >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> View/Reply Online (#157998): >> https://lists.openembedded.org/g/openembedded-core/message/157998 >> Mute This Topic: https://lists.openembedded.org/mt/86926962/3647476 >> Group Owner: openembedded-core+owner@lists.openembedded.org >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub >> [kweihmann@outlook.com] >> -=-=-=-=-=-=-=-=-=-=-=- >> ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 9:40 ` Robert Yang @ 2021-11-09 9:44 ` Alexander Kanavin 2021-11-09 9:51 ` Robert Yang 0 siblings, 1 reply; 12+ messages in thread From: Alexander Kanavin @ 2021-11-09 9:44 UTC (permalink / raw) To: Robert Yang; +Cc: Konrad Weihmann, OE-core [-- Attachment #1: Type: text/plain, Size: 4374 bytes --] Maybe, but once we start doing tweaks like this, where do they stop? Hey, I'd like to limit llvm and webkit threads too, they eat disproportionally large amounts of RAM compared to everything else! I'd rather err on the side of simple, striaghtforward, universal defaults, and having better documentation for tweaking system resource consumption. Each build system is different. Alex On Tue, 9 Nov 2021 at 10:40, Robert Yang <liezhi.yang@windriver.com> wrote: > > > On 11/9/21 4:41 PM, Konrad Weihmann wrote: > > > > > > On 09.11.21 09:48, Robert Yang wrote: > >> The original value is very easy to cause do_packge error when cpu > number is > >> larger, for example, 128 cores and 512G mem: > >> > >> error: create archive failed: cpio: write failed - Cannot allocate > memory" > >> > >> Set the ZSTD_THREADS to half of the CPU number can avoid the error in my > >> testing. > >> > >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > >> --- > >> meta/conf/bitbake.conf | 2 +- > >> 1 file changed, 1 insertion(+), 1 deletion(-) > >> > >> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > >> index 71c1e52ad6..46ebf5113f 100644 > >> --- a/meta/conf/bitbake.conf > >> +++ b/meta/conf/bitbake.conf > >> @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} > >> --threads=${XZ_THREADS}" > >> XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > >> # Default parallelism for zstd > >> -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > >> +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}" > > > > Then why not just limit it for the large setups you are referring to in > the > > example, for instance like > > > > ZSTD_THREADS ?= "${@min(int(oe.utils.cpu_count(at_least=4)), <insert a > sane > > value of your choice>)}" > > This sounds like a good choice. > > > > > BTW this can be also done in your local.conf - as Alex already said, > there is > > Set it in my local.conf can make it work, but it isn't good for oe-core's > OOBE > since the issue still exists. > > > simply no reason to make it slower for everyone, while also half the > threads of > > The whole build contains a lot of tasks, for lower cpu cores, the total > resources are fixed, so when zstd uses less mem, other tasks can use more. > > Limit zstd's thread doesn't make the build slower, but faster in my > testing, > it would be great if you can help test it. The simple testing commands are: > > # Make everything ready: > $ bitbake linux-yocto > > # Before the patch, set ZSTD_THREADS to 128 and run 6 times: > $ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f > > before_$i.log; done > > Note, the time will be printed on the screen, not the before.log, the > before.log > is for strip the logs and then we can read the time easier. > > What I get is: > real 2m12.079s > real 2m0.177s > real 1m52.426s > real 2m3.396s > real 2m16.018s > real 1m58.595s > > Drop the first build time since it *may* contain parsing time, and for the > last > five builds: > > Total: 609 seconds > Average: 609 / 5.0 = 121.8 > > > # After the patch, set ZSTD_THREADS to 64, and run 6 times: > $ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f > > after_$i.log; done > > What I get is: > real 1m50.017s > real 1m50.400s > real 1m53.174s > real 2m4.817s > real 1m53.476s > real 1m56.794s > > Drop the first build time since it *may* contain parsing time, and for the > last > five builds: > > Total: 576 seconds > Average: 576 / 5.0 = 115.2 > > So the smaller number is faster than the larger one. > > // Robert > > > a 128 core machine could be a trouble some setup. > > > > Last time I had this issue (with XZ back then) I used > > "${@min(int(oe.utils.cpu_count(at_least=4)), 20}" > > > >> ZSTD_THREADS[vardepvalue] = "1" > >> # Limit the number of threads that OpenMP libraries will use. > Otherwise they > >> > >> > >> > >> > >> > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#158004): > https://lists.openembedded.org/g/openembedded-core/message/158004 > Mute This Topic: https://lists.openembedded.org/mt/86926962/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > [-- Attachment #2: Type: text/html, Size: 5973 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 9:44 ` Alexander Kanavin @ 2021-11-09 9:51 ` Robert Yang 2021-11-09 9:56 ` Alexander Kanavin 0 siblings, 1 reply; 12+ messages in thread From: Robert Yang @ 2021-11-09 9:51 UTC (permalink / raw) To: Alexander Kanavin; +Cc: Konrad Weihmann, OE-core Hi Alexander, On 11/9/21 5:44 PM, Alexander Kanavin wrote: > Maybe, but once we start doing tweaks like this, where do they stop? Hey, I'd I think that it's not only a performance tweak, but this should be considered as a bug since it would be failed to build on powerful machine, which isn't good for oe-core's user experience. // Robert > like to limit llvm and webkit threads too, they eat disproportionally large > amounts of RAM compared to everything else! I'd rather err on the side of > simple, striaghtforward, universal defaults, and having better documentation for > tweaking system resource consumption. Each build system is different. > > Alex > > On Tue, 9 Nov 2021 at 10:40, Robert Yang <liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com>> wrote: > > > > On 11/9/21 4:41 PM, Konrad Weihmann wrote: > > > > > > On 09.11.21 09:48, Robert Yang wrote: > >> The original value is very easy to cause do_packge error when cpu number is > >> larger, for example, 128 cores and 512G mem: > >> > >> error: create archive failed: cpio: write failed - Cannot allocate memory" > >> > >> Set the ZSTD_THREADS to half of the CPU number can avoid the error in my > >> testing. > >> > >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com > <mailto:liezhi.yang@windriver.com>> > >> --- > >> meta/conf/bitbake.conf | 2 +- > >> 1 file changed, 1 insertion(+), 1 deletion(-) > >> > >> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > >> index 71c1e52ad6..46ebf5113f 100644 > >> --- a/meta/conf/bitbake.conf > >> +++ b/meta/conf/bitbake.conf > >> @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} > >> --threads=${XZ_THREADS}" > >> XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > >> # Default parallelism for zstd > >> -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > >> +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}" > > > > Then why not just limit it for the large setups you are referring to in the > > example, for instance like > > > > ZSTD_THREADS ?= "${@min(int(oe.utils.cpu_count(at_least=4)), <insert a sane > > value of your choice>)}" > > This sounds like a good choice. > > > > > BTW this can be also done in your local.conf - as Alex already said, > there is > > Set it in my local.conf can make it work, but it isn't good for oe-core's OOBE > since the issue still exists. > > > simply no reason to make it slower for everyone, while also half the > threads of > > The whole build contains a lot of tasks, for lower cpu cores, the total > resources are fixed, so when zstd uses less mem, other tasks can use more. > > Limit zstd's thread doesn't make the build slower, but faster in my testing, > it would be great if you can help test it. The simple testing commands are: > > # Make everything ready: > $ bitbake linux-yocto > > # Before the patch, set ZSTD_THREADS to 128 and run 6 times: > $ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f > > before_$i.log; done > > Note, the time will be printed on the screen, not the before.log, the before.log > is for strip the logs and then we can read the time easier. > > What I get is: > real 2m12.079s > real 2m0.177s > real 1m52.426s > real 2m3.396s > real 2m16.018s > real 1m58.595s > > Drop the first build time since it *may* contain parsing time, and for the last > five builds: > > Total: 609 seconds > Average: 609 / 5.0 = 121.8 > > > # After the patch, set ZSTD_THREADS to 64, and run 6 times: > $ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f > > after_$i.log; done > > What I get is: > real 1m50.017s > real 1m50.400s > real 1m53.174s > real 2m4.817s > real 1m53.476s > real 1m56.794s > > Drop the first build time since it *may* contain parsing time, and for the last > five builds: > > Total: 576 seconds > Average: 576 / 5.0 = 115.2 > > So the smaller number is faster than the larger one. > > // Robert > > > a 128 core machine could be a trouble some setup. > > > > Last time I had this issue (with XZ back then) I used > > "${@min(int(oe.utils.cpu_count(at_least=4)), 20}" > > > >> ZSTD_THREADS[vardepvalue] = "1" > >> # Limit the number of threads that OpenMP libraries will use. > Otherwise they > >> > >> > >> > >> > >> > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#158004): > https://lists.openembedded.org/g/openembedded-core/message/158004 > <https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/message/158004__;!!AjveYdw8EvQ!Pp09fG5njytGigcrK1Pci9c4pmhLI2WooNDvgaDRdaq-cC5BmErvgdfHBe3ivNZBDAzT$> > Mute This Topic: https://lists.openembedded.org/mt/86926962/1686489 > <https://urldefense.com/v3/__https://lists.openembedded.org/mt/86926962/1686489__;!!AjveYdw8EvQ!Pp09fG5njytGigcrK1Pci9c4pmhLI2WooNDvgaDRdaq-cC5BmErvgdfHBe3ivG7_2fs8$> > Group Owner: openembedded-core+owner@lists.openembedded.org > <mailto:openembedded-core%2Bowner@lists.openembedded.org> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub > <https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/unsub__;!!AjveYdw8EvQ!Pp09fG5njytGigcrK1Pci9c4pmhLI2WooNDvgaDRdaq-cC5BmErvgdfHBe3ivAnK9lkk$> > [alex.kanavin@gmail.com <mailto:alex.kanavin@gmail.com>] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 9:51 ` Robert Yang @ 2021-11-09 9:56 ` Alexander Kanavin 2021-11-09 10:26 ` Richard Purdie 0 siblings, 1 reply; 12+ messages in thread From: Alexander Kanavin @ 2021-11-09 9:56 UTC (permalink / raw) To: Robert Yang; +Cc: Konrad Weihmann, OE-core [-- Attachment #1: Type: text/plain, Size: 819 bytes --] On Tue, 9 Nov 2021 at 10:51, Robert Yang <liezhi.yang@windriver.com> wrote: > > Maybe, but once we start doing tweaks like this, where do they stop? > Hey, I'd > > I think that it's not only a performance tweak, but this should be > considered as > a bug since it would be failed to build on powerful machine, which isn't > good > for oe-core's user experience. > I managed to bring down a powerful machine just yesterday by building webkit and llvm together at the same time. They together ate all the RAM, and someone in the office had to go and reset the box. Do I see it as a bug? Absolutely not: it's in fact my fault for not ensuring system resource consumption doesn't overwhelm the machine. This is the same: if you do not have RAM to match the cores on your specific rig, add RAM or limit the threads. Alex [-- Attachment #2: Type: text/html, Size: 1251 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 9:56 ` Alexander Kanavin @ 2021-11-09 10:26 ` Richard Purdie 2021-11-11 4:09 ` Robert Yang 0 siblings, 1 reply; 12+ messages in thread From: Richard Purdie @ 2021-11-09 10:26 UTC (permalink / raw) To: Alexander Kanavin, Robert Yang; +Cc: Konrad Weihmann, OE-core On Tue, 2021-11-09 at 10:56 +0100, Alexander Kanavin wrote: > On Tue, 9 Nov 2021 at 10:51, Robert Yang <liezhi.yang@windriver.com> wrote: > > > Maybe, but once we start doing tweaks like this, where do they stop? Hey, > > I'd > > > > I think that it's not only a performance tweak, but this should be > > considered > > as > > a bug since it would be failed to build on powerful machine, which isn't > > good > > for oe-core's user experience. > > > > > I managed to bring down a powerful machine just yesterday by building webkit > and llvm together at the same time. They together ate all the RAM, and someone > in the office had to go and reset the box. > Do I see it as a bug? Absolutely not: it's in fact my fault for not ensuring > system resource consumption doesn't overwhelm the machine. This is the same: > if you do not have RAM to match the cores on your specific rig, add RAM or > limit the threads. I can see both sides of this. If we had a pool for all the threads and the pieces of the system all used the same pool, it would work well as implemented. Sadly the bitbake threads, the make processes and now the compression threads are all separate. We do see the autobuilder break due to load issues too and on that we have scaled back some of the defaults but not ZSTD as yet that I recall. As such I think limiting the upper value on number of threads may be a good compromise, both for compression and maybe for parallel make too. Cheers, Richard ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 10:26 ` Richard Purdie @ 2021-11-11 4:09 ` Robert Yang 0 siblings, 0 replies; 12+ messages in thread From: Robert Yang @ 2021-11-11 4:09 UTC (permalink / raw) To: Richard Purdie, Alexander Kanavin; +Cc: Konrad Weihmann, OE-core Hi RP, On 11/9/21 6:26 PM, Richard Purdie wrote: > On Tue, 2021-11-09 at 10:56 +0100, Alexander Kanavin wrote: >> On Tue, 9 Nov 2021 at 10:51, Robert Yang <liezhi.yang@windriver.com> wrote: >>>> Maybe, but once we start doing tweaks like this, where do they stop? Hey, >>> I'd >>> >>> I think that it's not only a performance tweak, but this should be >>> considered >>> as >>> a bug since it would be failed to build on powerful machine, which isn't >>> good >>> for oe-core's user experience. >>> >> >> >> I managed to bring down a powerful machine just yesterday by building webkit >> and llvm together at the same time. They together ate all the RAM, and someone >> in the office had to go and reset the box. >> Do I see it as a bug? Absolutely not: it's in fact my fault for not ensuring >> system resource consumption doesn't overwhelm the machine. This is the same: >> if you do not have RAM to match the cores on your specific rig, add RAM or >> limit the threads. > > I can see both sides of this. If we had a pool for all the threads and the > pieces of the system all used the same pool, it would work well as implemented. > Sadly the bitbake threads, the make processes and now the compression threads > are all separate. > > We do see the autobuilder break due to load issues too and on that we have > scaled back some of the defaults but not ZSTD as yet that I recall. > > As such I think limiting the upper value on number of threads may be a good > compromise, both for compression and maybe for parallel make too. What should we do next, please? Limit bb thread + parallel make + zstd thread? Or just limit zstd's thread since other people may concern about the performance. // Robert > > Cheers, > > Richard > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number 2021-11-09 8:48 ` [PATCH 1/1] " Robert Yang 2021-11-09 8:21 ` [OE-core] " Alexander Kanavin 2021-11-09 8:41 ` Konrad Weihmann @ 2021-11-09 8:45 ` Jose Quaresma 2 siblings, 0 replies; 12+ messages in thread From: Jose Quaresma @ 2021-11-09 8:45 UTC (permalink / raw) To: Robert Yang; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 1730 bytes --] Robert Yang <liezhi.yang@windriver.com> escreveu no dia terça, 9/11/2021 à(s) 08:12: > The original value is very easy to cause do_packge error when cpu number is > larger, for example, 128 cores and 512G mem: > > error: create archive failed: cpio: write failed - Cannot allocate memory" > > Set the ZSTD_THREADS to half of the CPU number can avoid the error in my > testing. > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > --- > meta/conf/bitbake.conf | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 71c1e52ad6..46ebf5113f 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} > --threads=${XZ_THREADS}" > XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > > # Default parallelism for zstd > -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}" > This will slow down all the systems with lower cpu cores. > ZSTD_THREADS[vardepvalue] = "1" > > # Limit the number of threads that OpenMP libraries will use. Otherwise > they > -- > 2.17.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#157998): > https://lists.openembedded.org/g/openembedded-core/message/157998 > Mute This Topic: https://lists.openembedded.org/mt/86926962/5052612 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > -- Best regards, José Quaresma [-- Attachment #2: Type: text/html, Size: 2970 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-11-11 4:09 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-11-09 8:48 [PATCH 0/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number Robert Yang 2021-11-09 8:48 ` [PATCH 1/1] " Robert Yang 2021-11-09 8:21 ` [OE-core] " Alexander Kanavin 2021-11-09 9:45 ` Robert Yang 2021-11-09 8:41 ` Konrad Weihmann 2021-11-09 9:40 ` Robert Yang 2021-11-09 9:44 ` Alexander Kanavin 2021-11-09 9:51 ` Robert Yang 2021-11-09 9:56 ` Alexander Kanavin 2021-11-09 10:26 ` Richard Purdie 2021-11-11 4:09 ` Robert Yang 2021-11-09 8:45 ` Jose Quaresma
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.