Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Alexander Kanavin <alex.kanavin@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4374 bytes --]

Maybe, but once we start doing tweaks like this, where do they stop? Hey,
I'd like to limit llvm and webkit threads too, they eat disproportionally
large amounts of RAM compared to everything else! I'd rather err on the
side of simple, striaghtforward, universal defaults, and having better
documentation for tweaking system resource consumption. Each build system
is different.

Alex

On Tue, 9 Nov 2021 at 10:40, Robert Yang <liezhi.yang@windriver.com> wrote:

>
>
> On 11/9/21 4:41 PM, Konrad Weihmann wrote:
> >
> >
> > On 09.11.21 09:48, Robert Yang wrote:
> >> The original value is very easy to cause do_packge error when cpu
> number is
> >> larger, for example, 128 cores and 512G mem:
> >>
> >> error: create archive failed: cpio: write failed - Cannot allocate
> memory"
> >>
> >> Set the ZSTD_THREADS to half of the CPU number can avoid the error in my
> >> testing.
> >>
> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> >> ---
> >>   meta/conf/bitbake.conf | 2 +-
> >>   1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> >> index 71c1e52ad6..46ebf5113f 100644
> >> --- a/meta/conf/bitbake.conf
> >> +++ b/meta/conf/bitbake.conf
> >> @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT}
> >> --threads=${XZ_THREADS}"
> >>   XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> >>   # Default parallelism for zstd
> >> -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> >> +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}"
> >
> > Then why not just limit it for the large setups you are referring to in
> the
> > example, for instance like
> >
> > ZSTD_THREADS ?= "${@min(int(oe.utils.cpu_count(at_least=4)), <insert a
> sane
> > value of your choice>)}"
>
> This sounds like a good choice.
>
> >
> > BTW this can be also done in your local.conf - as Alex already said,
> there is
>
> Set it in my local.conf can make it work, but it isn't good for oe-core's
> OOBE
> since the issue still exists.
>
> > simply no reason to make it slower for everyone, while also half the
> threads of
>
> The whole build contains a lot of tasks, for lower cpu cores, the total
> resources are fixed, so when zstd uses less mem, other tasks can use more.
>
> Limit zstd's thread doesn't make the build slower, but faster in my
> testing,
> it would be great if you can help test it. The simple testing commands are:
>
> # Make everything ready:
> $ bitbake linux-yocto
>
> # Before the patch, set ZSTD_THREADS to 128 and run 6 times:
> $ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f >
> before_$i.log; done
>
> Note, the time will be printed on the screen, not the before.log, the
> before.log
> is for strip the logs and then we can read the time easier.
>
> What I get is:
> real    2m12.079s
> real    2m0.177s
> real    1m52.426s
> real    2m3.396s
> real    2m16.018s
> real    1m58.595s
>
> Drop the first build time since it *may* contain parsing time, and for the
> last
> five builds:
>
> Total: 609 seconds
> Average: 609 / 5.0 = 121.8
>
>
> # After the patch, set ZSTD_THREADS to 64, and run 6 times:
> $ for i in `seq 0 5`; do time bitbake linux-yocto -cpackage_write_rpm -f >
> after_$i.log; done
>
> What I get is:
> real    1m50.017s
> real    1m50.400s
> real    1m53.174s
> real    2m4.817s
> real    1m53.476s
> real    1m56.794s
>
> Drop the first build time since it *may* contain parsing time, and for the
> last
> five builds:
>
> Total: 576 seconds
> Average: 576 / 5.0 = 115.2
>
> So the smaller number is faster than the larger one.
>
> // Robert
>
> > a 128 core machine could be a trouble some setup.
> >
> > Last time I had this issue (with XZ back then) I used
> > "${@min(int(oe.utils.cpu_count(at_least=4)), 20}"
> >
> >>   ZSTD_THREADS[vardepvalue] = "1"
> >>   # Limit the number of threads that OpenMP libraries will use.
> Otherwise they
> >>
> >>
> >>
> >>
> >>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#158004):
> https://lists.openembedded.org/g/openembedded-core/message/158004
> Mute This Topic: https://lists.openembedded.org/mt/86926962/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 5973 bytes --]