From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75178C433EF for ; Tue, 9 Nov 2021 08:21:47 +0000 (UTC) Received: from mail-ua1-f47.google.com (mail-ua1-f47.google.com [209.85.222.47]) by mx.groups.io with SMTP id smtpd.web10.9222.1636446106760401534 for ; Tue, 09 Nov 2021 00:21:46 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=iZ3NW9Te; spf=pass (domain: gmail.com, ip: 209.85.222.47, mailfrom: alex.kanavin@gmail.com) Received: by mail-ua1-f47.google.com with SMTP id o26so36958404uab.5 for ; Tue, 09 Nov 2021 00:21:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=yN7egtkXVPQcl1WmE8htzQy64xOXMvw2ZIF3jEjd0FE=; b=iZ3NW9Te02oIMb6G7uwEMH754WEkXpoRXEBgxBrFK4/eaxBU7umfiGh99AjVRy+Rsq PuYlsXcxzEu1VylFSGQ1YOrSZnXHrLzm5bSg/eZ0zeG8JtHFW4khGKAGYMRGA0MKGIiv 903Rl6XK62b3WNeJxBT7/P/0nLC5DiknnX5l/0HqVs1Np3lMsAXMk9ep94hk0PqLRCMI qJ4A9SAXQpyA+VwoVVC2MaDpuw3RjnuzZtZCd9x/3gK5sOuOaafqvs+6Jz8vzXOZhHvX MUWSK52pKCSC4F7tCVaK3I3+Ppu6TzOg9nM+66JiXN+L3DH9xfmFZih/ba1slkWBF/Ta WxmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yN7egtkXVPQcl1WmE8htzQy64xOXMvw2ZIF3jEjd0FE=; b=UARLT5q1sHb7z68nfBkgpztV0BaepPO8ncSimIpE6r2I5BF95e05pQLBSTDPTSNCk8 NrHkKZcROjOPwjF7oBRNmspDJ3zkX1Gr7OC7vz6OdE80LW/GXbpc2W1kT/2YXQNXrzg5 M5wQc3v9Vv6JX5RrITXvHynBNN0P9uOH2A3JrvGwwilrX7SBI2CCKaYmA7NYCJ5t6bLt nURgonaCPRUHip/FgEbPB93c4V+1rcTrlwlpg8rOQjp9CuI4xecGXECdGRf9ZbOOwraQ wa4F6QRGaGCICC0Ewh13KwJ2VL0O3Pm3OMX8UHAaVkTXyaE/yMo7hfCVqAeoZld/ua86 Q6hw== X-Gm-Message-State: AOAM530tYY6YkjkFDE7Iqz3kITasD0BOFp9SnlG38PrSyXFnJZTqVgG4 H8G8GuWkBJBIkCMvq7NvzFxDUuJ0gumWsY/Mwqm7p0LC X-Google-Smtp-Source: ABdhPJzcRfUh07NUQ8jSlB4yfYHRIvwwKfC9FGZJlZgl4oaS6vHLVpb5gVzV6MVqmO58eyp3CBuyZo+2GTnazbtKrDI= X-Received: by 2002:a05:6102:b13:: with SMTP id b19mr9627232vst.1.1636446105879; Tue, 09 Nov 2021 00:21:45 -0800 (PST) MIME-Version: 1.0 References: <5e10cddcd2697c7b762c8c8d245bb76f8666aefc.1636447679.git.liezhi.yang@windriver.com> In-Reply-To: <5e10cddcd2697c7b762c8c8d245bb76f8666aefc.1636447679.git.liezhi.yang@windriver.com> From: Alexander Kanavin Date: Tue, 9 Nov 2021 09:21:34 +0100 Message-ID: Subject: Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number To: Robert Yang Cc: OE-core Content-Type: multipart/alternative; boundary="000000000000d5c6dd05d056cc50" List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Tue, 09 Nov 2021 08:21:47 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/158000 --000000000000d5c6dd05d056cc50 Content-Type: text/plain; charset="UTF-8" But does this mean we need to globally restrict this and make things twice slower for everyone? I'm not sure. Alex On Tue, 9 Nov 2021 at 09:12, Robert Yang wrote: > The original value is very easy to cause do_packge error when cpu number is > larger, for example, 128 cores and 512G mem: > > error: create archive failed: cpio: write failed - Cannot allocate memory" > > Set the ZSTD_THREADS to half of the CPU number can avoid the error in my > testing. > > Signed-off-by: Robert Yang > --- > meta/conf/bitbake.conf | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 71c1e52ad6..46ebf5113f 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -833,7 +833,7 @@ XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} > --threads=${XZ_THREADS}" > XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > > # Default parallelism for zstd > -ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > +ZSTD_THREADS ?= "${@int(oe.utils.cpu_count(at_least=4)/2)}" > ZSTD_THREADS[vardepvalue] = "1" > > # Limit the number of threads that OpenMP libraries will use. Otherwise > they > -- > 2.17.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#157998): > https://lists.openembedded.org/g/openembedded-core/message/157998 > Mute This Topic: https://lists.openembedded.org/mt/86926962/1686489 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > alex.kanavin@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > --000000000000d5c6dd05d056cc50 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
But does this mean we need to globally restrict this = and make things twice slower for everyone? I'm not sure.

=

Alex

On Tue, 9 Nov 2021 at 09:12, Robert= Yang <liezhi.yang@windrive= r.com> wrote:
The original value is very easy to cause do_packge error when cpu numb= er is
larger, for example, 128 cores and 512G mem:

error: create archive failed: cpio: write failed - Cannot allocate memory&q= uot;

Set the ZSTD_THREADS to half of the CPU number can avoid the error in my testing.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
=C2=A0meta/conf/bitbake.conf | 2 +-
=C2=A01 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 71c1e52ad6..46ebf5113f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -833,7 +833,7 @@ XZ_DEFAULTS ?=3D "--memlimit=3D${XZ_MEMLIMIT} --th= reads=3D${XZ_THREADS}"
=C2=A0XZ_DEFAULTS[vardepsexclude] +=3D "XZ_MEMLIMIT XZ_THREADS"
=C2=A0# Default parallelism for zstd
-ZSTD_THREADS ?=3D "${@oe.utils.cpu_count(at_least=3D2)}"
+ZSTD_THREADS ?=3D "${@int(oe.utils.cpu_count(at_least=3D4)/2)}"<= br> =C2=A0ZSTD_THREADS[vardepvalue] =3D "1"

=C2=A0# Limit the number of threads that OpenMP libraries will use. Otherwi= se they
--
2.17.1


-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-
Links: You receive all messages sent to this group.
View/Reply Online (#157998): https:= //lists.openembedded.org/g/openembedded-core/message/157998
Mute This Topic: https://lists.openembedded.org/mt= /86926962/1686489
Group Owner: openembedded-core+owner@lists.openembedded.org<= br> Unsubscribe: https://lists.openembedded.org/= g/openembedded-core/unsub [alex.kanavin@gmail.com]
-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-

--000000000000d5c6dd05d056cc50--