[PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums

[PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums

[Richard Purdie]

rpm output packages currently depend on the value of XZ_THREADS which
is ok if left to the default value but problematic if system limits
are set such as on the autobuilder.

Force the value to a specific one in the hashes for better sstate reuse
and consistent rpm task checksums.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/conf/bitbake.conf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 2140d498f7c..e25d632dc1c 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -811,6 +811,7 @@ PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
 # Default parallelism and resource usage for xz
 XZ_MEMLIMIT ?= "50%"
 XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
+XZ_THREADS[vardepvalue] = "1"
 XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
 XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 
-- 
2.32.0

[PATCH 2/2] sstate: Avoid problems with recipes using SRCPV when fetching sstate

[Richard Purdie]

When fetching sstate, SRCPV can be expanded withuin PV which breaks the
sstate fetcher as other variables like SRC_URI are changed by the fetcher
code. Expand the variable before doing this to avoid those problems and
hence avoid setscene task failures with remote sstate storage in git
recipes.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/sstate.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index d7f1b3f26ed..62e45cb4a8b 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -731,6 +731,7 @@ def pstaging_fetch(sstatefetch, d):
     localdata.setVar('FILESPATH', dldir)
     localdata.setVar('DL_DIR', dldir)
     localdata.setVar('PREMIRRORS', mirrors)
+    localdata.setVar('SRCPV', d.getVar('SRCPV'))
 
     # if BB_NO_NETWORK is set but we also have SSTATE_MIRROR_ALLOW_NETWORK,
     # we'll want to allow network access for the current set of fetches.
@@ -755,6 +756,9 @@ def pstaging_fetch(sstatefetch, d):
         except bb.fetch2.BBFetchException:
             pass
 
+pstaging_fetch[vardepsexclude] += "SRCPV"
+
+
 def sstate_setscene(d):
     shared_state = sstate_state_fromvars(d)
     accelerate = sstate_installpkg(shared_state, d)
-- 
2.32.0

Re: [OE-core] [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums

[Mike Looijmans]

Wouldn't "0" be a better default? Setting threads to "0" will make xz use the 
CPU count anyway, so it'd be honest too.



Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topic.nl

Please consider the environment before printing this e-mail
On 11-09-2021 12:04, Richard Purdie via lists.openembedded.org wrote:
> rpm output packages currently depend on the value of XZ_THREADS which
> is ok if left to the default value but problematic if system limits
> are set such as on the autobuilder.
> 
> Force the value to a specific one in the hashes for better sstate reuse
> and consistent rpm task checksums.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>   meta/conf/bitbake.conf | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 2140d498f7c..e25d632dc1c 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -811,6 +811,7 @@ PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
>   # Default parallelism and resource usage for xz
>   XZ_MEMLIMIT ?= "50%"
>   XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> +XZ_THREADS[vardepvalue] = "1"
>   XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
>   XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
>   
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#155939): https://lists.openembedded.org/g/openembedded-core/message/155939
> Mute This Topic: https://lists.openembedded.org/mt/85529700/3618446
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [mike.looijmans@topic.nl]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [OE-core] [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums

[Jose Quaresma]

[-- Attachment #1: Type: text/plain, Size: 2351 bytes --]

Hi Mike,

But who is this calculated ?
For example, if we are inside of a container the XZ will use the host CPU
available or the scheduler affinity.

Mike Looijmans <mike.looijmans@topic.nl> escreveu no dia quinta, 30/09/2021
à(s) 07:50:

> Wouldn't "0" be a better default? Setting threads to "0" will make xz use
> the
> CPU count anyway, so it'd be honest too.
>
>
>
> Met vriendelijke groet / kind regards,
>
> Mike Looijmans
> System Expert
>
>
> TOPIC Embedded Products B.V.
> Materiaalweg 4, 5681 RJ Best
> The Netherlands
>
> T: +31 (0) 499 33 69 69
> E: mike.looijmans@topicproducts.com
> W: www.topic.nl
>
> Please consider the environment before printing this e-mail
> On 11-09-2021 12:04, Richard Purdie via lists.openembedded.org wrote:
> > rpm output packages currently depend on the value of XZ_THREADS which
> > is ok if left to the default value but problematic if system limits
> > are set such as on the autobuilder.
> >
> > Force the value to a specific one in the hashes for better sstate reuse
> > and consistent rpm task checksums.
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >   meta/conf/bitbake.conf | 1 +
> >   1 file changed, 1 insertion(+)
> >
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index 2140d498f7c..e25d632dc1c 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -811,6 +811,7 @@ PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
> >   # Default parallelism and resource usage for xz
> >   XZ_MEMLIMIT ?= "50%"
> >   XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> > +XZ_THREADS[vardepvalue] = "1"
> >   XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
> >   XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> >
> >
> >
> >
> >
> >
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#156483):
> https://lists.openembedded.org/g/openembedded-core/message/156483
> Mute This Topic: https://lists.openembedded.org/mt/85529700/5052612
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> quaresma.jose@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

-- 
Best regards,

José Quaresma

[-- Attachment #2: Type: text/html, Size: 3862 bytes --]

Re: [OE-core] [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums

[Richard Purdie]

On Thu, 2021-09-30 at 08:50 +0200, Mike Looijmans wrote:
> Wouldn't "0" be a better default? Setting threads to "0" will make xz use the 
> CPU count anyway, so it'd be honest too.

On some systems we have this set to a specific value rather than auto so any
value is potentially dishonest. I think I went for "1" as it was clearly the
least likely and more obvious when looking at the sig files to see if it was
working.

At the end of the day it doesn't really matter as long as it is a specific
value.

Cheers,

Richard