* [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums @ 2021-09-11 10:04 Richard Purdie 2021-09-11 10:04 ` [PATCH 2/2] sstate: Avoid problems with recipes using SRCPV when fetching sstate Richard Purdie [not found] ` <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.949ef384-8293-46b8-903f-40a477c056ae.d232f846-51ae-43c0-85e8-be1b07245956@emailsignatures365.codetwo.com> 0 siblings, 2 replies; 5+ messages in thread From: Richard Purdie @ 2021-09-11 10:04 UTC (permalink / raw) To: openembedded-core rpm output packages currently depend on the value of XZ_THREADS which is ok if left to the default value but problematic if system limits are set such as on the autobuilder. Force the value to a specific one in the hashes for better sstate reuse and consistent rpm task checksums. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> --- meta/conf/bitbake.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 2140d498f7c..e25d632dc1c 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -811,6 +811,7 @@ PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" # Default parallelism and resource usage for xz XZ_MEMLIMIT ?= "50%" XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" +XZ_THREADS[vardepvalue] = "1" XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" -- 2.32.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] sstate: Avoid problems with recipes using SRCPV when fetching sstate 2021-09-11 10:04 [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums Richard Purdie @ 2021-09-11 10:04 ` Richard Purdie [not found] ` <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.949ef384-8293-46b8-903f-40a477c056ae.d232f846-51ae-43c0-85e8-be1b07245956@emailsignatures365.codetwo.com> 1 sibling, 0 replies; 5+ messages in thread From: Richard Purdie @ 2021-09-11 10:04 UTC (permalink / raw) To: openembedded-core When fetching sstate, SRCPV can be expanded withuin PV which breaks the sstate fetcher as other variables like SRC_URI are changed by the fetcher code. Expand the variable before doing this to avoid those problems and hence avoid setscene task failures with remote sstate storage in git recipes. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> --- meta/classes/sstate.bbclass | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index d7f1b3f26ed..62e45cb4a8b 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -731,6 +731,7 @@ def pstaging_fetch(sstatefetch, d): localdata.setVar('FILESPATH', dldir) localdata.setVar('DL_DIR', dldir) localdata.setVar('PREMIRRORS', mirrors) + localdata.setVar('SRCPV', d.getVar('SRCPV')) # if BB_NO_NETWORK is set but we also have SSTATE_MIRROR_ALLOW_NETWORK, # we'll want to allow network access for the current set of fetches. @@ -755,6 +756,9 @@ def pstaging_fetch(sstatefetch, d): except bb.fetch2.BBFetchException: pass +pstaging_fetch[vardepsexclude] += "SRCPV" + + def sstate_setscene(d): shared_state = sstate_state_fromvars(d) accelerate = sstate_installpkg(shared_state, d) -- 2.32.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
[parent not found: <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.949ef384-8293-46b8-903f-40a477c056ae.d232f846-51ae-43c0-85e8-be1b07245956@emailsignatures365.codetwo.com>]
[parent not found: <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.0d2bd5fa-15cc-4b27-b94e-83614f9e5b38.5a416b0f-56e3-45c8-b87b-0760607e2824@emailsignatures365.codetwo.com>]
* Re: [OE-core] [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums [not found] ` <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.0d2bd5fa-15cc-4b27-b94e-83614f9e5b38.5a416b0f-56e3-45c8-b87b-0760607e2824@emailsignatures365.codetwo.com> @ 2021-09-30 6:50 ` Mike Looijmans 2021-09-30 8:38 ` Jose Quaresma 2021-09-30 9:47 ` Richard Purdie 0 siblings, 2 replies; 5+ messages in thread From: Mike Looijmans @ 2021-09-30 6:50 UTC (permalink / raw) To: richard.purdie, openembedded-core Wouldn't "0" be a better default? Setting threads to "0" will make xz use the CPU count anyway, so it'd be honest too. Met vriendelijke groet / kind regards, Mike Looijmans System Expert TOPIC Embedded Products B.V. Materiaalweg 4, 5681 RJ Best The Netherlands T: +31 (0) 499 33 69 69 E: mike.looijmans@topicproducts.com W: www.topic.nl Please consider the environment before printing this e-mail On 11-09-2021 12:04, Richard Purdie via lists.openembedded.org wrote: > rpm output packages currently depend on the value of XZ_THREADS which > is ok if left to the default value but problematic if system limits > are set such as on the autobuilder. > > Force the value to a specific one in the hashes for better sstate reuse > and consistent rpm task checksums. > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > meta/conf/bitbake.conf | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 2140d498f7c..e25d632dc1c 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -811,6 +811,7 @@ PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" > # Default parallelism and resource usage for xz > XZ_MEMLIMIT ?= "50%" > XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > +XZ_THREADS[vardepvalue] = "1" > XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" > XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#155939): https://lists.openembedded.org/g/openembedded-core/message/155939 > Mute This Topic: https://lists.openembedded.org/mt/85529700/3618446 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [mike.looijmans@topic.nl] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums 2021-09-30 6:50 ` [OE-core] [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums Mike Looijmans @ 2021-09-30 8:38 ` Jose Quaresma 2021-09-30 9:47 ` Richard Purdie 1 sibling, 0 replies; 5+ messages in thread From: Jose Quaresma @ 2021-09-30 8:38 UTC (permalink / raw) To: Mike Looijmans; +Cc: Richard Purdie, OE-core [-- Attachment #1: Type: text/plain, Size: 2351 bytes --] Hi Mike, But who is this calculated ? For example, if we are inside of a container the XZ will use the host CPU available or the scheduler affinity. Mike Looijmans <mike.looijmans@topic.nl> escreveu no dia quinta, 30/09/2021 à(s) 07:50: > Wouldn't "0" be a better default? Setting threads to "0" will make xz use > the > CPU count anyway, so it'd be honest too. > > > > Met vriendelijke groet / kind regards, > > Mike Looijmans > System Expert > > > TOPIC Embedded Products B.V. > Materiaalweg 4, 5681 RJ Best > The Netherlands > > T: +31 (0) 499 33 69 69 > E: mike.looijmans@topicproducts.com > W: www.topic.nl > > Please consider the environment before printing this e-mail > On 11-09-2021 12:04, Richard Purdie via lists.openembedded.org wrote: > > rpm output packages currently depend on the value of XZ_THREADS which > > is ok if left to the default value but problematic if system limits > > are set such as on the autobuilder. > > > > Force the value to a specific one in the hashes for better sstate reuse > > and consistent rpm task checksums. > > > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > > --- > > meta/conf/bitbake.conf | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > > index 2140d498f7c..e25d632dc1c 100644 > > --- a/meta/conf/bitbake.conf > > +++ b/meta/conf/bitbake.conf > > @@ -811,6 +811,7 @@ PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" > > # Default parallelism and resource usage for xz > > XZ_MEMLIMIT ?= "50%" > > XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" > > +XZ_THREADS[vardepvalue] = "1" > > XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" > > XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" > > > > > > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#156483): > https://lists.openembedded.org/g/openembedded-core/message/156483 > Mute This Topic: https://lists.openembedded.org/mt/85529700/5052612 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > -- Best regards, José Quaresma [-- Attachment #2: Type: text/html, Size: 3862 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums 2021-09-30 6:50 ` [OE-core] [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums Mike Looijmans 2021-09-30 8:38 ` Jose Quaresma @ 2021-09-30 9:47 ` Richard Purdie 1 sibling, 0 replies; 5+ messages in thread From: Richard Purdie @ 2021-09-30 9:47 UTC (permalink / raw) To: Mike Looijmans, openembedded-core On Thu, 2021-09-30 at 08:50 +0200, Mike Looijmans wrote: > Wouldn't "0" be a better default? Setting threads to "0" will make xz use the > CPU count anyway, so it'd be honest too. On some systems we have this set to a specific value rather than auto so any value is potentially dishonest. I think I went for "1" as it was clearly the least likely and more obvious when looking at the sig files to see if it was working. At the end of the day it doesn't really matter as long as it is a specific value. Cheers, Richard ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-09-30 9:47 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-11 10:04 [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums Richard Purdie 2021-09-11 10:04 ` [PATCH 2/2] sstate: Avoid problems with recipes using SRCPV when fetching sstate Richard Purdie [not found] ` <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.949ef384-8293-46b8-903f-40a477c056ae.d232f846-51ae-43c0-85e8-be1b07245956@emailsignatures365.codetwo.com> [not found] ` <1b153bce-a66a-45ee-a5c6-963ea6fb1c82.0d2bd5fa-15cc-4b27-b94e-83614f9e5b38.5a416b0f-56e3-45c8-b87b-0760607e2824@emailsignatures365.codetwo.com> 2021-09-30 6:50 ` [OE-core] [PATCH 1/2] bitbake.conf: Ensure XZ_THREADS doesn't change sstate checksums Mike Looijmans 2021-09-30 8:38 ` Jose Quaresma 2021-09-30 9:47 ` Richard Purdie
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.