[PATCH] package_rpm: pass XZ_THREADS to rpm

[PATCH] package_rpm: pass XZ_THREADS to rpm

[Ross Burton]

By default RPM uses the number of cores as the number of threads to use,
which can result in quite antisocial memory usage.

As we control the macros for compression anyway, we can pass XZ_THREADS
to limit the number of threads if needed.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes/package_rpm.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 84a9a6dd12..86706da842 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -684,8 +684,8 @@ python do_package_rpm () {
     cmd = cmd + " --define '_use_internal_dependency_generator 0'"
     cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'"
     cmd = cmd + " --define '_build_id_links none'"
-    cmd = cmd + " --define '_binary_payload w6T.xzdio'"
-    cmd = cmd + " --define '_source_payload w6T.xzdio'"
+    cmd = cmd + " --define '_binary_payload w6T%d.xzdio'" % int(d.getVar("XZ_THREADS"))
+    cmd = cmd + " --define '_source_payload w6T%d.xzdio'" % int(d.getVar("XZ_THREADS"))
     cmd = cmd + " --define 'clamp_mtime_to_source_date_epoch 1'"
     cmd = cmd + " --define 'use_source_date_epoch_as_buildtime 1'"
     cmd = cmd + " --define '_buildhost reproducible'"
-- 
2.25.1

Re: [OE-core] [PATCH] package_rpm: pass XZ_THREADS to rpm

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1549 bytes --]

What  happens if XZ_THREADS isn't set?

Alex

On Wed, 26 May 2021 at 16:59, Ross Burton <ross@burtonini.com> wrote:

> By default RPM uses the number of cores as the number of threads to use,
> which can result in quite antisocial memory usage.
>
> As we control the macros for compression anyway, we can pass XZ_THREADS
> to limit the number of threads if needed.
>
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  meta/classes/package_rpm.bbclass | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/package_rpm.bbclass
> b/meta/classes/package_rpm.bbclass
> index 84a9a6dd12..86706da842 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -684,8 +684,8 @@ python do_package_rpm () {
>      cmd = cmd + " --define '_use_internal_dependency_generator 0'"
>      cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build
> 0'"
>      cmd = cmd + " --define '_build_id_links none'"
> -    cmd = cmd + " --define '_binary_payload w6T.xzdio'"
> -    cmd = cmd + " --define '_source_payload w6T.xzdio'"
> +    cmd = cmd + " --define '_binary_payload w6T%d.xzdio'" %
> int(d.getVar("XZ_THREADS"))
> +    cmd = cmd + " --define '_source_payload w6T%d.xzdio'" %
> int(d.getVar("XZ_THREADS"))
>      cmd = cmd + " --define 'clamp_mtime_to_source_date_epoch 1'"
>      cmd = cmd + " --define 'use_source_date_epoch_as_buildtime 1'"
>      cmd = cmd + " --define '_buildhost reproducible'"
> --
> 2.25.1
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 2212 bytes --]

Re: [OE-core] [PATCH] package_rpm: pass XZ_THREADS to rpm

[Richard Purdie]

On Wed, 2021-05-26 at 18:30 +0200, Alexander Kanavin wrote:
> What  happens if XZ_THREADS isn't set?

It is given a default in bitbake.conf so you'd have to go out 
of your way to do it...

Cheers,

Richard

Re: [OE-core] [PATCH] package_rpm: pass XZ_THREADS to rpm

[Michael Opdenacker]

Hello,

Adding this docs@ mailing list too, because I have a documentation
related question...

On 5/26/21 7:10 PM, Richard Purdie wrote:
> On Wed, 2021-05-26 at 18:30 +0200, Alexander Kanavin wrote:
>> What  happens if XZ_THREADS isn't set?
> It is given a default in bitbake.conf so you'd have to go out 
> of your way to do it...

This makes me realize that the XZ_ variables defined in bitbake.conf
don't appear anywhere in the documentation.

Would it make sense to add them to the documentation too, or would it be
overwhelming to document all possible variables, which default values
should just work fine?

Cheers,

Michael

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH] package_rpm: pass XZ_THREADS to rpm

[Ross Burton]

On Fri, 28 May 2021 at 10:40, Michael Opdenacker
<michael.opdenacker@bootlin.com> wrote:
> This makes me realize that the XZ_ variables defined in bitbake.conf
> don't appear anywhere in the documentation.
>
> Would it make sense to add them to the documentation too, or would it be
> overwhelming to document all possible variables, which default values
> should just work fine?

This should most likely be documented, albeit maybe only in the
reference as it's not really a common tweak.

Ross

Re: [docs] [OE-core] [PATCH] package_rpm: pass XZ_THREADS to rpm

[Robert P. J. Day]

[-- Attachment #1: Type: text/plain, Size: 1844 bytes --]

On Fri, 28 May 2021, Michael Opdenacker wrote:

> Hello,
>
> Adding this docs@ mailing list too, because I have a documentation
> related question...
>
> On 5/26/21 7:10 PM, Richard Purdie wrote:
> > On Wed, 2021-05-26 at 18:30 +0200, Alexander Kanavin wrote:
> >> What  happens if XZ_THREADS isn't set?
> > It is given a default in bitbake.conf so you'd have to go out 
> > of your way to do it...
>
> This makes me realize that the XZ_ variables defined in bitbake.conf
> don't appear anywhere in the documentation.
>
> Would it make sense to add them to the documentation too, or would
> it be overwhelming to document all possible variables, which default
> values should just work fine?

  i've actually been pondering this for a while -- the best way to
keep the variables glossary up to date, and here's a thought.

  if one notices a single variable missing, and one has time and the
know-how to add that entry to the variables glossary, then submit a
patch (as i've done recently, all of which i apparently have to
improve. :-)

  on the other hand, if one finds a number of closely-related
variables that are all missing -- for which it would make little sense
to add just one or two without adding all the rest -- and one is
lacking either the time or the expertise to submit an all-encompassing
patch to add them all, then submit a bugzilla entry for it:

https://bugzilla.yoctoproject.org/describecomponents.cgi?product=Reference

and perhaps the person or people with the know-how can handle it. at
the very least, we'd have a list of what's missing.

  i might even add a new product category entry just for this:
"variables glossary". anyway, i think this would be useful just to
keep track of what's not there, and allow people to cherry-pick what
they'd like to work on.

rday

Re: [OE-core] [PATCH] package_rpm: pass XZ_THREADS to rpm

[Michael Opdenacker]

Hi Ross,

On 5/28/21 12:03 PM, Ross Burton wrote:
> On Fri, 28 May 2021 at 10:40, Michael Opdenacker
> <michael.opdenacker@bootlin.com> wrote:
>> This makes me realize that the XZ_ variables defined in bitbake.conf
>> don't appear anywhere in the documentation.
>>
>> Would it make sense to add them to the documentation too, or would it be
>> overwhelming to document all possible variables, which default values
>> should just work fine?
> This should most likely be documented, albeit maybe only in the
> reference as it's not really a common tweak.


Thanks for your guidance. Will do :)

Cheers,

Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [OE-core] [PATCH] package_rpm: pass XZ_THREADS to rpm

[Michael Opdenacker]

Hi Robert,

On 5/28/21 12:43 PM, Robert P. J. Day wrote:
>   i've actually been pondering this for a while -- the best way to
> keep the variables glossary up to date, and here's a thought.
>
>   if one notices a single variable missing, and one has time and the
> know-how to add that entry to the variables glossary, then submit a
> patch (as i've done recently, all of which i apparently have to
> improve. :-)

Your efforts are much appreciated, and I'm sharing the blame for being a
big slow at processing them.

>
>   on the other hand, if one finds a number of closely-related
> variables that are all missing -- for which it would make little sense
> to add just one or two without adding all the rest -- and one is
> lacking either the time or the expertise to submit an all-encompassing
> patch to add them all, then submit a bugzilla entry for it:
>
> https://bugzilla.yoctoproject.org/describecomponents.cgi?product=Reference
>
> and perhaps the person or people with the know-how can handle it. at
> the very least, we'd have a list of what's missing.
>
>   i might even add a new product category entry just for this:
> "variables glossary". anyway, i think this would be useful just to
> keep track of what's not there, and allow people to cherry-pick what
> they'd like to work on.


This sounds like a good idea.

I asked Michael H. to add one for us, and I'm going to use it very soon!

Thanks again for all your contributions!

Cheers,

Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com