* [RFC PATCH] rpm: limit xz threading
@ 2021-05-26 14:30 Ross Burton
2021-05-26 16:07 ` [OE-core] " Konrad Weihmann
0 siblings, 1 reply; 3+ messages in thread
From: Ross Burton @ 2021-05-26 14:30 UTC (permalink / raw)
To: openembedded-core
Signed-off-by: Ross Burton <ross.burton@arm.com>
---
.../rpm/files/less-threads.patch | 24 +++++++++++++++++++
meta/recipes-devtools/rpm/rpm_4.16.1.3.bb | 1 +
2 files changed, 25 insertions(+)
create mode 100644 meta/recipes-devtools/rpm/files/less-threads.patch
diff --git a/meta/recipes-devtools/rpm/files/less-threads.patch b/meta/recipes-devtools/rpm/files/less-threads.patch
new file mode 100644
index 0000000000..d6db85f28a
--- /dev/null
+++ b/meta/recipes-devtools/rpm/files/less-threads.patch
@@ -0,0 +1,24 @@
+Instead of telling XZ to use as many threads as there are processor cores, just
+hardcode this to 4.
+
+On large systems with lots of cores, XZ will use unreasonable amounts of memory.
+For example, my ThunderX2 reports 256 cores and rpmbuild takes 3.8GB of resident
+memory when packaging cmake. Limiting RPM to just 8 threads reduces this to
+1.2GB without massively reducing the compression speed.
+
+Upstream-Status: Inappropriate
+Signed-off-by: Ross Burton <ross.burton@arm.com>
+
+diff --git a/rpmio/rpmio.c b/rpmio/rpmio.c
+index 52be6a38d..c4034e9e5 100644
+--- a/rpmio/rpmio.c
++++ b/rpmio/rpmio.c
+@@ -799,7 +799,7 @@ static LZFILE *lzopen_internal(const char *mode, int fd, int xz)
+ #ifdef HAVE_LZMA_MT
+ } else {
+ if (threads == -1)
+- threads = rpmExpandNumeric("%{getncpus}");
++ threads = 8;
+ lzma_mt mt_options = {
+ .flags = 0,
+ .threads = threads,
diff --git a/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb b/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
index 95a6f5cf71..f7e9378061 100644
--- a/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
+++ b/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
@@ -40,6 +40,7 @@ SRC_URI = "git://github.com/rpm-software-management/rpm;branch=rpm-4.16.x \
file://0016-rpmscript.c-change-logging-level-around-scriptlets-t.patch \
file://0001-lib-transaction.c-fix-file-conflicts-for-MIPS64-N32.patch \
file://0001-tools-Add-error.h-for-non-glibc-case.patch \
+ file://less-threads.patch \
"
PE = "1"
--
2.25.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [OE-core] [RFC PATCH] rpm: limit xz threading
2021-05-26 14:30 [RFC PATCH] rpm: limit xz threading Ross Burton
@ 2021-05-26 16:07 ` Konrad Weihmann
2021-05-26 16:31 ` Alexander Kanavin
0 siblings, 1 reply; 3+ messages in thread
From: Konrad Weihmann @ 2021-05-26 16:07 UTC (permalink / raw)
To: openembedded-core; +Cc: Ross Burton
On 26.05.21 16:30, Ross Burton wrote:
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
> .../rpm/files/less-threads.patch | 24 +++++++++++++++++++
> meta/recipes-devtools/rpm/rpm_4.16.1.3.bb | 1 +
> 2 files changed, 25 insertions(+)
> create mode 100644 meta/recipes-devtools/rpm/files/less-threads.patch
>
> diff --git a/meta/recipes-devtools/rpm/files/less-threads.patch b/meta/recipes-devtools/rpm/files/less-threads.patch
> new file mode 100644
> index 0000000000..d6db85f28a
> --- /dev/null
> +++ b/meta/recipes-devtools/rpm/files/less-threads.patch
> @@ -0,0 +1,24 @@
> +Instead of telling XZ to use as many threads as there are processor cores, just
> +hardcode this to 4.
at this point I read 4
> +
> +On large systems with lots of cores, XZ will use unreasonable amounts of memory.
> +For example, my ThunderX2 reports 256 cores and rpmbuild takes 3.8GB of resident
> +memory when packaging cmake. Limiting RPM to just 8 threads reduces this to
> +1.2GB without massively reducing the compression speed.
> +
> +Upstream-Status: Inappropriate
> +Signed-off-by: Ross Burton <ross.burton@arm.com>
> +
> +diff --git a/rpmio/rpmio.c b/rpmio/rpmio.c
> +index 52be6a38d..c4034e9e5 100644
> +--- a/rpmio/rpmio.c
> ++++ b/rpmio/rpmio.c
> +@@ -799,7 +799,7 @@ static LZFILE *lzopen_internal(const char *mode, int fd, int xz)
> + #ifdef HAVE_LZMA_MT
> + } else {
> + if (threads == -1)
> +- threads = rpmExpandNumeric("%{getncpus}");
> ++ threads = 8;
...but here it's 8 :thinking: (I know 2xthreads == cores, still I find
the commit messages very confusing). Wouldn't it make more sense to
externalize the value, so that patch might become upstreamable?
> + lzma_mt mt_options = {
> + .flags = 0,
> + .threads = threads,
> diff --git a/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb b/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
> index 95a6f5cf71..f7e9378061 100644
> --- a/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
> +++ b/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
> @@ -40,6 +40,7 @@ SRC_URI = "git://github.com/rpm-software-management/rpm;branch=rpm-4.16.x \
> file://0016-rpmscript.c-change-logging-level-around-scriptlets-t.patch \
> file://0001-lib-transaction.c-fix-file-conflicts-for-MIPS64-N32.patch \
> file://0001-tools-Add-error.h-for-non-glibc-case.patch \
> + file://less-threads.patch \
> "
>
> PE = "1"
>
>
>
>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [OE-core] [RFC PATCH] rpm: limit xz threading
2021-05-26 16:07 ` [OE-core] " Konrad Weihmann
@ 2021-05-26 16:31 ` Alexander Kanavin
0 siblings, 0 replies; 3+ messages in thread
From: Alexander Kanavin @ 2021-05-26 16:31 UTC (permalink / raw)
To: Konrad Weihmann; +Cc: OE-core, Ross Burton
[-- Attachment #1: Type: text/plain, Size: 2923 bytes --]
I think Ross already sent a better patch where the number of threads is set
externally via BB variable?
Alex
On Wed, 26 May 2021 at 18:08, Konrad Weihmann <kweihmann@outlook.com> wrote:
>
>
> On 26.05.21 16:30, Ross Burton wrote:
> > Signed-off-by: Ross Burton <ross.burton@arm.com>
> > ---
> > .../rpm/files/less-threads.patch | 24 +++++++++++++++++++
> > meta/recipes-devtools/rpm/rpm_4.16.1.3.bb | 1 +
> > 2 files changed, 25 insertions(+)
> > create mode 100644 meta/recipes-devtools/rpm/files/less-threads.patch
> >
> > diff --git a/meta/recipes-devtools/rpm/files/less-threads.patch
> b/meta/recipes-devtools/rpm/files/less-threads.patch
> > new file mode 100644
> > index 0000000000..d6db85f28a
> > --- /dev/null
> > +++ b/meta/recipes-devtools/rpm/files/less-threads.patch
> > @@ -0,0 +1,24 @@
> > +Instead of telling XZ to use as many threads as there are processor
> cores, just
> > +hardcode this to 4.
>
> at this point I read 4
>
> > +
> > +On large systems with lots of cores, XZ will use unreasonable amounts
> of memory.
> > +For example, my ThunderX2 reports 256 cores and rpmbuild takes 3.8GB of
> resident
> > +memory when packaging cmake. Limiting RPM to just 8 threads reduces
> this to
> > +1.2GB without massively reducing the compression speed.
> > +
> > +Upstream-Status: Inappropriate
> > +Signed-off-by: Ross Burton <ross.burton@arm.com>
> > +
> > +diff --git a/rpmio/rpmio.c b/rpmio/rpmio.c
> > +index 52be6a38d..c4034e9e5 100644
> > +--- a/rpmio/rpmio.c
> > ++++ b/rpmio/rpmio.c
> > +@@ -799,7 +799,7 @@ static LZFILE *lzopen_internal(const char *mode,
> int fd, int xz)
> > + #ifdef HAVE_LZMA_MT
> > + } else {
> > + if (threads == -1)
> > +- threads = rpmExpandNumeric("%{getncpus}");
> > ++ threads = 8;
>
> ...but here it's 8 :thinking: (I know 2xthreads == cores, still I find
> the commit messages very confusing). Wouldn't it make more sense to
> externalize the value, so that patch might become upstreamable?
>
> > + lzma_mt mt_options = {
> > + .flags = 0,
> > + .threads = threads,
> > diff --git a/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
> b/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
> > index 95a6f5cf71..f7e9378061 100644
> > --- a/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
> > +++ b/meta/recipes-devtools/rpm/rpm_4.16.1.3.bb
> > @@ -40,6 +40,7 @@ SRC_URI = "git://
> github.com/rpm-software-management/rpm;branch=rpm-4.16.x \
> >
> file://0016-rpmscript.c-change-logging-level-around-scriptlets-t.patch \
> >
> file://0001-lib-transaction.c-fix-file-conflicts-for-MIPS64-N32.patch \
> > file://0001-tools-Add-error.h-for-non-glibc-case.patch \
> > + file://less-threads.patch \
> > "
> >
> > PE = "1"
> >
> >
> >
> >
> >
>
>
>
>
[-- Attachment #2: Type: text/html, Size: 4296 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-05-26 16:31 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-26 14:30 [RFC PATCH] rpm: limit xz threading Ross Burton
2021-05-26 16:07 ` [OE-core] " Konrad Weihmann
2021-05-26 16:31 ` Alexander Kanavin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.