[isar-cip-core][PATCH v2] add support to limit resource consumption of squashfs creation

[isar-cip-core][PATCH v2] add support to limit resource consumption of squashfs creation

[Felix Moessbauer]

This patch makes it possible to control the number of CPUs and the
memory usage when generating a squashfs. The variable names used
follow the imagetype variable scheme.

Especially when compressing a squashfs internally with cpu and memory
heavy compressors like xz, we inherit the issues we had with the image
compression in the past. On CI systems, where many tasks might run in
parallel, this often led to OOMs and poor performance. By providing
these options similiar to the other imagetype options, we give CI
administrators an easy way to control the memory consumption,
independent of the other options passed to mksquashfs.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
Changes since v1:

- add reasoning in the commit message
- no functional changes

 classes/squashfs.bbclass | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass
index a06c501..b14768c 100644
--- a/classes/squashfs.bbclass
+++ b/classes/squashfs.bbclass
@@ -15,6 +15,13 @@ SQUASHFS_EXCLUDE_DIRS ?= ""
 SQUASHFS_CONTENT ?= "${PP_ROOTFS}"
 SQUASHFS_CREATION_ARGS ?= ""
 
+SQUASHFS_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
+SQUASHFS_THREADS[vardepvalue] = "1"
+# default according to mksquasfs docs
+SQUASHFS_MEMLIMIT ?= "7982M"
+SQUASHFS_DEFAULTS ?= "-mem ${SQUASHFS_MEMLIMIT} -processors ${SQUASHFS_THREADS}"
+SQUASHFS_DEFAULTS[vardepsexclude] += "SQUASHFS_MEMLIMIT SQUASHFS_THREADS"
+
 python __anonymous() {
     exclude_directories = d.getVar('SQUASHFS_EXCLUDE_DIRS').split()
     if len(exclude_directories) == 0:
@@ -31,5 +38,5 @@ IMAGE_CMD:squashfs[depends] = "${PN}:do_transform_template"
 IMAGE_CMD:squashfs() {
     ${SUDO_CHROOT} /bin/mksquashfs \
         '${SQUASHFS_CONTENT}' '${IMAGE_FILE_CHROOT}' \
-        -noappend ${SQUASHFS_CREATION_ARGS}
+        -noappend ${SQUASHFS_DEFAULTS} ${SQUASHFS_CREATION_ARGS}
 }
-- 
2.39.2

Re: [isar-cip-core][PATCH v2] add support to limit resource consumption of squashfs creation

[Jan Kiszka]

On 14.04.23 11:24, Felix Moessbauer wrote:
> This patch makes it possible to control the number of CPUs and the
> memory usage when generating a squashfs. The variable names used
> follow the imagetype variable scheme.
> 
> Especially when compressing a squashfs internally with cpu and memory
> heavy compressors like xz, we inherit the issues we had with the image
> compression in the past. On CI systems, where many tasks might run in
> parallel, this often led to OOMs and poor performance. By providing
> these options similiar to the other imagetype options, we give CI
> administrators an easy way to control the memory consumption,
> independent of the other options passed to mksquashfs.

If this is targeting CI tunings, we should also export the related
variables via kas to allow pickup from the environment, not?

Jan

> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
> Changes since v1:
> 
> - add reasoning in the commit message
> - no functional changes
> 
>  classes/squashfs.bbclass | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass
> index a06c501..b14768c 100644
> --- a/classes/squashfs.bbclass
> +++ b/classes/squashfs.bbclass
> @@ -15,6 +15,13 @@ SQUASHFS_EXCLUDE_DIRS ?= ""
>  SQUASHFS_CONTENT ?= "${PP_ROOTFS}"
>  SQUASHFS_CREATION_ARGS ?= ""
>  
> +SQUASHFS_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> +SQUASHFS_THREADS[vardepvalue] = "1"
> +# default according to mksquasfs docs
> +SQUASHFS_MEMLIMIT ?= "7982M"
> +SQUASHFS_DEFAULTS ?= "-mem ${SQUASHFS_MEMLIMIT} -processors ${SQUASHFS_THREADS}"
> +SQUASHFS_DEFAULTS[vardepsexclude] += "SQUASHFS_MEMLIMIT SQUASHFS_THREADS"
> +
>  python __anonymous() {
>      exclude_directories = d.getVar('SQUASHFS_EXCLUDE_DIRS').split()
>      if len(exclude_directories) == 0:
> @@ -31,5 +38,5 @@ IMAGE_CMD:squashfs[depends] = "${PN}:do_transform_template"
>  IMAGE_CMD:squashfs() {
>      ${SUDO_CHROOT} /bin/mksquashfs \
>          '${SQUASHFS_CONTENT}' '${IMAGE_FILE_CHROOT}' \
> -        -noappend ${SQUASHFS_CREATION_ARGS}
> +        -noappend ${SQUASHFS_DEFAULTS} ${SQUASHFS_CREATION_ARGS}
>  }

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [isar-cip-core][PATCH v2] add support to limit resource consumption of squashfs creation

[Moessbauer, Felix]

On Fri, 2023-04-14 at 11:33 +0200, Jan Kiszka wrote:
> On 14.04.23 11:24, Felix Moessbauer wrote:
> > This patch makes it possible to control the number of CPUs and the
> > memory usage when generating a squashfs. The variable names used
> > follow the imagetype variable scheme.
> > 
> > Especially when compressing a squashfs internally with cpu and
> > memory
> > heavy compressors like xz, we inherit the issues we had with the
> > image
> > compression in the past. On CI systems, where many tasks might run
> > in
> > parallel, this often led to OOMs and poor performance. By providing
> > these options similiar to the other imagetype options, we give CI
> > administrators an easy way to control the memory consumption,
> > independent of the other options passed to mksquashfs.
> 
> If this is targeting CI tunings, we should also export the related
> variables via kas to allow pickup from the environment, not?

We don't do that for the other variables like XZ_MEMORY either.
IMHO we should also not do this, as usually no further customizations
are required. But if they are required, people can just add the pass-
through in the kas file on a per use-case basis (e.g. without value, so
the corresponding env var is picked up).

The more important thing is, that we provide clear interfaces.
And exactly that is what this patch does.

Felix

> 
> Jan
> 
> > 
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> > Changes since v1:
> > 
> > - add reasoning in the commit message
> > - no functional changes
> > 
> >  classes/squashfs.bbclass | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/classes/squashfs.bbclass b/classes/squashfs.bbclass
> > index a06c501..b14768c 100644
> > --- a/classes/squashfs.bbclass
> > +++ b/classes/squashfs.bbclass
> > @@ -15,6 +15,13 @@ SQUASHFS_EXCLUDE_DIRS ?= ""
> >  SQUASHFS_CONTENT ?= "${PP_ROOTFS}"
> >  SQUASHFS_CREATION_ARGS ?= ""
> >  
> > +SQUASHFS_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> > +SQUASHFS_THREADS[vardepvalue] = "1"
> > +# default according to mksquasfs docs
> > +SQUASHFS_MEMLIMIT ?= "7982M"
> > +SQUASHFS_DEFAULTS ?= "-mem ${SQUASHFS_MEMLIMIT} -processors
> > ${SQUASHFS_THREADS}"
> > +SQUASHFS_DEFAULTS[vardepsexclude] += "SQUASHFS_MEMLIMIT
> > SQUASHFS_THREADS"
> > +
> >  python __anonymous() {
> >      exclude_directories =
> > d.getVar('SQUASHFS_EXCLUDE_DIRS').split()
> >      if len(exclude_directories) == 0:
> > @@ -31,5 +38,5 @@ IMAGE_CMD:squashfs[depends] =
> > "${PN}:do_transform_template"
> >  IMAGE_CMD:squashfs() {
> >      ${SUDO_CHROOT} /bin/mksquashfs \
> >          '${SQUASHFS_CONTENT}' '${IMAGE_FILE_CHROOT}' \
> > -        -noappend ${SQUASHFS_CREATION_ARGS}
> > +        -noappend ${SQUASHFS_DEFAULTS} ${SQUASHFS_CREATION_ARGS}
> >  }
> 

Re: [isar-cip-core][PATCH v2] add support to limit resource consumption of squashfs creation

[Jan Kiszka]

On 14.04.23 14:55, Moessbauer, Felix (T CED INW-CN) wrote:
> On Fri, 2023-04-14 at 11:33 +0200, Jan Kiszka wrote:
>> On 14.04.23 11:24, Felix Moessbauer wrote:
>>> This patch makes it possible to control the number of CPUs and the
>>> memory usage when generating a squashfs. The variable names used
>>> follow the imagetype variable scheme.
>>>
>>> Especially when compressing a squashfs internally with cpu and
>>> memory
>>> heavy compressors like xz, we inherit the issues we had with the
>>> image
>>> compression in the past. On CI systems, where many tasks might run
>>> in
>>> parallel, this often led to OOMs and poor performance. By providing
>>> these options similiar to the other imagetype options, we give CI
>>> administrators an easy way to control the memory consumption,
>>> independent of the other options passed to mksquashfs.
>>
>> If this is targeting CI tunings, we should also export the related
>> variables via kas to allow pickup from the environment, not?
> 
> We don't do that for the other variables like XZ_MEMORY either.
> IMHO we should also not do this, as usually no further customizations
> are required. But if they are required, people can just add the pass-
> through in the kas file on a per use-case basis (e.g. without value, so
> the corresponding env var is picked up).

There are several aspects: XZ_* are variables of Isar upstream, and Isar
upstream has no kas support yet - isar-cip-core does have. If you wanted
to run isar-cip-core in CI and should run into the need to customize
those tunables, you would have to patch it. I agree, it's more likely
that your downstream layer will have that need and could then perfectly
add the tuning vars downstream to kas.

But then when looking at Isar, the other question would be if those vars
should not eventually be part of Isar's bitbake.conf as well. This class
here is downstream (/wrt Isar) so far, but maybe now another trigger to
change that.

> 
> The more important thing is, that we provide clear interfaces.
> And exactly that is what this patch does.

I don't disagree on that, and I'm fine with moving forward as-is for now.

Applied to next, thanks.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux