[PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[André Draszik]

The number of threads used, and the amount of memory allowed
to be used, should not affect sstate signatures, as they
don't affect the result.

Otherwise, it becomes impossible to re-use sstate from
automated builders on developer's machines (as the former
might execute bitbake with certain constraints different
compared to developer's machines).

This is in particular a problem with the opkg package writing
backend, as the OPKGBUILDCMD depends on XZ_DEFAULTS. Without
the vardepexclude, there is no re-use possible of the
package_write_ipk sstate.

The reason XZ_DEFAULTS as a whole isn't being whitelisted as
such here is that some arguments can affect the outcome of
the xz operation, so the approach taken is to only
whitelist the safe flags memlimit and # of threads.

Signed-off-by: André Draszik <git@andred.net>

---
v2: remove Gerrit Change-Id line
---
 meta/conf/bitbake.conf | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 954c06b313..d5a1a2471d 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -794,7 +794,10 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
 PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
 
 # Default parallelism and resource usage for xz
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
+XZ_MEMLIMIT ?= "50%"
+XZ_THREADS ?= "${@oe.utils.cpu_count()}"
+XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
+XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 
 ##################################################################
 # Magic Cookie for SANITY CHECK
-- 
2.23.0.rc1

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Adrian Bunk]

On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> The number of threads used, and the amount of memory allowed
> to be used, should not affect sstate signatures, as they
> don't affect the result.

Unfortunately they can affect the result.

> Otherwise, it becomes impossible to re-use sstate from
> automated builders on developer's machines (as the former
> might execute bitbake with certain constraints different
> compared to developer's machines).
>...
> -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
>...

Threaded compression can result in slightly worse compression
than single-threaded compression.

With memlimit the problem is actually the opposite way,
and worse than what you were trying to fix:

When a developer hits memlimit during compression, the documented
behavour of xz is to scale down the compression level.

I assume 50% wrongly gives the same sstate signature no matter how
much RAM is installed on the local machine?

cu
Adrian

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[André Draszik]

On Mon, 2020-02-24 at 15:40 +0200, Adrian Bunk wrote:
> On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> > The number of threads used, and the amount of memory allowed
> > to be used, should not affect sstate signatures, as they
> > don't affect the result.
> 
> Unfortunately they can affect the result.

I'm actually aware of that, and was contemplating on expanding on
that topic ... I guess I should have

> 
> > Otherwise, it becomes impossible to re-use sstate from
> > automated builders on developer's machines (as the former
> > might execute bitbake with certain constraints different
> > compared to developer's machines).
> > ...
> > -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
> > ...
> 
> Threaded compression can result in slightly worse compression
> than single-threaded compression.
> 
> With memlimit the problem is actually the opposite way,
> and worse than what you were trying to fix:
> 
> When a developer hits memlimit during compression, the documented
> behavour of xz is to scale down the compression level.

I believe none of the above is relevant in the context here, though.
The whole point is to avoid running xz again (within
a different
environment) if artefacts are available already.

threads and memlimit in particular are meant to enable compressing on
resource constrained systems. Allowing sstate re-use no matter what
threading and memory limit was set seems sensible, even if a build-
machine has more ram than a deverloper's machine (or vice versa), as
the point here is to allow sstate re-use to speed up builds.
The point is not to change the compression, or to invalidate upon
slight variations due to different amount of RAM.

It's good in particular to re-use the sstate artefacts, rather than
creating new ones, just because the build happens to run on a different
machine.

Once the artefact has been created, it doesn't matter anymore how it
was created, as long as the high-level options aren't being changed (like
an explicit compression level, checksum method, etc.)


> I assume 50% wrongly gives the same sstate signature no matter how
> much RAM is installed on the local machine?

Likely, but a different problem...

Even before this patch, you will get differing results, but this patch
doesn't prevent you from being more strict and specifying --no-adjust
and it doesn't prevent you from changing the compression level as a whole
either.


Cheers,
Andre'

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Adrian Bunk]

On Mon, Feb 24, 2020 at 02:21:57PM +0000, André Draszik wrote:
>...
> Once the artefact has been created, it doesn't matter anymore how it
> was created, as long as the high-level options aren't being changed (like
> an explicit compression level, checksum method, etc.)
>...

meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"

> Cheers,
> Andre'

cu
Adrian

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[André Draszik]

On Mon, 2020-02-24 at 16:31 +0200, Adrian Bunk wrote:
> On Mon, Feb 24, 2020 at 02:21:57PM +0000, André Draszik wrote:
> > ...
> > Once the artefact has been created, it doesn't matter anymore how it
> > was created, as long as the high-level options aren't being changed (like
> > an explicit compression level, checksum method, etc.)
> > ...
> 
> meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"

I still maintain these are two different problems. I am not trying to fix
reproducible builds.

Cheers,
A.

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Adrian Bunk]

On Mon, Feb 24, 2020 at 02:58:21PM +0000, André Draszik wrote:
> On Mon, 2020-02-24 at 16:31 +0200, Adrian Bunk wrote:
> > On Mon, Feb 24, 2020 at 02:21:57PM +0000, André Draszik wrote:
> > > ...
> > > Once the artefact has been created, it doesn't matter anymore how it
> > > was created, as long as the high-level options aren't being changed (like
> > > an explicit compression level, checksum method, etc.)
> > > ...
> > 
> > meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"
> 
> I still maintain these are two different problems. I am not trying to fix
> reproducible builds.

But won't fixing reproducible builds require reverting the patch you 
suggest for fixing your problem?

> Cheers,
> A.

cu
Adrian

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Richard Purdie]

On Mon, 2020-02-24 at 15:40 +0200, Adrian Bunk wrote:
> On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> > The number of threads used, and the amount of memory allowed
> > to be used, should not affect sstate signatures, as they
> > don't affect the result.
> 
> Unfortunately they can affect the result.

I looked into this a bit and its complicated. The threads are used to
compress chunks and their compression should be deterministic whether
done serially or in parallel.

I did some tests and:

xz <file>
gave equivalent output to:
xz <file> --threads=1

and

xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50

all give different identical output.

So if we force --threads >=2 we should have determinism?

> > Otherwise, it becomes impossible to re-use sstate from
> > automated builders on developer's machines (as the former
> > might execute bitbake with certain constraints different
> > compared to developer's machines).
> > ...
> > -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
> > ...
> 
> Threaded compression can result in slightly worse compression
> than single-threaded compression.
> 
> With memlimit the problem is actually the opposite way,
> and worse than what you were trying to fix:
> 
> When a developer hits memlimit during compression, the documented
> behavour of xz is to scale down the compression level.
> 
> I assume 50% wrongly gives the same sstate signature no matter how
> much RAM is installed on the local machine?

I did some tests locally and I could see different output checksums
depending on how much memory I gave xz.

Perhaps we should specify a specific high amount like 1GB?

Does anyone know more about the internals and how to have this behave
"nicely" for our needs?

FWIW we haven't seen variation on the autobuilder due to this as far as
I know.

Cheers,

Richard

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Adrian Bunk]

On Mon, Feb 24, 2020 at 04:44:28PM +0000, Richard Purdie wrote:
> On Mon, 2020-02-24 at 15:40 +0200, Adrian Bunk wrote:
> > On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> > > The number of threads used, and the amount of memory allowed
> > > to be used, should not affect sstate signatures, as they
> > > don't affect the result.
> > 
> > Unfortunately they can affect the result.
> 
> I looked into this a bit and its complicated. The threads are used to
> compress chunks and their compression should be deterministic whether
> done serially or in parallel.
> 
> I did some tests and:
> 
> xz <file>
> gave equivalent output to:
> xz <file> --threads=1
> 
> and
> 
> xz <file> --threads=2
> xz <file> --threads=5
> xz <file> --threads=50
> 
> all give different identical output.
> 
> So if we force --threads >=2 we should have determinism?

This was also my guess after reading the manpage,
but no definite answer from me.

> > > Otherwise, it becomes impossible to re-use sstate from
> > > automated builders on developer's machines (as the former
> > > might execute bitbake with certain constraints different
> > > compared to developer's machines).
> > > ...
> > > -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
> > > ...
> > 
> > Threaded compression can result in slightly worse compression
> > than single-threaded compression.
> > 
> > With memlimit the problem is actually the opposite way,
> > and worse than what you were trying to fix:
> > 
> > When a developer hits memlimit during compression, the documented
> > behavour of xz is to scale down the compression level.
> > 
> > I assume 50% wrongly gives the same sstate signature no matter how
> > much RAM is installed on the local machine?
> 
> I did some tests locally and I could see different output checksums
> depending on how much memory I gave xz.
> 
> Perhaps we should specify a specific high amount like 1GB?

xz -9 needs 1.25 GB per thread.

And since xz decompression speed is linear to compressed size,
-9 is often wanted since it gives the fastest xz decompression.

> Does anyone know more about the internals and how to have this behave
> "nicely" for our needs?
> 
> FWIW we haven't seen variation on the autobuilder due to this as far as
> I know.

I assume the autobuilders have plenty of RAM per core?

For any reasonably sizes machine that doesn't OOM on larger C++ projects
the memlimit is a nop and can be dropped.

More problematic might be developers with oldish desktops/laptops with 
many cores and few RAM.

> Cheers,
> 
> Richard

cu
Adrian

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[André Draszik]

On Mon, 2020-02-24 at 16:44 +0000, Richard Purdie wrote:
> On Mon, 2020-02-24 at 15:40 +0200, Adrian Bunk wrote:
> > On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> > > The number of threads used, and the amount of memory allowed
> > > to be used, should not affect sstate signatures, as they
> > > don't affect the result.
> > 
> > Unfortunately they can affect the result.
> 
> I looked into this a bit and its complicated. The threads are used to
> compress chunks and their compression should be deterministic whether
> done serially or in parallel.
> 
> I did some tests and:
> 
> xz <file>
> gave equivalent output to:
> xz <file> --threads=1
> 
> and
> 
> xz <file> --threads=2
> xz <file> --threads=5
> xz <file> --threads=50
> 
> all give different identical output.
> 
> So if we force --threads >=2 we should have determinism?

How large were your files? Given threaded operation works by operating in
'blocks', you should have a different number of blocks in your output.
They're all compressed independently, so I don't see how the result could
be identical, unless the block size is large enough for xz to not create
as many blocks as you allowed via setting the upper limit on the # of
threads.


> > > Otherwise, it becomes impossible to re-use sstate from
> > > automated builders on developer's machines (as the former
> > > might execute bitbake with certain constraints different
> > > compared to developer's machines).
> > > ...
> > > -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
> > > ...
> > 
> > Threaded compression can result in slightly worse compression
> > than single-threaded compression.
> > 
> > With memlimit the problem is actually the opposite way,
> > and worse than what you were trying to fix:
> > 
> > When a developer hits memlimit during compression, the documented
> > behavour of xz is to scale down the compression level.
> > 
> > I assume 50% wrongly gives the same sstate signature no matter how
> > much RAM is installed on the local machine?
> 
> I did some tests locally and I could see different output checksums
> depending on how much memory I gave xz.
> 
> Perhaps we should specify a specific high amount like 1GB?

If I understand the man page right, at compression level 9 (default in OE),
each thread / block will use up to 674MiB of RAM, so a limit of 1GiB
effectively reduces parallelism to 1.


> Does anyone know more about the internals and how to have this behave
> "nicely" for our needs?

What are the needs specifically?

In my case, I have some defined builds that run on their own, so they
are allowed to use all memory and a certain (lowish) number of threads.

I also have other builds where several can run in parallel, and xz would
just be killed if all of them were allowed to use all RAM, so I have to
reduce threads and memory consumption.

This of course means slower builds...

I don't want to penalise myself or anybody else by generally forcing a
low number of threads either.

> FWIW we haven't seen variation on the autobuilder due to this as far as
> I know.

BTW, pigz and pbzip should have a similar thread related problem, according
to the man pages, if you read between the lines.

Files are split into chunks, and chunks are compressed individually in the
pbzip2 case.
pigz terminates each chunk with an empty block in the final .gz file.


Cheers,
Andre'

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Richard Purdie]

On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
> On Mon, 2020-02-24 at 16:44 +0000, Richard Purdie wrote:
> > On Mon, 2020-02-24 at 15:40 +0200, Adrian Bunk wrote:
> > > On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> > > > The number of threads used, and the amount of memory allowed
> > > > to be used, should not affect sstate signatures, as they
> > > > don't affect the result.
> > > 
> > > Unfortunately they can affect the result.
> > 
> > I looked into this a bit and its complicated. The threads are used
> > to
> > compress chunks and their compression should be deterministic
> > whether
> > done serially or in parallel.
> > 
> > I did some tests and:
> > 
> > xz <file>
> > gave equivalent output to:
> > xz <file> --threads=1
> > 
> > and
> > 
> > xz <file> --threads=2
> > xz <file> --threads=5
> > xz <file> --threads=50
> > 
> > all give different identical output.
> > 
> > So if we force --threads >=2 we should have determinism?
> 
> How large were your files?

105MB total which should be enough as the largest xz block size is
32MB?

>  Given threaded operation works by operating in 'blocks', you should
> have a different number of blocks in your output.
> They're all compressed independently, so I don't see how the result
> could be identical, unless the block size is large enough for xz to
> not create as many blocks as you allowed via setting the upper limit
> on the # of threads.

I don't follow. The algorithm xz uses appears to be designed to give
consistent results regardless of numbers of threads. It gets split into
the same number of chunks. Each chunk is compressed independently, but
deterministically and then they're assembled in order. It doesn't
matter if the chunk is compressed in parallel or serially.

> > > > Otherwise, it becomes impossible to re-use sstate from
> > > > automated builders on developer's machines (as the former
> > > > might execute bitbake with certain constraints different
> > > > compared to developer's machines).
> > > > ...
> > > > -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count
> > > > ()}"
> > > > > Files are split into chunks, and chunks are compressed
> > > > individually
> > > > > in the
> > > > > pbzip2 case.
> > > > > pigz terminates each chunk with an empty block in the final
> > > > .gz file.
> > > > > 
> > > > > 
> > > > > Cheers,
> > > > > Andre'
> > > > > 
> > > > > 
> > > > ...
> > > 
> > > Threaded compression can result in slightly worse compression
> > > than single-threaded compression.
> > > 
> > > With memlimit the problem is actually the opposite way,
> > > and worse than what you were trying to fix:
> > > 
> > > When a developer hits memlimit during compression, the documented
> > > behavour of xz is to scale down the compression level.
> > > 
> > > I assume 50% wrongly gives the same sstate signature no matter
> > > how
> > > much RAM is installed on the local machine?
> > 
> > I did some tests locally and I could see different output checksums
> > depending on how much memory I gave xz.
> > 
> > Perhaps we should specify a specific high amount like 1GB?
> 
> If I understand the man page right, at compression level 9 (default
> in OE),
> each thread / block will use up to 674MiB of RAM, so a limit of 1GiB
> effectively reduces parallelism to 1.

Right, I agree that isn't feasible.

> > Does anyone know more about the internals and how to have this
> > behave "nicely" for our needs?
> 
> What are the needs specifically?

The project is aiming to have deterministic builds, our output for
given input metadata is binary identical.

> In my case, I have some defined builds that run on their own, so they
> are allowed to use all memory and a certain (lowish) number of
> threads.
> 
> I also have other builds where several can run in parallel, and xz
> would just be killed if all of them were allowed to use all RAM, so I
> have to reduce threads and memory consumption.
> 
> This of course means slower builds...
> 
> I don't want to penalise myself or anybody else by generally forcing
> a low number of threads either.

We didn't have resource issues on the autobuilder with xz but others
did report it and its why the memlimit was set. It would be better to
remove the memlimit for determinism really as far as I can tell and if
necessary throttle the threads count.

Maybe setting an upper limit of say 10 threads and a minimum of 2 might
given us what we need. Perhaps we just allow the number of xz threads
to be set independently?

> > FWIW we haven't seen variation on the autobuilder due to this as
> > far as
> > I know.
> 
> BTW, pigz and pbzip should have a similar thread related problem,
> according to the man pages, if you read between the lines.

Agreed. We have less of an issue with these as their use is much less
frequent. xz is used by deb and ipk for packaging.

Cheers,

Richard

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Adrian Bunk]

On Mon, Feb 24, 2020 at 05:32:29PM +0000, Richard Purdie wrote:
> On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
>...
> > I don't want to penalise myself or anybody else by generally forcing
> > a low number of threads either.
> 
> We didn't have resource issues on the autobuilder with xz but others
> did report it and its why the memlimit was set. It would be better to
> remove the memlimit for determinism really as far as I can tell and if
> necessary throttle the threads count.
> 
> Maybe setting an upper limit of say 10 threads and a minimum of 2 might
> given us what we need.

On a Threadripper with 128 cores and 256 GB RAM it would not be a
problem to use all cores.

A laptop with 8 cores and 8 GB RAM is problematic.

> Perhaps we just allow the number of xz threads
> to be set independently?

dpkg manually reduces the number of threads until less than half
of the RAM is used:
https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574

In a script it would be possible to use --no-adjust to achieve the same:
$ xz -9 --memlimit=50% --no-adjust -T32 /dev/null 
xz: Memory usage limit is too low for the given filter setup.
xz: 39,972 MiB of memory is required. The limit is 32,051 MiB.
$

> > > FWIW we haven't seen variation on the autobuilder due to this as
> > > far as
> > > I know.
> > 
> > BTW, pigz and pbzip should have a similar thread related problem,
> > according to the man pages, if you read between the lines.
> 
> Agreed. We have less of an issue with these as their use is much less
> frequent. xz is used by deb and ipk for packaging.

Good point, dpkg does parallel xz compression and I am not aware of any
reproducibility problems this causes.

> Cheers,
> 
> Richard

cu
Adrian

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[André Draszik]

On Tue, 2020-02-25 at 00:00 +0200, Adrian Bunk wrote:
> On Mon, Feb 24, 2020 at 05:32:29PM +0000, Richard Purdie wrote:
> > On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
> > ...
> > > I don't want to penalise myself or anybody else by generally forcing
> > > a low number of threads either.
> > 
> > We didn't have resource issues on the autobuilder with xz but others
> > did report it and its why the memlimit was set. It would be better to
> > remove the memlimit for determinism really as far as I can tell and if
> > necessary throttle the threads count.
> > 
> > Maybe setting an upper limit of say 10 threads and a minimum of 2 might
> > given us what we need.
> 
> On a Threadripper with 128 cores and 256 GB RAM it would not be a
> problem to use all cores.
> 
> A laptop with 8 cores and 8 GB RAM is problematic.
> 
> > Perhaps we just allow the number of xz threads
> > to be set independently?
> 
> dpkg manually reduces the number of threads until less than half
> of the RAM is used:
> https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574
> 
> In a script it would be possible to use --no-adjust to achieve the same:
> $ xz -9 --memlimit=50% --no-adjust -T32 /dev/null 
> xz: Memory usage limit is too low for the given filter setup.
> xz: 39,972 MiB of memory is required. The limit is 32,051 MiB.
> $
> 

The problem with --no-adjust is that it also prevents xz from reducing the
number of threads. It will do that in preference to changing compression
parameters, so as long as you have more than 2499MiB of memory to support at
least 2 threads (with default block size), compression will be identical.

> > > > FWIW we haven't seen variation on the autobuilder due to this as
> > > > far as
> > > > I know.
> > > 
> > > BTW, pigz and pbzip should have a similar thread related problem,
> > > according to the man pages, if you read between the lines.
> > 
> > Agreed. We have less of an issue with these as their use is much less
> > frequent. xz is used by deb and ipk for packaging.

They both actually don't. It's reproducible no matter how many threads.
Interesting...

> 
> Good point, dpkg does parallel xz compression and I am not aware of any
> reproducibility problems this causes.

Debian's dpkg tries to be clever by simply reducing the number of threads until
compression uses at most half of physical
ram.

I guess this typically works out as at least two threads in most if not all
environments.

Cheers,
Andre'

Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures

[Adrian Bunk]

On Tue, Feb 25, 2020 at 09:16:20AM +0000, André Draszik wrote:
> On Tue, 2020-02-25 at 00:00 +0200, Adrian Bunk wrote:
>...
> > > Perhaps we just allow the number of xz threads
> > > to be set independently?
> > 
> > dpkg manually reduces the number of threads until less than half
> > of the RAM is used:
> > https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574
> > 
> > In a script it would be possible to use --no-adjust to achieve the same:
> > $ xz -9 --memlimit=50% --no-adjust -T32 /dev/null 
> > xz: Memory usage limit is too low for the given filter setup.
> > xz: 39,972 MiB of memory is required. The limit is 32,051 MiB.
> > $
> 
> The problem with --no-adjust is that it also prevents xz from reducing the
> number of threads. It will do that in preference to changing compression
> parameters, so as long as you have more than 2499MiB of memory to support at
> least 2 threads (with default block size), compression will be identical.
>...

5 GB due to the 50%.

But I had something different in mind:

Similar to what dpkg is doing, you could once loop from 
@oe.utils.cpu_count() down to 2 until "--memlimit=50% --no-adjust -T$(i)"
succeeds.

Then run the actual compression without --memlimit.

I would not consider anything with less than 4 GB RAM reasonable for 
building Yocto (g++ also likes to use more than 2 GB).

> Cheers,
> Andre'

cu
Adrian

reproducible builds involving xz (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)

[André Draszik]

On Mon, 2020-02-24 at 17:10 +0200, Adrian Bunk wrote:
> On Mon, Feb 24, 2020 at 02:58:21PM +0000, André Draszik wrote:
> > On Mon, 2020-02-24 at 16:31 +0200, Adrian Bunk wrote:
> > > On Mon, Feb 24, 2020 at 02:21:57PM +0000, André Draszik wrote:
> > > > ...
> > > > Once the artefact has been created, it doesn't matter anymore how it
> > > > was created, as long as the high-level options aren't being changed (like
> > > > an explicit compression level, checksum method, etc.)
> > > > ...
> > > 
> > > meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"
> > 
> > I still maintain these are two different problems. I am not trying to fix
> > reproducible builds.
> 
> But won't fixing reproducible builds require reverting the patch you 
> suggest for fixing your problem?

No, I don't think so.

To fix reproducible, you need to either patch xz to always behave like in
multi-threaded mode, even when --threads=1 was given in the arguments.
Additionally, a patch to xz to allow it to scale down the number of
threads, but not the compression level (--no-adjust prevents both),
could be useful.

Alternatively, you need to decide if you want to support it with
--threads=1 or --threads >= 2, and issue at least a warning in
reproducible_build.bbclass if somebody deviates.


You should also calculate the memory requirements, reduce the number
of threads to fit, and again give at least a warning if the compression
can not be done without changing compression level after that.

I still maintain that none of that is related to my original patch.



Cheers,
Andre'

Re: reproducible builds involving xz (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)

[Richard Purdie]

On Tue, 2020-02-25 at 11:16 +0000, André Draszik wrote:
> On Mon, 2020-02-24 at 17:10 +0200, Adrian Bunk wrote:
> > On Mon, Feb 24, 2020 at 02:58:21PM +0000, André Draszik wrote:
> > > On Mon, 2020-02-24 at 16:31 +0200, Adrian Bunk wrote:
> > > > On Mon, Feb 24, 2020 at 02:21:57PM +0000, André Draszik wrote:
> > > > > ...
> > > > > Once the artefact has been created, it doesn't matter anymore
> > > > > how it
> > > > > was created, as long as the high-level options aren't being
> > > > > changed (like
> > > > > an explicit compression level, checksum method, etc.)
> > > > > ...
> > > > 
> > > > meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"
> > > 
> > > I still maintain these are two different problems. I am not
> > > trying to fix
> > > reproducible builds.
> > 
> > But won't fixing reproducible builds require reverting the patch
> > you 
> > suggest for fixing your problem?
> 
> No, I don't think so.
> 
> To fix reproducible, you need to either patch xz to always behave
> like in
> multi-threaded mode, even when --threads=1 was given in the
> arguments.
> Additionally, a patch to xz to allow it to scale down the number of
> threads, but not the compression level (--no-adjust prevents both),
> could be useful.
> 
> Alternatively, you need to decide if you want to support it with
> --threads=1 or --threads >= 2, and issue at least a warning in
> reproducible_build.bbclass if somebody deviates.
> 
> 
> You should also calculate the memory requirements, reduce the number
> of threads to fit, and again give at least a warning if the
> compression
> can not be done without changing compression level after that.
> 
> I still maintain that none of that is related to my original patch.

If I merge your patch, what will happen is that builds on the
autobuilder can potentially start failing as it will potentially detect
reproduciblity issues which it wouldn't previously see.

To address this, I'd ike to see a new parameter added to
oe.utils.cpu_count() which allows us to have a minimum of 2. I'd also
like to reparametise the memlimit so we can drop it entirely by
default.

This should get us some defaults which won't break things and give
reproducibile builds. They may not work for everyone but its then
something which we can build upon.

I do want to take a patch as I agree there is a serious issue here but
I'd prefer we get it right and try and stop it causing me headaches on
the testing infrastructure.

Cheers,

Richard

xz threads / memlimit behaviour (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)

[André Draszik]

On Mon, 2020-02-24 at 16:44 +0000, Richard Purdie wrote:
> On Mon, 2020-02-24 at 15:40 +0200, Adrian Bunk wrote:
> > On Mon, Feb 24, 2020 at 12:59:55PM +0000, André Draszik wrote:
> > > The number of threads used, and the amount of memory allowed
> > > to be used, should not affect sstate signatures, as they
> > > don't affect the result.
> > 
> > Unfortunately they can affect the result.
> 
> I looked into this a bit and its complicated. The threads are used to
> compress chunks and their compression should be deterministic whether
> done serially or in parallel.
> 
> I did some tests and:
> 
> xz <file>
> gave equivalent output to:
> xz <file> --threads=1
> 
> and
> 
> xz <file> --threads=2
> xz <file> --threads=5
> xz <file> --threads=50
> 
> all give different identical output.
> 
> So if we force --threads >=2 we should have determinism?

I did another test...

Even single threaded compression gives the same result as multi-threaded,
if single threaded is a result of xz scaling down the memory usage due
to --memlimit, ie:

xz -f -c -9 --threads=2 --memlimit=2000M --keep --verbose --verbose xztest > xztest2_down_to_1.xz
  xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
  xz: Using up to 2 threads.
  xz: 2499 MiB of memory is required. The limit is 2000 MiB.
  xz: Decompression will need 65 MiB of memory.
  xz: Adjusted the number of threads from 2 to 1 to not exceed the memory usage limit of 2000 MiB

is different from

xz -f -c -9 --threads=1 --memlimit=2000M --verbose --verbose xztest > xztest1.xz
  xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
  xz: 674 MiB of memory is required. The limit is 2000 MiB.
  xz: Decompression will need 65 MiB of memory.


There even is a comment in the xz sources about this. This simplifies things a bit...


> > > Otherwise, it becomes impossible to re-use sstate from
> > > automated builders on developer's machines (as the former
> > > might execute bitbake with certain constraints different
> > > compared to developer's machines).
> > > ...
> > > -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
> > > ...
> > 
> > Threaded compression can result in slightly worse compression
> > than single-threaded compression.
> > 
> > With memlimit the problem is actually the opposite way,
> > and worse than what you were trying to fix:
> > 
> > When a developer hits memlimit during compression, the documented
> > behavour of xz is to scale down the compression level.

This is not what is actually happening - it appears the documentation only
talks about single-threaded mode.
If you look at the code and look at command output, xz only ever
tries to reduce the number of threads if multi-threading was enabled. In
that case, it will *never* even attempt to change the compression level.

If it can't satisfy the memory limit by reducing the number of threads,
it just errors out.

So as long as all builders have at least 2 cores, i.e. --threads >= 2, memlimit
doesn't ever affect the outcome (except that it can make the operation fail as
a whole).


Cheers,
Andre'