Re: [yocto] bitbake controlling memory use

["Mike Looijmans" <mike.looijmans@topic.nl>]

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topic.nl

Please consider the environment before printing this e-mail
On 11-04-2021 17:23, Gmane Admin via lists.yoctoproject.org wrote:
> My build machine has 8 cores + HT so bitbake enthusiastically assumes 
> 16. However I have (only?) 16GB of RAM (+24GB swap space).

Been there, done that (8/16 cores with 8GB RAM).

The major cause is that bitbake will not just spawn 16 compiler threads, 
it will actually spawn up to 16 "do_compile" tasks which may spawn 16 
compiler processes each, thus you'll be running a whopping 256 
compilers. Bitbake may spawn a total of n^2 threads by default with "n" 
the detected number of cores.

The workaround I used was to limit the number of bitbake threads but 
leave the make threads at 16. Since most software these days is using 
parallel make, the impact on the build time of reducing the bb threads 
to 8 or 4 is negligible.

As for translating this into a bitbake feature, an implementation that 
comes to mind is to add a "weight" to each task. Most tasks would get a 
weight of just "1", but a (multithreaded) compile would be weighted "8" 
(some formula involving the number of CPUs) or so. This would stop 
bitbake spawning more processes early so that you don't get into the n^2 
threads problem, and the number of processed that bitbake will spawn 
will be close to linear agin instead of quadratic as is is now.

Recipes that have excessive memory loads (usually C++ template 
meta-programming) can then just increase the weighting for the compile task.


-- 
Mike Looijmans