Re: [yocto] bitbake controlling memory use

[Ferry Toth <fntoth@gmail.com>]

Hi,

Just in case you are interested in the result.

Op 03-01-2023 om 18:33 schreef Alexander Kanavin:
> On Tue, 3 Jan 2023 at 16:58, Ferry Toth <fntoth@gmail.com> wrote:
>
>> I know you don't want to spend time on this. So don't. Others may have
>> an interest or a tip to improve Yocto in this area.
>>
>> I'm just trying to get an old patch from Richard Purdie to work
>> (http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/wipqueue4&id=d66a327fb6189db5de8bc489859235dcba306237)
>> with later fixes by Trevor Gamblin.

I got Richards patch "Add shared make jobserver support 
<https://github.com/htot/poky/commit/28fd4b9e6e60bdd07352827ec713510ae86a39c6>" 
working again. It was a bit more work then last time, it was broken in 3 
places (poky, make and python). Find it here: 
https://github.com/htot/poky/commits/kirkstone

It works by making each make believe it is a submake, while the named 
pipe holding the execution tokens is created by bitbake.

Of course as is it only limits number of threads started by make, not 
ninja. On my machine (16 threads) while compiling (linux, nodejs, 
nodejs-native etc.) simultaneously this limits memory consumption to 
~14GB RAM. A little less then 1GB/thread.

Linking nodejs is another story. Each thread uses 4-5GB, and the recipe 
allows 5 threads simultaneously. With a nodejs_%.bbappend:

EXTRA_OEMAKE:prepend = "\
LINK='flock /tmp ${CXX}' \
"
only one thread will be linked simultaneously.

This brought my image build time from 3h > 1u47m (that is after removing 
out and sstate, not bbcache so excluding download time for sources).

>> Or alternatively something based on https://github.com/olsner/jobclient.

In principle we could use jobclient to start other programs then make, 
but for meta-intel-edison I didn't see any that would benefit from this.

It would be nice if ninja could use the same named pipe for the 
execution tokens as make, that might be for another holiday.

> By all means please do. Everyone will benefit from teaching make to
> abstain from starting new processes if RAM is tight, even people with
> xeon/epyc grade hardware. It's not that I don't want to spend time on
> this, it's simply that I do not have the time.

Ah, yes, that is a known problem. Like I said, this was for me just a 
little holiday project.

Now it works I'm not sure what to do. Richard marked the original patch 
as WIP. Maybe it's not appropriate for including into poky? If so I 
wouldn't mind carrying it in meta-intel-edison.

Or may be with a little work (from me) and a run on the CI servers we 
could make it go in?

> Alex