Re: [yocto] bitbake controlling memory use

["Randy MacLeod" <randy.macleod@windriver.com>]

On 2021-04-11 12:19 p.m., Alexander Kanavin wrote:
> make already has -l option for limiting new instances if load average is 
> too high, so it's only natural to add a RAM limiter too.
> 
>    -l [N], --load-average[=N], --max-load[=N]
>                                Don't start multiple jobs unless load is 
> below N.
> 
> In any case, patches welcome :)

During today's Yocto technical call (1),
we talked about approaches to limiting the system load and avoiding
swap and/or OOM events. Here's what (little!) i recall from the
discussion, 9 busy hours later.

In the short run, instead of independently maintaining changes to
configurations to limit parallelism or xz memory usage, etc, we
could develop an optional common include file where such limits
are shared across the community.

In the longer run, changes to how bitbake schedules work may be needed.

Richard says that there was a make/build server idea and maybe even a
patch from a while ago. It may be in one of his poky-contrib branches.
I took a few minutes to look but nothing popped up. A set of keywords to
search for might help me find it.

Someone mentioned that while ninja has not been open to accepting any
patches that would complicate and potentially slow down builds, there
is a fork of ninja calls 'samurai' that does seem to be open to some
improvements that we could benefit from.

It was also suggested that  there were existing defects in the YP BZ (2)
but I didn't find any earlier and it's too late in my day to start
looking now! If no one replies with a relevant BZ ID, I'll create one.

I'm sure I missed some things that were mentioned but Trevor Woerner
sometimes takes notes so I'll check on them once / if they are sent out.

../Randy


1) https://www.yoctoproject.org/public-virtual-meetings/

2) https://bugzilla.yoctoproject.org/

> 
> Alex
> 
> On Sun, 11 Apr 2021 at 18:08, Gmane Admin <gley-yocto@m.gmane-mx.org 
> <mailto:gley-yocto@m.gmane-mx.org>> wrote:
> 
>     Op 11-04-2021 om 17:55 schreef Alexander Kanavin:
>      > On Sun, 11 Apr 2021 at 17:49, Gmane Admin
>     <gley-yocto@m.gmane-mx.org <mailto:gley-yocto@m.gmane-mx.org>
>      > <mailto:gley-yocto@m.gmane-mx.org
>     <mailto:gley-yocto@m.gmane-mx.org>>> wrote:
>      >
>      >     Yes, and make project doesn't care, because make is called
>     with -j
>      >     16 so
>      >     that is what it does.
>      >
>      >     So here's my pitch: bitbake can stop processes spawned by
>     make, because
>      >     it knows that it started make on 4 recipies, each with -j 16. The
>      >     individual makes don't know about each other.
>      >
>      >
>      > And neither they should. They can simply abstain from spawning new
>      > compilers if used RAM is, say, at 90% total. Then bitbake does
>     not have
>      > to get involved in babysitting those makes.
>      >
>      > Alex
>     Bitbake does a lot of babysitting anyway :-) And is pretty good at
>     it too.
> 
>     To me, fixing make et al. is more work and less effective then adding a
>     feature to bitbake. The only way to know how much memory the compiler
>     will use for each spawned compiler is to let it run. And then it's
>     too late.
> 
>     This memory issue is all over our eco system and nobody cares (kernel,
>     make etc.) The only thing moving is systemd's oom killer will arrive
>     and
>     start killing processes. So that will just stop our builds from
>     completing.
> 
>     Yeah, I prefer a babysitter over a child murderer :-)
> 
>     Ferry
> 
> 
> 
> 
> 
> 
> 
> 


-- 
# Randy MacLeod
# Wind River Linux