feature request: avoid getting the server stuck because of high "-j" value

feature request: avoid getting the server stuck because of high "-j" value

[Gianfranco Costamagna]

Hello Yocto developers!


recently I found a really nice make feature, that tunes at runtime the number of parallel
execution jobs, according to the current system load [1].

I think such a feature would be really nice on server were multiple yocto builds are run in parallel,
and when people forgets to put the correct number of parallel build jobs.

E.g., a number around CPU+1 is nice, but sometimes CPU*2 gives better results.
Unfortunately if somebody else starts a build in parallel, that CPU*2 locks really hard the server, specially
when building qt-based projects (e.g. qtbase, qtwebengine).

Tuning such value at runtime will probably end in faster compilation time and lower swapping on remote
servers.
(I would think also to something that checks the amount of free RAM)


is such a feature feasible?

[1] https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel

thanks,

Gianfranco

[feature request] avoid getting the server stuck because of high "-j" value

[Gianfranco Costamagna]

Hello Yocto developers!




recently I found a really nice make feature, that tunes at runtime the number of parallel

execution jobs, according to the current system load [1].


I think such a feature would be really nice on server were multiple yocto builds are run in parallel,

and when people forgets to put the correct number of parallel build jobs.


E.g., a number around CPU+1 is nice, but sometimes CPU*2 gives better results.

Unfortunately if somebody else starts a build in parallel, that CPU*2 locks really hard the server, specially

when building qt-based projects (e.g. qtbase, qtwebengine).


Tuning such value at runtime will probably end in faster compilation time and lower swapping on remote

servers.

(I would think also to something that checks the amount of free RAM)



is such a feature feasible?
should I open a bug for it?


[1] https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel


thanks,


Gianfranco

Re: [feature request] avoid getting the server stuck because of high "-j" value

[Alexander Kanavin]

On 08/04/2017 01:58 PM, Gianfranco Costamagna wrote:

> recently I found a really nice make feature, that tunes at runtime the number of parallel
> 
> execution jobs, according to the current system load [1].
> 
> 
> I think such a feature would be really nice on server were multiple yocto builds are run in parallel,
> 
> and when people forgets to put the correct number of parallel build jobs.
> 
> 
> E.g., a number around CPU+1 is nice, but sometimes CPU*2 gives better results.
> 
> Unfortunately if somebody else starts a build in parallel, that CPU*2 locks really hard the server, specially
> 
> when building qt-based projects (e.g. qtbase, qtwebengine).

If the cause of the slowdown with qt stuff is that RAM is fully 
utilized, and swap has to be taken into use, does checking load average 
help with that? My understanding is that it does not. You simply have to 
introduce per-recipe limits:

PARALLEL_MAKE_pn-qtwebengine  = "-j 8"


Also, many projects are moving away from makefiles towards ninja build 
system, which does not have support for -l at all.


Alex

Re: [feature request] avoid getting the server stuck because of high "-j" value

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1568 bytes --]

It's not just feasible but you can do it right now.

PARALLEL_MAKE is the variable that sets how parallel the build should be.
By default we pass -j<number of processors".

Just set it to -l2.5 or whatever load average you want.

Ross

On 4 August 2017 at 11:58, Gianfranco Costamagna <
costamagnagianfranco@yahoo.it> wrote:

> Hello Yocto developers!
>
>
>
>
> recently I found a really nice make feature, that tunes at runtime the
> number of parallel
>
> execution jobs, according to the current system load [1].
>
>
> I think such a feature would be really nice on server were multiple yocto
> builds are run in parallel,
>
> and when people forgets to put the correct number of parallel build jobs.
>
>
> E.g., a number around CPU+1 is nice, but sometimes CPU*2 gives better
> results.
>
> Unfortunately if somebody else starts a build in parallel, that CPU*2
> locks really hard the server, specially
>
> when building qt-based projects (e.g. qtbase, qtwebengine).
>
>
> Tuning such value at runtime will probably end in faster compilation time
> and lower swapping on remote
>
> servers.
>
> (I would think also to something that checks the amount of free RAM)
>
>
>
> is such a feature feasible?
> should I open a bug for it?
>
>
> [1] https://www.gnu.org/software/make/manual/html_node/
> Parallel.html#Parallel
>
>
> thanks,
>
>
> Gianfranco
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 2246 bytes --]

Re: [feature request] avoid getting the server stuck because of high "-j" value

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1941 bytes --]

Alex's comment is right that the semantics of PARALLEL_MAKE are GNU Make
specific so you might find some recipes don't handle anything but there
being -j values.  You might have a little bit of fixing to do.

Ross

On 4 August 2017 at 13:00, Burton, Ross <ross.burton@intel.com> wrote:

> It's not just feasible but you can do it right now.
>
> PARALLEL_MAKE is the variable that sets how parallel the build should be.
> By default we pass -j<number of processors".
>
> Just set it to -l2.5 or whatever load average you want.
>
> Ross
>
> On 4 August 2017 at 11:58, Gianfranco Costamagna <
> costamagnagianfranco@yahoo.it> wrote:
>
>> Hello Yocto developers!
>>
>>
>>
>>
>> recently I found a really nice make feature, that tunes at runtime the
>> number of parallel
>>
>> execution jobs, according to the current system load [1].
>>
>>
>> I think such a feature would be really nice on server were multiple yocto
>> builds are run in parallel,
>>
>> and when people forgets to put the correct number of parallel build jobs.
>>
>>
>> E.g., a number around CPU+1 is nice, but sometimes CPU*2 gives better
>> results.
>>
>> Unfortunately if somebody else starts a build in parallel, that CPU*2
>> locks really hard the server, specially
>>
>> when building qt-based projects (e.g. qtbase, qtwebengine).
>>
>>
>> Tuning such value at runtime will probably end in faster compilation time
>> and lower swapping on remote
>>
>> servers.
>>
>> (I would think also to something that checks the amount of free RAM)
>>
>>
>>
>> is such a feature feasible?
>> should I open a bug for it?
>>
>>
>> [1] https://www.gnu.org/software/make/manual/html_node/Parallel.
>> html#Parallel
>>
>>
>> thanks,
>>
>>
>> Gianfranco
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>
>

[-- Attachment #2: Type: text/html, Size: 3042 bytes --]

Re: [feature request] avoid getting the server stuck because of high "-j" value

[Gianfranco Costamagna]

Thanks you all for the valuable answers!

cheers,

Gianfranco