Re: [OE-core] [PATCH 1/1] bitbake.conf: Set ZSTD_THREADS to half of cpu number

[Robert Yang <liezhi.yang@windriver.com>]

Hi RP,

On 11/9/21 6:26 PM, Richard Purdie wrote:
> On Tue, 2021-11-09 at 10:56 +0100, Alexander Kanavin wrote:
>> On Tue, 9 Nov 2021 at 10:51, Robert Yang <liezhi.yang@windriver.com> wrote:
>>>> Maybe, but once we start doing tweaks like this, where do they stop? Hey,
>>> I'd
>>>
>>> I think that it's not only a performance tweak, but this should be
>>> considered
>>> as
>>> a bug since it would be failed to build on powerful machine, which isn't
>>> good
>>> for oe-core's user experience.
>>>
>>
>>
>> I managed to bring down a powerful machine just yesterday by building webkit
>> and llvm together at the same time. They together ate all the RAM, and someone
>> in the office had to go and reset the box.
>> Do I see it as a bug? Absolutely not: it's in fact my fault for not ensuring
>> system resource consumption doesn't overwhelm the machine. This is the same:
>> if you do not have RAM to match the cores on your specific rig, add RAM or
>> limit the threads.
> 
> I can see both sides of this. If we had a pool for all the threads and the
> pieces of the system all used the same pool, it would work well as implemented.
> Sadly the bitbake threads, the make processes and now the compression threads
> are all separate.
> 
> We do see the autobuilder break due to load issues too and on that we have
> scaled back some of the defaults but not ZSTD as yet that I recall.
> 
> As such I think limiting the upper value on number of threads may be a good
> compromise, both for compression and maybe for parallel make too.

What should we do next, please? Limit bb thread + parallel make + zstd thread?
Or just limit zstd's thread since other people may concern about the
performance.

// Robert

> 
> Cheers,
> 
> Richard
>