Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression

[Andre McCurdy <armccurdy@gmail.com>]

On Fri, Jun 2, 2017 at 4:40 AM, Alexander Kanavin
<alexander.kanavin@linux.intel.com> wrote:
> On 06/01/2017 10:02 PM, Andre McCurdy wrote:
>>>
>>> RPM's default is single-threaded gz; the change greatly helps with
>>> both buildtimes (when there is a small number of large-sized packages)
>>> and disk space taken by resulting rpms.
>>
>> Do you have any test results for that? Obviously build times will be
>> longer when building with a single CPU since xz compression needs more
>> CPU than gzip, but I'm curious how many CPUs are required before the
>> multiple threads win over and build times actually get shorter. 4? 8?
>
> Yes; it's in the cover letter.

Yes, I only saw that after sending my reply. I shouldn't read emails
in reverse order... sorry about that.

The cover letter benchmarks didn't mention the 22 CPU cores though, so
it's good to have that clarified. Seeing some results from a 4 or 8
core machine would be interesting too.

>> One other concern with parallel xz compression is memory usage. Single
>> threaded xz needs more memory than gzip and running multiple xz
>> compression threads together will push the memory usage up again.
>> Maybe not an issue if the peak is still acceptable, but something to
>> be aware of (I've given up trying to build in 4GB, but hopefully 8GB
>> build machines still have some life left in them...).>
>
> I haven't seen anything resembling low-memory or OOM in my testing, even
> when running 44 rpm instances at the same time, each with their own several
> threads, each thread handling a package, and compression for the package
> again being done in a multi-threaded way. But then I have 22 cores and 32G
> RAM here.
>
> So do try the patches please, and report the findings, YMMV.
>
> Alex
>