Shorter build time?

Shorter build time?

[Takashi Matsuzawa]

Hello Yocto.

Well, this may not be a very new topic, but I wonder if you have recent suggetions.
So far I have tried the followings and see some improvement, making the build time to days to hours.

1) All of the directories, DL_DIR, SSTATE_CACHE, TMPDIR, DEPLOY_DIR are on SSD.
2) .repo and working directory (where I synch my recipes) also on SSD.
3) Set BB_NUMBER_PARSE_THREADS/PARALLEL_MAKE/BB_NUMBER_THREADS to something like 20, 30, etc.
4) Enable use of ICECC.
5) Use pigz, pbzip2, pxz instead of gzip, bzip2 and xz.

I can see with 3-5), I am benefited from my multicore PCs and see some scaling effects (if I can add more PCs or CPU core.)
However I think there is i the build not paralleled or distributed easily.

i) python tasks
They say python has slimitation of being single thread, and could be a bottle neck since most of yocto build tasks are in python, if not native/cross toolchain jobs.

ii) zip / tar tasks
e.g. seeing rootfs taking forever, jobs not distributed to the other machines.
I understand it is writing files to one target tree and doing a bit tar/zip for the files, being one big job though.

I wonder if I can do anything with i) and ii) at least to make them processed in parallel.
ICECC only can dispatch gcc jobs I believe.
Any previous attemps to do the similar to non-gcc jobs?

Re: Shorter build time?

[Daniel.]

Hi Takashi,

I read that python can't execute multiple threads simultaneously, but
that wouldn't be a problem if each python task is executed at its own
interpreter instance (process). This last statement is what I don't
really know, some experts on Yocto's internals may clarify that, maybe
:)

Regards,

2016-06-30 22:29 GMT-03:00 Takashi Matsuzawa <tmatsuzawa@uievolution.com>:
>
> Hello Yocto.
>
> Well, this may not be a very new topic, but I wonder if you have recent suggetions.
> So far I have tried the followings and see some improvement, making the build time to days to hours.
>
> 1) All of the directories, DL_DIR, SSTATE_CACHE, TMPDIR, DEPLOY_DIR are on SSD.
> 2) .repo and working directory (where I synch my recipes) also on SSD.
> 3) Set BB_NUMBER_PARSE_THREADS/PARALLEL_MAKE/BB_NUMBER_THREADS to something like 20, 30, etc.
> 4) Enable use of ICECC.
> 5) Use pigz, pbzip2, pxz instead of gzip, bzip2 and xz.
>
> I can see with 3-5), I am benefited from my multicore PCs and see some scaling effects (if I can add more PCs or CPU core.)
> However I think there is i the build not paralleled or distributed easily.
>
> i) python tasks
> They say python has slimitation of being single thread, and could be a bottle neck since most of yocto build tasks are in python, if not native/cross toolchain jobs.
>
> ii) zip / tar tasks
> e.g. seeing rootfs taking forever, jobs not distributed to the other machines.
> I understand it is writing files to one target tree and doing a bit tar/zip for the files, being one big job though.
>
> I wonder if I can do anything with i) and ii) at least to make them processed in parallel.
> ICECC only can dispatch gcc jobs I believe.
> Any previous attemps to do the similar to non-gcc jobs?
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto



-- 
"Do or do not. There is no try"
  Yoda Master

Re: Shorter build time?

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

On 1 July 2016 at 17:37, Daniel. <danielhilst@gmail.com> wrote:

> I read that python can't execute multiple threads simultaneously, but
> that wouldn't be a problem if each python task is executed at its own
> interpreter instance (process). This last statement is what I don't
> really know, some experts on Yocto's internals may clarify that, maybe
>

Yes, each worker is a separate Python process.

Ross

[-- Attachment #2: Type: text/html, Size: 845 bytes --]

Re: Shorter build time?

[Daniel.]

So we have full parallelism :)

2016-07-01 14:01 GMT-03:00 Burton, Ross <ross.burton@intel.com>:
>
> On 1 July 2016 at 17:37, Daniel. <danielhilst@gmail.com> wrote:
>>
>> I read that python can't execute multiple threads simultaneously, but
>> that wouldn't be a problem if each python task is executed at its own
>> interpreter instance (process). This last statement is what I don't
>> really know, some experts on Yocto's internals may clarify that, maybe
>
>
> Yes, each worker is a separate Python process.
>
> Ross



-- 
"Do or do not. There is no try"
  Yoda Master

Re: Shorter build time?

[Takashi Matsuzawa]

[-- Attachment #1: Type: text/plain, Size: 1125 bytes --]

Hmm, thank you for your comments.

So, they may be talking about single-thread nature within a python program, and as for bitbake python tasks they should be working in parallel.


There seems to be PyPy, Stackless Python, etc. but I am not sure they can be tried 'in-place' to see if they work faster.

________________________________
From: Daniel. <danielhilst@gmail.com>
Sent: Saturday, July 2, 2016 2:13 AM
To: Burton, Ross
Cc: Takashi Matsuzawa; yocto@yoctoproject.org
Subject: Re: [yocto] Shorter build time?

So we have full parallelism :)

2016-07-01 14:01 GMT-03:00 Burton, Ross <ross.burton@intel.com>:
>
> On 1 July 2016 at 17:37, Daniel. <danielhilst@gmail.com> wrote:
>>
>> I read that python can't execute multiple threads simultaneously, but
>> that wouldn't be a problem if each python task is executed at its own
>> interpreter instance (process). This last statement is what I don't
>> really know, some experts on Yocto's internals may clarify that, maybe
>
>
> Yes, each worker is a separate Python process.
>
> Ross



--
"Do or do not. There is no try"
  Yoda Master

[-- Attachment #2: Type: text/html, Size: 2105 bytes --]

Re: Shorter build time?

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 369 bytes --]

On 2 July 2016 at 03:12, Takashi Matsuzawa <tmatsuzawa@uievolution.com>
wrote:

> There seems to be PyPy, Stackless Python, etc. but I am not sure they can
> be tried 'in-place' to see if they work faster.
>
In the context of a bitbake build, the overhead of Python itself is utterly
insignificant in the context of doing the configure/compile/package.

Ross

[-- Attachment #2: Type: text/html, Size: 743 bytes --]

Re: Shorter build time?

[Marcelo E. Magallon]

On Sat, Jul 02, 2016 at 09:02:32AM +0100, Burton, Ross wrote:
> On 2 July 2016 at 03:12, Takashi Matsuzawa <tmatsuzawa@uievolution.com>
> wrote:
> 
> > There seems to be PyPy, Stackless Python, etc. but I am not sure
> > they can be tried 'in-place' to see if they work faster.
> >
> In the context of a bitbake build, the overhead of Python itself is
> utterly insignificant in the context of doing the
> configure/compile/package.

That is true, but I've seen cases where the start up time of Python (or
the time it takes bitbake to start "doing work") is *perceptually* much
larger than the work done by the recipe, or at least it looks like it
represents a sigficant fraction of the total work. That's where PyPy
might help.

Marcelo

Re: Shorter build time?

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

On 4 July 2016 at 19:10, Marcelo E. Magallon <marcelo.magallon@hpe.com>
wrote:

> That is true, but I've seen cases where the start up time of Python (or
> the time it takes bitbake to start "doing work") is *perceptually* much
> larger than the work done by the recipe, or at least it looks like it
> represents a sigficant fraction of the total work. That's where PyPy
> might help.
>

Measuring that overhead should be fairly simple, and if it's true then
having a pool of workers would be a good optimisation.

Ross

[-- Attachment #2: Type: text/html, Size: 961 bytes --]

Re: Shorter build time?

[Takashi Matsuzawa]

[-- Attachment #1: Type: text/plain, Size: 1004 bytes --]

By the way, PyPy seems to have some restriction that is not fully compatible with CPyhon and bitbake scripts may not run as they are.  I tried it yesterday and learned that it is not a simple in-place replacement.


________________________________
From: Burton, Ross <ross.burton@intel.com>
Sent: Tuesday, July 5, 2016 3:50 AM
To: Marcelo E. Magallon
Cc: Takashi Matsuzawa; yocto@yoctoproject.org
Subject: Re: [yocto] Shorter build time?


On 4 July 2016 at 19:10, Marcelo E. Magallon <marcelo.magallon@hpe.com<mailto:marcelo.magallon@hpe.com>> wrote:
That is true, but I've seen cases where the start up time of Python (or
the time it takes bitbake to start "doing work") is *perceptually* much
larger than the work done by the recipe, or at least it looks like it
represents a sigficant fraction of the total work. That's where PyPy
might help.

Measuring that overhead should be fairly simple, and if it's true then having a pool of workers would be a good optimisation.

Ross


[-- Attachment #2: Type: text/html, Size: 2127 bytes --]

Re: Shorter build time?

[Daniel.]

The only task that I see taking really long is the do_rootfs. And I
think that it takes that long because disks are slooow :)

2016-07-04 20:14 GMT-03:00 Takashi Matsuzawa <tmatsuzawa@uievolution.com>:
> By the way, PyPy seems to have some restriction that is not fully compatible
> with CPyhon and bitbake scripts may not run as they are.  I tried it
> yesterday and learned that it is not a simple in-place replacement.
>
>
>
> ________________________________
> From: Burton, Ross <ross.burton@intel.com>
> Sent: Tuesday, July 5, 2016 3:50 AM
> To: Marcelo E. Magallon
> Cc: Takashi Matsuzawa; yocto@yoctoproject.org
> Subject: Re: [yocto] Shorter build time?
>
>
> On 4 July 2016 at 19:10, Marcelo E. Magallon <marcelo.magallon@hpe.com>
> wrote:
>>
>> That is true, but I've seen cases where the start up time of Python (or
>> the time it takes bitbake to start "doing work") is *perceptually* much
>> larger than the work done by the recipe, or at least it looks like it
>> represents a sigficant fraction of the total work. That's where PyPy
>> might help.
>
>
> Measuring that overhead should be fairly simple, and if it's true then
> having a pool of workers would be a good optimisation.
>
> Ross
>
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>



-- 
"Do or do not. There is no try"
  Yoda Master

Re: Shorter build time?

[Carlos Alberto Lopez Perez]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On 01/07/16 03:29, Takashi Matsuzawa wrote:
> 
> Hello Yocto.
> 
> Well, this may not be a very new topic, but I wonder if you have recent suggetions.
> So far I have tried the followings and see some improvement, making the build time to days to hours.
> 
> 1) All of the directories, DL_DIR, SSTATE_CACHE, TMPDIR, DEPLOY_DIR are on SSD.
> 2) .repo and working directory (where I synch my recipes) also on SSD.
> 3) Set BB_NUMBER_PARSE_THREADS/PARALLEL_MAKE/BB_NUMBER_THREADS to something like 20, 30, etc.
> 4) Enable use of ICECC.
> 5) Use pigz, pbzip2, pxz instead of gzip, bzip2 and xz.
> 

6) Selecting ipk instead of rpm speeds ups all the packaging steps.

@local.conf:
-PACKAGE_CLASSES ?= "package_rpm"
+PACKAGE_CLASSES ?= "package_ipk"


https://lists.yoctoproject.org/pipermail/poky/2011-May/006362.html


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]