Re: Build time data

[Chris Tapp <opensource@keylevel.com>]

On 12 Apr 2012, at 15:34, Darren Hart wrote:
> 
> 
> On 04/12/2012 07:08 AM, Björn Stenberg wrote:
>> Darren Hart wrote:
>>> /dev/md0        /build          ext4 
>>> noauto,noatime,nodiratime,commit=6000
>> 
>> A minor detail: 'nodiratime' is a subset of 'noatime', so there is no
>> need to specify both.
> 
> Excellent, thanks for the tip.
> 
>> 
>>> I run on a beast with 12 cores, 48GB of RAM, OS and sources on a
>>> G2 Intel SSD, with two Seagate Barracudas in a RAID0 array for my
>>> /build partition. I run a headless Ubuntu 11.10 (x86_64)
>>> installation running the 3.0.0-16-server kernel. I can build
>>> core-image-minimal in < 30 minutes and core-image-sato in < 50
>>> minutes from scratch.
>> 
>> I'm guessing those are rather fast cores? 
> 
> They are:
> model name	: Intel(R) Xeon(R) CPU           X5680  @ 3.33GHz

Nice, but well out of my budget - I've got to make do with what one of your CPUs costs for the whole system ;-)

> 
>> I build on a different type
>> of beast: 64 cores at 2.1GHz and 128 GB ram. The OS is on a single
>> SSD and the build dir (and sources) is on a RAID0 array of Intel 520
>> SSDs. Kernel is the same ubuntu 3.0.0-16-server as yours.
> 
> Now that I think about it, my downloads are on the RAID0 array too.
> 
> One thing that comes to mind is the parallel settings, BB_NUMBER_THREADS
> and PARALLEL_MAKE. I noticed a negative impact if I increased these
> beyond 12 and 14 respectively. I tested this with bb-matrix
> (scripts/contrib/bb-perf/bb-matrix.sh). The script is a bit fickle, but
> can provide useful results and killer 3D surface plots of build time
> with BB and PM on the axis. Can't seem to find a plot image at the
> moment for some reason...
> 
>> 
>> Yet for all the combined horsepower, I am unable to match your time
>> of 30 minutes for core-image-minimal. I clock in at around 37 minutes
>> for a qemux86-64 build with ipk output:
>> 
>> ------ NOTE: Tasks Summary: Attempted 1363 tasks of which 290 didn't
>> need to be rerun and all succeeded.
>> 
>> real    36m32.118s user    214m39.697s sys     108m49.152s ------
>> 
>> These numbers also show that my build is running less than 9x
>> realtime, indicating that 80% of my cores sit idle most of the time.
> 
> Yup, that sounds about right. The build has a linear component to it,
> and anything above about 12 just doesn't help. In fact the added
> scheduling overhead seems to hurt.
> 
>> This confirms what "ps xf" says during the builds: Only rarely is
>> bitbake running more than a handful tasks at once, even with
>> BB_NUMBER_THREADS at 64. And many of these tasks are in turn running
>> sequential loops on a single core.
>> 
>> I'm hoping to find time soon to look deeper into this issue and
>> suggest remedies. It my distinct feeling that we should be able to
>> build significantly faster on powerful machines.
>> 
> 
> Reducing the dependency chains that result in the linear component of
> the build (forcing serialized execution) is one place we've focused, and
> could probably still use some attention. CC'ing RP as he's done a lot there.

Current plan for a 'budget' system is:

DX79TO motherboard, i7 3820, 16GB RAM, a pair of 60GB OCZ Vertex III's in RAID-0 for downloads / build, SATA HD for OS (Ubuntu 11.10 x86_64).

That'll give me a 2.7x boost just on CPU and the SSDs (and maybe some over-clocking) will give some more.

Not sure if SSDs in RAID-0 will give any boost, so I'll run some tests.

Thanks to all for the comments in this thread.

Chris Tapp

opensource@keylevel.com
www.keylevel.com