Recommended Hardware for building

Recommended Hardware for building

[Oliver Novakovic]

[-- Attachment #1: Type: text/plain, Size: 1393 bytes --]

Dear all,

 

since we are using Yocto in our projects, I'd like to set up a dedicated
Build PC or server for the Yocto build tasks (including Jenkins).

 

Can anyone recommend a reasonable performant hardware setup to use ?

What should be considered ? Are there any pitfalls ? What about
bottlenecks in the build system ?

 

Specifically:

How many cores are recommended ? And how much cache is necessary ?
How much of the main memory does Yocto really use ? Is 32 GB sufficient
or should I go for 64 ?

Does it make sense to use two SSDs as Raid0  to get builds faster ?

 

Any hints, recommendations or links to further reading will be highly
appreciated.

 

Thanks in advance,

Oliver


Alpine Electronics R&D Europe GmbH, Vor dem Lauch 14, 70567 Stuttgart,
Registergericht Stuttgart, HRB 25265
Geschaeftsfuehrer: Toshinori Kobayashi, Shuji Taguchi
Working office: Vor dem Lauch 14, 70567 Stuttgart

ATTENTION: This e-mail and any files transmitted with it may be privileged
and/or confidential, and are intended solely for the use of the individual
or entity to whom this  e-mail is addressed. If you are not one of the
named recipients or otherwise  have reason to believe that you have
received this e-mail in error, please notify  the sender and delete this
message immediately from your computer. We thank you for your cooperation.

[-- Attachment #2: Type: text/html, Size: 4236 bytes --]

Re: Recommended Hardware for building

[Burton, Ross]

On 2 October 2014 10:36, Oliver Novakovic <Oliver.Novakovic@alpine.de> wrote:
> Can anyone recommend a reasonable performant hardware setup to use ?
>
> What should be considered ? Are there any pitfalls ? What about bottlenecks
> in the build system ?
>
> Specifically:
>
> How many cores are recommended ? And how much cache is necessary ?
> How much of the main memory does Yocto really use ? Is 32 GB sufficient or
> should I go for 64 ?
>
> Does it make sense to use two SSDs as Raid0  to get builds faster ?

As much of everything as you can afford.  :)  The build isn't heavy in
any particular metric, so don't sacrifice RAM for SSDs for example.

RAID 0 over SSD would be nice and fast, but I prefer having a good
amount of RAM and a tuned ext4 (no journal, long commit delay) so data
doesn't actually hit the disk as frequently. Keeping the actual build
directories on a separate disk is good for performance and not causing
data loss when you lose a disk.

There are people that have 64GB in machines and then set TMPDIR to a
tmpfs.  Surprisingly this isn't that much faster (5% or so), but it's
a lot easier on the hardware and power consumption.

Ross

Re: Recommended Hardware for building

[Chris Tapp]

On 2 Oct 2014, at 11:04, Burton, Ross <ross.burton@intel.com> wrote:

> On 2 October 2014 10:36, Oliver Novakovic <Oliver.Novakovic@alpine.de> wrote:
>> Can anyone recommend a reasonable performant hardware setup to use ?
>> 
>> What should be considered ? Are there any pitfalls ? What about bottlenecks
>> in the build system ?
>> 
>> Specifically:
>> 
>> How many cores are recommended ? And how much cache is necessary ?
>> How much of the main memory does Yocto really use ? Is 32 GB sufficient or
>> should I go for 64 ?
>> 
>> Does it make sense to use two SSDs as Raid0  to get builds faster ?
> 
> As much of everything as you can afford.  :)  The build isn't heavy in
> any particular metric, so don't sacrifice RAM for SSDs for example.
> 
> RAID 0 over SSD would be nice and fast, but I prefer having a good
> amount of RAM and a tuned ext4 (no journal, long commit delay) so data
> doesn't actually hit the disk as frequently. Keeping the actual build
> directories on a separate disk is good for performance and not causing
> data loss when you lose a disk.
> 
> There are people that have 64GB in machines and then set TMPDIR to a
> tmpfs.  Surprisingly this isn't that much faster (5% or so), but it's
> a lot easier on the hardware and power consumption.

My experience:

I've got a quad core with hyper-threading (so 8 usable cores) running at about 3.8 GHz, 16GB of RAM and use multiple SSDs - one to hold the meta data, downloads and top level build areas (local.conf, etc) and have the TMPDIR on a second SSD (so, as Ross says, I don't get a surprise when it wears out!).

I can build my images (basically an x11 image) in just under 60 minutes (once all the files have been fetched). I run with BB_NUMBER_THREADS and PARALLEL_MAKE both set to 16 to make sure the cores are fully loaded as much as possible (other says that should be 8 and 8 to reduce scheduling overhead).

During the build the system is CPU bound quite a bit of the time (so more cores should help), but there are significant periods where the build dependency chain means this isn't the case and only two or three cores are active. Previously I recall comparing results with someone else and finding that having lots more cores (24, I think) didn't give a significant improvement in build time (certainly not for the 3x system build cost).

I've never seen peak memory usage go much above 9 GB during a build, and the peaks generally coincide with linking activities for "big" items (gcc, eglibc). This is likely to go higher with more active threads.

I started out with a RAID-0 SSD build array, but I didn't really see any difference over a single high-spec (consumer) SSD. As Ross said, running a fast file system on the disk is a good idea.

--

Chris Tapp
opensource@keylevel.com
www.keylevel.com

----
You can tell you're getting older when your car insurance gets real cheap!

Re: Recommended Hardware for building

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3820 bytes --]

On Thu, Oct 02, 2014 at 05:51:29PM +0100, Chris Tapp wrote:
> 
> On 2 Oct 2014, at 11:04, Burton, Ross <ross.burton@intel.com> wrote:
> 
> > On 2 October 2014 10:36, Oliver Novakovic <Oliver.Novakovic@alpine.de> wrote:
> >> Can anyone recommend a reasonable performant hardware setup to use ?
> >> 
> >> What should be considered ? Are there any pitfalls ? What about bottlenecks
> >> in the build system ?

you should start by saying what you're going to build, my experience is
quite different when building "small" images like console-image or even
x11-image and "big" images/feeds which contain whole qt5 stack, 3 webkits and
2 chromium builds.

In general: bitbake will better utilize all available performance with
bigger image (e.g. build time for console image won't change so much if
you go from 8 cores to 24, but building e.g. just webkit alone will be
more than twice faster on 24 cores).

Regards,

> >> Specifically:
> >> 
> >> How many cores are recommended ? And how much cache is necessary ?
> >> How much of the main memory does Yocto really use ? Is 32 GB sufficient or
> >> should I go for 64 ?
> >> 
> >> Does it make sense to use two SSDs as Raid0  to get builds faster ?
> > 
> > As much of everything as you can afford.  :)  The build isn't heavy in
> > any particular metric, so don't sacrifice RAM for SSDs for example.
> > 
> > RAID 0 over SSD would be nice and fast, but I prefer having a good
> > amount of RAM and a tuned ext4 (no journal, long commit delay) so data
> > doesn't actually hit the disk as frequently. Keeping the actual build
> > directories on a separate disk is good for performance and not causing
> > data loss when you lose a disk.
> > 
> > There are people that have 64GB in machines and then set TMPDIR to a
> > tmpfs.  Surprisingly this isn't that much faster (5% or so), but it's
> > a lot easier on the hardware and power consumption.
> 
> My experience:
> 
> I've got a quad core with hyper-threading (so 8 usable cores) running at about 3.8 GHz, 16GB of RAM and use multiple SSDs - one to hold the meta data, downloads and top level build areas (local.conf, etc) and have the TMPDIR on a second SSD (so, as Ross says, I don't get a surprise when it wears out!).
> 
> I can build my images (basically an x11 image) in just under 60 minutes (once all the files have been fetched). I run with BB_NUMBER_THREADS and PARALLEL_MAKE both set to 16 to make sure the cores are fully loaded as much as possible (other says that should be 8 and 8 to reduce scheduling overhead).
> 
> During the build the system is CPU bound quite a bit of the time (so more cores should help), but there are significant periods where the build dependency chain means this isn't the case and only two or three cores are active. Previously I recall comparing results with someone else and finding that having lots more cores (24, I think) didn't give a significant improvement in build time (certainly not for the 3x system build cost).
> 
> I've never seen peak memory usage go much above 9 GB during a build, and the peaks generally coincide with linking activities for "big" items (gcc, eglibc). This is likely to go higher with more active threads.
> 
> I started out with a RAID-0 SSD build array, but I didn't really see any difference over a single high-spec (consumer) SSD. As Ross said, running a fast file system on the disk is a good idea.
> 
> --
> 
> Chris Tapp
> opensource@keylevel.com
> www.keylevel.com
> 
> ----
> You can tell you're getting older when your car insurance gets real cheap!
> 
> -- 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: Recommended Hardware for building

[Bryan Evenson]

Oliver,

> -----Original Message-----
> From: yocto-bounces@yoctoproject.org [mailto:yocto-
> bounces@yoctoproject.org] On Behalf Of Martin Jansa
> Sent: Thursday, October 02, 2014 3:09 PM
> To: Chris Tapp
> Cc: yocto@yoctoproject.org
> Subject: Re: [yocto] Recommended Hardware for building
> 
> On Thu, Oct 02, 2014 at 05:51:29PM +0100, Chris Tapp wrote:
> >
> > On 2 Oct 2014, at 11:04, Burton, Ross <ross.burton@intel.com> wrote:
> >
> > > On 2 October 2014 10:36, Oliver Novakovic <Oliver.Novakovic@alpine.de>
> wrote:
> > >> Can anyone recommend a reasonable performant hardware setup to
> use ?
> > >>
> > >> What should be considered ? Are there any pitfalls ? What about
> > >> bottlenecks in the build system ?
> 
> you should start by saying what you're going to build, my experience is quite
> different when building "small" images like console-image or even x11-image
> and "big" images/feeds which contain whole qt5 stack, 3 webkits and
> 2 chromium builds.

Agreed, what you are building and what your goals are makes a difference in what you need.  I have a build machine setup that is mainly used to verify everything builds correctly after committing changes.  It's an Intel i3-3220 with 8GB RAM.  The autobuilder is setup on a Linux VM which is given 4GB RAM and does not recognize the extra Hyper-threaded cores, meaning it acts as a dual core machine.  Rebuild of my console image typically takes under 20 minutes, and most of that time is packaging/install.  After the initial build, there really isn't much for my system that needs to get rebuilt between commits.

So if you are looking for a build machine that is outside your normal workflow, a $400 PC may be enough for you.  If this machine is for your development build and you have a have a lot of graphic applications that you need to build, you may want something more in line with what other people are suggesting.

Regards,
Bryan

> 
> In general: bitbake will better utilize all available performance with bigger
> image (e.g. build time for console image won't change so much if you go from
> 8 cores to 24, but building e.g. just webkit alone will be more than twice
> faster on 24 cores).
> 
> Regards,
> 
> > >> Specifically:
> > >>
> > >> How many cores are recommended ? And how much cache is necessary
> ?
> > >> How much of the main memory does Yocto really use ? Is 32 GB
> > >> sufficient or should I go for 64 ?
> > >>
> > >> Does it make sense to use two SSDs as Raid0  to get builds faster ?
> > >
> > > As much of everything as you can afford.  :)  The build isn't heavy
> > > in any particular metric, so don't sacrifice RAM for SSDs for example.
> > >
> > > RAID 0 over SSD would be nice and fast, but I prefer having a good
> > > amount of RAM and a tuned ext4 (no journal, long commit delay) so
> > > data doesn't actually hit the disk as frequently. Keeping the actual
> > > build directories on a separate disk is good for performance and not
> > > causing data loss when you lose a disk.
> > >
> > > There are people that have 64GB in machines and then set TMPDIR to a
> > > tmpfs.  Surprisingly this isn't that much faster (5% or so), but
> > > it's a lot easier on the hardware and power consumption.
> >
> > My experience:
> >
> > I've got a quad core with hyper-threading (so 8 usable cores) running at
> about 3.8 GHz, 16GB of RAM and use multiple SSDs - one to hold the meta
> data, downloads and top level build areas (local.conf, etc) and have the
> TMPDIR on a second SSD (so, as Ross says, I don't get a surprise when it
> wears out!).
> >
> > I can build my images (basically an x11 image) in just under 60 minutes
> (once all the files have been fetched). I run with BB_NUMBER_THREADS and
> PARALLEL_MAKE both set to 16 to make sure the cores are fully loaded as
> much as possible (other says that should be 8 and 8 to reduce scheduling
> overhead).
> >
> > During the build the system is CPU bound quite a bit of the time (so more
> cores should help), but there are significant periods where the build
> dependency chain means this isn't the case and only two or three cores are
> active. Previously I recall comparing results with someone else and finding
> that having lots more cores (24, I think) didn't give a significant improvement
> in build time (certainly not for the 3x system build cost).
> >
> > I've never seen peak memory usage go much above 9 GB during a build,
> and the peaks generally coincide with linking activities for "big" items (gcc,
> eglibc). This is likely to go higher with more active threads.
> >
> > I started out with a RAID-0 SSD build array, but I didn't really see any
> difference over a single high-spec (consumer) SSD. As Ross said, running a
> fast file system on the disk is a good idea.
> >
> > --
> >
> > Chris Tapp
> > opensource@keylevel.com
> > www.keylevel.com
> >
> > ----
> > You can tell you're getting older when your car insurance gets real cheap!
> >
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> 
> --
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

Re: Recommended Hardware for building

[Denys Dmytriyenko]

On Thu, Oct 02, 2014 at 11:04:43AM +0100, Burton, Ross wrote:
> On 2 October 2014 10:36, Oliver Novakovic <Oliver.Novakovic@alpine.de> wrote:
> > Can anyone recommend a reasonable performant hardware setup to use ?
> >
> > What should be considered ? Are there any pitfalls ? What about bottlenecks
> > in the build system ?
> >
> > Specifically:
> >
> > How many cores are recommended ? And how much cache is necessary ?
> > How much of the main memory does Yocto really use ? Is 32 GB sufficient or
> > should I go for 64 ?
> >
> > Does it make sense to use two SSDs as Raid0  to get builds faster ?
> 
> As much of everything as you can afford.  :)  The build isn't heavy in
> any particular metric, so don't sacrifice RAM for SSDs for example.
> 
> RAID 0 over SSD would be nice and fast, but I prefer having a good
> amount of RAM and a tuned ext4 (no journal, long commit delay) so data
> doesn't actually hit the disk as frequently. Keeping the actual build
> directories on a separate disk is good for performance and not causing
> data loss when you lose a disk.
> 
> There are people that have 64GB in machines and then set TMPDIR to a
> tmpfs.  Surprisingly this isn't that much faster (5% or so), but it's
> a lot easier on the hardware and power consumption.

That's how I roll - after I lost few SSD drives by constantly building and 
rebuilding stuff, I ended up shoving 64GB of RAM into my gear and building 
into RAM-based tmpfs. Of course, there are size limits to what can be built in 
one go, but there are workarounds and ways to serialize builds...

-- 
Denys

Re: Recommended Hardware for building

[Oliver Novakovic]

[-- Attachment #1: Type: text/plain, Size: 6800 bytes --]

Thanks a lot for all of the information so far. This will give me a good
starting point on configuring a build server or PC.

 

Basically, the image itself is very basic, but it does include the
complete QT5 release including QTWebkit which seems to be the most
demanding to build.

The rest are smaller support packages for connectivity and a QML based
HMI.

 

Thanks again,

Oliver

 

From: Bryan Evenson <bevenson@melinkcorp.com> [mailto:Bryan Evenson
<bevenson@melinkcorp.com>] 
Sent: Freitag, 3. Oktober 2014 14:48
To: Martin Jansa <martin.jansa@gmail.com>; Chris Tapp
<opensource@keylevel.com>; "Oliver.Novakovic@alpine.de"
<Oliver.Novakovic@alpine.de>
Cc: "yocto@yoctoproject.org" <yocto@yoctoproject.org>
Subject: RE: [yocto] Recommended Hardware for building

 


Oliver, 

> -----Original Message----- 
> From: yocto-bounces@yoctoproject.org [mailto:yocto- 
> bounces@yoctoproject.org] On Behalf Of Martin Jansa 
> Sent: Thursday, October 02, 2014 3:09 PM 
> To: Chris Tapp 
> Cc: yocto@yoctoproject.org 
> Subject: Re: [yocto] Recommended Hardware for building 
> 
> On Thu, Oct 02, 2014 at 05:51:29PM +0100, Chris Tapp wrote: 
> > 
> > On 2 Oct 2014, at 11:04, Burton, Ross wrote: 
> > 
> > > On 2 October 2014 10:36, Oliver Novakovic 
> wrote: 
> > >> Can anyone recommend a reasonable performant hardware setup to 
> use ? 
> > >> 
> > >> What should be considered ? Are there any pitfalls ? What about 
> > >> bottlenecks in the build system ? 
> 
> you should start by saying what you're going to build, my experience
is quite 
> different when building "small" images like console-image or even
x11-image 
> and "big" images/feeds which contain whole qt5 stack, 3 webkits and 
> 2 chromium builds. 

Agreed, what you are building and what your goals are makes a difference
in what you need. I have a build machine setup that is mainly used to
verify everything builds correctly after committing changes. It's an
Intel i3-3220 with 8GB RAM. The autobuilder is setup on a Linux VM which
is given 4GB RAM and does not recognize the extra Hyper-threaded cores,
meaning it acts as a dual core machine. Rebuild of my console image
typically takes under 20 minutes, and most of that time is
packaging/install. After the initial build, there really isn't much for
my system that needs to get rebuilt between commits. 

So if you are looking for a build machine that is outside your normal
workflow, a $400 PC may be enough for you. If this machine is for your
development build and you have a have a lot of graphic applications that
you need to build, you may want something more in line with what other
people are suggesting. 

Regards, 
Bryan 

> 
> In general: bitbake will better utilize all available performance with
bigger 
> image (e.g. build time for console image won't change so much if you
go from 
> 8 cores to 24, but building e.g. just webkit alone will be more than
twice 
> faster on 24 cores). 
> 
> Regards, 
> 
> > >> Specifically: 
> > >> 
> > >> How many cores are recommended ? And how much cache is necessary 
> ? 
> > >> How much of the main memory does Yocto really use ? Is 32 GB 
> > >> sufficient or should I go for 64 ? 
> > >> 
> > >> Does it make sense to use two SSDs as Raid0 to get builds faster
? 
> > > 
> > > As much of everything as you can afford. :) The build isn't heavy 
> > > in any particular metric, so don't sacrifice RAM for SSDs for
example. 
> > > 
> > > RAID 0 over SSD would be nice and fast, but I prefer having a good
> > > amount of RAM and a tuned ext4 (no journal, long commit delay) so 
> > > data doesn't actually hit the disk as frequently. Keeping the
actual 
> > > build directories on a separate disk is good for performance and
not 
> > > causing data loss when you lose a disk. 
> > > 
> > > There are people that have 64GB in machines and then set TMPDIR to
a 
> > > tmpfs. Surprisingly this isn't that much faster (5% or so), but 
> > > it's a lot easier on the hardware and power consumption. 
> > 
> > My experience: 
> > 
> > I've got a quad core with hyper-threading (so 8 usable cores)
running at 
> about 3.8 GHz, 16GB of RAM and use multiple SSDs - one to hold the
meta 
> data, downloads and top level build areas (local.conf, etc) and have
the 
> TMPDIR on a second SSD (so, as Ross says, I don't get a surprise when
it 
> wears out!). 
> > 
> > I can build my images (basically an x11 image) in just under 60
minutes 
> (once all the files have been fetched). I run with BB_NUMBER_THREADS
and 
> PARALLEL_MAKE both set to 16 to make sure the cores are fully loaded
as 
> much as possible (other says that should be 8 and 8 to reduce
scheduling 
> overhead). 
> > 
> > During the build the system is CPU bound quite a bit of the time (so
more 
> cores should help), but there are significant periods where the build 
> dependency chain means this isn't the case and only two or three cores
are 
> active. Previously I recall comparing results with someone else and
finding 
> that having lots more cores (24, I think) didn't give a significant
improvement 
> in build time (certainly not for the 3x system build cost). 
> > 
> > I've never seen peak memory usage go much above 9 GB during a build,
> and the peaks generally coincide with linking activities for "big"
items (gcc, 
> eglibc). This is likely to go higher with more active threads. 
> > 
> > I started out with a RAID-0 SSD build array, but I didn't really see
any 
> difference over a single high-spec (consumer) SSD. As Ross said,
running a 
> fast file system on the disk is a good idea. 
> > 
> > -- 
> > 
> > Chris Tapp 
> > opensource@keylevel.com 
> > www.keylevel.com 
> > 
> > ---- 
> > You can tell you're getting older when your car insurance gets real
cheap! 
> > 
> > -- 
> > _______________________________________________ 
> > yocto mailing list 
> > yocto@yoctoproject.org 
> > https://lists.yoctoproject.org/listinfo/yocto 
> 
> -- 
> Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com 


Alpine Electronics R&D Europe GmbH, Vor dem Lauch 14, 70567 Stuttgart,
Registergericht Stuttgart, HRB 25265
Geschaeftsfuehrer: Toshinori Kobayashi, Shuji Taguchi
Working office: Vor dem Lauch 14, 70567 Stuttgart

ATTENTION: This e-mail and any files transmitted with it may be privileged
and/or confidential, and are intended solely for the use of the individual
or entity to whom this  e-mail is addressed. If you are not one of the
named recipients or otherwise  have reason to believe that you have
received this e-mail in error, please notify  the sender and delete this
message immediately from your computer. We thank you for your cooperation.

[-- Attachment #2: Type: text/html, Size: 11385 bytes --]