[Buildroot] autoconf caching

[Buildroot] autoconf caching

[Dallas Clement]

Hi All - I have seen posts on this subject before.  It looks like things
flamed out.

http://lists.gnu.org/archive/html/autoconf/2012-11/msg00022.html

I'm getting killed by autoconf.  My builds take 1.5 hours even with ccache
turned on.  Most of the cores in my very impressive build machine are
sitting on their !#@ while autoconf is running.  Most of my packages rely
on autotools and it seems that most of the time spent is running configure
scripts.

Can anyone offer some advice on how to speed things up with autoconf?  Or
is it a lost cause?

Many thanks,

Dallas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140605/043255ef/attachment.html>

[Buildroot] autoconf caching

[Baruch Siach]

Hi Dallas,

On Thu, Jun 05, 2014 at 11:34:41AM -0500, Dallas Clement wrote:
> Hi All - I have seen posts on this subject before.  It looks like things
> flamed out.
> 
> http://lists.gnu.org/archive/html/autoconf/2012-11/msg00022.html
> 
> I'm getting killed by autoconf.  My builds take 1.5 hours even with ccache
> turned on.  Most of the cores in my very impressive build machine are
> sitting on their !#@ while autoconf is running.  Most of my packages rely
> on autotools and it seems that most of the time spent is running configure
> scripts.
> 
> Can anyone offer some advice on how to speed things up with autoconf?  Or
> is it a lost cause?

You may try to enable parallel build which is disabled by default. Look for 
'.NOTPARALLEL:' in the top level Makefile. Make sure to read the comment above 
it.

Hope this helps.

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] autoconf caching

[Yann E. MORIN]

Dallas, All,

On 2014-06-05 11:34 -0500, Dallas Clement spake thusly:
> Hi All - I have seen posts on this subject before.  It looks like things
> flamed out.
> 
> http://lists.gnu.org/archive/html/autoconf/2012-11/msg00022.html
> 
> I'm getting killed by autoconf.  My builds take 1.5 hours even with ccache
> turned on.  Most of the cores in my very impressive build machine are
> sitting on their !#@ while autoconf is running.  Most of my packages rely
> on autotools and it seems that most of the time spent is running configure
> scripts.
> 
> Can anyone offer some advice on how to speed things up with autoconf?  Or
> is it a lost cause?

Yes, it is a lost cause.

autoconf implements a way for the user to provide pre-canned answers to
tests, so it looks tempting to just provide such a set of answers,
gathered during the build, so that latter packages profit from the tests
done in previous packages.

We had something like that in the past, since we thought that would be
such a good idea.

Turned out it is not, after all.

The underlying problem is that the naming of variables is not consistent
across packages, so two packages may use two different variables for the
same test. While this is sub-optimal, it causes no much harm, it is just
not as fast as it could be.

But since the naming is inconsistent, it means that two packages may
also use the same variable for two different tests. This is causing much
more pain, since the results from a previous package would pollute the
tests for a latter package.

The result is that caching autoconf tests is a really bad idea, in fact.

As Baruch suggested, you may want to give a spin at top-level parallel
builds. This is turned off by default (see top-level Makefile, lines 46
to 60).

However, if you have a mostly-linear chain of dependency (aka a deep
dependency tree), you'll gain nothing, or close to nothing. If your
dependency tree is wide (as opposed to deep), then you may get a
noticeable speed up.

Also notice that top-level parallel builds are bad for reproducibility,
since you then no longer guarantee the ordering of non-dependent package
builds, and there might be hideen (unknown to Buildroot) dependencies
that would kick in one build but not in another, even though using the
same .config file.

So, besides getting a yet-faster machine, there's not much you can do to
speed up the build...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] autoconf caching

[Dallas Clement]

On Thu, Jun 5, 2014 at 12:34 PM, Yann E. MORIN <yann.morin.1998@free.fr>
wrote:

> Dallas, All,
>
> On 2014-06-05 11:34 -0500, Dallas Clement spake thusly:
> > Hi All - I have seen posts on this subject before.  It looks like things
> > flamed out.
> >
> > http://lists.gnu.org/archive/html/autoconf/2012-11/msg00022.html
> >
> > I'm getting killed by autoconf.  My builds take 1.5 hours even with
> ccache
> > turned on.  Most of the cores in my very impressive build machine are
> > sitting on their !#@ while autoconf is running.  Most of my packages rely
> > on autotools and it seems that most of the time spent is running
> configure
> > scripts.
> >
> > Can anyone offer some advice on how to speed things up with autoconf?  Or
> > is it a lost cause?
>
> Yes, it is a lost cause.
>
> autoconf implements a way for the user to provide pre-canned answers to
> tests, so it looks tempting to just provide such a set of answers,
> gathered during the build, so that latter packages profit from the tests
> done in previous packages.
>
> We had something like that in the past, since we thought that would be
> such a good idea.
>
> Turned out it is not, after all.
>
> The underlying problem is that the naming of variables is not consistent
> across packages, so two packages may use two different variables for the
> same test. While this is sub-optimal, it causes no much harm, it is just
> not as fast as it could be.
>
> But since the naming is inconsistent, it means that two packages may
> also use the same variable for two different tests. This is causing much
> more pain, since the results from a previous package would pollute the
> tests for a latter package.
>
> The result is that caching autoconf tests is a really bad idea, in fact.
>
> As Baruch suggested, you may want to give a spin at top-level parallel
> builds. This is turned off by default (see top-level Makefile, lines 46
> to 60).
>
> However, if you have a mostly-linear chain of dependency (aka a deep
> dependency tree), you'll gain nothing, or close to nothing. If your
> dependency tree is wide (as opposed to deep), then you may get a
> noticeable speed up.
>
> Also notice that top-level parallel builds are bad for reproducibility,
> since you then no longer guarantee the ordering of non-dependent package
> builds, and there might be hideen (unknown to Buildroot) dependencies
> that would kick in one build but not in another, even though using the
> same .config file.
>
> So, besides getting a yet-faster machine, there's not much you can do to
> speed up the build...
>
> Regards,
> Yann E. MORIN.
>
> --
>
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics'
> conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___
>       |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There
> is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v
> conspiracy.  |
>
> '------------------------------^-------^------------------^--------------------'
>

That's a real bummer, but thank you for the detailed explanation.  I'll
give parallel builds a shot in the meantime.  Maybe there is hope if the
autotools people can figure out a way to provide a unique namespace for
each package and variables.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140605/94b61aa5/attachment.html>

[Buildroot] autoconf caching

[Yann E. MORIN]

Dallas, All,

On 2014-06-05 12:39 -0500, Dallas Clement spake thusly:
> Maybe there is hope if the
> autotools people can figure out a way to provide a unique namespace for
> each package and variables.

That's not really the responsibility of the autotools folks.

Say I write a pacakge that uses ac_cv_has_gstreamer which for my pckage
means "gst-inspect is present".

Now you have no knowledge of my package, and you write your own package
that uses ac_cv_has_gstreamer which would mean "libgstreamer.so.0 is
present and usable".

There is no reconciliation possible betwen your package and mine. That's
not the fault of the autotools. Maybe we could say that it's the fault
of gstreamer to not provide the adequate macros, but even if that was
the case, that would not prevent two packages to use the same variable
to represent two different things (think optinal behaviour depending on
the environment for a package, while another would use it to represent
a user-selectable --enable/disable option).

There's no way out of this mess...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] autoconf caching

[Thomas Petazzoni]

Dear Dallas Clement,

On Thu, 5 Jun 2014 11:34:41 -0500, Dallas Clement wrote:

> I'm getting killed by autoconf.  My builds take 1.5 hours even with ccache
> turned on.  Most of the cores in my very impressive build machine are

Could you post the build-time.log file of your build, and the .config
file you're using? What are the specifications of your build machine?
Do you build inside a virtual machine ? Do you build over NFS ?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] autoconf caching

[Dallas Clement]

On Thu, Jun 5, 2014 at 3:42 PM, Thomas Petazzoni <
thomas.petazzoni@free-electrons.com> wrote:

> Dear Dallas Clement,
>
> On Thu, 5 Jun 2014 11:34:41 -0500, Dallas Clement wrote:
>
> > I'm getting killed by autoconf.  My builds take 1.5 hours even with
> ccache
> > turned on.  Most of the cores in my very impressive build machine are
>
> Could you post the build-time.log file of your build, and the .config
> file you're using? What are the specifications of your build machine?
> Do you build inside a virtual machine ? Do you build over NFS ?
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
>

You bet.  Please see attached.  I'm in the process of fully upgrading over
to latest buildroot / master from 2012.08 so there is probably still some
deprecated stuff in my .config.

My build machine is a HP server with 8 core Intel(R) Xeon(R) CPU E5-2637 v2
@ 3.50GHz.  The OS is RHEL 6.4 64 bit.  No virtual machine and no NFS.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140605/84cb13bb/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config
Type: application/octet-stream
Size: 47018 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140605/84cb13bb/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build-time.log
Type: application/octet-stream
Size: 119578 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140605/84cb13bb/attachment-0003.obj>

[Buildroot] autoconf caching

[Waldemar Brodkorb]

Hi,
Dallas Clement wrote,

> Hi All - I have seen posts on this subject before.? It looks like things flamed
> out.
> 
> http://lists.gnu.org/archive/html/autoconf/2012-11/msg00022.html
> 
> I'm getting killed by autoconf.? My builds take 1.5 hours even with ccache
> turned on.? Most of the cores in my very impressive build machine are sitting
> on their !#@ while autoconf is running.? Most of my packages rely on autotools
> and it seems that most of the time spent is running configure scripts.
> 
> Can anyone offer some advice on how to speed things up with autoconf?? Or is it
> a lost cause?
> 
> Many thanks,

There was an interesting discussion about performance on the autoconf
mailinglist:
http://lists.gnu.org/archive/html/autoconf/2014-03/msg00030.html

OpenWrt uses CONFIG_SITE with some global architecture and libc
dependent files, see here:
https://dev.openwrt.org/browser/trunk/include/site

OpenEmbedded uses CONFIG_SITE, too:
http://cgit.openembedded.org/openembedded-core/tree/meta/site
They have a lot of package specific values added to their files.

May be the OpenWrt way with just a few global cached values would be
a benefit for buildroot performance? 

best regards
        Waldemar

[Buildroot] autoconf caching

[Thomas Petazzoni]

Dear Dallas Clement,

On Thu, 5 Jun 2014 17:06:48 -0500, Dallas Clement wrote:

> You bet.  Please see attached.  I'm in the process of fully upgrading over
> to latest buildroot / master from 2012.08 so there is probably still some
> deprecated stuff in my .config.

I didn't spot anything really wrong in there. host-gdb looks a bit
useless since most likely your external toolchain already has a
cross-debugger integrated. Other than that, it appears that:

 * Your configuration has a fairly large number of packages, some of
   them being fairly big stuff (mysql, perl, etc.)

 * The build time of perl is really large. Maybe there's something that
   could be done to improve this. Looking at perl.mk in Buildroot, I
   see that it uses $(MAKE1), so the build of this package is not
   parallelized.

 * The installation time of host-mysql seems quite large as well, I
   don't really see why, especially since the mysql host install
   commands copy just one file in Buildroot.

Really, other than these little details, I believe you're simply
affected by the size of your configuration in terms of the number of
packages.

> My build machine is a HP server with 8 core Intel(R) Xeon(R) CPU E5-2637 v2
> @ 3.50GHz.  The OS is RHEL 6.4 64 bit.  No virtual machine and no NFS.

Seems a fairly decent setup. You have enough RAM, and a good storage
system (ideally based on SSDs) ?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] autoconf caching

[Thomas Petazzoni]

Dear Dallas Clement,

On Thu, 5 Jun 2014 17:06:48 -0500, Dallas Clement wrote:

> You bet.  Please see attached.  I'm in the process of fully upgrading over
> to latest buildroot / master from 2012.08 so there is probably still some
> deprecated stuff in my .config.

There is another thing that strikes out in your configuration: there
are many packages that we don't have in Buildroot, and that don't seem
to be specific to your company or project. What about submitting your
new packages upstream?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] autoconf caching

[Dallas Clement]

On Fri, Jun 6, 2014 at 2:15 AM, Thomas Petazzoni <
thomas.petazzoni@free-electrons.com> wrote:

> Dear Dallas Clement,
>
> On Thu, 5 Jun 2014 17:06:48 -0500, Dallas Clement wrote:
>
> > You bet.  Please see attached.  I'm in the process of fully upgrading
> over
> > to latest buildroot / master from 2012.08 so there is probably still some
> > deprecated stuff in my .config.
>
> There is another thing that strikes out in your configuration: there
> are many packages that we don't have in Buildroot, and that don't seem
> to be specific to your company or project. What about submitting your
> new packages upstream?
>
> Thanks!
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
>

Hi Thomas.  Thanks very much for looking at my configuration.  I think you
certainly nailed it.
I have a lot of packages, and perl and mysql especially are taking a long
time.  My build recipes
for both of these are a year or two out of date.  In fact, my perl recipe
is still using qemu to build
instead of perl cross.  Hopefully updating this recipe will help some,
though as you pointed out
it is still using MAKE1.

> Seems a fairly decent setup. You have enough RAM, and a good storage
> system (ideally based on SSDs) ?

Yep, very speedy SSDs and 64 GB for RAM.  Because of all the serialization
in this build process,
building from a large ram disk seems to make little difference.

> There is another thing that strikes out in your configuration: there
> are many packages that we don't have in Buildroot, and that don't seem
> to be specific to your company or project. What about submitting your
> new packages upstream?

Yes, I can certainly do that.  There's a lot here.  Much of it is obscure.
Please let me know which ones you are most
interested in.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140606/8122b93f/attachment.html>

[Buildroot] autoconf caching

[Mike Zick]

On Fri, 6 Jun 2014 11:14:15 -0500
Dallas Clement <dallas.a.clement@gmail.com> wrote:

> > Seems a fairly decent setup. You have enough RAM, and a good storage
> > system (ideally based on SSDs) ?  
> 
> Yep, very speedy SSDs and 64 GB for RAM.  Because of all the
> serialization in this build process,
> building from a large ram disk seems to make little difference.
>

Having 7 of your 8 cores sitting in their idle loop is probably your
#1 thing to address. . . .

With a very large selection of packages, perhaps you could split them
into several (8, 9?) Buildroot trees.
Then run those buildroot instances as jobs at the same time.

Or...

At least pull the really big time consumers out, into their own BR
instance, and use your idle cores on those.

Mike

[Buildroot] autoconf caching

[Dallas Clement]

On Fri, Jun 6, 2014 at 11:31 AM, Mike Zick <minimod@morethan.org> wrote:

> On Fri, 6 Jun 2014 11:14:15 -0500
> Dallas Clement <dallas.a.clement@gmail.com> wrote:
>
> > > Seems a fairly decent setup. You have enough RAM, and a good storage
> > > system (ideally based on SSDs) ?
> >
> > Yep, very speedy SSDs and 64 GB for RAM.  Because of all the
> > serialization in this build process,
> > building from a large ram disk seems to make little difference.
> >
>
> Having 7 of your 8 cores sitting in their idle loop is probably your
> #1 thing to address. . . .
>
> With a very large selection of packages, perhaps you could split them
> into several (8, 9?) Buildroot trees.
> Then run those buildroot instances as jobs at the same time.
>
> Or...
>
> At least pull the really big time consumers out, into their own BR
> instance, and use your idle cores on those.
>
> Mike
>
>
> Having 7 of your 8 cores sitting in their idle loop is probably your
> #1 thing to address. . . .

Yes, makes me sick to have them just sitting there...

> With a very large selection of packages, perhaps you could split them
> into several (8, 9?) Buildroot trees.

This is a good idea.  I think I've exhausted most of the other options
except for
commenting out .NOTPARALLEL in the top-level make.  If that doesn't help,
I will definitely try breaking things up.  Thanks for the tip.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140606/e3678a68/attachment.html>

[Buildroot] autoconf caching

[Thomas Petazzoni]

Dear Dallas Clement,

On Fri, 6 Jun 2014 11:14:15 -0500, Dallas Clement wrote:

> Hi Thomas.  Thanks very much for looking at my configuration.  I think you
> certainly nailed it.
> I have a lot of packages, and perl and mysql especially are taking a long
> time.  My build recipes
> for both of these are a year or two out of date.  In fact, my perl recipe
> is still using qemu to build
> instead of perl cross.  Hopefully updating this recipe will help some,
> though as you pointed out
> it is still using MAKE1.

Right. I'll send a separate e-mail to Fran?ois Perrad about this (our
Perl guy), because it seems like other build systems are building Perl
in parallel.

> Yep, very speedy SSDs and 64 GB for RAM.  Because of all the serialization
> in this build process,
> building from a large ram disk seems to make little difference.

Right. So you have a good build configuration, there's not much we can
do here.

> > There is another thing that strikes out in your configuration: there
> > are many packages that we don't have in Buildroot, and that don't seem
> > to be specific to your company or project. What about submitting your
> > new packages upstream?
> 
> Yes, I can certainly do that.  There's a lot here.  Much of it is obscure.
> Please let me know which ones you are most
> interested in.

Could you make a list of the packages that you've added, and that are
open-source? I'm pretty sure a majority of them could potentially be
useful to have in Buildroot.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [UNSURE]Re: autoconf caching

[François Perrad]

2014-06-07 11:04 GMT+02:00 Thomas Petazzoni
<thomas.petazzoni@free-electrons.com>:
> Dear Dallas Clement,
>
> On Fri, 6 Jun 2014 11:14:15 -0500, Dallas Clement wrote:
>
>> Hi Thomas.  Thanks very much for looking at my configuration.  I think you
>> certainly nailed it.
>> I have a lot of packages, and perl and mysql especially are taking a long
>> time.  My build recipes
>> for both of these are a year or two out of date.  In fact, my perl recipe
>> is still using qemu to build
>> instead of perl cross.  Hopefully updating this recipe will help some,
>> though as you pointed out
>> it is still using MAKE1.
>
> Right. I'll send a separate e-mail to Fran?ois Perrad about this (our
> Perl guy), because it seems like other build systems are building Perl
> in parallel.
>

The build of perl with qemu takes very very long time.
The build of perl with perl-cross takes the same time as Linux Kernel
for example.
Parallel build with perl-cross seems not supported (this needs a
confirmation of Alex Suykov, the author of perl-cross).

Fran?ois

>> Yep, very speedy SSDs and 64 GB for RAM.  Because of all the serialization
>> in this build process,
>> building from a large ram disk seems to make little difference.
>
> Right. So you have a good build configuration, there's not much we can
> do here.
>
>> > There is another thing that strikes out in your configuration: there
>> > are many packages that we don't have in Buildroot, and that don't seem
>> > to be specific to your company or project. What about submitting your
>> > new packages upstream?
>>
>> Yes, I can certainly do that.  There's a lot here.  Much of it is obscure.
>> Please let me know which ones you are most
>> interested in.
>
> Could you make a list of the packages that you've added, and that are
> open-source? I'm pretty sure a majority of them could potentially be
> useful to have in Buildroot.
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [UNSURE]Re: autoconf caching

[Dallas Clement]

On Sun, Jun 8, 2014 at 3:19 AM, Fran?ois Perrad <francois.perrad@gadz.org>
wrote:

> 2014-06-07 11:04 GMT+02:00 Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com>:
> > Dear Dallas Clement,
> >
> > On Fri, 6 Jun 2014 11:14:15 -0500, Dallas Clement wrote:
> >
> >> Hi Thomas.  Thanks very much for looking at my configuration.  I think
> you
> >> certainly nailed it.
> >> I have a lot of packages, and perl and mysql especially are taking a
> long
> >> time.  My build recipes
> >> for both of these are a year or two out of date.  In fact, my perl
> recipe
> >> is still using qemu to build
> >> instead of perl cross.  Hopefully updating this recipe will help some,
> >> though as you pointed out
> >> it is still using MAKE1.
> >
> > Right. I'll send a separate e-mail to Fran?ois Perrad about this (our
> > Perl guy), because it seems like other build systems are building Perl
> > in parallel.
> >
>
> The build of perl with qemu takes very very long time.
> The build of perl with perl-cross takes the same time as Linux Kernel
> for example.
> Parallel build with perl-cross seems not supported (this needs a
> confirmation of Alex Suykov, the author of perl-cross).
>
> Fran?ois
>
> >> Yep, very speedy SSDs and 64 GB for RAM.  Because of all the
> serialization
> >> in this build process,
> >> building from a large ram disk seems to make little difference.
> >
> > Right. So you have a good build configuration, there's not much we can
> > do here.
> >
> >> > There is another thing that strikes out in your configuration: there
> >> > are many packages that we don't have in Buildroot, and that don't seem
> >> > to be specific to your company or project. What about submitting your
> >> > new packages upstream?
> >>
> >> Yes, I can certainly do that.  There's a lot here.  Much of it is
> obscure.
> >> Please let me know which ones you are most
> >> interested in.
> >
> > Could you make a list of the packages that you've added, and that are
> > open-source? I'm pretty sure a majority of them could potentially be
> > useful to have in Buildroot.
> >
> > Best regards,
> >
> > Thomas
> > --
> > Thomas Petazzoni, CTO, Free Electrons
> > Embedded Linux, Kernel and Android engineering
> > http://free-electrons.com
> > _______________________________________________
> > buildroot mailing list
> > buildroot at busybox.net
> > http://lists.busybox.net/mailman/listinfo/buildroot
>

The build-time histograms in buildroot now are so cool!  The three biggest
hogs in my rootfs are perl, icu, and xerces, all of which are MAKE1, and
the lion's share of this time is compilation.  It would sure help me a lot
if we could somehow get these guys to parallel make.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140609/6d7270df/attachment.html>

[Buildroot] [UNSURE]Re: autoconf caching

[Thomas Petazzoni]

Dear Fran?ois Perrad,

On Sun, 8 Jun 2014 10:19:13 +0200, Fran?ois Perrad wrote:

> The build of perl with qemu takes very very long time.
> The build of perl with perl-cross takes the same time as Linux Kernel
> for example.
> Parallel build with perl-cross seems not supported (this needs a
> confirmation of Alex Suykov, the author of perl-cross).

Would it be possible to investigate how Yocto does the parallel
cross-compilation build of Perl ?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] autoconf caching

[Fabio Porcedda]

On Fri, Jun 6, 2014 at 6:45 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> On Fri, Jun 6, 2014 at 11:31 AM, Mike Zick <minimod@morethan.org> wrote:
>>
>> On Fri, 6 Jun 2014 11:14:15 -0500
>> Dallas Clement <dallas.a.clement@gmail.com> wrote:
>>
>> > > Seems a fairly decent setup. You have enough RAM, and a good storage
>> > > system (ideally based on SSDs) ?
>> >
>> > Yep, very speedy SSDs and 64 GB for RAM.  Because of all the
>> > serialization in this build process,
>> > building from a large ram disk seems to make little difference.
>> >
>>
>> Having 7 of your 8 cores sitting in their idle loop is probably your
>> #1 thing to address. . . .
>>
>> With a very large selection of packages, perhaps you could split them
>> into several (8, 9?) Buildroot trees.
>> Then run those buildroot instances as jobs at the same time.
>>
>> Or...
>>
>> At least pull the really big time consumers out, into their own BR
>> instance, and use your idle cores on those.
>>
>> Mike
>>
>
>> Having 7 of your 8 cores sitting in their idle loop is probably your
>> #1 thing to address. . . .
>
> Yes, makes me sick to have them just sitting there...
>
>> With a very large selection of packages, perhaps you could split them
>> into several (8, 9?) Buildroot trees.
>
> This is a good idea.  I think I've exhausted most of the other options
> except for
> commenting out .NOTPARALLEL in the top-level make.  If that doesn't help,
> I will definitely try breaking things up.  Thanks for the tip.

Please let me know if you have any issue using top-level parallel make.
I'm curently testing it and ironing it out.

Best regards
-- 
Fabio Porcedda

[Buildroot] autoconf caching

[Dallas Clement]

On Tue, Jun 10, 2014 at 4:32 AM, Fabio Porcedda <fabio.porcedda@gmail.com>
wrote:

> On Fri, Jun 6, 2014 at 6:45 PM, Dallas Clement
> <dallas.a.clement@gmail.com> wrote:
> > On Fri, Jun 6, 2014 at 11:31 AM, Mike Zick <minimod@morethan.org> wrote:
> >>
> >> On Fri, 6 Jun 2014 11:14:15 -0500
> >> Dallas Clement <dallas.a.clement@gmail.com> wrote:
> >>
> >> > > Seems a fairly decent setup. You have enough RAM, and a good storage
> >> > > system (ideally based on SSDs) ?
> >> >
> >> > Yep, very speedy SSDs and 64 GB for RAM.  Because of all the
> >> > serialization in this build process,
> >> > building from a large ram disk seems to make little difference.
> >> >
> >>
> >> Having 7 of your 8 cores sitting in their idle loop is probably your
> >> #1 thing to address. . . .
> >>
> >> With a very large selection of packages, perhaps you could split them
> >> into several (8, 9?) Buildroot trees.
> >> Then run those buildroot instances as jobs at the same time.
> >>
> >> Or...
> >>
> >> At least pull the really big time consumers out, into their own BR
> >> instance, and use your idle cores on those.
> >>
> >> Mike
> >>
> >
> >> Having 7 of your 8 cores sitting in their idle loop is probably your
> >> #1 thing to address. . . .
> >
> > Yes, makes me sick to have them just sitting there...
> >
> >> With a very large selection of packages, perhaps you could split them
> >> into several (8, 9?) Buildroot trees.
> >
> > This is a good idea.  I think I've exhausted most of the other options
> > except for
> > commenting out .NOTPARALLEL in the top-level make.  If that doesn't help,
> > I will definitely try breaking things up.  Thanks for the tip.
>
> Please let me know if you have any issue using top-level parallel make.
> I'm curently testing it and ironing it out.
>
> Best regards
> --
> Fabio Porcedda
>

> Please let me know if you have any issue using top-level parallel make.
> I'm curently testing it and ironing it out.

So far, no serious issues.  It uncovered some hidden dependencies.  But I
was warned about this.  ;)

Surprisingly, it did not speed up my build times at all.  Perhaps this is
due
to a complex dependency graph.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140611/e29b9ecb/attachment.html>

[Buildroot] autoconf caching

[Fabio Porcedda]

On Wed, Jun 11, 2014 at 7:41 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> On Tue, Jun 10, 2014 at 4:32 AM, Fabio Porcedda <fabio.porcedda@gmail.com>
> wrote:
>>
>> On Fri, Jun 6, 2014 at 6:45 PM, Dallas Clement
>> <dallas.a.clement@gmail.com> wrote:
>> > On Fri, Jun 6, 2014 at 11:31 AM, Mike Zick <minimod@morethan.org> wrote:
<snip>
>> > This is a good idea.  I think I've exhausted most of the other options
>> > except for
>> > commenting out .NOTPARALLEL in the top-level make.  If that doesn't
>> > help,
>> > I will definitely try breaking things up.  Thanks for the tip.
>>
>> Please let me know if you have any issue using top-level parallel make.
>> I'm curently testing it and ironing it out.
>>
>> Best regards
>> --
>> Fabio Porcedda
>
>
>> Please let me know if you have any issue using top-level parallel make.
>> I'm curently testing it and ironing it out.
>
> So far, no serious issues.  It uncovered some hidden dependencies.  But I
> was warned about this.  ;)
>
> Surprisingly, it did not speed up my build times at all.  Perhaps this is
> due
> to a complex dependency graph.

Can you please send the graphs made by "make graph-depends" and
"make graph-build" for both serial and parallel and the output of the
time commands, i mean just the time command output not the complete
log:
"time make" and "time make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))"?

It's best if you don't attach those files to the email but use some
other way, e.g. dropbox.

Best regards
-- 
Fabio Porcedda

[Buildroot] autoconf caching

[Dallas Clement]

On Fri, Jun 13, 2014 at 4:23 AM, Fabio Porcedda <fabio.porcedda@gmail.com>
wrote:

> On Wed, Jun 11, 2014 at 7:41 PM, Dallas Clement
> <dallas.a.clement@gmail.com> wrote:
> > On Tue, Jun 10, 2014 at 4:32 AM, Fabio Porcedda <
> fabio.porcedda at gmail.com>
> > wrote:
> >>
> >> On Fri, Jun 6, 2014 at 6:45 PM, Dallas Clement
> >> <dallas.a.clement@gmail.com> wrote:
> >> > On Fri, Jun 6, 2014 at 11:31 AM, Mike Zick <minimod@morethan.org>
> wrote:
> <snip>
> >> > This is a good idea.  I think I've exhausted most of the other options
> >> > except for
> >> > commenting out .NOTPARALLEL in the top-level make.  If that doesn't
> >> > help,
> >> > I will definitely try breaking things up.  Thanks for the tip.
> >>
> >> Please let me know if you have any issue using top-level parallel make.
> >> I'm curently testing it and ironing it out.
> >>
> >> Best regards
> >> --
> >> Fabio Porcedda
> >
> >
> >> Please let me know if you have any issue using top-level parallel make.
> >> I'm curently testing it and ironing it out.
> >
> > So far, no serious issues.  It uncovered some hidden dependencies.  But I
> > was warned about this.  ;)
> >
> > Surprisingly, it did not speed up my build times at all.  Perhaps this is
> > due
> > to a complex dependency graph.
>
> Can you please send the graphs made by "make graph-depends" and
> "make graph-build" for both serial and parallel and the output of the
> time commands, i mean just the time command output not the complete
> log:
> "time make" and "time make BR2_JLEVEL= -j$((`getconf
> _NPROCESSORS_ONLN`+1))"?
>
> It's best if you don't attach those files to the email but use some
> other way, e.g. dropbox.
>
> Best regards
> --
> Fabio Porcedda
>

Fabio, I have some good news!  I had commented out the .NOTPARALLEL in the
top-level
makefile earlier, but I was not setting BR2_JLEVEL correctly, as I was
doing so from
a wrapper makefile.  Anyhow, parallel building is working for me now after
fixing my mistake.
When I started this journey, my build times were about 1 hour and 50
minutes.  After
enabling ccache they were reduced to about 1 hour.  With parallel package
building,
all 8 of my CPU cores are near 100% the whole time, and build times are now
only 14 minutes!!!

You are a beautiful man!  Thank you for all you have done to get this
working.

p.s.  If you still want me to collect the graphs, and time comparison for
parallel vs
not parallel, please let me know.  I also have more testing to do to verify
that there
are no unexpected run-time errors due to parallel building.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140613/12170b91/attachment.html>

[Buildroot] autoconf caching

[Thomas Petazzoni]

Dear Dallas Clement,

(Seems like your e-mail client has major issues with line wrapping).

On Fri, 13 Jun 2014 19:52:15 -0500, Dallas Clement wrote:

> Fabio, I have some good news!  I had commented out the .NOTPARALLEL
> in the top-level
> makefile earlier, but I was not setting BR2_JLEVEL correctly, as I was
> doing so from
> a wrapper makefile.  Anyhow, parallel building is working for me now
> after fixing my mistake.
> When I started this journey, my build times were about 1 hour and 50
> minutes.  After
> enabling ccache they were reduced to about 1 hour.  With parallel
> package building,
> all 8 of my CPU cores are near 100% the whole time, and build times
> are now only 14 minutes!!!
> 
> You are a beautiful man!  Thank you for all you have done to get this
> working.

That's indeed a very significant improvement. However, be careful that
using top-level parallel builds makes builds non-reproducible: we do
not guarantee that from one build to the other you'll get the same
result. You might even get failures sometimes, and then success. Or the
other way around. That's why we don't allow top-level parallel build by
default.

Unfortunately, the only way to fix this is to have per-package sysroot,
and doing this will involve a lot more preparation work before starting
the build of each package, which will certainly reduce the benefit of
top-level parallel build. That's something that needs to be worked on,
but it's far from being a trivial problem.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] autoconf caching

[Fabio Porcedda]

On Sat, Jun 14, 2014 at 2:52 AM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
<snip>
> Fabio, I have some good news!  I had commented out the .NOTPARALLEL in the
> top-level
> makefile earlier, but I was not setting BR2_JLEVEL correctly, as I was doing
> so from
> a wrapper makefile.  Anyhow, parallel building is working for me now after
> fixing my mistake.
> When I started this journey, my build times were about 1 hour and 50
> minutes.  After
> enabling ccache they were reduced to about 1 hour.  With parallel package
> building,
> all 8 of my CPU cores are near 100% the whole time, and build times are now
> only 14 minutes!!!

That's great!

> You are a beautiful man!  Thank you for all you have done to get this
> working.

Thank you very much for your kind words.
I'm glad to hear that you like it.

> p.s.  If you still want me to collect the graphs, and time comparison for
> parallel vs
> not parallel, please let me know.  I also have more testing to do to verify
> that there
> are no unexpected run-time errors due to parallel building.

I've asked for for those graphs just to use correctly the parallel
make so I don't need it anymore.

If you encounter build problems related to the reordering of targets,
there is a way to replicate almost the same ordering (excluding
interleaving targets).
Given a "build-time.log" you can generate the list of targets:

grep :start: < output/build/build-time.log  | perl -F: -ane '$s =
"$F[3]"; $s =~ s/^\s+|\s+$//g; print "$s-$F[2]\n"' > targets.log

Now you can use that list to build following the same order:
xargs make < targets.log

This is useful to be able to reproduce the problems and debug it.

I will send a patch to add that script or to document it, I'm not sure
what is the best thing to do.

Best regards
-- 
Fabio Porcedda