[PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Richard Purdie]

Its rather sad that people don't appear to read local.conf and then complain
about slow builds when they're just using a single thread. Most systems have
more than one core now so we might as well use a more automatic default
for these values. This may lead to better experiences for new users.

[YOCTO #2528]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
index 71856b8..36d33e1 100644
--- a/meta/conf/local.conf.sample
+++ b/meta/conf/local.conf.sample
@@ -18,12 +18,18 @@
 # option determines how many tasks bitbake should run in parallel:
 #
 #BB_NUMBER_THREADS ?= "4"
+#
+# Default to setting automatically based on cpu count
+BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
 # 
 # The second option controls how many processes make should run in parallel when
 # running compile tasks:
 #
 #PARALLEL_MAKE ?= "-j 4"
 #
+# Default to setting automatically based on cpu count
+PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
+#
 # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
 # be appropriate for example.
 

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Otavio Salvador]

On Mon, Jan 27, 2014 at 12:39 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Its rather sad that people don't appear to read local.conf and then complain
> about slow builds when they're just using a single thread. Most systems have
> more than one core now so we might as well use a more automatic default
> for these values. This may lead to better experiences for new users.
>
> [YOCTO #2528]
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Acked-by: Otavio Salvador <otavio@ossystems.com.br>

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Stewart, David C]

On 1/27/14, 8:45 AM, "Otavio Salvador" <otavio@ossystems.com.br> wrote:

>On Mon, Jan 27, 2014 at 12:39 PM, Richard Purdie
><richard.purdie@linuxfoundation.org> wrote:
>> Its rather sad that people don't appear to read local.conf and then
>>complain
>> about slow builds when they're just using a single thread. Most systems
>>have
>> more than one core now so we might as well use a more automatic default
>> for these values. This may lead to better experiences for new users.
>>
>> [YOCTO #2528]
>>
>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Nice! Now if we could just convince people to configure their VM guests
with enough cpus - 

:-)

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Burton, Ross]

On 27 January 2014 17:23, Stewart, David C <david.c.stewart@intel.com> wrote:
> Nice! Now if we could just convince people to configure their VM guests
> with enough cpus -

Now if only we could make bitbake error out when it's running inside a VM. ;)

Ross

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Koen Kooi]

Op 27 jan. 2014, om 15:39 heeft Richard Purdie <richard.purdie@linuxfoundation.org> het volgende geschreven:

> Its rather sad that people don't appear to read local.conf and then complain
> about slow builds when they're just using a single thread. Most systems have
> more than one core now so we might as well use a more automatic default
> for these values. This may lead to better experiences for new users.
> 
> [YOCTO #2528]
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
> index 71856b8..36d33e1 100644
> --- a/meta/conf/local.conf.sample
> +++ b/meta/conf/local.conf.sample
> @@ -18,12 +18,18 @@
> # option determines how many tasks bitbake should run in parallel:
> #
> #BB_NUMBER_THREADS ?= "4"
> +#
> +# Default to setting automatically based on cpu count
> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"

I've noticed that after 4 threads IO becomes a big bottleneck when you have things like webkit, qt, asio etc in the buildqueue. Combine that with issues like every make -j thread taking >2GB ram with asio and webkit this default seems a bit high. I'd use 0.5*numcpu with a lower bound of 2.

regards,

Koen

> # 
> # The second option controls how many processes make should run in parallel when
> # running compile tasks:
> #
> #PARALLEL_MAKE ?= "-j 4"
> #
> +# Default to setting automatically based on cpu count
> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
> +#
> # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
> # be appropriate for example.
> 
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> 

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Richard Purdie]

On Tue, 2014-01-28 at 11:08 +0100, Koen Kooi wrote:
> Op 27 jan. 2014, om 15:39 heeft Richard Purdie <richard.purdie@linuxfoundation.org> het volgende geschreven:
> 
> > Its rather sad that people don't appear to read local.conf and then complain
> > about slow builds when they're just using a single thread. Most systems have
> > more than one core now so we might as well use a more automatic default
> > for these values. This may lead to better experiences for new users.
> > 
> > [YOCTO #2528]
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> > diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
> > index 71856b8..36d33e1 100644
> > --- a/meta/conf/local.conf.sample
> > +++ b/meta/conf/local.conf.sample
> > @@ -18,12 +18,18 @@
> > # option determines how many tasks bitbake should run in parallel:
> > #
> > #BB_NUMBER_THREADS ?= "4"
> > +#
> > +# Default to setting automatically based on cpu count
> > +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
> 
> I've noticed that after 4 threads IO becomes a big bottleneck when you
> have things like webkit, qt, asio etc in the buildqueue. Combine that
> with issues like every make -j thread taking >2GB ram with asio and
> webkit this default seems a bit high. I'd use 0.5*numcpu with a lower
> bound of 2.

This is one of those things I think we'll not find a perfect answer to
and its why I left the defaults alone for as long.

Personally, I run with 48 threads enabled and I've yet to see it do
anything too crazy. At the end of the day it all comes down to the
hardware you're running it on though.

webkit is a particular case where whilst its building you'd probably
want different values. Which numbers you use depends on how often you're
building it...

I'm not sure we can win. If the build appears to take over a system, I'm
hoping the user will look at the conf file and figure out how to make it
use less of the system...

Cheers,

Richard

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Steffen Sledz]

On 27.01.2014 15:39, Richard Purdie wrote:
> Its rather sad that people don't appear to read local.conf and then complain
> about slow builds when they're just using a single thread. Most systems have
> more than one core now so we might as well use a more automatic default
> for these values. This may lead to better experiences for new users.
> 
> [YOCTO #2528]
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
> index 71856b8..36d33e1 100644
> --- a/meta/conf/local.conf.sample
> +++ b/meta/conf/local.conf.sample
> @@ -18,12 +18,18 @@
>  # option determines how many tasks bitbake should run in parallel:
>  #
>  #BB_NUMBER_THREADS ?= "4"
> +#
> +# Default to setting automatically based on cpu count
> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
>  # 
>  # The second option controls how many processes make should run in parallel when
>  # running compile tasks:
>  #
>  #PARALLEL_MAKE ?= "-j 4"
>  #
> +# Default to setting automatically based on cpu count
> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
> +#
>  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
>  # be appropriate for example.

On our Fedora-18 build host this change leads to the following exception. :(

----------------------> snip <-------------------------
Loading cache...done.
Loaded 2167 entries from dependency cache.

Parsing recipes...done.
Parsing of 1729 .bb files complete (1727 cached, 2 parsed). 2168 targets, 85 skipped, 1 masked, 0 errors.

NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION        = "1.20.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Fedora-18"
TARGET_SYS        = "arm-angstrom-linux-gnueabi"
MACHINE           = "hikirk"
DISTRO            = "angstrom"
DISTRO_VERSION    = "v2013.12"
TUNE_FEATURES     = "armv5 thumb dsp"
TARGET_FPU        = "soft"
meta-hipos        = "(nobranch):6f2ad6754c9c9ddd6f1c5505e584e0eb62cfcf83"
meta-hipos-kirkwood = "(nobranch):1a43f75e217fea3c6b4493536b3b66945ce53921"
meta-hipos-fsl    = "(nobranch):ad3beec4157a96d0c506969a29d14f6354e7406f"
meta-fsl-arm      = "(nobranch):af392c22bf6b563525ede4a81b6755ff1dd2c1c6"
meta-fsl-arm-extra = "(nobranch):07ad83db0fb67c5023bd627a61efb7f474c52622"
meta-java         = "(nobranch):9dfb7ecb87e67052e94641d6e02907c582264217"
meta-angstrom     = "(nobranch):8fcd781b777a7a94f2b97514c9691a2b5c74eaf2"
meta-oe
meta-systemd
meta-multimedia
meta-networking
meta-efl
meta-gnome        = "(nobranch):eb4563b83be0a57ede4269ab19688af6baa62cd2"
meta              = "(nobranch):e86622a932bbd0acdea67ecfb15c8b06c27353d8"

NOTE: Preparing runqueue
ERROR: An uncaught exception occured in runqueue, please see the failure below:
ERROR: Running idle function
Traceback (most recent call last):
  File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/runqueue.py", line 1111, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue instance at 0x1534b2d8>):

    >        self.number_tasks = int(self.cfgData.getVar("BB_NUMBER_THREADS", True) or 1)
             self.scheduler = self.cfgData.getVar("BB_SCHEDULER", True) or "speed"
  File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 516, in DataSmart.getVar(var='BB_NUMBER_THREADS', expand=True, noweakdefault=False):
         def getVar(self, var, expand=False, noweakdefault=False):
    >        return self.getVarFlag(var, "_content", expand, noweakdefault)

  File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 606, in DataSmart.getVarFlag(var='BB_NUMBER_THREADS', flag='_content', expand=True, noweakdefault=False):
                     cachename = var + "[" + flag + "]"
    >            value = self.expand(value, cachename)
             if value is not None and flag == "_content" and local_var is not None and "_removeactive" in local_var:
  File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 344, in DataSmart.expand(s='${@oe.utils.cpu_count()}', varname='BB_NUMBER_THREADS'):
         def expand(self, s, varname = None):
    >        return self.expandWithRefs(s, varname).value

  File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 334, in DataSmart.expandWithRefs(s='${@oe.utils.cpu_count()}', varname='BB_NUMBER_THREADS'):
                 except Exception as exc:
    >                raise ExpansionError(varname, s, exc)

ExpansionError: Failure expanding variable BB_NUMBER_THREADS, expression was ${@oe.utils.cpu_count()} which triggered exception AttributeError: 'module' object has no attribute 'cpu_count'
----------------------> snip <-------------------------

Any ideas? A missing python package at the build host? ...?

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Paul Eggleton]

On Wednesday 29 January 2014 11:14:47 Steffen Sledz wrote:
> On 27.01.2014 15:39, Richard Purdie wrote:
> > Its rather sad that people don't appear to read local.conf and then
> > complain about slow builds when they're just using a single thread. Most
> > systems have more than one core now so we might as well use a more
> > automatic default for these values. This may lead to better experiences
> > for new users.
> > 
> > [YOCTO #2528]
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> > diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
> > index 71856b8..36d33e1 100644
> > --- a/meta/conf/local.conf.sample
> > +++ b/meta/conf/local.conf.sample
> > @@ -18,12 +18,18 @@
> > 
> >  # option determines how many tasks bitbake should run in parallel:
> >  #
> >  #BB_NUMBER_THREADS ?= "4"
> > 
> > +#
> > +# Default to setting automatically based on cpu count
> > +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
> > 
> >  #
> >  # The second option controls how many processes make should run in
> >  parallel when # running compile tasks:
> >  #
> >  #PARALLEL_MAKE ?= "-j 4"
> >  #
> > 
> > +# Default to setting automatically based on cpu count
> > +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
> > +#
> > 
> >  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j
> >  4" would # be appropriate for example.
> 
> On our Fedora-18 build host this change leads to the following exception. :(
> 
> ----------------------> snip <-------------------------
> Loading cache...done.
> Loaded 2167 entries from dependency cache.
> 
> Parsing recipes...done.
> Parsing of 1729 .bb files complete (1727 cached, 2 parsed). 2168 targets, 85
> skipped, 1 masked, 0 errors.
> 
> NOTE: Resolving any missing task queue dependencies
> 
> Build Configuration:
> BB_VERSION        = "1.20.0"
> BUILD_SYS         = "x86_64-linux"
> NATIVELSBSTRING   = "Fedora-18"
> TARGET_SYS        = "arm-angstrom-linux-gnueabi"
> MACHINE           = "hikirk"
> DISTRO            = "angstrom"
> DISTRO_VERSION    = "v2013.12"
> TUNE_FEATURES     = "armv5 thumb dsp"
> TARGET_FPU        = "soft"
> meta-hipos        = "(nobranch):6f2ad6754c9c9ddd6f1c5505e584e0eb62cfcf83"
> meta-hipos-kirkwood = "(nobranch):1a43f75e217fea3c6b4493536b3b66945ce53921"
> meta-hipos-fsl    = "(nobranch):ad3beec4157a96d0c506969a29d14f6354e7406f"
> meta-fsl-arm      = "(nobranch):af392c22bf6b563525ede4a81b6755ff1dd2c1c6"
> meta-fsl-arm-extra = "(nobranch):07ad83db0fb67c5023bd627a61efb7f474c52622"
> meta-java         = "(nobranch):9dfb7ecb87e67052e94641d6e02907c582264217"
> meta-angstrom     = "(nobranch):8fcd781b777a7a94f2b97514c9691a2b5c74eaf2"
> meta-oe
> meta-systemd
> meta-multimedia
> meta-networking
> meta-efl
> meta-gnome        = "(nobranch):eb4563b83be0a57ede4269ab19688af6baa62cd2"
> meta              = "(nobranch):e86622a932bbd0acdea67ecfb15c8b06c27353d8"
> 
> NOTE: Preparing runqueue
> ERROR: An uncaught exception occured in runqueue, please see the failure
> below: ERROR: Running idle function
> Traceback (most recent call last):
> 
>   File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/runqueue.py", line 
1111, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue instance 
at 0x1534b2d8>):
>     >        self.number_tasks =
>     >        int(self.cfgData.getVar("BB_NUMBER_THREADS", True) or 1)
>              self.scheduler = self.cfgData.getVar("BB_SCHEDULER", True) or
> "speed" File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py",
> line 516, in DataSmart.getVar(var='BB_NUMBER_THREADS', expand=True,
> noweakdefault=False):
>          def getVar(self, var, expand=False, noweakdefault=False):
>     >        return self.getVarFlag(var, "_content", expand, noweakdefault)
> 
>   File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line
> 606, in DataSmart.getVarFlag(var='BB_NUMBER_THREADS', flag='_content',
> expand=True, noweakdefault=False): cachename = var + "[" + flag + "]"
> 
>     >            value = self.expand(value, cachename)
> 
>              if value is not None and flag == "_content" and local_var is
> not None and "_removeactive" in local_var: File
> "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 344, in
> DataSmart.expand(s='${@oe.utils.cpu_count()}',
> varname='BB_NUMBER_THREADS'):
>          def expand(self, s, varname = None):
>     >        return self.expandWithRefs(s, varname).value
> 
>   File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line
> 334, in DataSmart.expandWithRefs(s='${@oe.utils.cpu_count()}',
> varname='BB_NUMBER_THREADS'):
>                  except Exception as exc:
>     >                raise ExpansionError(varname, s, exc)
> 
> ExpansionError: Failure expanding variable BB_NUMBER_THREADS, expression was
> ${@oe.utils.cpu_count()} which triggered exception AttributeError: 'module'
> object has no attribute 'cpu_count' ----------------------> snip
> <-------------------------
> 
> Any ideas? A missing python package at the build host? ...?

Looks like you're missing Richard's other posted patch that adds this 
function.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Richard Purdie]

On Wed, 2014-01-29 at 11:14 +0100, Steffen Sledz wrote:
> On 27.01.2014 15:39, Richard Purdie wrote:
> > Its rather sad that people don't appear to read local.conf and then complain
> > about slow builds when they're just using a single thread. Most systems have
> > more than one core now so we might as well use a more automatic default
> > for these values. This may lead to better experiences for new users.
> > 
> > [YOCTO #2528]
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> > diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
> > index 71856b8..36d33e1 100644
> > --- a/meta/conf/local.conf.sample
> > +++ b/meta/conf/local.conf.sample
> > @@ -18,12 +18,18 @@
> >  # option determines how many tasks bitbake should run in parallel:
> >  #
> >  #BB_NUMBER_THREADS ?= "4"
> > +#
> > +# Default to setting automatically based on cpu count
> > +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
> >  # 
> >  # The second option controls how many processes make should run in parallel when
> >  # running compile tasks:
> >  #
> >  #PARALLEL_MAKE ?= "-j 4"
> >  #
> > +# Default to setting automatically based on cpu count
> > +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
> > +#
> >  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
> >  # be appropriate for example.
> 
> On our Fedora-18 build host this change leads to the following exception. :(

Do you have:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=721773072da08cf0f5e1206961ada3083b6722b4

?

Cheers,

Richard

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Steffen Sledz]

On 29.01.2014 11:22, Paul Eggleton wrote:
> On Wednesday 29 January 2014 11:14:47 Steffen Sledz wrote:
>> On 27.01.2014 15:39, Richard Purdie wrote:
>>> Its rather sad that people don't appear to read local.conf and then
>>> complain about slow builds when they're just using a single thread. Most
>>> systems have more than one core now so we might as well use a more
>>> automatic default for these values. This may lead to better experiences
>>> for new users.
>>>
>>> [YOCTO #2528]
>>>
>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>> ---
>>> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
>>> index 71856b8..36d33e1 100644
>>> --- a/meta/conf/local.conf.sample
>>> +++ b/meta/conf/local.conf.sample
>>> @@ -18,12 +18,18 @@
>>>
>>>  # option determines how many tasks bitbake should run in parallel:
>>>  #
>>>  #BB_NUMBER_THREADS ?= "4"
>>>
>>> +#
>>> +# Default to setting automatically based on cpu count
>>> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
>>>
>>>  #
>>>  # The second option controls how many processes make should run in
>>>  parallel when # running compile tasks:
>>>  #
>>>  #PARALLEL_MAKE ?= "-j 4"
>>>  #
>>>
>>> +# Default to setting automatically based on cpu count
>>> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
>>> +#
>>>
>>>  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j
>>>  4" would # be appropriate for example.
>>
>> On our Fedora-18 build host this change leads to the following exception. :(
>>
>> ----------------------> snip <-------------------------
>> Loading cache...done.
>> Loaded 2167 entries from dependency cache.
>>
>> Parsing recipes...done.
>> Parsing of 1729 .bb files complete (1727 cached, 2 parsed). 2168 targets, 85
>> skipped, 1 masked, 0 errors.
>>
>> NOTE: Resolving any missing task queue dependencies
>>
>> Build Configuration:
>> BB_VERSION        = "1.20.0"
>> BUILD_SYS         = "x86_64-linux"
>> NATIVELSBSTRING   = "Fedora-18"
>> TARGET_SYS        = "arm-angstrom-linux-gnueabi"
>> MACHINE           = "hikirk"
>> DISTRO            = "angstrom"
>> DISTRO_VERSION    = "v2013.12"
>> TUNE_FEATURES     = "armv5 thumb dsp"
>> TARGET_FPU        = "soft"
>> meta-hipos        = "(nobranch):6f2ad6754c9c9ddd6f1c5505e584e0eb62cfcf83"
>> meta-hipos-kirkwood = "(nobranch):1a43f75e217fea3c6b4493536b3b66945ce53921"
>> meta-hipos-fsl    = "(nobranch):ad3beec4157a96d0c506969a29d14f6354e7406f"
>> meta-fsl-arm      = "(nobranch):af392c22bf6b563525ede4a81b6755ff1dd2c1c6"
>> meta-fsl-arm-extra = "(nobranch):07ad83db0fb67c5023bd627a61efb7f474c52622"
>> meta-java         = "(nobranch):9dfb7ecb87e67052e94641d6e02907c582264217"
>> meta-angstrom     = "(nobranch):8fcd781b777a7a94f2b97514c9691a2b5c74eaf2"
>> meta-oe
>> meta-systemd
>> meta-multimedia
>> meta-networking
>> meta-efl
>> meta-gnome        = "(nobranch):eb4563b83be0a57ede4269ab19688af6baa62cd2"
>> meta              = "(nobranch):e86622a932bbd0acdea67ecfb15c8b06c27353d8"
>>
>> NOTE: Preparing runqueue
>> ERROR: An uncaught exception occured in runqueue, please see the failure
>> below: ERROR: Running idle function
>> Traceback (most recent call last):
>>
>>   File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/runqueue.py", line 
> 1111, in RunQueueExecuteScenequeue.__init__(rq=<bb.runqueue.RunQueue instance 
> at 0x1534b2d8>):
>>     >        self.number_tasks =
>>     >        int(self.cfgData.getVar("BB_NUMBER_THREADS", True) or 1)
>>              self.scheduler = self.cfgData.getVar("BB_SCHEDULER", True) or
>> "speed" File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py",
>> line 516, in DataSmart.getVar(var='BB_NUMBER_THREADS', expand=True,
>> noweakdefault=False):
>>          def getVar(self, var, expand=False, noweakdefault=False):
>>     >        return self.getVarFlag(var, "_content", expand, noweakdefault)
>>
>>   File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line
>> 606, in DataSmart.getVarFlag(var='BB_NUMBER_THREADS', flag='_content',
>> expand=True, noweakdefault=False): cachename = var + "[" + flag + "]"
>>
>>     >            value = self.expand(value, cachename)
>>
>>              if value is not None and flag == "_content" and local_var is
>> not None and "_removeactive" in local_var: File
>> "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line 344, in
>> DataSmart.expand(s='${@oe.utils.cpu_count()}',
>> varname='BB_NUMBER_THREADS'):
>>          def expand(self, s, varname = None):
>>     >        return self.expandWithRefs(s, varname).value
>>
>>   File "/CACHE/jenkins-workspace/hipos/bitbake/lib/bb/data_smart.py", line
>> 334, in DataSmart.expandWithRefs(s='${@oe.utils.cpu_count()}',
>> varname='BB_NUMBER_THREADS'):
>>                  except Exception as exc:
>>     >                raise ExpansionError(varname, s, exc)
>>
>> ExpansionError: Failure expanding variable BB_NUMBER_THREADS, expression was
>> ${@oe.utils.cpu_count()} which triggered exception AttributeError: 'module'
>> object has no attribute 'cpu_count' ----------------------> snip
>> <-------------------------
>>
>> Any ideas? A missing python package at the build host? ...?
> 
> Looks like you're missing Richard's other posted patch that adds this 
> function.

Hmmm? Exactly the same revisions work well on another (openSUSE-13.1) build host.

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Steffen Sledz]

On 29.01.2014 11:59, Richard Purdie wrote:
> On Wed, 2014-01-29 at 11:14 +0100, Steffen Sledz wrote:
>> On 27.01.2014 15:39, Richard Purdie wrote:
>>> Its rather sad that people don't appear to read local.conf and then complain
>>> about slow builds when they're just using a single thread. Most systems have
>>> more than one core now so we might as well use a more automatic default
>>> for these values. This may lead to better experiences for new users.
>>>
>>> [YOCTO #2528]
>>>
>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>> ---
>>> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
>>> index 71856b8..36d33e1 100644
>>> --- a/meta/conf/local.conf.sample
>>> +++ b/meta/conf/local.conf.sample
>>> @@ -18,12 +18,18 @@
>>>  # option determines how many tasks bitbake should run in parallel:
>>>  #
>>>  #BB_NUMBER_THREADS ?= "4"
>>> +#
>>> +# Default to setting automatically based on cpu count
>>> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
>>>  # 
>>>  # The second option controls how many processes make should run in parallel when
>>>  # running compile tasks:
>>>  #
>>>  #PARALLEL_MAKE ?= "-j 4"
>>>  #
>>> +# Default to setting automatically based on cpu count
>>> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
>>> +#
>>>  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
>>>  # be appropriate for example.
>>
>> On our Fedora-18 build host this change leads to the following exception. :(
> 
> Do you have:
> 
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=721773072da08cf0f5e1206961ada3083b6722b4

No.

But now it becomes really interesting. As mentioned before it works well on our openSUSE-13.1 build host (without this commit)???

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Paul Eggleton]

On Wednesday 29 January 2014 12:47:54 Steffen Sledz wrote:
> On 29.01.2014 11:59, Richard Purdie wrote:
> > On Wed, 2014-01-29 at 11:14 +0100, Steffen Sledz wrote:
> >> On 27.01.2014 15:39, Richard Purdie wrote:
> >>> Its rather sad that people don't appear to read local.conf and then
> >>> complain about slow builds when they're just using a single thread.
> >>> Most systems have more than one core now so we might as well use a more
> >>> automatic default for these values. This may lead to better experiences
> >>> for new users.
> >>> 
> >>> [YOCTO #2528]
> >>> 
> >>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> >>> ---
> >>> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
> >>> index 71856b8..36d33e1 100644
> >>> --- a/meta/conf/local.conf.sample
> >>> +++ b/meta/conf/local.conf.sample
> >>> @@ -18,12 +18,18 @@
> >>> 
> >>>  # option determines how many tasks bitbake should run in parallel:
> >>>  #
> >>>  #BB_NUMBER_THREADS ?= "4"
> >>> 
> >>> +#
> >>> +# Default to setting automatically based on cpu count
> >>> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
> >>> 
> >>>  #
> >>>  # The second option controls how many processes make should run in
> >>>  parallel when # running compile tasks:
> >>>  #
> >>>  #PARALLEL_MAKE ?= "-j 4"
> >>>  #
> >>> 
> >>> +# Default to setting automatically based on cpu count
> >>> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
> >>> +#
> >>> 
> >>>  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j
> >>>  4" would # be appropriate for example.
> >> 
> >> On our Fedora-18 build host this change leads to the following exception.
> >> :(> 
> > Do you have:
> > 
> > http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=721773072da08cf0f5e12
> > 06961ada3083b6722b4
> No.
> 
> But now it becomes really interesting. As mentioned before it works well on
> our openSUSE-13.1 build host (without this commit)???

Are you sure you're not overriding BB_NUMBER_THREADS / PARALLEL_MAKE on the 
machine where it's "working"? You can use bitbake -e | less (variable history) 
to be sure.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Ulf Samuelsson]

2014-01-28 11:08, Koen Kooi skrev:
> Op 27 jan. 2014, om 15:39 heeft Richard Purdie <richard.purdie@linuxfoundation.org> het volgende geschreven:
>
>> Its rather sad that people don't appear to read local.conf and then complain
>> about slow builds when they're just using a single thread. Most systems have
>> more than one core now so we might as well use a more automatic default
>> for these values. This may lead to better experiences for new users.
>>
>> [YOCTO #2528]
>>
>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>> ---
>> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
>> index 71856b8..36d33e1 100644
>> --- a/meta/conf/local.conf.sample
>> +++ b/meta/conf/local.conf.sample
>> @@ -18,12 +18,18 @@
>> # option determines how many tasks bitbake should run in parallel:
>> #
>> #BB_NUMBER_THREADS ?= "4"
>> +#
>> +# Default to setting automatically based on cpu count
>> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
> I've noticed that after 4 threads IO becomes a big bottleneck when you have things like webkit, qt, asio etc in the buildqueue. Combine that with issues like every make -j thread taking >2GB ram with asio and webkit this default seems a bit high. I'd use 0.5*numcpu with a lower bound of 2.
>
> regards,
We discussed this 2.3 months ago.
Did some studies on my dual hex-core machine (24 H/W treads) while 
building a cloud9-gnome-image derivative.
This did about 7500 tasks.

Enabled the CPU supervisors in the panel.

Everything seems to be ok with BB_NUMBER_THREADS = "24" for about 4-4500 
tasks.

Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks.
Then parallellism  is resumed until about task 7000, and again
only a few CPUs are active.

I believe that some tools use "make" within the Makefile,
and they are written badly, and do not use "-j <n>" for
that part of the build.

Got my build down to 83 minutes.

Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build,
and copied the download and the recipes to the ram.

That shaved only 2 monutes from the build, and some stuff,
still built using only a single CPU.

BR
Ulf Samuelsson


> Koen
>
>> #
>> # The second option controls how many processes make should run in parallel when
>> # running compile tasks:
>> #
>> #PARALLEL_MAKE ?= "-j 4"
>> #
>> +# Default to setting automatically based on cpu count
>> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
>> +#
>> # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
>> # be appropriate for example.
>>
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core


-- 
Best Regards
Ulf Samuelsson
eMagii

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Steffen Sledz]

On 29.01.2014 12:52, Paul Eggleton wrote:
> On Wednesday 29 January 2014 12:47:54 Steffen Sledz wrote:
>> On 29.01.2014 11:59, Richard Purdie wrote:
>>> On Wed, 2014-01-29 at 11:14 +0100, Steffen Sledz wrote:
>>>> On 27.01.2014 15:39, Richard Purdie wrote:
>>>>> Its rather sad that people don't appear to read local.conf and then
>>>>> complain about slow builds when they're just using a single thread.
>>>>> Most systems have more than one core now so we might as well use a more
>>>>> automatic default for these values. This may lead to better experiences
>>>>> for new users.
>>>>>
>>>>> [YOCTO #2528]
>>>>>
>>>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>>>> ---
>>>>> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
>>>>> index 71856b8..36d33e1 100644
>>>>> --- a/meta/conf/local.conf.sample
>>>>> +++ b/meta/conf/local.conf.sample
>>>>> @@ -18,12 +18,18 @@
>>>>>
>>>>>  # option determines how many tasks bitbake should run in parallel:
>>>>>  #
>>>>>  #BB_NUMBER_THREADS ?= "4"
>>>>>
>>>>> +#
>>>>> +# Default to setting automatically based on cpu count
>>>>> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
>>>>>
>>>>>  #
>>>>>  # The second option controls how many processes make should run in
>>>>>  parallel when # running compile tasks:
>>>>>  #
>>>>>  #PARALLEL_MAKE ?= "-j 4"
>>>>>  #
>>>>>
>>>>> +# Default to setting automatically based on cpu count
>>>>> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
>>>>> +#
>>>>>
>>>>>  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j
>>>>>  4" would # be appropriate for example.
>>>>
>>>> On our Fedora-18 build host this change leads to the following exception.
>>>> :(> 
>>> Do you have:
>>>
>>> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=721773072da08cf0f5e12
>>> 06961ada3083b6722b4
>> No.
>>
>> But now it becomes really interesting. As mentioned before it works well on
>> our openSUSE-13.1 build host (without this commit)???
> 
> Are you sure you're not overriding BB_NUMBER_THREADS / PARALLEL_MAKE on the 
> machine where it's "working"? You can use bitbake -e | less (variable history) 
> to be sure.

Yes, this was a good hint.

Sorry for disturbance.

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Richard Purdie]

On Wed, 2014-01-29 at 13:09 +0100, Ulf Samuelsson wrote:
> We discussed this 2.3 months ago.
> Did some studies on my dual hex-core machine (24 H/W treads) while 
> building a cloud9-gnome-image derivative.
> This did about 7500 tasks.
> 
> Enabled the CPU supervisors in the panel.
> 
> Everything seems to be ok with BB_NUMBER_THREADS = "24" for about 4-4500 
> tasks.
> 
> Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks.
> Then parallellism  is resumed until about task 7000, and again
> only a few CPUs are active.

This is likely whilst the lib and toolchain is getting built.

> I believe that some tools use "make" within the Makefile,
> and they are written badly, and do not use "-j <n>" for
> that part of the build.

Which recipes were building at this point? It would be interesting to
track them down.

> Got my build down to 83 minutes.
> 
> Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build,
> and copied the download and the recipes to the ram.
> 
> That shaved only 2 monutes from the build, and some stuff,
> still built using only a single CPU.

There are certainly dependency bottlenecks in the build such as the
toolchain, compiler, gettext, gtk+ and so where large numbers of things
need those dependencies to get built before they can proceed. Not sure
what we can do to help this though.

Cheers,

Richard

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Ulf Samuelsson]

2014-01-29 13:56, Richard Purdie skrev:
> On Wed, 2014-01-29 at 13:09 +0100, Ulf Samuelsson wrote:
>> We discussed this 2.3 months ago.
>> Did some studies on my dual hex-core machine (24 H/W treads) while
>> building a cloud9-gnome-image derivative.
>> This did about 7500 tasks.
>>
>> Enabled the CPU supervisors in the panel.
>>
>> Everything seems to be ok with BB_NUMBER_THREADS = "24" for about 4-4500
>> tasks.
>>
>> Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks.
>> Then parallellism  is resumed until about task 7000, and again
>> only a few CPUs are active.
> This is likely whilst the lib and toolchain is getting built.
>
>> I believe that some tools use "make" within the Makefile,
>> and they are written badly, and do not use "-j <n>" for
>> that part of the build.
> Which recipes were building at this point? It would be interesting to
> track them down.
I think eglibc, node[-js] (native) and a few others

Think I remember mentioning them when it was discussed.

webkit, node are also not running in parallell in later part of the build

>
>> Got my build down to 83 minutes.
>>
>> Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build,
>> and copied the download and the recipes to the ram.
>>
>> That shaved only 2 monutes from the build, and some stuff,
>> still built using only a single CPU.
> There are certainly dependency bottlenecks in the build such as the
> toolchain, compiler, gettext, gtk+ and so where large numbers of things
> need those dependencies to get built before they can proceed. Not sure
> what we can do to help this though.
In order to find out more, perhaps it should be possible to log info 
about the build.
1. How many makes are in progress when you start a task
2. How long does it take to build a recipe.
3. CPU activity (frequency scaling) when the recipe is build.

A recipe which takes very long time to build but with low CPU activity
should be analyzed to find out if more parallellism can be introduced.

BR
Ulf
>
> Cheers,
>
> Richard
>


-- 
Best Regards
Ulf Samuelsson
ulf@emagii.com
+46 722 427437

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Ross Burton]

[-- Attachment #1: Type: text/plain, Size: 342 bytes --]

webkit, node are also not running in parallell in later part of the build 
WebKit does parallel builds internally so what you’re seeing here is everything that doesn’t depend on webkit has finished, so the remaining tasks are blocking on the webkit build.

Ross

-- 
Ross Burton
Open Source Technology Centre
Intel Corporation

[-- Attachment #2: Type: text/html, Size: 1962 bytes --]

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Laszlo Papp]

Thank you for this contribution.

As a side note, please consider the documentation bits in the future
as well. :-) I have just opened a separate ticket for that here:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=5770

On Mon, Jan 27, 2014 at 2:39 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Its rather sad that people don't appear to read local.conf and then complain
> about slow builds when they're just using a single thread. Most systems have
> more than one core now so we might as well use a more automatic default
> for these values. This may lead to better experiences for new users.
>
> [YOCTO #2528]
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
> index 71856b8..36d33e1 100644
> --- a/meta/conf/local.conf.sample
> +++ b/meta/conf/local.conf.sample
> @@ -18,12 +18,18 @@
>  # option determines how many tasks bitbake should run in parallel:
>  #
>  #BB_NUMBER_THREADS ?= "4"
> +#
> +# Default to setting automatically based on cpu count
> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
>  #
>  # The second option controls how many processes make should run in parallel when
>  # running compile tasks:
>  #
>  #PARALLEL_MAKE ?= "-j 4"
>  #
> +# Default to setting automatically based on cpu count
> +PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
> +#
>  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j 4" would
>  # be appropriate for example.
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Mike Looijmans]

On 01/29/2014 01:56 PM, Richard Purdie wrote:
> On Wed, 2014-01-29 at 13:09 +0100, Ulf Samuelsson wrote:
>> We discussed this 2.3 months ago.
>> Did some studies on my dual hex-core machine (24 H/W treads) while
>> building a cloud9-gnome-image derivative.
>> This did about 7500 tasks.
>>
>> Enabled the CPU supervisors in the panel.
>>
>> Everything seems to be ok with BB_NUMBER_THREADS = "24" for about 4-4500
>> tasks.
>>
>> Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks.
>> Then parallellism  is resumed until about task 7000, and again
>> only a few CPUs are active.
>
> This is likely whilst the lib and toolchain is getting built.
>
>> I believe that some tools use "make" within the Makefile,
>> and they are written badly, and do not use "-j <n>" for
>> that part of the build.
>
> Which recipes were building at this point? It would be interesting to
> track them down.

I would suspect "recursive" makes. If you use autotools with the evil 
"SUDIRS=..." construction, it will WAIT for that dir to finish before 
doing anything else.
It's much better to construct a giant makefile in the root, autotools 
will happily do that once properly instructed. I've seen first time 
builds going from half an hour to two minutes, and incremental builds 
taking only one or two seconds instead of several minutes just because I 
removed the recursion.

>> Got my build down to 83 minutes.
>>
>> Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build,
>> and copied the download and the recipes to the ram.
>>
>> That shaved only 2 monutes from the build, and some stuff,
>> still built using only a single CPU.

That confirms what I have already suspected - it is pointless to buy an 
SSD, building OE is mostly CPU limited and hardly I/O related.

I guess the only way to really speed up the build would be to have 
multiple machines participate in it. Single machines aren't really 
getting any faster.

> There are certainly dependency bottlenecks in the build such as the
> toolchain, compiler, gettext, gtk+ and so where large numbers of things
> need those dependencies to get built before they can proceed. Not sure
> what we can do to help this though.

Move them to the front as far as possible I guess. And any package they 
depend on as well... It should try to set up the shortest tree to be 
able to build the crosscompiler and build that first...

Mike.

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Ulf Samuelsson]

1 feb 2014 kl. 10:21 skrev Mike Looijmans <mike.looijmans@topic.nl>:

> On 01/29/2014 01:56 PM, Richard Purdie wrote:
>> On Wed, 2014-01-29 at 13:09 +0100, Ulf Samuelsson wrote:
>>> We discussed this 2.3 months ago.
>>> Did some studies on my dual hex-core machine (24 H/W treads) while
>>> building a cloud9-gnome-image derivative.
>>> This did about 7500 tasks.
>>> 
>>> Enabled the CPU supervisors in the panel.
>>> 
>>> Everything seems to be ok with BB_NUMBER_THREADS = "24" for about 4-4500
>>> tasks.
>>> 
>>> Then the CPUs are mostly inactive and only 1-2 running for ~500 tasks.
>>> Then parallellism  is resumed until about task 7000, and again
>>> only a few CPUs are active.
>> 
>> This is likely whilst the lib and toolchain is getting built.
>> 
>>> I believe that some tools use "make" within the Makefile,
>>> and they are written badly, and do not use "-j <n>" for
>>> that part of the build.
>> 
>> Which recipes were building at this point? It would be interesting to
>> track them down.
> 
> I would suspect "recursive" makes. If you use autotools with the evil "SUDIRS=..." construction, it will WAIT for that dir to finish before doing anything else.
> It's much better to construct a giant makefile in the root, autotools will happily do that once properly instructed. I've seen first time builds going from half an hour to two minutes, and incremental builds taking only one or two seconds instead of several minutes just because I removed the recursion.
> 
>>> Got my build down to 83 minutes.
>>> 
>>> Since I have 96 GB of RAM, I tried creating an 80 GB tmpfs for the build,
>>> and copied the download and the recipes to the ram.
>>> 
>>> That shaved only 2 monutes from the build, and some stuff,
>>> still built using only a single CPU.
> 
> That confirms what I have already suspected - it is pointless to buy an SSD, building OE is mostly CPU limited and hardly I/O related.

To be fair, the 83 minute build was on a RAID with two striped 600 GB, 15k RPM, 3 Gbps SAS disks (luckily on the machine I got on eBay :-)
hdparm -t -T give about 390 MB/s.

If I run a single late model SATA-III disk (on a SATA-II) I get 120-150 MB/s
with the same test.

Using 15k RPM SAS disks will give 2 x the number of seeks per second
vs the SATA disks, (170 vs 85 IIRC)
Using RAID does not increase this value.

> 
> I guess the only way to really speed up the build would be to have multiple machines participate in it. Single machines aren't really getting any faster.
> 
>> There are certainly dependency bottlenecks in the build such as the
>> toolchain, compiler, gettext, gtk+ and so where large numbers of things
>> need those dependencies to get built before they can proceed. Not sure
>> what we can do to help this though.
> 
> Move them to the front as far as possible I guess. And any package they depend on as well... It should try to set up the shortest tree to be able to build the crosscompiler and build that first...
> 

If I understand things correctly, bitbake will calculate a priority for a certain package based on
how many packages are depending on this package.
If you could add a value to the priority, specified in the recipe
then you could make the build of the package occur earlier.

If bitbakes can track the loading of the CPU, then maybe some parts
in the later part of each recipe can be delayed until loading is low (maybe already doing this).

If the toolchain/native stuff could be pulled down from packages as part of a build
it would be real cool.
Today you can install cross compilers for certain Linux distributions.
Maybe there should be a possibility to generate automatically the .rpm/.deb etc.
and make that the default way to get stuff done for those distributions.

Best Regards
Ulf Samuelsson


> Mike.
> 
> 
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Richard Purdie]

On Mon, 2014-02-03 at 22:23 +0100, Ulf Samuelsson wrote:
> 1 feb 2014 kl. 10:21 skrev Mike Looijmans <mike.looijmans@topic.nl>:
> 
> > On 01/29/2014 01:56 PM, Richard Purdie wrote:
> >> On Wed, 2014-01-29 at 13:09 +0100, Ulf Samuelsson wrote:
> If bitbakes can track the loading of the CPU, then maybe some parts
> in the later part of each recipe can be delayed until loading is low (maybe already doing this).

It doesn't. We did try this, the Linux scheduler is in fact rather good
and we only ever slowed down builds when we tried adding hints.

> If the toolchain/native stuff could be pulled down from packages as part of a build
> it would be real cool.

This is called sstate ;-).

> Today you can install cross compilers for certain Linux distributions.
> Maybe there should be a possibility to generate automatically the .rpm/.deb etc.
> and make that the default way to get stuff done for those distributions.

Its a maintenance/bug report nightmare...

Cheers,

Richard

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Enrico Scholz]

Koen Kooi <koen-QLwJDigV5abLmq1fohREcCpxlwaOVQ5f@public.gmane.org>
writes:

>> +# Default to setting automatically based on cpu count
>> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
>
> I've noticed that after 4 threads IO becomes a big bottleneck when you
> have things like webkit, qt, asio etc in the buildqueue. Combine that
> with issues like every make -j thread taking >2GB ram with asio and
> webkit this default seems a bit high. I'd use 0.5*numcpu with a lower
> bound of 2.

limitting the load mitigates this (high i/o increases it); e.g.

PARALLEL_MAKE = "\
  ...
  -l ${@int(os.sysconf(os.sysconf_names['SC_NPROCESSORS_ONLN'])) * 150/100} \
"

Enrico

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Mark Hatle]

On 2/4/14, 10:13 AM, Enrico Scholz wrote:
> Koen Kooi <koen-QLwJDigV5abLmq1fohREcCpxlwaOVQ5f@public.gmane.org>
> writes:
>
>>> +# Default to setting automatically based on cpu count
>>> +BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
>>
>> I've noticed that after 4 threads IO becomes a big bottleneck when you
>> have things like webkit, qt, asio etc in the buildqueue. Combine that
>> with issues like every make -j thread taking >2GB ram with asio and
>> webkit this default seems a bit high. I'd use 0.5*numcpu with a lower
>> bound of 2.
>
> limitting the load mitigates this (high i/o increases it); e.g.
>
> PARALLEL_MAKE = "\
>    ...
>    -l ${@int(os.sysconf(os.sysconf_names['SC_NPROCESSORS_ONLN'])) * 150/100} \
> "

FYI, I think this points out the variability in system performance, between CPU, 
RAM and I/o.

As it stands the patch gives my machine the best performance.  So I like it as 
it is.  But my machine (dual 8-core w/ HT, 64 GB of RAM, and hardware raid). 
But on hardware with less RAM, slower disk, it may not perform optimally.

So the catch is what is the proper optimal setting?  As I see it, assuming that 
the system has enough ram and I/O to fill the CPUs is the best approach (what 
was implemented.)  And then in the comments document that this may not be the 
best setting for all systems, and the user should adjust it as necessary.  Even 
suggesting some of the alternative systems such as the 150/100 above.

No setting is going to make everyone happy, but something has to be better then 
defaulting to '1'.

--Mark

> Enrico
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

Re: [PATCH] local.conf.sample: Add automatic defaults for BB_NUMBER_THREADS and PARALLEL_MAKE

[Enrico Scholz]

Mark Hatle <mark.hatle-CWA4WttNNZF54TAoqtyWWQ@public.gmane.org> writes:

>>> I've noticed that after 4 threads IO becomes a big bottleneck when
>>> you have things like webkit, qt, asio etc in the buildqueue. Combine
>>> that with issues like every make -j thread taking >2GB ram with asio
>>> and webkit this default seems a bit high. I'd use 0.5*numcpu with a
>>> lower bound of 2.
>>
>> limitting the load mitigates this (high i/o increases it); e.g.
>>
>> PARALLEL_MAKE = "\
>>    ...
>>    -l ${@int(os.sysconf(os.sysconf_names['SC_NPROCESSORS_ONLN'])) * 150/100} \
>> "
>
> FYI, I think this points out the variability in system performance,
> between CPU, RAM and I/o.
>
> As it stands the patch gives my machine the best performance.  So I
> like it as it is.

Point of my posting was not the exact scale (I just copied a piece of
my ~/.bitbake.conf), but using '-l' in addition to '-j'.  Doing heavy
i/o or being low on RAM (which causes swapping) increases the load and
'make' will throttle then.


> No setting is going to make everyone happy but something has to be
> better then defaulting to '1'.

That's true; but not due to performance gain but because it will help to
detect races in the build process.


Enrico