Customize Number of Threads per Task?

Customize Number of Threads per Task?

[Post, Travis]

[-- Attachment #1: Type: text/plain, Size: 409 bytes --]

Hello:

I’ve written a bitbake class that adds a custom task in-between do_configure and do_compile, called ‘do_send_to_remote’. Is there any way for Yocto to allow only one instance of this task to run at a time? I know I can set BB_NUMBER_THREADS = 1, but that will slow down everything. Is there a way to do something like: BB_NUMBER_THREADS[do_send_to_remote] = 1?

Thanks,


Travis





[-- Attachment #2: Type: text/html, Size: 2424 bytes --]

Re: Customize Number of Threads per Task?

[Paul Eggleton]

Hi Travis,

On Friday 17 May 2013 00:40:51 Post, Travis wrote:
> I’ve written a bitbake class that adds a custom task in-between do_configure
> and do_compile, called ‘do_send_to_remote’. Is there any way for Yocto to
> allow only one instance of this task to run at a time? I know I can set
> BB_NUMBER_THREADS = 1, but that will slow down everything. Is there a way
> to do something like: BB_NUMBER_THREADS[do_send_to_remote] = 1?

I don't think we have anything quite like that, but we do have support for 
locking via a "lockfiles" parameter on a task function; e.g. from 
rootfs_rpm.bbclass:

do_rootfs[lockfiles] += "${DEPLOY_DIR_RPM}/rpm.lock"

That should ensure only one instance of the task's code gets to run at a time.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Customize Number of Threads per Task?

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]

On Fri, May 17, 2013 at 1:53 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> Hi Travis,
>
> On Friday 17 May 2013 00:40:51 Post, Travis wrote:
> > I’ve written a bitbake class that adds a custom task in-between
> do_configure
> > and do_compile, called ‘do_send_to_remote’. Is there any way for Yocto to
> > allow only one instance of this task to run at a time? I know I can set
> > BB_NUMBER_THREADS = 1, but that will slow down everything. Is there a way
> > to do something like: BB_NUMBER_THREADS[do_send_to_remote] = 1?
>
> I don't think we have anything quite like that, but we do have support for
> locking via a "lockfiles" parameter on a task function; e.g. from
> rootfs_rpm.bbclass:
>
> do_rootfs[lockfiles] += "${DEPLOY_DIR_RPM}/rpm.lock"
>
> That should ensure only one instance of the task's code gets to run at a
> time.
>

One could implement this through a custom bitbake scheduler. At one point I
had a scheduler which let you define a task flag to indicate which resource
it predominantly used (cpu, memory, network) and then let you define
per-resource maximum thread counts, but it was extremely naive, and didn't
improve overall performance much :\
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1841 bytes --]