Dynamic pipeline execution

Dynamic pipeline execution

[Anton Mikanovich]

Hello, we are using Bitbake 1.50.4 inside custom Yocto-like build system for
Debian packages building and need to have a dynamic pipeline changing 
based on
information obtained during the execution.

Here is simple 3-stage example of what we are doing currently in POC with 3
bitbake calls from external tinfoil-based wrapper:
1) Perform fetch tasks for some set of recipes to get their sources.
2) Obtaining some kind of data from the sources and generate dependency list
based on that data (analyzing debian/control files).
3) Perform build tasks for the original set of recipes and dependency 
recipes
obtained from the previous step.

The question is how can we do the same logic within only one bitbake call?
We see two possible ways of bitbake sources modification to achieve this:
1. Add an API for dynamic injection of tasks on already started cooker.
2. Rebuild runqueue handling to enable per-task dependencies 
setting/analyzing.

What is the best way to focus on?
Or maybe there is any other more simple solution for the same logic?

Thanks for any advice.

Re: [bitbake-devel] Dynamic pipeline execution

[Richard Purdie]

On Mon, 2022-03-28 at 21:43 +0300, Anton Mikanovich wrote:
> Hello, we are using Bitbake 1.50.4 inside custom Yocto-like build system for
> Debian packages building and need to have a dynamic pipeline changing 
> based on
> information obtained during the execution.
> 
> Here is simple 3-stage example of what we are doing currently in POC with 3
> bitbake calls from external tinfoil-based wrapper:
> 1) Perform fetch tasks for some set of recipes to get their sources.
> 2) Obtaining some kind of data from the sources and generate dependency list
> based on that data (analyzing debian/control files).
> 3) Perform build tasks for the original set of recipes and dependency 
> recipes
> obtained from the previous step.
> 
> The question is how can we do the same logic within only one bitbake call?
> We see two possible ways of bitbake sources modification to achieve this:
> 1. Add an API for dynamic injection of tasks on already started cooker.
> 2. Rebuild runqueue handling to enable per-task dependencies 
> setting/analyzing.
> 
> What is the best way to focus on?
> Or maybe there is any other more simple solution for the same logic?

That is going to be quite hard to do generically given other constraints we've
put upon bitbake tasks. The setscene task graph is walked backwards, the main
task graph is walked forward and so bitbake has to know what the task graph
looks like in advance in order to be able to work on things from both directions
and to be able to produce the taskhashes.

Put another way, bitbake expects to be able to construct a task graph without
execution (e.g. fetching/examining sources). The places where we do allow
external inputs (e.g. querying an SCM for a revision) are fraught with race
issues already.

Where we have this challenge (e.g. crates and go modules), we're ended up with
solutions which compute our metadata, then bitbake executes it. We haven't
linked those commands into a single bitbake execution.

Why do you want to use a single bitbake invocation rather than your existing
tinfoil wrapper?

Cheers,

Richard

Re: [bitbake-devel] Dynamic pipeline execution

[Anton Mikanovich]

29.03.2022 02:17, Richard Purdie wrote:
> On Mon, 2022-03-28 at 21:43 +0300, Anton Mikanovich wrote:
>> Hello, we are using Bitbake 1.50.4 inside custom Yocto-like build system for
>> Debian packages building and need to have a dynamic pipeline changing
>> based on
>> information obtained during the execution.
>>
>> Here is simple 3-stage example of what we are doing currently in POC with 3
>> bitbake calls from external tinfoil-based wrapper:
>> 1) Perform fetch tasks for some set of recipes to get their sources.
>> 2) Obtaining some kind of data from the sources and generate dependency list
>> based on that data (analyzing debian/control files).
>> 3) Perform build tasks for the original set of recipes and dependency
>> recipes
>> obtained from the previous step.
>>
>> The question is how can we do the same logic within only one bitbake call?
>> We see two possible ways of bitbake sources modification to achieve this:
>> 1. Add an API for dynamic injection of tasks on already started cooker.
>> 2. Rebuild runqueue handling to enable per-task dependencies
>> setting/analyzing.
>>
>> What is the best way to focus on?
>> Or maybe there is any other more simple solution for the same logic?
> That is going to be quite hard to do generically given other constraints we've
> put upon bitbake tasks. The setscene task graph is walked backwards, the main
> task graph is walked forward and so bitbake has to know what the task graph
> looks like in advance in order to be able to work on things from both directions
> and to be able to produce the taskhashes.
>
> Put another way, bitbake expects to be able to construct a task graph without
> execution (e.g. fetching/examining sources). The places where we do allow
> external inputs (e.g. querying an SCM for a revision) are fraught with race
> issues already.
>
> Where we have this challenge (e.g. crates and go modules), we're ended up with
> solutions which compute our metadata, then bitbake executes it. We haven't
> linked those commands into a single bitbake execution.
>
> Why do you want to use a single bitbake invocation rather than your existing
> tinfoil wrapper?
>
> Cheers,
>
> Richard

Hello Richard, thanks for quick reply.
Using single bitbake invocation can reduce overhead for recipes parsing and
task graph creation. We already have ~50 multiconfigs, so it takes some 
time.
It will also allow to save task parallelization during processing.
And it is more convenient to retain bitbake as entry point for the user.