Should multiconfig save build time?

Should multiconfig save build time?

[Alan Kozlay]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

All,

We have need to generate 6 slighly different images, with different
dedicated applications and use cases, but they're all destined for the same
hardware.
We have multiconfig set to build these 6 variants.  Clearly, they'll all
share a majority of underlying packages.
While the bitbake initialization takes a long time (*with multiconfig*), we
were under the impression that it could save time during the actual build
process because it could have the potential to reuse packages already built
from mc A for mc B if the inputs to the recipes are unchanged (if same
xxx_FEATURES, same arch, etc.).

This was a guess on our part since we initially didn't assign different tmp
directories for each conf.  So, the <WORKDIR> didn't change between mc
builds.  That actually was causing some issues until we realized that
separate tmp dirs was recommended for them.
Since they all now have separate tmp dirs, they clearly are not going to
share work dirs, so, now we're questioning our original assumption about
any build-time speed benefit.

So, is mc expected to be any faster than, say, 6 separate invocations of
non-mc bitbake whilst switching out the local.conf between them?

Thanks,
-Alan

-- 
Alan Kozlay
VES Solutions, LLC
Alan Kozlay <alan.kozlay@ves.solutions>
DESK 443-595-7734
CELL 443-752-0323

[-- Attachment #2: Type: text/html, Size: 2346 bytes --]

Re: Should multiconfig save build time?

[Richard Purdie]

On Fri, 2019-06-28 at 13:31 -0400, Alan Kozlay wrote:
> We have need to generate 6 slighly different images, with different
> dedicated applications and use cases, but they're all destined for
> the same hardware.
> We have multiconfig set to build these 6 variants.  Clearly, they'll
> all share a majority of underlying packages.
> While the bitbake initialization takes a long time (with
> multiconfig), we were under the impression that it could save time
> during the actual build process because it could have the potential
> to reuse packages already built from mc A for mc B if the inputs to
> the recipes are unchanged (if same xxx_FEATURES, same arch, etc.).
> 
> This was a guess on our part since we initially didn't assign
> different tmp directories for each conf.  So, the <WORKDIR> didn't
> change between mc builds.  That actually was causing some issues
> until we realized that separate tmp dirs was recommended for them.
> Since they all now have separate tmp dirs, they clearly are not going
> to share work dirs, so, now we're questioning our original assumption
> about any build-time speed benefit.
> 
> So, is mc expected to be any faster than, say, 6 separate invocations
> of non-mc bitbake whilst switching out the local.conf between them? 

Ultimately, yes, it would be expected to be faster.

With the code as it stands today, no, it won't be. Its a known open
issue:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=10682

What would be faster is to build one of the configs initially which
will populate the sstate cache, then build the other 5 together in one
multiconfig.

We obviously want to get that issue fixed ultimately but its not an
easy thing to do due to the way our codebase works.

Cheers,

Richard

Re: Should multiconfig save build time?

[chris.laplante]

> We have need to generate 6 slighly different images, with different dedicated applications and use cases, but they're all destined for
> the same hardware.
> We have multiconfig set to build these 6 variants.  Clearly, they'll all share a majority of underlying packages.
> While the bitbake initialization takes a long time (with multiconfig), we were under the impression that it could save time during the
> actual build process because it could have the potential to reuse packages already built from mc A for mc B if the inputs to the recipes
> are unchanged (if same xxx_FEATURES, same arch, etc.).
> 
> This was a guess on our part since we initially didn't assign different tmp directories for each conf.  So, the <WORKDIR> didn't change
> between mc builds.  That actually was causing some issues until we realized that separate tmp dirs was recommended for them.
> Since they all now have separate tmp dirs, they clearly are not going to share work dirs, so, now we're questioning our original
> assumption about any build-time speed benefit.

Have you considered whether you actually need multiconfig in this case? If the arch and distro features don't change, you should just be able to have multiple image recipes.

Chris 

Re: Should multiconfig save build time?

[Alan Kozlay]

[-- Attachment #1: Type: text/plain, Size: 1903 bytes --]

I wasn't completely clear in my origin post.  We actually do use the mc to
specify different DISTRO and IMAGE features.  But my point was that, for
recipies which are not affected by a change in those features, couldn't
such as recipe's workdir be leveraged by another mc conf.  But Richard's
response was clear.  That's not something which is supported, currently.

On Mon, Jul 1, 2019 at 10:13 AM chris.laplante@agilent.com <
chris.laplante@agilent.com> wrote:

> > We have need to generate 6 slighly different images, with different
> dedicated applications and use cases, but they're all destined for
> > the same hardware.
> > We have multiconfig set to build these 6 variants.  Clearly, they'll all
> share a majority of underlying packages.
> > While the bitbake initialization takes a long time (with multiconfig),
> we were under the impression that it could save time during the
> > actual build process because it could have the potential to reuse
> packages already built from mc A for mc B if the inputs to the recipes
> > are unchanged (if same xxx_FEATURES, same arch, etc.).
> >
> > This was a guess on our part since we initially didn't assign different
> tmp directories for each conf.  So, the <WORKDIR> didn't change
> > between mc builds.  That actually was causing some issues until we
> realized that separate tmp dirs was recommended for them.
> > Since they all now have separate tmp dirs, they clearly are not going to
> share work dirs, so, now we're questioning our original
> > assumption about any build-time speed benefit.
>
> Have you considered whether you actually need multiconfig in this case? If
> the arch and distro features don't change, you should just be able to have
> multiple image recipes.
>
> Chris
>


-- 
Alan Kozlay
VES Solutions, LLC
Alan Kozlay <alan.kozlay@ves.solutions>
DESK 443-595-7734
CELL 443-752-0323

[-- Attachment #2: Type: text/html, Size: 3154 bytes --]

Re: Should multiconfig save build time?

[Alan Kozlay]

[-- Attachment #1: Type: text/plain, Size: 3509 bytes --]

All,

In regards to our original June 28th post:

We're looking for things we can do to improve the multiconfig build time.

Even on our build servers it still takes a long time.
For instance, our build servers each have 64 core, 128G RAM with 4 NVMe SSD
HDs in a striped raid (*all configured with bells and whistles to make it
super fast*).
A full fresh build on one of those servers still takes 6 hours.

I need to also clarify that we do indeed define different DISTRO_FEATURES
between multiconfigs, so multiconfig is necessary.

(1) We're attempting to use the ccache (via ccache.bbclass), but it's
causing some build errors that we're investigating.
(2) Can we at least set the root native work directory to be the same for
all multiconfigs?

   - Would each multiconfig honor the previously built native tools?
   - Or, would it just rebuild them, anyway?
   - Is this safe or tested or supported?

(3) We're considering a network shared sstate-cache.  But it's not entirely
clear if a multiconfig build is benefited by the sstate-cache directory.

   - Under what circumstances does a multiconfig leverage sstate-cache when
   building for another multiconfig?
   - What if the recipe would be affected by a change in DISTRO_FEATURES
   between multiconfigs?
   - Are native packages included in the sstate-cache or just target
   packages?

We welcome all thoughts and ideas.

-Alan

On Sun, Jun 30, 2019 at 5:38 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Fri, 2019-06-28 at 13:31 -0400, Alan Kozlay wrote:
> > We have need to generate 6 slighly different images, with different
> > dedicated applications and use cases, but they're all destined for
> > the same hardware.
> > We have multiconfig set to build these 6 variants.  Clearly, they'll
> > all share a majority of underlying packages.
> > While the bitbake initialization takes a long time (with
> > multiconfig), we were under the impression that it could save time
> > during the actual build process because it could have the potential
> > to reuse packages already built from mc A for mc B if the inputs to
> > the recipes are unchanged (if same xxx_FEATURES, same arch, etc.).
> >
> > This was a guess on our part since we initially didn't assign
> > different tmp directories for each conf.  So, the <WORKDIR> didn't
> > change between mc builds.  That actually was causing some issues
> > until we realized that separate tmp dirs was recommended for them.
> > Since they all now have separate tmp dirs, they clearly are not going
> > to share work dirs, so, now we're questioning our original assumption
> > about any build-time speed benefit.
> >
> > So, is mc expected to be any faster than, say, 6 separate invocations
> > of non-mc bitbake whilst switching out the local.conf between them?
>
> Ultimately, yes, it would be expected to be faster.
>
> With the code as it stands today, no, it won't be. Its a known open
> issue:
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=10682
>
> What would be faster is to build one of the configs initially which
> will populate the sstate cache, then build the other 5 together in one
> multiconfig.
>
> We obviously want to get that issue fixed ultimately but its not an
> easy thing to do due to the way our codebase works.
>
> Cheers,
>
> Richard
>
>

-- 
Alan Kozlay
VES Solutions, LLC
Alan Kozlay <alan.kozlay@ves.solutions>
DESK 443-595-7734
CELL 443-752-0323

[-- Attachment #2: Type: text/html, Size: 5155 bytes --]

Re: Should multiconfig save build time?

[Robert Yang]

On 7/9/19 2:27 AM, Alan Kozlay wrote:
> All,
> 
> In regards to our original June 28th post:
> 
> We're looking for things we can do to improve the multiconfig build time.
> 
> Even on our build servers it still takes a long time.
> For instance, our build servers each have 64 core, 128G RAM with 4 NVMe SSD HDs 
> in a striped raid (/all configured with bells and whistles to make it super fast/).
> A full fresh build on one of those servers still takes 6 hours.
> 
> I need to also clarify that we do indeed define different DISTRO_FEATURES 
> between multiconfigs, so multiconfig is necessary.
> 
> (1) We're attempting to use the ccache (via ccache.bbclass), but it's causing 
> some build errors that we're investigating.

Which branch are you using, please? What's the detailed errors?

> (2) Can we at least set the root native work directory to be the same for all 
> multiconfigs?
> 
>   * Would each multiconfig honor the previously built native tools?
>   * Or, would it just rebuild them, anyway?
>   * Is this safe or tested or supported?
> 
> (3) We're considering a network shared sstate-cache.  But it's not entirely 
> clear if a multiconfig build is benefited by the sstate-cache directory.
> 
>   * Under what circumstances does a multiconfig leverage sstate-cache when
>     building for another multiconfig?

Only when the checksums are the same.

>   * What if the recipe would be affected by a change in DISTRO_FEATURES between >     multiconfigs?

You can use:
$ bitbake -Sprintdiff <recipe> (or world)

Before and after DISTRO_FEATURES changed to see the differences.

>   * Are native packages included in the sstate-cache or just target packages?

I prefer including all of them in the sstate-cache, e.g., allarch recipes
may also re-use the sstate-cache.


Build time is a problem, I had tried to adjust the build options to make it
faster, but didn't improve a lot. Make bitbake support distributed build
might be a good choice, not distribute C/C++ files, but distribute
the whole recipes or task queues to other hosts to build them. bitbake
doesn't support this atm. Perhaps we may make it work in the future,
add RP in the loop to see his comments.

// Robert

> 
> We welcome all thoughts and ideas.
> 
> -Alan
> 
> On Sun, Jun 30, 2019 at 5:38 PM Richard Purdie 
> <richard.purdie@linuxfoundation.org <mailto:richard.purdie@linuxfoundation.org>> 
> wrote:
> 
>     On Fri, 2019-06-28 at 13:31 -0400, Alan Kozlay wrote:
>      > We have need to generate 6 slighly different images, with different
>      > dedicated applications and use cases, but they're all destined for
>      > the same hardware.
>      > We have multiconfig set to build these 6 variants.  Clearly, they'll
>      > all share a majority of underlying packages.
>      > While the bitbake initialization takes a long time (with
>      > multiconfig), we were under the impression that it could save time
>      > during the actual build process because it could have the potential
>      > to reuse packages already built from mc A for mc B if the inputs to
>      > the recipes are unchanged (if same xxx_FEATURES, same arch, etc.).
>      >
>      > This was a guess on our part since we initially didn't assign
>      > different tmp directories for each conf.  So, the <WORKDIR> didn't
>      > change between mc builds.  That actually was causing some issues
>      > until we realized that separate tmp dirs was recommended for them.
>      > Since they all now have separate tmp dirs, they clearly are not going
>      > to share work dirs, so, now we're questioning our original assumption
>      > about any build-time speed benefit.
>      >
>      > So, is mc expected to be any faster than, say, 6 separate invocations
>      > of non-mc bitbake whilst switching out the local.conf between them?
> 
>     Ultimately, yes, it would be expected to be faster.
> 
>     With the code as it stands today, no, it won't be. Its a known open
>     issue:
> 
>     https://bugzilla.yoctoproject.org/show_bug.cgi?id=10682
> 
>     What would be faster is to build one of the configs initially which
>     will populate the sstate cache, then build the other 5 together in one
>     multiconfig.
> 
>     We obviously want to get that issue fixed ultimately but its not an
>     easy thing to do due to the way our codebase works.
> 
>     Cheers,
> 
>     Richard
> 
> 
> 
> -- 
> Alan Kozlay
> VES Solutions, LLC
> Alan Kozlay <alan.kozlay@ves.solutions>
> DESK 443-595-7734
> CELL 443-752-0323
> 

Re: Should multiconfig save build time?

[Richard Purdie]

On Mon, 2019-07-08 at 14:27 -0400, Alan Kozlay wrote:
> We're looking for things we can do to improve the multiconfig build
> time.
> 
> Even on our build servers it still takes a long time.
> For instance, our build servers each have 64 core, 128G RAM with 4
> NVMe SSD HDs in a striped raid (all configured with bells and
> whistles to make it super fast).
> A full fresh build on one of those servers still takes 6 hours.

Given we have no idea what these mulitconfigs are, how much overlap
there is or how you're running it, its hard to know whether 6 hours is
good or bad.

> I need to also clarify that we do indeed define different
> DISTRO_FEATURES between multiconfigs, so multiconfig is necessary.
> 
> (1) We're attempting to use the ccache (via ccache.bbclass), but it's
> causing some build errors that we're investigating.

That shouldn't cause errors and would likely be some ccache problem.

> (2) Can we at least set the root native work directory to be the same
> for all multiconfigs?

It depends if the configurations are different or not. WORKDIR isn't
your problem here, you really need the builds to have the same sysroot-
components dir for the native case too, else this won't work. Its more
about the output from the task than the directory they were built in.

> Would each multiconfig honor the previously built native tools?
> Or, would it just rebuild them, anyway?
> Is this safe or tested or supported?

multiconfig is configured so that if it sees overlapping stamp files it
would handle that correctly. The tricky part is you'd have to split
some directories which aren't usually split such as sysroot-components. 
I've never seen that attempted and am not sure if it would work or not.

> (3) We're considering a network shared sstate-cache.  But it's not
> entirely clear if a multiconfig build is benefited by the sstate-
> cache directory.
> Under what circumstances does a multiconfig leverage sstate-cache
> when building for another multiconfig?

Right now today for a single multiconfig build it won't.

> What if the recipe would be affected by a change in DISTRO_FEATURES
> between multiconfigs?
> Are native packages included in the sstate-cache or just target
> packages?

native and target are included in sstate-cache.

> We welcome all thoughts and ideas.

With current bitbake you'd be best off splitting your multiconfig build
into two parts a first stage where you build common shared components
and a second where you leverage them.

I appreciate that isn't a good solution but its what will work right
now.

The initial work in adapting bitbake to better handle situations like
this is being done, as we speak and there are patches starting it in
master-next.

Right now those patches also won't change anything for you, they re-
architect the codebase so we have a chance of implementing what you
want. It is however at least a start. I'm working on this effectively
alone with the day to day patch review, email and bugs to handle at the
same time :(. As a project we know this is important but we struggle to
find people willing to contribute development resources to make these
kinds of changes happen.

Next step is to write a regression test suite for runqueue. Once I've
done that I'll start to think about optimisation work.

Cheers,

Richard

Re: Should multiconfig save build time?

[Alan Kozlay]

[-- Attachment #1: Type: text/plain, Size: 3778 bytes --]

Thanks for the feedback.

On Wed, Jul 10, 2019 at 5:32 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2019-07-08 at 14:27 -0400, Alan Kozlay wrote:
> > We're looking for things we can do to improve the multiconfig build
> > time.
> >
> > Even on our build servers it still takes a long time.
> > For instance, our build servers each have 64 core, 128G RAM with 4
> > NVMe SSD HDs in a striped raid (all configured with bells and
> > whistles to make it super fast).
> > A full fresh build on one of those servers still takes 6 hours.
>
> Given we have no idea what these mulitconfigs are, how much overlap
> there is or how you're running it, its hard to know whether 6 hours is
> good or bad.
>
> > I need to also clarify that we do indeed define different
> > DISTRO_FEATURES between multiconfigs, so multiconfig is necessary.
> >
> > (1) We're attempting to use the ccache (via ccache.bbclass), but it's
> > causing some build errors that we're investigating.
>
> That shouldn't cause errors and would likely be some ccache problem.
>
> > (2) Can we at least set the root native work directory to be the same
> > for all multiconfigs?
>
> It depends if the configurations are different or not. WORKDIR isn't
> your problem here, you really need the builds to have the same sysroot-
> components dir for the native case too, else this won't work. Its more
> about the output from the task than the directory they were built in.
>
> > Would each multiconfig honor the previously built native tools?
> > Or, would it just rebuild them, anyway?
> > Is this safe or tested or supported?
>
> multiconfig is configured so that if it sees overlapping stamp files it
> would handle that correctly. The tricky part is you'd have to split
> some directories which aren't usually split such as sysroot-components.
> I've never seen that attempted and am not sure if it would work or not.
>
> > (3) We're considering a network shared sstate-cache.  But it's not
> > entirely clear if a multiconfig build is benefited by the sstate-
> > cache directory.
> > Under what circumstances does a multiconfig leverage sstate-cache
> > when building for another multiconfig?
>
> Right now today for a single multiconfig build it won't.
>
> > What if the recipe would be affected by a change in DISTRO_FEATURES
> > between multiconfigs?
> > Are native packages included in the sstate-cache or just target
> > packages?
>
> native and target are included in sstate-cache.
>
> > We welcome all thoughts and ideas.
>
> With current bitbake you'd be best off splitting your multiconfig build
> into two parts a first stage where you build common shared components
> and a second where you leverage them.
>
> I appreciate that isn't a good solution but its what will work right
> now.
>
> The initial work in adapting bitbake to better handle situations like
> this is being done, as we speak and there are patches starting it in
> master-next.
>
> Right now those patches also won't change anything for you, they re-
> architect the codebase so we have a chance of implementing what you
> want. It is however at least a start. I'm working on this effectively
> alone with the day to day patch review, email and bugs to handle at the
> same time :(. As a project we know this is important but we struggle to
> find people willing to contribute development resources to make these
> kinds of changes happen.
>
> Next step is to write a regression test suite for runqueue. Once I've
> done that I'll start to think about optimisation work.
>
> Cheers,
>
> Richard
>
>
>

-- 
Alan Kozlay
VES Solutions, LLC
Alan Kozlay <alan.kozlay@ves.solutions>
DESK 443-595-7734
CELL 443-752-0323

[-- Attachment #2: Type: text/html, Size: 5260 bytes --]

Re: Should multiconfig save build time?

[richard.purdie]

On Wed, 2019-07-10 at 11:58 -0400, Alan Kozlay wrote:
> Thanks for the feedback. 

Good news is that with the basics of the work done, optimising the
multiconfig case turned out not to be too bad.

I've shared some further patches on the mailing list (also in master-
next) and you could give those a try if you're feeling adventurous.

I've shown they do the right thing in the test I wrote for it. Its had
no real world testing as yet, you're at the edge of development here.

Cheers,

Richard

Re: Should multiconfig save build time?

[Alan Kozlay]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

Cool, thanks!

Alan Kozlay
443-752-0323
Software Engineer
VES Solutions, LLC
Sent from my Android phone

On Thu, Jul 11, 2019, 12:14 PM <richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2019-07-10 at 11:58 -0400, Alan Kozlay wrote:
> > Thanks for the feedback.
>
> Good news is that with the basics of the work done, optimising the
> multiconfig case turned out not to be too bad.
>
> I've shared some further patches on the mailing list (also in master-
> next) and you could give those a try if you're feeling adventurous.
>
> I've shown they do the right thing in the test I wrote for it. Its had
> no real world testing as yet, you're at the edge of development here.
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 1116 bytes --]

Re: Should multiconfig save build time?

[Alan Kozlay]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

Richard,

I was already planning on doing some build time testing, over the weekend.
l plan to add your patches and see how it goes.  I'll let you know what I
find out.

Thanks for working on it.

-Alan

On Fri, Jul 12, 2019 at 9:03 AM Alan Kozlay <alan.kozlay@ves-solutions.com>
wrote:

> Cool, thanks!
>
> Alan Kozlay
> 443-752-0323
> Software Engineer
> VES Solutions, LLC
> Sent from my Android phone
>
> On Thu, Jul 11, 2019, 12:14 PM <richard.purdie@linuxfoundation.org> wrote:
>
>> On Wed, 2019-07-10 at 11:58 -0400, Alan Kozlay wrote:
>> > Thanks for the feedback.
>>
>> Good news is that with the basics of the work done, optimising the
>> multiconfig case turned out not to be too bad.
>>
>> I've shared some further patches on the mailing list (also in master-
>> next) and you could give those a try if you're feeling adventurous.
>>
>> I've shown they do the right thing in the test I wrote for it. Its had
>> no real world testing as yet, you're at the edge of development here.
>>
>> Cheers,
>>
>> Richard
>>
>>

-- 
Alan Kozlay
VES Solutions, LLC
Alan Kozlay <alan.kozlay@ves.solutions>
DESK 443-595-7734
CELL 443-752-0323

[-- Attachment #2: Type: text/html, Size: 2737 bytes --]