Multiple MACHINEs in a single build?

Multiple MACHINEs in a single build?

[Marcelo E. Magallon]

Hi,

I suspect this is documented in a FAQ somewhere, but I'm failing to hit
the correct keywords when googling...

Is there a way to support building images for multiple MACHINEs in a
single workspace at the same time?

This works:

	$ MACHINE=qemux86 bitbake core-image-minimal
	$ MACHINE=qemux86-64 bitbake core-image-minimal

and bitbake parallelizes the build across tasks. What I'm trying to
figure out is if it's possible to run two bitbake instances with
different MACHINE values on different hosts but writing to the same
workspace (say, over NFS).

Thanks!

Marcelo

Re: Multiple MACHINEs in a single build?

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 929 bytes --]


> On May 4, 2016, at 7:08 PM, Marcelo E. Magallon <marcelo.magallon@hpe.com> wrote:
> 
> Hi,
> 
> I suspect this is documented in a FAQ somewhere, but I'm failing to hit
> the correct keywords when googling...
> 
> Is there a way to support building images for multiple MACHINEs in a
> single workspace at the same time?
> 
> This works:
> 
> 	$ MACHINE=qemux86 bitbake core-image-minimal
> 	$ MACHINE=qemux86-64 bitbake core-image-minimal
> 
> and bitbake parallelizes the build across tasks. What I'm trying to
> figure out is if it's possible to run two bitbake instances with
> different MACHINE values on different hosts but writing to the same
> workspace (say, over NFS).

Thats not possible and its by design.

> 
> Thanks!
> 
> Marcelo
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: Multiple MACHINEs in a single build?

[Paul Eggleton]

Hi Marcelo,

On Wed, 04 May 2016 20:08:42 Marcelo E. Magallon wrote:
> I suspect this is documented in a FAQ somewhere, but I'm failing to hit
> the correct keywords when googling...
> 
> Is there a way to support building images for multiple MACHINEs in a
> single workspace at the same time?
> 
> This works:
> 
> 	$ MACHINE=qemux86 bitbake core-image-minimal
> 	$ MACHINE=qemux86-64 bitbake core-image-minimal
> 
> and bitbake parallelizes the build across tasks. What I'm trying to
> figure out is if it's possible to run two bitbake instances with
> different MACHINE values on different hosts but writing to the same
> workspace (say, over NFS).

You can do builds for two different machines *sequentially*, reusing the same 
TMPDIR (or alternatively a different TMPDIR using the same sstate-cache or a 
mirror thereof) - tasks that were completed in the first build that don't need 
to be run differently for the second machine won't be re-run in the second 
build, so there is some level of efficiency to be had.

You cannot really run these builds concurrently though (you definitely can't 
sharing the same TMPDIR). However, as part of the plan for 2.2 we will be 
looking into how we can do this.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Multiple MACHINEs in a single build?

[Marcelo E. Magallon]

On Thu, May 05, 2016 at 06:45:36AM -0700, Khem Raj wrote:

> > This works:
> > 
> > 	$ MACHINE=qemux86 bitbake core-image-minimal
> > 	$ MACHINE=qemux86-64 bitbake core-image-minimal
> > 
> > and bitbake parallelizes the build across tasks. What I'm trying to
> > figure out is if it's possible to run two bitbake instances with
> > different MACHINE values on different hosts but writing to the same
> > workspace (say, over NFS).
> 
> Thats not possible and its by design.

Can you nudge me in the direction of some documentation for this design?

Thanks!

Marcelo

Re: Multiple MACHINEs in a single build?

[Khem Raj]

On Thu, May 5, 2016 at 12:20 PM, Marcelo E. Magallon
<marcelo.magallon@hpe.com> wrote:
> On Thu, May 05, 2016 at 06:45:36AM -0700, Khem Raj wrote:
>
>> > This works:
>> >
>> >     $ MACHINE=qemux86 bitbake core-image-minimal
>> >     $ MACHINE=qemux86-64 bitbake core-image-minimal
>> >
>> > and bitbake parallelizes the build across tasks. What I'm trying to
>> > figure out is if it's possible to run two bitbake instances with
>> > different MACHINE values on different hosts but writing to the same
>> > workspace (say, over NFS).
>>
>> Thats not possible and its by design.
>
> Can you nudge me in the direction of some documentation for this design?

It was not a conscious choice to limit however that is what it started with
single machine singe distro in one invocation. It has been raised in
past and I was
one of the persons who raised it as a limitation if you want to call it.

However in your case you said NFS and we have seen problems with NFS even with
single machine builds in past so just be aware. Secondly during build
there are native
packages built and staged and every machine will build them similarly
there are all-*-*
packages which are going to conflict if two bitbake instances did not
synchronise, they
might conflict with each other, so unless we change the logic to let
bitbake take machine
as parameter and setup global datastore accordingly e.g.

bitbake machine1,<image1, images2,.,,, machine2,<imageA, imageB....>

it might not work.

>
> Thanks!
>
> Marcelo