Incorporating deploy artefacts from one multiconfig in another multiconfig

Incorporating deploy artefacts from one multiconfig in another multiconfig

[Burton, Ross]

So I'd like to use multiconfig for a fairly typical host/container use
case.  I've a class which can be used to build packages from images in
a fairly simple manner, essential just taking files from DEPLOY_DIR
when creating a package:

core-image-minimal-package.bb:
do_install[depends] += "core-image-minimal:do_image_complete
virtual/kernel:do_deploy"
do_install () {
    install -d ${D}/var/lib/machines/
    install ${DEPLOY_DIR_IMAGE}/core-image-minimal-${MACHINE}.ext4
${D}/var/lib/machines/core-image-minimal.ext4
    install ${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}
${D}/var/lib/machines/${IMAGE_NAME}.${KERNEL_IMAGETYPE}
}

This part works fine, but the problem is straddling multiconfig boundaries...

I can't just add "core-image-minimal-package" to IMAGE_INSTALL because
it will build that for the current configuration.  I can't just add
"multiconfig:uos:core-image-minimal-package" because that will look
for a recipe with that name.  I've a rather ugly hack to call opkg
directly in a post-rootfs hook which sort of works.  Linking in bits
of the other configuration's deploy directory in a pre-rootfs hook and
extending IMAGE_INSTALL there almost works, apart from the rootfs code
then looks at the task dependencies to generate a pruned deploy dir,
which is missing the pieces I linked in.

I suspect this is now breaking new ground: the multiconfiguration
support lets you do builds but so far the artefact sharing hasn't
really been explored.  Has anyone dug further into this?

Ross

Re: [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig

[Mark Hatle]

On 2/21/19 9:44 AM, Burton, Ross wrote:
> So I'd like to use multiconfig for a fairly typical host/container use
> case.  I've a class which can be used to build packages from images in
> a fairly simple manner, essential just taking files from DEPLOY_DIR
> when creating a package:
> 
> core-image-minimal-package.bb:
> do_install[depends] += "core-image-minimal:do_image_complete
> virtual/kernel:do_deploy"
> do_install () {
>     install -d ${D}/var/lib/machines/
>     install ${DEPLOY_DIR_IMAGE}/core-image-minimal-${MACHINE}.ext4
> ${D}/var/lib/machines/core-image-minimal.ext4
>     install ${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE}
> ${D}/var/lib/machines/${IMAGE_NAME}.${KERNEL_IMAGETYPE}
> }
> 
> This part works fine, but the problem is straddling multiconfig boundaries...
> 
> I can't just add "core-image-minimal-package" to IMAGE_INSTALL because
> it will build that for the current configuration.  I can't just add
> "multiconfig:uos:core-image-minimal-package" because that will look
> for a recipe with that name.  I've a rather ugly hack to call opkg
> directly in a post-rootfs hook which sort of works.  Linking in bits
> of the other configuration's deploy directory in a pre-rootfs hook and
> extending IMAGE_INSTALL there almost works, apart from the rootfs code
> then looks at the task dependencies to generate a pruned deploy dir,
> which is missing the pieces I linked in.
> 
> I suspect this is now breaking new ground: the multiconfiguration
> support lets you do builds but so far the artefact sharing hasn't
> really been explored.  Has anyone dug further into this?

I don't think anyone has.  But as we continue moving toward containers/micro
services environments -- this exactly type of thing is going to be more and more
needed.

One 'build', with various configurations that end up constructing a single
device image with a container runtime (booting OS), and a number of container
micro services should be able to be created with a single call to bitbake and (a
potentially complex) config.

I think the first question is, can one multiconfig require a dependency out of
another yet?  I thought this was still unimplemented.  Once this can be done,
then it should be much easier to have a 'configuration' whose job it is to
assemble the various components while waiting for everything to finish being built.

--Mark

> Ross
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
> 

Re: [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig

[Richard Purdie]

Multiconfig is meant to support this workflow. Unfortunately there are
open bugs and people haven't the time to work on it so its stalled.
That said, the key pieces are already there.

A picture is worth a thousand words. I thought a demo might interest
people and "prove" this can work.

To make it work you need four patches:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=b896b2238c100d486e7c43992b49001150749d04

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=e5ef6b008e85f42b5e4ffd72fae7036dd68b2ea5

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=5d2bd43d14315ef3212e0d09a16446bd74305a2f

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=abdb83576b699448b76293294809aaa04b8e853b

The first two are fixes from Alejandro which haven't merged as they're
not right. The third patch was me hitting the fixes and bitbake to make
the demo work. That proves they're not right and hints at where we have
some problems in bitbake too. We can get those issues sorted fairly
easily and I'm going to block 2.7 M3 on them.

The final patch is the interesting one.

It adds a layer, meta-multiconfig-example layer with a README about how
to configure it. Basically you create two multiconfig configurations,
"musl" and "tiny".

The demo is to run "bitbake core-image-full-cmdline".

The result is a core-image-full-cmdline image with a tiny qemux86 image
and a musl based qemux86-64 image installed into /var/lib/machines/

The way it does this is creative, having a recipe called image-packer
which creates virtclass variants of itself which depend on the
appropriate multiconfig. It pulls in the images and creates a package
containing them which the bbappend to core-image-full-cmdline can then
install. It'd be very easy to turn this into a list driven piece of
code rather than the two values currently listed.

Its late here, some of the above code is horrible. I have rushed this
but hopefully it shows this is possible! :)

I've love to turn this into an oe-selftest, that is my ultimate 
intention for this code.

As things stand today you can't query one multiconfig from another. As
this demo shows, that doesn't need to stop you making interesting
things though. We will do something with such querying eventually to
make things easier but until then, much is still possible.

Cheers,

Richard

Re: [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig

[Richard Purdie]

On Thu, 2019-02-21 at 20:09 -0500, Rich Persaud wrote:
> > On Feb 21, 2019, at 18:51, Richard Purdie <
> > richard.purdie@linuxfoundation.org> wrote:
> > 
> > Multiconfig is meant to support this workflow. Unfortunately there
> > are
> > open bugs and people haven't the time to work on it so its stalled.
> > That said, the key pieces are already there.
> > 
> > A picture is worth a thousand words. I thought a demo might
> > interest
> > people and "prove" this can work.
> 
> Thank you for the patches and PoC.  We will test and are motivated to
> see this work.

I've been thinking about what Alejandro and I discussed and I think the
correct fix for the multiconfig code is to filter the BB_TASKDEPDATA at
source in bitbake to remove the multiconfig entries for other
multiconfigs. I'll try and work up a patch for that in the next day or
two as the hacks on my branch that do this are fragile right now.

> I will also be experimenting with the generation of "immutable
> infrastructure" from source code, to learn whether a single Packer.io
> template can be used to both:
> 
> (a) modify binary images, in the traditional Packer model, e.g. start
> a bitbake-generated image in a VM or container, make runtime
> configuration changes, then repack the machine image
> 
> (b) perform automated customization  of _inputs_ to bitbake recipes
> and package source code, such that a machine image built by bitbake +
> Packer template is equivalent to the image generated by Packer at
> runtime
> 
> If feasible, this would combine the agility of image customizations
> with the security of bitbake's reproducible builds and supply chain
> integrity.

(a) definitely sounds easier than (b). I've not used packer.io but I'd
be curious to see how that works out. People do find our image
customisation hard to get to grips with :(

Cheers,

Richard

Re: [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig

[Burton, Ross]

On Thu, 21 Mar 2019 at 06:48, Rich Persaud <persaur@gmail.com> wrote:
> Is it possible to avoid duplicate builds of packages which are used in multiple VM images? Could multiconfig replace a dozen serial instances of bitbake with a single parsing instance, or would it use a dozen parallel instances?  If the latter, is this a constraint in bitbake architecture, recipe DSL, or time/resources?

If the packages are the same and sstate cache would be used, then you
can build one on its own first to populate the cache and then build
the rest using the sstate.  Currently if the sstate isn't present at
startup bitbake schedules the same build twice, which obviously isn't
ideal but also *should* be fixable with enough effort.

Ross

Re: [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig

[Richard Purdie]

On Thu, 2019-02-21 at 20:09 -0500, Rich Persaud wrote:
> > On Feb 21, 2019, at 18:51, Richard Purdie <
> > richard.purdie@linuxfoundation.org> wrote:
> > 
> > Multiconfig is meant to support this workflow. Unfortunately there
> > are
> > open bugs and people haven't the time to work on it so its stalled.
> > That said, the key pieces are already there.
> > 
> > A picture is worth a thousand words. I thought a demo might
> > interest
> > people and "prove" this can work.
> 
> Thank you for the patches and PoC.  We will test and are motivated to
> see this work.

Just to follow up, I've cleaned up the patches, they're now just fixes
to bitbake and are done generically so we shouldn't have to patch the
various BB_TASKDEPDATA users. They're on the bitbake-devel list and
queued in master-next.

This should mean multiconfig looks better in 2.7. I'm going to try and
get a patch to add some tests/example too.

Could anyone who's previously reported issues (Armin?) retest with
these and confirm the issues seen have been fixed please?

Cheers,

Richard

Re: [Openembedded-architecture] Incorporating deploy artefacts from one multiconfig in another multiconfig

[Rich Persaud]

[-- Attachment #1: Type: text/plain, Size: 3900 bytes --]

On Feb 21, 2019, at 18:51, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> 
> Multiconfig is meant to support this workflow. Unfortunately there are
> open bugs and people haven't the time to work on it so its stalled.
> That said, the key pieces are already there.
> 
> A picture is worth a thousand words. I thought a demo might interest
> people and "prove" this can work.
> 
> To make it work you need four patches:
> 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=b896b2238c100d486e7c43992b49001150749d04
> 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=e5ef6b008e85f42b5e4ffd72fae7036dd68b2ea5
> 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=5d2bd43d14315ef3212e0d09a16446bd74305a2f
> 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=abdb83576b699448b76293294809aaa04b8e853b
> 
> The first two are fixes from Alejandro which haven't merged as they're
> not right. The third patch was me hitting the fixes and bitbake to make
> the demo work. That proves they're not right and hints at where we have
> some problems in bitbake too. We can get those issues sorted fairly
> easily and I'm going to block 2.7 M3 on them.
> 
> The final patch is the interesting one.
> 
> It adds a layer, meta-multiconfig-example layer with a README about how
> to configure it. Basically you create two multiconfig configurations,
> "musl" and "tiny".
> 
> The demo is to run "bitbake core-image-full-cmdline".
> 
> The result is a core-image-full-cmdline image with a tiny qemux86 image
> and a musl based qemux86-64 image installed into /var/lib/machines/
> 
> The way it does this is creative, having a recipe called image-packer
> which creates virtclass variants of itself which depend on the
> appropriate multiconfig. It pulls in the images and creates a package
> containing them which the bbappend to core-image-full-cmdline can then
> install. It'd be very easy to turn this into a list driven piece of
> code rather than the two values currently listed.
> 
> Its late here, some of the above code is horrible. I have rushed this
> but hopefully it shows this is possible! :)
> 
> I've love to turn this into an oe-selftest, that is my ultimate 
> intention for this code.
> 
> As things stand today you can't query one multiconfig from another. As
> this demo shows, that doesn't need to stop you making interesting
> things though. We will do something with such querying eventually to
> make things easier but until then, much is still possible.

For search engines: the multiconfig demo from this thread is now is an oe-selftest:

https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta-selftest/recipes-test/multiconfig/multiconfig-image-packager_0.1.bb
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/cases/multiconfig.py

Are there other public examples of combining artifacts from different machines into a single image via multiconfig?  We can include them in a doc update for Dunfell.  A list of use cases could motivate more people to try multiconfig.

For developer workflows, multiconfig can accelerate multi-image builds (more parallelism) but slow single-package builds (more recipe parsing).  As a workaround, BBMULTICONFIG="" restores the speed of building a target for a single MACHINE+DISTRO config.

Is there a better way to disable multiconfig for a single bitbake invocation?  Bitbake currently requires a MACHINE to be specified for multiconfig builds.  It also accepts non-multiconfig target names, even when BBMULTICONFIG is defined.  From a usability point of view, this CLI syntax could distinguish the two workflows:

  standard build:  MACHINE=foo bitbake singleconfig-image  
  multiconfig build:  bitbake mc::multiconfig-image

Rich

[-- Attachment #2: Type: text/html, Size: 7311 bytes --]