Importing Docker Images

Importing Docker Images

[Nathan Dunne]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]

Hi,

I am looking for some community feedback on whether it would make sense to add support in meta-virtualization for including custom docker container image tarballs. These container image tarballs would have been generated manually by using a "docker export" command.

Currently in meta-arm-autonomy we have:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/tree/meta-arm-autonomy/recipes-containers/import-docker-image
Which allows a single image file to be included, then imported at boot time. I am currently extending this to allow a list of image files rather than only one, and to use docker-compose for easier control of multiple containers. This could live in meta-virtualization rather than meta-arm-autonomy if the community believes this a good idea.

Let me know what you think,
Nathan

[-- Attachment #2: Type: text/html, Size: 1330 bytes --]

Re: [meta-virtualization] Importing Docker Images

[Bruce Ashfield]

On Tue, Feb 9, 2021 at 11:22 AM Nathan Dunne <Nathan.Dunne@arm.com> wrote:
>
> Hi,
>
> I am looking for some community feedback on whether it would make sense to add support in meta-virtualization for including custom docker container image tarballs. These container image tarballs would have been generated manually by using a "docker export" command.

This is something we've talked about multiple times on the mailing
list and at various conferences, and it really doesn't fit properly
into the workflow.

We run into licensing, reproducibility, development and compatibility
issues very quickly.

I am working on extending the image-oci bbclass to use natively (OE)
based images and pre-populate them to be runnable via
podman/docker/runc at boot. That way, we have an integrated solution
that leverages all of the OE advantages.

(podman is actually easier than docker to do this on the build/cross side)

>
> Currently in meta-arm-autonomy we have:
> https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/tree/meta-arm-autonomy/recipes-containers/import-docker-image
> Which allows a single image file to be included, then imported at boot time. I am currently extending this to allow a list of image files rather than only one, and to use docker-compose for easier control of multiple containers. This could live in meta-virtualization rather than meta-arm-autonomy if the community believes this a good idea.

There are some points of collaboration, but as it currently sits (and
is designed), there are the issues that I mention above that are a
problem, before we'd want to endorse it as a supported / suggestion
option.

I'll work on getting my WIP image class pushed to master-next as soon
as possible, but I have some other pending issues to resolve first.

Bruce

>
> Let me know what you think,
> Nathan
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] Importing Docker Images

[Robert Berger]

Hi,

My comments are in-line.


On 09/02/2021 18:55, Bruce Ashfield wrote:
> 
> I am working on extending the image-oci bbclass to use natively (OE)
> based images and pre-populate them to be runnable via
> podman/docker/runc at boot. That way, we have an integrated solution
> that leverages all of the OE advantages.

With multiconfig this can already be done for quite some time. Startup 
scripts and other config needs to be done in the recipe manually with 
what I have, but I guess it's anyhow hard to automate this.

> 
> (podman is actually easier than docker to do this on the build/cross side)
> 
>>
>> Currently in meta-arm-autonomy we have:
>> https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/tree/meta-arm-autonomy/recipes-containers/import-docker-image
>> Which allows a single image file to be included, then imported at boot time. I am currently extending this to allow a list of image files rather than only one, and to use docker-compose for easier control of multiple containers. This could live in meta-virtualization rather than meta-arm-autonomy if the community believes this a good idea.
> 
> 
> I'll work on getting my WIP image class pushed to master-next as soon
> as possible, but I have some other pending issues to resolve first.

Happy to work with you on that as well.

If we really want "container images as binaries" it's as easy as 
creating them with meta-virtualization and pushing them to some registry 
e.g. docker.io and pulling them from the target. With all the well known 
licensing and reproducebility issues.

> 
> Bruce
> 
>>
>> Let me know what you think,
>> Nathan

Regards,

Robert

>>
>>
>>
> 
> 
> 
> 
> 
> 

Re: Importing Docker Images

[Diego Sueiro]

On Tue, Feb  9, 2021 at 04:55 PM, Bruce Ashfield wrote:

>
> On Tue, Feb 9, 2021 at 11:22 AM Nathan Dunne <Nathan.Dunne@arm.com> wrote:
> >
> > Hi,
> >
> > I am looking for some community feedback on whether it would make sense to
> add support in meta-virtualization for including custom docker container image
> tarballs. These container image tarballs would have been generated manually by
> using a "docker export" command.
> 
> This is something we've talked about multiple times on the mailing
> list and at various conferences, and it really doesn't fit properly
> into the workflow.
> 
> We run into licensing, reproducibility, development and compatibility
> issues very quickly.

These are very strong characteristics that make our current solution fragile.

> 
> I am working on extending the image-oci bbclass to use natively (OE)
> based images and pre-populate them to be runnable via
> podman/docker/runc at boot. That way, we have an integrated solution
> that leverages all of the OE advantages.
> 
> (podman is actually easier than docker to do this on the build/cross side)
> 
> >
> > Currently in meta-arm-autonomy we have:
> >
> https://git.yoctoproject.org/cgit/cgit.cgi/meta-arm/tree/meta-arm-autonomy/recipes-containers/import-docker-image
> > Which allows a single image file to be included, then imported at boot time.
> I am currently extending this to allow a list of image files rather than only
> one, and to use docker-compose for easier control of multiple containers. This
> could live in meta-virtualization rather than meta-arm-autonomy if the
> community believes this a good idea.
> 
> There are some points of collaboration, but as it currently sits (and
> is designed), there are the issues that I mention above that are a
> problem, before we'd want to endorse it as a supported / suggestion
> option.
> 

Our main motivation with this recipe and following improvements described by
Nathan is that most of our users are interested in including their externally generated
workloads, as docker container images, in Xen guests filesystems. Some of these
workloads are based in Autoware which can't be built with bitbake at the moment and
we also don't want to require the user to be experienced with bitbake/oe in order to
have its workload integrated.

If your implementation supports pulling images from any user-defined image hub I
think we can collaborate on both implementation and testing.

> I'll work on getting my WIP image class pushed to master-next as soon
> as possible, but I have some other pending issues to resolve first.
> 
> Bruce
> 
> >
> > Let me know what you think,
> > Nathan
> >
> > 
> >
> 
> 
> -- 
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>