Re: [tpm2] Docker Images

Re: [tpm2] Docker Images

[Matthew Dempsky]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]

On Fri, Aug 30, 2019, 8:27 AM Roberts, William C <
william.c.roberts(a)intel.com> wrote:

> The only thing that still sucks, is I can't seem to find the "# include"
> directive to
> Avoid repeating text blocks.


Yeah, I'm not aware of any sort of include functionality either
unfortunately.

It seems like standard practice here is to just generate Dockerfiles from a
script/template; and then to appease Docker Hub's requirements, commit the
generated files.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 872 bytes --]

Re: [tpm2] Docker Images

[Roberts, William C]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]



> -----Original Message-----
> From: Matthew Dempsky [mailto:matthew(a)dempsky.org]
> Sent: Friday, August 30, 2019 12:01 PM
> To: Roberts, William C <william.c.roberts(a)intel.com>
> Cc: tpm2(a)lists.01.org
> Subject: Re: [tpm2] Docker Images
> 
> On Fri, Aug 30, 2019, 8:27 AM Roberts, William C <william.c.roberts(a)intel.com
> <mailto:william.c.roberts(a)intel.com> > wrote:
> 
> 
> 	The only thing that still sucks, is I can't seem to find the "# include"
> directive to
> 	Avoid repeating text blocks.
> 
> 
> Yeah, I'm not aware of any sort of include functionality either unfortunately.
> 
> It seems like standard practice here is to just generate Dockerfiles from a
> script/template; and then to appease Docker Hub's requirements, commit the
> generated files.

Yeah I figure if maintenance becomes a PITA I'm sure import something in python
will make life easier.

Re: [tpm2] Docker Images

[Roberts, William C]

[-- Attachment #1: Type: text/plain, Size: 2382 bytes --]



> -----Original Message-----
> From: Matthew Dempsky [mailto:matthew(a)dempsky.org]
> Sent: Thursday, August 29, 2019 12:39 PM
> To: Roberts, William C <william.c.roberts(a)intel.com>
> Cc: tpm2(a)lists.01.org
> Subject: Re: [tpm2] Docker Images
> 
> On Thu, Aug 29, 2019 at 9:40 AM Roberts, William C <william.c.roberts(a)intel.com
> <mailto:william.c.roberts(a)intel.com> > wrote:
> 
> 
> 	But Dockerhub knows how to map branch/tag name to a docker image
> name.
> 
> 
> 
> Understood, but I believe this functionality is customizable. For example, see this
> Stack Overflow response:

Oh I see what you mean, I didn't even notice that it had the "Dockerfile location"

The only thing that still sucks, is I can't seem to find the "# include" directive to
Avoid repeating text blocks.

> 
> 
> 	https://stackoverflow.com/a/44796846
> 	https://i.stack.imgur.com/9GpzT.png
> 
> 
> What I understand is you're trying to use the Docker Hub's built-in Git Branch =>
> Docker Tag mapping functionality.
> 
> I'm suggesting to instead configure multiple builds manually; e.g.,
> 
>     - Type "Branch", Name "master", Dockerfile Location "/ubuntu", Docker Tag
> Name "ubuntu"
>     - Type "Branch", Name "master", Dockerfile Location "/fedora", Docker Tag
> Name "fedora"
>     Etc.
> 
> Admittedly, this has the downside that to add/remove OSes, you'll need to
> update the Docker Hub configuration. But this is easily documented.
> 
> 
> 	And do to constraints I cannot control, I need to use dockerhub to build
> the images...
> 
> 
> 
> To clarify, I'm suggesting an alternative way to configure Docker Hub. I'm not
> suggesting to stop using Docker Hub.
> 
> 
> 	The other thing is that each distro is likely going to differences to the
> docker file anyways. Setup on
> 	Ubuntu will likely not be the same as fedora. Right now the updates are
> infrequent enough where
> 	It shouldn't really be that much of a burden, hopefully.
> 
> 
> I'm sure you have a better sense of the effort involved than I do, and you're
> going to be the one responsible for that work.
> 
> 
> I'm just cautioning that from personal experience with projects I've worked on
> (e.g., Android, Chrome, Chrome OS), the "apply N patches to N repos/branches"
> model is workable but very tedious, error prone, and generally dispiriting.

Re: [tpm2] Docker Images

[Matthew Dempsky]

[-- Attachment #1: Type: text/plain, Size: 1759 bytes --]

On Thu, Aug 29, 2019 at 9:40 AM Roberts, William C <
william.c.roberts(a)intel.com> wrote:

> But Dockerhub knows how to map branch/tag name to a docker image name.
>

Understood, but I believe this functionality is customizable. For example,
see this Stack Overflow response:

https://stackoverflow.com/a/44796846
https://i.stack.imgur.com/9GpzT.png


What I understand is you're trying to use the Docker Hub's built-in Git
Branch => Docker Tag mapping functionality.

I'm suggesting to instead configure multiple builds manually; e.g.,

    - Type "Branch", Name "master", Dockerfile Location "/ubuntu", Docker
Tag Name "ubuntu"
    - Type "Branch", Name "master", Dockerfile Location "/fedora", Docker
Tag Name "fedora"
    Etc.

Admittedly, this has the downside that to add/remove OSes, you'll need to
update the Docker Hub configuration. But this is easily documented.

And do to constraints I cannot control, I need to use dockerhub to build
> the images...
>

To clarify, I'm suggesting an alternative way to configure Docker Hub. I'm
not suggesting to stop using Docker Hub.

The other thing is that each distro is likely going to differences to the
> docker file anyways. Setup on
> Ubuntu will likely not be the same as fedora. Right now the updates are
> infrequent enough where
> It shouldn't really be that much of a burden, hopefully.


I'm sure you have a better sense of the effort involved than I do, and
you're going to be the one responsible for that work.

I'm just cautioning that from personal experience with projects I've worked
on (e.g., Android, Chrome, Chrome OS), the "apply N patches to N
repos/branches" model is workable but very tedious, error prone, and
generally dispiriting.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2939 bytes --]

Re: [tpm2] Docker Images

[Roberts, William C]

[-- Attachment #1: Type: text/plain, Size: 2180 bytes --]



> -----Original Message-----
> From: Matthew Dempsky [mailto:matthew(a)dempsky.org]
> Sent: Wednesday, August 28, 2019 6:36 PM
> To: Roberts, William C <william.c.roberts(a)intel.com>
> Cc: tpm2(a)lists.01.org
> Subject: Re: [tpm2] Docker Images
> 
> On Wed, Aug 28, 2019 at 1:19 PM Roberts, William C <william.c.roberts(a)intel.com
> <mailto:william.c.roberts(a)intel.com> > wrote:
> 
> 
> 	If we want to utilize dockerhub to build multiple images and extend our
> build coverage,
> 
> 
> 	This seems limited. Dockerhub has the ability to follow any branch and
> associate a tag.
> 	I'm thinking that we create branches for each base image, like:
> 	  - branch: ubuntu-16.04
> 	  - ubuntu-18.04
> 	  - fedora-29
> 	  - fedora-30
> 	  - opensuse-tumbleweed
> 	  - opensuse-leap
> 	  - etc
> 
> 
> 
> If I understand you correctly, you're suggesting to create a separate Git branch
> for each supported OS, and each Git branch has a single Dockerfile?
> 
> That seems like it might lead to a lot of extra busy work. E.g., if you want to add a
> new dependency, you'll have to do N commits touching N files across N branches.
> 
> I'd think you could configure Docker Hub to build multiple Dockerfiles (e.g.,
> stored within separate subdirectories, or "build contexts" in Docker lingo?
> <https://docs.docker.com/docker-hub/builds/#set-the-build-context-and-
> dockerfile-location> ) within a single branch. Then if you need to make sweeping
> updates, that's only 1 commit touching N files across 1 branch.

Yep that is the downside. But Dockerhub knows how to map branch/tag name to a docker image name.
And do to constraints I cannot control, I need to use dockerhub to build the images...
The other thing is that each distro is likely going to differences to the docker file anyways. Setup on
Ubuntu will likely not be the same as fedora. Right now the updates are infrequent enough where
It shouldn't really be that much of a burden, hopefully.

I really see us limiting us to like 6 images.

> 
> 
> 	Thoughts, ideas objections are welcome. I am **not** a Docker expert.
> 
> 
> 
> Caveat: I'm certainly not either.

Re: [tpm2] Docker Images

[Matthew Dempsky]

[-- Attachment #1: Type: text/plain, Size: 1308 bytes --]

On Wed, Aug 28, 2019 at 1:19 PM Roberts, William C <
william.c.roberts(a)intel.com> wrote:

> If we want to utilize dockerhub to build multiple images and extend our
> build coverage,
>
This seems limited. Dockerhub has the ability to follow any branch and
> associate a tag.
> I'm thinking that we create branches for each base image, like:
>   - branch: ubuntu-16.04
>   - ubuntu-18.04
>   - fedora-29
>   - fedora-30
>   - opensuse-tumbleweed
>   - opensuse-leap
>   - etc
>

If I understand you correctly, you're suggesting to create a separate Git
branch for each supported OS, and each Git branch has a single Dockerfile?

That seems like it might lead to a lot of extra busy work. E.g., if you
want to add a new dependency, you'll have to do N commits touching N files
across N branches.

I'd think you could configure Docker Hub to build multiple Dockerfiles
(e.g., stored within separate subdirectories, or "build contexts" in Docker
lingo?
<https://docs.docker.com/docker-hub/builds/#set-the-build-context-and-dockerfile-location>)
within a single branch. Then if you need to make sweeping updates, that's
only 1 commit touching N files across 1 branch.

Thoughts, ideas objections are welcome. I am **not** a Docker expert.
>

Caveat: I'm certainly not either.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 1982 bytes --]

Re: [tpm2] Docker Images

[Roberts, William C]

[-- Attachment #1: Type: text/plain, Size: 1814 bytes --]



> -----Original Message-----
> From: Andersen, John S
> Sent: Wednesday, August 28, 2019 4:05 PM
> To: Roberts, William C <william.c.roberts(a)intel.com>
> Cc: tpm2(a)lists.01.org
> Subject: Re: [tpm2] Docker Images
> 
> On Wed, Aug 28, 2019 at 08:19:29PM +0000, Roberts, William C wrote:
> > Currently the
> > https://github.com/tpm2-software/tpm2-software-container/pulls
> > Docker build file has one branch and one Dockerfile. The Dockerhub is
> > configured To build the contents of branch master and tag as the latest.
> >
> > If we want to utilize dockerhub to build multiple images and extend
> > our build coverage, This seems limited. Dockerhub has the ability to follow any
> branch and associate a tag.
> > I'm thinking that we create branches for each base image, like:
> >   - branch: ubuntu-16.04
> >   - ubuntu-18.04
> >   - fedora-29
> >   - fedora-30
> >   - opensuse-tumbleweed
> >   - opensuse-leap
> >   - etc
> >
> > Those branches can be mapped in the Docker build UI to build a similarly named
> file.
> > The base image that our dockerfile extends is the image in the tag
> > name. Ie branch ubuntu-18.04's Dockerfile starts with FROM ubuntu:18.04.
> >
> > And then in a travis build file, just specify the matrix of images to test against.
> >
> > Thoughts, ideas objections are welcome. I am **not** a Docker expert.
> >
> > Thanks,
> > Bill
> >
> 
> Sounds like a great plan to me.

Another nice part is that I can "extend" the base image and have another layer
In a branch called ubuntu-18.04-tools that would be built with a docker file that
satisfies the TSS and ABRMD dependencies (tied to a stable release of them).

Probably will be a tad bit faster than cloning and building on each travis run. At least
Simplify our CI scripts.

Re: [tpm2] Docker Images

[Andersen, John]

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

On Wed, Aug 28, 2019 at 08:19:29PM +0000, Roberts, William C wrote:
> Currently the https://github.com/tpm2-software/tpm2-software-container/pulls 
> Docker build file has one branch and one Dockerfile. The Dockerhub is configured
> To build the contents of branch master and tag as the latest.
> 
> If we want to utilize dockerhub to build multiple images and extend our build coverage,
> This seems limited. Dockerhub has the ability to follow any branch and associate a tag.
> I'm thinking that we create branches for each base image, like:
>   - branch: ubuntu-16.04
>   - ubuntu-18.04 
>   - fedora-29
>   - fedora-30
>   - opensuse-tumbleweed
>   - opensuse-leap
>   - etc
> 
> Those branches can be mapped in the Docker build UI to build a similarly named file.
> The base image that our dockerfile extends is the image in the tag name. Ie branch
> ubuntu-18.04's Dockerfile starts with FROM ubuntu:18.04.
> 
> And then in a travis build file, just specify the matrix of images to test against.
> 
> Thoughts, ideas objections are welcome. I am **not** a Docker expert. 
> 
> Thanks,
> Bill
> 

Sounds like a great plan to me.

Thanks,
John

[tpm2] Docker Images

[Roberts, William C]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

Currently the https://github.com/tpm2-software/tpm2-software-container/pulls 
Docker build file has one branch and one Dockerfile. The Dockerhub is configured
To build the contents of branch master and tag as the latest.

If we want to utilize dockerhub to build multiple images and extend our build coverage,
This seems limited. Dockerhub has the ability to follow any branch and associate a tag.
I'm thinking that we create branches for each base image, like:
  - branch: ubuntu-16.04
  - ubuntu-18.04 
  - fedora-29
  - fedora-30
  - opensuse-tumbleweed
  - opensuse-leap
  - etc

Those branches can be mapped in the Docker build UI to build a similarly named file.
The base image that our dockerfile extends is the image in the tag name. Ie branch
ubuntu-18.04's Dockerfile starts with FROM ubuntu:18.04.

And then in a travis build file, just specify the matrix of images to test against.

Thoughts, ideas objections are welcome. I am **not** a Docker expert. 

Thanks,
Bill