Re: [PATCH RFC 2/3] gitlab: build all container images during CI

From: "Daniel P. Berrangé" <berrange@redhat.com>
To: "Philippe Mathieu-Daudé" <philmd@redhat.com>
Cc: "Alex Bennée" <alex.bennee@linaro.org>,
	"Thomas Huth" <thuth@redhat.com>,
	"Laszlo Ersek" <lersek@redhat.com>,
	qemu-devel@nongnu.org,
	"Wainer dos Santos Moschetta" <wainersm@redhat.com>
Subject: Re: [PATCH RFC 2/3] gitlab: build all container images during CI
Date: Mon, 22 Jun 2020 16:46:11 +0100	[thread overview]
Message-ID: <20200622154611.GJ736373@redhat.com> (raw)
In-Reply-To: <e6c27611-2873-2567-e3c8-f10b78831bc9@redhat.com>

On Mon, Jun 22, 2020 at 05:38:16PM +0200, Philippe Mathieu-Daudé wrote:
> On 6/22/20 5:33 PM, Daniel P. Berrangé wrote:
> > We have a number of container images in tests/docker/dockerfiles
> > that are intended to provide well defined environments for doing
> > test builds. We want our CI system to use these containers too.
> > 
> > This introduces builds of all of them as the first stage in the
> > CI, so that the built containers are available for later build
> > jobs. The containers are setup to use the GitLab container
> > registry as the cache, so we only pay the penalty of the full
> > build when the dockerfiles change. The main qemu-project/qemu
> > repo is used as a second cache, so that users forking QEMU will
> > see a fast turnaround time on their CI jobs.
> 
> OMG you did it! Lovely... 😍
> 
> Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
> why gitlab isn't caching the docker images? the cache isn't
> populated on all the runners yet? Or we have to use the same
> runner again to hit its cache?

It is caching it but it isn't obvious what to look for.

The "docker build" command is always still run, but you'll
notice each of the actual commands in the dockerfile are
followed by a message "  ---> Using cache", instead of the
normal command output.

Compare the "amd64-debian9-container" job as an example.

Here is the output seen in the original job which actually did a real
build:

   https://gitlab.com/berrange/qemu/-/jobs/605783351

And here is the output from the pipeline you point to above:

   https://gitlab.com/berrange/qemu/-/jobs/606175855

The cached case is still taking 2 mins 30 seconds, but the original
full build took 4 mins 50 seconds.

The difference will be more noticable if we expand the list of packages
install in the images to cover more of QEMU's possible dependancies.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|