[PATCH RFC 0/3] gitlab: build containers to use in build jobs

[PATCH RFC 0/3] gitlab: build containers to use in build jobs

[Daniel P. Berrangé]

The current gitlab CI jobs are quite inefficient because they
use the generic distro images and then apt-get/dnf install
extra packages every time.

The other downside is that the container environment used is
only defined in thte .gitlab-ci.yml file, so it tedious to
reproduce locally.

We already have containers defined in tests/docker for use by
developers building locally. We can use these for CI systems
too if we just had a way to build them....

...GitLab CI offers such a way. We can use docker-in-docker
to build the images at the start of the CI cycle, and use
the built images in later jobs.

These later jobs are now faster because they're not having
to install any software.

There is a performance hit to build the images, however, they
are cached in the GitLab docker registry. A user forking the
repo will use the cache from qemu-project/qemu and their own
local fork. So overall the time penalty to build images will
only be seen if the user modifiers tests/docker/dockerfiles
in their patch series, or if the base image changes.

The GitLab container registry is public, so we're not limited
to using the built images in GitLab CI. Any other CI system
that uses docker can consume these images. Similarly we could
change the tests/docker/Makefile  rules so that developers
pull from https://gitlab.com/qemu-project/qemu, instead of
building images locally. This is why we're building all the
images, instead of just the handful the current jobs want.

The interesting stuff is mostly in patch 2.

Patch 3 is a quick hack I did to convert existing jobs to use
the newly built images. I know there's other work being done
on the GitLab jobs right now that probably conflicts with this
though, so I've not invested much time in patch 3. Consider it
more an illustration of what's possible, than a finished proposal
for merge. The patch 3 diff is fairly unintelligble, so it is
easier to look at the final result:

  https://gitlab.com/berrange/qemu/-/blob/ci-cont/.gitlab-ci.yml

An example pipeline can be viewed here:

  https://gitlab.com/berrange/qemu/-/pipelines/158824359

The cached images are here:

  https://gitlab.com/berrange/qemu/container_registry

Daniel P. Berrangé (3):
  gitlab: introduce explicit "container" and "build" stages
  gitlab: build all container images during CI
  gitlab: convert jobs to use custom built containers

 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.d/edk2.yml       |   3 +-
 .gitlab-ci.d/opensbi.yml    |   3 +-
 .gitlab-ci.yml              | 187 +++++++++++++--------------
 4 files changed, 340 insertions(+), 101 deletions(-)
 create mode 100644 .gitlab-ci.d/containers.yml

-- 
2.24.1

[PATCH RFC 1/3] gitlab: introduce explicit "container" and "build" stages

[Daniel P. Berrangé]

If no stage is listed, jobs get put in an implicit "test" stage.
Some jobs which create container images to be used by later stages
are currently listed as in a "build" stages.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/edk2.yml    |  3 ++-
 .gitlab-ci.d/opensbi.yml |  3 ++-
 .gitlab-ci.yml           | 11 +++++++++++
 3 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
index 088ba4b43a..d4e7dfcba6 100644
--- a/.gitlab-ci.d/edk2.yml
+++ b/.gitlab-ci.d/edk2.yml
@@ -1,5 +1,5 @@
 docker-edk2:
- stage: build
+ stage: containers
  rules: # Only run this job when the Dockerfile is modified
  - changes:
    - .gitlab-ci-edk2.yml
@@ -24,6 +24,7 @@ docker-edk2:
  - docker push $IMAGE_TAG
 
 build-edk2:
+ stage: build
  rules: # Only run this job when ...
  - changes: # ... roms/edk2/ is modified (submodule updated)
    - roms/edk2/*
diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
index dd051c0124..ec1c1f4cab 100644
--- a/.gitlab-ci.d/opensbi.yml
+++ b/.gitlab-ci.d/opensbi.yml
@@ -1,5 +1,5 @@
 docker-opensbi:
- stage: build
+ stage: containers
  rules: # Only run this job when the Dockerfile is modified
  - changes:
    - .gitlab-ci-opensbi.yml
@@ -24,6 +24,7 @@ docker-opensbi:
  - docker push $IMAGE_TAG
 
 build-opensbi:
+ stage: build
  rules: # Only run this job when ...
  - changes: # ... roms/opensbi/ is modified (submodule updated)
    - roms/opensbi/*
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 349c77aa58..9fdc752ea6 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,3 +1,7 @@
+stages:
+  - containers
+  - build
+
 include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
@@ -17,6 +21,7 @@ include:
   - JOBS=$(expr $(nproc) + 1)
 
 build-system1:
+ stage: build
  image: ubuntu:19.10
  <<: *before_script_apt
  script:
@@ -31,6 +36,7 @@ build-system1:
  - make -j"$JOBS" check
 
 build-system2:
+ stage: build
  image: fedora:latest
  <<: *before_script_dnf
  script:
@@ -46,6 +52,7 @@ build-system2:
  - make -j"$JOBS" check
 
 build-disabled:
+ stage: build
  image: fedora:latest
  <<: *before_script_dnf
  script:
@@ -62,6 +69,7 @@ build-disabled:
  - make -j"$JOBS" check-qtest SPEED=slow
 
 build-tcg-disabled:
+ stage: build
  image: centos:8
  <<: *before_script_dnf
  script:
@@ -82,6 +90,7 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
+ stage: build
  <<: *before_script_apt
  script:
  - mkdir build
@@ -92,6 +101,7 @@ build-user:
  - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
+ stage: build
  image: fedora:latest
  <<: *before_script_dnf
  script:
@@ -106,6 +116,7 @@ build-clang:
  - make -j"$JOBS" check
 
 build-tci:
+ stage: build
  image: centos:8
  <<: *before_script_dnf
  script:
-- 
2.24.1

Re: [PATCH RFC 1/3] gitlab: introduce explicit "container" and "build" stages

[Laszlo Ersek]

On 06/22/20 17:33, Daniel P. Berrangé wrote:
> If no stage is listed, jobs get put in an implicit "test" stage.
> Some jobs which create container images to be used by later stages
> are currently listed as in a "build" stages.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/edk2.yml    |  3 ++-
>  .gitlab-ci.d/opensbi.yml |  3 ++-
>  .gitlab-ci.yml           | 11 +++++++++++
>  3 files changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
> index 088ba4b43a..d4e7dfcba6 100644
> --- a/.gitlab-ci.d/edk2.yml
> +++ b/.gitlab-ci.d/edk2.yml
> @@ -1,5 +1,5 @@
>  docker-edk2:
> - stage: build
> + stage: containers
>   rules: # Only run this job when the Dockerfile is modified
>   - changes:
>     - .gitlab-ci-edk2.yml
> @@ -24,6 +24,7 @@ docker-edk2:
>   - docker push $IMAGE_TAG
>  
>  build-edk2:
> + stage: build
>   rules: # Only run this job when ...
>   - changes: # ... roms/edk2/ is modified (submodule updated)
>     - roms/edk2/*
> diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
> index dd051c0124..ec1c1f4cab 100644
> --- a/.gitlab-ci.d/opensbi.yml
> +++ b/.gitlab-ci.d/opensbi.yml
> @@ -1,5 +1,5 @@
>  docker-opensbi:
> - stage: build
> + stage: containers
>   rules: # Only run this job when the Dockerfile is modified
>   - changes:
>     - .gitlab-ci-opensbi.yml
> @@ -24,6 +24,7 @@ docker-opensbi:
>   - docker push $IMAGE_TAG
>  
>  build-opensbi:
> + stage: build
>   rules: # Only run this job when ...
>   - changes: # ... roms/opensbi/ is modified (submodule updated)
>     - roms/opensbi/*
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 349c77aa58..9fdc752ea6 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,3 +1,7 @@
> +stages:
> +  - containers
> +  - build
> +
>  include:
>    - local: '/.gitlab-ci.d/edk2.yml'
>    - local: '/.gitlab-ci.d/opensbi.yml'
> @@ -17,6 +21,7 @@ include:
>    - JOBS=$(expr $(nproc) + 1)
>  
>  build-system1:
> + stage: build
>   image: ubuntu:19.10
>   <<: *before_script_apt
>   script:
> @@ -31,6 +36,7 @@ build-system1:
>   - make -j"$JOBS" check
>  
>  build-system2:
> + stage: build
>   image: fedora:latest
>   <<: *before_script_dnf
>   script:
> @@ -46,6 +52,7 @@ build-system2:
>   - make -j"$JOBS" check
>  
>  build-disabled:
> + stage: build
>   image: fedora:latest
>   <<: *before_script_dnf
>   script:
> @@ -62,6 +69,7 @@ build-disabled:
>   - make -j"$JOBS" check-qtest SPEED=slow
>  
>  build-tcg-disabled:
> + stage: build
>   image: centos:8
>   <<: *before_script_dnf
>   script:
> @@ -82,6 +90,7 @@ build-tcg-disabled:
>              260 261 262 263 264 270 272 273 277 279
>  
>  build-user:
> + stage: build
>   <<: *before_script_apt
>   script:
>   - mkdir build
> @@ -92,6 +101,7 @@ build-user:
>   - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
>  
>  build-clang:
> + stage: build
>   image: fedora:latest
>   <<: *before_script_dnf
>   script:
> @@ -106,6 +116,7 @@ build-clang:
>   - make -j"$JOBS" check
>  
>  build-tci:
> + stage: build
>   image: centos:8
>   <<: *before_script_dnf
>   script:
> 

Acked-by: Laszlo Ersek <lersek@redhat.com>

Re: [PATCH RFC 1/3] gitlab: introduce explicit "container" and "build" stages

[Thomas Huth]

On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> If no stage is listed, jobs get put in an implicit "test" stage.
> Some jobs which create container images to be used by later stages
> are currently listed as in a "build" stages.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/edk2.yml    |  3 ++-
>   .gitlab-ci.d/opensbi.yml |  3 ++-
>   .gitlab-ci.yml           | 11 +++++++++++
>   3 files changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
> index 088ba4b43a..d4e7dfcba6 100644
> --- a/.gitlab-ci.d/edk2.yml
> +++ b/.gitlab-ci.d/edk2.yml
> @@ -1,5 +1,5 @@
>   docker-edk2:
> - stage: build
> + stage: containers
>    rules: # Only run this job when the Dockerfile is modified
>    - changes:
>      - .gitlab-ci-edk2.yml

Uh, oh, I guess I should have changed that line to .gitlab-ci.d/edk2.yml 
when I renamed that file .... will send a patch...

> @@ -24,6 +24,7 @@ docker-edk2:
>    - docker push $IMAGE_TAG
>   
>   build-edk2:
> + stage: build
>    rules: # Only run this job when ...
>    - changes: # ... roms/edk2/ is modified (submodule updated)
>      - roms/edk2/*
> diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
> index dd051c0124..ec1c1f4cab 100644
> --- a/.gitlab-ci.d/opensbi.yml
> +++ b/.gitlab-ci.d/opensbi.yml
> @@ -1,5 +1,5 @@
>   docker-opensbi:
> - stage: build
> + stage: containers
>    rules: # Only run this job when the Dockerfile is modified
>    - changes:
>      - .gitlab-ci-opensbi.yml

dito

> @@ -24,6 +24,7 @@ docker-opensbi:
>    - docker push $IMAGE_TAG
>   
>   build-opensbi:
> + stage: build
>    rules: # Only run this job when ...
>    - changes: # ... roms/opensbi/ is modified (submodule updated)
>      - roms/opensbi/*
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 349c77aa58..9fdc752ea6 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,3 +1,7 @@
> +stages:
> +  - containers
> +  - build
> +
>   include:
>     - local: '/.gitlab-ci.d/edk2.yml'
>     - local: '/.gitlab-ci.d/opensbi.yml'
> @@ -17,6 +21,7 @@ include:
>     - JOBS=$(expr $(nproc) + 1)
>   
>   build-system1:
> + stage: build
>    image: ubuntu:19.10
>    <<: *before_script_apt
>    script:
> @@ -31,6 +36,7 @@ build-system1:
>    - make -j"$JOBS" check
>   
>   build-system2:
> + stage: build
>    image: fedora:latest
>    <<: *before_script_dnf
>    script:
> @@ -46,6 +52,7 @@ build-system2:
>    - make -j"$JOBS" check
>   
>   build-disabled:
> + stage: build
>    image: fedora:latest
>    <<: *before_script_dnf
>    script:
> @@ -62,6 +69,7 @@ build-disabled:
>    - make -j"$JOBS" check-qtest SPEED=slow
>   
>   build-tcg-disabled:
> + stage: build
>    image: centos:8
>    <<: *before_script_dnf
>    script:
> @@ -82,6 +90,7 @@ build-tcg-disabled:
>               260 261 262 263 264 270 272 273 277 279
>   
>   build-user:
> + stage: build
>    <<: *before_script_apt
>    script:
>    - mkdir build
> @@ -92,6 +101,7 @@ build-user:
>    - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
>   
>   build-clang:
> + stage: build
>    image: fedora:latest
>    <<: *before_script_dnf
>    script:
> @@ -106,6 +116,7 @@ build-clang:
>    - make -j"$JOBS" check
>   
>   build-tci:
> + stage: build
>    image: centos:8
>    <<: *before_script_dnf
>    script:
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH RFC 1/3] gitlab: introduce explicit "container" and "build" stages

[Philippe Mathieu-Daudé]

On 6/25/20 10:54 AM, Thomas Huth wrote:
> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
>> If no stage is listed, jobs get put in an implicit "test" stage.
>> Some jobs which create container images to be used by later stages
>> are currently listed as in a "build" stages.
>>
>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> ---
>>   .gitlab-ci.d/edk2.yml    |  3 ++-
>>   .gitlab-ci.d/opensbi.yml |  3 ++-
>>   .gitlab-ci.yml           | 11 +++++++++++
>>   3 files changed, 15 insertions(+), 2 deletions(-)
>>
>> diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
>> index 088ba4b43a..d4e7dfcba6 100644
>> --- a/.gitlab-ci.d/edk2.yml
>> +++ b/.gitlab-ci.d/edk2.yml
>> @@ -1,5 +1,5 @@
>>   docker-edk2:
>> - stage: build
>> + stage: containers
>>    rules: # Only run this job when the Dockerfile is modified
>>    - changes:
>>      - .gitlab-ci-edk2.yml
> 
> Uh, oh, I guess I should have changed that line to .gitlab-ci.d/edk2.yml
> when I renamed that file .... will send a patch...

Well, your patch has been reviewed...

> 
>> @@ -24,6 +24,7 @@ docker-edk2:
>>    - docker push $IMAGE_TAG
>>     build-edk2:
>> + stage: build
>>    rules: # Only run this job when ...
>>    - changes: # ... roms/edk2/ is modified (submodule updated)
>>      - roms/edk2/*
>> diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
>> index dd051c0124..ec1c1f4cab 100644
>> --- a/.gitlab-ci.d/opensbi.yml
>> +++ b/.gitlab-ci.d/opensbi.yml
>> @@ -1,5 +1,5 @@
>>   docker-opensbi:
>> - stage: build
>> + stage: containers
>>    rules: # Only run this job when the Dockerfile is modified
>>    - changes:
>>      - .gitlab-ci-opensbi.yml
> 
> dito

Oops...

> 
>> @@ -24,6 +24,7 @@ docker-opensbi:
>>    - docker push $IMAGE_TAG
>>     build-opensbi:
>> + stage: build
>>    rules: # Only run this job when ...
>>    - changes: # ... roms/opensbi/ is modified (submodule updated)
>>      - roms/opensbi/*
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 349c77aa58..9fdc752ea6 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -1,3 +1,7 @@
>> +stages:
>> +  - containers
>> +  - build
>> +
>>   include:
>>     - local: '/.gitlab-ci.d/edk2.yml'
>>     - local: '/.gitlab-ci.d/opensbi.yml'
>> @@ -17,6 +21,7 @@ include:
>>     - JOBS=$(expr $(nproc) + 1)
>>     build-system1:
>> + stage: build
>>    image: ubuntu:19.10
>>    <<: *before_script_apt
>>    script:
>> @@ -31,6 +36,7 @@ build-system1:
>>    - make -j"$JOBS" check
>>     build-system2:
>> + stage: build
>>    image: fedora:latest
>>    <<: *before_script_dnf
>>    script:
>> @@ -46,6 +52,7 @@ build-system2:
>>    - make -j"$JOBS" check
>>     build-disabled:
>> + stage: build
>>    image: fedora:latest
>>    <<: *before_script_dnf
>>    script:
>> @@ -62,6 +69,7 @@ build-disabled:
>>    - make -j"$JOBS" check-qtest SPEED=slow
>>     build-tcg-disabled:
>> + stage: build
>>    image: centos:8
>>    <<: *before_script_dnf
>>    script:
>> @@ -82,6 +90,7 @@ build-tcg-disabled:
>>               260 261 262 263 264 270 272 273 277 279
>>     build-user:
>> + stage: build
>>    <<: *before_script_apt
>>    script:
>>    - mkdir build
>> @@ -92,6 +101,7 @@ build-user:
>>    - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
>>     build-clang:
>> + stage: build
>>    image: fedora:latest
>>    <<: *before_script_dnf
>>    script:
>> @@ -106,6 +116,7 @@ build-clang:
>>    - make -j"$JOBS" check
>>     build-tci:
>> + stage: build
>>    image: centos:8
>>    <<: *before_script_dnf
>>    script:
>>
> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

[PATCH RFC 2/3] gitlab: build all container images during CI

[Daniel P. Berrangé]

We have a number of container images in tests/docker/dockerfiles
that are intended to provide well defined environments for doing
test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the
CI, so that the built containers are available for later build
jobs. The containers are setup to use the GitLab container
registry as the cache, so we only pay the penalty of the full
build when the dockerfiles change. The main qemu-project/qemu
repo is used as a second cache, so that users forking QEMU will
see a fast turnaround time on their CI jobs.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml              |   3 +
 2 files changed, 251 insertions(+)
 create mode 100644 .gitlab-ci.d/containers.yml

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
new file mode 100644
index 0000000000..ea1edbb196
--- /dev/null
+++ b/.gitlab-ci.d/containers.yml
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+    - docker info
+    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
+    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+    - docker push "$TAG"
+  after_script:
+    - docker logout
+
+amd64-centos7-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos7
+
+amd64-centos8-container:
+  <<: *container_job_definition
+  variables:
+    NAME: centos8
+
+amd64-debian10-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian10
+
+amd64-debian11-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian11
+
+amd64-debian9-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian9
+
+amd64-debian9-mxe-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian9-mxe
+
+alpha-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-alpha-cross
+
+amd64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64-cross
+
+amd64-debian-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-amd64
+
+arm64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-arm64-cross
+
+arm64-test-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian11-container']
+  variables:
+    NAME: debian-arm64-test-cross
+
+armel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armel-cross
+
+armhf-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-armhf-cross
+
+hppa-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-hppa-cross
+
+m68k-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-m68k-cross
+
+mips64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64-cross
+
+mips64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips64el-cross
+
+mips-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mips-cross
+
+mipsel-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-mipsel-cross
+
+powerpc-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-powerpc-cross
+
+ppc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64-cross
+
+ppc64el-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-ppc64el-cross
+
+riscv64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-riscv64-cross
+
+s390x-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-s390x-cross
+
+sh4-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sh4-cross
+
+sparc64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-sparc64-cross
+
+tricore-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer2
+  needs: ['amd64-debian9-container']
+  variables:
+    NAME: debian-tricore-cross
+
+win32-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win32-cross
+
+win64-debian-cross-container:
+  <<: *container_job_definition
+  stage: containers-layer3
+  needs: ['amd64-debian9-mxe-container']
+  variables:
+    NAME: debian-win64-cross
+
+xtensa-debian-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: debian-xtensa-cross
+
+cris-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-cris-cross
+
+amd64-fedora-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora
+
+i386-fedora-cross-container:
+  <<: *container_job_definition
+  variables:
+    NAME: fedora-i386-cross
+
+amd64-ubuntu1804-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu1804
+
+amd64-ubuntu2004-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu2004
+
+amd64-ubuntu-container:
+  <<: *container_job_definition
+  variables:
+    NAME: ubuntu
+
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9fdc752ea6..72d688875f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,10 +1,13 @@
 stages:
   - containers
+  - containers-layer2
+  - containers-layer3
   - build
 
 include:
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
+  - local: '/.gitlab-ci.d/containers.yml'
 
 .update_apt_template: &before_script_apt
  before_script:
-- 
2.24.1

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Philippe Mathieu-Daudé]

On 6/22/20 5:33 PM, Daniel P. Berrangé wrote:
> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
> 
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.

OMG you did it! Lovely... 😍

Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
why gitlab isn't caching the docker images? the cache isn't
populated on all the runners yet? Or we have to use the same
runner again to hit its cache?

> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.yml              |   3 +
>  2 files changed, 251 insertions(+)
>  create mode 100644 .gitlab-ci.d/containers.yml
> 
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout
> +
> +amd64-centos7-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos7
[...]

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Daniel P. Berrangé]

On Mon, Jun 22, 2020 at 05:38:16PM +0200, Philippe Mathieu-Daudé wrote:
> On 6/22/20 5:33 PM, Daniel P. Berrangé wrote:
> > We have a number of container images in tests/docker/dockerfiles
> > that are intended to provide well defined environments for doing
> > test builds. We want our CI system to use these containers too.
> > 
> > This introduces builds of all of them as the first stage in the
> > CI, so that the built containers are available for later build
> > jobs. The containers are setup to use the GitLab container
> > registry as the cache, so we only pay the penalty of the full
> > build when the dockerfiles change. The main qemu-project/qemu
> > repo is used as a second cache, so that users forking QEMU will
> > see a fast turnaround time on their CI jobs.
> 
> OMG you did it! Lovely... 😍
> 
> Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
> why gitlab isn't caching the docker images? the cache isn't
> populated on all the runners yet? Or we have to use the same
> runner again to hit its cache?

It is caching it but it isn't obvious what to look for.

The "docker build" command is always still run, but you'll
notice each of the actual commands in the dockerfile are
followed by a message "  ---> Using cache", instead of the
normal command output.

Compare the "amd64-debian9-container" job as an example.

Here is the output seen in the original job which actually did a real
build:

   https://gitlab.com/berrange/qemu/-/jobs/605783351

And here is the output from the pipeline you point to above:

   https://gitlab.com/berrange/qemu/-/jobs/606175855

The cached case is still taking 2 mins 30 seconds, but the original
full build took 4 mins 50 seconds.

The difference will be more noticable if we expand the list of packages
install in the images to cover more of QEMU's possible dependancies.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Philippe Mathieu-Daudé]

On 6/22/20 5:46 PM, Daniel P. Berrangé wrote:
> On Mon, Jun 22, 2020 at 05:38:16PM +0200, Philippe Mathieu-Daudé wrote:
>> On 6/22/20 5:33 PM, Daniel P. Berrangé wrote:
>>> We have a number of container images in tests/docker/dockerfiles
>>> that are intended to provide well defined environments for doing
>>> test builds. We want our CI system to use these containers too.
>>>
>>> This introduces builds of all of them as the first stage in the
>>> CI, so that the built containers are available for later build
>>> jobs. The containers are setup to use the GitLab container
>>> registry as the cache, so we only pay the penalty of the full
>>> build when the dockerfiles change. The main qemu-project/qemu
>>> repo is used as a second cache, so that users forking QEMU will
>>> see a fast turnaround time on their CI jobs.
>>
>> OMG you did it! Lovely... 😍
>>
>> Looking at https://gitlab.com/berrange/qemu/-/pipelines/158854797
>> why gitlab isn't caching the docker images? the cache isn't
>> populated on all the runners yet? Or we have to use the same
>> runner again to hit its cache?
> 
> It is caching it but it isn't obvious what to look for.
> 
> The "docker build" command is always still run, but you'll
> notice each of the actual commands in the dockerfile are
> followed by a message "  ---> Using cache", instead of the
> normal command output.

Indeed!

> 
> Compare the "amd64-debian9-container" job as an example.
> 
> Here is the output seen in the original job which actually did a real
> build:
> 
>    https://gitlab.com/berrange/qemu/-/jobs/605783351
> 
> And here is the output from the pipeline you point to above:
> 
>    https://gitlab.com/berrange/qemu/-/jobs/606175855
> 
> The cached case is still taking 2 mins 30 seconds, but the original
> full build took 4 mins 50 seconds.
> 
> The difference will be more noticable if we expand the list of packages
> install in the images to cover more of QEMU's possible dependancies.

The difference is already very noticeable on my host:

$ time docker pull registry.gitlab.com/berrange/qemu/ci-centos8:latest
latest: Pulling from berrange/qemu/ci-centos8
8a29a15cefae: Pull complete
4a748adf2909: Pull complete
3c034506a458: Pull complete
cd48de325fa1: Pull complete
Digest:
sha256:a6580433f5456d7c49e2b9b796c717c79a5fa15f4eecf71bdbcffd9df6b8217a
Status: Downloaded newer image for
registry.gitlab.com/berrange/qemu/ci-centos8:latest
registry.gitlab.com/berrange/qemu/ci-centos8:latest
real    0m53.566s
user    0m0.115s
sys     0m0.081s

> 
> Regards,
> Daniel
> 

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
>
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.yml              |   3 +
>  2 files changed, 251 insertions(+)
>  create mode 100644 .gitlab-ci.d/containers.yml
>
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export
> COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"

It would be nice if we could keep the same form as they have in the
local registry which would make it easier to integrate with the rest of
the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"

Otherwise it looks good:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>


-- 
Alex Bennée

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Daniel P. Berrangé]

On Mon, Jun 22, 2020 at 07:26:39PM +0100, Alex Bennée wrote:
> 
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > We have a number of container images in tests/docker/dockerfiles
> > that are intended to provide well defined environments for doing
> > test builds. We want our CI system to use these containers too.
> >
> > This introduces builds of all of them as the first stage in the
> > CI, so that the built containers are available for later build
> > jobs. The containers are setup to use the GitLab container
> > registry as the cache, so we only pay the penalty of the full
> > build when the dockerfiles change. The main qemu-project/qemu
> > repo is used as a second cache, so that users forking QEMU will
> > see a fast turnaround time on their CI jobs.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
> >  .gitlab-ci.yml              |   3 +
> >  2 files changed, 251 insertions(+)
> >  create mode 100644 .gitlab-ci.d/containers.yml
> >
> > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > new file mode 100644
> > index 0000000000..ea1edbb196
> > --- /dev/null
> > +++ b/.gitlab-ci.d/containers.yml
> > @@ -0,0 +1,248 @@
> > +
> > +
> > +.container_job_template: &container_job_definition
> > +  image: docker:stable
> > +  stage: containers
> > +  services:
> > +    - docker:dind
> > +  before_script:
> > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> > +    - export
> > COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> 
> It would be nice if we could keep the same form as they have in the
> local registry which would make it easier to integrate with the rest of
> the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"

Every time I re-discover it, I find the QEMU container naming really
surprising. It is not following the normal best practice for naming
containers. Expected container naming convention is that the image
name reflects the general content set, and the tag reflects a version
number. QEMU has shifted it along, so container name is just a fixed
string, and the tag reflects the contenet set, and there is no version.

QEMU's naming will cause problems with caching in GitLab. As GitLab
expects the normal container naming scheme, it has a job which expires
old versions of an image once there are more than 10 tags. So we have
to use the normal naming scheme. Ideally we would change rest of QEMU
to use the normal scheme too.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Jun 22, 2020 at 07:26:39PM +0100, Alex Bennée wrote:
>> 
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> 
>> > We have a number of container images in tests/docker/dockerfiles
>> > that are intended to provide well defined environments for doing
>> > test builds. We want our CI system to use these containers too.
>> >
>> > This introduces builds of all of them as the first stage in the
>> > CI, so that the built containers are available for later build
>> > jobs. The containers are setup to use the GitLab container
>> > registry as the cache, so we only pay the penalty of the full
>> > build when the dockerfiles change. The main qemu-project/qemu
>> > repo is used as a second cache, so that users forking QEMU will
>> > see a fast turnaround time on their CI jobs.
>> >
>> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> > ---
>> >  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>> >  .gitlab-ci.yml              |   3 +
>> >  2 files changed, 251 insertions(+)
>> >  create mode 100644 .gitlab-ci.d/containers.yml
>> >
>> > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>> > new file mode 100644
>> > index 0000000000..ea1edbb196
>> > --- /dev/null
>> > +++ b/.gitlab-ci.d/containers.yml
>> > @@ -0,0 +1,248 @@
>> > +
>> > +
>> > +.container_job_template: &container_job_definition
>> > +  image: docker:stable
>> > +  stage: containers
>> > +  services:
>> > +    - docker:dind
>> > +  before_script:
>> > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
>> > +    - export
>> > COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
>> 
>> It would be nice if we could keep the same form as they have in the
>> local registry which would make it easier to integrate with the rest of
>> the tooling, e.g. "$CI_REGISTRY/qemu-project/qemu:$NAME"
>
> Every time I re-discover it, I find the QEMU container naming really
> surprising. It is not following the normal best practice for naming
> containers. Expected container naming convention is that the image
> name reflects the general content set, and the tag reflects a version
> number. QEMU has shifted it along, so container name is just a fixed
> string, and the tag reflects the contenet set, and there is no version.
>
> QEMU's naming will cause problems with caching in GitLab. As GitLab
> expects the normal container naming scheme, it has a job which expires
> old versions of an image once there are more than 10 tags. So we have
> to use the normal naming scheme. Ideally we would change rest of QEMU
> to use the normal scheme too.

Fair enough.. I'll look into it.

>
> Regards,
> Daniel


-- 
Alex Bennée

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Thomas Huth]

On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
> 
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>   .gitlab-ci.yml              |   3 +
>   2 files changed, 251 insertions(+)
>   create mode 100644 .gitlab-ci.d/containers.yml
> 
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"

I can see this in the output:

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

I have to admit that I have only little knowledge about docker ... but 
could there be an issue here? Should this be done in a different way?

> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout
> +
> +amd64-centos7-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos7
> +
> +amd64-centos8-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos8
> +
> +amd64-debian10-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian10
> +
> +amd64-debian11-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian11
> +
> +amd64-debian9-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian9
> +
> +amd64-debian9-mxe-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian9-mxe
> +
> +alpha-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-alpha-cross
> +
> +amd64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64-cross
> +
> +amd64-debian-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64
> +
> +arm64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-arm64-cross
> +
> +arm64-test-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian11-container']
> +  variables:
> +    NAME: debian-arm64-test-cross
> +
> +armel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armel-cross
> +
> +armhf-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armhf-cross
> +
> +hppa-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-hppa-cross
> +
> +m68k-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-m68k-cross
> +
> +mips64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64-cross
> +
> +mips64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64el-cross
> +
> +mips-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips-cross
> +
> +mipsel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mipsel-cross
> +
> +powerpc-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-powerpc-cross
> +
> +ppc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64-cross
> +
> +ppc64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64el-cross
> +
> +riscv64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-riscv64-cross
> +
> +s390x-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-s390x-cross
> +
> +sh4-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sh4-cross
> +
> +sparc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sparc64-cross
> +
> +tricore-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian-tricore-cross
> +
> +win32-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win32-cross
> +
> +win64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win64-cross
> +
> +xtensa-debian-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian-xtensa-cross
> +
> +cris-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-cris-cross
> +
> +amd64-fedora-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora
> +
> +i386-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-i386-cross
> +
> +amd64-ubuntu1804-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu1804
> +
> +amd64-ubuntu2004-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu2004
> +
> +amd64-ubuntu-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu
> +
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 9fdc752ea6..72d688875f 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,10 +1,13 @@
>   stages:
>     - containers
> +  - containers-layer2
> +  - containers-layer3
>     - build
>   
>   include:
>     - local: '/.gitlab-ci.d/edk2.yml'
>     - local: '/.gitlab-ci.d/opensbi.yml'
> +  - local: '/.gitlab-ci.d/containers.yml'
>   
>   .update_apt_template: &before_script_apt
>    before_script:
> 

Acked-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Daniel P. Berrangé]

On Thu, Jun 25, 2020 at 11:35:54AM +0200, Thomas Huth wrote:
> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > We have a number of container images in tests/docker/dockerfiles
> > that are intended to provide well defined environments for doing
> > test builds. We want our CI system to use these containers too.
> > 
> > This introduces builds of all of them as the first stage in the
> > CI, so that the built containers are available for later build
> > jobs. The containers are setup to use the GitLab container
> > registry as the cache, so we only pay the penalty of the full
> > build when the dockerfiles change. The main qemu-project/qemu
> > repo is used as a second cache, so that users forking QEMU will
> > see a fast turnaround time on their CI jobs.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
> >   .gitlab-ci.yml              |   3 +
> >   2 files changed, 251 insertions(+)
> >   create mode 100644 .gitlab-ci.d/containers.yml
> > 
> > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > new file mode 100644
> > index 0000000000..ea1edbb196
> > --- /dev/null
> > +++ b/.gitlab-ci.d/containers.yml
> > @@ -0,0 +1,248 @@
> > +
> > +
> > +.container_job_template: &container_job_definition
> > +  image: docker:stable
> > +  stage: containers
> > +  services:
> > +    - docker:dind
> > +  before_script:
> > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> > +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> > +    - docker info
> > +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> 
> I can see this in the output:
> 
> WARNING! Using --password via the CLI is insecure. Use --password-stdin.
> 
> I have to admit that I have only little knowledge about docker ... but could
> there be an issue here? Should this be done in a different way?

In general the warning is correct, because other users on the same
host can see the process CLI args, and thus the password is susceptible
to snooping.

In this case, however, it is a non-issue. This is running inside a docker
container already which has a PID namespace. Thus the only things that
can see our password on the CLI are things inside our own container
which already all have the env variable, and processes running in host
OS context which are only things GitLab admins control. So there's no
data leakage to anyone who doesn't already have access to the password

This particular docker login command is the documented solution:

  https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Laszlo Ersek]

On 06/25/20 11:50, Daniel P. Berrangé wrote:
> On Thu, Jun 25, 2020 at 11:35:54AM +0200, Thomas Huth wrote:
>> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
>>> We have a number of container images in tests/docker/dockerfiles
>>> that are intended to provide well defined environments for doing
>>> test builds. We want our CI system to use these containers too.
>>>
>>> This introduces builds of all of them as the first stage in the
>>> CI, so that the built containers are available for later build
>>> jobs. The containers are setup to use the GitLab container
>>> registry as the cache, so we only pay the penalty of the full
>>> build when the dockerfiles change. The main qemu-project/qemu
>>> repo is used as a second cache, so that users forking QEMU will
>>> see a fast turnaround time on their CI jobs.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>>>   .gitlab-ci.yml              |   3 +
>>>   2 files changed, 251 insertions(+)
>>>   create mode 100644 .gitlab-ci.d/containers.yml
>>>
>>> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>>> new file mode 100644
>>> index 0000000000..ea1edbb196
>>> --- /dev/null
>>> +++ b/.gitlab-ci.d/containers.yml
>>> @@ -0,0 +1,248 @@
>>> +
>>> +
>>> +.container_job_template: &container_job_definition
>>> +  image: docker:stable
>>> +  stage: containers
>>> +  services:
>>> +    - docker:dind
>>> +  before_script:
>>> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
>>> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
>>> +    - docker info
>>> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>>
>> I can see this in the output:
>>
>> WARNING! Using --password via the CLI is insecure. Use --password-stdin.
>>
>> I have to admit that I have only little knowledge about docker ... but could
>> there be an issue here? Should this be done in a different way?
> 
> In general the warning is correct, because other users on the same
> host can see the process CLI args, and thus the password is susceptible
> to snooping.
> 
> In this case, however, it is a non-issue. This is running inside a docker
> container already which has a PID namespace. Thus the only things that
> can see our password on the CLI are things inside our own container
> which already all have the env variable, and processes running in host
> OS context which are only things GitLab admins control. So there's no
> data leakage to anyone who doesn't already have access to the password
> 
> This particular docker login command is the documented solution:
> 
>   https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

(

Purely theoretically, we could use a "here string":

docker [...] --password-stdin <<< "$CI_REGISTRY_PASSWORD"

The password is then not exposed on any process's command line; it's a
(bash) shell redirection. (It's not in POSIX.)

)

Thanks
Laszlo

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Daniel P. Berrangé]

On Mon, Jun 22, 2020 at 04:33:17PM +0100, Daniel P. Berrangé wrote:
> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
> 
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.yml              |   3 +
>  2 files changed, 251 insertions(+)
>  create mode 100644 .gitlab-ci.d/containers.yml

Thinking about this at 3am with insomnia, I started considering possible
ways this could result in CI failures. I came up with

 - Distro repos are unavialable due to transient network/mirror problems
 
 - Distro pushes bad packages or changes something that invalidates QEMU
   assumptions

The first one can hit at any time, but I think that we're reasonably well
insulated from it due to our usage of caching. Only a small number of CI
jobs are going to actually trigger a full rebuild of an image, most will
just hit the cache.

The second one is probably not likely, *provided* we only use stable branches
of distros. It looks like we're ok in that regard as we're not using Debian
Sid, or Fedora rawhide for any images currently.

If we did want to reduce the risk though, we could mark some of the
container jobs as non-fatal. We could mark the subset of images that are
not actually required by later build jobs that we currently have.

> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout
> +
> +amd64-centos7-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos7
> +
> +amd64-centos8-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos8
> +
> +amd64-debian10-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian10
> +
> +amd64-debian11-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian11
> +
> +amd64-debian9-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian9
> +
> +amd64-debian9-mxe-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian9-mxe
> +
> +alpha-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-alpha-cross
> +
> +amd64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64-cross
> +
> +amd64-debian-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64
> +
> +arm64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-arm64-cross
> +
> +arm64-test-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian11-container']
> +  variables:
> +    NAME: debian-arm64-test-cross
> +
> +armel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armel-cross
> +
> +armhf-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armhf-cross
> +
> +hppa-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-hppa-cross
> +
> +m68k-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-m68k-cross
> +
> +mips64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64-cross
> +
> +mips64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64el-cross
> +
> +mips-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips-cross
> +
> +mipsel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mipsel-cross
> +
> +powerpc-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-powerpc-cross
> +
> +ppc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64-cross
> +
> +ppc64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64el-cross
> +
> +riscv64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-riscv64-cross
> +
> +s390x-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-s390x-cross
> +
> +sh4-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sh4-cross
> +
> +sparc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sparc64-cross
> +
> +tricore-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian-tricore-cross
> +
> +win32-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win32-cross
> +
> +win64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win64-cross
> +
> +xtensa-debian-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian-xtensa-cross
> +
> +cris-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-cris-cross
> +
> +amd64-fedora-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora
> +
> +i386-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-i386-cross
> +
> +amd64-ubuntu1804-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu1804
> +
> +amd64-ubuntu2004-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu2004
> +
> +amd64-ubuntu-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu
> +
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 9fdc752ea6..72d688875f 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,10 +1,13 @@
>  stages:
>    - containers
> +  - containers-layer2
> +  - containers-layer3
>    - build
>  
>  include:
>    - local: '/.gitlab-ci.d/edk2.yml'
>    - local: '/.gitlab-ci.d/opensbi.yml'
> +  - local: '/.gitlab-ci.d/containers.yml'
>  
>  .update_apt_template: &before_script_apt
>   before_script:
> -- 
> 2.24.1
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Jun 22, 2020 at 04:33:17PM +0100, Daniel P. Berrangé wrote:
>> We have a number of container images in tests/docker/dockerfiles
>> that are intended to provide well defined environments for doing
>> test builds. We want our CI system to use these containers too.
>> 
>> This introduces builds of all of them as the first stage in the
>> CI, so that the built containers are available for later build
>> jobs. The containers are setup to use the GitLab container
>> registry as the cache, so we only pay the penalty of the full
>> build when the dockerfiles change. The main qemu-project/qemu
>> repo is used as a second cache, so that users forking QEMU will
>> see a fast turnaround time on their CI jobs.
>> 
>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> ---
>>  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>>  .gitlab-ci.yml              |   3 +
>>  2 files changed, 251 insertions(+)
>>  create mode 100644 .gitlab-ci.d/containers.yml
>
> Thinking about this at 3am with insomnia, I started considering possible
> ways this could result in CI failures. I came up with
>
>  - Distro repos are unavialable due to transient network/mirror problems
>  
>  - Distro pushes bad packages or changes something that invalidates QEMU
>    assumptions
>
> The first one can hit at any time, but I think that we're reasonably well
> insulated from it due to our usage of caching. Only a small number of CI
> jobs are going to actually trigger a full rebuild of an image, most will
> just hit the cache.
>
> The second one is probably not likely, *provided* we only use stable branches
> of distros. It looks like we're ok in that regard as we're not using Debian
> Sid, or Fedora rawhide for any images currently.

We use debian11 (which I think is debian-testing) for one of the aarch64
compilers because we need fairlyu bleeding edge gcc's but we won't go
back to sid anytime soon.

>
> If we did want to reduce the risk though, we could mark some of the
> container jobs as non-fatal. We could mark the subset of images that are
> not actually required by later build jobs that we currently have.
>
>> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>> new file mode 100644
>> index 0000000000..ea1edbb196
>> --- /dev/null
>> +++ b/.gitlab-ci.d/containers.yml
>> @@ -0,0 +1,248 @@
>> +
>> +
>> +.container_job_template: &container_job_definition
>> +  image: docker:stable
>> +  stage: containers
>> +  services:
>> +    - docker:dind
>> +  before_script:
>> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
>> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
>> +    - docker info
>> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>> +  script:
>> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
>> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
>> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
>> +    - docker push "$TAG"
>> +  after_script:
>> +    - docker logout
>> +
>> +amd64-centos7-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: centos7
>> +
>> +amd64-centos8-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: centos8
>> +
>> +amd64-debian10-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian10
>> +
>> +amd64-debian11-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian11
>> +
>> +amd64-debian9-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian9
>> +
>> +amd64-debian9-mxe-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian9-container']
>> +  variables:
>> +    NAME: debian9-mxe
>> +
>> +alpha-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-alpha-cross
>> +
>> +amd64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-amd64-cross
>> +
>> +amd64-debian-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-amd64
>> +
>> +arm64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-arm64-cross
>> +
>> +arm64-test-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian11-container']
>> +  variables:
>> +    NAME: debian-arm64-test-cross
>> +
>> +armel-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-armel-cross
>> +
>> +armhf-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-armhf-cross
>> +
>> +hppa-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-hppa-cross
>> +
>> +m68k-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-m68k-cross
>> +
>> +mips64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mips64-cross
>> +
>> +mips64el-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mips64el-cross
>> +
>> +mips-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mips-cross
>> +
>> +mipsel-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mipsel-cross
>> +
>> +powerpc-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-powerpc-cross
>> +
>> +ppc64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-ppc64-cross
>> +
>> +ppc64el-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-ppc64el-cross
>> +
>> +riscv64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-riscv64-cross
>> +
>> +s390x-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-s390x-cross
>> +
>> +sh4-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-sh4-cross
>> +
>> +sparc64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-sparc64-cross
>> +
>> +tricore-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian9-container']
>> +  variables:
>> +    NAME: debian-tricore-cross
>> +
>> +win32-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer3
>> +  needs: ['amd64-debian9-mxe-container']
>> +  variables:
>> +    NAME: debian-win32-cross
>> +
>> +win64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer3
>> +  needs: ['amd64-debian9-mxe-container']
>> +  variables:
>> +    NAME: debian-win64-cross
>> +
>> +xtensa-debian-cross-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian-xtensa-cross
>> +
>> +cris-fedora-cross-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: fedora-cris-cross
>> +
>> +amd64-fedora-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: fedora
>> +
>> +i386-fedora-cross-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: fedora-i386-cross
>> +
>> +amd64-ubuntu1804-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: ubuntu1804
>> +
>> +amd64-ubuntu2004-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: ubuntu2004
>> +
>> +amd64-ubuntu-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: ubuntu
>> +
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 9fdc752ea6..72d688875f 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -1,10 +1,13 @@
>>  stages:
>>    - containers
>> +  - containers-layer2
>> +  - containers-layer3
>>    - build
>>  
>>  include:
>>    - local: '/.gitlab-ci.d/edk2.yml'
>>    - local: '/.gitlab-ci.d/opensbi.yml'
>> +  - local: '/.gitlab-ci.d/containers.yml'
>>  
>>  .update_apt_template: &before_script_apt
>>   before_script:
>> -- 
>> 2.24.1
>> 
>
> Regards,
> Daniel


-- 
Alex Bennée

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Thomas Huth]

On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
> 
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>   .gitlab-ci.yml              |   3 +
>   2 files changed, 251 insertions(+)
>   create mode 100644 .gitlab-ci.d/containers.yml
> 
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout

.gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if 
something really has been changed. Could you use something similar here? 
E.g.:

rules:
  - changes:
    - .gitlab-ci.d/containers.yml
    - tests/docker/*
    - tests/docker/dockerfiles/*

?

  Thomas

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Daniel P. Berrangé]

On Thu, Jun 25, 2020 at 12:14:33PM +0200, Thomas Huth wrote:
> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > We have a number of container images in tests/docker/dockerfiles
> > that are intended to provide well defined environments for doing
> > test builds. We want our CI system to use these containers too.
> > 
> > This introduces builds of all of them as the first stage in the
> > CI, so that the built containers are available for later build
> > jobs. The containers are setup to use the GitLab container
> > registry as the cache, so we only pay the penalty of the full
> > build when the dockerfiles change. The main qemu-project/qemu
> > repo is used as a second cache, so that users forking QEMU will
> > see a fast turnaround time on their CI jobs.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
> >   .gitlab-ci.yml              |   3 +
> >   2 files changed, 251 insertions(+)
> >   create mode 100644 .gitlab-ci.d/containers.yml
> > 
> > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > new file mode 100644
> > index 0000000000..ea1edbb196
> > --- /dev/null
> > +++ b/.gitlab-ci.d/containers.yml
> > @@ -0,0 +1,248 @@
> > +
> > +
> > +.container_job_template: &container_job_definition
> > +  image: docker:stable
> > +  stage: containers
> > +  services:
> > +    - docker:dind
> > +  before_script:
> > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> > +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> > +    - docker info
> > +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> > +  script:
> > +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> > +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> > +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> > +    - docker push "$TAG"
> > +  after_script:
> > +    - docker logout
> 
> .gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if
> something really has been changed. Could you use something similar here?
> E.g.:
> 
> rules:
>  - changes:
>    - .gitlab-ci.d/containers.yml
>    - tests/docker/*
>    - tests/docker/dockerfiles/*
> 
> ?

If the OS distro base image changes, we'll never pick it up with that
kind of filtering.  For the main gitlab.com/qemu-project/qemu  you
could configure a nightly/weekly/whatever job to force rebuild on a
periodic basis to pick up base image changes.  The downside of this
is that any users who fork qemu won't have that periodic job and so
will be testing their work against potentially outdated content.

Having said all that, I'm not 100% convinced I'm actually picking
up changed base images right now anyway, given our use of caching.

It is possible that I would need todo an explict "docker pull" of
the base image to force it to trigger a refresh othrewise I have
a feeling we're always cached.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Thomas Huth]

On 25/06/2020 12.24, Daniel P. Berrangé wrote:
> On Thu, Jun 25, 2020 at 12:14:33PM +0200, Thomas Huth wrote:
>> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
>>> We have a number of container images in tests/docker/dockerfiles
>>> that are intended to provide well defined environments for doing
>>> test builds. We want our CI system to use these containers too.
>>>
>>> This introduces builds of all of them as the first stage in the
>>> CI, so that the built containers are available for later build
>>> jobs. The containers are setup to use the GitLab container
>>> registry as the cache, so we only pay the penalty of the full
>>> build when the dockerfiles change. The main qemu-project/qemu
>>> repo is used as a second cache, so that users forking QEMU will
>>> see a fast turnaround time on their CI jobs.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>    .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>>>    .gitlab-ci.yml              |   3 +
>>>    2 files changed, 251 insertions(+)
>>>    create mode 100644 .gitlab-ci.d/containers.yml
>>>
>>> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>>> new file mode 100644
>>> index 0000000000..ea1edbb196
>>> --- /dev/null
>>> +++ b/.gitlab-ci.d/containers.yml
>>> @@ -0,0 +1,248 @@
>>> +
>>> +
>>> +.container_job_template: &container_job_definition
>>> +  image: docker:stable
>>> +  stage: containers
>>> +  services:
>>> +    - docker:dind
>>> +  before_script:
>>> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
>>> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
>>> +    - docker info
>>> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>>> +  script:
>>> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
>>> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
>>> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
>>> +    - docker push "$TAG"
>>> +  after_script:
>>> +    - docker logout
>>
>> .gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if
>> something really has been changed. Could you use something similar here?
>> E.g.:
>>
>> rules:
>>   - changes:
>>     - .gitlab-ci.d/containers.yml
>>     - tests/docker/*
>>     - tests/docker/dockerfiles/*
>>
>> ?
> 
> If the OS distro base image changes, we'll never pick it up with that
> kind of filtering.  For the main gitlab.com/qemu-project/qemu  you
> could configure a nightly/weekly/whatever job to force rebuild on a
> periodic basis to pick up base image changes.  The downside of this
> is that any users who fork qemu won't have that periodic job and so
> will be testing their work against potentially outdated content.
> 
> Having said all that, I'm not 100% convinced I'm actually picking
> up changed base images right now anyway, given our use of caching.
> 
> It is possible that I would need todo an explict "docker pull" of
> the base image to force it to trigger a refresh othrewise I have
> a feeling we're always cached.

But currently, each of the container stages currently takes > 2 minutes, 
even with the cached containers. I had a quick look, and it takes 7 
minutes 'till the "build" stage begins. So all the advantages of not 
having to do "yum/apt-get install" in the build containers anymore seem 
to be crushed by the time that the three container stages take now?

  Thomas

Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Daniel P. Berrangé]

On Thu, Jun 25, 2020 at 03:25:19PM +0200, Thomas Huth wrote:
> On 25/06/2020 12.24, Daniel P. Berrangé wrote:
> > On Thu, Jun 25, 2020 at 12:14:33PM +0200, Thomas Huth wrote:
> > > On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > > > We have a number of container images in tests/docker/dockerfiles
> > > > that are intended to provide well defined environments for doing
> > > > test builds. We want our CI system to use these containers too.
> > > > 
> > > > This introduces builds of all of them as the first stage in the
> > > > CI, so that the built containers are available for later build
> > > > jobs. The containers are setup to use the GitLab container
> > > > registry as the cache, so we only pay the penalty of the full
> > > > build when the dockerfiles change. The main qemu-project/qemu
> > > > repo is used as a second cache, so that users forking QEMU will
> > > > see a fast turnaround time on their CI jobs.
> > > > 
> > > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > > > ---
> > > >    .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
> > > >    .gitlab-ci.yml              |   3 +
> > > >    2 files changed, 251 insertions(+)
> > > >    create mode 100644 .gitlab-ci.d/containers.yml
> > > > 
> > > > diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> > > > new file mode 100644
> > > > index 0000000000..ea1edbb196
> > > > --- /dev/null
> > > > +++ b/.gitlab-ci.d/containers.yml
> > > > @@ -0,0 +1,248 @@
> > > > +
> > > > +
> > > > +.container_job_template: &container_job_definition
> > > > +  image: docker:stable
> > > > +  stage: containers
> > > > +  services:
> > > > +    - docker:dind
> > > > +  before_script:
> > > > +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> > > > +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> > > > +    - docker info
> > > > +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> > > > +  script:
> > > > +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> > > > +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> > > > +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> > > > +    - docker push "$TAG"
> > > > +  after_script:
> > > > +    - docker logout
> > > 
> > > .gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if
> > > something really has been changed. Could you use something similar here?
> > > E.g.:
> > > 
> > > rules:
> > >   - changes:
> > >     - .gitlab-ci.d/containers.yml
> > >     - tests/docker/*
> > >     - tests/docker/dockerfiles/*
> > > 
> > > ?
> > 
> > If the OS distro base image changes, we'll never pick it up with that
> > kind of filtering.  For the main gitlab.com/qemu-project/qemu  you
> > could configure a nightly/weekly/whatever job to force rebuild on a
> > periodic basis to pick up base image changes.  The downside of this
> > is that any users who fork qemu won't have that periodic job and so
> > will be testing their work against potentially outdated content.
> > 
> > Having said all that, I'm not 100% convinced I'm actually picking
> > up changed base images right now anyway, given our use of caching.
> > 
> > It is possible that I would need todo an explict "docker pull" of
> > the base image to force it to trigger a refresh othrewise I have
> > a feeling we're always cached.
> 
> But currently, each of the container stages currently takes > 2 minutes,
> even with the cached containers. I had a quick look, and it takes 7 minutes
> 'till the "build" stage begins. So all the advantages of not having to do
> "yum/apt-get install" in the build containers anymore seem to be crushed by
> the time that the three container stages take now?

That's still not an apples/apples comparison though. The containers have
many mre packages installed and so test more features. Add all those extra
packages into the original apt-get/dnf commands and then compare times.

I do agree though that I don't much like the 3 containers stages. I set
them up so they would parallelize. ie stage 2 can start without waiting
for entire of stage 1 to finish - only the parent container needs to
finish. The builds stage still waits for all the containers to complete.

We could do similar optimize for the actual build jobs, so they don't have
to wait for all containers to finish - they only need wait for their own
required container to finish.

Ultimately it might be a better tradeoff to not have inheritance between
the containers, and instead just duplicate the common packages in each.
This leads to bigger container images, but simpler builds. Libvirt went
this way for its cross compiler images, just duplicating the shared parts
in each.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH RFC 3/3] gitlab: convert jobs to use custom built containers

[Daniel P. Berrangé]

Now that we're building standard container images from
dockerfiles in tests/docker/dockerfiles, we can convert
the build jobs to use them. The key benefit of this is
that a contributor can now more easily replicate the CI
environment on their local machine. The container images
are cached too, so we are not spending time waiting for
the apt-get/dnf package installs to complete.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.yml | 187 +++++++++++++++++++++----------------------------
 1 file changed, 81 insertions(+), 106 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 72d688875f..3297a402f7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -9,133 +9,108 @@ include:
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
 
-.update_apt_template: &before_script_apt
- before_script:
-  - apt-get update -qq
-  - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make
-        genisoimage
-  - JOBS=$(expr $(nproc) + 1)
-
-.update_dnf_template: &before_script_dnf
- before_script:
-  - dnf update -y
-  - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel
-        make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel
-  - JOBS=$(expr $(nproc) + 1)
+.native_build_job_template: &native_build_job_definition
+  stage: build
+  image: $CI_REGISTRY_IMAGE/ci-$IMAGE:latest
+  before_script:
+    - JOBS=$(expr $(nproc) + 1)
+  script:
+    - mkdir build
+    - cd build
+    - if test -n "$TARGETS";
+      then
+        ../configure --enable-werror $CONFIGURE_ARGS --target-list="$TARGETS" ;
+      else
+        ../configure --enable-werror $CONFIGURE_ARGS ;
+      fi
+    - make -j"$JOBS"
+    - make -j"$JOBS" $MAKE_CHECK_ARGS
 
 build-system1:
- stage: build
- image: ubuntu:19.10
- <<: *before_script_apt
- script:
- - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
-      libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev
- - mkdir build
- - cd build
- - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu
-      cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu
-      mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu"
- - make -j"$JOBS"
- - make -j"$JOBS" check
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: ubuntu2004
+    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu
+      moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu
+      riscv64-softmmu sparc-softmmu
+    MAKE_CHECK_ARGS: check
 
 build-system2:
- stage: build
- image: fedora:latest
- <<: *before_script_dnf
- script:
- - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel
-       libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel
-       libzstd-devel
- - mkdir build
- - cd build
- - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
-      microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu
-      sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu"
- - make -j"$JOBS"
- - make -j"$JOBS" check
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+    TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu
+      riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu
+      xtensa-softmmu nios2-softmmu or1k-softmmu
+    MAKE_CHECK_ARGS: check
 
 build-disabled:
- stage: build
- image: fedora:latest
- <<: *before_script_dnf
- script:
- - mkdir build
- - cd build
- - ../configure --enable-werror --disable-rdma --disable-slirp --disable-curl
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+    CONFIGURE_ARGS: --disable-rdma --disable-slirp --disable-curl
       --disable-capstone --disable-live-block-migration --disable-glusterfs
       --disable-replication --disable-coroutine-pool --disable-smartcard
       --disable-guest-agent --disable-curses --disable-libxml2 --disable-tpm
       --disable-qom-cast-debug --disable-spice --disable-vhost-vsock
       --disable-vhost-net --disable-vhost-crypto --disable-vhost-user
-      --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user"
- - make -j"$JOBS"
- - make -j"$JOBS" check-qtest SPEED=slow
+    TARGETS: i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user
+    MAKE_CHECK_ARGS: check-qtest SPEED=slow
 
 build-tcg-disabled:
- stage: build
- image: centos:8
- <<: *before_script_dnf
- script:
- - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel
- - mkdir build
- - cd build
- - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""
- - make -j"$JOBS"
- - make check-unit
- - make check-qapi-schema
- - cd tests/qemu-iotests/
- - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: centos8
+  script:
+    - mkdir build
+    - cd build
+    - ../configure --disable-tcg --audio-drv-list=""
+    - make -j"$JOBS"
+    - make check-unit
+    - make check-qapi-schema
+    - cd tests/qemu-iotests/
+    - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
             052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163
             170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277
- - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122
+    - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122
             124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202
             208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
- stage: build
- <<: *before_script_apt
- script:
- - mkdir build
- - cd build
- - ../configure --enable-werror --disable-system --disable-guest-agent
-               --disable-capstone --disable-slirp --disable-fdt
- - make -j"$JOBS"
- - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: ubuntu2004
+    CONFIGURE_ARGS: --disable-system --disable-guest-agent
+      --disable-capstone --disable-slirp --disable-fdt
+    MAKE_CHECK_ARGS:  run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
 
 build-clang:
- stage: build
- image: fedora:latest
- <<: *before_script_dnf
- script:
- - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel
-       libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel
- - mkdir build
- - cd build
- - ../configure --cc=clang --cxx=clang++ --enable-werror
-      --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
-                     ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user"
- - make -j"$JOBS"
- - make -j"$JOBS" check
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+    CONFIGURE_ARGS: --cc=clang --cxx=clang++
+    TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
+      ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user
+    MAKE_CHECK_ARGS: check
 
 build-tci:
- stage: build
- image: centos:8
- <<: *before_script_dnf
- script:
- - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
- - mkdir build
- - cd build
- - ../configure --enable-tcg-interpreter
-      --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
- - make -j"$JOBS"
- - make run-tcg-tests-x86_64-softmmu
- - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
- - for tg in $TARGETS ; do
-     export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ;
-     ./tests/qtest/boot-serial-test || exit 1 ;
-     ./tests/qtest/cdrom-test || exit 1 ;
-   done
- - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test
- - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x"
-   ./tests/qtest/pxe-test -m slow
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: fedora
+  script:
+    - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
+    - mkdir build
+    - cd build
+    - ../configure --enable-tcg-interpreter
+        --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
+    - make -j"$JOBS"
+    - make run-tcg-tests-x86_64-softmmu
+    - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
+    - for tg in $TARGETS ; do
+        export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ;
+        ./tests/qtest/boot-serial-test || exit 1 ;
+        ./tests/qtest/cdrom-test || exit 1 ;
+      done
+    - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test
+    - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x" ./tests/qtest/pxe-test -m slow
-- 
2.24.1

Re: [PATCH RFC 3/3] gitlab: convert jobs to use custom built containers

[Thomas Huth]

On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> Now that we're building standard container images from
> dockerfiles in tests/docker/dockerfiles, we can convert
> the build jobs to use them. The key benefit of this is
> that a contributor can now more easily replicate the CI
> environment on their local machine. The container images
> are cached too, so we are not spending time waiting for
> the apt-get/dnf package installs to complete.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.yml | 187 +++++++++++++++++++++----------------------------
>   1 file changed, 81 insertions(+), 106 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 72d688875f..3297a402f7 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -9,133 +9,108 @@ include:
>     - local: '/.gitlab-ci.d/opensbi.yml'
>     - local: '/.gitlab-ci.d/containers.yml'
>   
> -.update_apt_template: &before_script_apt
> - before_script:
> -  - apt-get update -qq
> -  - apt-get install -y -qq git gcc libglib2.0-dev libpixman-1-dev make
> -        genisoimage
> -  - JOBS=$(expr $(nproc) + 1)
> -
> -.update_dnf_template: &before_script_dnf
> - before_script:
> -  - dnf update -y
> -  - dnf install -y bzip2 diffutils gcc git genisoimage findutils glib2-devel
> -        make python3 perl-podlators perl-Test-Harness pixman-devel zlib-devel
> -  - JOBS=$(expr $(nproc) + 1)
> +.native_build_job_template: &native_build_job_definition
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/ci-$IMAGE:latest
> +  before_script:
> +    - JOBS=$(expr $(nproc) + 1)
> +  script:
> +    - mkdir build
> +    - cd build
> +    - if test -n "$TARGETS";
> +      then
> +        ../configure --enable-werror $CONFIGURE_ARGS --target-list="$TARGETS" ;
> +      else
> +        ../configure --enable-werror $CONFIGURE_ARGS ;
> +      fi
> +    - make -j"$JOBS"
> +    - make -j"$JOBS" $MAKE_CHECK_ARGS
>   
>   build-system1:
> - stage: build
> - image: ubuntu:19.10
> - <<: *before_script_apt
> - script:
> - - apt-get install -y -qq libgtk-3-dev libvte-dev nettle-dev libcacard-dev
> -      libusb-dev libvde-dev libspice-protocol-dev libgl1-mesa-dev libvdeplug-dev
> - - mkdir build
> - - cd build
> - - ../configure --enable-werror --target-list="aarch64-softmmu alpha-softmmu
> -      cris-softmmu hppa-softmmu lm32-softmmu moxie-softmmu microblazeel-softmmu
> -      mips64el-softmmu m68k-softmmu ppc-softmmu riscv64-softmmu sparc-softmmu"
> - - make -j"$JOBS"
> - - make -j"$JOBS" check
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: ubuntu2004
> +    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu
> +      moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu
> +      riscv64-softmmu sparc-softmmu
> +    MAKE_CHECK_ARGS: check
>   
>   build-system2:
> - stage: build
> - image: fedora:latest
> - <<: *before_script_dnf
> - script:
> - - yum install -y SDL2-devel libgcrypt-devel brlapi-devel libaio-devel
> -       libfdt-devel lzo-devel librdmacm-devel libibverbs-devel libibumad-devel
> -       libzstd-devel
> - - mkdir build
> - - cd build
> - - ../configure --enable-werror --target-list="tricore-softmmu unicore32-softmmu
> -      microblaze-softmmu mips-softmmu riscv32-softmmu s390x-softmmu sh4-softmmu
> -      sparc64-softmmu x86_64-softmmu xtensa-softmmu nios2-softmmu or1k-softmmu"
> - - make -j"$JOBS"
> - - make -j"$JOBS" check
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: fedora
> +    TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu
> +      riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu
> +      xtensa-softmmu nios2-softmmu or1k-softmmu
> +    MAKE_CHECK_ARGS: check
>   
>   build-disabled:
> - stage: build
> - image: fedora:latest
> - <<: *before_script_dnf
> - script:
> - - mkdir build
> - - cd build
> - - ../configure --enable-werror --disable-rdma --disable-slirp --disable-curl
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: fedora
> +    CONFIGURE_ARGS: --disable-rdma --disable-slirp --disable-curl
>         --disable-capstone --disable-live-block-migration --disable-glusterfs
>         --disable-replication --disable-coroutine-pool --disable-smartcard
>         --disable-guest-agent --disable-curses --disable-libxml2 --disable-tpm
>         --disable-qom-cast-debug --disable-spice --disable-vhost-vsock
>         --disable-vhost-net --disable-vhost-crypto --disable-vhost-user
> -      --target-list="i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user"
> - - make -j"$JOBS"
> - - make -j"$JOBS" check-qtest SPEED=slow
> +    TARGETS: i386-softmmu ppc64-softmmu mips64-softmmu i386-linux-user
> +    MAKE_CHECK_ARGS: check-qtest SPEED=slow
>   
>   build-tcg-disabled:
> - stage: build
> - image: centos:8
> - <<: *before_script_dnf
> - script:
> - - dnf install -y clang gtk3-devel libusbx-devel libgcrypt-devel
> - - mkdir build
> - - cd build
> - - ../configure --cc=clang --enable-werror --disable-tcg --audio-drv-list=""
> - - make -j"$JOBS"
> - - make check-unit
> - - make check-qapi-schema
> - - cd tests/qemu-iotests/
> - - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: centos8
> +  script:
> +    - mkdir build
> +    - cd build
> +    - ../configure --disable-tcg --audio-drv-list=""
> +    - make -j"$JOBS"
> +    - make check-unit
> +    - make check-qapi-schema
> +    - cd tests/qemu-iotests/
> +    - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
>               052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163
>               170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277
> - - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122
> +    - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122
>               124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202
>               208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258
>               260 261 262 263 264 270 272 273 277 279
>   
>   build-user:
> - stage: build
> - <<: *before_script_apt
> - script:
> - - mkdir build
> - - cd build
> - - ../configure --enable-werror --disable-system --disable-guest-agent
> -               --disable-capstone --disable-slirp --disable-fdt
> - - make -j"$JOBS"
> - - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: ubuntu2004
> +    CONFIGURE_ARGS: --disable-system --disable-guest-agent
> +      --disable-capstone --disable-slirp --disable-fdt
> +    MAKE_CHECK_ARGS:  run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
>   
>   build-clang:
> - stage: build
> - image: fedora:latest
> - <<: *before_script_dnf
> - script:
> - - yum install -y clang SDL2-devel libattr-devel libcap-ng-devel xfsprogs-devel
> -       libiscsi-devel libnfs-devel libseccomp-devel gnutls-devel librbd-devel
> - - mkdir build
> - - cd build
> - - ../configure --cc=clang --cxx=clang++ --enable-werror
> -      --target-list="alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
> -                     ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user"
> - - make -j"$JOBS"
> - - make -j"$JOBS" check
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: fedora
> +    CONFIGURE_ARGS: --cc=clang --cxx=clang++
> +    TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
> +      ppc-softmmu s390x-softmmu x86_64-softmmu arm-linux-user
> +    MAKE_CHECK_ARGS: check
>   
>   build-tci:
> - stage: build
> - image: centos:8
> - <<: *before_script_dnf
> - script:
> - - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
> - - mkdir build
> - - cd build
> - - ../configure --enable-tcg-interpreter
> -      --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
> - - make -j"$JOBS"
> - - make run-tcg-tests-x86_64-softmmu
> - - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
> - - for tg in $TARGETS ; do
> -     export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ;
> -     ./tests/qtest/boot-serial-test || exit 1 ;
> -     ./tests/qtest/cdrom-test || exit 1 ;
> -   done
> - - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test
> - - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x"
> -   ./tests/qtest/pxe-test -m slow
> +  <<: *native_build_job_definition
> +  variables:
> +    IMAGE: fedora
> +  script:
> +    - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
> +    - mkdir build
> +    - cd build
> +    - ../configure --enable-tcg-interpreter
> +        --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)"
> +    - make -j"$JOBS"
> +    - make run-tcg-tests-x86_64-softmmu
> +    - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
> +    - for tg in $TARGETS ; do
> +        export QTEST_QEMU_BINARY="${tg}-softmmu/qemu-system-${tg}" ;
> +        ./tests/qtest/boot-serial-test || exit 1 ;
> +        ./tests/qtest/cdrom-test || exit 1 ;
> +      done
> +    - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64" ./tests/qtest/pxe-test
> +    - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x" ./tests/qtest/pxe-test -m slow
> 

Looks pretty cool, thanks for doing this!

Acked-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH RFC 0/3] gitlab: build containers to use in build jobs

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> The current gitlab CI jobs are quite inefficient because they
> use the generic distro images and then apt-get/dnf install
> extra packages every time.
<snip>

I should say I've queued this into testing/next and tweaked it slightly.

-- 
Alex Bennée

Re: [PATCH RFC 0/3] gitlab: build containers to use in build jobs

[Thomas Huth]

On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> The current gitlab CI jobs are quite inefficient because they
> use the generic distro images and then apt-get/dnf install
> extra packages every time.
> 
> The other downside is that the container environment used is
> only defined in thte .gitlab-ci.yml file, so it tedious to
> reproduce locally.
> 
> We already have containers defined in tests/docker for use by
> developers building locally. We can use these for CI systems
> too if we just had a way to build them....
> 
> ...GitLab CI offers such a way. We can use docker-in-docker
> to build the images at the start of the CI cycle, and use
> the built images in later jobs.
> 
> These later jobs are now faster because they're not having
> to install any software.

Did you see any speed-up? I had a look at some pipelines, and it seems 
to me that they rather got slower now? For example, this is the system1 
pipeline before your change:

  https://gitlab.com/huth/qemu/-/jobs/610924897

and after your change:

  https://gitlab.com/huth/qemu/-/jobs/611069374

Duration went up from 35 minutes to 42 minutes.

Seems also to happen in your builds, before the change:

  https://gitlab.com/berrange/qemu/-/jobs/582995084

and after the change:

  https://gitlab.com/berrange/qemu/-/jobs/606175927

... went from 36 minutes up to 42 minutes.

Could be a coincidence due to the load on the shared runners, but it 
looks at least a little bit suspicious...

  Thomas

Re: [PATCH RFC 0/3] gitlab: build containers to use in build jobs

[Daniel P. Berrangé]

On Thu, Jun 25, 2020 at 01:15:52PM +0200, Thomas Huth wrote:
> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > The current gitlab CI jobs are quite inefficient because they
> > use the generic distro images and then apt-get/dnf install
> > extra packages every time.
> > 
> > The other downside is that the container environment used is
> > only defined in thte .gitlab-ci.yml file, so it tedious to
> > reproduce locally.
> > 
> > We already have containers defined in tests/docker for use by
> > developers building locally. We can use these for CI systems
> > too if we just had a way to build them....
> > 
> > ...GitLab CI offers such a way. We can use docker-in-docker
> > to build the images at the start of the CI cycle, and use
> > the built images in later jobs.
> > 
> > These later jobs are now faster because they're not having
> > to install any software.
> 
> Did you see any speed-up? I had a look at some pipelines, and it seems to me
> that they rather got slower now? For example, this is the system1 pipeline
> before your change:
> 
>  https://gitlab.com/huth/qemu/-/jobs/610924897
> 
> and after your change:
> 
>  https://gitlab.com/huth/qemu/-/jobs/611069374
> 
> Duration went up from 35 minutes to 42 minutes.
> 
> Seems also to happen in your builds, before the change:
> 
>  https://gitlab.com/berrange/qemu/-/jobs/582995084
> 
> and after the change:
> 
>  https://gitlab.com/berrange/qemu/-/jobs/606175927
> 
> ... went from 36 minutes up to 42 minutes.
> 
> Could be a coincidence due to the load on the shared runners, but it looks
> at least a little bit suspicious...

I think the difference is because we're building more features now. The
dockerfiles have provided more build pre-requisites that the old gitlab
recipe did.

If you compare the configure summary, I see the new build now covers
SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs
and a few more.  So we've saved time by not intsallling many packages
each time, but consumed a greater amount of time by compiling more
features.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 0/3] gitlab: build containers to use in build jobs

[Daniel P. Berrangé]

On Thu, Jun 25, 2020 at 12:26:53PM +0100, Daniel P. Berrangé wrote:
> On Thu, Jun 25, 2020 at 01:15:52PM +0200, Thomas Huth wrote:
> > On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > > The current gitlab CI jobs are quite inefficient because they
> > > use the generic distro images and then apt-get/dnf install
> > > extra packages every time.
> > > 
> > > The other downside is that the container environment used is
> > > only defined in thte .gitlab-ci.yml file, so it tedious to
> > > reproduce locally.
> > > 
> > > We already have containers defined in tests/docker for use by
> > > developers building locally. We can use these for CI systems
> > > too if we just had a way to build them....
> > > 
> > > ...GitLab CI offers such a way. We can use docker-in-docker
> > > to build the images at the start of the CI cycle, and use
> > > the built images in later jobs.
> > > 
> > > These later jobs are now faster because they're not having
> > > to install any software.
> > 
> > Did you see any speed-up? I had a look at some pipelines, and it seems to me
> > that they rather got slower now? For example, this is the system1 pipeline
> > before your change:
> > 
> >  https://gitlab.com/huth/qemu/-/jobs/610924897
> > 
> > and after your change:
> > 
> >  https://gitlab.com/huth/qemu/-/jobs/611069374
> > 
> > Duration went up from 35 minutes to 42 minutes.
> > 
> > Seems also to happen in your builds, before the change:
> > 
> >  https://gitlab.com/berrange/qemu/-/jobs/582995084
> > 
> > and after the change:
> > 
> >  https://gitlab.com/berrange/qemu/-/jobs/606175927
> > 
> > ... went from 36 minutes up to 42 minutes.
> > 
> > Could be a coincidence due to the load on the shared runners, but it looks
> > at least a little bit suspicious...
> 
> I think the difference is because we're building more features now. The
> dockerfiles have provided more build pre-requisites that the old gitlab
> recipe did.
> 
> If you compare the configure summary, I see the new build now covers
> SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs
> and a few more.  So we've saved time by not intsallling many packages
> each time, but consumed a greater amount of time by compiling more
> features.

Oh a missed a lot more actually - there's also spice, opengl, libiscsi,
libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev
too.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 0/3] gitlab: build containers to use in build jobs

[Thomas Huth]

On 25/06/2020 13.29, Daniel P. Berrangé wrote:
> On Thu, Jun 25, 2020 at 12:26:53PM +0100, Daniel P. Berrangé wrote:
>> On Thu, Jun 25, 2020 at 01:15:52PM +0200, Thomas Huth wrote:
>>> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
>>>> The current gitlab CI jobs are quite inefficient because they
>>>> use the generic distro images and then apt-get/dnf install
>>>> extra packages every time.
>>>>
>>>> The other downside is that the container environment used is
>>>> only defined in thte .gitlab-ci.yml file, so it tedious to
>>>> reproduce locally.
>>>>
>>>> We already have containers defined in tests/docker for use by
>>>> developers building locally. We can use these for CI systems
>>>> too if we just had a way to build them....
>>>>
>>>> ...GitLab CI offers such a way. We can use docker-in-docker
>>>> to build the images at the start of the CI cycle, and use
>>>> the built images in later jobs.
>>>>
>>>> These later jobs are now faster because they're not having
>>>> to install any software.
>>>
>>> Did you see any speed-up? I had a look at some pipelines, and it seems to me
>>> that they rather got slower now? For example, this is the system1 pipeline
>>> before your change:
>>>
>>>   https://gitlab.com/huth/qemu/-/jobs/610924897
>>>
>>> and after your change:
>>>
>>>   https://gitlab.com/huth/qemu/-/jobs/611069374
>>>
>>> Duration went up from 35 minutes to 42 minutes.
>>>
>>> Seems also to happen in your builds, before the change:
>>>
>>>   https://gitlab.com/berrange/qemu/-/jobs/582995084
>>>
>>> and after the change:
>>>
>>>   https://gitlab.com/berrange/qemu/-/jobs/606175927
>>>
>>> ... went from 36 minutes up to 42 minutes.
>>>
>>> Could be a coincidence due to the load on the shared runners, but it looks
>>> at least a little bit suspicious...
>>
>> I think the difference is because we're building more features now. The
>> dockerfiles have provided more build pre-requisites that the old gitlab
>> recipe did.
>>
>> If you compare the configure summary, I see the new build now covers
>> SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs
>> and a few more.  So we've saved time by not intsallling many packages
>> each time, but consumed a greater amount of time by compiling more
>> features.
> 
> Oh a missed a lot more actually - there's also spice, opengl, libiscsi,
> libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev
> too.

Ok, that's fair, I think it's ok to spend some additional minutes for 
the extended test coverage here.

  Thomas

Re: [PATCH RFC 0/3] gitlab: build containers to use in build jobs

[Daniel P. Berrangé]

On Thu, Jun 25, 2020 at 01:33:42PM +0200, Thomas Huth wrote:
> On 25/06/2020 13.29, Daniel P. Berrangé wrote:
> > On Thu, Jun 25, 2020 at 12:26:53PM +0100, Daniel P. Berrangé wrote:
> > > On Thu, Jun 25, 2020 at 01:15:52PM +0200, Thomas Huth wrote:
> > > > On 22/06/2020 17.33, Daniel P. Berrangé wrote:
> > > > > The current gitlab CI jobs are quite inefficient because they
> > > > > use the generic distro images and then apt-get/dnf install
> > > > > extra packages every time.
> > > > > 
> > > > > The other downside is that the container environment used is
> > > > > only defined in thte .gitlab-ci.yml file, so it tedious to
> > > > > reproduce locally.
> > > > > 
> > > > > We already have containers defined in tests/docker for use by
> > > > > developers building locally. We can use these for CI systems
> > > > > too if we just had a way to build them....
> > > > > 
> > > > > ...GitLab CI offers such a way. We can use docker-in-docker
> > > > > to build the images at the start of the CI cycle, and use
> > > > > the built images in later jobs.
> > > > > 
> > > > > These later jobs are now faster because they're not having
> > > > > to install any software.
> > > > 
> > > > Did you see any speed-up? I had a look at some pipelines, and it seems to me
> > > > that they rather got slower now? For example, this is the system1 pipeline
> > > > before your change:
> > > > 
> > > >   https://gitlab.com/huth/qemu/-/jobs/610924897
> > > > 
> > > > and after your change:
> > > > 
> > > >   https://gitlab.com/huth/qemu/-/jobs/611069374
> > > > 
> > > > Duration went up from 35 minutes to 42 minutes.
> > > > 
> > > > Seems also to happen in your builds, before the change:
> > > > 
> > > >   https://gitlab.com/berrange/qemu/-/jobs/582995084
> > > > 
> > > > and after the change:
> > > > 
> > > >   https://gitlab.com/berrange/qemu/-/jobs/606175927
> > > > 
> > > > ... went from 36 minutes up to 42 minutes.
> > > > 
> > > > Could be a coincidence due to the load on the shared runners, but it looks
> > > > at least a little bit suspicious...
> > > 
> > > I think the difference is because we're building more features now. The
> > > dockerfiles have provided more build pre-requisites that the old gitlab
> > > recipe did.
> > > 
> > > If you compare the configure summary, I see the new build now covers
> > > SDL, curses, curl, pulseaudio, virtiofs, SASL, libjpeg, xen, docs
> > > and a few more.  So we've saved time by not intsallling many packages
> > > each time, but consumed a greater amount of time by compiling more
> > > features.
> > 
> > Oh a missed a lot more actually - there's also spice, opengl, libiscsi,
> > libnfs, libusb, seccomp, libssh, lzo, snappy, bzip, zstd, numa and udev
> > too.
> 
> Ok, that's fair, I think it's ok to spend some additional minutes for the
> extended test coverage here.

Unlike Travis which limits to 5 concurrent jobs,  GitLab doesn't seem to
have a fixed concurrency limit (at least I've not managed to hit one yet).
So if we want faster overall build time, we have alot more scope for
splitting jobs in half and making use of much better parallelism in CI
to get this to complete sooner.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|