[RFC XEN PATCH 0/7] automation, RFC prototype, Have GitLab CI built its own containers

[RFC XEN PATCH 0/7] automation, RFC prototype, Have GitLab CI built its own containers

[Anthony PERARD]

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-auto-rebuild-v1

Hi,

I have done some research to be able to build containers in the CI. This works
only for x86 containers as I've setup only a single x86 gitlab-runner to be
able to run docker-in-docker.

The runner is setup to only accept jobs from a branch that is "protected" in
gitlab. Also, one need credential to push to the container register, those are
called "Group deploy tokens", and I've set the variables CI_DEPLOY_USER and
CI_DEPLOY_PASSWORD in the project "xen-project/xen" (variables only visible on
a pipeline running on a protected branch).

These patch introduce quite a lot of redundancies in jobs, 2 new jobs per
containers which build/push containers, and duplicate most of build.yaml.
This mean that if we go with this, we have to duplicate and keep in sync many
jobs.

To avoid having to do the duplicated jobs by hand, I could look at
creating a script that use "build.yaml" as input and produce the 3
stages needed to update a container, but that script would need to be
run by hand, as gitlab can't really use it, unless ..

I've look at generated pipeline, and they look like this in gitlab:
    https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/777665738
But the result of the generated/child pipeline doesn't seems to be taken into
account in the original pipeline, which make me think that we can't use them to
generate "build.yaml". But maybe the could be use for generating the pipeline
that will update a container.
Doc:
    https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines

So, with all of this, is it reasonable to test containers before
pushing them to production? Or is it to much work? We could simply have jobs
tacking care of rebuilding a container and pushing them to production without
testing.

An example with the variable DO_REBUILD_CONTAINER and PUSH_CONTAINER set (and
existing build/test jobs disabled):
    https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/791711467

Cheers,

Anthony PERARD (7):
  automation: Automatically rebuild debian:unstable container
  automation: Introduce test-containers stage
  automation: Add a template per container for build jobs.
  automation: Adding containers build jobs and test of thoses
  automation: Introduce DO_REBUILD_CONTAINER, to allow to rebuild a
    container
  automation: Push container been tested
  automation: Add some more push containers jobs

 .gitlab-ci.yml                            |   6 +
 automation/build/Makefile                 |  14 +-
 automation/gitlab-ci/build.yaml           | 327 ++++++++------
 automation/gitlab-ci/containers.yaml      |  98 +++++
 automation/gitlab-ci/push-containers.yaml |  79 ++++
 automation/gitlab-ci/test-containers.yaml | 496 ++++++++++++++++++++++
 6 files changed, 894 insertions(+), 126 deletions(-)
 create mode 100644 automation/gitlab-ci/containers.yaml
 create mode 100644 automation/gitlab-ci/push-containers.yaml
 create mode 100644 automation/gitlab-ci/test-containers.yaml

-- 
Anthony PERARD

[RFC XEN PATCH 1/7] automation: Automatically rebuild debian:unstable container

[Anthony PERARD]

Only run this on the staging branch, whenever the dockerfile changes.

Allow to set a suffix when building containers, to be able to test it
before changing the one in production.

Using "rules" instead of "only" as this allow to use variables in the
"changes" rules. Also, "rules" is the preferred keyword as
"only/except" isn't being actively developed in GitLab.

Use $CI_PIPELINE_SOURCE==push to evaluate "rules:changes" only on
push. In most other cases, "rules:changes" evaluate to true so
checking CI_PIPELINE_SOURCE is important.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
 .gitlab-ci.yml                       |  2 ++
 automation/build/Makefile            |  4 ++--
 automation/gitlab-ci/containers.yaml | 25 +++++++++++++++++++++++++
 3 files changed, 29 insertions(+), 2 deletions(-)
 create mode 100644 automation/gitlab-ci/containers.yaml

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c8bd7519d5..c5d499b321 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,7 +1,9 @@
 stages:
   - build
   - test
+  - containers
 
 include:
   - 'automation/gitlab-ci/build.yaml'
   - 'automation/gitlab-ci/test.yaml'
+  - 'automation/gitlab-ci/containers.yaml'
diff --git a/automation/build/Makefile b/automation/build/Makefile
index 4df43b0407..5515938878 100644
--- a/automation/build/Makefile
+++ b/automation/build/Makefile
@@ -16,9 +16,9 @@ help:
 include yocto/yocto.inc
 
 %: %.dockerfile ## Builds containers
-	$(DOCKER_CMD) build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
+	$(DOCKER_CMD) build --pull -t $(REGISTRY)/$(@D):$(@F)$(BUILD_CONTAINER_SUFFIX) -f $< $(<D)
 	@if [ ! -z $${PUSH+x} ]; then \
-		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
+		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F)$(BUILD_CONTAINER_SUFFIX); \
 	fi
 
 .PHONY: all clean
diff --git a/automation/gitlab-ci/containers.yaml b/automation/gitlab-ci/containers.yaml
new file mode 100644
index 0000000000..ace93eaccf
--- /dev/null
+++ b/automation/gitlab-ci/containers.yaml
@@ -0,0 +1,25 @@
+.container-build-tmpl:
+  stage: containers
+  image: docker:stable
+  tags:
+    - container-builder
+  rules:
+    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "staging"
+      changes:
+        - automation/build/${BUILD_CONTAINER}.dockerfile
+  services:
+    - docker:dind
+  before_script:
+    - apk add make
+    - docker info
+    - docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY
+  script:
+    - make -C automation/build ${BUILD_CONTAINER} PUSH=1 BUILD_CONTAINER_SUFFIX=-test
+  after_script:
+    - docker logout
+
+debian-unstable-container:
+  extends:
+    - .container-build-tmpl
+  variables:
+    BUILD_CONTAINER: debian/unstable
-- 
Anthony PERARD

[RFC XEN PATCH 2/7] automation: Introduce test-containers stage

[Anthony PERARD]

Jobs in the "test-containers" stage will be used to check that the
newly built container is working fine, and that it could be used in
production.

Need to rename jobs name compared to "build.yaml", adding "test-"
prefix to all build jobs.

Need also to rename templates as many of them are used with "extends"
which look for "jobs" and template across all the yaml files. Mostly
change "build" to "testbuild".

Introduce a job template per container, as we've got three
"variables", CONTAINER, BUILD_CONTAINER, and a job dependency.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---

Notes:
    It is probably possible to share many of the templates with
    "build.yaml", by changing some of the templates and the way link between
    them.

 .gitlab-ci.yml                            |   2 +
 automation/gitlab-ci/test-containers.yaml | 239 ++++++++++++++++++++++
 2 files changed, 241 insertions(+)
 create mode 100644 automation/gitlab-ci/test-containers.yaml

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c5d499b321..ed5383ab50 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -2,8 +2,10 @@ stages:
   - build
   - test
   - containers
+  - test-containers
 
 include:
   - 'automation/gitlab-ci/build.yaml'
   - 'automation/gitlab-ci/test.yaml'
   - 'automation/gitlab-ci/containers.yaml'
+  - 'automation/gitlab-ci/test-containers.yaml'
diff --git a/automation/gitlab-ci/test-containers.yaml b/automation/gitlab-ci/test-containers.yaml
new file mode 100644
index 0000000000..5dbf3902ff
--- /dev/null
+++ b/automation/gitlab-ci/test-containers.yaml
@@ -0,0 +1,239 @@
+.testbuild-tmpl: &testbuild
+  stage: test-containers
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}-test
+  script:
+    - ./automation/scripts/build 2>&1 | tee build.log
+  artifacts:
+    paths:
+      - binaries/
+      - xen-config
+      - '*.log'
+      - '*/*.log'
+    when: always
+  rules: !reference [.container-build-tmpl, rules]
+
+.gcc-tmpl:
+  variables: &gcc
+    CC: gcc
+    CXX: g++
+
+.clang-tmpl:
+  variables: &clang
+    CC: clang
+    CXX: clang++
+    clang: y
+
+.clang-8-tmpl:
+  variables: &clang-8
+    CC: clang-8
+    CXX: clang++-8
+    LD: ld.lld-8
+    clang: y
+
+.x86-64-testbuild-tmpl:
+  <<: *testbuild
+  variables:
+    XEN_TARGET_ARCH: x86_64
+  tags:
+    - x86_64
+
+.x86-64-testbuild:
+  extends: .x86-64-testbuild-tmpl
+  variables:
+    debug: n
+
+.x86-64-testbuild-debug:
+  extends: .x86-64-testbuild-tmpl
+  variables:
+    debug: y
+
+.x86-32-testbuild-tmpl:
+  <<: *testbuild
+  variables:
+    XEN_TARGET_ARCH: x86_32
+  tags:
+    - x86_32
+
+.x86-32-testbuild:
+  extends: .x86-32-testbuild-tmpl
+  variables:
+    debug: n
+
+.x86-32-testbuild-debug:
+  extends: .x86-32-testbuild-tmpl
+  variables:
+    debug: y
+
+.gcc-x86-64-testbuild:
+  extends: .x86-64-testbuild
+  variables:
+    <<: *gcc
+
+.gcc-x86-64-testbuild-debug:
+  extends: .x86-64-testbuild-debug
+  variables:
+    <<: *gcc
+
+.gcc-x86-32-testbuild:
+  extends: .x86-32-testbuild
+  variables:
+    <<: *gcc
+
+.gcc-x86-32-testbuild-debug:
+  extends: .x86-32-testbuild-debug
+  variables:
+    <<: *gcc
+
+.clang-x86-64-testbuild:
+  extends: .x86-64-testbuild
+  variables:
+    <<: *clang
+
+.clang-x86-64-testbuild-debug:
+  extends: .x86-64-testbuild-debug
+  variables:
+    <<: *clang
+
+.clang-8-x86-64-testbuild:
+  extends: .x86-64-testbuild
+  variables:
+    <<: *clang-8
+
+.clang-8-x86-64-testbuild-debug:
+  extends: .x86-64-testbuild-debug
+  variables:
+    <<: *clang-8
+
+.clang-x86-32-testbuild:
+  extends: .x86-32-testbuild
+  variables:
+    <<: *clang
+
+.clang-x86-32-testbuild-debug:
+  extends: .x86-32-testbuild-debug
+  variables:
+    <<: *clang
+
+.arm32-cross-testbuild-tmpl:
+  <<: *testbuild
+  variables:
+    XEN_TARGET_ARCH: arm32
+  tags:
+    - x86_64
+
+.arm32-cross-testbuild:
+  extends: .arm32-cross-testbuild-tmpl
+  variables:
+    debug: n
+
+.arm32-cross-testbuild-debug:
+  extends: .arm32-cross-testbuild-tmpl
+  variables:
+    debug: y
+
+.gcc-arm32-cross-testbuild:
+  extends: .arm32-cross-testbuild
+  variables:
+    <<: *gcc
+
+.gcc-arm32-cross-testbuild-debug:
+  extends: .arm32-cross-testbuild-debug
+  variables:
+    <<: *gcc
+
+.arm64-testbuild-tmpl:
+  <<: *testbuild
+  variables:
+    XEN_TARGET_ARCH: arm64
+  tags:
+    - arm64
+
+.arm64-testbuild:
+  extends: .arm64-testbuild-tmpl
+  variables:
+    debug: n
+
+.arm64-testbuild-debug:
+  extends: .arm64-testbuild-tmpl
+  variables:
+    debug: y
+
+.gcc-arm64-testbuild:
+  extends: .arm64-testbuild
+  variables:
+    <<: *gcc
+
+.gcc-arm64-testbuild-debug:
+  extends: .arm64-testbuild-debug
+  variables:
+    <<: *gcc
+
+.riscv64-cross-testbuild-tmpl:
+  <<: *testbuild
+  variables:
+    XEN_TARGET_ARCH: riscv64
+  tags:
+    - x86_64
+
+.riscv64-cross-testbuild:
+  extends: .riscv64-cross-testbuild-tmpl
+  variables:
+    debug: n
+
+.riscv64-cross-testbuild-debug:
+  extends: .riscv64-cross-testbuild-tmpl
+  variables:
+    debug: y
+
+.gcc-riscv64-cross-testbuild:
+  extends: .riscv64-cross-testbuild
+  variables:
+    <<: *gcc
+
+.gcc-riscv64-cross-testbuild-debug:
+  extends: .riscv64-cross-testbuild-debug
+  variables:
+    <<: *gcc
+
+# Jobs below this line
+
+.container-debian-unstable-testtmpl:
+  variables:
+    CONTAINER: debian:unstable
+    BUILD_CONTAINER: debian/unstable
+  needs:
+    - debian-unstable-container
+
+test-debian-unstable-clang:
+  extends:
+    - .clang-x86-64-testbuild
+    - .container-debian-unstable-testtmpl
+
+test-debian-unstable-clang-debug:
+  extends:
+    - .clang-x86-64-testbuild-debug
+    - .container-debian-unstable-testtmpl
+
+test-debian-unstable-gcc:
+  extends:
+    - .gcc-x86-64-testbuild
+    - .container-debian-unstable-testtmpl
+
+test-debian-unstable-gcc-debug:
+  extends:
+    - .gcc-x86-64-testbuild-debug
+    - .container-debian-unstable-testtmpl
+
+test-debian-unstable-gcc-randconfig:
+  extends:
+    - .gcc-x86-64-testbuild
+    - .container-debian-unstable-testtmpl
+  variables:
+    RANDCONFIG: y
+
+test-debian-unstable-gcc-debug-randconfig:
+  extends:
+    - .gcc-x86-64-testbuild-debug
+    - .container-debian-unstable-testtmpl
+  variables:
+    RANDCONFIG: y
-- 
Anthony PERARD

[RFC XEN PATCH 3/7] automation: Add a template per container for build jobs.

[Anthony PERARD]

Have one template per container, which each build job will extend.
This will allow to add more variable which are linked to a used
container.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
 automation/gitlab-ci/build.yaml | 327 ++++++++++++++++++++------------
 1 file changed, 202 insertions(+), 125 deletions(-)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 38bb22d860..1f186bf346 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -255,285 +255,346 @@
 
 # Jobs below this line
 
-archlinux-gcc:
-  extends: .gcc-x86-64-build
+.container-archlinux:
   variables:
     CONTAINER: archlinux:current
 
+archlinux-gcc:
+  extends:
+    - .gcc-x86-64-build
+    - .container-archlinux
+
 archlinux-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: archlinux:current
+  extends:
+    - .gcc-x86-64-build-debug
+    - .container-archlinux
 
-centos-7-gcc:
-  extends: .gcc-x86-64-build
+.container-centos-7:
   variables:
     CONTAINER: centos:7
 
+centos-7-gcc:
+  extends:
+    - .gcc-x86-64-build
+    - .container-centos-7
+
 centos-7-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: centos:7
+  extends:
+    - .gcc-x86-64-build-debug
+    - .container-centos-7
 
-debian-stretch-clang:
-  extends: .clang-x86-64-build
+.container-debian-stretch:
   variables:
     CONTAINER: debian:stretch
 
+debian-stretch-clang:
+  extends:
+    - .clang-x86-64-build
+    - .container-debian-stretch
+
 debian-stretch-clang-debug:
-  extends: .clang-x86-64-build-debug
-  variables:
-    CONTAINER: debian:stretch
+  extends:
+    - .clang-x86-64-build-debug
+    - .container-debian-stretch
 
 debian-stretch-clang-8:
-  extends: .clang-8-x86-64-build
-  variables:
-    CONTAINER: debian:stretch
+  extends:
+    - .clang-8-x86-64-build
+    - .container-debian-stretch
 
 debian-stretch-clang-8-debug:
-  extends: .clang-8-x86-64-build-debug
-  variables:
-    CONTAINER: debian:stretch
+  extends:
+    - .clang-8-x86-64-build-debug
+    - .container-debian-stretch
 
 debian-stretch-gcc:
-  extends: .gcc-x86-64-build
-  variables:
-    CONTAINER: debian:stretch
+  extends:
+    - .gcc-x86-64-build
+    - .container-debian-stretch
 
 debian-stretch-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: debian:stretch
+  extends:
+    - .gcc-x86-64-build-debug
+    - .container-debian-stretch
 
-debian-stretch-32-clang-debug:
-  extends: .clang-x86-32-build-debug
+.container-debian-stretch-32:
   variables:
     CONTAINER: debian:stretch-i386
 
+debian-stretch-32-clang-debug:
+  extends:
+    - .clang-x86-32-build-debug
+    - .container-debian-stretch-32
+
 debian-stretch-32-gcc-debug:
-  extends: .gcc-x86-32-build-debug
-  variables:
-    CONTAINER: debian:stretch-i386
+  extends:
+    - .gcc-x86-32-build-debug
+    - .container-debian-stretch-32
 
 debian-buster-gcc-ibt:
-  extends: .gcc-x86-64-build
+  extends:
+    - .gcc-x86-64-build
   variables:
     CONTAINER: debian:buster-gcc-ibt
     RANDCONFIG: y
     EXTRA_FIXED_RANDCONFIG: |
       CONFIG_XEN_IBT=y
 
-debian-unstable-clang:
-  extends: .clang-x86-64-build
+.container-debian-unstable:
   variables:
     CONTAINER: debian:unstable
 
+debian-unstable-clang:
+  extends:
+    - .clang-x86-64-build
+    - .container-debian-unstable
+
 debian-unstable-clang-debug:
-  extends: .clang-x86-64-build-debug
-  variables:
-    CONTAINER: debian:unstable
+  extends:
+    - .clang-x86-64-build-debug
+    - .container-debian-unstable
 
 debian-unstable-gcc:
-  extends: .gcc-x86-64-build
-  variables:
-    CONTAINER: debian:unstable
+  extends:
+    - .gcc-x86-64-build
+    - .container-debian-unstable
 
 debian-unstable-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: debian:unstable
+  extends:
+    - .gcc-x86-64-build-debug
+    - .container-debian-unstable
 
 debian-unstable-gcc-randconfig:
-  extends: .gcc-x86-64-build
+  extends:
+    - .gcc-x86-64-build
+    - .container-debian-unstable
   variables:
-    CONTAINER: debian:unstable
     RANDCONFIG: y
 
 debian-unstable-gcc-debug-randconfig:
-  extends: .gcc-x86-64-build-debug
+  extends:
+    - .gcc-x86-64-build-debug
+    - .container-debian-unstable
   variables:
-    CONTAINER: debian:unstable
     RANDCONFIG: y
 
-debian-unstable-32-clang-debug:
-  extends: .clang-x86-32-build-debug
+.container-debian-unstable-32:
   variables:
     CONTAINER: debian:unstable-i386
 
+debian-unstable-32-clang-debug:
+  extends:
+    - .clang-x86-32-build-debug
+    - .container-debian-unstable-32
+
 debian-unstable-32-gcc-debug:
-  extends: .gcc-x86-32-build-debug
-  variables:
-    CONTAINER: debian:unstable-i386
+  extends:
+    - .gcc-x86-32-build-debug
+    - .container-debian-unstable-32
 
 fedora-gcc:
-  extends: .gcc-x86-64-build
+  extends:
+    - .gcc-x86-64-build
   variables:
     CONTAINER: fedora:29
 
 fedora-gcc-debug:
-  extends: .gcc-x86-64-build-debug
+  extends:
+    - .gcc-x86-64-build-debug
   variables:
     CONTAINER: fedora:29
 
 # Ubuntu Trusty's Clang is 3.4 while Xen requires 3.5
 
-ubuntu-trusty-gcc:
-  extends: .gcc-x86-64-build
+.container-ubuntu-trusty:
   variables:
     CONTAINER: ubuntu:trusty
 
+ubuntu-trusty-gcc:
+  extends:
+    - .gcc-x86-64-build
+    - .container-ubuntu-trusty
+
 ubuntu-trusty-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:trusty
+  extends:
+    - .gcc-x86-64-build-debug
+    - .container-ubuntu-trusty
 
-ubuntu-xenial-clang:
-  extends: .clang-x86-64-build
+.container-ubuntu-xenial:
   variables:
     CONTAINER: ubuntu:xenial
 
+ubuntu-xenial-clang:
+  extends:
+    - .clang-x86-64-build
+    - .container-ubuntu-xenial
+
 ubuntu-xenial-clang-debug:
-  extends: .clang-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:xenial
+  extends:
+    - .clang-x86-64-build-debug
+    - .container-ubuntu-xenial
 
 ubuntu-xenial-gcc:
-  extends: .gcc-x86-64-build
-  variables:
-    CONTAINER: ubuntu:xenial
+  extends:
+    - .gcc-x86-64-build
+    - .container-ubuntu-xenial
 
 ubuntu-xenial-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:xenial
+  extends:
+    - .gcc-x86-64-build-debug
+    - .container-ubuntu-xenial
 
-ubuntu-bionic-clang:
-  extends: .clang-x86-64-build
+.container-ubuntu-bionic:
   variables:
     CONTAINER: ubuntu:bionic
 
+ubuntu-bionic-clang:
+  extends:
+    - .clang-x86-64-build
+    - .container-ubuntu-bionic
+
 ubuntu-bionic-clang-debug:
-  extends: .clang-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:bionic
+  extends:
+    - .clang-x86-64-build-debug
+    - .container-ubuntu-bionic
 
 ubuntu-bionic-gcc:
-  extends: .gcc-x86-64-build
-  variables:
-    CONTAINER: ubuntu:bionic
+  extends:
+    - .gcc-x86-64-build
+    - .container-ubuntu-bionic
 
 ubuntu-bionic-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:bionic
+  extends:
+    - .gcc-x86-64-build-debug
+    - .container-ubuntu-bionic
 
-ubuntu-focal-gcc:
-  extends: .gcc-x86-64-build
+.container-ubuntu-focal:
   variables:
     CONTAINER: ubuntu:focal
 
+ubuntu-focal-gcc:
+  extends:
+    - .gcc-x86-64-build
+    - .container-ubuntu-focal
+
 ubuntu-focal-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:focal
+  extends:
+    - .gcc-x86-64-build-debug
+    - .container-ubuntu-focal
 
 ubuntu-focal-clang:
-  extends: .clang-x86-64-build
-  variables:
-    CONTAINER: ubuntu:focal
+  extends:
+    - .clang-x86-64-build
+    - .container-ubuntu-focal
 
 ubuntu-focal-clang-debug:
-  extends: .clang-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:focal
+  extends:
+    - .clang-x86-64-build-debug
+    - .container-ubuntu-focal
 
 opensuse-leap-clang:
-  extends: .clang-x86-64-build
+  extends:
+    - .clang-x86-64-build
   variables:
     CONTAINER: suse:opensuse-leap
 
 opensuse-leap-clang-debug:
-  extends: .clang-x86-64-build-debug
+  extends:
+    - .clang-x86-64-build-debug
   variables:
     CONTAINER: suse:opensuse-leap
 
 opensuse-leap-gcc:
-  extends: .gcc-x86-64-build
+  extends:
+    - .gcc-x86-64-build
   variables:
     CONTAINER: suse:opensuse-leap
 
 opensuse-leap-gcc-debug:
-  extends: .gcc-x86-64-build-debug
+  extends:
+    - .gcc-x86-64-build-debug
   variables:
     CONTAINER: suse:opensuse-leap
 
 opensuse-tumbleweed-clang:
-  extends: .clang-x86-64-build
+  extends:
+    - .clang-x86-64-build
   variables:
     CONTAINER: suse:opensuse-tumbleweed
   allow_failure: true
 
 opensuse-tumbleweed-clang-debug:
-  extends: .clang-x86-64-build-debug
+  extends:
+    - .clang-x86-64-build-debug
   variables:
     CONTAINER: suse:opensuse-tumbleweed
   allow_failure: true
 
 opensuse-tumbleweed-gcc:
-  extends: .gcc-x86-64-build
+  extends:
+    - .gcc-x86-64-build
   variables:
     CONTAINER: suse:opensuse-tumbleweed
   allow_failure: true
 
 opensuse-tumbleweed-gcc-debug:
-  extends: .gcc-x86-64-build-debug
+  extends:
+    - .gcc-x86-64-build-debug
   variables:
     CONTAINER: suse:opensuse-tumbleweed
   allow_failure: true
 
 alpine-3.12-gcc:
-  extends: .gcc-x86-64-build
+  extends:
+    - .gcc-x86-64-build
   variables:
     CONTAINER: alpine:3.12
 
 alpine-3.12-gcc-debug:
-  extends: .gcc-x86-64-build-debug
+  extends:
+    - .gcc-x86-64-build-debug
   variables:
     CONTAINER: alpine:3.12
 
 alpine-3.12-clang:
-  extends: .clang-x86-64-build
+  extends:
+    - .clang-x86-64-build
   variables:
     CONTAINER: alpine:3.12
 
 alpine-3.12-clang-debug:
-  extends: .clang-x86-64-build-debug
+  extends:
+    - .clang-x86-64-build-debug
   variables:
     CONTAINER: alpine:3.12
 
 # Arm32 cross-build
 
 debian-unstable-gcc-arm32:
-  extends: .gcc-arm32-cross-build
+  extends:
+    - .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm64v8-arm32-gcc
     HYPERVISOR_ONLY: y
 
 debian-unstable-gcc-arm32-debug:
-  extends: .gcc-arm32-cross-build-debug
+  extends:
+    - .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm64v8-arm32-gcc
     HYPERVISOR_ONLY: y
 
 debian-unstable-gcc-arm32-randconfig:
-  extends: .gcc-arm32-cross-build
+  extends:
+    - .gcc-arm32-cross-build
   variables:
     CONTAINER: debian:unstable-arm64v8-arm32-gcc
     HYPERVISOR_ONLY: y
     RANDCONFIG: y
 
 debian-unstable-gcc-arm32-debug-randconfig:
-  extends: .gcc-arm32-cross-build-debug
+  extends:
+    - .gcc-arm32-cross-build-debug
   variables:
     CONTAINER: debian:unstable-arm64v8-arm32-gcc
     HYPERVISOR_ONLY: y
@@ -562,51 +623,60 @@ debian-unstable-gcc-arm32-debug-staticmem:
 # Arm builds
 
 debian-unstable-gcc-arm64:
-  extends: .gcc-arm64-build
+  extends:
+    - .gcc-arm64-build
   variables:
     CONTAINER: debian:unstable-arm64v8
 
 debian-unstable-gcc-debug-arm64:
-  extends: .gcc-arm64-build-debug
+  extends:
+    - .gcc-arm64-build-debug
   variables:
     CONTAINER: debian:unstable-arm64v8
 
 debian-unstable-gcc-arm64-randconfig:
-  extends: .gcc-arm64-build
+  extends:
+    - .gcc-arm64-build
   variables:
     CONTAINER: debian:unstable-arm64v8
     RANDCONFIG: y
 
 debian-unstable-gcc-debug-arm64-randconfig:
-  extends: .gcc-arm64-build-debug
+  extends:
+    - .gcc-arm64-build-debug
   variables:
     CONTAINER: debian:unstable-arm64v8
     RANDCONFIG: y
 
 alpine-3.12-gcc-arm64:
-  extends: .gcc-arm64-build
+  extends:
+    - .gcc-arm64-build
   variables:
     CONTAINER: alpine:3.12-arm64v8
 
 alpine-3.12-gcc-debug-arm64:
-  extends: .gcc-arm64-build-debug
+  extends:
+    - .gcc-arm64-build-debug
   variables:
     CONTAINER: alpine:3.12-arm64v8
 
 alpine-3.12-gcc-arm64-randconfig:
-  extends: .gcc-arm64-build
+  extends:
+    - .gcc-arm64-build
   variables:
     CONTAINER: alpine:3.12-arm64v8
     RANDCONFIG: y
 
 alpine-3.12-gcc-debug-arm64-randconfig:
-  extends: .gcc-arm64-build-debug
+  extends:
+    - .gcc-arm64-build-debug
   variables:
     CONTAINER: alpine:3.12-arm64v8
     RANDCONFIG: y
 
 alpine-3.12-gcc-arm64-staticmem:
-  extends: .gcc-arm64-build
+  extends:
+    - .gcc-arm64-build
   variables:
     CONTAINER: alpine:3.12-arm64v8
     EXTRA_XEN_CONFIG: |
@@ -615,7 +685,8 @@ alpine-3.12-gcc-arm64-staticmem:
       CONFIG_STATIC_MEMORY=y
 
 alpine-3.12-gcc-debug-arm64-staticmem:
-  extends: .gcc-arm64-build-debug
+  extends:
+    - .gcc-arm64-build-debug
   variables:
     CONTAINER: alpine:3.12-arm64v8
     EXTRA_XEN_CONFIG: |
@@ -624,7 +695,8 @@ alpine-3.12-gcc-debug-arm64-staticmem:
       CONFIG_STATIC_MEMORY=y
 
 alpine-3.12-gcc-arm64-boot-cpupools:
-  extends: .gcc-arm64-build
+  extends:
+    - .gcc-arm64-build
   variables:
     CONTAINER: alpine:3.12-arm64v8
     EXTRA_XEN_CONFIG: |
@@ -634,7 +706,8 @@ alpine-3.12-gcc-arm64-boot-cpupools:
       CONFIG_BOOT_TIME_CPUPOOLS=y
 
 alpine-3.12-gcc-debug-arm64-boot-cpupools:
-  extends: .gcc-arm64-build-debug
+  extends:
+    - .gcc-arm64-build-debug
   variables:
     CONTAINER: alpine:3.12-arm64v8
     EXTRA_XEN_CONFIG: |
@@ -642,21 +715,24 @@ alpine-3.12-gcc-debug-arm64-boot-cpupools:
 
 # RISC-V 64 cross-build
 archlinux-current-gcc-riscv64:
-  extends: .gcc-riscv64-cross-build
+  extends:
+    - .gcc-riscv64-cross-build
   variables:
     CONTAINER: archlinux:current-riscv64
     KBUILD_DEFCONFIG: tiny64_defconfig
     HYPERVISOR_ONLY: y
 
 archlinux-current-gcc-riscv64-debug:
-  extends: .gcc-riscv64-cross-build-debug
+  extends:
+    - .gcc-riscv64-cross-build-debug
   variables:
     CONTAINER: archlinux:current-riscv64
     KBUILD_DEFCONFIG: tiny64_defconfig
     HYPERVISOR_ONLY: y
 
 archlinux-current-gcc-riscv64-randconfig:
-  extends: .gcc-riscv64-cross-build
+  extends:
+    - .gcc-riscv64-cross-build
   variables:
     CONTAINER: archlinux:current-riscv64
     KBUILD_DEFCONFIG: tiny64_defconfig
@@ -665,7 +741,8 @@ archlinux-current-gcc-riscv64-randconfig:
       CONFIG_COVERAGE=n
 
 archlinux-current-gcc-riscv64-debug-randconfig:
-  extends: .gcc-riscv64-cross-build-debug
+  extends:
+    - .gcc-riscv64-cross-build-debug
   variables:
     CONTAINER: archlinux:current-riscv64
     KBUILD_DEFCONFIG: tiny64_defconfig
-- 
Anthony PERARD

[RFC XEN PATCH 4/7] automation: Adding containers build jobs and test of thoses

[Anthony PERARD]

For the test-containers jobs, mostly copy from "build.yaml", rename
'.container-*-tmpl' templates to '.container-*-testtmpl', prefix build
jobs with "test-", add BUILD_CONTAINER and "needs" to container
template.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---

Notes:
    WARNING: This is an incomplete list of the containers that can be
    rebuilt and tested.

 automation/gitlab-ci/containers.yaml      |  74 ++++++-
 automation/gitlab-ci/test-containers.yaml | 257 ++++++++++++++++++++++
 2 files changed, 330 insertions(+), 1 deletion(-)

diff --git a/automation/gitlab-ci/containers.yaml b/automation/gitlab-ci/containers.yaml
index ace93eaccf..a6d61980b1 100644
--- a/automation/gitlab-ci/containers.yaml
+++ b/automation/gitlab-ci/containers.yaml
@@ -18,8 +18,80 @@
   after_script:
     - docker logout
 
-debian-unstable-container:
+archlinux-current-container:
+  variables:
+    BUILD_CONTAINER: archlinux/current
+  extends:
+    - .container-build-tmpl
+
+centos-7-2-container:
+  variables:
+    BUILD_CONTAINER: centos/7.2
+  extends:
+    - .container-build-tmpl
+
+centos-7-container:
+  variables:
+    BUILD_CONTAINER: centos/7
+  extends:
+    - .container-build-tmpl
+
+debian-jessie-container:
+  variables:
+    BUILD_CONTAINER: debian/jessie
+  extends:
+    - .container-build-tmpl
+
+debian-jessie-32-container:
+  variables:
+    BUILD_CONTAINER: debian/jessie-i386
+  extends:
+    - .container-build-tmpl
+
+debian-stretch-container:
+  variables:
+    BUILD_CONTAINER: debian/stretch
+  extends:
+    - .container-build-tmpl
+
+debian-stretch-32-container:
+  variables:
+    BUILD_CONTAINER: debian/stretch-i386
   extends:
     - .container-build-tmpl
+
+debian-unstable-container:
   variables:
     BUILD_CONTAINER: debian/unstable
+  extends:
+    - .container-build-tmpl
+
+debian-unstable-32-container:
+  variables:
+    BUILD_CONTAINER: debian/unstable-i386
+  extends:
+    - .container-build-tmpl
+
+ubuntu-trusty-container:
+  variables:
+    BUILD_CONTAINER: ubuntu/trusty
+  extends:
+    - .container-build-tmpl
+
+ubuntu-xenial-container:
+  variables:
+    BUILD_CONTAINER: ubuntu/xenial
+  extends:
+    - .container-build-tmpl
+
+ubuntu-bionic-container:
+  variables:
+    BUILD_CONTAINER: ubuntu/bionic
+  extends:
+    - .container-build-tmpl
+
+ubuntu-focal-container:
+  variables:
+    BUILD_CONTAINER: ubuntu/focal
+  extends:
+    - .container-build-tmpl
diff --git a/automation/gitlab-ci/test-containers.yaml b/automation/gitlab-ci/test-containers.yaml
index 5dbf3902ff..4d5c6ba364 100644
--- a/automation/gitlab-ci/test-containers.yaml
+++ b/automation/gitlab-ci/test-containers.yaml
@@ -197,6 +197,148 @@
 
 # Jobs below this line
 
+.container-archlinux-testtmpl:
+  variables:
+    CONTAINER: archlinux:current
+    BUILD_CONTAINER: archlinux/current
+  needs:
+    - archlinux-current-container
+
+test-archlinux-gcc:
+  extends:
+    - .gcc-x86-64-testbuild
+    - .container-archlinux-testtmpl
+
+test-archlinux-gcc-debug:
+  extends:
+    - .gcc-x86-64-testbuild-debug
+    - .container-archlinux-testtmpl
+
+.container-centos-7-testtmpl:
+  variables:
+    CONTAINER: centos:7
+    BUILD_CONTAINER: centos/7
+  needs:
+    - centos-7-container
+
+test-centos-7-gcc:
+  extends:
+    - .gcc-x86-64-testbuild
+    - .container-centos-7-testtmpl
+
+test-centos-7-gcc-debug:
+  extends:
+    - .gcc-x86-64-testbuild-debug
+    - .container-centos-7-testtmpl
+
+.container-debian-jessie-testtmpl:
+  variables:
+    CONTAINER: debian:jessie
+    BUILD_CONTAINER: debian/jessie
+  needs:
+    - debian-jessie-container
+
+test-debian-jessie-clang:
+  extends:
+    - .clang-x86-64-testbuild
+    - .container-debian-jessie-testtmpl
+
+test-debian-jessie-clang-debug:
+  extends:
+    - .clang-x86-64-testbuild-debug
+    - .container-debian-jessie-testtmpl
+
+test-debian-jessie-gcc:
+  extends:
+    - .gcc-x86-64-testbuild
+    - .container-debian-jessie-testtmpl
+
+test-debian-jessie-gcc-debug:
+  extends:
+    - .gcc-x86-64-testbuild-debug
+    - .container-debian-jessie-testtmpl
+
+.container-debian-32-jessie-testtmpl:
+  variables:
+    CONTAINER: debian:jessie-i386
+    BUILD_CONTAINER: debian/jessie-i386
+  needs:
+    - debian-jessie-32-container
+
+test-debian-jessie-32-clang:
+  extends:
+    - .clang-x86-32-testbuild
+    - .container-debian-32-jessie-testtmpl
+
+test-debian-jessie-32-clang-debug:
+  extends:
+    - .clang-x86-32-testbuild-debug
+    - .container-debian-32-jessie-testtmpl
+
+test-debian-jessie-32-gcc:
+  extends:
+    - .gcc-x86-32-testbuild
+    - .container-debian-32-jessie-testtmpl
+
+test-debian-jessie-32-gcc-debug:
+  extends:
+    - .gcc-x86-32-testbuild-debug
+    - .container-debian-32-jessie-testtmpl
+
+.container-debian-stretch-testtmpl:
+  variables:
+    CONTAINER: debian:stretch
+    BUILD_CONTAINER: debian/stretch
+  needs:
+    - debian-stretch-container
+
+test-debian-stretch-clang:
+  extends:
+    - .clang-x86-64-testbuild
+    - .container-debian-stretch-testtmpl
+
+test-debian-stretch-clang-debug:
+  extends:
+    - .clang-x86-64-testbuild-debug
+    - .container-debian-stretch-testtmpl
+
+test-debian-stretch-clang-8:
+  extends:
+    - .clang-8-x86-64-testbuild
+    - .container-debian-stretch-testtmpl
+
+test-debian-stretch-clang-8-debug:
+  extends:
+    - .clang-8-x86-64-testbuild-debug
+    - .container-debian-stretch-testtmpl
+
+test-debian-stretch-gcc:
+  extends:
+    - .gcc-x86-64-testbuild
+    - .container-debian-stretch-testtmpl
+
+test-debian-stretch-gcc-debug:
+  extends:
+    - .gcc-x86-64-testbuild-debug
+    - .container-debian-stretch-testtmpl
+
+.container-debian-stretch-32-testtmpl:
+  variables:
+    CONTAINER: debian:stretch-i386
+    BUILD_CONTAINER: debian/stretch-i386
+  needs:
+    - debian-stretch-32-container
+
+test-debian-stretch-32-clang-debug:
+  extends:
+    - .clang-x86-32-testbuild-debug
+    - .container-debian-stretch-32-testtmpl
+
+test-debian-stretch-32-gcc-debug:
+  extends:
+    - .gcc-x86-32-testbuild-debug
+    - .container-debian-stretch-32-testtmpl
+
 .container-debian-unstable-testtmpl:
   variables:
     CONTAINER: debian:unstable
@@ -237,3 +379,118 @@ test-debian-unstable-gcc-debug-randconfig:
     - .container-debian-unstable-testtmpl
   variables:
     RANDCONFIG: y
+
+.container-debian-unstable-32-testtmpl:
+  variables:
+    CONTAINER: debian:unstable-i386
+    BUILD_CONTAINER: debian/unstable-i386
+  needs:
+    - debian-unstable-32-container
+
+test-debian-unstable-32-clang-debug:
+  extends:
+    - .clang-x86-32-testbuild-debug
+    - .container-debian-unstable-32-testtmpl
+
+test-debian-unstable-32-gcc-debug:
+  extends:
+    - .gcc-x86-32-testbuild-debug
+    - .container-debian-unstable-32-testtmpl
+
+.container-ubuntu-trusty-testtmpl:
+  variables:
+    CONTAINER: ubuntu:trusty
+    BUILD_CONTAINER: ubuntu/trusty
+  needs:
+    - ubuntu-trusty-container
+
+test-ubuntu-trusty-gcc:
+  extends:
+    - .gcc-x86-64-testbuild
+    - .container-ubuntu-trusty-testtmpl
+
+test-ubuntu-trusty-gcc-debug:
+  extends:
+    - .gcc-x86-64-testbuild-debug
+    - .container-ubuntu-trusty-testtmpl
+
+.container-ubuntu-xenial-testtmpl:
+  variables:
+    CONTAINER: ubuntu:xenial
+    BUILD_CONTAINER: ubuntu/xenial
+  needs:
+    - ubuntu-xenial-container
+
+test-ubuntu-xenial-clang:
+  extends:
+    - .clang-x86-64-testbuild
+    - .container-ubuntu-xenial-testtmpl
+
+test-ubuntu-xenial-clang-debug:
+  extends:
+    - .clang-x86-64-testbuild-debug
+    - .container-ubuntu-xenial-testtmpl
+
+test-ubuntu-xenial-gcc:
+  extends:
+    - .gcc-x86-64-testbuild
+    - .container-ubuntu-xenial-testtmpl
+
+test-ubuntu-xenial-gcc-debug:
+  extends:
+    - .gcc-x86-64-testbuild-debug
+    - .container-ubuntu-xenial-testtmpl
+
+.container-ubuntu-bionic-testtmpl:
+  variables:
+    CONTAINER: ubuntu:bionic
+    BUILD_CONTAINER: ubuntu/bionic
+  needs:
+    - ubuntu-bionic-container
+
+test-ubuntu-bionic-clang:
+  extends:
+    - .clang-x86-64-testbuild
+    - .container-ubuntu-bionic-testtmpl
+
+test-ubuntu-bionic-clang-debug:
+  extends:
+    - .clang-x86-64-testbuild-debug
+    - .container-ubuntu-bionic-testtmpl
+
+test-ubuntu-bionic-gcc:
+  extends:
+    - .gcc-x86-64-testbuild
+    - .container-ubuntu-bionic-testtmpl
+
+test-ubuntu-bionic-gcc-debug:
+  extends:
+    - .gcc-x86-64-testbuild-debug
+    - .container-ubuntu-bionic-testtmpl
+
+.container-ubuntu-focal-testtmpl:
+  variables:
+    CONTAINER: ubuntu:focal
+    BUILD_CONTAINER: ubuntu/focal
+  needs:
+    - ubuntu-focal-container
+
+test-ubuntu-focal-gcc:
+  extends:
+    - .gcc-x86-64-testbuild
+    - .container-ubuntu-focal-testtmpl
+
+test-ubuntu-focal-gcc-debug:
+  extends:
+    - .gcc-x86-64-testbuild-debug
+    - .container-ubuntu-focal-testtmpl
+
+test-ubuntu-focal-clang:
+  extends:
+    - .clang-x86-64-testbuild
+    - .container-ubuntu-focal-testtmpl
+
+test-ubuntu-focal-clang-debug:
+  extends:
+    - .clang-x86-64-testbuild-debug
+    - .container-ubuntu-focal-testtmpl
-- 
Anthony PERARD

[RFC XEN PATCH 5/7] automation: Introduce DO_REBUILD_CONTAINER, to allow to rebuild a container

[Anthony PERARD]

This allow to start a pipeline manually and set a variable to test the
build of a single container, e.g.
    DO_REBUILD_CONTAINER = ubuntu/xenial

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
 automation/gitlab-ci/containers.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/automation/gitlab-ci/containers.yaml b/automation/gitlab-ci/containers.yaml
index a6d61980b1..9074bfe6f1 100644
--- a/automation/gitlab-ci/containers.yaml
+++ b/automation/gitlab-ci/containers.yaml
@@ -7,6 +7,7 @@
     - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "staging"
       changes:
         - automation/build/${BUILD_CONTAINER}.dockerfile
+    - if: $DO_REBUILD_CONTAINER == $BUILD_CONTAINER
   services:
     - docker:dind
   before_script:
-- 
Anthony PERARD

[RFC XEN PATCH 6/7] automation: Push container been tested

[Anthony PERARD]

Now, we can run a pipeline and set two variables to have a container
been rebuilt, tested, and pushed.

Variables:
    DO_REBUILD_CONTAINER = "ubuntu/xenial"
    PUSH_CONTAINER = 1

Or if PUSH_CONTAINER is set on a gitlab project "xen-project/xen", a
change on the dockerfile can result in a container been rebuild when
the change is pushed to staging.

The push-containers stage pull the container been tested and retag it
before pushing it. So both tagged container with and without "-test"
suffix are the same.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---

Notes:
    Something that could be added is to check that the container that we are
    going to push is the same one that have been tested. Maybe by comparing
    "digest", or maybe by using a suffix that is only generated by the
    current pipeline.

 .gitlab-ci.yml                            |  2 ++
 automation/build/Makefile                 | 12 +++++++++
 automation/gitlab-ci/push-containers.yaml | 30 +++++++++++++++++++++++
 3 files changed, 44 insertions(+)
 create mode 100644 automation/gitlab-ci/push-containers.yaml

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ed5383ab50..0cd45ad001 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -3,9 +3,11 @@ stages:
   - test
   - containers
   - test-containers
+  - push-containers
 
 include:
   - 'automation/gitlab-ci/build.yaml'
   - 'automation/gitlab-ci/test.yaml'
   - 'automation/gitlab-ci/containers.yaml'
   - 'automation/gitlab-ci/test-containers.yaml'
+  - 'automation/gitlab-ci/push-containers.yaml'
diff --git a/automation/build/Makefile b/automation/build/Makefile
index 5515938878..14d1320b23 100644
--- a/automation/build/Makefile
+++ b/automation/build/Makefile
@@ -21,6 +21,18 @@ include yocto/yocto.inc
 		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F)$(BUILD_CONTAINER_SUFFIX); \
 	fi
 
+# rule used by GitLab CI jobs, to push a container that as just been built and
+# tested. It override the rule used to build a container.
+ifdef PUSH_TEST_CONTAINER_SUFFIX
+%: %.dockerfile
+	$(if $(BUILD_CONTAINER_SUFFIX),$(error BUILD_CONTAINER_SUFFIX should not be set anymore))
+	$(DOCKER_CMD) pull $(REGISTRY)/$(@D):$(@F)$(PUSH_TEST_CONTAINER_SUFFIX)
+	$(DOCKER_CMD) image tag $(REGISTRY)/$(@D):$(@F)$(PUSH_TEST_CONTAINER_SUFFIX) $(REGISTRY)/$(@D):$(@F)
+	@if [ ! -z $${PUSH+x} ]; then \
+		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
+	fi
+endif
+
 .PHONY: all clean
 all: $(CONTAINERS)
 
diff --git a/automation/gitlab-ci/push-containers.yaml b/automation/gitlab-ci/push-containers.yaml
new file mode 100644
index 0000000000..d7e7e2b9e2
--- /dev/null
+++ b/automation/gitlab-ci/push-containers.yaml
@@ -0,0 +1,30 @@
+.push-container-build-tmpl:
+  stage: push-containers
+  image: docker:stable
+  tags:
+    - container-builder
+  rules:
+    - if: $PUSH_CONTAINER != "1"
+      when: never
+    - !reference [.container-build-tmpl, rules]
+  services:
+    - docker:dind
+  before_script:
+    - apk add make
+    - docker info
+    - docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY
+  script:
+    - make -C automation/build ${BUILD_CONTAINER} PUSH=1 PUSH_TEST_CONTAINER_SUFFIX=-test
+  after_script:
+    - docker logout
+
+push-ubuntu-xenial-container:
+  variables:
+    BUILD_CONTAINER: ubuntu/xenial
+  extends:
+    - .push-container-build-tmpl
+  needs:
+    - test-ubuntu-xenial-clang
+    - test-ubuntu-xenial-clang-debug
+    - test-ubuntu-xenial-gcc
+    - test-ubuntu-xenial-gcc-debug
-- 
Anthony PERARD

[RFC XEN PATCH 7/7] automation: Add some more push containers jobs

[Anthony PERARD]

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---

Notes:
    WARNING: This is an incomplete list of jobs needed to push.

 automation/gitlab-ci/push-containers.yaml | 49 +++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/automation/gitlab-ci/push-containers.yaml b/automation/gitlab-ci/push-containers.yaml
index d7e7e2b9e2..3785e29250 100644
--- a/automation/gitlab-ci/push-containers.yaml
+++ b/automation/gitlab-ci/push-containers.yaml
@@ -18,6 +18,33 @@
   after_script:
     - docker logout
 
+push-archlinux-current-container:
+  variables:
+    BUILD_CONTAINER: archlinux/current
+  extends:
+    - .push-container-build-tmpl
+  needs:
+    - test-archlinux-gcc
+    - test-archlinux-gcc-debug
+
+push-debian-stretch-32-container:
+  variables:
+    BUILD_CONTAINER: debian/stretch-i386
+  extends:
+    - .push-container-build-tmpl
+  needs:
+    - test-debian-stretch-32-clang-debug
+    - test-debian-stretch-32-gcc-debug
+
+push-debian-unstable-32-container:
+  variables:
+    BUILD_CONTAINER: debian/unstable-i386
+  extends:
+    - .push-container-build-tmpl
+  needs:
+    - test-debian-unstable-32-clang-debug
+    - test-debian-unstable-32-gcc-debug
+
 push-ubuntu-xenial-container:
   variables:
     BUILD_CONTAINER: ubuntu/xenial
@@ -28,3 +55,25 @@ push-ubuntu-xenial-container:
     - test-ubuntu-xenial-clang-debug
     - test-ubuntu-xenial-gcc
     - test-ubuntu-xenial-gcc-debug
+
+push-ubuntu-bionic-container:
+  variables:
+    BUILD_CONTAINER: ubuntu/bionic
+  extends:
+    - .push-container-build-tmpl
+  needs:
+    - test-ubuntu-bionic-clang
+    - test-ubuntu-bionic-clang-debug
+    - test-ubuntu-bionic-gcc
+    - test-ubuntu-bionic-gcc-debug
+
+push-ubuntu-focal-container:
+  variables:
+    BUILD_CONTAINER: ubuntu/focal
+  extends:
+    - .push-container-build-tmpl
+  needs:
+    - test-ubuntu-focal-gcc
+    - test-ubuntu-focal-gcc-debug
+    - test-ubuntu-focal-clang
+    - test-ubuntu-focal-clang-debug
-- 
Anthony PERARD

Re: [RFC XEN PATCH 0/7] automation, RFC prototype, Have GitLab CI built its own containers

[Stefano Stabellini]

On Thu, 2 Mar 2023, Anthony PERARD wrote:
> Patch series available in this git branch:
> https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-auto-rebuild-v1
> 
> Hi,
> 
> I have done some research to be able to build containers in the CI. This works
> only for x86 containers as I've setup only a single x86 gitlab-runner to be
> able to run docker-in-docker.
> 
> The runner is setup to only accept jobs from a branch that is "protected" in
> gitlab. Also, one need credential to push to the container register, those are
> called "Group deploy tokens", and I've set the variables CI_DEPLOY_USER and
> CI_DEPLOY_PASSWORD in the project "xen-project/xen" (variables only visible on
> a pipeline running on a protected branch).
> 
> These patch introduce quite a lot of redundancies in jobs, 2 new jobs per
> containers which build/push containers, and duplicate most of build.yaml.
> This mean that if we go with this, we have to duplicate and keep in sync many
> jobs.
> 
> To avoid having to do the duplicated jobs by hand, I could look at
> creating a script that use "build.yaml" as input and produce the 3
> stages needed to update a container, but that script would need to be
> run by hand, as gitlab can't really use it, unless ..
> 
> I've look at generated pipeline, and they look like this in gitlab:
>     https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/777665738
> But the result of the generated/child pipeline doesn't seems to be taken into
> account in the original pipeline, which make me think that we can't use them to
> generate "build.yaml". But maybe the could be use for generating the pipeline
> that will update a container.
> Doc:
>     https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines
> 
> So, with all of this, is it reasonable to test containers before
> pushing them to production? Or is it to much work? We could simply have jobs
> tacking care of rebuilding a container and pushing them to production without
> testing.

I don't think it is a good idea to duplicate build.yaml, also because
some of the containers are used in the testing stage too, so an updated
container could be OK during the build phase and break the testing
phase. We would need to duplicate both build.yaml and test.yaml, which
is not feasible.

In practice today people either:
1) re-build a container locally & test it locally before pushing
2) re-build a container locally, docker push it, then run a private
   gitlab pipeline, if it passes send out a patch to xen-devel

1) is not affected by this series
2) is also not affected because by the time the pipeline is created, the
container is already updated

However, there are cases where it would definitely be nice to have a
"button" to press to update a container. For instance, when a pipeline
failis due to a Debian unstable apt-get failure, which can be easily fixed
by updating the Debian unstable container.

So I think it would be nice to have jobs that can automatically update
the build containers but I would set them to manually trigger instead of
automatically (when: manual).


Alternatively, we could move the "containers.yaml" stage to be the first
step, rebuild the containers and push them to a "staging" area
(registry.gitlab.com/xen-project/xen/staging), run the build and test
steps fetching from the staging area instead of the regular, if all
tests pass, then push the containers to
registry.gitlab.com/xen-project/xen as last step.


> An example with the variable DO_REBUILD_CONTAINER and PUSH_CONTAINER set (and
> existing build/test jobs disabled):
>     https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/791711467
> 
> Cheers,
> 
> Anthony PERARD (7):
>   automation: Automatically rebuild debian:unstable container
>   automation: Introduce test-containers stage
>   automation: Add a template per container for build jobs.
>   automation: Adding containers build jobs and test of thoses
>   automation: Introduce DO_REBUILD_CONTAINER, to allow to rebuild a
>     container
>   automation: Push container been tested
>   automation: Add some more push containers jobs
> 
>  .gitlab-ci.yml                            |   6 +
>  automation/build/Makefile                 |  14 +-
>  automation/gitlab-ci/build.yaml           | 327 ++++++++------
>  automation/gitlab-ci/containers.yaml      |  98 +++++
>  automation/gitlab-ci/push-containers.yaml |  79 ++++
>  automation/gitlab-ci/test-containers.yaml | 496 ++++++++++++++++++++++
>  6 files changed, 894 insertions(+), 126 deletions(-)
>  create mode 100644 automation/gitlab-ci/containers.yaml
>  create mode 100644 automation/gitlab-ci/push-containers.yaml
>  create mode 100644 automation/gitlab-ci/test-containers.yaml
> 
> -- 
> Anthony PERARD
> 

Re: [RFC XEN PATCH 0/7] automation, RFC prototype, Have GitLab CI built its own containers

[Anthony PERARD]

On Thu, Mar 02, 2023 at 06:48:35PM -0800, Stefano Stabellini wrote:
> On Thu, 2 Mar 2023, Anthony PERARD wrote:
> > Patch series available in this git branch:
> > https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-auto-rebuild-v1
> > 
> > Hi,
> > 
> > I have done some research to be able to build containers in the CI. This works
> > only for x86 containers as I've setup only a single x86 gitlab-runner to be
> > able to run docker-in-docker.
> > 
> > The runner is setup to only accept jobs from a branch that is "protected" in
> > gitlab. Also, one need credential to push to the container register, those are
> > called "Group deploy tokens", and I've set the variables CI_DEPLOY_USER and
> > CI_DEPLOY_PASSWORD in the project "xen-project/xen" (variables only visible on
> > a pipeline running on a protected branch).
> > 
> > These patch introduce quite a lot of redundancies in jobs, 2 new jobs per
> > containers which build/push containers, and duplicate most of build.yaml.
> > This mean that if we go with this, we have to duplicate and keep in sync many
> > jobs.
> > 
> > To avoid having to do the duplicated jobs by hand, I could look at
> > creating a script that use "build.yaml" as input and produce the 3
> > stages needed to update a container, but that script would need to be
> > run by hand, as gitlab can't really use it, unless ..
> > 
> > I've look at generated pipeline, and they look like this in gitlab:
> >     https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/777665738
> > But the result of the generated/child pipeline doesn't seems to be taken into
> > account in the original pipeline, which make me think that we can't use them to
> > generate "build.yaml". But maybe the could be use for generating the pipeline
> > that will update a container.
> > Doc:
> >     https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines
> > 
> > So, with all of this, is it reasonable to test containers before
> > pushing them to production? Or is it to much work? We could simply have jobs
> > tacking care of rebuilding a container and pushing them to production without
> > testing.
> 
> I don't think it is a good idea to duplicate build.yaml, also because
> some of the containers are used in the testing stage too, so an updated
> container could be OK during the build phase and break the testing
> phase. We would need to duplicate both build.yaml and test.yaml, which
> is not feasible.
> 
> In practice today people either:
> 1) re-build a container locally & test it locally before pushing
> 2) re-build a container locally, docker push it, then run a private
>    gitlab pipeline, if it passes send out a patch to xen-devel
> 
> 1) is not affected by this series
> 2) is also not affected because by the time the pipeline is created, the
> container is already updated
> 
> However, there are cases where it would definitely be nice to have a
> "button" to press to update a container. For instance, when a pipeline
> failis due to a Debian unstable apt-get failure, which can be easily fixed
> by updating the Debian unstable container.
> 
> So I think it would be nice to have jobs that can automatically update
> the build containers but I would set them to manually trigger instead of
> automatically (when: manual).

What I was looking at with this work was to be able to have container
been rebuild automatically on a schedule. We have/had containers that
were 3yr old, and when it's a container that supposed to test the lasted
version of a distro, or a rolling release distro, they are kind of
useless if they aren't rebuild regularly. So I was looking to take the
human out of the loop and have computers the tedious work of rebuilding a
container every month or two.

Containers that needs to be rebuilt regularly to stay relevant are
archlinux, debian/unstable, fedora/latest, propably opensuse/leap and
opensuse/tumbleweed. I don't know if they are others.


> Alternatively, we could move the "containers.yaml" stage to be the first
> step, rebuild the containers and push them to a "staging" area
> (registry.gitlab.com/xen-project/xen/staging), run the build and test
> steps fetching from the staging area instead of the regular, if all
> tests pass, then push the containers to
> registry.gitlab.com/xen-project/xen as last step.

Sounds good, I can look into how easily it would be to use a different
registry to run a pipeline.

Cheers,

-- 
Anthony PERARD

Re: [RFC XEN PATCH 0/7] automation, RFC prototype, Have GitLab CI built its own containers

[Stefano Stabellini]

On Fri, 3 Mar 2023, Anthony PERARD wrote:
> On Thu, Mar 02, 2023 at 06:48:35PM -0800, Stefano Stabellini wrote:
> > On Thu, 2 Mar 2023, Anthony PERARD wrote:
> > > Patch series available in this git branch:
> > > https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-auto-rebuild-v1
> > > 
> > > Hi,
> > > 
> > > I have done some research to be able to build containers in the CI. This works
> > > only for x86 containers as I've setup only a single x86 gitlab-runner to be
> > > able to run docker-in-docker.
> > > 
> > > The runner is setup to only accept jobs from a branch that is "protected" in
> > > gitlab. Also, one need credential to push to the container register, those are
> > > called "Group deploy tokens", and I've set the variables CI_DEPLOY_USER and
> > > CI_DEPLOY_PASSWORD in the project "xen-project/xen" (variables only visible on
> > > a pipeline running on a protected branch).
> > > 
> > > These patch introduce quite a lot of redundancies in jobs, 2 new jobs per
> > > containers which build/push containers, and duplicate most of build.yaml.
> > > This mean that if we go with this, we have to duplicate and keep in sync many
> > > jobs.
> > > 
> > > To avoid having to do the duplicated jobs by hand, I could look at
> > > creating a script that use "build.yaml" as input and produce the 3
> > > stages needed to update a container, but that script would need to be
> > > run by hand, as gitlab can't really use it, unless ..
> > > 
> > > I've look at generated pipeline, and they look like this in gitlab:
> > >     https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/777665738
> > > But the result of the generated/child pipeline doesn't seems to be taken into
> > > account in the original pipeline, which make me think that we can't use them to
> > > generate "build.yaml". But maybe the could be use for generating the pipeline
> > > that will update a container.
> > > Doc:
> > >     https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines
> > > 
> > > So, with all of this, is it reasonable to test containers before
> > > pushing them to production? Or is it to much work? We could simply have jobs
> > > tacking care of rebuilding a container and pushing them to production without
> > > testing.
> > 
> > I don't think it is a good idea to duplicate build.yaml, also because
> > some of the containers are used in the testing stage too, so an updated
> > container could be OK during the build phase and break the testing
> > phase. We would need to duplicate both build.yaml and test.yaml, which
> > is not feasible.
> > 
> > In practice today people either:
> > 1) re-build a container locally & test it locally before pushing
> > 2) re-build a container locally, docker push it, then run a private
> >    gitlab pipeline, if it passes send out a patch to xen-devel
> > 
> > 1) is not affected by this series
> > 2) is also not affected because by the time the pipeline is created, the
> > container is already updated
> > 
> > However, there are cases where it would definitely be nice to have a
> > "button" to press to update a container. For instance, when a pipeline
> > failis due to a Debian unstable apt-get failure, which can be easily fixed
> > by updating the Debian unstable container.
> > 
> > So I think it would be nice to have jobs that can automatically update
> > the build containers but I would set them to manually trigger instead of
> > automatically (when: manual).
> 
> What I was looking at with this work was to be able to have container
> been rebuild automatically on a schedule. We have/had containers that
> were 3yr old, and when it's a container that supposed to test the lasted
> version of a distro, or a rolling release distro, they are kind of
> useless if they aren't rebuild regularly. So I was looking to take the
> human out of the loop and have computers the tedious work of rebuilding a
> container every month or two.
> 
> Containers that needs to be rebuilt regularly to stay relevant are
> archlinux, debian/unstable, fedora/latest, propably opensuse/leap and
> opensuse/tumbleweed. I don't know if they are others.
 
OK. This use-case is very different from 1) and 2) below. In this case
the need to rebuild the container does not come from a change in the
Dockerfile made by a human (e.g. a contributor adding a package to
apt-get) but it comes from the fact that the underlying distro is
updated continuously so the containers should also be updated
continuously.

For this use-case specifically, I think it would be better to trigger
the jobs to do the containers update based on schedule, rather than a
change to the Dockerfile?

And also, given that there are supposed to be no breaking changes, it
would be more acceptable to do the containers update without testing it.
But it would still be nicer to run the tests as suggested below of
course.

 
> > Alternatively, we could move the "containers.yaml" stage to be the first
> > step, rebuild the containers and push them to a "staging" area
> > (registry.gitlab.com/xen-project/xen/staging), run the build and test
> > steps fetching from the staging area instead of the regular, if all
> > tests pass, then push the containers to
> > registry.gitlab.com/xen-project/xen as last step.
> 
> Sounds good, I can look into how easily it would be to use a different
> registry to run a pipeline.

Thanks!