qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream
@ 2021-04-18 23:34 Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [PATCH 01/15] gitlab-ci: Replace YAML anchors by extends (container_job) Philippe Mathieu-Daudé
                   ` (14 more replies)
  0 siblings, 15 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

TL;DR: Patch 13 is an experiment to dynamically generate pipelines.\r
\r
Hi,\r
\r
This series is an intent to allow git forks to use different set\r
of jobs than the ones used by the mainstream repository.\r
\r
Currently, a fork gets the mainstream CI pipeline YAML configuration\r
and runs its set of jobs by default. Mainstream tests many cases\r
which might not be always useful to forks. Beside, one mainstream\r
pipeline might be a waste of ressources when forks provide their\r
own runners to their gitlab namespace, or use private instances.\r
\r
This is not a new / unique problem to our community, and Gitlab\r
already figured it out. They describe this case as Upstream (our\r
project mainline) / Downstream (a fork).\r
\r
The pipeline is divided in 2 steps. First the "upstream" part, which\r
does some evaluation, and depending on the result, triggers the\r
"downstream" part. The "upstream" is committed into mainline, and\r
dynamically generate the "downstream" pipeline. This allows forks\r
to easily rebase with mainstream, avoiding merge conflicts.\r
\r
This might not be adapted to the QEMU use case, but I wanted to give\r
it a try.\r
\r
Gitlab recommends to use job templates. To be able to re-use /\r
include templates, they can not use YAML anchors. All current jobs\r
has first to be converted to use the 'extends' keyworkd.\r
This is done by patches 1-5 (already reviewed).\r
\r
Patches 6-12 extract various generic jobs (which can be reused by\r
forks) as template.\r
\r
Patches 13-14 follow Gitlab guideline to generate dynamic pipelines\r
\r
Patch 15 is an example of how a fork could use mainstream templates\r
to maintain his own set of jobs.\r
\r
The PRO is forks can contribute to templates.\r
\r
Regards,\r
\r
Phil.\r
\r
Philippe Mathieu-Daudé (15):\r
  gitlab-ci: Replace YAML anchors by extends (container_job)\r
  gitlab-ci: Replace YAML anchors by extends (native_build_job)\r
  gitlab-ci: Replace YAML anchors by extends (native_test_job)\r
  gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)\r
  gitlab-ci: Rename acceptance_test_job -> integration_test_job\r
  gitlab-ci: Extract container job template to container-template.yml\r
  gitlab-ci: Extract crossbuild job templates to crossbuild-template.yml\r
  gitlab-ci: Extract DCO/style check jobs to checks.yml\r
  gitlab-ci: Extract build stages to stages.yml\r
  gitlab-ci: Extract all default build/test jobs to buildtest.yml\r
  gitlab-ci: Extract core container jobs to container-core.yml\r
  gitlab-ci: Move current job set to qemu-project.yml\r
  gitlab-ci: Switch to dynamically generated pipelines\r
  gitlab-ci: Allow forks to use different set of jobs\r
  gitlab-ci: Use my own set of jobs for CI pipeline\r
\r
 .gitlab-ci.d/buildtest-template.yml  |  80 +++\r
 .gitlab-ci.d/buildtest.yml           | 744 +++++++++++++++++++++++\r
 .gitlab-ci.d/checks.yml              |  24 +\r
 .gitlab-ci.d/container-core.yml      |  17 +\r
 .gitlab-ci.d/container-template.yml  |  22 +\r
 .gitlab-ci.d/containers.yml          | 108 ++--\r
 .gitlab-ci.d/crossbuild-template.yml |  41 ++\r
 .gitlab-ci.d/crossbuilds.yml         |  43 +-\r
 .gitlab-ci.d/philmd.yml              |  33 ++\r
 .gitlab-ci.d/qemu-project.yml        |   8 +\r
 .gitlab-ci.d/stages.yml              |   8 +\r
 .gitlab-ci.yml                       | 845 +--------------------------\r
 12 files changed, 1033 insertions(+), 940 deletions(-)\r
 create mode 100644 .gitlab-ci.d/buildtest-template.yml\r
 create mode 100644 .gitlab-ci.d/buildtest.yml\r
 create mode 100644 .gitlab-ci.d/checks.yml\r
 create mode 100644 .gitlab-ci.d/container-core.yml\r
 create mode 100644 .gitlab-ci.d/container-template.yml\r
 create mode 100644 .gitlab-ci.d/crossbuild-template.yml\r
 create mode 100644 .gitlab-ci.d/philmd.yml\r
 create mode 100644 .gitlab-ci.d/qemu-project.yml\r
 create mode 100644 .gitlab-ci.d/stages.yml\r
\r
-- \r
2.26.3\r
\r


^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 01/15] gitlab-ci: Replace YAML anchors by extends (container_job)
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [PATCH 02/15] gitlab-ci: Replace YAML anchors by extends (native_build_job) Philippe Mathieu-Daudé
                   ` (13 subsequent siblings)
  14 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

From: Philippe Mathieu-Daudé <philmd@redhat.com>

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci.d/containers.yml | 76 ++++++++++++++++++-------------------
 1 file changed, 38 insertions(+), 38 deletions(-)

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 33e4046e233..4ef76d1f54d 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -1,4 +1,4 @@
-.container_job_template: &container_job_definition
+.container_job_template:
   image: docker:stable
   stage: containers
   services:
@@ -22,230 +22,230 @@
     - docker logout
 
 amd64-alpine-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: alpine
 
 amd64-centos7-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: centos7
 
 amd64-centos8-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: centos8
 
 amd64-debian10-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: debian10
 
 amd64-debian11-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: debian11
 
 alpha-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-alpha-cross
 
 amd64-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-amd64-cross
 
 amd64-debian-user-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-all-test-cross
 
 amd64-debian-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-amd64
 
 arm64-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-arm64-cross
 
 arm64-test-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian11-container']
   variables:
     NAME: debian-arm64-test-cross
 
 armel-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-armel-cross
 
 armhf-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-armhf-cross
 
 hppa-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-hppa-cross
 
 m68k-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-m68k-cross
 
 mips64-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-mips64-cross
 
 mips64el-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-mips64el-cross
 
 mips-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-mips-cross
 
 mipsel-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-mipsel-cross
 
 powerpc-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-powerpc-cross
 
 ppc64-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-ppc64-cross
 
 ppc64el-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-ppc64el-cross
 
 riscv64-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-riscv64-cross
 
 s390x-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-s390x-cross
 
 sh4-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-sh4-cross
 
 sparc64-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-sparc64-cross
 
 tricore-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
     NAME: debian-tricore-cross
 
 xtensa-debian-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: debian-xtensa-cross
 
 cris-fedora-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: fedora-cris-cross
 
 amd64-fedora-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: fedora
 
 i386-fedora-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: fedora-i386-cross
 
 win32-fedora-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: fedora-win32-cross
 
 win64-fedora-cross-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: fedora-win64-cross
 
 amd64-ubuntu1804-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: ubuntu1804
 
 amd64-ubuntu2004-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: ubuntu2004
 
 amd64-ubuntu-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: ubuntu
 
 amd64-opensuse-leap-container:
-  <<: *container_job_definition
+  extends: .container_job_template
   variables:
     NAME: opensuse-leap
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 02/15] gitlab-ci: Replace YAML anchors by extends (native_build_job)
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [PATCH 01/15] gitlab-ci: Replace YAML anchors by extends (container_job) Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [PATCH 03/15] gitlab-ci: Replace YAML anchors by extends (native_test_job) Philippe Mathieu-Daudé
                   ` (12 subsequent siblings)
  14 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

From: Philippe Mathieu-Daudé <philmd@redhat.com>

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci.yml | 64 +++++++++++++++++++++++++-------------------------
 1 file changed, 32 insertions(+), 32 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 52d65d6c04f..98d73429405 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -13,7 +13,7 @@ include:
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
 
-.native_build_job_template: &native_build_job_definition
+.native_build_job_template:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -79,7 +79,7 @@ include:
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
 
 build-system-alpine:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     - job: amd64-alpine-container
   variables:
@@ -114,7 +114,7 @@ acceptance-system-alpine:
   <<: *acceptance_definition
 
 build-system-ubuntu:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-ubuntu2004-container
   variables:
@@ -148,7 +148,7 @@ acceptance-system-ubuntu:
   <<: *acceptance_definition
 
 build-system-debian:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-container
   variables:
@@ -182,7 +182,7 @@ acceptance-system-debian:
   <<: *acceptance_definition
 
 build-system-fedora:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-fedora-container
   variables:
@@ -217,7 +217,7 @@ acceptance-system-fedora:
   <<: *acceptance_definition
 
 build-system-centos:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-centos8-container
   variables:
@@ -252,7 +252,7 @@ acceptance-system-centos:
   <<: *acceptance_definition
 
 build-system-opensuse:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-opensuse-leap-container
   variables:
@@ -286,7 +286,7 @@ acceptance-system-opensuse:
 
 
 build-disabled:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-fedora-container
   variables:
@@ -373,7 +373,7 @@ build-disabled:
 # Also use a different coroutine implementation (which is only really of
 # interest to KVM users, i.e. with TCG disabled)
 build-tcg-disabled:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-centos8-container
   variables:
@@ -396,7 +396,7 @@ build-tcg-disabled:
             260 261 262 263 264 270 272 273 277 279
 
 build-user:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-user-cross-container
   variables:
@@ -405,7 +405,7 @@ build-user:
     MAKE_CHECK_ARGS: check-tcg
 
 build-user-static:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-user-cross-container
   variables:
@@ -415,7 +415,7 @@ build-user-static:
 
 # Only build the softmmu targets we have check-tcg tests for
 build-some-softmmu:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-user-cross-container
   variables:
@@ -428,7 +428,7 @@ build-some-softmmu:
 # we skip sparc64-linux-user until it has been fixed somewhat
 # we skip cris-linux-user as it doesn't use the common run loop
 build-user-plugins:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-user-cross-container
   variables:
@@ -438,7 +438,7 @@ build-user-plugins:
   timeout: 1h 30m
 
 build-user-centos7:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-centos7-container
   variables:
@@ -447,7 +447,7 @@ build-user-centos7:
     MAKE_CHECK_ARGS: check-tcg
 
 build-some-softmmu-plugins:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-user-cross-container
   variables:
@@ -457,7 +457,7 @@ build-some-softmmu-plugins:
     MAKE_CHECK_ARGS: check-tcg
 
 clang-system:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-fedora-container
   variables:
@@ -469,7 +469,7 @@ clang-system:
     MAKE_CHECK_ARGS: check-qtest check-tcg
 
 clang-user:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-user-cross-container
   variables:
@@ -491,7 +491,7 @@ clang-user:
 # Split in three sets of build/check/acceptance to limit the execution time of each
 # job
 build-cfi-aarch64:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
   - job: amd64-fedora-container
   variables:
@@ -528,7 +528,7 @@ acceptance-cfi-aarch64:
   <<: *acceptance_definition
 
 build-cfi-ppc64-s390x:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
   - job: amd64-fedora-container
   variables:
@@ -565,7 +565,7 @@ acceptance-cfi-ppc64-s390x:
   <<: *acceptance_definition
 
 build-cfi-x86_64:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
   - job: amd64-fedora-container
   variables:
@@ -602,7 +602,7 @@ acceptance-cfi-x86_64:
   <<: *acceptance_definition
 
 tsan-build:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-ubuntu2004-container
   variables:
@@ -614,7 +614,7 @@ tsan-build:
 
 # These targets are on the way out
 build-deprecated:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-user-cross-container
   variables:
@@ -641,7 +641,7 @@ check-deprecated:
 
 # gprof/gcov are GCC features
 gprof-gcov:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-ubuntu2004-container
   variables:
@@ -654,7 +654,7 @@ gprof-gcov:
     - ${CI_PROJECT_DIR}/scripts/ci/coverage-summary.sh
 
 build-oss-fuzz:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-fedora-container
   variables:
@@ -674,7 +674,7 @@ build-oss-fuzz:
     - cd build-oss-fuzz && make check-qtest-i386 check-unit
 
 build-tci:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-user-cross-container
   variables:
@@ -699,7 +699,7 @@ build-tci:
 # Alternate coroutines implementations are only really of interest to KVM users
 # However we can't test against KVM on Gitlab-CI so we can only run unit tests
 build-coroutine-sigaltstack:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-ubuntu2004-container
   variables:
@@ -713,7 +713,7 @@ build-coroutine-sigaltstack:
 # These jobs test old gcrypt and nettle from RHEL7
 # which had some API differences.
 crypto-old-nettle:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-centos7-container
   variables:
@@ -723,7 +723,7 @@ crypto-old-nettle:
     MAKE_CHECK_ARGS: check
 
 crypto-old-gcrypt:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-centos7-container
   variables:
@@ -733,7 +733,7 @@ crypto-old-gcrypt:
     MAKE_CHECK_ARGS: check
 
 crypto-only-gnutls:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-centos7-container
   variables:
@@ -745,7 +745,7 @@ crypto-only-gnutls:
 
 # Check our reduced build configurations
 build-without-default-devices:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-centos8-container
   variables:
@@ -753,7 +753,7 @@ build-without-default-devices:
     CONFIGURE_ARGS: --without-default-devices --disable-user
 
 build-without-default-features:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-container
   variables:
@@ -803,7 +803,7 @@ build-libvhost-user:
 # No targets are built here, just tools, docs, and unit tests. This
 # also feeds into the eventual documentation deployment steps later
 build-tools-and-docs-debian:
-  <<: *native_build_job_definition
+  extends: .native_build_job_template
   needs:
     job: amd64-debian-container
   variables:
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 03/15] gitlab-ci: Replace YAML anchors by extends (native_test_job)
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [PATCH 01/15] gitlab-ci: Replace YAML anchors by extends (container_job) Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [PATCH 02/15] gitlab-ci: Replace YAML anchors by extends (native_build_job) Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [PATCH 04/15] gitlab-ci: Replace YAML anchors by extends (acceptance_test_job) Philippe Mathieu-Daudé
                   ` (11 subsequent siblings)
  14 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

From: Philippe Mathieu-Daudé <philmd@redhat.com>

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci.yml | 40 ++++++++++++++++++++--------------------
 1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 98d73429405..11bda3abf33 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -37,7 +37,7 @@ include:
         make -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
 
-.native_test_job_template: &native_test_job_definition
+.native_test_job_template:
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -95,7 +95,7 @@ build-system-alpine:
       - build
 
 check-system-alpine:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-alpine
       artifacts: true
@@ -104,7 +104,7 @@ check-system-alpine:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-alpine:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-alpine
       artifacts: true
@@ -129,7 +129,7 @@ build-system-ubuntu:
       - build
 
 check-system-ubuntu:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-ubuntu
       artifacts: true
@@ -138,7 +138,7 @@ check-system-ubuntu:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-ubuntu:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-ubuntu
       artifacts: true
@@ -163,7 +163,7 @@ build-system-debian:
       - build
 
 check-system-debian:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-debian
       artifacts: true
@@ -172,7 +172,7 @@ check-system-debian:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-debian:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-debian
       artifacts: true
@@ -198,7 +198,7 @@ build-system-fedora:
       - build
 
 check-system-fedora:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-fedora
       artifacts: true
@@ -207,7 +207,7 @@ check-system-fedora:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-fedora:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-fedora
       artifacts: true
@@ -233,7 +233,7 @@ build-system-centos:
       - build
 
 check-system-centos:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-centos
       artifacts: true
@@ -242,7 +242,7 @@ check-system-centos:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-centos:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-centos
       artifacts: true
@@ -266,7 +266,7 @@ build-system-opensuse:
       - build
 
 check-system-opensuse:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-opensuse
       artifacts: true
@@ -275,7 +275,7 @@ check-system-opensuse:
     MAKE_CHECK_ARGS: check
 
 acceptance-system-opensuse:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-system-opensuse
       artifacts: true
@@ -509,7 +509,7 @@ build-cfi-aarch64:
       - build
 
 check-cfi-aarch64:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-cfi-aarch64
       artifacts: true
@@ -518,7 +518,7 @@ check-cfi-aarch64:
     MAKE_CHECK_ARGS: check
 
 acceptance-cfi-aarch64:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-cfi-aarch64
       artifacts: true
@@ -546,7 +546,7 @@ build-cfi-ppc64-s390x:
       - build
 
 check-cfi-ppc64-s390x:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-cfi-ppc64-s390x
       artifacts: true
@@ -555,7 +555,7 @@ check-cfi-ppc64-s390x:
     MAKE_CHECK_ARGS: check
 
 acceptance-cfi-ppc64-s390x:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-cfi-ppc64-s390x
       artifacts: true
@@ -583,7 +583,7 @@ build-cfi-x86_64:
       - build
 
 check-cfi-x86_64:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-cfi-x86_64
       artifacts: true
@@ -592,7 +592,7 @@ check-cfi-x86_64:
     MAKE_CHECK_ARGS: check
 
 acceptance-cfi-x86_64:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-cfi-x86_64
       artifacts: true
@@ -630,7 +630,7 @@ build-deprecated:
 # We split the check-tcg step as test failures are expected but we still
 # want to catch the build breaking.
 check-deprecated:
-  <<: *native_test_job_definition
+  extends: .native_test_job_template
   needs:
     - job: build-deprecated
       artifacts: true
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 04/15] gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (2 preceding siblings ...)
  2021-04-18 23:34 ` [PATCH 03/15] gitlab-ci: Replace YAML anchors by extends (native_test_job) Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-05-03  9:22   ` Thomas Huth
  2021-04-18 23:34 ` [PATCH 05/15] gitlab-ci: Rename acceptance_test_job -> integration_test_job Philippe Mathieu-Daudé
                   ` (10 subsequent siblings)
  14 siblings, 1 reply; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

From: Philippe Mathieu-Daudé <philmd@redhat.com>

'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci.yml | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 11bda3abf33..07e5e737685 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -48,7 +48,7 @@ include:
     # Avoid recompiling by hiding ninja with NINJA=":"
     - make NINJA=":" $MAKE_CHECK_ARGS
 
-.acceptance_template: &acceptance_definition
+.acceptance_test_job_template:
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
@@ -111,7 +111,7 @@ acceptance-system-alpine:
   variables:
     IMAGE: alpine
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+  extends: .acceptance_test_job_template
 
 build-system-ubuntu:
   extends: .native_build_job_template
@@ -145,7 +145,7 @@ acceptance-system-ubuntu:
   variables:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+  extends: .acceptance_test_job_template
 
 build-system-debian:
   extends: .native_build_job_template
@@ -179,7 +179,7 @@ acceptance-system-debian:
   variables:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+  extends: .acceptance_test_job_template
 
 build-system-fedora:
   extends: .native_build_job_template
@@ -214,7 +214,7 @@ acceptance-system-fedora:
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+  extends: .acceptance_test_job_template
 
 build-system-centos:
   extends: .native_build_job_template
@@ -249,7 +249,7 @@ acceptance-system-centos:
   variables:
     IMAGE: centos8
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+  extends: .acceptance_test_job_template
 
 build-system-opensuse:
   extends: .native_build_job_template
@@ -282,7 +282,7 @@ acceptance-system-opensuse:
   variables:
     IMAGE: opensuse-leap
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+  extends: .acceptance_test_job_template
 
 
 build-disabled:
@@ -525,7 +525,7 @@ acceptance-cfi-aarch64:
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+  extends: .acceptance_test_job_template
 
 build-cfi-ppc64-s390x:
   extends: .native_build_job_template
@@ -562,7 +562,7 @@ acceptance-cfi-ppc64-s390x:
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+  extends: .acceptance_test_job_template
 
 build-cfi-x86_64:
   extends: .native_build_job_template
@@ -599,7 +599,7 @@ acceptance-cfi-x86_64:
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *acceptance_definition
+  extends: .acceptance_test_job_template
 
 tsan-build:
   extends: .native_build_job_template
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 05/15] gitlab-ci: Rename acceptance_test_job -> integration_test_job
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (3 preceding siblings ...)
  2021-04-18 23:34 ` [PATCH 04/15] gitlab-ci: Replace YAML anchors by extends (acceptance_test_job) Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-19  5:19   ` Thomas Huth
  2021-04-18 23:34 ` [PATCH 06/15] gitlab-ci: Extract container job template to container-template.yml Philippe Mathieu-Daudé
                   ` (9 subsequent siblings)
  14 siblings, 1 reply; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

From: Philippe Mathieu-Daudé <philmd@redhat.com>

The acceptance tests aren't accepting anything,
rename them as integration tests.

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci.yml | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 07e5e737685..bf4862a78de 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -48,7 +48,7 @@ include:
     # Avoid recompiling by hiding ninja with NINJA=":"
     - make NINJA=":" $MAKE_CHECK_ARGS
 
-.acceptance_test_job_template:
+.integration_test_job_template:
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
@@ -111,7 +111,7 @@ acceptance-system-alpine:
   variables:
     IMAGE: alpine
     MAKE_CHECK_ARGS: check-acceptance
-  extends: .acceptance_test_job_template
+  extends: .integration_test_job_template
 
 build-system-ubuntu:
   extends: .native_build_job_template
@@ -145,7 +145,7 @@ acceptance-system-ubuntu:
   variables:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check-acceptance
-  extends: .acceptance_test_job_template
+  extends: .integration_test_job_template
 
 build-system-debian:
   extends: .native_build_job_template
@@ -179,7 +179,7 @@ acceptance-system-debian:
   variables:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-acceptance
-  extends: .acceptance_test_job_template
+  extends: .integration_test_job_template
 
 build-system-fedora:
   extends: .native_build_job_template
@@ -214,7 +214,7 @@ acceptance-system-fedora:
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  extends: .acceptance_test_job_template
+  extends: .integration_test_job_template
 
 build-system-centos:
   extends: .native_build_job_template
@@ -249,7 +249,7 @@ acceptance-system-centos:
   variables:
     IMAGE: centos8
     MAKE_CHECK_ARGS: check-acceptance
-  extends: .acceptance_test_job_template
+  extends: .integration_test_job_template
 
 build-system-opensuse:
   extends: .native_build_job_template
@@ -282,7 +282,7 @@ acceptance-system-opensuse:
   variables:
     IMAGE: opensuse-leap
     MAKE_CHECK_ARGS: check-acceptance
-  extends: .acceptance_test_job_template
+  extends: .integration_test_job_template
 
 
 build-disabled:
@@ -525,7 +525,7 @@ acceptance-cfi-aarch64:
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  extends: .acceptance_test_job_template
+  extends: .integration_test_job_template
 
 build-cfi-ppc64-s390x:
   extends: .native_build_job_template
@@ -562,7 +562,7 @@ acceptance-cfi-ppc64-s390x:
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  extends: .acceptance_test_job_template
+  extends: .integration_test_job_template
 
 build-cfi-x86_64:
   extends: .native_build_job_template
@@ -599,7 +599,7 @@ acceptance-cfi-x86_64:
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  extends: .acceptance_test_job_template
+  extends: .integration_test_job_template
 
 tsan-build:
   extends: .native_build_job_template
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 06/15] gitlab-ci: Extract container job template to container-template.yml
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (4 preceding siblings ...)
  2021-04-18 23:34 ` [PATCH 05/15] gitlab-ci: Rename acceptance_test_job -> integration_test_job Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [PATCH 07/15] gitlab-ci: Extract crossbuild job templates to crossbuild-template.yml Philippe Mathieu-Daudé
                   ` (8 subsequent siblings)
  14 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

Extract the container job template to a new file
(container-template.yml) to be able to reuse it
without having to run all the jobs included, which
are mainly useful for mainstream CI.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/container-template.yml | 22 ++++++++++++++++++++++
 .gitlab-ci.d/containers.yml         | 24 ++----------------------
 2 files changed, 24 insertions(+), 22 deletions(-)
 create mode 100644 .gitlab-ci.d/container-template.yml

diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
new file mode 100644
index 00000000000..9e3f3ae3c26
--- /dev/null
+++ b/.gitlab-ci.d/container-template.yml
@@ -0,0 +1,22 @@
+.container_job_template:
+  image: docker:stable
+  stage: containers
+  services:
+    - docker:dind
+  before_script:
+    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
+    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
+    - apk add python3
+    - docker info
+    - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
+  script:
+    - echo "TAG:$TAG"
+    - echo "COMMON_TAG:$COMMON_TAG"
+    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+    - ./tests/docker/docker.py --engine docker build
+          -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
+          -r $CI_REGISTRY_IMAGE
+    - docker tag "qemu/$NAME" "$TAG"
+    - docker push "$TAG"
+  after_script:
+    - docker logout
diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 4ef76d1f54d..0bf4899df10 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -1,25 +1,5 @@
-.container_job_template:
-  image: docker:stable
-  stage: containers
-  services:
-    - docker:dind
-  before_script:
-    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
-    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
-    - apk add python3
-    - docker info
-    - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
-  script:
-    - echo "TAG:$TAG"
-    - echo "COMMON_TAG:$COMMON_TAG"
-    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
-    - ./tests/docker/docker.py --engine docker build
-          -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
-          -r $CI_REGISTRY_IMAGE
-    - docker tag "qemu/$NAME" "$TAG"
-    - docker push "$TAG"
-  after_script:
-    - docker logout
+include:
+  - local: '/.gitlab-ci.d/container-template.yml'
 
 amd64-alpine-container:
   extends: .container_job_template
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 07/15] gitlab-ci: Extract crossbuild job templates to crossbuild-template.yml
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (5 preceding siblings ...)
  2021-04-18 23:34 ` [PATCH 06/15] gitlab-ci: Extract container job template to container-template.yml Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [PATCH 08/15] gitlab-ci: Extract DCO/style check jobs to checks.yml Philippe Mathieu-Daudé
                   ` (7 subsequent siblings)
  14 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

Extract the crossbuild job templates to a new file
(crossbuild-template.yml) to be able to reuse them
without having to run all the jobs included, which
are mainly useful for mainstream CI.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/crossbuild-template.yml | 41 ++++++++++++++++++++++++++
 .gitlab-ci.d/crossbuilds.yml         | 43 ++--------------------------
 2 files changed, 43 insertions(+), 41 deletions(-)
 create mode 100644 .gitlab-ci.d/crossbuild-template.yml

diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
new file mode 100644
index 00000000000..bc790a43486
--- /dev/null
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -0,0 +1,41 @@
+.cross_system_build_job:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  timeout: 80m
+  script:
+    - mkdir build
+    - cd build
+    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
+      ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
+        --disable-user --target-list-exclude="arm-softmmu cris-softmmu
+          i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
+          mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
+    - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
+
+# Job to cross-build specific accelerators.
+#
+# Set the $ACCEL variable to select the specific accelerator (default to
+# KVM), and set extra options (such disabling other accelerators) via the
+# $ACCEL_CONFIGURE_OPTS variable.
+.cross_accel_build_job:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  timeout: 30m
+  script:
+    - mkdir build
+    - cd build
+    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
+      ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
+        --disable-tools --enable-${ACCEL:-kvm} $ACCEL_CONFIGURE_OPTS
+    - make -j$(expr $(nproc) + 1) all check-build
+
+.cross_user_build_job:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  script:
+    - mkdir build
+    - cd build
+    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
+      ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
+        --disable-system
+    - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 2d95784ed51..7a0d2069c14 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -1,44 +1,5 @@
-.cross_system_build_job:
-  stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
-  timeout: 80m
-  script:
-    - mkdir build
-    - cd build
-    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
-      ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
-        --disable-user --target-list-exclude="arm-softmmu cris-softmmu
-          i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
-          mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
-    - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
-
-# Job to cross-build specific accelerators.
-#
-# Set the $ACCEL variable to select the specific accelerator (default to
-# KVM), and set extra options (such disabling other accelerators) via the
-# $ACCEL_CONFIGURE_OPTS variable.
-.cross_accel_build_job:
-  stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
-  timeout: 30m
-  script:
-    - mkdir build
-    - cd build
-    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
-      ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
-        --disable-tools --enable-${ACCEL:-kvm} $ACCEL_CONFIGURE_OPTS
-    - make -j$(expr $(nproc) + 1) all check-build
-
-.cross_user_build_job:
-  stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
-  script:
-    - mkdir build
-    - cd build
-    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
-      ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
-        --disable-system
-    - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
+include:
+  - local: '/.gitlab-ci.d/crossbuild-template.yml'
 
 cross-armel-system:
   extends: .cross_system_build_job
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 08/15] gitlab-ci: Extract DCO/style check jobs to checks.yml
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (6 preceding siblings ...)
  2021-04-18 23:34 ` [PATCH 07/15] gitlab-ci: Extract crossbuild job templates to crossbuild-template.yml Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-19  5:26   ` Thomas Huth
  2021-04-18 23:34 ` [PATCH 09/15] gitlab-ci: Extract build stages to stages.yml Philippe Mathieu-Daudé
                   ` (6 subsequent siblings)
  14 siblings, 1 reply; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

Extract the DCO / checkpatch jobs to a new file (checks.yml)
to be able to run them without having to run all the jobs
included in the default .gitlab-ci.yml, which are mainly useful
for mainstream CI.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/checks.yml | 24 ++++++++++++++++++++++++
 .gitlab-ci.yml          |  1 +
 2 files changed, 25 insertions(+)
 create mode 100644 .gitlab-ci.d/checks.yml

diff --git a/.gitlab-ci.d/checks.yml b/.gitlab-ci.d/checks.yml
new file mode 100644
index 00000000000..f695627b7cd
--- /dev/null
+++ b/.gitlab-ci.d/checks.yml
@@ -0,0 +1,24 @@
+check-patch:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  needs:
+    job: amd64-centos8-container
+  script: .gitlab-ci.d/check-patch.py
+  except:
+    variables:
+      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
+  variables:
+    GIT_DEPTH: 1000
+  allow_failure: true
+
+check-dco:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  needs:
+    job: amd64-centos8-container
+  script: .gitlab-ci.d/check-dco.py
+  except:
+    variables:
+      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
+  variables:
+    GIT_DEPTH: 1000
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index bf4862a78de..ba35b3917e7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -12,6 +12,7 @@ include:
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
+  - local: '/.gitlab-ci.d/checks.yml'
 
 .native_build_job_template:
   stage: build
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 09/15] gitlab-ci: Extract build stages to stages.yml
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (7 preceding siblings ...)
  2021-04-18 23:34 ` [PATCH 08/15] gitlab-ci: Extract DCO/style check jobs to checks.yml Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml Philippe Mathieu-Daudé
                   ` (5 subsequent siblings)
  14 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

Extract the build stages used by our job templates to a new file
(stages.yml) to be able to include it with the other templates,
without having to run all the jobs included in the default
.gitlab-ci.yml, which are mainly useful for mainstream CI.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/stages.yml |  8 ++++++++
 .gitlab-ci.yml          | 10 +---------
 2 files changed, 9 insertions(+), 9 deletions(-)
 create mode 100644 .gitlab-ci.d/stages.yml

diff --git a/.gitlab-ci.d/stages.yml b/.gitlab-ci.d/stages.yml
new file mode 100644
index 00000000000..f50826018df
--- /dev/null
+++ b/.gitlab-ci.d/stages.yml
@@ -0,0 +1,8 @@
+# Currently we have two build stages after our containers are built:
+#  - build (for traditional build and test or first stage build)
+#  - test (for test stages, using build artefacts from a build stage)
+stages:
+  - containers
+  - containers-layer2
+  - build
+  - test
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ba35b3917e7..c367d42977c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,13 +1,5 @@
-# Currently we have two build stages after our containers are built:
-#  - build (for traditional build and test or first stage build)
-#  - test (for test stages, using build artefacts from a build stage)
-stages:
-  - containers
-  - containers-layer2
-  - build
-  - test
-
 include:
+  - local: '/.gitlab-ci.d/stages.yml'
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (8 preceding siblings ...)
  2021-04-18 23:34 ` [PATCH 09/15] gitlab-ci: Extract build stages to stages.yml Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-19  5:39   ` Thomas Huth
  2021-04-19 15:11   ` Alex Bennée
  2021-04-18 23:34 ` [PATCH 11/15] gitlab-ci: Extract core container jobs to container-core.yml Philippe Mathieu-Daudé
                   ` (4 subsequent siblings)
  14 siblings, 2 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

Extract the build/test jobs run by default on the mainstream
CI into a new file (buildtest.yml) and their templates into
(buildtest-template.yml), to be able to reuse the templates
without having to run all these mainstream jobs by default.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml |  80 +++
 .gitlab-ci.d/buildtest.yml          | 744 +++++++++++++++++++++++++
 .gitlab-ci.yml                      | 824 +---------------------------
 3 files changed, 825 insertions(+), 823 deletions(-)
 create mode 100644 .gitlab-ci.d/buildtest-template.yml
 create mode 100644 .gitlab-ci.d/buildtest.yml

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
new file mode 100644
index 00000000000..b24a153d904
--- /dev/null
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -0,0 +1,80 @@
+.native_build_job_template:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  before_script:
+    - JOBS=$(expr $(nproc) + 1)
+  script:
+    - mkdir build
+    - cd build
+    - if test -n "$TARGETS";
+      then
+        ../configure --enable-werror --disable-docs $CONFIGURE_ARGS --target-list="$TARGETS" ;
+      else
+        ../configure --enable-werror --disable-docs $CONFIGURE_ARGS ;
+      fi || { cat config.log meson-logs/meson-log.txt && exit 1; }
+    - if test -n "$LD_JOBS";
+      then
+        meson configure . -Dbackend_max_links="$LD_JOBS" ;
+      fi || exit 1;
+    - make -j"$JOBS"
+    - if test -n "$MAKE_CHECK_ARGS";
+      then
+        make -j"$JOBS" $MAKE_CHECK_ARGS ;
+      fi
+
+.native_test_job_template:
+  stage: test
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  script:
+    - scripts/git-submodule.sh update
+        $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
+    - cd build
+    - find . -type f -exec touch {} +
+    # Avoid recompiling by hiding ninja with NINJA=":"
+    - make NINJA=":" $MAKE_CHECK_ARGS
+
+.integration_test_job_template:
+  cache:
+    key: "${CI_JOB_NAME}-cache"
+    paths:
+      - ${CI_PROJECT_DIR}/avocado-cache
+    policy: pull-push
+  artifacts:
+    name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
+    when: always
+    expire_in: 2 days
+    paths:
+      - build/tests/results/latest/results.xml
+      - build/tests/results/latest/test-results
+    reports:
+      junit: build/tests/results/latest/results.xml
+  before_script:
+    - mkdir -p ~/.config/avocado
+    - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
+    - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
+           >> ~/.config/avocado/avocado.conf
+    - echo -e '[job.output.testlogs]\nstatuses = ["FAIL", "INTERRUPT"]'
+           >> ~/.config/avocado/avocado.conf
+    - if [ -d ${CI_PROJECT_DIR}/avocado-cache ]; then
+        du -chs ${CI_PROJECT_DIR}/avocado-cache ;
+      fi
+    - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
+  after_script:
+    - cd build
+    - du -chs ${CI_PROJECT_DIR}/avocado-cache
+
+build-system-alpine:
+  extends: .native_build_job_template
+  needs:
+    - job: amd64-alpine-container
+  variables:
+    IMAGE: alpine
+    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
+      moxie-softmmu microblazeel-softmmu mips64el-softmmu
+    MAKE_CHECK_ARGS: check-build
+    CONFIGURE_ARGS: --enable-docs --enable-trace-backends=log,simple,syslog
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - .git-submodule-status
+      - build
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
new file mode 100644
index 00000000000..c24553eab05
--- /dev/null
+++ b/.gitlab-ci.d/buildtest.yml
@@ -0,0 +1,744 @@
+include:
+  - local: '/.gitlab-ci.d/buildtest-template.yml'
+
+check-system-alpine:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-alpine
+      artifacts: true
+  variables:
+    IMAGE: alpine
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-alpine:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-alpine
+      artifacts: true
+  variables:
+    IMAGE: alpine
+    MAKE_CHECK_ARGS: check-acceptance
+  extends: .integration_test_job_template
+
+build-system-ubuntu:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-ubuntu2004-container
+  variables:
+    IMAGE: ubuntu2004
+    CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-slirp=system
+    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
+      moxie-softmmu microblazeel-softmmu mips64el-softmmu
+    MAKE_CHECK_ARGS: check-build
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
+check-system-ubuntu:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-ubuntu
+      artifacts: true
+  variables:
+    IMAGE: ubuntu2004
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-ubuntu:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-ubuntu
+      artifacts: true
+  variables:
+    IMAGE: ubuntu2004
+    MAKE_CHECK_ARGS: check-acceptance
+  extends: .integration_test_job_template
+
+build-system-debian:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-container
+  variables:
+    IMAGE: debian-amd64
+    CONFIGURE_ARGS: --enable-fdt=system
+    TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
+      riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
+    MAKE_CHECK_ARGS: check-build
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
+check-system-debian:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-debian
+      artifacts: true
+  variables:
+    IMAGE: debian-amd64
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-debian:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-debian
+      artifacts: true
+  variables:
+    IMAGE: debian-amd64
+    MAKE_CHECK_ARGS: check-acceptance
+  extends: .integration_test_job_template
+
+build-system-fedora:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-fedora-container
+  variables:
+    IMAGE: fedora
+    CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs
+             --enable-fdt=system --enable-slirp=system --enable-capstone=system
+    TARGETS: tricore-softmmu microblaze-softmmu mips-softmmu
+      xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu
+    MAKE_CHECK_ARGS: check-build
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
+check-system-fedora:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-fedora
+      artifacts: true
+  variables:
+    IMAGE: fedora
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-fedora:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-fedora
+      artifacts: true
+  variables:
+    IMAGE: fedora
+    MAKE_CHECK_ARGS: check-acceptance
+  extends: .integration_test_job_template
+
+build-system-centos:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-centos8-container
+  variables:
+    IMAGE: centos8
+    CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
+                    --enable-modules --enable-trace-backends=dtrace
+    TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
+      x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
+    MAKE_CHECK_ARGS: check-build
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
+check-system-centos:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos8
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-centos:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos8
+    MAKE_CHECK_ARGS: check-acceptance
+  extends: .integration_test_job_template
+
+build-system-opensuse:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-opensuse-leap-container
+  variables:
+    IMAGE: opensuse-leap
+    CONFIGURE_ARGS: --enable-fdt=system
+    TARGETS: s390x-softmmu x86_64-softmmu aarch64-softmmu
+    MAKE_CHECK_ARGS: check-build
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
+check-system-opensuse:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-opensuse
+      artifacts: true
+  variables:
+    IMAGE: opensuse-leap
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-opensuse:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-opensuse
+      artifacts: true
+  variables:
+    IMAGE: opensuse-leap
+    MAKE_CHECK_ARGS: check-acceptance
+  extends: .integration_test_job_template
+
+
+build-disabled:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-fedora-container
+  variables:
+    IMAGE: fedora
+    CONFIGURE_ARGS:
+      --disable-attr
+      --disable-auth-pam
+      --disable-avx2
+      --disable-bochs
+      --disable-brlapi
+      --disable-bzip2
+      --disable-cap-ng
+      --disable-capstone
+      --disable-cloop
+      --disable-coroutine-pool
+      --disable-curl
+      --disable-curses
+      --disable-dmg
+      --disable-docs
+      --disable-gcrypt
+      --disable-glusterfs
+      --disable-gnutls
+      --disable-gtk
+      --disable-guest-agent
+      --disable-iconv
+      --disable-keyring
+      --disable-kvm
+      --disable-libiscsi
+      --disable-libpmem
+      --disable-libssh
+      --disable-libudev
+      --disable-libusb
+      --disable-libxml2
+      --disable-linux-aio
+      --disable-live-block-migration
+      --disable-lzo
+      --disable-malloc-trim
+      --disable-mpath
+      --disable-nettle
+      --disable-numa
+      --disable-opengl
+      --disable-parallels
+      --disable-pie
+      --disable-qcow1
+      --disable-qed
+      --disable-qom-cast-debug
+      --disable-rbd
+      --disable-rdma
+      --disable-replication
+      --disable-sdl
+      --disable-seccomp
+      --disable-sheepdog
+      --disable-slirp
+      --disable-smartcard
+      --disable-snappy
+      --disable-sparse
+      --disable-spice
+      --disable-strip
+      --disable-tpm
+      --disable-usb-redir
+      --disable-vdi
+      --disable-vhost-crypto
+      --disable-vhost-net
+      --disable-vhost-scsi
+      --disable-vhost-kernel
+      --disable-vhost-user
+      --disable-vhost-vdpa
+      --disable-vhost-vsock
+      --disable-virglrenderer
+      --disable-vnc
+      --disable-vte
+      --disable-vvfat
+      --disable-xen
+      --disable-zstd
+    TARGETS: arm-softmmu i386-softmmu ppc64-softmmu mips64-softmmu
+      s390x-softmmu i386-linux-user
+    MAKE_CHECK_ARGS: check-qtest SPEED=slow
+
+# This jobs explicitly disable TCG (--disable-tcg), KVM is detected by
+# the configure script. The container doesn't contain Xen headers so
+# Xen accelerator is not detected / selected. As result it build the
+# i386-softmmu and x86_64-softmmu with KVM being the single accelerator
+# available.
+# Also use a different coroutine implementation (which is only really of
+# interest to KVM users, i.e. with TCG disabled)
+build-tcg-disabled:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-centos8-container
+  variables:
+    IMAGE: centos8
+  script:
+    - mkdir build
+    - cd build
+    - ../configure --disable-tcg --audio-drv-list="" --with-coroutine=ucontext
+      || { cat config.log meson-logs/meson-log.txt && exit 1; }
+    - make -j"$JOBS"
+    - make check-unit
+    - make check-qapi-schema
+    - cd tests/qemu-iotests/
+    - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
+            052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163
+            170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277
+    - ./check -qcow2 028 051 056 057 058 065 068 082 085 091 095 096 102 122
+            124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202
+            208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258
+            260 261 262 263 264 270 272 273 277 279
+
+build-user:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-user-cross-container
+  variables:
+    IMAGE: debian-all-test-cross
+    CONFIGURE_ARGS: --disable-tools --disable-system
+    MAKE_CHECK_ARGS: check-tcg
+
+build-user-static:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-user-cross-container
+  variables:
+    IMAGE: debian-all-test-cross
+    CONFIGURE_ARGS: --disable-tools --disable-system --static
+    MAKE_CHECK_ARGS: check-tcg
+
+# Only build the softmmu targets we have check-tcg tests for
+build-some-softmmu:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-user-cross-container
+  variables:
+    IMAGE: debian-all-test-cross
+    CONFIGURE_ARGS: --disable-tools --enable-debug
+    TARGETS: xtensa-softmmu arm-softmmu aarch64-softmmu alpha-softmmu
+    MAKE_CHECK_ARGS: check-tcg
+
+# Run check-tcg against linux-user (with plugins)
+# we skip sparc64-linux-user until it has been fixed somewhat
+# we skip cris-linux-user as it doesn't use the common run loop
+build-user-plugins:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-user-cross-container
+  variables:
+    IMAGE: debian-all-test-cross
+    CONFIGURE_ARGS: --disable-tools --disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user,cris-linux-user
+    MAKE_CHECK_ARGS: check-tcg
+  timeout: 1h 30m
+
+build-user-centos7:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-centos7-container
+  variables:
+    IMAGE: centos7
+    CONFIGURE_ARGS: --disable-system --disable-tools --disable-docs
+    MAKE_CHECK_ARGS: check-tcg
+
+build-some-softmmu-plugins:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-user-cross-container
+  variables:
+    IMAGE: debian-all-test-cross
+    CONFIGURE_ARGS: --disable-tools --disable-user --enable-plugins --enable-debug-tcg
+    TARGETS: xtensa-softmmu arm-softmmu aarch64-softmmu alpha-softmmu
+    MAKE_CHECK_ARGS: check-tcg
+
+clang-system:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-fedora-container
+  variables:
+    IMAGE: fedora
+    CONFIGURE_ARGS: --cc=clang --cxx=clang++
+      --extra-cflags=-fsanitize=undefined --extra-cflags=-fno-sanitize-recover=undefined
+    TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
+      ppc-softmmu s390x-softmmu
+    MAKE_CHECK_ARGS: check-qtest check-tcg
+
+clang-user:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-user-cross-container
+  variables:
+    IMAGE: debian-all-test-cross
+    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --disable-system
+      --target-list-exclude=microblazeel-linux-user,aarch64_be-linux-user,i386-linux-user,m68k-linux-user,mipsn32el-linux-user,xtensaeb-linux-user
+      --extra-cflags=-fsanitize=undefined --extra-cflags=-fno-sanitize-recover=undefined
+    MAKE_CHECK_ARGS: check-unit check-tcg
+
+# Set LD_JOBS=1 because this requires LTO and ld consumes a large amount of memory.
+# On gitlab runners, default value sometimes end up calling 2 lds concurrently and
+# triggers an Out-Of-Memory error
+#
+# Since slirp callbacks are used in QEMU Timers, slirp needs to be compiled together
+# with QEMU and linked as a static library to avoid false positives in CFI checks.
+# This can be accomplished by using -enable-slirp=git, which avoids the use of
+# a system-wide version of the library
+#
+# Split in three sets of build/check/acceptance to limit the execution time of each
+# job
+build-cfi-aarch64:
+  extends: .native_build_job_template
+  needs:
+  - job: amd64-fedora-container
+  variables:
+    LD_JOBS: 1
+    AR: llvm-ar
+    IMAGE: fedora
+    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-cfi --enable-cfi-debug
+      --enable-safe-stack --enable-slirp=git
+    TARGETS: aarch64-softmmu
+    MAKE_CHECK_ARGS: check-build
+  timeout: 70m
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
+check-cfi-aarch64:
+  extends: .native_test_job_template
+  needs:
+    - job: build-cfi-aarch64
+      artifacts: true
+  variables:
+    IMAGE: fedora
+    MAKE_CHECK_ARGS: check
+
+acceptance-cfi-aarch64:
+  extends: .native_test_job_template
+  needs:
+    - job: build-cfi-aarch64
+      artifacts: true
+  variables:
+    IMAGE: fedora
+    MAKE_CHECK_ARGS: check-acceptance
+  extends: .integration_test_job_template
+
+build-cfi-ppc64-s390x:
+  extends: .native_build_job_template
+  needs:
+  - job: amd64-fedora-container
+  variables:
+    LD_JOBS: 1
+    AR: llvm-ar
+    IMAGE: fedora
+    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-cfi --enable-cfi-debug
+      --enable-safe-stack --enable-slirp=git
+    TARGETS: ppc64-softmmu s390x-softmmu
+    MAKE_CHECK_ARGS: check-build
+  timeout: 70m
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
+check-cfi-ppc64-s390x:
+  extends: .native_test_job_template
+  needs:
+    - job: build-cfi-ppc64-s390x
+      artifacts: true
+  variables:
+    IMAGE: fedora
+    MAKE_CHECK_ARGS: check
+
+acceptance-cfi-ppc64-s390x:
+  extends: .native_test_job_template
+  needs:
+    - job: build-cfi-ppc64-s390x
+      artifacts: true
+  variables:
+    IMAGE: fedora
+    MAKE_CHECK_ARGS: check-acceptance
+  extends: .integration_test_job_template
+
+build-cfi-x86_64:
+  extends: .native_build_job_template
+  needs:
+  - job: amd64-fedora-container
+  variables:
+    LD_JOBS: 1
+    AR: llvm-ar
+    IMAGE: fedora
+    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-cfi --enable-cfi-debug
+      --enable-safe-stack --enable-slirp=git
+    TARGETS: x86_64-softmmu
+    MAKE_CHECK_ARGS: check-build
+  timeout: 70m
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
+check-cfi-x86_64:
+  extends: .native_test_job_template
+  needs:
+    - job: build-cfi-x86_64
+      artifacts: true
+  variables:
+    IMAGE: fedora
+    MAKE_CHECK_ARGS: check
+
+acceptance-cfi-x86_64:
+  extends: .native_test_job_template
+  needs:
+    - job: build-cfi-x86_64
+      artifacts: true
+  variables:
+    IMAGE: fedora
+    MAKE_CHECK_ARGS: check-acceptance
+  extends: .integration_test_job_template
+
+tsan-build:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-ubuntu2004-container
+  variables:
+    IMAGE: ubuntu2004
+    CONFIGURE_ARGS: --enable-tsan --cc=clang-10 --cxx=clang++-10
+          --enable-trace-backends=ust --enable-fdt=system --enable-slirp=system
+    TARGETS: x86_64-softmmu ppc64-softmmu riscv64-softmmu x86_64-linux-user
+    MAKE_CHECK_ARGS: bench V=1
+
+# These targets are on the way out
+build-deprecated:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-user-cross-container
+  variables:
+    IMAGE: debian-all-test-cross
+    CONFIGURE_ARGS: --disable-tools
+    MAKE_CHECK_ARGS: build-tcg
+    TARGETS: ppc64abi32-linux-user lm32-softmmu unicore32-softmmu
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
+# We split the check-tcg step as test failures are expected but we still
+# want to catch the build breaking.
+check-deprecated:
+  extends: .native_test_job_template
+  needs:
+    - job: build-deprecated
+      artifacts: true
+  variables:
+    IMAGE: debian-all-test-cross
+    MAKE_CHECK_ARGS: check-tcg
+  allow_failure: true
+
+# gprof/gcov are GCC features
+gprof-gcov:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-ubuntu2004-container
+  variables:
+    IMAGE: ubuntu2004
+    CONFIGURE_ARGS: --enable-gprof --enable-gcov
+    MAKE_CHECK_ARGS: check
+    TARGETS: aarch64-softmmu ppc64-softmmu s390x-softmmu x86_64-softmmu
+  timeout: 70m
+  after_script:
+    - ${CI_PROJECT_DIR}/scripts/ci/coverage-summary.sh
+
+build-oss-fuzz:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-fedora-container
+  variables:
+    IMAGE: fedora
+  script:
+    - mkdir build-oss-fuzz
+    - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
+      ./scripts/oss-fuzz/build.sh
+    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
+    - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f
+                      | grep -v slirp); do
+        grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
+        echo Testing ${fuzzer} ... ;
+        "${fuzzer}" -runs=1 -seed=1 || exit 1 ;
+      done
+    # Unrelated to fuzzer: run some tests with -fsanitize=address
+    - cd build-oss-fuzz && make check-qtest-i386 check-unit
+
+build-tci:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-user-cross-container
+  variables:
+    IMAGE: debian-all-test-cross
+  script:
+    - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
+    - mkdir build
+    - cd build
+    - ../configure --enable-tcg-interpreter
+        --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)" || { cat config.log meson-logs/meson-log.txt && exit 1; }
+    - make -j"$JOBS"
+    - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
+    - for tg in $TARGETS ; do
+        export QTEST_QEMU_BINARY="./qemu-system-${tg}" ;
+        ./tests/qtest/boot-serial-test || exit 1 ;
+        ./tests/qtest/cdrom-test || exit 1 ;
+      done
+    - QTEST_QEMU_BINARY="./qemu-system-x86_64" ./tests/qtest/pxe-test
+    - QTEST_QEMU_BINARY="./qemu-system-s390x" ./tests/qtest/pxe-test -m slow
+    - make check-tcg
+
+# Alternate coroutines implementations are only really of interest to KVM users
+# However we can't test against KVM on Gitlab-CI so we can only run unit tests
+build-coroutine-sigaltstack:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-ubuntu2004-container
+  variables:
+    IMAGE: ubuntu2004
+    CONFIGURE_ARGS: --with-coroutine=sigaltstack --disable-tcg
+                    --enable-trace-backends=ftrace
+    MAKE_CHECK_ARGS: check-unit
+
+# Most jobs test latest gcrypt or nettle builds
+#
+# These jobs test old gcrypt and nettle from RHEL7
+# which had some API differences.
+crypto-old-nettle:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-centos7-container
+  variables:
+    IMAGE: centos7
+    TARGETS: x86_64-softmmu x86_64-linux-user
+    CONFIGURE_ARGS: --disable-gcrypt --enable-nettle
+    MAKE_CHECK_ARGS: check
+
+crypto-old-gcrypt:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-centos7-container
+  variables:
+    IMAGE: centos7
+    TARGETS: x86_64-softmmu x86_64-linux-user
+    CONFIGURE_ARGS: --disable-nettle --enable-gcrypt
+    MAKE_CHECK_ARGS: check
+
+crypto-only-gnutls:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-centos7-container
+  variables:
+    IMAGE: centos7
+    TARGETS: x86_64-softmmu x86_64-linux-user
+    CONFIGURE_ARGS: --disable-nettle --disable-gcrypt --enable-gnutls
+    MAKE_CHECK_ARGS: check
+
+
+# Check our reduced build configurations
+build-without-default-devices:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-centos8-container
+  variables:
+    IMAGE: centos8
+    CONFIGURE_ARGS: --without-default-devices --disable-user
+
+build-without-default-features:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-container
+  variables:
+    IMAGE: debian-amd64
+    CONFIGURE_ARGS: --without-default-features --disable-user
+        --target-list-exclude=arm-softmmu,i386-softmmu,mipsel-softmmu,mips64-softmmu,ppc-softmmu
+    MAKE_CHECK_ARGS: check-unit
+
+check-patch:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  needs:
+    job: amd64-centos8-container
+  script: .gitlab-ci.d/check-patch.py
+  except:
+    variables:
+      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
+  variables:
+    GIT_DEPTH: 1000
+  allow_failure: true
+
+check-dco:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
+  needs:
+    job: amd64-centos8-container
+  script: .gitlab-ci.d/check-dco.py
+  except:
+    variables:
+      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
+  variables:
+    GIT_DEPTH: 1000
+
+build-libvhost-user:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
+  needs:
+    job: amd64-fedora-container
+  before_script:
+    - dnf install -y meson ninja-build
+  script:
+    - mkdir subprojects/libvhost-user/build
+    - cd subprojects/libvhost-user/build
+    - meson
+    - ninja
+
+# No targets are built here, just tools, docs, and unit tests. This
+# also feeds into the eventual documentation deployment steps later
+build-tools-and-docs-debian:
+  extends: .native_build_job_template
+  needs:
+    job: amd64-debian-container
+  variables:
+    IMAGE: debian-amd64
+    MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
+    CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - build
+
+# Prepare for GitLab pages deployment. Anything copied into the
+# "public" directory will be deployed to $USER.gitlab.io/$PROJECT
+pages:
+  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
+  stage: test
+  needs:
+    - job: build-tools-and-docs-debian
+  script:
+    - mkdir -p public
+    # HTML-ised source tree
+    - make gtags
+    - htags -anT --tree-view=filetree -m qemu_init
+        -t "Welcome to the QEMU sourcecode"
+    - mv HTML public/src
+    # Project documentation
+    - make -C build install DESTDIR=$(pwd)/temp-install
+    - mv temp-install/usr/local/share/doc/qemu/* public/
+  artifacts:
+    paths:
+      - public
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c367d42977c..8f13a584a29 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -4,827 +4,5 @@ include:
   - local: '/.gitlab-ci.d/opensbi.yml'
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
+  - local: '/.gitlab-ci.d/buildtest.yml'
   - local: '/.gitlab-ci.d/checks.yml'
-
-.native_build_job_template:
-  stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
-  before_script:
-    - JOBS=$(expr $(nproc) + 1)
-  script:
-    - mkdir build
-    - cd build
-    - if test -n "$TARGETS";
-      then
-        ../configure --enable-werror --disable-docs $CONFIGURE_ARGS --target-list="$TARGETS" ;
-      else
-        ../configure --enable-werror --disable-docs $CONFIGURE_ARGS ;
-      fi || { cat config.log meson-logs/meson-log.txt && exit 1; }
-    - if test -n "$LD_JOBS";
-      then
-        meson configure . -Dbackend_max_links="$LD_JOBS" ;
-      fi || exit 1;
-    - make -j"$JOBS"
-    - if test -n "$MAKE_CHECK_ARGS";
-      then
-        make -j"$JOBS" $MAKE_CHECK_ARGS ;
-      fi
-
-.native_test_job_template:
-  stage: test
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
-  script:
-    - scripts/git-submodule.sh update
-        $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
-    - cd build
-    - find . -type f -exec touch {} +
-    # Avoid recompiling by hiding ninja with NINJA=":"
-    - make NINJA=":" $MAKE_CHECK_ARGS
-
-.integration_test_job_template:
-  cache:
-    key: "${CI_JOB_NAME}-cache"
-    paths:
-      - ${CI_PROJECT_DIR}/avocado-cache
-    policy: pull-push
-  artifacts:
-    name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
-    when: always
-    expire_in: 2 days
-    paths:
-      - build/tests/results/latest/results.xml
-      - build/tests/results/latest/test-results
-    reports:
-      junit: build/tests/results/latest/results.xml
-  before_script:
-    - mkdir -p ~/.config/avocado
-    - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
-    - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
-           >> ~/.config/avocado/avocado.conf
-    - echo -e '[job.output.testlogs]\nstatuses = ["FAIL", "INTERRUPT"]'
-           >> ~/.config/avocado/avocado.conf
-    - if [ -d ${CI_PROJECT_DIR}/avocado-cache ]; then
-        du -chs ${CI_PROJECT_DIR}/avocado-cache ;
-      fi
-    - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
-  after_script:
-    - cd build
-    - du -chs ${CI_PROJECT_DIR}/avocado-cache
-
-build-system-alpine:
-  extends: .native_build_job_template
-  needs:
-    - job: amd64-alpine-container
-  variables:
-    IMAGE: alpine
-    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
-      moxie-softmmu microblazeel-softmmu mips64el-softmmu
-    MAKE_CHECK_ARGS: check-build
-    CONFIGURE_ARGS: --enable-docs --enable-trace-backends=log,simple,syslog
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - .git-submodule-status
-      - build
-
-check-system-alpine:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-alpine
-      artifacts: true
-  variables:
-    IMAGE: alpine
-    MAKE_CHECK_ARGS: check
-
-acceptance-system-alpine:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-alpine
-      artifacts: true
-  variables:
-    IMAGE: alpine
-    MAKE_CHECK_ARGS: check-acceptance
-  extends: .integration_test_job_template
-
-build-system-ubuntu:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-ubuntu2004-container
-  variables:
-    IMAGE: ubuntu2004
-    CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-slirp=system
-    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
-      moxie-softmmu microblazeel-softmmu mips64el-softmmu
-    MAKE_CHECK_ARGS: check-build
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
-check-system-ubuntu:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-ubuntu
-      artifacts: true
-  variables:
-    IMAGE: ubuntu2004
-    MAKE_CHECK_ARGS: check
-
-acceptance-system-ubuntu:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-ubuntu
-      artifacts: true
-  variables:
-    IMAGE: ubuntu2004
-    MAKE_CHECK_ARGS: check-acceptance
-  extends: .integration_test_job_template
-
-build-system-debian:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-container
-  variables:
-    IMAGE: debian-amd64
-    CONFIGURE_ARGS: --enable-fdt=system
-    TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
-      riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
-    MAKE_CHECK_ARGS: check-build
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
-check-system-debian:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-debian
-      artifacts: true
-  variables:
-    IMAGE: debian-amd64
-    MAKE_CHECK_ARGS: check
-
-acceptance-system-debian:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-debian
-      artifacts: true
-  variables:
-    IMAGE: debian-amd64
-    MAKE_CHECK_ARGS: check-acceptance
-  extends: .integration_test_job_template
-
-build-system-fedora:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-fedora-container
-  variables:
-    IMAGE: fedora
-    CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs
-             --enable-fdt=system --enable-slirp=system --enable-capstone=system
-    TARGETS: tricore-softmmu microblaze-softmmu mips-softmmu
-      xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu
-    MAKE_CHECK_ARGS: check-build
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
-check-system-fedora:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-fedora
-      artifacts: true
-  variables:
-    IMAGE: fedora
-    MAKE_CHECK_ARGS: check
-
-acceptance-system-fedora:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-fedora
-      artifacts: true
-  variables:
-    IMAGE: fedora
-    MAKE_CHECK_ARGS: check-acceptance
-  extends: .integration_test_job_template
-
-build-system-centos:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-centos8-container
-  variables:
-    IMAGE: centos8
-    CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
-                    --enable-modules --enable-trace-backends=dtrace
-    TARGETS: ppc64-softmmu or1k-softmmu s390x-softmmu
-      x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
-    MAKE_CHECK_ARGS: check-build
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
-check-system-centos:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-centos
-      artifacts: true
-  variables:
-    IMAGE: centos8
-    MAKE_CHECK_ARGS: check
-
-acceptance-system-centos:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-centos
-      artifacts: true
-  variables:
-    IMAGE: centos8
-    MAKE_CHECK_ARGS: check-acceptance
-  extends: .integration_test_job_template
-
-build-system-opensuse:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-opensuse-leap-container
-  variables:
-    IMAGE: opensuse-leap
-    CONFIGURE_ARGS: --enable-fdt=system
-    TARGETS: s390x-softmmu x86_64-softmmu aarch64-softmmu
-    MAKE_CHECK_ARGS: check-build
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
-check-system-opensuse:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-opensuse
-      artifacts: true
-  variables:
-    IMAGE: opensuse-leap
-    MAKE_CHECK_ARGS: check
-
-acceptance-system-opensuse:
-  extends: .native_test_job_template
-  needs:
-    - job: build-system-opensuse
-      artifacts: true
-  variables:
-    IMAGE: opensuse-leap
-    MAKE_CHECK_ARGS: check-acceptance
-  extends: .integration_test_job_template
-
-
-build-disabled:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-fedora-container
-  variables:
-    IMAGE: fedora
-    CONFIGURE_ARGS:
-      --disable-attr
-      --disable-auth-pam
-      --disable-avx2
-      --disable-bochs
-      --disable-brlapi
-      --disable-bzip2
-      --disable-cap-ng
-      --disable-capstone
-      --disable-cloop
-      --disable-coroutine-pool
-      --disable-curl
-      --disable-curses
-      --disable-dmg
-      --disable-docs
-      --disable-gcrypt
-      --disable-glusterfs
-      --disable-gnutls
-      --disable-gtk
-      --disable-guest-agent
-      --disable-iconv
-      --disable-keyring
-      --disable-kvm
-      --disable-libiscsi
-      --disable-libpmem
-      --disable-libssh
-      --disable-libudev
-      --disable-libusb
-      --disable-libxml2
-      --disable-linux-aio
-      --disable-live-block-migration
-      --disable-lzo
-      --disable-malloc-trim
-      --disable-mpath
-      --disable-nettle
-      --disable-numa
-      --disable-opengl
-      --disable-parallels
-      --disable-pie
-      --disable-qcow1
-      --disable-qed
-      --disable-qom-cast-debug
-      --disable-rbd
-      --disable-rdma
-      --disable-replication
-      --disable-sdl
-      --disable-seccomp
-      --disable-sheepdog
-      --disable-slirp
-      --disable-smartcard
-      --disable-snappy
-      --disable-sparse
-      --disable-spice
-      --disable-strip
-      --disable-tpm
-      --disable-usb-redir
-      --disable-vdi
-      --disable-vhost-crypto
-      --disable-vhost-net
-      --disable-vhost-scsi
-      --disable-vhost-kernel
-      --disable-vhost-user
-      --disable-vhost-vdpa
-      --disable-vhost-vsock
-      --disable-virglrenderer
-      --disable-vnc
-      --disable-vte
-      --disable-vvfat
-      --disable-xen
-      --disable-zstd
-    TARGETS: arm-softmmu i386-softmmu ppc64-softmmu mips64-softmmu
-      s390x-softmmu i386-linux-user
-    MAKE_CHECK_ARGS: check-qtest SPEED=slow
-
-# This jobs explicitly disable TCG (--disable-tcg), KVM is detected by
-# the configure script. The container doesn't contain Xen headers so
-# Xen accelerator is not detected / selected. As result it build the
-# i386-softmmu and x86_64-softmmu with KVM being the single accelerator
-# available.
-# Also use a different coroutine implementation (which is only really of
-# interest to KVM users, i.e. with TCG disabled)
-build-tcg-disabled:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-centos8-container
-  variables:
-    IMAGE: centos8
-  script:
-    - mkdir build
-    - cd build
-    - ../configure --disable-tcg --audio-drv-list="" --with-coroutine=ucontext
-      || { cat config.log meson-logs/meson-log.txt && exit 1; }
-    - make -j"$JOBS"
-    - make check-unit
-    - make check-qapi-schema
-    - cd tests/qemu-iotests/
-    - ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
-            052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163
-            170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277
-    - ./check -qcow2 028 051 056 057 058 065 068 082 085 091 095 096 102 122
-            124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202
-            208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258
-            260 261 262 263 264 270 272 273 277 279
-
-build-user:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-user-cross-container
-  variables:
-    IMAGE: debian-all-test-cross
-    CONFIGURE_ARGS: --disable-tools --disable-system
-    MAKE_CHECK_ARGS: check-tcg
-
-build-user-static:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-user-cross-container
-  variables:
-    IMAGE: debian-all-test-cross
-    CONFIGURE_ARGS: --disable-tools --disable-system --static
-    MAKE_CHECK_ARGS: check-tcg
-
-# Only build the softmmu targets we have check-tcg tests for
-build-some-softmmu:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-user-cross-container
-  variables:
-    IMAGE: debian-all-test-cross
-    CONFIGURE_ARGS: --disable-tools --enable-debug
-    TARGETS: xtensa-softmmu arm-softmmu aarch64-softmmu alpha-softmmu
-    MAKE_CHECK_ARGS: check-tcg
-
-# Run check-tcg against linux-user (with plugins)
-# we skip sparc64-linux-user until it has been fixed somewhat
-# we skip cris-linux-user as it doesn't use the common run loop
-build-user-plugins:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-user-cross-container
-  variables:
-    IMAGE: debian-all-test-cross
-    CONFIGURE_ARGS: --disable-tools --disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user,cris-linux-user
-    MAKE_CHECK_ARGS: check-tcg
-  timeout: 1h 30m
-
-build-user-centos7:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-centos7-container
-  variables:
-    IMAGE: centos7
-    CONFIGURE_ARGS: --disable-system --disable-tools --disable-docs
-    MAKE_CHECK_ARGS: check-tcg
-
-build-some-softmmu-plugins:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-user-cross-container
-  variables:
-    IMAGE: debian-all-test-cross
-    CONFIGURE_ARGS: --disable-tools --disable-user --enable-plugins --enable-debug-tcg
-    TARGETS: xtensa-softmmu arm-softmmu aarch64-softmmu alpha-softmmu
-    MAKE_CHECK_ARGS: check-tcg
-
-clang-system:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-fedora-container
-  variables:
-    IMAGE: fedora
-    CONFIGURE_ARGS: --cc=clang --cxx=clang++
-      --extra-cflags=-fsanitize=undefined --extra-cflags=-fno-sanitize-recover=undefined
-    TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu
-      ppc-softmmu s390x-softmmu
-    MAKE_CHECK_ARGS: check-qtest check-tcg
-
-clang-user:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-user-cross-container
-  variables:
-    IMAGE: debian-all-test-cross
-    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --disable-system
-      --target-list-exclude=microblazeel-linux-user,aarch64_be-linux-user,i386-linux-user,m68k-linux-user,mipsn32el-linux-user,xtensaeb-linux-user
-      --extra-cflags=-fsanitize=undefined --extra-cflags=-fno-sanitize-recover=undefined
-    MAKE_CHECK_ARGS: check-unit check-tcg
-
-# Set LD_JOBS=1 because this requires LTO and ld consumes a large amount of memory.
-# On gitlab runners, default value sometimes end up calling 2 lds concurrently and
-# triggers an Out-Of-Memory error
-#
-# Since slirp callbacks are used in QEMU Timers, slirp needs to be compiled together
-# with QEMU and linked as a static library to avoid false positives in CFI checks.
-# This can be accomplished by using -enable-slirp=git, which avoids the use of
-# a system-wide version of the library
-#
-# Split in three sets of build/check/acceptance to limit the execution time of each
-# job
-build-cfi-aarch64:
-  extends: .native_build_job_template
-  needs:
-  - job: amd64-fedora-container
-  variables:
-    LD_JOBS: 1
-    AR: llvm-ar
-    IMAGE: fedora
-    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-cfi --enable-cfi-debug
-      --enable-safe-stack --enable-slirp=git
-    TARGETS: aarch64-softmmu
-    MAKE_CHECK_ARGS: check-build
-  timeout: 70m
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
-check-cfi-aarch64:
-  extends: .native_test_job_template
-  needs:
-    - job: build-cfi-aarch64
-      artifacts: true
-  variables:
-    IMAGE: fedora
-    MAKE_CHECK_ARGS: check
-
-acceptance-cfi-aarch64:
-  extends: .native_test_job_template
-  needs:
-    - job: build-cfi-aarch64
-      artifacts: true
-  variables:
-    IMAGE: fedora
-    MAKE_CHECK_ARGS: check-acceptance
-  extends: .integration_test_job_template
-
-build-cfi-ppc64-s390x:
-  extends: .native_build_job_template
-  needs:
-  - job: amd64-fedora-container
-  variables:
-    LD_JOBS: 1
-    AR: llvm-ar
-    IMAGE: fedora
-    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-cfi --enable-cfi-debug
-      --enable-safe-stack --enable-slirp=git
-    TARGETS: ppc64-softmmu s390x-softmmu
-    MAKE_CHECK_ARGS: check-build
-  timeout: 70m
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
-check-cfi-ppc64-s390x:
-  extends: .native_test_job_template
-  needs:
-    - job: build-cfi-ppc64-s390x
-      artifacts: true
-  variables:
-    IMAGE: fedora
-    MAKE_CHECK_ARGS: check
-
-acceptance-cfi-ppc64-s390x:
-  extends: .native_test_job_template
-  needs:
-    - job: build-cfi-ppc64-s390x
-      artifacts: true
-  variables:
-    IMAGE: fedora
-    MAKE_CHECK_ARGS: check-acceptance
-  extends: .integration_test_job_template
-
-build-cfi-x86_64:
-  extends: .native_build_job_template
-  needs:
-  - job: amd64-fedora-container
-  variables:
-    LD_JOBS: 1
-    AR: llvm-ar
-    IMAGE: fedora
-    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-cfi --enable-cfi-debug
-      --enable-safe-stack --enable-slirp=git
-    TARGETS: x86_64-softmmu
-    MAKE_CHECK_ARGS: check-build
-  timeout: 70m
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
-check-cfi-x86_64:
-  extends: .native_test_job_template
-  needs:
-    - job: build-cfi-x86_64
-      artifacts: true
-  variables:
-    IMAGE: fedora
-    MAKE_CHECK_ARGS: check
-
-acceptance-cfi-x86_64:
-  extends: .native_test_job_template
-  needs:
-    - job: build-cfi-x86_64
-      artifacts: true
-  variables:
-    IMAGE: fedora
-    MAKE_CHECK_ARGS: check-acceptance
-  extends: .integration_test_job_template
-
-tsan-build:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-ubuntu2004-container
-  variables:
-    IMAGE: ubuntu2004
-    CONFIGURE_ARGS: --enable-tsan --cc=clang-10 --cxx=clang++-10
-          --enable-trace-backends=ust --enable-fdt=system --enable-slirp=system
-    TARGETS: x86_64-softmmu ppc64-softmmu riscv64-softmmu x86_64-linux-user
-    MAKE_CHECK_ARGS: bench V=1
-
-# These targets are on the way out
-build-deprecated:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-user-cross-container
-  variables:
-    IMAGE: debian-all-test-cross
-    CONFIGURE_ARGS: --disable-tools
-    MAKE_CHECK_ARGS: build-tcg
-    TARGETS: ppc64abi32-linux-user lm32-softmmu unicore32-softmmu
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
-# We split the check-tcg step as test failures are expected but we still
-# want to catch the build breaking.
-check-deprecated:
-  extends: .native_test_job_template
-  needs:
-    - job: build-deprecated
-      artifacts: true
-  variables:
-    IMAGE: debian-all-test-cross
-    MAKE_CHECK_ARGS: check-tcg
-  allow_failure: true
-
-# gprof/gcov are GCC features
-gprof-gcov:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-ubuntu2004-container
-  variables:
-    IMAGE: ubuntu2004
-    CONFIGURE_ARGS: --enable-gprof --enable-gcov
-    MAKE_CHECK_ARGS: check
-    TARGETS: aarch64-softmmu ppc64-softmmu s390x-softmmu x86_64-softmmu
-  timeout: 70m
-  after_script:
-    - ${CI_PROJECT_DIR}/scripts/ci/coverage-summary.sh
-
-build-oss-fuzz:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-fedora-container
-  variables:
-    IMAGE: fedora
-  script:
-    - mkdir build-oss-fuzz
-    - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
-      ./scripts/oss-fuzz/build.sh
-    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
-    - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f
-                      | grep -v slirp); do
-        grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
-        echo Testing ${fuzzer} ... ;
-        "${fuzzer}" -runs=1 -seed=1 || exit 1 ;
-      done
-    # Unrelated to fuzzer: run some tests with -fsanitize=address
-    - cd build-oss-fuzz && make check-qtest-i386 check-unit
-
-build-tci:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-user-cross-container
-  variables:
-    IMAGE: debian-all-test-cross
-  script:
-    - TARGETS="aarch64 alpha arm hppa m68k microblaze moxie ppc64 s390x x86_64"
-    - mkdir build
-    - cd build
-    - ../configure --enable-tcg-interpreter
-        --target-list="$(for tg in $TARGETS; do echo -n ${tg}'-softmmu '; done)" || { cat config.log meson-logs/meson-log.txt && exit 1; }
-    - make -j"$JOBS"
-    - make tests/qtest/boot-serial-test tests/qtest/cdrom-test tests/qtest/pxe-test
-    - for tg in $TARGETS ; do
-        export QTEST_QEMU_BINARY="./qemu-system-${tg}" ;
-        ./tests/qtest/boot-serial-test || exit 1 ;
-        ./tests/qtest/cdrom-test || exit 1 ;
-      done
-    - QTEST_QEMU_BINARY="./qemu-system-x86_64" ./tests/qtest/pxe-test
-    - QTEST_QEMU_BINARY="./qemu-system-s390x" ./tests/qtest/pxe-test -m slow
-    - make check-tcg
-
-# Alternate coroutines implementations are only really of interest to KVM users
-# However we can't test against KVM on Gitlab-CI so we can only run unit tests
-build-coroutine-sigaltstack:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-ubuntu2004-container
-  variables:
-    IMAGE: ubuntu2004
-    CONFIGURE_ARGS: --with-coroutine=sigaltstack --disable-tcg
-                    --enable-trace-backends=ftrace
-    MAKE_CHECK_ARGS: check-unit
-
-# Most jobs test latest gcrypt or nettle builds
-#
-# These jobs test old gcrypt and nettle from RHEL7
-# which had some API differences.
-crypto-old-nettle:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-centos7-container
-  variables:
-    IMAGE: centos7
-    TARGETS: x86_64-softmmu x86_64-linux-user
-    CONFIGURE_ARGS: --disable-gcrypt --enable-nettle
-    MAKE_CHECK_ARGS: check
-
-crypto-old-gcrypt:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-centos7-container
-  variables:
-    IMAGE: centos7
-    TARGETS: x86_64-softmmu x86_64-linux-user
-    CONFIGURE_ARGS: --disable-nettle --enable-gcrypt
-    MAKE_CHECK_ARGS: check
-
-crypto-only-gnutls:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-centos7-container
-  variables:
-    IMAGE: centos7
-    TARGETS: x86_64-softmmu x86_64-linux-user
-    CONFIGURE_ARGS: --disable-nettle --disable-gcrypt --enable-gnutls
-    MAKE_CHECK_ARGS: check
-
-
-# Check our reduced build configurations
-build-without-default-devices:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-centos8-container
-  variables:
-    IMAGE: centos8
-    CONFIGURE_ARGS: --without-default-devices --disable-user
-
-build-without-default-features:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-container
-  variables:
-    IMAGE: debian-amd64
-    CONFIGURE_ARGS: --without-default-features --disable-user
-        --target-list-exclude=arm-softmmu,i386-softmmu,mipsel-softmmu,mips64-softmmu,ppc-softmmu
-    MAKE_CHECK_ARGS: check-unit
-
-check-patch:
-  stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
-  needs:
-    job: amd64-centos8-container
-  script: .gitlab-ci.d/check-patch.py
-  except:
-    variables:
-      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
-  variables:
-    GIT_DEPTH: 1000
-  allow_failure: true
-
-check-dco:
-  stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
-  needs:
-    job: amd64-centos8-container
-  script: .gitlab-ci.d/check-dco.py
-  except:
-    variables:
-      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
-  variables:
-    GIT_DEPTH: 1000
-
-build-libvhost-user:
-  stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
-  needs:
-    job: amd64-fedora-container
-  before_script:
-    - dnf install -y meson ninja-build
-  script:
-    - mkdir subprojects/libvhost-user/build
-    - cd subprojects/libvhost-user/build
-    - meson
-    - ninja
-
-# No targets are built here, just tools, docs, and unit tests. This
-# also feeds into the eventual documentation deployment steps later
-build-tools-and-docs-debian:
-  extends: .native_build_job_template
-  needs:
-    job: amd64-debian-container
-  variables:
-    IMAGE: debian-amd64
-    MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
-    CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
-  artifacts:
-    expire_in: 2 days
-    paths:
-      - build
-
-# Prepare for GitLab pages deployment. Anything copied into the
-# "public" directory will be deployed to $USER.gitlab.io/$PROJECT
-pages:
-  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
-  stage: test
-  needs:
-    - job: build-tools-and-docs-debian
-  script:
-    - mkdir -p public
-    # HTML-ised source tree
-    - make gtags
-    - htags -anT --tree-view=filetree -m qemu_init
-        -t "Welcome to the QEMU sourcecode"
-    - mv HTML public/src
-    # Project documentation
-    - make -C build install DESTDIR=$(pwd)/temp-install
-    - mv temp-install/usr/local/share/doc/qemu/* public/
-  artifacts:
-    paths:
-      - public
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 11/15] gitlab-ci: Extract core container jobs to container-core.yml
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (9 preceding siblings ...)
  2021-04-18 23:34 ` [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-19  5:42   ` Thomas Huth
  2021-04-18 23:34 ` [PATCH 12/15] gitlab-ci: Move current job set to qemu-project.yml Philippe Mathieu-Daudé
                   ` (3 subsequent siblings)
  14 siblings, 1 reply; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

It is not possible to use the previously extracted templates
without this set of core containers. Extract them into a new
file (container-core.yml) to be able to build them without
having to build all the other containers by default.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/container-core.yml | 17 +++++++++++++++++
 .gitlab-ci.d/containers.yml     | 16 +---------------
 2 files changed, 18 insertions(+), 15 deletions(-)
 create mode 100644 .gitlab-ci.d/container-core.yml

diff --git a/.gitlab-ci.d/container-core.yml b/.gitlab-ci.d/container-core.yml
new file mode 100644
index 00000000000..e8dd1f476a2
--- /dev/null
+++ b/.gitlab-ci.d/container-core.yml
@@ -0,0 +1,17 @@
+include:
+  - local: '/.gitlab-ci.d/container-template.yml'
+
+amd64-centos8-container:
+  extends: .container_job_template
+  variables:
+    NAME: centos8
+
+amd64-fedora-container:
+  extends: .container_job_template
+  variables:
+    NAME: fedora
+
+amd64-debian10-container:
+  extends: .container_job_template
+  variables:
+    NAME: debian10
diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 0bf4899df10..c371f8587e1 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -1,5 +1,6 @@
 include:
   - local: '/.gitlab-ci.d/container-template.yml'
+  - local: '/.gitlab-ci.d/container-core.yml'
 
 amd64-alpine-container:
   extends: .container_job_template
@@ -11,16 +12,6 @@ amd64-centos7-container:
   variables:
     NAME: centos7
 
-amd64-centos8-container:
-  extends: .container_job_template
-  variables:
-    NAME: centos8
-
-amd64-debian10-container:
-  extends: .container_job_template
-  variables:
-    NAME: debian10
-
 amd64-debian11-container:
   extends: .container_job_template
   variables:
@@ -190,11 +181,6 @@ cris-fedora-cross-container:
   variables:
     NAME: fedora-cris-cross
 
-amd64-fedora-container:
-  extends: .container_job_template
-  variables:
-    NAME: fedora
-
 i386-fedora-cross-container:
   extends: .container_job_template
   variables:
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [PATCH 12/15] gitlab-ci: Move current job set to qemu-project.yml
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (10 preceding siblings ...)
  2021-04-18 23:34 ` [PATCH 11/15] gitlab-ci: Extract core container jobs to container-core.yml Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [RFC PATCH 13/15] gitlab-ci: Switch to dynamically generated pipelines Philippe Mathieu-Daudé
                   ` (2 subsequent siblings)
  14 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

To allow forks to easily decide which jobs they want to run,
but without disrupting the current default, move the current
set of jobs to a new file corresponding to the jobs run by
the mainstream project CI:
https://gitlab.com/qemu-project/qemu/-/pipelines

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/qemu-project.yml | 8 ++++++++
 .gitlab-ci.yml                | 8 +-------
 2 files changed, 9 insertions(+), 7 deletions(-)
 create mode 100644 .gitlab-ci.d/qemu-project.yml

diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
new file mode 100644
index 00000000000..8f13a584a29
--- /dev/null
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -0,0 +1,8 @@
+include:
+  - local: '/.gitlab-ci.d/stages.yml'
+  - local: '/.gitlab-ci.d/edk2.yml'
+  - local: '/.gitlab-ci.d/opensbi.yml'
+  - local: '/.gitlab-ci.d/containers.yml'
+  - local: '/.gitlab-ci.d/crossbuilds.yml'
+  - local: '/.gitlab-ci.d/buildtest.yml'
+  - local: '/.gitlab-ci.d/checks.yml'
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8f13a584a29..533a7e61339 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,8 +1,2 @@
 include:
-  - local: '/.gitlab-ci.d/stages.yml'
-  - local: '/.gitlab-ci.d/edk2.yml'
-  - local: '/.gitlab-ci.d/opensbi.yml'
-  - local: '/.gitlab-ci.d/containers.yml'
-  - local: '/.gitlab-ci.d/crossbuilds.yml'
-  - local: '/.gitlab-ci.d/buildtest.yml'
-  - local: '/.gitlab-ci.d/checks.yml'
+  - local: '/.gitlab-ci.d/qemu-project.yml'
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [RFC PATCH 13/15] gitlab-ci: Switch to dynamically generated pipelines
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (11 preceding siblings ...)
  2021-04-18 23:34 ` [PATCH 12/15] gitlab-ci: Move current job set to qemu-project.yml Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs Philippe Mathieu-Daudé
  2021-04-18 23:34 ` [NOTFORMERGE PATCH 15/15] gitlab-ci: Use my own set of jobs for CI pipeline Philippe Mathieu-Daudé
  14 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

Switch to the dynamically generated pipeline scheme described in:
https://docs.gitlab.com/ee/ci/parent_child_pipelines.html#dynamic-child-pipelines

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.yml | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 533a7e61339..718c8e004be 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,2 +1,21 @@
-include:
-  - local: '/.gitlab-ci.d/qemu-project.yml'
+# Dynamic child pipelines
+# https://docs.gitlab.com/ee/ci/parent_child_pipelines.html#dynamic-child-pipelines
+
+generate-config:
+  stage: build
+  variables:
+    GIT_SUBMODULE_STRATEGY: none
+  artifacts:
+    paths:
+      - generated-config.yml
+  script:
+    - cp .gitlab-ci.d/qemu-project.yml generated-config.yml
+
+generate-pipeline:
+  stage: test
+  variables:
+    GIT_SUBMODULE_STRATEGY: none
+  trigger:
+    include:
+      - artifact: generated-config.yml
+        job: generate-config
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (12 preceding siblings ...)
  2021-04-18 23:34 ` [RFC PATCH 13/15] gitlab-ci: Switch to dynamically generated pipelines Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  2021-04-19  5:48   ` Thomas Huth
                     ` (2 more replies)
  2021-04-18 23:34 ` [NOTFORMERGE PATCH 15/15] gitlab-ci: Use my own set of jobs for CI pipeline Philippe Mathieu-Daudé
  14 siblings, 3 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

Forks run the same jobs than mainstream, which might be overkill.
Allow them to easily rebase their custom set, while keeping using
the mainstream templates, and ability to pick specific jobs from
the mainstream set.

To switch to your set, simply add your .gitlab-ci.yml as
.gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
is your gitlab 'namespace', usually username). This file will be
used instead of the default mainstream set.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.yml | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 718c8e004be..35fd35075db 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -9,7 +9,12 @@ generate-config:
     paths:
       - generated-config.yml
   script:
-    - cp .gitlab-ci.d/qemu-project.yml generated-config.yml
+    - if test -e .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml ;
+      then
+        cp .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml generated-config.yml ;
+      else
+        cp .gitlab-ci.d/qemu-project.yml generated-config.yml ;
+      fi
 
 generate-pipeline:
   stage: test
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* [NOTFORMERGE PATCH 15/15] gitlab-ci: Use my own set of jobs for CI pipeline
  2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
                   ` (13 preceding siblings ...)
  2021-04-18 23:34 ` [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs Philippe Mathieu-Daudé
@ 2021-04-18 23:34 ` Philippe Mathieu-Daudé
  14 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-18 23:34 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

This is an example to use a different set of jobs than the
mainstream one (currently 116 jobs).

Here I only select 2 jobs:
- cross-s390x-kvm-only
- build-libvhost-user

Including their dependencies, I have to run 8 jobs.

Result (build time 15 minutes and 25 seconds):
  https://gitlab.com/philmd/qemu/-/pipelines/284362446

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/philmd.yml | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100644 .gitlab-ci.d/philmd.yml

diff --git a/.gitlab-ci.d/philmd.yml b/.gitlab-ci.d/philmd.yml
new file mode 100644
index 00000000000..402228e6ed1
--- /dev/null
+++ b/.gitlab-ci.d/philmd.yml
@@ -0,0 +1,33 @@
+include:
+  - local: '/.gitlab-ci.d/stages.yml'
+  - local: '/.gitlab-ci.d/container-core.yml'
+  - local: '/.gitlab-ci.d/crossbuild-template.yml'
+  - local: '/.gitlab-ci.d/checks.yml'
+
+build-libvhost-user:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
+  needs:
+    job: amd64-fedora-container
+  before_script:
+    - dnf install -y meson ninja-build
+  script:
+    - mkdir subprojects/libvhost-user/build
+    - cd subprojects/libvhost-user/build
+    - meson
+    - ninja
+
+s390x-debian-cross-container:
+  extends: .container_job_template
+  stage: containers-layer2
+  needs: ['amd64-debian10-container']
+  variables:
+    NAME: debian-s390x-cross
+
+cross-s390x-kvm-only:
+  extends: .cross_accel_build_job
+  needs:
+    job: s390x-debian-cross-container
+  variables:
+    IMAGE: debian-s390x-cross
+    ACCEL_CONFIGURE_OPTS: --disable-tcg
-- 
2.26.3



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [PATCH 05/15] gitlab-ci: Rename acceptance_test_job -> integration_test_job
  2021-04-18 23:34 ` [PATCH 05/15] gitlab-ci: Rename acceptance_test_job -> integration_test_job Philippe Mathieu-Daudé
@ 2021-04-19  5:19   ` Thomas Huth
  2021-04-23 17:18     ` Willian Rampazzo
  0 siblings, 1 reply; 47+ messages in thread
From: Thomas Huth @ 2021-04-19  5:19 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

On 19/04/2021 01.34, Philippe Mathieu-Daudé wrote:
> From: Philippe Mathieu-Daudé <philmd@redhat.com>
> 
> The acceptance tests aren't accepting anything,
> rename them as integration tests.
> 
> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   .gitlab-ci.yml | 20 ++++++++++----------
>   1 file changed, 10 insertions(+), 10 deletions(-)

Acked-by: Thomas Huth <thuth@redhat.com>

FWIW, I'm in favour of renaming the tests/acceptance directory to 
tests/integration , too.



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [PATCH 08/15] gitlab-ci: Extract DCO/style check jobs to checks.yml
  2021-04-18 23:34 ` [PATCH 08/15] gitlab-ci: Extract DCO/style check jobs to checks.yml Philippe Mathieu-Daudé
@ 2021-04-19  5:26   ` Thomas Huth
  2021-04-19 13:44     ` Wainer dos Santos Moschetta
  0 siblings, 1 reply; 47+ messages in thread
From: Thomas Huth @ 2021-04-19  5:26 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Daniel P . Berrangé,
	Erik Skultety, Wainer dos Santos Moschetta, Willian Rampazzo,
	Cleber Rosa, Miroslav Rezanina, Alex Bennée

On 19/04/2021 01.34, Philippe Mathieu-Daudé wrote:
> Extract the DCO / checkpatch jobs to a new file (checks.yml)
> to be able to run them without having to run all the jobs
> included in the default .gitlab-ci.yml, which are mainly useful
> for mainstream CI.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/checks.yml | 24 ++++++++++++++++++++++++
>   .gitlab-ci.yml          |  1 +
>   2 files changed, 25 insertions(+)
>   create mode 100644 .gitlab-ci.d/checks.yml
> 
> diff --git a/.gitlab-ci.d/checks.yml b/.gitlab-ci.d/checks.yml
> new file mode 100644
> index 00000000000..f695627b7cd
> --- /dev/null
> +++ b/.gitlab-ci.d/checks.yml
> @@ -0,0 +1,24 @@
> +check-patch:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> +  needs:
> +    job: amd64-centos8-container
> +  script: .gitlab-ci.d/check-patch.py
> +  except:
> +    variables:
> +      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
> +  variables:
> +    GIT_DEPTH: 1000
> +  allow_failure: true
> +
> +check-dco:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> +  needs:
> +    job: amd64-centos8-container
> +  script: .gitlab-ci.d/check-dco.py
> +  except:
> +    variables:
> +      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
> +  variables:
> +    GIT_DEPTH: 1000
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index bf4862a78de..ba35b3917e7 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -12,6 +12,7 @@ include:
>     - local: '/.gitlab-ci.d/opensbi.yml'
>     - local: '/.gitlab-ci.d/containers.yml'
>     - local: '/.gitlab-ci.d/crossbuilds.yml'
> +  - local: '/.gitlab-ci.d/checks.yml'
>   
>   .native_build_job_template:
>     stage: build
> 

You added the new file, but it looks like you did not remove the jobs from 
the main .gitlab-ci.yml file ? ... that looks like a mistake?

Also, "checks.yml" sounds a little bit too generic. Maybe rather name it 
"style-check.yml" or something similar?

  Thomas



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml
  2021-04-18 23:34 ` [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml Philippe Mathieu-Daudé
@ 2021-04-19  5:39   ` Thomas Huth
  2021-04-19 15:11   ` Alex Bennée
  1 sibling, 0 replies; 47+ messages in thread
From: Thomas Huth @ 2021-04-19  5:39 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Daniel P . Berrangé,
	Erik Skultety, Wainer dos Santos Moschetta, Willian Rampazzo,
	Cleber Rosa, Miroslav Rezanina, Alex Bennée

On 19/04/2021 01.34, Philippe Mathieu-Daudé wrote:
> Extract the build/test jobs run by default on the mainstream
> CI into a new file (buildtest.yml) and their templates into
> (buildtest-template.yml), to be able to reuse the templates
> without having to run all these mainstream jobs by default.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml |  80 +++
>   .gitlab-ci.d/buildtest.yml          | 744 +++++++++++++++++++++++++
>   .gitlab-ci.yml                      | 824 +---------------------------
>   3 files changed, 825 insertions(+), 823 deletions(-)
>   create mode 100644 .gitlab-ci.d/buildtest-template.yml
>   create mode 100644 .gitlab-ci.d/buildtest.yml

I like the idea, the main .gitlab-ci.yml was way to crowded already anyway.

Not sure if "buildtest.yml" is such a good name (since it's not only about 
building, but also about testing), but I fail to come up with a better 
suggestion...

Just a suggestion for discussion, not sure whether it's such a good idea 
yet: Maybe we could split it up even further into "system.yml" (with the 
various jobs that only test the "softmmu" binaries), "user.yml" (with the 
--disable-system jobs) and "misc.yml" (with mixed jobs) ?

> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> new file mode 100644
> index 00000000000..b24a153d904
> --- /dev/null
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -0,0 +1,80 @@
> +.native_build_job_template:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  before_script:
> +    - JOBS=$(expr $(nproc) + 1)
> +  script:
> +    - mkdir build
> +    - cd build
> +    - if test -n "$TARGETS";
> +      then
> +        ../configure --enable-werror --disable-docs $CONFIGURE_ARGS --target-list="$TARGETS" ;
> +      else
> +        ../configure --enable-werror --disable-docs $CONFIGURE_ARGS ;
> +      fi || { cat config.log meson-logs/meson-log.txt && exit 1; }
> +    - if test -n "$LD_JOBS";
> +      then
> +        meson configure . -Dbackend_max_links="$LD_JOBS" ;
> +      fi || exit 1;
> +    - make -j"$JOBS"
> +    - if test -n "$MAKE_CHECK_ARGS";
> +      then
> +        make -j"$JOBS" $MAKE_CHECK_ARGS ;
> +      fi
> +
> +.native_test_job_template:
> +  stage: test
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  script:
> +    - scripts/git-submodule.sh update
> +        $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
> +    - cd build
> +    - find . -type f -exec touch {} +
> +    # Avoid recompiling by hiding ninja with NINJA=":"
> +    - make NINJA=":" $MAKE_CHECK_ARGS
> +
> +.integration_test_job_template:
> +  cache:
> +    key: "${CI_JOB_NAME}-cache"
> +    paths:
> +      - ${CI_PROJECT_DIR}/avocado-cache
> +    policy: pull-push
> +  artifacts:
> +    name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
> +    when: always
> +    expire_in: 2 days
> +    paths:
> +      - build/tests/results/latest/results.xml
> +      - build/tests/results/latest/test-results
> +    reports:
> +      junit: build/tests/results/latest/results.xml
> +  before_script:
> +    - mkdir -p ~/.config/avocado
> +    - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
> +    - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
> +           >> ~/.config/avocado/avocado.conf
> +    - echo -e '[job.output.testlogs]\nstatuses = ["FAIL", "INTERRUPT"]'
> +           >> ~/.config/avocado/avocado.conf
> +    - if [ -d ${CI_PROJECT_DIR}/avocado-cache ]; then
> +        du -chs ${CI_PROJECT_DIR}/avocado-cache ;
> +      fi
> +    - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
> +  after_script:
> +    - cd build
> +    - du -chs ${CI_PROJECT_DIR}/avocado-cache
> +
> +build-system-alpine:
> +  extends: .native_build_job_template
> +  needs:
> +    - job: amd64-alpine-container
> +  variables:
> +    IMAGE: alpine
> +    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
> +      moxie-softmmu microblazeel-softmmu mips64el-softmmu
> +    MAKE_CHECK_ARGS: check-build
> +    CONFIGURE_ARGS: --enable-docs --enable-trace-backends=log,simple,syslog
> +  artifacts:
> +    expire_in: 2 days
> +    paths:
> +      - .git-submodule-status
> +      - build

build-system-alpine should certainly not go into the template file, but into 
the file below instead.

> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> new file mode 100644
> index 00000000000..c24553eab05
> --- /dev/null
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -0,0 +1,744 @@
[...]
> +check-patch:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> +  needs:
> +    job: amd64-centos8-container
> +  script: .gitlab-ci.d/check-patch.py
> +  except:
> +    variables:
> +      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
> +  variables:
> +    GIT_DEPTH: 1000
> +  allow_failure: true
> +
> +check-dco:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
> +  needs:
> +    job: amd64-centos8-container
> +  script: .gitlab-ci.d/check-dco.py
> +  except:
> +    variables:
> +      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH == 'master'
> +  variables:
> +    GIT_DEPTH: 1000

These two should not be here anymore since they have been handled in patch 
08/15 already.

 > +# No targets are built here, just tools, docs, and unit tests. This
 > +# also feeds into the eventual documentation deployment steps later
 > +build-tools-and-docs-debian:
 > +  extends: .native_build_job_template
 > +  needs:
 > +    job: amd64-debian-container
 > +  variables:
 > +    IMAGE: debian-amd64
 > +    MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
 > +    CONFIGURE_ARGS: --disable-system --disable-user --enable-docs 
--enable-tools
 > +  artifacts:
 > +    expire_in: 2 days
 > +    paths:
 > +      - build
 > +
 > +# Prepare for GitLab pages deployment. Anything copied into the
 > +# "public" directory will be deployed to $USER.gitlab.io/$PROJECT
 > +pages:
 > +  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
 > +  stage: test
 > +  needs:
 > +    - job: build-tools-and-docs-debian
 > +  script:
 > +    - mkdir -p public
 > +    # HTML-ised source tree
 > +    - make gtags
 > +    - htags -anT --tree-view=filetree -m qemu_init
 > +        -t "Welcome to the QEMU sourcecode"
 > +    - mv HTML public/src
 > +    # Project documentation
 > +    - make -C build install DESTDIR=$(pwd)/temp-install
 > +    - mv temp-install/usr/local/share/doc/qemu/* public/
 > +  artifacts:
 > +    paths:
 > +      - public

I think the "pages" job should maybe rather go into a separate file instead 
(maybe together with the build-tools-and-docs-debian job)?

  Thomas



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [PATCH 11/15] gitlab-ci: Extract core container jobs to container-core.yml
  2021-04-18 23:34 ` [PATCH 11/15] gitlab-ci: Extract core container jobs to container-core.yml Philippe Mathieu-Daudé
@ 2021-04-19  5:42   ` Thomas Huth
  0 siblings, 0 replies; 47+ messages in thread
From: Thomas Huth @ 2021-04-19  5:42 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Daniel P . Berrangé,
	Erik Skultety, Wainer dos Santos Moschetta, Willian Rampazzo,
	Cleber Rosa, Miroslav Rezanina, Alex Bennée

On 19/04/2021 01.34, Philippe Mathieu-Daudé wrote:
> It is not possible to use the previously extracted templates
> without this set of core containers. Extract them into a new
> file (container-core.yml) to be able to build them without
> having to build all the other containers by default.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/container-core.yml | 17 +++++++++++++++++
>   .gitlab-ci.d/containers.yml     | 16 +---------------
>   2 files changed, 18 insertions(+), 15 deletions(-)
>   create mode 100644 .gitlab-ci.d/container-core.yml

Reviewed-by: Thomas Huth <thuth@redhat.com>



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-18 23:34 ` [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs Philippe Mathieu-Daudé
@ 2021-04-19  5:48   ` Thomas Huth
  2021-04-19  9:40   ` Daniel P. Berrangé
  2021-04-19 15:57   ` Alex Bennée
  2 siblings, 0 replies; 47+ messages in thread
From: Thomas Huth @ 2021-04-19  5:48 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Daniel P . Berrangé,
	Erik Skultety, Wainer dos Santos Moschetta, Willian Rampazzo,
	Cleber Rosa, Miroslav Rezanina, Alex Bennée

On 19/04/2021 01.34, Philippe Mathieu-Daudé wrote:
> Forks run the same jobs than mainstream, which might be overkill.
> Allow them to easily rebase their custom set, while keeping using
> the mainstream templates, and ability to pick specific jobs from
> the mainstream set.
> 
> To switch to your set, simply add your .gitlab-ci.yml as
> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
> is your gitlab 'namespace', usually username). This file will be
> used instead of the default mainstream set.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.yml | 7 ++++++-
>   1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 718c8e004be..35fd35075db 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -9,7 +9,12 @@ generate-config:
>       paths:
>         - generated-config.yml
>     script:
> -    - cp .gitlab-ci.d/qemu-project.yml generated-config.yml
> +    - if test -e .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml ;
> +      then
> +        cp .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml generated-config.yml ;
> +      else
> +        cp .gitlab-ci.d/qemu-project.yml generated-config.yml ;
> +      fi

I think you could merge this with the previous patch, since the previous 
patch is not very useful on its own.

Anyway, I like the idea, that could be useful for downstream, indeed!

  Thomas



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-18 23:34 ` [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs Philippe Mathieu-Daudé
  2021-04-19  5:48   ` Thomas Huth
@ 2021-04-19  9:40   ` Daniel P. Berrangé
  2021-04-19 10:09     ` Philippe Mathieu-Daudé
  2021-04-19 10:10     ` Erik Skultety
  2021-04-19 15:57   ` Alex Bennée
  2 siblings, 2 replies; 47+ messages in thread
From: Daniel P. Berrangé @ 2021-04-19  9:40 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Erik Skultety, qemu-devel,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé wrote:
> Forks run the same jobs than mainstream, which might be overkill.
> Allow them to easily rebase their custom set, while keeping using
> the mainstream templates, and ability to pick specific jobs from
> the mainstream set.
> 
> To switch to your set, simply add your .gitlab-ci.yml as
> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
> is your gitlab 'namespace', usually username). This file will be
> used instead of the default mainstream set.

I find this approach undesirable, because AFAICT, it means you have
to commit this extra file to any of your downstream branches that
you want this to be used for.  Then you have to be either delete it
again before sending patches upstream, or tell git-publish to
exclude the commit that adds this.

IMHO any per-contributor overhead needs to not involve committing
stuff to their git branches, that isn't intended to go upstream.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19  9:40   ` Daniel P. Berrangé
@ 2021-04-19 10:09     ` Philippe Mathieu-Daudé
  2021-04-19 10:10     ` Erik Skultety
  1 sibling, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-19 10:09 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Thomas Huth, Erik Skultety, qemu-devel,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

On 4/19/21 11:40 AM, Daniel P. Berrangé wrote:
> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé wrote:
>> Forks run the same jobs than mainstream, which might be overkill.
>> Allow them to easily rebase their custom set, while keeping using
>> the mainstream templates, and ability to pick specific jobs from
>> the mainstream set.
>>
>> To switch to your set, simply add your .gitlab-ci.yml as
>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
>> is your gitlab 'namespace', usually username). This file will be
>> used instead of the default mainstream set.
> 
> I find this approach undesirable, because AFAICT, it means you have
> to commit this extra file to any of your downstream branches that
> you want this to be used for.  Then you have to be either delete it
> again before sending patches upstream, or tell git-publish to
> exclude the commit that adds this.

Good point. What I'm looking for is allow fork to keep following the
mainstream development.

> IMHO any per-contributor overhead needs to not involve committing
> stuff to their git branches, that isn't intended to go upstream.

But why am I forced to run the upstream overhead stuff into my fork?
I find it counter-productive for my limited set of topic I'm modifying.
Also, why should I wait >2h for a pipeline when I exactly know which
area I'm modifying? This is a waste of time and resources.

Gitlab suggested an alternative 3 months ago, it is still fresh:
https://docs.gitlab.com/ee/ci/yaml/README.html#variables-with-include
combined with
https://docs.gitlab.com/ee/ci/yaml/README.html#includeremote
and
https://docs.gitlab.com/ee/ci/yaml/README.html#multiple-files-from-a-project
we could have forks include their gitlab-ci.yml from a specific branch
of their repository.

Example, if I push a branch named project-specific-ci, and we add
that to mainstream:

  include:
  - project: '$CI_PROJECT_PATH'
    ref: project-specific-ci
    file:
      - '/.gitlab-ci.d/project-specific.yml'

The it would include
project-specific-ci:/.gitlab-ci.d/project-specific.yml in all
branches/tags I push.

In that case we could rename qemu-project.yml -> project-specific.yml
(patch 12).

The problem is I couldn't have it optionally working (when there is
no 'project-specific-ci' branch).

Still room for investigation...

Thanks for the feedback,

Phil.


^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19  9:40   ` Daniel P. Berrangé
  2021-04-19 10:09     ` Philippe Mathieu-Daudé
@ 2021-04-19 10:10     ` Erik Skultety
  2021-04-19 10:20       ` Thomas Huth
  2021-04-19 10:44       ` Philippe Mathieu-Daudé
  1 sibling, 2 replies; 47+ messages in thread
From: Erik Skultety @ 2021-04-19 10:10 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Thomas Huth, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, qemu-devel, Willian Rampazzo,
	Cleber Rosa, Miroslav Rezanina, Alex Bennée

On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé wrote:
> > Forks run the same jobs than mainstream, which might be overkill.
> > Allow them to easily rebase their custom set, while keeping using
> > the mainstream templates, and ability to pick specific jobs from
> > the mainstream set.
> > 
> > To switch to your set, simply add your .gitlab-ci.yml as
> > .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
> > is your gitlab 'namespace', usually username). This file will be
> > used instead of the default mainstream set.
> 
> I find this approach undesirable, because AFAICT, it means you have
> to commit this extra file to any of your downstream branches that
> you want this to be used for.  Then you have to be either delete it
> again before sending patches upstream, or tell git-publish to
> exclude the commit that adds this.
> 
> IMHO any per-contributor overhead needs to not involve committing
> stuff to their git branches, that isn't intended to go upstream.

Not just that, ideally, they should also run all the upstream workloads before
submitting a PR or posting patches because they'd have to respin because of a
potential failure in upstream pipelines anyway.

Erik



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 10:10     ` Erik Skultety
@ 2021-04-19 10:20       ` Thomas Huth
  2021-04-19 10:36         ` Daniel P. Berrangé
  2021-04-19 10:47         ` Daniel P. Berrangé
  2021-04-19 10:44       ` Philippe Mathieu-Daudé
  1 sibling, 2 replies; 47+ messages in thread
From: Thomas Huth @ 2021-04-19 10:20 UTC (permalink / raw)
  To: Erik Skultety, Daniel P. Berrangé
  Cc: Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, qemu-devel, Willian Rampazzo,
	Cleber Rosa, Miroslav Rezanina, Alex Bennée

On 19/04/2021 12.10, Erik Skultety wrote:
> On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
>> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé wrote:
>>> Forks run the same jobs than mainstream, which might be overkill.
>>> Allow them to easily rebase their custom set, while keeping using
>>> the mainstream templates, and ability to pick specific jobs from
>>> the mainstream set.
>>>
>>> To switch to your set, simply add your .gitlab-ci.yml as
>>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
>>> is your gitlab 'namespace', usually username). This file will be
>>> used instead of the default mainstream set.
>>
>> I find this approach undesirable, because AFAICT, it means you have
>> to commit this extra file to any of your downstream branches that
>> you want this to be used for.  Then you have to be either delete it
>> again before sending patches upstream, or tell git-publish to
>> exclude the commit that adds this.
>>
>> IMHO any per-contributor overhead needs to not involve committing
>> stuff to their git branches, that isn't intended to go upstream.
> 
> Not just that, ideally, they should also run all the upstream workloads before
> submitting a PR or posting patches because they'd have to respin because of a
> potential failure in upstream pipelines anyway.

It's pretty clear that you want to run the full QEMU CI before submitting 
patches to the QEMU project, but I think we are rather talking about forks 
here that are meant not meant for immediately contributing to upstream 
again, like RHEL where we only build the KVM-related targets and certainly 
do not want to test other things like CPUs that are not capable of KVM, or a 
branch where Philippe only wants to check his MIPS-related work during 
development.
For contributing patches to upstream, you certainly have to run the full CI, 
but for other things, it's sometimes really useful to cut down the CI 
machinery (I'm also doing this in my development branches manually some 
times to speed up the CI), so I think this series make sense, indeed.

  Thomas



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 10:20       ` Thomas Huth
@ 2021-04-19 10:36         ` Daniel P. Berrangé
  2021-04-19 10:48           ` Thomas Huth
  2021-04-19 10:47         ` Daniel P. Berrangé
  1 sibling, 1 reply; 47+ messages in thread
From: Daniel P. Berrangé @ 2021-04-19 10:36 UTC (permalink / raw)
  To: Thomas Huth
  Cc: Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina,
	Alex Bennée

On Mon, Apr 19, 2021 at 12:20:55PM +0200, Thomas Huth wrote:
> On 19/04/2021 12.10, Erik Skultety wrote:
> > On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
> > > On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé wrote:
> > > > Forks run the same jobs than mainstream, which might be overkill.
> > > > Allow them to easily rebase their custom set, while keeping using
> > > > the mainstream templates, and ability to pick specific jobs from
> > > > the mainstream set.
> > > > 
> > > > To switch to your set, simply add your .gitlab-ci.yml as
> > > > .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
> > > > is your gitlab 'namespace', usually username). This file will be
> > > > used instead of the default mainstream set.
> > > 
> > > I find this approach undesirable, because AFAICT, it means you have
> > > to commit this extra file to any of your downstream branches that
> > > you want this to be used for.  Then you have to be either delete it
> > > again before sending patches upstream, or tell git-publish to
> > > exclude the commit that adds this.
> > > 
> > > IMHO any per-contributor overhead needs to not involve committing
> > > stuff to their git branches, that isn't intended to go upstream.
> > 
> > Not just that, ideally, they should also run all the upstream workloads before
> > submitting a PR or posting patches because they'd have to respin because of a
> > potential failure in upstream pipelines anyway.
> 
> It's pretty clear that you want to run the full QEMU CI before submitting
> patches to the QEMU project, but I think we are rather talking about forks
> here that are meant not meant for immediately contributing to upstream
> again, like RHEL where we only build the KVM-related targets and certainly
> do not want to test other things like CPUs that are not capable of KVM, or a
> branch where Philippe only wants to check his MIPS-related work during
> development.
> For contributing patches to upstream, you certainly have to run the full CI,
> but for other things, it's sometimes really useful to cut down the CI
> machinery (I'm also doing this in my development branches manually some
> times to speed up the CI), so I think this series make sense, indeed.

For a downstream like RHEL, I'd just expect them to replace the main
.gitlab-ci.yml entirely to suit their downstream needs.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 10:10     ` Erik Skultety
  2021-04-19 10:20       ` Thomas Huth
@ 2021-04-19 10:44       ` Philippe Mathieu-Daudé
  1 sibling, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-19 10:44 UTC (permalink / raw)
  To: Erik Skultety, Daniel P. Berrangé
  Cc: Thomas Huth, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina,
	Alex Bennée

On 4/19/21 12:10 PM, Erik Skultety wrote:
> On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
>> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé wrote:
>>> Forks run the same jobs than mainstream, which might be overkill.
>>> Allow them to easily rebase their custom set, while keeping using
>>> the mainstream templates, and ability to pick specific jobs from
>>> the mainstream set.
>>>
>>> To switch to your set, simply add your .gitlab-ci.yml as
>>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
>>> is your gitlab 'namespace', usually username). This file will be
>>> used instead of the default mainstream set.
>>
>> I find this approach undesirable, because AFAICT, it means you have
>> to commit this extra file to any of your downstream branches that
>> you want this to be used for.  Then you have to be either delete it
>> again before sending patches upstream, or tell git-publish to
>> exclude the commit that adds this.
>>
>> IMHO any per-contributor overhead needs to not involve committing
>> stuff to their git branches, that isn't intended to go upstream.
> 
> Not just that, ideally, they should also run all the upstream workloads before
> submitting a PR or posting patches because they'd have to respin because of a
> potential failure in upstream pipelines anyway.

Working a patch series on your fork could take days/weeks/months before
you post it to mainstream... I believe forks are only interested
in running mainstream pipelines when they are ready to post their work,
not at every push to their repository.


^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 10:20       ` Thomas Huth
  2021-04-19 10:36         ` Daniel P. Berrangé
@ 2021-04-19 10:47         ` Daniel P. Berrangé
  1 sibling, 0 replies; 47+ messages in thread
From: Daniel P. Berrangé @ 2021-04-19 10:47 UTC (permalink / raw)
  To: Thomas Huth
  Cc: Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina,
	Alex Bennée

On Mon, Apr 19, 2021 at 12:20:55PM +0200, Thomas Huth wrote:
> On 19/04/2021 12.10, Erik Skultety wrote:
> > On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
> > > On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé wrote:
> > > > Forks run the same jobs than mainstream, which might be overkill.
> > > > Allow them to easily rebase their custom set, while keeping using
> > > > the mainstream templates, and ability to pick specific jobs from
> > > > the mainstream set.
> > > > 
> > > > To switch to your set, simply add your .gitlab-ci.yml as
> > > > .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
> > > > is your gitlab 'namespace', usually username). This file will be
> > > > used instead of the default mainstream set.
> > > 
> > > I find this approach undesirable, because AFAICT, it means you have
> > > to commit this extra file to any of your downstream branches that
> > > you want this to be used for.  Then you have to be either delete it
> > > again before sending patches upstream, or tell git-publish to
> > > exclude the commit that adds this.
> > > 
> > > IMHO any per-contributor overhead needs to not involve committing
> > > stuff to their git branches, that isn't intended to go upstream.
> > 
> > Not just that, ideally, they should also run all the upstream workloads before
> > submitting a PR or posting patches because they'd have to respin because of a
> > potential failure in upstream pipelines anyway.
> 
> It's pretty clear that you want to run the full QEMU CI before submitting
> patches to the QEMU project, but I think we are rather talking about forks
> here that are meant not meant for immediately contributing to upstream
> again, like RHEL where we only build the KVM-related targets and certainly
> do not want to test other things like CPUs that are not capable of KVM, or a
> branch where Philippe only wants to check his MIPS-related work during
> development.
> For contributing patches to upstream, you certainly have to run the full CI,
> but for other things, it's sometimes really useful to cut down the CI
> machinery (I'm also doing this in my development branches manually some
> times to speed up the CI), so I think this series make sense, indeed.

In the case of a permanent fork like RHEL, I'd expect them to just
replace the existing .gitlab-ci.yml entirely in their git repo.

I don't think we need to care about doing anything special downstream
forks, but just focus on what's beneficial to upstream contributors
like the scenario you describe for Philippe only wanting to check
MIPS jobs.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 10:36         ` Daniel P. Berrangé
@ 2021-04-19 10:48           ` Thomas Huth
  2021-04-19 10:51             ` Daniel P. Berrangé
  0 siblings, 1 reply; 47+ messages in thread
From: Thomas Huth @ 2021-04-19 10:48 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina,
	Alex Bennée

On 19/04/2021 12.36, Daniel P. Berrangé wrote:
> On Mon, Apr 19, 2021 at 12:20:55PM +0200, Thomas Huth wrote:
>> On 19/04/2021 12.10, Erik Skultety wrote:
>>> On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
>>>> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé wrote:
>>>>> Forks run the same jobs than mainstream, which might be overkill.
>>>>> Allow them to easily rebase their custom set, while keeping using
>>>>> the mainstream templates, and ability to pick specific jobs from
>>>>> the mainstream set.
>>>>>
>>>>> To switch to your set, simply add your .gitlab-ci.yml as
>>>>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
>>>>> is your gitlab 'namespace', usually username). This file will be
>>>>> used instead of the default mainstream set.
>>>>
>>>> I find this approach undesirable, because AFAICT, it means you have
>>>> to commit this extra file to any of your downstream branches that
>>>> you want this to be used for.  Then you have to be either delete it
>>>> again before sending patches upstream, or tell git-publish to
>>>> exclude the commit that adds this.
>>>>
>>>> IMHO any per-contributor overhead needs to not involve committing
>>>> stuff to their git branches, that isn't intended to go upstream.
>>>
>>> Not just that, ideally, they should also run all the upstream workloads before
>>> submitting a PR or posting patches because they'd have to respin because of a
>>> potential failure in upstream pipelines anyway.
>>
>> It's pretty clear that you want to run the full QEMU CI before submitting
>> patches to the QEMU project, but I think we are rather talking about forks
>> here that are meant not meant for immediately contributing to upstream
>> again, like RHEL where we only build the KVM-related targets and certainly
>> do not want to test other things like CPUs that are not capable of KVM, or a
>> branch where Philippe only wants to check his MIPS-related work during
>> development.
>> For contributing patches to upstream, you certainly have to run the full CI,
>> but for other things, it's sometimes really useful to cut down the CI
>> machinery (I'm also doing this in my development branches manually some
>> times to speed up the CI), so I think this series make sense, indeed.
> 
> For a downstream like RHEL, I'd just expect them to replace the main
> .gitlab-ci.yml entirely to suit their downstream needs.

But that still means that we should clean up the main .gitlab-ci.yml file 
anyway, like it is done in this series, to avoid that you always get 
conflicts in this big file with your downstream-only modifications. So at 
least up to patch 11 or 12, I think this is a very valuable work that 
Philippe is doing here.

  Thomas



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 10:48           ` Thomas Huth
@ 2021-04-19 10:51             ` Daniel P. Berrangé
  2021-04-19 10:59               ` Thomas Huth
  0 siblings, 1 reply; 47+ messages in thread
From: Daniel P. Berrangé @ 2021-04-19 10:51 UTC (permalink / raw)
  To: Thomas Huth
  Cc: Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina,
	Alex Bennée

On Mon, Apr 19, 2021 at 12:48:25PM +0200, Thomas Huth wrote:
> On 19/04/2021 12.36, Daniel P. Berrangé wrote:
> > On Mon, Apr 19, 2021 at 12:20:55PM +0200, Thomas Huth wrote:
> > > On 19/04/2021 12.10, Erik Skultety wrote:
> > > > On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
> > > > > On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé wrote:
> > > > > > Forks run the same jobs than mainstream, which might be overkill.
> > > > > > Allow them to easily rebase their custom set, while keeping using
> > > > > > the mainstream templates, and ability to pick specific jobs from
> > > > > > the mainstream set.
> > > > > > 
> > > > > > To switch to your set, simply add your .gitlab-ci.yml as
> > > > > > .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
> > > > > > is your gitlab 'namespace', usually username). This file will be
> > > > > > used instead of the default mainstream set.
> > > > > 
> > > > > I find this approach undesirable, because AFAICT, it means you have
> > > > > to commit this extra file to any of your downstream branches that
> > > > > you want this to be used for.  Then you have to be either delete it
> > > > > again before sending patches upstream, or tell git-publish to
> > > > > exclude the commit that adds this.
> > > > > 
> > > > > IMHO any per-contributor overhead needs to not involve committing
> > > > > stuff to their git branches, that isn't intended to go upstream.
> > > > 
> > > > Not just that, ideally, they should also run all the upstream workloads before
> > > > submitting a PR or posting patches because they'd have to respin because of a
> > > > potential failure in upstream pipelines anyway.
> > > 
> > > It's pretty clear that you want to run the full QEMU CI before submitting
> > > patches to the QEMU project, but I think we are rather talking about forks
> > > here that are meant not meant for immediately contributing to upstream
> > > again, like RHEL where we only build the KVM-related targets and certainly
> > > do not want to test other things like CPUs that are not capable of KVM, or a
> > > branch where Philippe only wants to check his MIPS-related work during
> > > development.
> > > For contributing patches to upstream, you certainly have to run the full CI,
> > > but for other things, it's sometimes really useful to cut down the CI
> > > machinery (I'm also doing this in my development branches manually some
> > > times to speed up the CI), so I think this series make sense, indeed.
> > 
> > For a downstream like RHEL, I'd just expect them to replace the main
> > .gitlab-ci.yml entirely to suit their downstream needs.
> 
> But that still means that we should clean up the main .gitlab-ci.yml file
> anyway, like it is done in this series, to avoid that you always get
> conflicts in this big file with your downstream-only modifications. So at
> least up to patch 11 or 12, I think this is a very valuable work that
> Philippe is doing here.

I don't see a real issue with downstream conflicts. They'll just
periodically pick a release to base themselves off and change once
every 6 months or more.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 10:51             ` Daniel P. Berrangé
@ 2021-04-19 10:59               ` Thomas Huth
  2021-05-11  6:48                 ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 47+ messages in thread
From: Thomas Huth @ 2021-04-19 10:59 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Philippe Mathieu-Daudé,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina,
	Alex Bennée

On 19/04/2021 12.51, Daniel P. Berrangé wrote:
> On Mon, Apr 19, 2021 at 12:48:25PM +0200, Thomas Huth wrote:
>> On 19/04/2021 12.36, Daniel P. Berrangé wrote:
>>> On Mon, Apr 19, 2021 at 12:20:55PM +0200, Thomas Huth wrote:
>>>> On 19/04/2021 12.10, Erik Skultety wrote:
>>>>> On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
>>>>>> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé wrote:
>>>>>>> Forks run the same jobs than mainstream, which might be overkill.
>>>>>>> Allow them to easily rebase their custom set, while keeping using
>>>>>>> the mainstream templates, and ability to pick specific jobs from
>>>>>>> the mainstream set.
>>>>>>>
>>>>>>> To switch to your set, simply add your .gitlab-ci.yml as
>>>>>>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
>>>>>>> is your gitlab 'namespace', usually username). This file will be
>>>>>>> used instead of the default mainstream set.
>>>>>>
>>>>>> I find this approach undesirable, because AFAICT, it means you have
>>>>>> to commit this extra file to any of your downstream branches that
>>>>>> you want this to be used for.  Then you have to be either delete it
>>>>>> again before sending patches upstream, or tell git-publish to
>>>>>> exclude the commit that adds this.
>>>>>>
>>>>>> IMHO any per-contributor overhead needs to not involve committing
>>>>>> stuff to their git branches, that isn't intended to go upstream.
>>>>>
>>>>> Not just that, ideally, they should also run all the upstream workloads before
>>>>> submitting a PR or posting patches because they'd have to respin because of a
>>>>> potential failure in upstream pipelines anyway.
>>>>
>>>> It's pretty clear that you want to run the full QEMU CI before submitting
>>>> patches to the QEMU project, but I think we are rather talking about forks
>>>> here that are meant not meant for immediately contributing to upstream
>>>> again, like RHEL where we only build the KVM-related targets and certainly
>>>> do not want to test other things like CPUs that are not capable of KVM, or a
>>>> branch where Philippe only wants to check his MIPS-related work during
>>>> development.
>>>> For contributing patches to upstream, you certainly have to run the full CI,
>>>> but for other things, it's sometimes really useful to cut down the CI
>>>> machinery (I'm also doing this in my development branches manually some
>>>> times to speed up the CI), so I think this series make sense, indeed.
>>>
>>> For a downstream like RHEL, I'd just expect them to replace the main
>>> .gitlab-ci.yml entirely to suit their downstream needs.
>>
>> But that still means that we should clean up the main .gitlab-ci.yml file
>> anyway, like it is done in this series, to avoid that you always get
>> conflicts in this big file with your downstream-only modifications. So at
>> least up to patch 11 or 12, I think this is a very valuable work that
>> Philippe is doing here.
> 
> I don't see a real issue with downstream conflicts. They'll just
> periodically pick a release to base themselves off and change once
> every 6 months or more.

It's not only downstream distros that rebase every 6 month. Like Philippe, 
I'm sometimes hacking my .gitlab-ci.yml of my development branch to speed up 
the CI during my development cycles (i.e. I'm removing the jobs that I do 
not need). And I'm regularly rebasing my development branchs. Conflicts in 
.gitlab-ci.yml are then always painful, so a leaner main .gitlab-ci.yml file 
would be helpful for me, too, indeed.

  Thomas



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [PATCH 08/15] gitlab-ci: Extract DCO/style check jobs to checks.yml
  2021-04-19  5:26   ` Thomas Huth
@ 2021-04-19 13:44     ` Wainer dos Santos Moschetta
  0 siblings, 0 replies; 47+ messages in thread
From: Wainer dos Santos Moschetta @ 2021-04-19 13:44 UTC (permalink / raw)
  To: Thomas Huth, Philippe Mathieu-Daudé, qemu-devel
  Cc: Daniel P . Berrangé,
	Erik Skultety, Willian Rampazzo, Cleber Rosa, Miroslav Rezanina,
	Alex Bennée


On 4/19/21 2:26 AM, Thomas Huth wrote:
> On 19/04/2021 01.34, Philippe Mathieu-Daudé wrote:
>> Extract the DCO / checkpatch jobs to a new file (checks.yml)
>> to be able to run them without having to run all the jobs
>> included in the default .gitlab-ci.yml, which are mainly useful
>> for mainstream CI.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>   .gitlab-ci.d/checks.yml | 24 ++++++++++++++++++++++++
>>   .gitlab-ci.yml          |  1 +
>>   2 files changed, 25 insertions(+)
>>   create mode 100644 .gitlab-ci.d/checks.yml
>>
>> diff --git a/.gitlab-ci.d/checks.yml b/.gitlab-ci.d/checks.yml
>> new file mode 100644
>> index 00000000000..f695627b7cd
>> --- /dev/null
>> +++ b/.gitlab-ci.d/checks.yml
>> @@ -0,0 +1,24 @@
>> +check-patch:
>> +  stage: build
>> +  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
>> +  needs:
>> +    job: amd64-centos8-container
>> +  script: .gitlab-ci.d/check-patch.py
>> +  except:
>> +    variables:
>> +      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH 
>> == 'master'
>> +  variables:
>> +    GIT_DEPTH: 1000
>> +  allow_failure: true
>> +
>> +check-dco:
>> +  stage: build
>> +  image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
>> +  needs:
>> +    job: amd64-centos8-container
>> +  script: .gitlab-ci.d/check-dco.py
>> +  except:
>> +    variables:
>> +      - $CI_PROJECT_NAMESPACE == 'qemu-project' && $CI_COMMIT_BRANCH 
>> == 'master'
>> +  variables:
>> +    GIT_DEPTH: 1000
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index bf4862a78de..ba35b3917e7 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -12,6 +12,7 @@ include:
>>     - local: '/.gitlab-ci.d/opensbi.yml'
>>     - local: '/.gitlab-ci.d/containers.yml'
>>     - local: '/.gitlab-ci.d/crossbuilds.yml'
>> +  - local: '/.gitlab-ci.d/checks.yml'
>>     .native_build_job_template:
>>     stage: build
>>
>
> You added the new file, but it looks like you did not remove the jobs 
> from the main .gitlab-ci.yml file ? ... that looks like a mistake?
>
> Also, "checks.yml" sounds a little bit too generic. Maybe rather name 
> it "style-check.yml" or something similar?


Maybe "static_checks.yml".

- Wainer

>
>
>  Thomas



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml
  2021-04-18 23:34 ` [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml Philippe Mathieu-Daudé
  2021-04-19  5:39   ` Thomas Huth
@ 2021-04-19 15:11   ` Alex Bennée
  2021-05-11  7:19     ` Philippe Mathieu-Daudé
  1 sibling, 1 reply; 47+ messages in thread
From: Alex Bennée @ 2021-04-19 15:11 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Extract the build/test jobs run by default on the mainstream
> CI into a new file (buildtest.yml) and their templates into
> (buildtest-template.yml), to be able to reuse the templates
> without having to run all these mainstream jobs by default.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml |  80 +++
>  .gitlab-ci.d/buildtest.yml          | 744 +++++++++++++++++++++++++
>  .gitlab-ci.yml                      | 824 +---------------------------
>  3 files changed, 825 insertions(+), 823 deletions(-)
>  create mode 100644 .gitlab-ci.d/buildtest-template.yml
>  create mode 100644 .gitlab-ci.d/buildtest.yml
>
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> new file mode 100644
> index 00000000000..b24a153d904
> --- /dev/null
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -0,0 +1,80 @@
> +.native_build_job_template:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  before_script:
> +    - JOBS=$(expr $(nproc) + 1)
> +  script:
> +    - mkdir build
> +    - cd build
> +    - if test -n "$TARGETS";
> +      then
> +        ../configure --enable-werror --disable-docs $CONFIGURE_ARGS --target-list="$TARGETS" ;
> +      else
> +        ../configure --enable-werror --disable-docs $CONFIGURE_ARGS ;
> +      fi || { cat config.log meson-logs/meson-log.txt && exit 1; }
> +    - if test -n "$LD_JOBS";
> +      then
> +        meson configure . -Dbackend_max_links="$LD_JOBS" ;
> +      fi || exit 1;
> +    - make -j"$JOBS"
> +    - if test -n "$MAKE_CHECK_ARGS";
> +      then
> +        make -j"$JOBS" $MAKE_CHECK_ARGS ;
> +      fi
> +
> +.native_test_job_template:
> +  stage: test
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  script:
> +    - scripts/git-submodule.sh update
> +        $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
> +    - cd build
> +    - find . -type f -exec touch {} +
> +    # Avoid recompiling by hiding ninja with NINJA=":"
> +    - make NINJA=":" $MAKE_CHECK_ARGS
> +
> +.integration_test_job_template:
> +  cache:
> +    key: "${CI_JOB_NAME}-cache"
> +    paths:
> +      - ${CI_PROJECT_DIR}/avocado-cache
> +    policy: pull-push
> +  artifacts:
> +    name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
> +    when: always
> +    expire_in: 2 days
> +    paths:
> +      - build/tests/results/latest/results.xml
> +      - build/tests/results/latest/test-results
> +    reports:
> +      junit: build/tests/results/latest/results.xml
> +  before_script:
> +    - mkdir -p ~/.config/avocado
> +    - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
> +    - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
> +           >> ~/.config/avocado/avocado.conf
> +    - echo -e '[job.output.testlogs]\nstatuses = ["FAIL", "INTERRUPT"]'
> +           >> ~/.config/avocado/avocado.conf
> +    - if [ -d ${CI_PROJECT_DIR}/avocado-cache ]; then
> +        du -chs ${CI_PROJECT_DIR}/avocado-cache ;
> +      fi
> +    - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
> +  after_script:
> +    - cd build
> +    - du -chs ${CI_PROJECT_DIR}/avocado-cache

I don't know if my suggestion further bellow requires moving the above:

> +build-system-alpine:
> +check-system-alpine:
> +acceptance-system-alpine:

Maybe these can be moved into buildtest-rolling? Or maybe buildtest-[unsupported|other]?

<snip>
<snip>

> +build-disabled:
> +build-tcg-disabled:

to buildtest-custom?


> +build-user-plugins:
> +build-some-softmmu-plugins:
> +clang-system:
> +clang-user:
> +build-cfi-aarch64:
> +check-cfi-aarch64:
> +acceptance-cfi-aarch64:
> +build-cfi-ppc64-s390x:
> +check-cfi-ppc64-s390x:
> +acceptance-cfi-ppc64-s390x:
> +build-cfi-x86_64:
> +check-cfi-x86_64:
> +acceptance-cfi-x86_64:
> +tsan-build:

to buildtest-developer? is CFI a dev feature or a distro feature?
Arguably they could be grouped together.

<snip>
> +# These targets are on the way out
> +build-deprecated:
> +check-deprecated:

buildtest-developer or even buildtest-deprecated?

<snip>
> +gprof-gcov:
> +build-oss-fuzz:

buildtest-developer?

> +build-tci:
> +build-coroutine-sigaltstack:

buildtest-custom?

<snip>
> +crypto-old-nettle:
> +crypto-old-gcrypt:

buildtest-custom or buildtest-legacy?

> +crypto-only-gnutls:
> +build-without-default-devices:
> +build-without-default-features:

buildtest-custom.

> +check-patch:
> +check-dco:

buildtest-developer or maybe their own pr-qa?

> +build-libvhost-user:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
> +  needs:
> +    job: amd64-fedora-container
> +  before_script:
> +    - dnf install -y meson ninja-build
> +  script:
> +    - mkdir subprojects/libvhost-user/build
> +    - cd subprojects/libvhost-user/build
> +    - meson
> +    - ninja

I dunno, buildtest-misc?

> +# No targets are built here, just tools, docs, and unit tests. This
> +# also feeds into the eventual documentation deployment steps later
> +build-tools-and-docs-debian:
> +pages:

Seems these could be under their own documentation section?

-- 
Alex Bennée


^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-18 23:34 ` [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs Philippe Mathieu-Daudé
  2021-04-19  5:48   ` Thomas Huth
  2021-04-19  9:40   ` Daniel P. Berrangé
@ 2021-04-19 15:57   ` Alex Bennée
  2021-04-19 16:22     ` Daniel P. Berrangé
  2021-04-19 16:39     ` Philippe Mathieu-Daudé
  2 siblings, 2 replies; 47+ messages in thread
From: Alex Bennée @ 2021-04-19 15:57 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Forks run the same jobs than mainstream, which might be overkill.
> Allow them to easily rebase their custom set, while keeping using
> the mainstream templates, and ability to pick specific jobs from
> the mainstream set.
>
> To switch to your set, simply add your .gitlab-ci.yml as
> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
> is your gitlab 'namespace', usually username). This file will be
> used instead of the default mainstream set.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.yml | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 718c8e004be..35fd35075db 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -9,7 +9,12 @@ generate-config:
>      paths:
>        - generated-config.yml
>    script:
> -    - cp .gitlab-ci.d/qemu-project.yml generated-config.yml
> +    - if test -e .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml ;
> +      then
> +        cp .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml generated-config.yml ;
> +      else
> +        cp .gitlab-ci.d/qemu-project.yml generated-config.yml ;
> +      fi

This is going to be a little clunky. I can see a use for the static
forks that Danial proposes but I guess what is needed is a little
expressiveness. So how to express things like:

 - I've only touched stuff in linux-user, so run only linux-user tests

 - I'm working on KVM, run all KVM enabled builds and tests

 - I've changed the core TCG code, run everything that exercises that

 - I'm working on ARM, only build and run jobs that have ARM targets

This sounds like tags I guess but the documentation indicates they are
used for runner selection. Could we come up with a subset that could be
used to select from all our build fragments when constructing the
generated-config? I could even imagine a script analysing a diffstat and
guessing the tags based on that.

I think we should define a minimum set of lightweight smoke tests that
get the most bang for buck for catching sillies. I think checkpatch and
dco checking should probably be in there - and maybe one of the bog
standard build everything builds (maybe a random ../configure; make;
make check on one of the supported LTS targets).

Then there is the question of defaults. Should we default to a minimised
set unless asked or should the default be the full fat run everything?
We could I guess only switch to running everything for the staging
branch and anything that is associated with a tag or a branch that has
pull in the name?

>  
>  generate-pipeline:
>    stage: test


-- 
Alex Bennée


^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 15:57   ` Alex Bennée
@ 2021-04-19 16:22     ` Daniel P. Berrangé
  2021-04-19 16:46       ` Philippe Mathieu-Daudé
  2021-04-19 16:39     ` Philippe Mathieu-Daudé
  1 sibling, 1 reply; 47+ messages in thread
From: Daniel P. Berrangé @ 2021-04-19 16:22 UTC (permalink / raw)
  To: Alex Bennée
  Cc: Thomas Huth, Erik Skultety, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, qemu-devel, Willian Rampazzo,
	Cleber Rosa, Miroslav Rezanina

On Mon, Apr 19, 2021 at 04:57:55PM +0100, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
> > Forks run the same jobs than mainstream, which might be overkill.
> > Allow them to easily rebase their custom set, while keeping using
> > the mainstream templates, and ability to pick specific jobs from
> > the mainstream set.
> >
> > To switch to your set, simply add your .gitlab-ci.yml as
> > .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
> > is your gitlab 'namespace', usually username). This file will be
> > used instead of the default mainstream set.
> >
> > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > ---
> >  .gitlab-ci.yml | 7 ++++++-
> >  1 file changed, 6 insertions(+), 1 deletion(-)
> >
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index 718c8e004be..35fd35075db 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -9,7 +9,12 @@ generate-config:
> >      paths:
> >        - generated-config.yml
> >    script:
> > -    - cp .gitlab-ci.d/qemu-project.yml generated-config.yml
> > +    - if test -e .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml ;
> > +      then
> > +        cp .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml generated-config.yml ;
> > +      else
> > +        cp .gitlab-ci.d/qemu-project.yml generated-config.yml ;
> > +      fi
> 
> This is going to be a little clunky. I can see a use for the static
> forks that Danial proposes but I guess what is needed is a little
> expressiveness. So how to express things like:
> 
>  - I've only touched stuff in linux-user, so run only linux-user tests

This can be done with "rules" matching on files, but *only* if the
pipeline trigger is a merge request - specifically not a git branch
push, as the latter doesn't have the semantics you expect wrt
determining the "ancestor" to compare against. It only looks at commits
in the push, not those which may already have previously been pushed
on the branch.

>  - I'm working on KVM, run all KVM enabled builds and tests
> 
>  - I've changed the core TCG code, run everything that exercises that
> 
>  - I'm working on ARM, only build and run jobs that have ARM targets

If the stuff you work on is fairly static, we could potentially
allow env variables to be set by the user in their fork, which
the CI jobs use to filter jobs.

> I think we should define a minimum set of lightweight smoke tests that
> get the most bang for buck for catching sillies. I think checkpatch and
> dco checking should probably be in there - and maybe one of the bog
> standard build everything builds (maybe a random ../configure; make;
> make check on one of the supported LTS targets).

Could we have allow an env var  "QEMU_CI_SMOKE_TEST=1" which can be
set when pushing:

   git push  -o ci.variable="QEMU_CI_SMOKE_TEST=1"


which causes it to only do the minimum neccessary.

Alternatively, invert this, so do minimum smoke test by default
and require an env to run the full test. QEMU_CI_MAX=1

Potentially allow also  "QEMU_CI_EXTRA_JOBS=foo,bar,wizz"
to match against job jnames ?

> Then there is the question of defaults. Should we default to a minimised
> set unless asked or should the default be the full fat run everything?

With the direction gitlab is taking towards limiting CI minuts, it is
probably a safer bet to do a minimal smoke test by default and only
do the full test when definitely needed.

> We could I guess only switch to running everything for the staging
> branch and anything that is associated with a tag or a branch that has
> pull in the name?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 15:57   ` Alex Bennée
  2021-04-19 16:22     ` Daniel P. Berrangé
@ 2021-04-19 16:39     ` Philippe Mathieu-Daudé
  1 sibling, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-19 16:39 UTC (permalink / raw)
  To: Alex Bennée
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina

On 4/19/21 5:57 PM, Alex Bennée wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Forks run the same jobs than mainstream, which might be overkill.
>> Allow them to easily rebase their custom set, while keeping using
>> the mainstream templates, and ability to pick specific jobs from
>> the mainstream set.
>>
>> To switch to your set, simply add your .gitlab-ci.yml as
>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
>> is your gitlab 'namespace', usually username). This file will be
>> used instead of the default mainstream set.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  .gitlab-ci.yml | 7 ++++++-
>>  1 file changed, 6 insertions(+), 1 deletion(-)
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 718c8e004be..35fd35075db 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -9,7 +9,12 @@ generate-config:
>>      paths:
>>        - generated-config.yml
>>    script:
>> -    - cp .gitlab-ci.d/qemu-project.yml generated-config.yml
>> +    - if test -e .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml ;
>> +      then
>> +        cp .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml generated-config.yml ;
>> +      else
>> +        cp .gitlab-ci.d/qemu-project.yml generated-config.yml ;
>> +      fi
> 
> This is going to be a little clunky. I can see a use for the static
> forks that Danial proposes but I guess what is needed is a little
> expressiveness. So how to express things like:
> 
>  - I've only touched stuff in linux-user, so run only linux-user tests
> 
>  - I'm working on KVM, run all KVM enabled builds and tests
> 
>  - I've changed the core TCG code, run everything that exercises that
> 
>  - I'm working on ARM, only build and run jobs that have ARM targets
> 
> This sounds like tags I guess but the documentation indicates they are
> used for runner selection. Could we come up with a subset that could be
> used to select from all our build fragments when constructing the
> generated-config? I could even imagine a script analysing a diffstat and
> guessing the tags based on that.

Ahah this is just what we were discussing with Willian 2h ago after
looking again at stefanha analysis
(https://www.mail-archive.com/qemu-devel@nongnu.org/msg795905.html).

. diff-stat -> files modified
. files modified | get_maintainers -> subsystem maintained sections

I suggested Willian to add support for 'tags' entries to MAINTAINERS,
so we could have:

./get_maintainer --tags file1 file2 ...
-> virtio, migration, kvm

Then we could run all the tests tagged 'virtio, migration, kvm'
(unit tests, iotests, qtests, integration tests).


The transposed use is when a test fails, we can list its tags and
from here get the subsystem maintained sections tracking these tags
and notify them a test using their subsystem failed.

> I think we should define a minimum set of lightweight smoke tests that
> get the most bang for buck for catching sillies. I think checkpatch and
> dco checking should probably be in there - and maybe one of the bog
> standard build everything builds (maybe a random ../configure; make;
> make check on one of the supported LTS targets).
> 
> Then there is the question of defaults. Should we default to a minimised
> set unless asked or should the default be the full fat run everything?
> We could I guess only switch to running everything for the staging
> branch and anything that is associated with a tag or a branch that has
> pull in the name?

Yes, this is a community problem that need to be discussed. Not all the
community members have the same requirements and expectations.

What I'm trying to do here is ease random contributor fork workflow,
not uptimizing mainstream /master gating CI, which is suppose to have
way more resources than a random contributor.

Also I don't believe 1 set of CI jobs will ever make all users happy
together. We have all different needs. I'm looking for a solution
which include every contributors from the community.

I'm brainstorming about a setup where a maintainer might have extra
resources provided by the project (such access to dedicated hardware).
Let's use 'virtio' for example. The maintainer might want to use 2
different pipelines:
- one to run all its 'virtio' tagged tests each time patches are queued
  from the subsystem the contributors (this is the subsystem "gating"
  side).
- one to run extra set more complex, run only before sending a pull
  request.

Regards,

Phil.


^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 16:22     ` Daniel P. Berrangé
@ 2021-04-19 16:46       ` Philippe Mathieu-Daudé
  2021-04-19 16:58         ` Daniel P. Berrangé
  0 siblings, 1 reply; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-19 16:46 UTC (permalink / raw)
  To: Daniel P. Berrangé, Alex Bennée
  Cc: Thomas Huth, Erik Skultety, qemu-devel,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina

On 4/19/21 6:22 PM, Daniel P. Berrangé wrote:
> On Mon, Apr 19, 2021 at 04:57:55PM +0100, Alex Bennée wrote:
>>
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>>
>>> Forks run the same jobs than mainstream, which might be overkill.
>>> Allow them to easily rebase their custom set, while keeping using
>>> the mainstream templates, and ability to pick specific jobs from
>>> the mainstream set.
>>>
>>> To switch to your set, simply add your .gitlab-ci.yml as
>>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
>>> is your gitlab 'namespace', usually username). This file will be
>>> used instead of the default mainstream set.
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>  .gitlab-ci.yml | 7 ++++++-
>>>  1 file changed, 6 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>> index 718c8e004be..35fd35075db 100644
>>> --- a/.gitlab-ci.yml
>>> +++ b/.gitlab-ci.yml
>>> @@ -9,7 +9,12 @@ generate-config:
>>>      paths:
>>>        - generated-config.yml
>>>    script:
>>> -    - cp .gitlab-ci.d/qemu-project.yml generated-config.yml
>>> +    - if test -e .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml ;
>>> +      then
>>> +        cp .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml generated-config.yml ;
>>> +      else
>>> +        cp .gitlab-ci.d/qemu-project.yml generated-config.yml ;
>>> +      fi
>>
>> This is going to be a little clunky. I can see a use for the static
>> forks that Danial proposes but I guess what is needed is a little
>> expressiveness. So how to express things like:
>>
>>  - I've only touched stuff in linux-user, so run only linux-user tests
> 
> This can be done with "rules" matching on files, but *only* if the
> pipeline trigger is a merge request - specifically not a git branch
> push, as the latter doesn't have the semantics you expect wrt
> determining the "ancestor" to compare against. It only looks at commits
> in the push, not those which may already have previously been pushed
> on the branch.
> 
>>  - I'm working on KVM, run all KVM enabled builds and tests
>>
>>  - I've changed the core TCG code, run everything that exercises that
>>
>>  - I'm working on ARM, only build and run jobs that have ARM targets
> 
> If the stuff you work on is fairly static, we could potentially
> allow env variables to be set by the user in their fork, which
> the CI jobs use to filter jobs.
> 
>> I think we should define a minimum set of lightweight smoke tests that
>> get the most bang for buck for catching sillies. I think checkpatch and
>> dco checking should probably be in there - and maybe one of the bog
>> standard build everything builds (maybe a random ../configure; make;
>> make check on one of the supported LTS targets).
> 
> Could we have allow an env var  "QEMU_CI_SMOKE_TEST=1" which can be
> set when pushing:
> 
>    git push  -o ci.variable="QEMU_CI_SMOKE_TEST=1"
> 
> 
> which causes it to only do the minimum neccessary.
> 
> Alternatively, invert this, so do minimum smoke test by default
> and require an env to run the full test. QEMU_CI_MAX=1
> 
> Potentially allow also  "QEMU_CI_EXTRA_JOBS=foo,bar,wizz"
> to match against job jnames ?

Is that what you mean?
https://www.mail-archive.com/qemu-devel@nongnu.org/msg758340.html

(cover https://www.mail-archive.com/qemu-devel@nongnu.org/msg758331.html)

>> Then there is the question of defaults. Should we default to a minimised
>> set unless asked or should the default be the full fat run everything?
> 
> With the direction gitlab is taking towards limiting CI minuts, it is
> probably a safer bet to do a minimal smoke test by default and only
> do the full test when definitely needed.

Yes please.


^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 16:46       ` Philippe Mathieu-Daudé
@ 2021-04-19 16:58         ` Daniel P. Berrangé
  0 siblings, 0 replies; 47+ messages in thread
From: Daniel P. Berrangé @ 2021-04-19 16:58 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Erik Skultety, qemu-devel,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

On Mon, Apr 19, 2021 at 06:46:49PM +0200, Philippe Mathieu-Daudé wrote:
> On 4/19/21 6:22 PM, Daniel P. Berrangé wrote:
> > On Mon, Apr 19, 2021 at 04:57:55PM +0100, Alex Bennée wrote:
> >>
> >> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> >>
> >>> Forks run the same jobs than mainstream, which might be overkill.
> >>> Allow them to easily rebase their custom set, while keeping using
> >>> the mainstream templates, and ability to pick specific jobs from
> >>> the mainstream set.
> >>>
> >>> To switch to your set, simply add your .gitlab-ci.yml as
> >>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where CI_PROJECT_NAMESPACE
> >>> is your gitlab 'namespace', usually username). This file will be
> >>> used instead of the default mainstream set.
> >>>
> >>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >>> ---
> >>>  .gitlab-ci.yml | 7 ++++++-
> >>>  1 file changed, 6 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> >>> index 718c8e004be..35fd35075db 100644
> >>> --- a/.gitlab-ci.yml
> >>> +++ b/.gitlab-ci.yml
> >>> @@ -9,7 +9,12 @@ generate-config:
> >>>      paths:
> >>>        - generated-config.yml
> >>>    script:
> >>> -    - cp .gitlab-ci.d/qemu-project.yml generated-config.yml
> >>> +    - if test -e .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml ;
> >>> +      then
> >>> +        cp .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml generated-config.yml ;
> >>> +      else
> >>> +        cp .gitlab-ci.d/qemu-project.yml generated-config.yml ;
> >>> +      fi
> >>
> >> This is going to be a little clunky. I can see a use for the static
> >> forks that Danial proposes but I guess what is needed is a little
> >> expressiveness. So how to express things like:
> >>
> >>  - I've only touched stuff in linux-user, so run only linux-user tests
> > 
> > This can be done with "rules" matching on files, but *only* if the
> > pipeline trigger is a merge request - specifically not a git branch
> > push, as the latter doesn't have the semantics you expect wrt
> > determining the "ancestor" to compare against. It only looks at commits
> > in the push, not those which may already have previously been pushed
> > on the branch.
> > 
> >>  - I'm working on KVM, run all KVM enabled builds and tests
> >>
> >>  - I've changed the core TCG code, run everything that exercises that
> >>
> >>  - I'm working on ARM, only build and run jobs that have ARM targets
> > 
> > If the stuff you work on is fairly static, we could potentially
> > allow env variables to be set by the user in their fork, which
> > the CI jobs use to filter jobs.
> > 
> >> I think we should define a minimum set of lightweight smoke tests that
> >> get the most bang for buck for catching sillies. I think checkpatch and
> >> dco checking should probably be in there - and maybe one of the bog
> >> standard build everything builds (maybe a random ../configure; make;
> >> make check on one of the supported LTS targets).
> > 
> > Could we have allow an env var  "QEMU_CI_SMOKE_TEST=1" which can be
> > set when pushing:
> > 
> >    git push  -o ci.variable="QEMU_CI_SMOKE_TEST=1"
> > 
> > 
> > which causes it to only do the minimum neccessary.
> > 
> > Alternatively, invert this, so do minimum smoke test by default
> > and require an env to run the full test. QEMU_CI_MAX=1
> > 
> > Potentially allow also  "QEMU_CI_EXTRA_JOBS=foo,bar,wizz"
> > to match against job jnames ?
> 
> Is that what you mean?
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg758340.html

Sort of - this is more implementing high level tags - I was actally
suggesting the explicit job names here.

eg if I see that my pull request to peter has failed on  job "foo",
then when testing fixes it is easier if I can just say run job "foo",
instead of trying to figure out which high level tag happens to pull
in job "foo".

The two approaches probably aren't mutually exclusive though.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [PATCH 05/15] gitlab-ci: Rename acceptance_test_job -> integration_test_job
  2021-04-19  5:19   ` Thomas Huth
@ 2021-04-23 17:18     ` Willian Rampazzo
  0 siblings, 0 replies; 47+ messages in thread
From: Willian Rampazzo @ 2021-04-23 17:18 UTC (permalink / raw)
  To: Thomas Huth
  Cc: Daniel P . Berrangé,
	Erik Skultety, Philippe Mathieu-Daudé,
	Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, qemu-devel, Cleber Rosa,
	Miroslav Rezanina, Alex Bennée

On Mon, Apr 19, 2021 at 2:19 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 19/04/2021 01.34, Philippe Mathieu-Daudé wrote:
> > From: Philippe Mathieu-Daudé <philmd@redhat.com>
> >
> > The acceptance tests aren't accepting anything,
> > rename them as integration tests.
> >
> > Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> > Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> > ---
> >   .gitlab-ci.yml | 20 ++++++++++----------
> >   1 file changed, 10 insertions(+), 10 deletions(-)
>
> Acked-by: Thomas Huth <thuth@redhat.com>
>
> FWIW, I'm in favour of renaming the tests/acceptance directory to
> tests/integration , too.
>

For the sake of consistency, +1 here.



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [PATCH 04/15] gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)
  2021-04-18 23:34 ` [PATCH 04/15] gitlab-ci: Replace YAML anchors by extends (acceptance_test_job) Philippe Mathieu-Daudé
@ 2021-05-03  9:22   ` Thomas Huth
  2021-05-03  9:45     ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 47+ messages in thread
From: Thomas Huth @ 2021-05-03  9:22 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Daniel P . Berrangé,
	Erik Skultety, Alex Bennée, Wainer dos Santos Moschetta,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina,
	Philippe Mathieu-Daudé

On 19/04/2021 01.34, Philippe Mathieu-Daudé wrote:
> From: Philippe Mathieu-Daudé <philmd@redhat.com>
> 
> 'extends' is an alternative to using YAML anchors
> and is a little more flexible and readable. See:
> https://docs.gitlab.com/ee/ci/yaml/#extends
> 
> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   .gitlab-ci.yml | 20 ++++++++++----------
>   1 file changed, 10 insertions(+), 10 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 11bda3abf33..07e5e737685 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -48,7 +48,7 @@ include:
>       # Avoid recompiling by hiding ninja with NINJA=":"
>       - make NINJA=":" $MAKE_CHECK_ARGS
>   
> -.acceptance_template: &acceptance_definition
> +.acceptance_test_job_template:
>     cache:
>       key: "${CI_JOB_NAME}-cache"
>       paths:
> @@ -111,7 +111,7 @@ acceptance-system-alpine:
>     variables:
>       IMAGE: alpine
>       MAKE_CHECK_ARGS: check-acceptance
> -  <<: *acceptance_definition
> +  extends: .acceptance_test_job_template

FYI, I'm getting "YAML invalid" errors when adding this patch to my staging 
branch. I guess you cannot use extends twice in a job and you should extend 
the .acceptance_test_job_template from ..native_build_job_template instead.

  Thomas



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [PATCH 04/15] gitlab-ci: Replace YAML anchors by extends (acceptance_test_job)
  2021-05-03  9:22   ` Thomas Huth
@ 2021-05-03  9:45     ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-03  9:45 UTC (permalink / raw)
  To: Thomas Huth, qemu-devel
  Cc: Daniel P . Berrangé,
	Erik Skultety, Alex Bennée, Wainer dos Santos Moschetta,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina,
	Philippe Mathieu-Daudé

On 5/3/21 11:22 AM, Thomas Huth wrote:
> On 19/04/2021 01.34, Philippe Mathieu-Daudé wrote:
>> From: Philippe Mathieu-Daudé <philmd@redhat.com>
>>
>> 'extends' is an alternative to using YAML anchors
>> and is a little more flexible and readable. See:
>> https://docs.gitlab.com/ee/ci/yaml/#extends
>>
>> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   .gitlab-ci.yml | 20 ++++++++++----------
>>   1 file changed, 10 insertions(+), 10 deletions(-)
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 11bda3abf33..07e5e737685 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -48,7 +48,7 @@ include:
>>       # Avoid recompiling by hiding ninja with NINJA=":"
>>       - make NINJA=":" $MAKE_CHECK_ARGS
>>   -.acceptance_template: &acceptance_definition
>> +.acceptance_test_job_template:
>>     cache:
>>       key: "${CI_JOB_NAME}-cache"
>>       paths:
>> @@ -111,7 +111,7 @@ acceptance-system-alpine:
>>     variables:
>>       IMAGE: alpine
>>       MAKE_CHECK_ARGS: check-acceptance
>> -  <<: *acceptance_definition
>> +  extends: .acceptance_test_job_template
> 
> FYI, I'm getting "YAML invalid" errors when adding this patch to my
> staging branch. I guess you cannot use extends twice in a job and you
> should extend the .acceptance_test_job_template from
> ..native_build_job_template instead.

I'm surprised because I tested/used that branch quite a lot.
Maybe a job using anchors got added meanwhile and I need to
rebase?


^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-04-19 10:59               ` Thomas Huth
@ 2021-05-11  6:48                 ` Philippe Mathieu-Daudé
  2021-05-11 13:55                   ` Stefan Hajnoczi
                                     ` (3 more replies)
  0 siblings, 4 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-11  6:48 UTC (permalink / raw)
  To: Thomas Huth, Daniel P. Berrangé
  Cc: Peter Maydell, Erik Skultety, qemu-devel,
	Wainer dos Santos Moschetta, Willian Rampazzo, Stefan Hajnoczi,
	Cleber Rosa, Miroslav Rezanina, Alex Bennée

+Stefan/Peter

On 4/19/21 12:59 PM, Thomas Huth wrote:
> On 19/04/2021 12.51, Daniel P. Berrangé wrote:
>> On Mon, Apr 19, 2021 at 12:48:25PM +0200, Thomas Huth wrote:
>>> On 19/04/2021 12.36, Daniel P. Berrangé wrote:
>>>> On Mon, Apr 19, 2021 at 12:20:55PM +0200, Thomas Huth wrote:
>>>>> On 19/04/2021 12.10, Erik Skultety wrote:
>>>>>> On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
>>>>>>> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé
>>>>>>> wrote:
>>>>>>>> Forks run the same jobs than mainstream, which might be overkill.
>>>>>>>> Allow them to easily rebase their custom set, while keeping using
>>>>>>>> the mainstream templates, and ability to pick specific jobs from
>>>>>>>> the mainstream set.
>>>>>>>>
>>>>>>>> To switch to your set, simply add your .gitlab-ci.yml as
>>>>>>>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where
>>>>>>>> CI_PROJECT_NAMESPACE
>>>>>>>> is your gitlab 'namespace', usually username). This file will be
>>>>>>>> used instead of the default mainstream set.
>>>>>>>
>>>>>>> I find this approach undesirable, because AFAICT, it means you have
>>>>>>> to commit this extra file to any of your downstream branches that
>>>>>>> you want this to be used for.  Then you have to be either delete it
>>>>>>> again before sending patches upstream, or tell git-publish to
>>>>>>> exclude the commit that adds this.
>>>>>>>
>>>>>>> IMHO any per-contributor overhead needs to not involve committing
>>>>>>> stuff to their git branches, that isn't intended to go upstream.
>>>>>>
>>>>>> Not just that, ideally, they should also run all the upstream
>>>>>> workloads before
>>>>>> submitting a PR or posting patches because they'd have to respin
>>>>>> because of a
>>>>>> potential failure in upstream pipelines anyway.
>>>>>
>>>>> It's pretty clear that you want to run the full QEMU CI before
>>>>> submitting
>>>>> patches to the QEMU project, but I think we are rather talking
>>>>> about forks
>>>>> here that are meant not meant for immediately contributing to upstream
>>>>> again, like RHEL where we only build the KVM-related targets and
>>>>> certainly
>>>>> do not want to test other things like CPUs that are not capable of
>>>>> KVM, or a
>>>>> branch where Philippe only wants to check his MIPS-related work during
>>>>> development.
>>>>> For contributing patches to upstream, you certainly have to run the
>>>>> full CI,
>>>>> but for other things, it's sometimes really useful to cut down the CI
>>>>> machinery (I'm also doing this in my development branches manually
>>>>> some
>>>>> times to speed up the CI), so I think this series make sense, indeed.
>>>>
>>>> For a downstream like RHEL, I'd just expect them to replace the main
>>>> .gitlab-ci.yml entirely to suit their downstream needs.
>>>
>>> But that still means that we should clean up the main .gitlab-ci.yml
>>> file
>>> anyway, like it is done in this series, to avoid that you always get
>>> conflicts in this big file with your downstream-only modifications.
>>> So at
>>> least up to patch 11 or 12, I think this is a very valuable work that
>>> Philippe is doing here.
>>
>> I don't see a real issue with downstream conflicts. They'll just
>> periodically pick a release to base themselves off and change once
>> every 6 months or more.
> 
> It's not only downstream distros that rebase every 6 month. Like
> Philippe, I'm sometimes hacking my .gitlab-ci.yml of my development
> branch to speed up the CI during my development cycles (i.e. I'm
> removing the jobs that I do not need). And I'm regularly rebasing my
> development branchs. Conflicts in .gitlab-ci.yml are then always
> painful, so a leaner main .gitlab-ci.yml file would be helpful for me,
> too, indeed.

Not sure if following up this thread or start a new one, but I got
blocked again from Gitlab, tagged as a crypto miner for running QEMU
CI...
[1]
https://about.gitlab.com/handbook/support/workflows/investigate_blocked_pipeline.html#trends--high-priority-cases

I pushed 5 different branches to my repository in less than 1h,
kicking 580 jobs [*].

I didn't try to stress Gitlab, it was a simple "one time in the month
rebase unmerged branches, push them before respining on the mailing
list".

I'm considering changing my workflow:
- not push more than 2 branches per hour (I know 3/h works, so choose
  a lower number, as we want to add more tests).
- merge multiple branches locally and push the merged result and
  bisect / re-push on failure
- run less testing
- do not run testing

This sounds counter productive and doesn't scale to a community of
contributors asked to use Gitlab.

So far I don't have better idea than this series.

Who is interested in sending patches to improve our workflow?

Thanks,

Phil.

[*] NB I have 3 extra runners added to my namespace, but it didn't
help, as per [1] I got blocked by reaching an API rate limit.


^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml
  2021-04-19 15:11   ` Alex Bennée
@ 2021-05-11  7:19     ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-11  7:19 UTC (permalink / raw)
  To: Alex Bennée
  Cc: Thomas Huth, Daniel P . Berrangé,
	Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina

Hi Alex,

On 4/19/21 5:11 PM, Alex Bennée wrote:
> 
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Extract the build/test jobs run by default on the mainstream
>> CI into a new file (buildtest.yml) and their templates into
>> (buildtest-template.yml), to be able to reuse the templates
>> without having to run all these mainstream jobs by default.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  .gitlab-ci.d/buildtest-template.yml |  80 +++
>>  .gitlab-ci.d/buildtest.yml          | 744 +++++++++++++++++++++++++
>>  .gitlab-ci.yml                      | 824 +---------------------------
>>  3 files changed, 825 insertions(+), 823 deletions(-)
>>  create mode 100644 .gitlab-ci.d/buildtest-template.yml
>>  create mode 100644 .gitlab-ci.d/buildtest.yml

> I don't know if my suggestion further bellow requires moving the above:
> 
>> +build-system-alpine:
>> +check-system-alpine:
>> +acceptance-system-alpine:
> 
> Maybe these can be moved into buildtest-rolling? Or maybe buildtest-[unsupported|other]?

As I temporarily can't test your suggestions, I'll send v2 without
them, and will try to address them on v3 (I am not ignoring them).

> <snip>
> <snip>
> 
>> +build-disabled:
>> +build-tcg-disabled:
> 
> to buildtest-custom?
> 
> 
>> +build-user-plugins:
>> +build-some-softmmu-plugins:
>> +clang-system:
>> +clang-user:
>> +build-cfi-aarch64:
>> +check-cfi-aarch64:
>> +acceptance-cfi-aarch64:
>> +build-cfi-ppc64-s390x:
>> +check-cfi-ppc64-s390x:
>> +acceptance-cfi-ppc64-s390x:
>> +build-cfi-x86_64:
>> +check-cfi-x86_64:
>> +acceptance-cfi-x86_64:
>> +tsan-build:
> 
> to buildtest-developer? is CFI a dev feature or a distro feature?
> Arguably they could be grouped together.
> 
> <snip>
>> +# These targets are on the way out
>> +build-deprecated:
>> +check-deprecated:
> 
> buildtest-developer or even buildtest-deprecated?
> 
> <snip>
>> +gprof-gcov:
>> +build-oss-fuzz:
> 
> buildtest-developer?
> 
>> +build-tci:
>> +build-coroutine-sigaltstack:
> 
> buildtest-custom?
> 
> <snip>
>> +crypto-old-nettle:
>> +crypto-old-gcrypt:
> 
> buildtest-custom or buildtest-legacy?
> 
>> +crypto-only-gnutls:
>> +build-without-default-devices:
>> +build-without-default-features:
> 
> buildtest-custom.
> 
>> +check-patch:
>> +check-dco:
> 
> buildtest-developer or maybe their own pr-qa?
> 
>> +build-libvhost-user:
>> +  stage: build
>> +  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
>> +  needs:
>> +    job: amd64-fedora-container
>> +  before_script:
>> +    - dnf install -y meson ninja-build
>> +  script:
>> +    - mkdir subprojects/libvhost-user/build
>> +    - cd subprojects/libvhost-user/build
>> +    - meson
>> +    - ninja
> 
> I dunno, buildtest-misc?
> 
>> +# No targets are built here, just tools, docs, and unit tests. This
>> +# also feeds into the eventual documentation deployment steps later
>> +build-tools-and-docs-debian:
>> +pages:
> 
> Seems these could be under their own documentation section?
> 


^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-05-11  6:48                 ` Philippe Mathieu-Daudé
@ 2021-05-11 13:55                   ` Stefan Hajnoczi
  2021-05-11 14:00                   ` Alex Bennée
                                     ` (2 subsequent siblings)
  3 siblings, 0 replies; 47+ messages in thread
From: Stefan Hajnoczi @ 2021-05-11 13:55 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Peter Maydell, Thomas Huth, Daniel P. Berrangé,
	Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Cleber Rosa, Miroslav Rezanina,
	Alex Bennée

[-- Attachment #1: Type: text/plain, Size: 5600 bytes --]

On Tue, May 11, 2021 at 08:48:44AM +0200, Philippe Mathieu-Daudé wrote:
> +Stefan/Peter
> 
> On 4/19/21 12:59 PM, Thomas Huth wrote:
> > On 19/04/2021 12.51, Daniel P. Berrangé wrote:
> >> On Mon, Apr 19, 2021 at 12:48:25PM +0200, Thomas Huth wrote:
> >>> On 19/04/2021 12.36, Daniel P. Berrangé wrote:
> >>>> On Mon, Apr 19, 2021 at 12:20:55PM +0200, Thomas Huth wrote:
> >>>>> On 19/04/2021 12.10, Erik Skultety wrote:
> >>>>>> On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
> >>>>>>> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé
> >>>>>>> wrote:
> >>>>>>>> Forks run the same jobs than mainstream, which might be overkill.
> >>>>>>>> Allow them to easily rebase their custom set, while keeping using
> >>>>>>>> the mainstream templates, and ability to pick specific jobs from
> >>>>>>>> the mainstream set.
> >>>>>>>>
> >>>>>>>> To switch to your set, simply add your .gitlab-ci.yml as
> >>>>>>>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where
> >>>>>>>> CI_PROJECT_NAMESPACE
> >>>>>>>> is your gitlab 'namespace', usually username). This file will be
> >>>>>>>> used instead of the default mainstream set.
> >>>>>>>
> >>>>>>> I find this approach undesirable, because AFAICT, it means you have
> >>>>>>> to commit this extra file to any of your downstream branches that
> >>>>>>> you want this to be used for.  Then you have to be either delete it
> >>>>>>> again before sending patches upstream, or tell git-publish to
> >>>>>>> exclude the commit that adds this.
> >>>>>>>
> >>>>>>> IMHO any per-contributor overhead needs to not involve committing
> >>>>>>> stuff to their git branches, that isn't intended to go upstream.
> >>>>>>
> >>>>>> Not just that, ideally, they should also run all the upstream
> >>>>>> workloads before
> >>>>>> submitting a PR or posting patches because they'd have to respin
> >>>>>> because of a
> >>>>>> potential failure in upstream pipelines anyway.
> >>>>>
> >>>>> It's pretty clear that you want to run the full QEMU CI before
> >>>>> submitting
> >>>>> patches to the QEMU project, but I think we are rather talking
> >>>>> about forks
> >>>>> here that are meant not meant for immediately contributing to upstream
> >>>>> again, like RHEL where we only build the KVM-related targets and
> >>>>> certainly
> >>>>> do not want to test other things like CPUs that are not capable of
> >>>>> KVM, or a
> >>>>> branch where Philippe only wants to check his MIPS-related work during
> >>>>> development.
> >>>>> For contributing patches to upstream, you certainly have to run the
> >>>>> full CI,
> >>>>> but for other things, it's sometimes really useful to cut down the CI
> >>>>> machinery (I'm also doing this in my development branches manually
> >>>>> some
> >>>>> times to speed up the CI), so I think this series make sense, indeed.
> >>>>
> >>>> For a downstream like RHEL, I'd just expect them to replace the main
> >>>> .gitlab-ci.yml entirely to suit their downstream needs.
> >>>
> >>> But that still means that we should clean up the main .gitlab-ci.yml
> >>> file
> >>> anyway, like it is done in this series, to avoid that you always get
> >>> conflicts in this big file with your downstream-only modifications.
> >>> So at
> >>> least up to patch 11 or 12, I think this is a very valuable work that
> >>> Philippe is doing here.
> >>
> >> I don't see a real issue with downstream conflicts. They'll just
> >> periodically pick a release to base themselves off and change once
> >> every 6 months or more.
> > 
> > It's not only downstream distros that rebase every 6 month. Like
> > Philippe, I'm sometimes hacking my .gitlab-ci.yml of my development
> > branch to speed up the CI during my development cycles (i.e. I'm
> > removing the jobs that I do not need). And I'm regularly rebasing my
> > development branchs. Conflicts in .gitlab-ci.yml are then always
> > painful, so a leaner main .gitlab-ci.yml file would be helpful for me,
> > too, indeed.
> 
> Not sure if following up this thread or start a new one, but I got
> blocked again from Gitlab, tagged as a crypto miner for running QEMU
> CI...
> [1]
> https://about.gitlab.com/handbook/support/workflows/investigate_blocked_pipeline.html#trends--high-priority-cases
> 
> I pushed 5 different branches to my repository in less than 1h,
> kicking 580 jobs [*].
> 
> I didn't try to stress Gitlab, it was a simple "one time in the month
> rebase unmerged branches, push them before respining on the mailing
> list".
> 
> I'm considering changing my workflow:
> - not push more than 2 branches per hour (I know 3/h works, so choose
>   a lower number, as we want to add more tests).
> - merge multiple branches locally and push the merged result and
>   bisect / re-push on failure
> - run less testing
> - do not run testing
> 
> This sounds counter productive and doesn't scale to a community of
> contributors asked to use Gitlab.
> 
> So far I don't have better idea than this series.
> 
> Who is interested in sending patches to improve our workflow?
> 
> Thanks,
> 
> Phil.
> 
> [*] NB I have 3 extra runners added to my namespace, but it didn't
> help, as per [1] I got blocked by reaching an API rate limit.

The easiest short-term workaround seems to be disabling testing when you
push certain branches.

In the long term I think GitLab CI should allow unlimited jobs on
dedicated runners. It may be necessary to get in touch with GitLab
support and figure out how to stop it blocking dedicated runner jobs.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-05-11  6:48                 ` Philippe Mathieu-Daudé
  2021-05-11 13:55                   ` Stefan Hajnoczi
@ 2021-05-11 14:00                   ` Alex Bennée
  2021-05-11 14:21                   ` Daniel P. Berrangé
  2021-05-13 19:01                   ` Philippe Mathieu-Daudé
  3 siblings, 0 replies; 47+ messages in thread
From: Alex Bennée @ 2021-05-11 14:00 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Peter Maydell, Thomas Huth, Daniel P. Berrangé,
	Erik Skultety, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Stefan Hajnoczi, Cleber Rosa,
	Miroslav Rezanina


Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> +Stefan/Peter
>
> On 4/19/21 12:59 PM, Thomas Huth wrote:
>> On 19/04/2021 12.51, Daniel P. Berrangé wrote:
>>> On Mon, Apr 19, 2021 at 12:48:25PM +0200, Thomas Huth wrote:
>>>> On 19/04/2021 12.36, Daniel P. Berrangé wrote:
>>>>> On Mon, Apr 19, 2021 at 12:20:55PM +0200, Thomas Huth wrote:
>>>>>> On 19/04/2021 12.10, Erik Skultety wrote:
>>>>>>> On Mon, Apr 19, 2021 at 10:40:53AM +0100, Daniel P. Berrangé wrote:
>>>>>>>> On Mon, Apr 19, 2021 at 01:34:47AM +0200, Philippe Mathieu-Daudé
>>>>>>>> wrote:
>>>>>>>>> Forks run the same jobs than mainstream, which might be overkill.
>>>>>>>>> Allow them to easily rebase their custom set, while keeping using
>>>>>>>>> the mainstream templates, and ability to pick specific jobs from
>>>>>>>>> the mainstream set.
>>>>>>>>>
>>>>>>>>> To switch to your set, simply add your .gitlab-ci.yml as
>>>>>>>>> .gitlab-ci.d/${CI_PROJECT_NAMESPACE}.yml (where
>>>>>>>>> CI_PROJECT_NAMESPACE
>>>>>>>>> is your gitlab 'namespace', usually username). This file will be
>>>>>>>>> used instead of the default mainstream set.
>>>>>>>>
>>>>>>>> I find this approach undesirable, because AFAICT, it means you have
>>>>>>>> to commit this extra file to any of your downstream branches that
>>>>>>>> you want this to be used for.  Then you have to be either delete it
>>>>>>>> again before sending patches upstream, or tell git-publish to
>>>>>>>> exclude the commit that adds this.
>>>>>>>>
>>>>>>>> IMHO any per-contributor overhead needs to not involve committing
>>>>>>>> stuff to their git branches, that isn't intended to go upstream.
>>>>>>>
>>>>>>> Not just that, ideally, they should also run all the upstream
>>>>>>> workloads before
>>>>>>> submitting a PR or posting patches because they'd have to respin
>>>>>>> because of a
>>>>>>> potential failure in upstream pipelines anyway.
>>>>>>
>>>>>> It's pretty clear that you want to run the full QEMU CI before
>>>>>> submitting
>>>>>> patches to the QEMU project, but I think we are rather talking
>>>>>> about forks
>>>>>> here that are meant not meant for immediately contributing to upstream
>>>>>> again, like RHEL where we only build the KVM-related targets and
>>>>>> certainly
>>>>>> do not want to test other things like CPUs that are not capable of
>>>>>> KVM, or a
>>>>>> branch where Philippe only wants to check his MIPS-related work during
>>>>>> development.
>>>>>> For contributing patches to upstream, you certainly have to run the
>>>>>> full CI,
>>>>>> but for other things, it's sometimes really useful to cut down the CI
>>>>>> machinery (I'm also doing this in my development branches manually
>>>>>> some
>>>>>> times to speed up the CI), so I think this series make sense, indeed.
>>>>>
>>>>> For a downstream like RHEL, I'd just expect them to replace the main
>>>>> .gitlab-ci.yml entirely to suit their downstream needs.
>>>>
>>>> But that still means that we should clean up the main .gitlab-ci.yml
>>>> file
>>>> anyway, like it is done in this series, to avoid that you always get
>>>> conflicts in this big file with your downstream-only modifications.
>>>> So at
>>>> least up to patch 11 or 12, I think this is a very valuable work that
>>>> Philippe is doing here.
>>>
>>> I don't see a real issue with downstream conflicts. They'll just
>>> periodically pick a release to base themselves off and change once
>>> every 6 months or more.
>> 
>> It's not only downstream distros that rebase every 6 month. Like
>> Philippe, I'm sometimes hacking my .gitlab-ci.yml of my development
>> branch to speed up the CI during my development cycles (i.e. I'm
>> removing the jobs that I do not need). And I'm regularly rebasing my
>> development branchs. Conflicts in .gitlab-ci.yml are then always
>> painful, so a leaner main .gitlab-ci.yml file would be helpful for me,
>> too, indeed.
>
> Not sure if following up this thread or start a new one, but I got
> blocked again from Gitlab, tagged as a crypto miner for running QEMU
> CI...
> [1]
> https://about.gitlab.com/handbook/support/workflows/investigate_blocked_pipeline.html#trends--high-priority-cases
>
> I pushed 5 different branches to my repository in less than 1h,
> kicking 580 jobs [*].
>
> I didn't try to stress Gitlab, it was a simple "one time in the month
> rebase unmerged branches, push them before respining on the mailing
> list".
>
> I'm considering changing my workflow:
> - not push more than 2 branches per hour (I know 3/h works, so choose
>   a lower number, as we want to add more tests).
> - merge multiple branches locally and push the merged result and
>   bisect / re-push on failure

I stack my branches - so usually I have a:

 testing/next
 gdb/next
 whatever my current hack is

Every week I re-base the branches and re-build my current hacking tree.
If an actual problem shows up in CI I'll bisect on one of my beefy boxes
to fix it and then fix and re-push testing/next and whatever my tip is.

> - run less testing
> - do not run testing

I run a lot of testing locally (or rather on a beefy server) so I'm
really only using GitLab for final validation of trees rather than day 2
day.

>
> This sounds counter productive and doesn't scale to a community of
> contributors asked to use Gitlab.
>
> So far I don't have better idea than this series.
>
> Who is interested in sending patches to improve our workflow?
>
> Thanks,
>
> Phil.
>
> [*] NB I have 3 extra runners added to my namespace, but it didn't
> help, as per [1] I got blocked by reaching an API rate limit.


-- 
Alex Bennée


^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-05-11  6:48                 ` Philippe Mathieu-Daudé
  2021-05-11 13:55                   ` Stefan Hajnoczi
  2021-05-11 14:00                   ` Alex Bennée
@ 2021-05-11 14:21                   ` Daniel P. Berrangé
  2021-05-13 19:01                   ` Philippe Mathieu-Daudé
  3 siblings, 0 replies; 47+ messages in thread
From: Daniel P. Berrangé @ 2021-05-11 14:21 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Peter Maydell, Thomas Huth, Erik Skultety, qemu-devel,
	Wainer dos Santos Moschetta, Willian Rampazzo, Stefan Hajnoczi,
	Cleber Rosa, Miroslav Rezanina, Alex Bennée

On Tue, May 11, 2021 at 08:48:44AM +0200, Philippe Mathieu-Daudé wrote:
> +Stefan/Peter
> 
> Not sure if following up this thread or start a new one, but I got
> blocked again from Gitlab, tagged as a crypto miner for running QEMU
> CI...
> [1]
> https://about.gitlab.com/handbook/support/workflows/investigate_blocked_pipeline.html#trends--high-priority-cases
> 
> I pushed 5 different branches to my repository in less than 1h,
> kicking 580 jobs [*].
> 
> I didn't try to stress Gitlab, it was a simple "one time in the month
> rebase unmerged branches, push them before respining on the mailing
> list".
> 
> I'm considering changing my workflow:
> - not push more than 2 branches per hour (I know 3/h works, so choose
>   a lower number, as we want to add more tests).
> - merge multiple branches locally and push the merged result and
>   bisect / re-push on failure
> - run less testing
> - do not run testing
> 
> This sounds counter productive and doesn't scale to a community of
> contributors asked to use Gitlab.
> 
> So far I don't have better idea than this series.
> 
> Who is interested in sending patches to improve our workflow?

So we have a few scenarios for using the CI

 1. Running gating CI before merging to master
 2. Subsystem maintainers running CI before sending a PULL req
 3. Contributors running CI before sending a patch series

Right now we have the same jobs running in all three scenarios.

Given the increasing restrictions on usage, we clearly need to cut
down in general and also make it so that it is harder to accidentally
burn all your available CI allowance.

Currently we always run CI whenever pushing to gitlab. This is
convenient but in retrospect it is overkill. People often push
to gitlab simply as their backup strategy and thus don't need
CI run every time.

Not all changes require all possible jobs to by run, but it is hard
to filter jobs when we're triggering them based on pushes, as the
baseline against which file changes are identified is ill-defined.


For scenario (1) we need all the jobs run to maximise quality.
This is also a case where we're most likely to have custom runners
available, so CI allowance is less of a concern. The job count still
needs to be reasonable to avoid hitting issues at times when the
merges are frequent (just before freeze).


For scenario (2) subsys maintainers, we want them to minimize
the liklihood that a pull request will fail scenario (1) and
require a respin.  Running all jobs achieces this but it is
likely overkill.

eg we have 24 cross compiler builds. If we expect most maintainers
will have either x86-64 or aarch64 hardware for their primary dev
platform, then the key benefit of cross compilers is getting coverage
of

 - 32-bit
 - big endian
 - windows

We don't need 24 jobs todo that. We could simply pick armel as the
most relevant 32-bit arch and s390x as the most relevant big endian
arch, and then the win32/64 platforms. IOW we could potentially only
run 4-6 jobs instead of 24, and still get excellant arch coverage.

Similarly for native builds we test quite alot of different distros.
I think we probably can rationalize that down to just 2 distros,
one covering oldest packages (Debian Stretch) and one covering newest
(Fedora 34), and a "build everything" config.

We have many other jobs that are testing various obscure combinations
of configure args. I'd suggest these rarely fail for most pull requests
so are overkill.

For subsystem maintainers we could potentially get down to just 10-15
jobs if we're ambitious. Leave everything else as manual trigger only.

Perhaps set all the jobs to only run on certain branch name patterns.
eg perhaps "*-next" filter is common for subsystem maintainer's pending
branches ?

For general contributors a similarly short set of jobs to subsystem
maintainers is viable. Perhaps again just let then use a "-next"
branch.

If we can enable manual triggers on any other branches that's good.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 47+ messages in thread

* Re: [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs
  2021-05-11  6:48                 ` Philippe Mathieu-Daudé
                                     ` (2 preceding siblings ...)
  2021-05-11 14:21                   ` Daniel P. Berrangé
@ 2021-05-13 19:01                   ` Philippe Mathieu-Daudé
  3 siblings, 0 replies; 47+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-13 19:01 UTC (permalink / raw)
  To: Thomas Huth, Daniel P. Berrangé
  Cc: Peter Maydell, Erik Skultety, qemu-devel,
	Wainer dos Santos Moschetta, Willian Rampazzo, Stefan Hajnoczi,
	Cleber Rosa, Miroslav Rezanina, Alex Bennée

On 5/11/21 8:48 AM, Philippe Mathieu-Daudé wrote:
> Not sure if following up this thread or start a new one, but I got
> blocked again from Gitlab, tagged as a crypto miner for running QEMU
> CI...
> [1]
> https://about.gitlab.com/handbook/support/workflows/investigate_blocked_pipeline.html#trends--high-priority-cases
> 
> I pushed 5 different branches to my repository in less than 1h,
> kicking 580 jobs [*].
> 
> I didn't try to stress Gitlab, it was a simple "one time in the month
> rebase unmerged branches, push them before respining on the mailing
> list".

FYI I got my account unlocked (without any notification update).
36h passed, maybe it is something automatic (block the user for
36h if suspected of crypto mining?).


^ permalink raw reply	[flat|nested] 47+ messages in thread

end of thread, other threads:[~2021-05-13 19:02 UTC | newest]

Thread overview: 47+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-18 23:34 [RFC PATCH 00/15] gitlab-ci: Allow forks to use different pipelines than mainstream Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 01/15] gitlab-ci: Replace YAML anchors by extends (container_job) Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 02/15] gitlab-ci: Replace YAML anchors by extends (native_build_job) Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 03/15] gitlab-ci: Replace YAML anchors by extends (native_test_job) Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 04/15] gitlab-ci: Replace YAML anchors by extends (acceptance_test_job) Philippe Mathieu-Daudé
2021-05-03  9:22   ` Thomas Huth
2021-05-03  9:45     ` Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 05/15] gitlab-ci: Rename acceptance_test_job -> integration_test_job Philippe Mathieu-Daudé
2021-04-19  5:19   ` Thomas Huth
2021-04-23 17:18     ` Willian Rampazzo
2021-04-18 23:34 ` [PATCH 06/15] gitlab-ci: Extract container job template to container-template.yml Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 07/15] gitlab-ci: Extract crossbuild job templates to crossbuild-template.yml Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 08/15] gitlab-ci: Extract DCO/style check jobs to checks.yml Philippe Mathieu-Daudé
2021-04-19  5:26   ` Thomas Huth
2021-04-19 13:44     ` Wainer dos Santos Moschetta
2021-04-18 23:34 ` [PATCH 09/15] gitlab-ci: Extract build stages to stages.yml Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 10/15] gitlab-ci: Extract all default build/test jobs to buildtest.yml Philippe Mathieu-Daudé
2021-04-19  5:39   ` Thomas Huth
2021-04-19 15:11   ` Alex Bennée
2021-05-11  7:19     ` Philippe Mathieu-Daudé
2021-04-18 23:34 ` [PATCH 11/15] gitlab-ci: Extract core container jobs to container-core.yml Philippe Mathieu-Daudé
2021-04-19  5:42   ` Thomas Huth
2021-04-18 23:34 ` [PATCH 12/15] gitlab-ci: Move current job set to qemu-project.yml Philippe Mathieu-Daudé
2021-04-18 23:34 ` [RFC PATCH 13/15] gitlab-ci: Switch to dynamically generated pipelines Philippe Mathieu-Daudé
2021-04-18 23:34 ` [RFC PATCH 14/15] gitlab-ci: Allow forks to use different set of jobs Philippe Mathieu-Daudé
2021-04-19  5:48   ` Thomas Huth
2021-04-19  9:40   ` Daniel P. Berrangé
2021-04-19 10:09     ` Philippe Mathieu-Daudé
2021-04-19 10:10     ` Erik Skultety
2021-04-19 10:20       ` Thomas Huth
2021-04-19 10:36         ` Daniel P. Berrangé
2021-04-19 10:48           ` Thomas Huth
2021-04-19 10:51             ` Daniel P. Berrangé
2021-04-19 10:59               ` Thomas Huth
2021-05-11  6:48                 ` Philippe Mathieu-Daudé
2021-05-11 13:55                   ` Stefan Hajnoczi
2021-05-11 14:00                   ` Alex Bennée
2021-05-11 14:21                   ` Daniel P. Berrangé
2021-05-13 19:01                   ` Philippe Mathieu-Daudé
2021-04-19 10:47         ` Daniel P. Berrangé
2021-04-19 10:44       ` Philippe Mathieu-Daudé
2021-04-19 15:57   ` Alex Bennée
2021-04-19 16:22     ` Daniel P. Berrangé
2021-04-19 16:46       ` Philippe Mathieu-Daudé
2021-04-19 16:58         ` Daniel P. Berrangé
2021-04-19 16:39     ` Philippe Mathieu-Daudé
2021-04-18 23:34 ` [NOTFORMERGE PATCH 15/15] gitlab-ci: Use my own set of jobs for CI pipeline Philippe Mathieu-Daudé

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).