[PATCH 0/5] gitlab: restrict running jobs in forks and upstream master

[PATCH 0/5] gitlab: restrict running jobs in forks and upstream master

[Daniel P. Berrangé]

Currently on upstream most jobs will run in both staging
and master. This is quite wasteful of CI credits. The only
need to run in master is for the jobs related to publishing
the website

In forks we run jobs on every push. With restricted CI
allowance this is quickly going to cause  problems.

With this series jobs will no longer run on forks at all,
without an opt-in with QEMU_CI=1 (pipeline with manual
jobs) or QEMU_CI=2 (pipeline with immediate jobs)

This is a rewrite of a previous proposal:

https://lists.nongnu.org/archive/html/qemu-devel/2021-08/msg02104.html

where I've kept it simpler and also split up the patches
into more understandable chunks

Daniel P. Berrangé (5):
  gitlab: introduce a common base job template
  gitlab: convert Cirrus jobs to .base_job_template
  gitlab: convert static checks to .base_job_template
  gitlab: convert build/container jobs to .base_job_template
  gitlab: don't run CI jobs in forks by default

 .gitlab-ci.d/base.yml                | 72 +++++++++++++++++++++++
 .gitlab-ci.d/buildtest-template.yml  | 16 ++---
 .gitlab-ci.d/buildtest.yml           | 28 ++++-----
 .gitlab-ci.d/cirrus.yml              | 16 ++---
 .gitlab-ci.d/container-cross.yml     |  6 +-
 .gitlab-ci.d/container-template.yml  |  1 +
 .gitlab-ci.d/crossbuild-template.yml |  3 +
 .gitlab-ci.d/qemu-project.yml        |  1 +
 .gitlab-ci.d/static_checks.yml       | 19 +++---
 .gitlab-ci.d/windows.yml             |  1 +
 docs/devel/ci-jobs.rst.inc           | 88 +++++++++++++++++++++++++++-
 11 files changed, 199 insertions(+), 52 deletions(-)
 create mode 100644 .gitlab-ci.d/base.yml

-- 
2.36.1

[PATCH 1/5] gitlab: introduce a common base job template

[Daniel P. Berrangé]

Currently job rules are spread across the various templates
and jobs, making it hard to understand exactly what runs in
what scenario. This leads to inconsistency in the rules and
increased maint burden.

The intent is that we introduce a common '.base_job_template'
which will have a general purpose 'rules:' block. No other
template or job should define 'rules:', but instead they must
rely on the inherited rules. To allow behaviour to be tweaked,
rules will be influenced by a number of variables with the
naming scheme 'QEMU_JOB_nnnn'.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml         | 28 +++++++++++++++++++++++++++
 .gitlab-ci.d/qemu-project.yml |  1 +
 docs/devel/ci-jobs.rst.inc    | 36 ++++++++++++++++++++++++++++++++++-
 3 files changed, 64 insertions(+), 1 deletion(-)
 create mode 100644 .gitlab-ci.d/base.yml

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
new file mode 100644
index 0000000000..10eb6ab8bc
--- /dev/null
+++ b/.gitlab-ci.d/base.yml
@@ -0,0 +1,28 @@
+
+# The order of rules defined here is critically important.
+# They are evaluated in order and first match wins.
+#
+# Thus we group them into a number of stages, ordered from
+# most restrictive to least restrictive
+#
+.base_job_template:
+  rules:
+    #############################################################
+    # Stage 1: exclude scenarios where we definitely don't
+    # want jobs to run
+    #############################################################
+
+
+    #############################################################
+    # Stage 2: fine tune execution of jobs in specific scenarios
+    # where the catch all logic is inapprorpaite
+    #############################################################
+
+
+    #############################################################
+    # Stage 3: catch all logic applying to any job not matching
+    # an earlier criteria
+    #############################################################
+
+    # Jobs can run if any jobs they depend on were successfull
+    - when: on_success
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 871262fe0e..691d9bf5dc 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -2,6 +2,7 @@
 # https://gitlab.com/qemu-project/qemu/-/pipelines
 
 include:
+  - local: '/.gitlab-ci.d/base.yml'
   - local: '/.gitlab-ci.d/stages.yml'
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 92e25872aa..eb6a9e6122 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -28,7 +28,35 @@ For further information about how to set these variables, please refer to::
 
   https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
 
-Here is a list of the most used variables:
+Variable naming and grouping
+----------------------------
+
+The variables used by QEMU's CI configuration are grouped together
+in a handful of namespaces
+
+ * QEMU_JOB_nnnn - variables to be defined in individual jobs
+   or templates, to influence the shared rules defined in the
+   .base_job_template.
+
+ * QEMU_CI_nnn - variables to be set by contributors in their
+   repository CI settings, or as git push variables, to influence
+   which jobs get run in a pipeline
+
+ * nnn - other misc variables not falling into the above
+   categories, or using different names for historical reasons
+   and not yet converted.
+
+Maintainer controlled job variables
+-----------------------------------
+
+The following variables may be set when defining a job in the
+CI configuration file.
+
+Contributor controlled runtime variables
+----------------------------------------
+
+The following variables may be set by contributors to control
+job execution
 
 QEMU_CI_AVOCADO_TESTING
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -38,6 +66,12 @@ these artifacts are not already cached, downloading them make the jobs
 reach the timeout limit). Set this variable to have the tests using the
 Avocado framework run automatically.
 
+Other misc variables
+--------------------
+
+These variables are primarily to control execution of jobs on
+private runners
+
 AARCH64_RUNNER_AVAILABLE
 ~~~~~~~~~~~~~~~~~~~~~~~~
 If you've got access to an aarch64 host that can be used as a gitlab-CI
-- 
2.36.1

[PATCH 2/5] gitlab: convert Cirrus jobs to .base_job_template

[Daniel P. Berrangé]

This folds the Cirrus job rules into the base job
template, introducing two new variables

  - QEMU_JOB_CIRRUS - identifies the job as making
    use of Cirrus CI via cirrus-run

  - QEMU_JOB_OPTIONAL - identifies the job as one
    that is not run by default, primarily due to
    resource constraints. It can be manually invoked
    by users if they wish to validate that scenario.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml      |  9 +++++++++
 .gitlab-ci.d/cirrus.yml    | 16 ++++++++--------
 docs/devel/ci-jobs.rst.inc | 14 ++++++++++++++
 3 files changed, 31 insertions(+), 8 deletions(-)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 10eb6ab8bc..5734caf9fe 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -12,12 +12,21 @@
     # want jobs to run
     #############################################################
 
+    # Cirrus jobs can't run unless the creds / target repo are set
+    - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
+      when: never
+
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
     # where the catch all logic is inapprorpaite
     #############################################################
 
+    # Optional jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_OPTIONAL'
+      when: manual
+      allow_failure: true
+
 
     #############################################################
     # Stage 3: catch all logic applying to any job not matching
diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index b96b22e269..609c364308 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -11,6 +11,7 @@
 # special care, because we can't just override it at the GitLab CI job
 # definition level or we risk breaking it completely.
 .cirrus_build_job:
+  extends: .base_job_template
   stage: build
   image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
   needs: []
@@ -40,11 +41,8 @@
       <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
     - cat .gitlab-ci.d/cirrus/$NAME.yml
     - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
-  rules:
-    # Allow on 'staging' branch and 'stable-X.Y-staging' branches only
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/'
-      when: never
-    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
+  variables:
+    QEMU_JOB_CIRRUS: 1
 
 x64-freebsd-12-build:
   extends: .cirrus_build_job
@@ -90,11 +88,11 @@ x64-macos-11-base-build:
 
 # The following jobs run VM-based tests via KVM on a Linux-based Cirrus-CI job
 .cirrus_kvm_job:
+  extends: .base_job_template
   stage: build
   image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
   needs: []
   timeout: 80m
-  allow_failure: true
   script:
     - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
           -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
@@ -105,8 +103,10 @@ x64-macos-11-base-build:
       <.gitlab-ci.d/cirrus/kvm-build.yml >.gitlab-ci.d/cirrus/$NAME.yml
     - cat .gitlab-ci.d/cirrus/$NAME.yml
     - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
-  rules:
-    - when: manual
+  variables:
+    QEMU_JOB_CIRRUS: 1
+    QEMU_JOB_OPTIONAL: 1
+
 
 x86-netbsd:
   extends: .cirrus_kvm_job
diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index eb6a9e6122..a539f502da 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -52,6 +52,20 @@ Maintainer controlled job variables
 The following variables may be set when defining a job in the
 CI configuration file.
 
+QEMU_JOB_CIRRUS
+~~~~~~~~~~~~~~~
+
+The job makes use of Cirrus CI infrastructure, requiring the
+configuration setup for cirrus-run to be present in the repository
+
+QEMU_JOB_OPTIONAL
+~~~~~~~~~~~~~~~~~
+
+The job is expected to be successful in general, but is not run
+by default due to need to conserve limited CI resources. It is
+available to be started manually by the contributor in the CI
+pipelines UI.
+
 Contributor controlled runtime variables
 ----------------------------------------
 
-- 
2.36.1

[PATCH 3/5] gitlab: convert static checks to .base_job_template

[Daniel P. Berrangé]

This folds the static checks into using the base job
template rules, introducing one new variable

 - QEMU_JOB_ONLY_FORKS - a job that should never run
   on an upstream pipeline. The information it reports
   is only applicable to contributors in a pre-submission
   scenario, not time of merge.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml          |  4 ++++
 .gitlab-ci.d/static_checks.yml | 19 +++++++------------
 docs/devel/ci-jobs.rst.inc     |  7 +++++++
 3 files changed, 18 insertions(+), 12 deletions(-)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 5734caf9fe..9a0b8d7f97 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -16,6 +16,10 @@
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
       when: never
 
+    # Jobs only intended for forks should always be skipped on upstram
+    - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
+      when: never
+
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
index 94858e3272..289ad1359e 100644
--- a/.gitlab-ci.d/static_checks.yml
+++ b/.gitlab-ci.d/static_checks.yml
@@ -1,4 +1,5 @@
 check-patch:
+  extends: .base_job_template
   stage: build
   image: python:3.10-alpine
   needs: []
@@ -6,15 +7,13 @@ check-patch:
     - .gitlab-ci.d/check-patch.py
   variables:
     GIT_DEPTH: 1000
+    QEMU_JOB_ONLY_FORKS: 1
   before_script:
     - apk -U add git perl
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - when: on_success
-      allow_failure: true
+  allow_failure: true
 
 check-dco:
+  extends: .base_job_template
   stage: build
   image: python:3.10-alpine
   needs: []
@@ -23,12 +22,9 @@ check-dco:
     GIT_DEPTH: 1000
   before_script:
     - apk -U add git
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: never
-    - when: on_success
 
 check-python-pipenv:
+  extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
@@ -39,6 +35,7 @@ check-python-pipenv:
     job: python-container
 
 check-python-tox:
+  extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
@@ -46,8 +43,6 @@ check-python-tox:
   variables:
     GIT_DEPTH: 1
     QEMU_TOX_EXTRA_ARGS: --skip-missing-interpreters=false
+    QEMU_JOB_OPTIONAL: 1
   needs:
     job: python-container
-  rules:
-    - when: manual
-  allow_failure: true
diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index a539f502da..4c7e30ab08 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -66,6 +66,13 @@ by default due to need to conserve limited CI resources. It is
 available to be started manually by the contributor in the CI
 pipelines UI.
 
+QEMU_JOB_ONLY_FORKS
+~~~~~~~~~~~~~~~~~~~
+
+The job results are only of interest to contributors prior to
+submitting code. They are not required as part of the gating
+CI pipeline.
+
 Contributor controlled runtime variables
 ----------------------------------------
 
-- 
2.36.1

[PATCH 4/5] gitlab: convert build/container jobs to .base_job_template

[Daniel P. Berrangé]

This converts the main build and container jobs to use the
base job rules, defining the following new variables

 - QEMU_JOB_SKIPPED - jobs that are known to be currently
   broken and should not be run. Can still be manually
   launched if desired.

 - QEMU_JOB_AVOCADO - jobs that run the Avocado integration
   test harness.

 - QEMU_JOB_PUBLISH - jobs that publish content after the
   branch is merged upstream

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml                | 22 ++++++++++++++++++++++
 .gitlab-ci.d/buildtest-template.yml  | 16 ++++------------
 .gitlab-ci.d/buildtest.yml           | 28 +++++++++++++---------------
 .gitlab-ci.d/container-cross.yml     |  6 ++----
 .gitlab-ci.d/container-template.yml  |  1 +
 .gitlab-ci.d/crossbuild-template.yml |  3 +++
 .gitlab-ci.d/windows.yml             |  1 +
 docs/devel/ci-jobs.rst.inc           | 19 +++++++++++++++++++
 8 files changed, 65 insertions(+), 31 deletions(-)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 9a0b8d7f97..6a918abbda 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -16,10 +16,22 @@
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == "" || $CIRRUS_API_TOKEN == "")'
       when: never
 
+    # Publishing jobs should only run on the default branch in upstream
+    - if: '$QEMU_JOB_PUBLISH == "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH'
+      when: never
+
+    # Non-publishing jobs should only run on staging branches in upstream
+    - if: '$QEMU_JOB_PUBLISH != "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/'
+      when: never
+
     # Jobs only intended for forks should always be skipped on upstram
     - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
       when: never
 
+    # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
+    - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: never
+
 
     #############################################################
     # Stage 2: fine tune execution of jobs in specific scenarios
@@ -31,6 +43,16 @@
       when: manual
       allow_failure: true
 
+    # Skipped jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_SKIPPED'
+      when: manual
+      allow_failure: true
+
+    # Avocado jobs can be manually start in forks if $QEMU_CI_AVOCADO_TESTING is unset
+    - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: manual
+      allow_failure: true
+
 
     #############################################################
     # Stage 3: catch all logic applying to any job not matching
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index dc6d67aacf..1874c07a4b 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,4 +1,5 @@
 .native_build_job_template:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -27,6 +28,7 @@
       fi
 
 .common_test_job_template:
+  extends: .base_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -75,15 +77,5 @@
   after_script:
     - cd build
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
-  rules:
-    # Only run these jobs if running on the mainstream namespace,
-    # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either
-    # in its namespace setting or via git-push option, see documentation
-    # in /.gitlab-ci.yml of this repository).
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: on_success
-    - if: '$QEMU_CI_AVOCADO_TESTING'
-      when: on_success
-    # Otherwise, set to manual (the jobs are created but not run).
-    - when: manual
-      allow_failure: true
+  variables:
+    QEMU_JOB_AVOCADO: 1
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index e9620c3074..ecac3ec50c 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -360,12 +360,11 @@ build-cfi-aarch64:
     expire_in: 2 days
     paths:
       - build
-  rules:
+  variables:
     # FIXME: This job is often failing, likely due to out-of-memory problems in
     # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1
 
 check-cfi-aarch64:
   extends: .native_test_job_template
@@ -402,12 +401,11 @@ build-cfi-ppc64-s390x:
     expire_in: 2 days
     paths:
       - build
-  rules:
+  variables:
     # FIXME: This job is often failing, likely due to out-of-memory problems in
     # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
+    # skipped until the situation has been solved.
+    QEMU_JOB_SKIPPED: 1
 
 check-cfi-ppc64-s390x:
   extends: .native_test_job_template
@@ -579,6 +577,7 @@ build-without-default-features:
     MAKE_CHECK_ARGS: check-unit check-qtest SPEED=slow
 
 build-libvhost-user:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
   needs:
@@ -595,10 +594,13 @@ build-tools-and-docs-debian:
   extends: .native_build_job_template
   needs:
     job: amd64-debian-container
+    # when running on 'master' we use pre-existing container
+    optional: true
   variables:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
     CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
+    QEMU_JOB_PUBLISH: 1
   artifacts:
     expire_in: 2 days
     paths:
@@ -618,6 +620,7 @@ build-tools-and-docs-debian:
 # that users can see the results of their commits, regardless
 # of what topic branch they're currently using
 pages:
+  extends: .base_job_template
   image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
   stage: test
   needs:
@@ -635,10 +638,5 @@ pages:
   artifacts:
     paths:
       - public
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: on_success
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - if: '$CI_PROJECT_NAMESPACE != "qemu-project"'
-      when: on_success
+  variables:
+    QEMU_JOB_PUBLISH: 1
diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index e622ac2d21..90ecfd8da0 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -42,15 +42,13 @@ armhf-debian-cross-container:
 # We never want to build hexagon in the CI system and by default we
 # always want to refer to the master registry where it lives.
 hexagon-cross-container:
+  extends: .base_job_template
   image: docker:stable
   stage: containers
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - when: always
   variables:
     NAME: debian-hexagon-cross
     GIT_DEPTH: 1
+    QEMU_JOB_ONLY_FORKS: 1
   services:
     - docker:dind
   before_script:
diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index 1baecd9460..c434b9c8f3 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -1,4 +1,5 @@
 .container_job_template:
+  extends: .base_job_template
   image: docker:stable
   stage: containers
   services:
diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index 29c3c2b826..28b2142ec2 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -1,4 +1,5 @@
 .cross_system_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -24,6 +25,7 @@
 # KVM), and set extra options (such disabling other accelerators) via the
 # $EXTRA_CONFIGURE_OPTS variable.
 .cross_accel_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 30m
@@ -36,6 +38,7 @@
     - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
 
 .cross_user_build_job:
+  extends: .base_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index cf7724b8e5..1b2ede49e1 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -1,4 +1,5 @@
 .shared_msys2_builder:
+  extends: .base_job_template
   tags:
   - shared-windows
   - windows
diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 4c7e30ab08..0b4926e537 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -73,6 +73,25 @@ The job results are only of interest to contributors prior to
 submitting code. They are not required as part of the gating
 CI pipeline.
 
+QEMU_JOB_SKIPPED
+~~~~~~~~~~~~~~~~
+
+The job is not reliably successsful in general, so is not
+currently suitable to be run by default. Ideally this should
+be a temporary marker until the problems can be addressed, or
+the job permanently removed.
+
+QEMU_JOB_PUBLISH
+~~~~~~~~~~~~~~~~
+
+The job is for publishing content after a branch has been
+merged into the upstream default branch.
+
+QEMU_JOB_AVOCADO
+~~~~~~~~~~~~~~~~
+
+The job runs the Avocado integration test suite
+
 Contributor controlled runtime variables
 ----------------------------------------
 
-- 
2.36.1

[PATCH 5/5] gitlab: don't run CI jobs in forks by default

[Daniel P. Berrangé]

To preserve CI shared runner credits we don't want to run
pipelines on every push.

This sets up the config to that pipelines are never created
for contributors by default. To override this the QEMU_CI
variable can be set to a non-zero value. If set to 1, the
pipeline will be created but all jobs will remain manually
started. The contributor can selectively run jobs that they
care about. If set to 2, the pipeline will be created and
all jobs will immediately start.

This behavior can be controlled using push variables

  git push -o ci.variable=QEMU_CI=1

To make this more convenient define an alias

   git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
   git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"

Which lets you run

  git push-ci

to create the pipeline, or

  git push-ci-now

to create and run the pipeline

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml      |  9 +++++++++
 docs/devel/ci-jobs.rst.inc | 12 ++++++++++++
 2 files changed, 21 insertions(+)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 6a918abbda..62f2a850c3 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -28,6 +28,10 @@
     - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
       when: never
 
+    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: never
+
     # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
     - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
       when: never
@@ -59,5 +63,10 @@
     # an earlier criteria
     #############################################################
 
+    # Forks pipeline jobs don't start automatically unless
+    # QEMU_CI=2 is set
+    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: manual
+
     # Jobs can run if any jobs they depend on were successfull
     - when: on_success
diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 0b4926e537..e36c8cc68d 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -98,6 +98,18 @@ Contributor controlled runtime variables
 The following variables may be set by contributors to control
 job execution
 
+QEMU_CI
+~~~~~~~
+
+By default, no pipelines will be created on contributor forks
+in order to preserve CI credits
+
+Set this variable to 1 to create the pipelines, but leave all
+the jobs to be manually started from the UI
+
+Set this variable to 2 to create the pipelines and run all
+the jobs immediately, as was historicaly behaviour
+
 QEMU_CI_AVOCADO_TESTING
 ~~~~~~~~~~~~~~~~~~~~~~~
 By default, tests using the Avocado framework are not run automatically in
-- 
2.36.1

Re: [PATCH 5/5] gitlab: don't run CI jobs in forks by default

[Marc-André Lureau]

[-- Attachment #1: Type: text/plain, Size: 3548 bytes --]

Hi

On Thu, May 26, 2022 at 1:28 PM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> To preserve CI shared runner credits we don't want to run
> pipelines on every push.
>
> This sets up the config to that pipelines are never created
>

to/so ?


> for contributors by default. To override this the QEMU_CI
> variable can be set to a non-zero value. If set to 1, the
> pipeline will be created but all jobs will remain manually
> started. The contributor can selectively run jobs that they
> care about. If set to 2, the pipeline will be created and
> all jobs will immediately start.
>
> This behavior can be controlled using push variables
>
>   git push -o ci.variable=QEMU_CI=1
>
> To make this more convenient define an alias
>
>    git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
>    git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
>
> Which lets you run
>
>   git push-ci
>
> to create the pipeline, or
>
>   git push-ci-now
>
> to create and run the pipeline
>


Nice
Is there a place to put that in the devel documentation? Either in
submiting-a-patch.rst "Test your patches" (there is a broken link
Testing#Continuous_Integration here, wonder why sphinx doesn't catch it) or
somewhere in ci.rst documents perhaps.


> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/base.yml      |  9 +++++++++
>  docs/devel/ci-jobs.rst.inc | 12 ++++++++++++
>  2 files changed, 21 insertions(+)
>
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index 6a918abbda..62f2a850c3 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -28,6 +28,10 @@
>      - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE ==
> "qemu-project"'
>        when: never
>
> +    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
> +    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE !=
> "qemu-project"'
> +      when: never
> +
>      # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is
> set
>      - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" &&
> $CI_PROJECT_NAMESPACE != "qemu-project"'
>        when: never
> @@ -59,5 +63,10 @@
>      # an earlier criteria
>      #############################################################
>
> +    # Forks pipeline jobs don't start automatically unless
> +    # QEMU_CI=2 is set
> +    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
> +      when: manual
> +
>      # Jobs can run if any jobs they depend on were successfull
>      - when: on_success
> diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
> index 0b4926e537..e36c8cc68d 100644
> --- a/docs/devel/ci-jobs.rst.inc
> +++ b/docs/devel/ci-jobs.rst.inc
> @@ -98,6 +98,18 @@ Contributor controlled runtime variables
>  The following variables may be set by contributors to control
>  job execution
>
> +QEMU_CI
> +~~~~~~~
> +
> +By default, no pipelines will be created on contributor forks
> +in order to preserve CI credits
> +
> +Set this variable to 1 to create the pipelines, but leave all
> +the jobs to be manually started from the UI
> +
> +Set this variable to 2 to create the pipelines and run all
> +the jobs immediately, as was historicaly behaviour
> +
>  QEMU_CI_AVOCADO_TESTING
>  ~~~~~~~~~~~~~~~~~~~~~~~
>  By default, tests using the Avocado framework are not run automatically in
> --
> 2.36.1
>
>
>

-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 4798 bytes --]

Re: [PATCH 5/5] gitlab: don't run CI jobs in forks by default

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> To preserve CI shared runner credits we don't want to run
> pipelines on every push.
>
> This sets up the config to that pipelines are never created
> for contributors by default. To override this the QEMU_CI
> variable can be set to a non-zero value. If set to 1, the
> pipeline will be created but all jobs will remain manually
> started. The contributor can selectively run jobs that they
> care about. If set to 2, the pipeline will be created and
> all jobs will immediately start.
>
> This behavior can be controlled using push variables
>
>   git push -o ci.variable=QEMU_CI=1
>
> To make this more convenient define an alias
>
>    git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
>    git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
>
> Which lets you run
>
>   git push-ci
>
> to create the pipeline, or
>
>   git push-ci-now
>
> to create and run the pipeline

I think these useful tips on running the pipelines should go in the rst
so you don't have to grep the commit log to find them.

>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/base.yml      |  9 +++++++++
>  docs/devel/ci-jobs.rst.inc | 12 ++++++++++++
>  2 files changed, 21 insertions(+)
>
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index 6a918abbda..62f2a850c3 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -28,6 +28,10 @@
>      - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
>        when: never
>  
> +    # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
> +    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
> +      when: never
> +
>      # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
>      - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
>        when: never
> @@ -59,5 +63,10 @@
>      # an earlier criteria
>      #############################################################
>  
> +    # Forks pipeline jobs don't start automatically unless
> +    # QEMU_CI=2 is set
> +    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
> +      when: manual
> +
>      # Jobs can run if any jobs they depend on were successfull
>      - when: on_success
> diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
> index 0b4926e537..e36c8cc68d 100644
> --- a/docs/devel/ci-jobs.rst.inc
> +++ b/docs/devel/ci-jobs.rst.inc
> @@ -98,6 +98,18 @@ Contributor controlled runtime variables
>  The following variables may be set by contributors to control
>  job execution
>  
> +QEMU_CI
> +~~~~~~~
> +
> +By default, no pipelines will be created on contributor forks
> +in order to preserve CI credits
> +
> +Set this variable to 1 to create the pipelines, but leave all
> +the jobs to be manually started from the UI
> +
> +Set this variable to 2 to create the pipelines and run all
> +the jobs immediately, as was historicaly behaviour
> +
>  QEMU_CI_AVOCADO_TESTING
>  ~~~~~~~~~~~~~~~~~~~~~~~
>  By default, tests using the Avocado framework are not run automatically in


-- 
Alex Bennée

Re: [PATCH 0/5] gitlab: restrict running jobs in forks and upstream master

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> Currently on upstream most jobs will run in both staging
> and master. This is quite wasteful of CI credits. The only
> need to run in master is for the jobs related to publishing
> the website
>
> In forks we run jobs on every push. With restricted CI
> allowance this is quickly going to cause  problems.
>
> With this series jobs will no longer run on forks at all,
> without an opt-in with QEMU_CI=1 (pipeline with manual
> jobs) or QEMU_CI=2 (pipeline with immediate jobs)
>
> This is a rewrite of a previous proposal:
>
> https://lists.nongnu.org/archive/html/qemu-devel/2021-08/msg02104.html
>
> where I've kept it simpler and also split up the patches
> into more understandable chunks

Queued to testing/next, thanks.

I'll fix up the comment and move some stuff into the rst.

>
> Daniel P. Berrangé (5):
>   gitlab: introduce a common base job template
>   gitlab: convert Cirrus jobs to .base_job_template
>   gitlab: convert static checks to .base_job_template
>   gitlab: convert build/container jobs to .base_job_template
>   gitlab: don't run CI jobs in forks by default
>
>  .gitlab-ci.d/base.yml                | 72 +++++++++++++++++++++++
>  .gitlab-ci.d/buildtest-template.yml  | 16 ++---
>  .gitlab-ci.d/buildtest.yml           | 28 ++++-----
>  .gitlab-ci.d/cirrus.yml              | 16 ++---
>  .gitlab-ci.d/container-cross.yml     |  6 +-
>  .gitlab-ci.d/container-template.yml  |  1 +
>  .gitlab-ci.d/crossbuild-template.yml |  3 +
>  .gitlab-ci.d/qemu-project.yml        |  1 +
>  .gitlab-ci.d/static_checks.yml       | 19 +++---
>  .gitlab-ci.d/windows.yml             |  1 +
>  docs/devel/ci-jobs.rst.inc           | 88 +++++++++++++++++++++++++++-
>  11 files changed, 199 insertions(+), 52 deletions(-)
>  create mode 100644 .gitlab-ci.d/base.yml



-- 
Alex Bennée

Re: [PATCH 4/5] gitlab: convert build/container jobs to .base_job_template

[Thomas Huth]

On 26/05/2022 13.07, Daniel P. Berrangé wrote:
> This converts the main build and container jobs to use the
> base job rules, defining the following new variables
> 
>   - QEMU_JOB_SKIPPED - jobs that are known to be currently
>     broken and should not be run. Can still be manually
>     launched if desired.
> 
>   - QEMU_JOB_AVOCADO - jobs that run the Avocado integration
>     test harness.
> 
>   - QEMU_JOB_PUBLISH - jobs that publish content after the
>     branch is merged upstream
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/base.yml                | 22 ++++++++++++++++++++++
>   .gitlab-ci.d/buildtest-template.yml  | 16 ++++------------
>   .gitlab-ci.d/buildtest.yml           | 28 +++++++++++++---------------
>   .gitlab-ci.d/container-cross.yml     |  6 ++----
>   .gitlab-ci.d/container-template.yml  |  1 +
>   .gitlab-ci.d/crossbuild-template.yml |  3 +++
>   .gitlab-ci.d/windows.yml             |  1 +
>   docs/devel/ci-jobs.rst.inc           | 19 +++++++++++++++++++
>   8 files changed, 65 insertions(+), 31 deletions(-)
...
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index e9620c3074..ecac3ec50c 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -360,12 +360,11 @@ build-cfi-aarch64:
>       expire_in: 2 days
>       paths:
>         - build
> -  rules:
> +  variables:
>       # FIXME: This job is often failing, likely due to out-of-memory problems in
>       # the constrained containers of the shared runners. Thus this is marked as
> -    # manual until the situation has been solved.
> -    - when: manual
> -      allow_failure: true
> +    # skipped until the situation has been solved.
> +    QEMU_JOB_SKIPPED: 1
>   
>   check-cfi-aarch64:
>     extends: .native_test_job_template
> @@ -402,12 +401,11 @@ build-cfi-ppc64-s390x:
>       expire_in: 2 days
>       paths:
>         - build
> -  rules:
> +  variables:
>       # FIXME: This job is often failing, likely due to out-of-memory problems in
>       # the constrained containers of the shared runners. Thus this is marked as
> -    # manual until the situation has been solved.
> -    - when: manual
> -      allow_failure: true
> +    # skipped until the situation has been solved.
> +    QEMU_JOB_SKIPPED: 1

FYI, this patch broke the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs 
since they've now got two "variables:" sections and apparently only the 
second one is taken into account...

  Thomas

Re: [PATCH 4/5] gitlab: convert build/container jobs to .base_job_template

[Daniel P. Berrangé]

On Thu, Jun 02, 2022 at 07:41:59PM +0200, Thomas Huth wrote:
> On 26/05/2022 13.07, Daniel P. Berrangé wrote:
> > This converts the main build and container jobs to use the
> > base job rules, defining the following new variables
> > 
> >   - QEMU_JOB_SKIPPED - jobs that are known to be currently
> >     broken and should not be run. Can still be manually
> >     launched if desired.
> > 
> >   - QEMU_JOB_AVOCADO - jobs that run the Avocado integration
> >     test harness.
> > 
> >   - QEMU_JOB_PUBLISH - jobs that publish content after the
> >     branch is merged upstream
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/base.yml                | 22 ++++++++++++++++++++++
> >   .gitlab-ci.d/buildtest-template.yml  | 16 ++++------------
> >   .gitlab-ci.d/buildtest.yml           | 28 +++++++++++++---------------
> >   .gitlab-ci.d/container-cross.yml     |  6 ++----
> >   .gitlab-ci.d/container-template.yml  |  1 +
> >   .gitlab-ci.d/crossbuild-template.yml |  3 +++
> >   .gitlab-ci.d/windows.yml             |  1 +
> >   docs/devel/ci-jobs.rst.inc           | 19 +++++++++++++++++++
> >   8 files changed, 65 insertions(+), 31 deletions(-)
> ...
> > diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> > index e9620c3074..ecac3ec50c 100644
> > --- a/.gitlab-ci.d/buildtest.yml
> > +++ b/.gitlab-ci.d/buildtest.yml
> > @@ -360,12 +360,11 @@ build-cfi-aarch64:
> >       expire_in: 2 days
> >       paths:
> >         - build
> > -  rules:
> > +  variables:
> >       # FIXME: This job is often failing, likely due to out-of-memory problems in
> >       # the constrained containers of the shared runners. Thus this is marked as
> > -    # manual until the situation has been solved.
> > -    - when: manual
> > -      allow_failure: true
> > +    # skipped until the situation has been solved.
> > +    QEMU_JOB_SKIPPED: 1
> >   check-cfi-aarch64:
> >     extends: .native_test_job_template
> > @@ -402,12 +401,11 @@ build-cfi-ppc64-s390x:
> >       expire_in: 2 days
> >       paths:
> >         - build
> > -  rules:
> > +  variables:
> >       # FIXME: This job is often failing, likely due to out-of-memory problems in
> >       # the constrained containers of the shared runners. Thus this is marked as
> > -    # manual until the situation has been solved.
> > -    - when: manual
> > -      allow_failure: true
> > +    # skipped until the situation has been solved.
> > +    QEMU_JOB_SKIPPED: 1
> 
> FYI, this patch broke the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs
> since they've now got two "variables:" sections and apparently only the
> second one is taken into account...

Opps, sorry about that, and of course I didn't notice as it is a skipped
job and I forgot to try running it :-(

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|