[PATCH 0/2] gitlab: prepare for limited CI minutes by not running by default

[PATCH 0/2] gitlab: prepare for limited CI minutes by not running by default

[Daniel P. Berrangé]

A while ago GitLab announced it would be ending the free unlimited CI
minutes for public projects. It hasn't happened yet, but it is certainly
still on the radar. It is possible for OSS projects to get a greater
allowance, but it is still unclear how this will work for contributors'
forks. Even with the greater allowance, we'll burn through it in just
a few pipeline runs. People creating brand new gitlab accounts are
already getting hurt by the new limits IIUC.

This series isn't a full solution, but it gets us started in a more
sustainable direction. See the second patch commit message and the
docs that it adds for full details.

At a high level though, with this series applied, we no longer run
any CI jobs by default, except for the 'staging' branch in the
'qemu-project' namespace.

Users can manually start any individual job from the web UI still
though. Further, they can opt-in to various sets of jobs

 - 100% manual jobs (the new default):

    https://gitlab.com/berrange/qemu/-/pipelines/352393697

 - A minimal set of jobs

    https://gitlab.com/berrange/qemu/-/pipelines/352397093

 - A full set of jobs (the old default)

    https://gitlab.com/berrange/qemu/-/pipelines/352123582

 - A gating set of jobs (identical to what's run on staging)

    https://gitlab.com/berrange/qemu/-/pipelines/352195185

With this starting point, the common "rules" definition across
all jobs, makes it easy for us to add more refinements. We can
defining further sets of interesting jobs.

eg could define a set of cross-compilation, or a set for
non-Linux, or a set of Windows, or sets for particular host
architectures, etc.

Daniel P. Berrangé (2):
  docs: split the CI docs into two files
  gitlab: don't run CI jobs by default on push to user forks

 .gitlab-ci.d/buildtest-template.yml  |  17 +--
 .gitlab-ci.d/buildtest.yml           |  43 +++----
 .gitlab-ci.d/cirrus.yml              |   6 +-
 .gitlab-ci.d/container-core.yml      |   4 +
 .gitlab-ci.d/container-cross.yml     |   9 +-
 .gitlab-ci.d/container-template.yml  |   1 +
 .gitlab-ci.d/containers.yml          |   4 +
 .gitlab-ci.d/crossbuild-template.yml |   3 +
 .gitlab-ci.d/crossbuilds.yml         |   3 +
 .gitlab-ci.d/qemu-project.yml        |   1 +
 .gitlab-ci.d/rules.yml               | 116 +++++++++++++++++++
 .gitlab-ci.d/static_checks.yml       |  18 +--
 .gitlab-ci.yml                       |   4 -
 docs/devel/ci-jobs.rst               | 161 +++++++++++++++++++++++++++
 docs/devel/ci-runners.rst            | 117 +++++++++++++++++++
 docs/devel/ci.rst                    | 159 +-------------------------
 16 files changed, 459 insertions(+), 207 deletions(-)
 create mode 100644 .gitlab-ci.d/rules.yml
 create mode 100644 docs/devel/ci-jobs.rst
 create mode 100644 docs/devel/ci-runners.rst

-- 
2.31.1

[PATCH 1/2] docs: split the CI docs into two files

[Daniel P. Berrangé]

This splits the CI docs into one file talking about job setup and usage
and another file describing provisioning of custom runners.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/devel/ci-jobs.rst    |  40 ++++++++++
 docs/devel/ci-runners.rst | 117 ++++++++++++++++++++++++++++
 docs/devel/ci.rst         | 159 +-------------------------------------
 3 files changed, 159 insertions(+), 157 deletions(-)
 create mode 100644 docs/devel/ci-jobs.rst
 create mode 100644 docs/devel/ci-runners.rst

diff --git a/docs/devel/ci-jobs.rst b/docs/devel/ci-jobs.rst
new file mode 100644
index 0000000000..9cd9819786
--- /dev/null
+++ b/docs/devel/ci-jobs.rst
@@ -0,0 +1,40 @@
+Custom CI/CD variables
+======================
+
+QEMU CI pipelines can be tuned by setting some CI environment variables.
+
+Set variable globally in the user's CI namespace
+------------------------------------------------
+
+Variables can be set globally in the user's CI namespace setting.
+
+For further information about how to set these variables, please refer to::
+
+  https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
+
+Set variable manually when pushing a branch or tag to the user's repository
+---------------------------------------------------------------------------
+
+Variables can be set manually when pushing a branch or tag, using
+git-push command line arguments.
+
+Example setting the QEMU_CI_EXAMPLE_VAR variable:
+
+.. code::
+
+   git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch
+
+For further information about how to set these variables, please refer to::
+
+  https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
+
+Here is a list of the most used variables:
+
+QEMU_CI_AVOCADO_TESTING
+~~~~~~~~~~~~~~~~~~~~~~~
+By default, tests using the Avocado framework are not run automatically in
+the pipelines (because multiple artifacts have to be downloaded, and if
+these artifacts are not already cached, downloading them make the jobs
+reach the timeout limit). Set this variable to have the tests using the
+Avocado framework run automatically.
+
diff --git a/docs/devel/ci-runners.rst b/docs/devel/ci-runners.rst
new file mode 100644
index 0000000000..7817001fb2
--- /dev/null
+++ b/docs/devel/ci-runners.rst
@@ -0,0 +1,117 @@
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner".
+
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
+
+Custom runners entail custom machines.  To see a list of the machines
+currently deployed in the QEMU GitLab CI and their maintainers, please
+refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  This playbook consists on the installation of various
+required packages (and a general package update while at it).  It
+currently covers a number of different Linux distributions, but it can
+be expanded to cover other systems.
+
+The minimum required version of Ansible successfully tested in this
+playbook is 2.8.0 (a version check is embedded within the playbook
+itself).  To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
+
+Please note that most of the tasks in the playbook require superuser
+privileges, such as those from the ``root`` account or those obtained
+by ``sudo``.  If necessary, please refer to ``ansible-playbook``
+options such as ``--become``, ``--become-method``, ``--become-user``
+and ``--ask-become-pass``.
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears-like icon at the bottom of the left hand side
+   vertical toolbar), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "And this registration token:"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Tags are very important as they are used to route specific jobs to
+specific types of runners, so it's a good idea to double check that
+the automatically created tags are consistent with the OS and
+architecture.  For instance, an Ubuntu 20.04 aarch64 system should
+have tags set as::
+
+  ubuntu_20.04,aarch64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  ubuntu-20.04-aarch64-all:
+   tags:
+   - ubuntu_20.04
+   - aarch64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * give it a better Description
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 205572510c..a6a650968b 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -8,160 +8,5 @@ found at::
 
    https://wiki.qemu.org/Testing/CI
 
-Custom CI/CD variables
-======================
-
-QEMU CI pipelines can be tuned by setting some CI environment variables.
-
-Set variable globally in the user's CI namespace
-------------------------------------------------
-
-Variables can be set globally in the user's CI namespace setting.
-
-For further information about how to set these variables, please refer to::
-
-  https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
-
-Set variable manually when pushing a branch or tag to the user's repository
----------------------------------------------------------------------------
-
-Variables can be set manually when pushing a branch or tag, using
-git-push command line arguments.
-
-Example setting the QEMU_CI_EXAMPLE_VAR variable:
-
-.. code::
-
-   git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch
-
-For further information about how to set these variables, please refer to::
-
-  https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
-
-Here is a list of the most used variables:
-
-QEMU_CI_AVOCADO_TESTING
-~~~~~~~~~~~~~~~~~~~~~~~
-By default, tests using the Avocado framework are not run automatically in
-the pipelines (because multiple artifacts have to be downloaded, and if
-these artifacts are not already cached, downloading them make the jobs
-reach the timeout limit). Set this variable to have the tests using the
-Avocado framework run automatically.
-
-Jobs on Custom Runners
-======================
-
-Besides the jobs run under the various CI systems listed before, there
-are a number additional jobs that will run before an actual merge.
-These use the same GitLab CI's service/framework already used for all
-other GitLab based CI jobs, but rely on additional systems, not the
-ones provided by GitLab as "shared runners".
-
-The architecture of GitLab's CI service allows different machines to
-be set up with GitLab's "agent", called gitlab-runner, which will take
-care of running jobs created by events such as a push to a branch.
-Here, the combination of a machine, properly configured with GitLab's
-gitlab-runner, is called a "custom runner".
-
-The GitLab CI jobs definition for the custom runners are located under::
-
-  .gitlab-ci.d/custom-runners.yml
-
-Custom runners entail custom machines.  To see a list of the machines
-currently deployed in the QEMU GitLab CI and their maintainers, please
-refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
-
-Machine Setup Howto
--------------------
-
-For all Linux based systems, the setup can be mostly automated by the
-execution of two Ansible playbooks.  Create an ``inventory`` file
-under ``scripts/ci/setup``, such as this::
-
-  fully.qualified.domain
-  other.machine.hostname
-
-You may need to set some variables in the inventory file itself.  One
-very common need is to tell Ansible to use a Python 3 interpreter on
-those hosts.  This would look like::
-
-  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
-  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
-
-Build environment
-~~~~~~~~~~~~~~~~~
-
-The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
-set up machines with the environment needed to perform builds and run
-QEMU tests.  This playbook consists on the installation of various
-required packages (and a general package update while at it).  It
-currently covers a number of different Linux distributions, but it can
-be expanded to cover other systems.
-
-The minimum required version of Ansible successfully tested in this
-playbook is 2.8.0 (a version check is embedded within the playbook
-itself).  To run the playbook, execute::
-
-  cd scripts/ci/setup
-  ansible-playbook -i inventory build-environment.yml
-
-Please note that most of the tasks in the playbook require superuser
-privileges, such as those from the ``root`` account or those obtained
-by ``sudo``.  If necessary, please refer to ``ansible-playbook``
-options such as ``--become``, ``--become-method``, ``--become-user``
-and ``--ask-become-pass``.
-
-gitlab-runner setup and registration
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The gitlab-runner agent needs to be installed on each machine that
-will run jobs.  The association between a machine and a GitLab project
-happens with a registration token.  To find the registration token for
-your repository/project, navigate on GitLab's web UI to:
-
- * Settings (the gears-like icon at the bottom of the left hand side
-   vertical toolbar), then
- * CI/CD, then
- * Runners, and click on the "Expand" button, then
- * Under "Set up a specific Runner manually", look for the value under
-   "And this registration token:"
-
-Copy the ``scripts/ci/setup/vars.yml.template`` file to
-``scripts/ci/setup/vars.yml``.  Then, set the
-``gitlab_runner_registration_token`` variable to the value obtained
-earlier.
-
-To run the playbook, execute::
-
-  cd scripts/ci/setup
-  ansible-playbook -i inventory gitlab-runner.yml
-
-Following the registration, it's necessary to configure the runner tags,
-and optionally other configurations on the GitLab UI.  Navigate to:
-
- * Settings (the gears like icon), then
- * CI/CD, then
- * Runners, and click on the "Expand" button, then
- * "Runners activated for this project", then
- * Click on the "Edit" icon (next to the "Lock" Icon)
-
-Tags are very important as they are used to route specific jobs to
-specific types of runners, so it's a good idea to double check that
-the automatically created tags are consistent with the OS and
-architecture.  For instance, an Ubuntu 20.04 aarch64 system should
-have tags set as::
-
-  ubuntu_20.04,aarch64
-
-Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
-would contain::
-
-  ubuntu-20.04-aarch64-all:
-   tags:
-   - ubuntu_20.04
-   - aarch64
-
-It's also recommended to:
-
- * increase the "Maximum job timeout" to something like ``2h``
- * give it a better Description
+.. include:: ci-jobs.rst
+.. include:: ci-runners.rst
-- 
2.31.1

Re: [PATCH 1/2] docs: split the CI docs into two files

[Philippe Mathieu-Daudé]

On 8/12/21 8:04 PM, Daniel P. Berrangé wrote:
> This splits the CI docs into one file talking about job setup and usage
> and another file describing provisioning of custom runners.

Thanks.

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/ci-jobs.rst    |  40 ++++++++++
>  docs/devel/ci-runners.rst | 117 ++++++++++++++++++++++++++++
>  docs/devel/ci.rst         | 159 +-------------------------------------
>  3 files changed, 159 insertions(+), 157 deletions(-)
>  create mode 100644 docs/devel/ci-jobs.rst
>  create mode 100644 docs/devel/ci-runners.rst

Re: [PATCH 1/2] docs: split the CI docs into two files

[Willian Rampazzo]

On Thu, Aug 12, 2021 at 3:04 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> This splits the CI docs into one file talking about job setup and usage
> and another file describing provisioning of custom runners.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/ci-jobs.rst    |  40 ++++++++++
>  docs/devel/ci-runners.rst | 117 ++++++++++++++++++++++++++++
>  docs/devel/ci.rst         | 159 +-------------------------------------
>  3 files changed, 159 insertions(+), 157 deletions(-)
>  create mode 100644 docs/devel/ci-jobs.rst
>  create mode 100644 docs/devel/ci-runners.rst
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

[PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Daniel P. Berrangé]

Currently pushes to user forks run the same set of build / test jobs as
pushes to the main repo. This results in creation of 120+ individual
jobs. While it is useful for subsystem maintainers, and even regular
contributors to be able to run the full set of jobs, it is wasteful to
run all of them all the time.

In addition, with the expected change in GitLab to set a finite CI
minute allowance on all users we need to be much more conservative.

This patch thus sets up rules so that no CI jobs will run by default
on push, except

 - Pushes to branch name prefix 'staging' on 'qemu-project' repo
   run full set of jobs
 - Pushes to 'master' branch, only run jobs needed to publish
   the website.

In any pushes to users forks, CI is now strictly opt-in. The pipeline
will always be created by every singe job will be marked manual. You
can then use the web UI to start individual jobs.

For more convenience there are three levels of increasing comprehensive
CI job sets that can be chosen between

 - Gating - run exactly the same as pushes to 'staging' branch
            (except for jobs needing custom runners)

   Push to 'ci-gating-xxx' branch, or set env QEMU_CI_GATING=1

 - Full - same as 'Gating', except that acceptance tests don't
          get run. This is equivalent to historical CI behaviour
	  for pushes to user forks.

   Push to 'ci-full-xxx' branch, or set env QEMU_CI_FULL=1

 - Minimal - a significantly cut down set of jobs to get a
             decent sanity check of builds without burning
	     massive amounts of CI time.

   Push to 'ci-min-xxx' branch, or set env QEMU_CI=1

The minimal job set covers:

  * Fedora, CentOS, Ubuntu & Debian system emulator builds
    to cover all arch targets.
  * Linux user static build
  * Cross build i386 (to identify any 32-bit build issues)
  * Cross build s390x (to identify any big endian build issues)
  * Containers needed for the above
  * Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
  * Simple checks - python unittests, DCO check, checkpatch.pl, etc

This gives about 30 jobs instead of 120 from the "Full" group. It
is likely reasonable to cut the minimal set down even more, as IMHO
there are too many system emulator jobs there.

This is all controlled by defining a set of 'rules' entries that
apply to (almost) all jobs in the CI pipeline. The exceptions are
those using custom runners, and the firmware build jobs.

Using inherited templates together with rules, doesn't work in
the way you might expect. Any rules defined in the child completely
replace those from the parent, rather than augmenting them. Thus we
need to avoid having rules defined in any jobs at all, by inheriting
from a parent job template that contains the rules. The characteristics
of individual jobs are then defined by variables set against the job.
This is described in the updated content for the docs/devel/ci-jobs.rst
file.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/buildtest-template.yml  |  17 +--
 .gitlab-ci.d/buildtest.yml           |  43 ++++----
 .gitlab-ci.d/cirrus.yml              |   6 +-
 .gitlab-ci.d/container-core.yml      |   4 +
 .gitlab-ci.d/container-cross.yml     |   9 +-
 .gitlab-ci.d/container-template.yml  |   1 +
 .gitlab-ci.d/containers.yml          |   4 +
 .gitlab-ci.d/crossbuild-template.yml |   3 +
 .gitlab-ci.d/crossbuilds.yml         |   3 +
 .gitlab-ci.d/qemu-project.yml        |   1 +
 .gitlab-ci.d/rules.yml               | 116 ++++++++++++++++++++
 .gitlab-ci.d/static_checks.yml       |  18 ++--
 .gitlab-ci.yml                       |   4 -
 docs/devel/ci-jobs.rst               | 155 ++++++++++++++++++++++++---
 14 files changed, 317 insertions(+), 67 deletions(-)
 create mode 100644 .gitlab-ci.d/rules.yml

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index fcbcc4e627..8578b3ed11 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,4 +1,6 @@
+
 .native_build_job_template:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -27,6 +29,7 @@
       fi
 
 .native_test_job_template:
+  extends: .job_rules
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -67,15 +70,5 @@
   after_script:
     - cd build
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
-  rules:
-    # Only run these jobs if running on the mainstream namespace,
-    # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either
-    # in its namespace setting or via git-push option, see documentation
-    # in /.gitlab-ci.yml of this repository).
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: on_success
-    - if: '$QEMU_CI_AVOCADO_TESTING'
-      when: on_success
-    # Otherwise, set to manual (the jobs are created but not run).
-    - when: manual
-      allow_failure: true
+  variables:
+    QEMU_JOB_GATING: 1
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 903ee65f32..c8e03373c5 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -40,6 +40,7 @@ build-system-ubuntu:
   needs:
     job: amd64-ubuntu2004-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: ubuntu2004
     CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-slirp=system
     TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
@@ -56,6 +57,7 @@ check-system-ubuntu:
     - job: build-system-ubuntu
       artifacts: true
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check
 
@@ -73,6 +75,7 @@ build-system-debian:
   needs:
     job: amd64-debian-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: debian-amd64
     CONFIGURE_ARGS: --enable-fdt=system
     TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
@@ -89,6 +92,7 @@ check-system-debian:
     - job: build-system-debian
       artifacts: true
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check
 
@@ -106,6 +110,7 @@ build-system-fedora:
   needs:
     job: amd64-fedora-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: fedora
     CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs
              --enable-fdt=system --enable-slirp=system --enable-capstone=system
@@ -123,6 +128,7 @@ check-system-fedora:
     - job: build-system-fedora
       artifacts: true
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
@@ -140,6 +146,7 @@ build-system-centos:
   needs:
     job: amd64-centos8-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: centos8
     CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
                     --enable-modules --enable-trace-backends=dtrace
@@ -157,6 +164,7 @@ check-system-centos:
     - job: build-system-centos
       artifacts: true
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: centos8
     MAKE_CHECK_ARGS: check
 
@@ -325,6 +333,7 @@ build-user-static:
   needs:
     job: amd64-debian-user-cross-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: debian-all-test-cross
     CONFIGURE_ARGS: --disable-tools --disable-system --static
     MAKE_CHECK_ARGS: check-tcg
@@ -411,17 +420,15 @@ build-cfi-aarch64:
       --enable-safe-stack --enable-slirp=git
     TARGETS: aarch64-softmmu
     MAKE_CHECK_ARGS: check-build
+    # FIXME: This job is often failing, likely due to out-of-memory problems in
+    # the constrained containers of the shared runners. Thus this is marked as
+    # manual until the situation has been solved.
+    QEMU_JOB_MANUAL: 1
   timeout: 70m
   artifacts:
     expire_in: 2 days
     paths:
       - build
-  rules:
-    # FIXME: This job is often failing, likely due to out-of-memory problems in
-    # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
 
 check-cfi-aarch64:
   extends: .native_test_job_template
@@ -453,17 +460,15 @@ build-cfi-ppc64-s390x:
       --enable-safe-stack --enable-slirp=git
     TARGETS: ppc64-softmmu s390x-softmmu
     MAKE_CHECK_ARGS: check-build
+    # FIXME: This job is often failing, likely due to out-of-memory problems in
+    # the constrained containers of the shared runners. Thus this is marked as
+    # manual until the situation has been solved.
+    QEMU_JOB_MANUAL: 1
   timeout: 70m
   artifacts:
     expire_in: 2 days
     paths:
       - build
-  rules:
-    # FIXME: This job is often failing, likely due to out-of-memory problems in
-    # the constrained containers of the shared runners. Thus this is marked as
-    # manual until the situation has been solved.
-    - when: manual
-      allow_failure: true
 
 check-cfi-ppc64-s390x:
   extends: .native_test_job_template
@@ -657,6 +662,7 @@ build-without-default-features:
     MAKE_CHECK_ARGS: check-unit
 
 build-libvhost-user:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
   needs:
@@ -676,6 +682,8 @@ build-tools-and-docs-debian:
   needs:
     job: amd64-debian-container
   variables:
+    QEMU_JOB_MINIMAL: 1
+    QEMU_JOB_PUBLISH: 1
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
     CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
@@ -698,10 +706,14 @@ build-tools-and-docs-debian:
 # that users can see the results of their commits, regardless
 # of what topic branch they're currently using
 pages:
+  extends: .native_build_job_template
   image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
   stage: test
   needs:
     - job: build-tools-and-docs-debian
+  variables:
+    QEMU_JOB_MINIMAL: 1
+    QEMU_JOB_PUBLISH: 1
   script:
     - mkdir -p public
     # HTML-ised source tree
@@ -715,10 +727,3 @@ pages:
   artifacts:
     paths:
       - public
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: on_success
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - if: '$CI_PROJECT_NAMESPACE != "qemu-project"'
-      when: on_success
diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index 675db69622..4334736796 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -11,6 +11,7 @@
 # special care, because we can't just override it at the GitLab CI job
 # definition level or we risk breaking it completely.
 .cirrus_build_job:
+  extends: .job_rules
   stage: build
   image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
   needs: []
@@ -39,8 +40,6 @@
       <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
     - cat .gitlab-ci.d/cirrus/$NAME.yml
     - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
-  rules:
-    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
 
 x64-freebsd-12-build:
   extends: .cirrus_build_job
@@ -61,6 +60,8 @@ x64-freebsd-12-build:
 x64-freebsd-13-build:
   extends: .cirrus_build_job
   variables:
+    QEMU_JOB_SOFT_FAIL: 1
+    QEMU_JOB_MINIMAL: 1
     NAME: freebsd-13
     CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
     CIRRUS_VM_IMAGE_SELECTOR: image_family
@@ -74,6 +75,7 @@ x64-freebsd-13-build:
 x64-macos-11-base-build:
   extends: .cirrus_build_job
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: macos-11
     CIRRUS_VM_INSTANCE_TYPE: osx_instance
     CIRRUS_VM_IMAGE_SELECTOR: image
diff --git a/.gitlab-ci.d/container-core.yml b/.gitlab-ci.d/container-core.yml
index e8dd1f476a..8983c32b3b 100644
--- a/.gitlab-ci.d/container-core.yml
+++ b/.gitlab-ci.d/container-core.yml
@@ -4,14 +4,18 @@ include:
 amd64-centos8-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: centos8
 
 amd64-fedora-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: fedora
 
 amd64-debian10-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
+    QEMU_JOB_PUBLISH: 1
     NAME: debian10
diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index 0fcebe363a..ef316668eb 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -50,13 +50,11 @@ armhf-debian-cross-container:
 # We never want to build hexagon in the CI system and by default we
 # always want to refer to the master registry where it lives.
 hexagon-cross-container:
+  extends: .job_rules
   image: docker:stable
   stage: containers
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
-      when: never
-    - when: always
   variables:
+    QEMU_JOB_MANUAL: 1
     NAME: debian-hexagon-cross
     GIT_DEPTH: 1
   services:
@@ -143,6 +141,7 @@ s390x-debian-cross-container:
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: debian-s390x-cross
 
 sh4-debian-cross-container:
@@ -179,11 +178,13 @@ cris-fedora-cross-container:
 i386-fedora-cross-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: fedora-i386-cross
 
 win32-fedora-cross-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: fedora-win32-cross
 
 win64-fedora-cross-container:
diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index 1baecd9460..e2954e4ed3 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -1,4 +1,5 @@
 .container_job_template:
+  extends: .job_rules
   image: docker:stable
   stage: containers
   services:
diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index cd06d3f5f4..6948923892 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -17,6 +17,8 @@ amd64-debian-container:
   stage: containers-layer2
   needs: ['amd64-debian10-container']
   variables:
+    QEMU_JOB_MINIMAL: 1
+    QEMU_JOB_PUBLISH: 1
     NAME: debian-amd64
 
 amd64-ubuntu1804-container:
@@ -27,6 +29,7 @@ amd64-ubuntu1804-container:
 amd64-ubuntu2004-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: ubuntu2004
 
 amd64-ubuntu-container:
@@ -42,4 +45,5 @@ amd64-opensuse-leap-container:
 python-container:
   extends: .container_job_template
   variables:
+    QEMU_JOB_MINIMAL: 1
     NAME: python
diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index 7d3ad00a1e..997ec02f79 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -1,4 +1,5 @@
 .cross_system_build_job:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -23,6 +24,7 @@
 # KVM), and set extra options (such disabling other accelerators) via the
 # $EXTRA_CONFIGURE_OPTS variable.
 .cross_accel_build_job:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 30m
@@ -35,6 +37,7 @@
     - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
 
 .cross_user_build_job:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 4ff3aa3cfc..9fcd2ac567 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -48,6 +48,7 @@ cross-i386-system:
   needs:
     job: i386-fedora-cross-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: fedora-i386-cross
     MAKE_CHECK_ARGS: check-qtest
 
@@ -129,6 +130,7 @@ cross-s390x-system:
   needs:
     job: s390x-debian-cross-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: debian-s390x-cross
 
 cross-s390x-user:
@@ -159,6 +161,7 @@ cross-win32-system:
   needs:
     job: win32-fedora-cross-container
   variables:
+    QEMU_JOB_MINIMAL: 1
     IMAGE: fedora-win32-cross
   artifacts:
     paths:
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index b3d79bc429..7007193fc0 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -2,6 +2,7 @@
 # https://gitlab.com/qemu-project/qemu/-/pipelines
 
 include:
+  - local: '/.gitlab-ci.d/rules.yml'
   - local: '/.gitlab-ci.d/stages.yml'
   - local: '/.gitlab-ci.d/edk2.yml'
   - local: '/.gitlab-ci.d/opensbi.yml'
diff --git a/.gitlab-ci.d/rules.yml b/.gitlab-ci.d/rules.yml
new file mode 100644
index 0000000000..3399722ca9
--- /dev/null
+++ b/.gitlab-ci.d/rules.yml
@@ -0,0 +1,116 @@
+
+# This defines rules used to control individual job execution
+# See docs/devel/ci-jobs.rst for an explanation of the various
+# variables and branch naming conventions applied here.
+#
+.job_rules:
+  rules:
+    # ======================================================================
+    # Rules that apply regardless of whether the primary qemu repo or a fork
+    # ======================================================================
+
+    # Skip any cirrus job if either repo or api token are missing
+    # as we won't be able to talk to cirrus
+    - if: '$CIRRUS_VM_INSTANCE_TYPE && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
+      when: never
+
+    # Any jobs marked as manual, never get automatically run in any scenario
+    # and don't contribute to pipeline status
+    - if: '$QEMU_JOB_MANUAL'
+      when: manual
+      allow_failure: true
+
+    # For the main repo, tags represent official releases.
+    # The branch associated with the release will have passed
+    # a CI pipeline already
+    #
+    # For user forks, tags are tyically added by tools like
+    # git-publish at the same time as pushing the branch prior
+    # to sending out for review
+    #
+    # In neither case do we wish to run CI pipelines for tags
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_TAG'
+      when: never
+
+
+    # ==================================================
+    # Rules to control what happens in primary qemu repo
+    # ==================================================
+
+    # The 'pages' job must never run on a non-default branch as we don't
+    # want to publish the website from pre-merged content
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_JOB_NAME == "pages"'
+      when: never
+
+    # Run all jobs for main repo staging branch
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == "staging"'
+      when: on_success
+
+    # Run jobs needed to support website publishing on the main branch
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $QEMU_JOB_PUBLISH'
+      when: on_success
+
+    # Prevent jobs from running on pushes to any other branch / tag
+    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != "staging"'
+      when: never
+
+
+    # Remainining pipeline sources follow the same rules as for fork repos
+
+
+
+    # ====================================
+    # Rules for running jobs in user forks
+    # ====================================
+
+    # Part 1: gating jobs
+    # -------------------
+
+    # If on a branch with name prefix 'ci-acceptance-', then run
+    # everything, just as a gating job on 'staging' branch would
+    - if: '$CI_COMMIT_BRANCH =~ /^ci-gating-/'
+      when: on_success
+
+    # If user set QEMU_CI_GATING=1, then run everything just as
+    # a gating job on 'staging' branch would
+    - if: '$QEMU_CI_GATING'
+      when: on_success
+
+    # Otherwise never run jobs marked as gating, but allow manual trigger
+    # without affecting pipeline status
+    - if: '$QEMU_JOB_GATING'
+      when: manual
+      allow_failure: true
+
+
+    # =============================================
+    # Part 2: opt-in for all CI, except gating jobs
+    # =============================================
+
+    # If pushing to a branch with name prefix 'ci-all', then run all jobs
+    - if: '$CI_COMMIT_BRANCH =~ /^ci-all/'
+      when: on_success
+
+    # If user QEMU_CI_ALL=1, then run all jobs
+    - if: '$QEMU_CI_ALL'
+      when: on_success
+
+
+    # ===============================
+    # Part 3: the minimal set of jobs
+    # ===============================
+
+    # If pushing to a branch with name prefix 'ci-min', then run
+    # only a minimal set of jobs
+    - if: '$CI_COMMIT_BRANCH =~ /^ci-min/ && $QEMU_JOB_MINIMAL'
+      when: on_success
+
+    # If user set QEMU_CI=1, then run only a minimal set of jobs
+    - if: '$QEMU_CI && $QEMU_JOB_MINIMAL'
+      when: on_success
+
+    # Any other scenario, let the jobs be manually triggered only.
+    # Set to be non-fatal, so that pending manual jobs don't
+    # affect overall pipeline status
+    - when: manual
+      allow_failure: true
diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
index 96dbd9e310..4ba3e038dd 100644
--- a/.gitlab-ci.d/static_checks.yml
+++ b/.gitlab-ci.d/static_checks.yml
@@ -1,4 +1,5 @@
 check-patch:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
   needs:
@@ -6,42 +7,41 @@ check-patch:
   script:
     - .gitlab-ci.d/check-patch.py
   variables:
+    QEMU_JOB_MINIMAL: 1
     GIT_DEPTH: 1000
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: never
-    - when: on_success
-      allow_failure: true
+  allow_failure: true
 
 check-dco:
+  extends: .job_rules
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
   needs:
     job: amd64-centos8-container
   script: .gitlab-ci.d/check-dco.py
   variables:
+    QEMU_JOB_MINIMAL: 1
     GIT_DEPTH: 1000
-  rules:
-    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
-      when: never
-    - when: on_success
 
 check-python-pipenv:
+  extends: .job_rules
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
     - make -C python check-pipenv
   variables:
+    QEMU_JOB_MINIMAL: 1
     GIT_DEPTH: 1
   needs:
     job: python-container
 
 check-python-tox:
+  extends: .job_rules
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/python:latest
   script:
     - make -C python check-tox
   variables:
+    QEMU_JOB_MINIMAL: 1
     GIT_DEPTH: 1
     QEMU_TOX_EXTRA_ARGS: --skip-missing-interpreters=false
   needs:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9762dda2ee..d5bd31f294 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -15,10 +15,6 @@
 #
 # QEMU CI jobs are based on templates. Some templates provide
 # user-configurable options, modifiable via configuration variables.
-#
-# See https://qemu-project.gitlab.io/qemu/devel/ci.html#custom-ci-cd-variables
-# for more information.
-#
 
 include:
   - local: '/.gitlab-ci.d/qemu-project.yml'
diff --git a/docs/devel/ci-jobs.rst b/docs/devel/ci-jobs.rst
index 9cd9819786..6d3f13d495 100644
--- a/docs/devel/ci-jobs.rst
+++ b/docs/devel/ci-jobs.rst
@@ -1,22 +1,49 @@
-Custom CI/CD variables
-======================
+CI Job Rules
+============
 
-QEMU CI pipelines can be tuned by setting some CI environment variables.
+The following documents how CI rules control execution of jobs in a pipeline
 
-Set variable globally in the user's CI namespace
-------------------------------------------------
+Job variable naming
+-------------------
 
-Variables can be set globally in the user's CI namespace setting.
+The rules for controlling jobs in a pipeline will be driven by a number of
+variables:
+
+ - ``CI_XXX``, ``GITLAB_XXX`` - variables defined by GitLab:
+
+   https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
+
+ - ``QEMU_JOB_XXX`` - variables defined in the YAML files that express
+   characteristics of the job used to control default behaviour
+
+ - ``QEMU_CI_XXX`` - variables defined by the user that are used to fine
+   tune which jobs are run dynamically
+
+and in some cases based on the branch name prefixes.
+
+Job fine tuning strategies
+--------------------------
+
+Using a combination of the ``QEMU_CI_XXX`` variables and ``ci-``
+branch name prefix, users can fine tune what jobs are run.
+
+Set variable globally in the gitlab repository CI config
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Variables can be set globally in the user's gitlab repository CI config.
+These variables will apply to all pipelines associated with the repository
+thereafter. This is useful for fine tuning the jobs indefinitely.
 
 For further information about how to set these variables, please refer to::
 
   https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
 
-Set variable manually when pushing a branch or tag to the user's repository
----------------------------------------------------------------------------
+Set variable manually when pushing to git
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Variables can be set manually when pushing a branch or tag, using
-git-push command line arguments.
+git-push command line arguments. This is useful for fine tuning the
+jobs on an adhoc basis.
 
 Example setting the QEMU_CI_EXAMPLE_VAR variable:
 
@@ -28,13 +55,107 @@ For further information about how to set these variables, please refer to::
 
   https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
 
-Here is a list of the most used variables:
+Pushing to specially named branches
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Branch names starting with a 'ci-' prefix can be used as an alternative
+to setting variables. Details of supported branch names are detailed
+later in this document.
+
+Provide a custom gitlab CI configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above strategies all provide a way to fine tune the jobs defined by the
+standard QEMU gitlab CI configuration file. If this is not sufficient it is
+possible to completely replace the CI configuration file with a custom
+version. This allows the contributor to achieve essentially anything they
+desire, within the scope of what GitLab supports.
+
+Replacing the ``.gitlab-ci.yml`` file is done in the repository settings:
+
+  https://docs.gitlab.com/ee/ci/pipelines/settings.html#specify-a-custom-cicd-configuration-file
+
+Note that it is possible to reference an arbitrary URL. This could point
+to a replacement .gitlab-ci.yml on a specific branch in the same repo,
+or can point to any external site.
+
+
+Job grouping sets
+-----------------
+
+There are many different jobs defined in the GitLab CI pipeline used by QEMU.
+It is not practical to run all jobs in every scenario, so there are a set of
+rules defined that control which jobs are executed on each pipeline run. At
+a high level the jobs can be grouped into a number of mutually exclusive
+sets.
+
+ - Manual jobs
+
+   This is a set of jobs that will never be run automatically in any scenario.
+   The reason a job will be marked manual is if it is known to exhibit some
+   non-deterministic failures, or liable to trigger timeouts. Ideally jobs are
+   only present in this set for a short period of time while the problems are
+   investigated and resolved. Users can manually trigger these jobs from the
+   the pipelines web UI if desired, but they will never contribute to the
+   overall pipeline status.
+
+   Identified by the variable ``QEMU_JOB_MANUAL: 1``
+
+
+ - Minimal jobs
+
+   This is a minimal set of jobs that give a reasonable build and test sanity
+   check, which will be run by default in all possible scenarios. This set of
+   jobs is intended to be controlled fairly strictly to avoid wasting CI
+   minutes of maintainers/contributors. The intent is to get 80-90% coverage
+   of build and unit tests across the most important host platforms and
+   architectures.
+
+   Identified by the variable ``QEMU_JOB_MINIMAL: 1``
+
+   Run by setting the variable ``QEMU_CI: 1`` or pushing to a branch
+   named ``ci-min-XXX``.
+
+
+ - Gating jobs
+
+   This is a set of jobs will will always run as a gating test prior to code
+   merging into the default branch of the primary QEMU git repository. In user
+   forks the jobs will not run by default, but the user can opt-in to trigger
+   their execution. These jobs may run a particularly thorough set of scenarios
+   that maintainers are not normally going to exercise before sending series.
+
+   Identified by the variable ``QEMU_JOB_GATING: 1``
+
+   Run by setting the variable ``QEMU_CI_GATING: 1`` or pushing to a branch
+   named ``ci-gating-XXX``
+
+
+ - Optional jobs
+
+   This is all remaining jobs that don't fall into one of the above buckets.
+   These jobs will always be run as a gating test prior to code merging into
+   the default branch of the primary QEMU git repository. In user forks the
+   jobs will not run by default, but the user can opt-in to trigger their
+   execution. These jobs will aim to expand the host/platform build coverage
+   gaps left by the default jobs.
+
+   Identified by not having any of the above variables set.
+
+   Run by setting the variable ``QEMU_CI_ALL: 1`` or pushing to a branch
+   named ``ci-all-XXXX``
+
+Since the sets are mutually exclusive, if more than one of the variables above
+is set, then only one of them will be honoured. The precedence matches the
+ordering of the above docs.
+
+In addition to the above, there are some special extra sets that can augment
+any of the above sets
+
+ - Publishing jobs
 
-QEMU_CI_AVOCADO_TESTING
-~~~~~~~~~~~~~~~~~~~~~~~
-By default, tests using the Avocado framework are not run automatically in
-the pipelines (because multiple artifacts have to be downloaded, and if
-these artifacts are not already cached, downloading them make the jobs
-reach the timeout limit). Set this variable to have the tests using the
-Avocado framework run automatically.
+   These are jobs that are needed to publish the QEMU website. For user fork
+   repos these follow the normal rules set out above. For the primary QEMU
+   repo, however, the jobs will also run on the default branch.
 
+   Identified by the variable ``QEMU_JOB_PUBLISH: 1``
-- 
2.31.1

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Cornelia Huck]

On Thu, Aug 12 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:

> Currently pushes to user forks run the same set of build / test jobs as
> pushes to the main repo. This results in creation of 120+ individual
> jobs. While it is useful for subsystem maintainers, and even regular
> contributors to be able to run the full set of jobs, it is wasteful to
> run all of them all the time.
>
> In addition, with the expected change in GitLab to set a finite CI
> minute allowance on all users we need to be much more conservative.
>
> This patch thus sets up rules so that no CI jobs will run by default
> on push, except
>
>  - Pushes to branch name prefix 'staging' on 'qemu-project' repo
>    run full set of jobs
>  - Pushes to 'master' branch, only run jobs needed to publish
>    the website.
>
> In any pushes to users forks, CI is now strictly opt-in. The pipeline
> will always be created by every singe job will be marked manual. You
> can then use the web UI to start individual jobs.

Is a random user notified in some way that a pipeline with jobs that can
be triggered manually has been created?

> For more convenience there are three levels of increasing comprehensive
> CI job sets that can be chosen between
>
>  - Gating - run exactly the same as pushes to 'staging' branch
>             (except for jobs needing custom runners)
>
>    Push to 'ci-gating-xxx' branch, or set env QEMU_CI_GATING=1
>
>  - Full - same as 'Gating', except that acceptance tests don't
>           get run. This is equivalent to historical CI behaviour
> 	  for pushes to user forks.
>
>    Push to 'ci-full-xxx' branch, or set env QEMU_CI_FULL=1
>
>  - Minimal - a significantly cut down set of jobs to get a
>              decent sanity check of builds without burning
> 	     massive amounts of CI time.
>
>    Push to 'ci-min-xxx' branch, or set env QEMU_CI=1
>
> The minimal job set covers:
>
>   * Fedora, CentOS, Ubuntu & Debian system emulator builds
>     to cover all arch targets.
>   * Linux user static build
>   * Cross build i386 (to identify any 32-bit build issues)
>   * Cross build s390x (to identify any big endian build issues)
>   * Containers needed for the above
>   * Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
>   * Simple checks - python unittests, DCO check, checkpatch.pl, etc
>
> This gives about 30 jobs instead of 120 from the "Full" group. It
> is likely reasonable to cut the minimal set down even more, as IMHO
> there are too many system emulator jobs there.

Where do you think we should start to cut them down? Limit the set of
tested arch targets to the most common ones?

Generally speaking, this makes sense; but I think we have different
situations which need different kinds of testing, and we should make it
as easy as possible to run the right set of tests.

(a) an individual contributor is doing some changes

In that case, I assume (hope?) that the contributor has actually
compiled the code for the relevant targets and has done some manual
testing. Running acceptance tests locally would also be good, or things
like iotests or check-tcg, when applicable.

(b) a subsystem maintainer is queuing some changes

Some more comprehensive automated testing, and likely some
subsystem-specific testing on top (like testing that requires
specialized hardware, manual interventions, etc.) Since the introduction
of gitlab-ci, I had relied on what you call the 'full' set to be run
automatically when I push to my staging branch, plus my manual
testing. Prior to that, I relied on patchew and ran tests
locally. Drawbacks of that: it hogs my development machine, and it is
easy to forget something.

If I pull from a sub-submaintainer, I assume that everything is already
in good order, and only do some very light sanity/integration
checking. Although we're hardly doing subsystem pull request nowadays.

(c) a subsystem maintainer is preparing a pull request

Ideally, that should run the 'gating' set, to eliminate needless bounces
of the pull request; plus some subsystem-specific manual testing on
top. In practice, the 'full' set might be good enough.

Couple of ideas from my side:
- include some scripts/make targets that guide an individual contributor
  to run a good subset of automated tests locally
- define a 'ci-subsystem' set that covers usual pain points for a
  subsystem
- have subsystem maintainers run a pull req test in the qemu-project
  context (i.e. using extra CI minutes that the project may have), or
  put them on a special list on subsystem maintainers so they can use
  more minutes

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Daniel P. Berrangé]

On Mon, Aug 16, 2021 at 12:44:02PM +0200, Cornelia Huck wrote:
> On Thu, Aug 12 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> > Currently pushes to user forks run the same set of build / test jobs as
> > pushes to the main repo. This results in creation of 120+ individual
> > jobs. While it is useful for subsystem maintainers, and even regular
> > contributors to be able to run the full set of jobs, it is wasteful to
> > run all of them all the time.
> >
> > In addition, with the expected change in GitLab to set a finite CI
> > minute allowance on all users we need to be much more conservative.
> >
> > This patch thus sets up rules so that no CI jobs will run by default
> > on push, except
> >
> >  - Pushes to branch name prefix 'staging' on 'qemu-project' repo
> >    run full set of jobs
> >  - Pushes to 'master' branch, only run jobs needed to publish
> >    the website.
> >
> > In any pushes to users forks, CI is now strictly opt-in. The pipeline
> > will always be created by every singe job will be marked manual. You
> > can then use the web UI to start individual jobs.
> 
> Is a random user notified in some way that a pipeline with jobs that can
> be triggered manually has been created?

I don't believe there's any notification.


> > The minimal job set covers:
> >
> >   * Fedora, CentOS, Ubuntu & Debian system emulator builds
> >     to cover all arch targets.
> >   * Linux user static build
> >   * Cross build i386 (to identify any 32-bit build issues)
> >   * Cross build s390x (to identify any big endian build issues)
> >   * Containers needed for the above
> >   * Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
> >   * Simple checks - python unittests, DCO check, checkpatch.pl, etc
> >
> > This gives about 30 jobs instead of 120 from the "Full" group. It
> > is likely reasonable to cut the minimal set down even more, as IMHO
> > there are too many system emulator jobs there.
> 
> Where do you think we should start to cut them down? Limit the set of
> tested arch targets to the most common ones?

Some of our targets are obviously much more important and
frequently changed than others.  For contributors our goal is
to mimimize breakage after patches are submitted. Most of our
contributors changes will be well covered by x86-64 + aarch64
alone. Other targets give varying degrees of extra benefit.

On the other hand the contributors are likely to have tested
x86_64 or aarch64 themselves since that'll be their dev
platform. So the benefit of CI is testing bits that they
didnt bother to test.

No clear easy answer here, but I feel like we could benefit
from classifying our target archs tier 1/2/3 and tailoring
our default testing matrix accordingly.

The other way to cut down the "minimal" set is to reduce
the OS containers that we build. The jobs above end up
requiring something like 8 container builds - we should
try to cut this down to perhaps 2-3 container builds

> Generally speaking, this makes sense; but I think we have different
> situations which need different kinds of testing, and we should make it
> as easy as possible to run the right set of tests.
> 
> (a) an individual contributor is doing some changes
> 
> In that case, I assume (hope?) that the contributor has actually
> compiled the code for the relevant targets and has done some manual
> testing. Running acceptance tests locally would also be good, or things
> like iotests or check-tcg, when applicable.

With my contributor hat on, I like GitLab CI to validate the platforms
I always forget. Changes I do are 95% tested on Fedora x86_64. I have
often broken stuff for non-Linux builds (Windows in particular), or
have broken non-x86_64 target arches. CI lets me see this before
sending patches.  Unfortunately this means I benefit most from the
"full" set, but this won't be sustainable with limited CI minutes :-(

When I'm working on changing gitlab CI rules, then I burn loads of
minutes which is especially troubling - limited CI minutes will make
it very hard for me to debug future CI rule changes :-(

> (b) a subsystem maintainer is queuing some changes
> 
> Some more comprehensive automated testing, and likely some
> subsystem-specific testing on top (like testing that requires
> specialized hardware, manual interventions, etc.) Since the introduction
> of gitlab-ci, I had relied on what you call the 'full' set to be run
> automatically when I push to my staging branch, plus my manual
> testing. Prior to that, I relied on patchew and ran tests
> locally. Drawbacks of that: it hogs my development machine, and it is
> easy to forget something.

> If I pull from a sub-submaintainer, I assume that everything is already
> in good order, and only do some very light sanity/integration
> checking. Although we're hardly doing subsystem pull request nowadays.

Yes, I think that 'minimal' ought to be a good "smoke test" to run
when queuing changes, on the assumption you'll run a full/gating set
before sending the final pull.

> (c) a subsystem maintainer is preparing a pull request
> 
> Ideally, that should run the 'gating' set, to eliminate needless bounces
> of the pull request; plus some subsystem-specific manual testing on
> top. In practice, the 'full' set might be good enough.

Yeah, the full/gating set is what I would thing subsys maintainers
would want to use, to minimize risk that Peter's tests throw back
the merge due to failure. The only difference of gating vs full
is whether the acceptance tests run.

> Couple of ideas from my side:
> - include some scripts/make targets that guide an individual contributor
>   to run a good subset of automated tests locally

We've got various make targets right now, but they don't guide users
as to what is the most beneficial thing to run.

> - define a 'ci-subsystem' set that covers usual pain points for a
>   subsystem
> - have subsystem maintainers run a pull req test in the qemu-project
>   context (i.e. using extra CI minutes that the project may have), or
>   put them on a special list on subsystem maintainers so they can use
>   more minutes

I think ultimately we should be looking to take email out of the loop
for merging pull requests from subsys maintainers.

If we expect subsys maintainers to use gitlab CI before sending, then
we have a crazy situation where subsys maintainers pushes to gitlab,
has CI run, then send email PULL to list, then Peter downloads and
applies the mails, and pushes back to gitlab and runs CI again and
then pushes to gitlab again for master.

It would be much simpler if the subsys maintainer pushes to gitlab
and opens a merge request. This would trigger the CI gating runs,
consuming minutes from the QEMU project, not the maintainer. Peter
can ignore merge request until CI passes. The subsys maintainer
is responsible for getting CI to pass. Once CI passes, the merge
request could be apporved and merged. The person approving it for
merge should really only need to validate the the non-automatable
stuff. eg it meets the current freeze rules; the maintainer is
the owner of the subsystem being modified, etc.

Of course this relies on us being able to use GitLab for 100% of
merge time gating. Cleber's custom runners recently enabled get
us closer, but I think Peter still uses some other hardware
outside of GitLab for some testing.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Philippe Mathieu-Daudé]

On 8/16/21 1:03 PM, Daniel P. Berrangé wrote:
> On Mon, Aug 16, 2021 at 12:44:02PM +0200, Cornelia Huck wrote:
>> On Thu, Aug 12 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> The minimal job set covers:
>>>
>>>   * Fedora, CentOS, Ubuntu & Debian system emulator builds
>>>     to cover all arch targets.
>>>   * Linux user static build
>>>   * Cross build i386 (to identify any 32-bit build issues)
>>>   * Cross build s390x (to identify any big endian build issues)
>>>   * Containers needed for the above
>>>   * Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
>>>   * Simple checks - python unittests, DCO check, checkpatch.pl, etc
>>>
>>> This gives about 30 jobs instead of 120 from the "Full" group. It
>>> is likely reasonable to cut the minimal set down even more, as IMHO
>>> there are too many system emulator jobs there.
>>
>> Where do you think we should start to cut them down? Limit the set of
>> tested arch targets to the most common ones?
> 
> Some of our targets are obviously much more important and
> frequently changed than others.  For contributors our goal is
> to mimimize breakage after patches are submitted. Most of our
> contributors changes will be well covered by x86-64 + aarch64
> alone. Other targets give varying degrees of extra benefit.
> 
> On the other hand the contributors are likely to have tested
> x86_64 or aarch64 themselves since that'll be their dev
> platform. So the benefit of CI is testing bits that they
> didnt bother to test.
> 
> No clear easy answer here, but I feel like we could benefit
> from classifying our target archs tier 1/2/3 and tailoring
> our default testing matrix accordingly.

[*]

> The other way to cut down the "minimal" set is to reduce
> the OS containers that we build. The jobs above end up
> requiring something like 8 container builds - we should
> try to cut this down to perhaps 2-3 container builds
> 
>> Generally speaking, this makes sense; but I think we have different
>> situations which need different kinds of testing, and we should make it
>> as easy as possible to run the right set of tests.
>>
>> (a) an individual contributor is doing some changes
>>
>> In that case, I assume (hope?) that the contributor has actually
>> compiled the code for the relevant targets and has done some manual
>> testing. Running acceptance tests locally would also be good, or things
>> like iotests or check-tcg, when applicable.
> 
> With my contributor hat on, I like GitLab CI to validate the platforms
> I always forget. Changes I do are 95% tested on Fedora x86_64. I have
> often broken stuff for non-Linux builds (Windows in particular), or
> have broken non-x86_64 target arches. CI lets me see this before
> sending patches.  Unfortunately this means I benefit most from the
> "full" set, but this won't be sustainable with limited CI minutes :-(

Hmmm somehow. What we said 2 years ago was "if a vendor is interested
in have QEMU working on a target, it should provide hardware to the
project, otherwise it can't be considered a tier 1 platform" [*].
Some did, but we have been wasting their resources doing almost nothing,
mostly because we were not ready / organized to use them. Hopefully this
might change soon.

> When I'm working on changing gitlab CI rules, then I burn loads of
> minutes which is especially troubling - limited CI minutes will make
> it very hard for me to debug future CI rule changes :-(

Which is why I stopped doing gitlab-related patches and reduced
my testing (or even worst for me, before I was simply doing
'git-push' and wait for the pipeline result, now I'm back
running 6 different scripts locally).

>> (c) a subsystem maintainer is preparing a pull request
>>
>> Ideally, that should run the 'gating' set, to eliminate needless bounces
>> of the pull request; plus some subsystem-specific manual testing on
>> top. In practice, the 'full' set might be good enough.
> 
> Yeah, the full/gating set is what I would thing subsys maintainers
> would want to use, to minimize risk that Peter's tests throw back
> the merge due to failure. The only difference of gating vs full
> is whether the acceptance tests run.

"run" or "pass"?

> Of course this relies on us being able to use GitLab for 100% of
> merge time gating. Cleber's custom runners recently enabled get
> us closer, but I think Peter still uses some other hardware
> outside of GitLab for some testing.

There is hope!

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Daniel P. Berrangé]

On Mon, Aug 16, 2021 at 01:20:14PM +0200, Philippe Mathieu-Daudé wrote:
> On 8/16/21 1:03 PM, Daniel P. Berrangé wrote:
> >> (c) a subsystem maintainer is preparing a pull request
> >>
> >> Ideally, that should run the 'gating' set, to eliminate needless bounces
> >> of the pull request; plus some subsystem-specific manual testing on
> >> top. In practice, the 'full' set might be good enough.
> > 
> > Yeah, the full/gating set is what I would thing subsys maintainers
> > would want to use, to minimize risk that Peter's tests throw back
> > the merge due to failure. The only difference of gating vs full
> > is whether the acceptance tests run.
> 
> "run" or "pass"?

In 'full' set the acceptance test jobs are marked to allow to be
triggered manually and won't affect pipeline status if they fail

In 'gating' set the acceptance test jobs always run and are
required to pass.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Philippe Mathieu-Daudé]

On 8/16/21 1:35 PM, Daniel P. Berrangé wrote:
> On Mon, Aug 16, 2021 at 01:20:14PM +0200, Philippe Mathieu-Daudé wrote:
>> On 8/16/21 1:03 PM, Daniel P. Berrangé wrote:
>>>> (c) a subsystem maintainer is preparing a pull request
>>>>
>>>> Ideally, that should run the 'gating' set, to eliminate needless bounces
>>>> of the pull request; plus some subsystem-specific manual testing on
>>>> top. In practice, the 'full' set might be good enough.
>>>
>>> Yeah, the full/gating set is what I would thing subsys maintainers
>>> would want to use, to minimize risk that Peter's tests throw back
>>> the merge due to failure. The only difference of gating vs full
>>> is whether the acceptance tests run.
>>
>> "run" or "pass"?
> 
> In 'full' set the acceptance test jobs are marked to allow to be
> triggered manually and won't affect pipeline status if they fail

I'd expect 'full' to *run* all tests, having acceptance tests
tagged with 'allow_failure', not 'manual', but I realize this
is a waste of resources. OK then.

> In 'gating' set the acceptance test jobs always run and are
> required to pass.
> 
> 
> Regards,
> Daniel
> 

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Cornelia Huck]

On Mon, Aug 16 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Mon, Aug 16, 2021 at 12:44:02PM +0200, Cornelia Huck wrote:
>> On Thu, Aug 12 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:
>> 
>> > Currently pushes to user forks run the same set of build / test jobs as
>> > pushes to the main repo. This results in creation of 120+ individual
>> > jobs. While it is useful for subsystem maintainers, and even regular
>> > contributors to be able to run the full set of jobs, it is wasteful to
>> > run all of them all the time.
>> >
>> > In addition, with the expected change in GitLab to set a finite CI
>> > minute allowance on all users we need to be much more conservative.
>> >
>> > This patch thus sets up rules so that no CI jobs will run by default
>> > on push, except
>> >
>> >  - Pushes to branch name prefix 'staging' on 'qemu-project' repo
>> >    run full set of jobs
>> >  - Pushes to 'master' branch, only run jobs needed to publish
>> >    the website.
>> >
>> > In any pushes to users forks, CI is now strictly opt-in. The pipeline
>> > will always be created by every singe job will be marked manual. You
>> > can then use the web UI to start individual jobs.
>> 
>> Is a random user notified in some way that a pipeline with jobs that can
>> be triggered manually has been created?
>
> I don't believe there's any notification.

So a random user might be entirely unaware that they could trigger some
jobs. OTOH, they already consciously pushed to gitlab...

>
>
>> > The minimal job set covers:
>> >
>> >   * Fedora, CentOS, Ubuntu & Debian system emulator builds
>> >     to cover all arch targets.
>> >   * Linux user static build
>> >   * Cross build i386 (to identify any 32-bit build issues)
>> >   * Cross build s390x (to identify any big endian build issues)
>> >   * Containers needed for the above
>> >   * Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
>> >   * Simple checks - python unittests, DCO check, checkpatch.pl, etc
>> >
>> > This gives about 30 jobs instead of 120 from the "Full" group. It
>> > is likely reasonable to cut the minimal set down even more, as IMHO
>> > there are too many system emulator jobs there.
>> 
>> Where do you think we should start to cut them down? Limit the set of
>> tested arch targets to the most common ones?
>
> Some of our targets are obviously much more important and
> frequently changed than others.  For contributors our goal is
> to mimimize breakage after patches are submitted. Most of our
> contributors changes will be well covered by x86-64 + aarch64
> alone. Other targets give varying degrees of extra benefit.

I'd probably add s390x to that list, not just because of personal bias
:), but also because it has a unique set of devices, is big endian, and
has been broken in the past.

>
> On the other hand the contributors are likely to have tested
> x86_64 or aarch64 themselves since that'll be their dev
> platform. So the benefit of CI is testing bits that they
> didnt bother to test.

Contributors are unlikely to have tested s390x unless they explicitly
develop for that target, so that might be a net benefit.

>
> No clear easy answer here, but I feel like we could benefit
> from classifying our target archs tier 1/2/3 and tailoring
> our default testing matrix accordingly.

Nod, that seems useful in any case.

>
> The other way to cut down the "minimal" set is to reduce
> the OS containers that we build. The jobs above end up
> requiring something like 8 container builds - we should
> try to cut this down to perhaps 2-3 container builds

Yes, that is also what usually takes ages.

>
>> Generally speaking, this makes sense; but I think we have different
>> situations which need different kinds of testing, and we should make it
>> as easy as possible to run the right set of tests.
>> 
>> (a) an individual contributor is doing some changes
>> 
>> In that case, I assume (hope?) that the contributor has actually
>> compiled the code for the relevant targets and has done some manual
>> testing. Running acceptance tests locally would also be good, or things
>> like iotests or check-tcg, when applicable.
>
> With my contributor hat on, I like GitLab CI to validate the platforms
> I always forget. Changes I do are 95% tested on Fedora x86_64. I have
> often broken stuff for non-Linux builds (Windows in particular), or
> have broken non-x86_64 target arches. CI lets me see this before
> sending patches.  Unfortunately this means I benefit most from the
> "full" set, but this won't be sustainable with limited CI minutes :-(

Agreed :(

>
> When I'm working on changing gitlab CI rules, then I burn loads of
> minutes which is especially troubling - limited CI minutes will make
> it very hard for me to debug future CI rule changes :-(

I hope that will not make gitlab CI a complete non-starter -- if you
cannot easily debug a test case that is failing, it's mostly
useless. We've seen too many cases where a failure could not be
reproduced when the test case was running locally.

>
>> (b) a subsystem maintainer is queuing some changes
>> 
>> Some more comprehensive automated testing, and likely some
>> subsystem-specific testing on top (like testing that requires
>> specialized hardware, manual interventions, etc.) Since the introduction
>> of gitlab-ci, I had relied on what you call the 'full' set to be run
>> automatically when I push to my staging branch, plus my manual
>> testing. Prior to that, I relied on patchew and ran tests
>> locally. Drawbacks of that: it hogs my development machine, and it is
>> easy to forget something.
>
>> If I pull from a sub-submaintainer, I assume that everything is already
>> in good order, and only do some very light sanity/integration
>> checking. Although we're hardly doing subsystem pull request nowadays.
>
> Yes, I think that 'minimal' ought to be a good "smoke test" to run
> when queuing changes, on the assumption you'll run a full/gating set
> before sending the final pull.
>
>> (c) a subsystem maintainer is preparing a pull request
>> 
>> Ideally, that should run the 'gating' set, to eliminate needless bounces
>> of the pull request; plus some subsystem-specific manual testing on
>> top. In practice, the 'full' set might be good enough.
>
> Yeah, the full/gating set is what I would thing subsys maintainers
> would want to use, to minimize risk that Peter's tests throw back
> the merge due to failure. The only difference of gating vs full
> is whether the acceptance tests run.

I can at least run a subset of the acceptance tests locally, but I think
I may be missing some platforms? Still, better than nothing.

>
>> Couple of ideas from my side:
>> - include some scripts/make targets that guide an individual contributor
>>   to run a good subset of automated tests locally
>
> We've got various make targets right now, but they don't guide users
> as to what is the most beneficial thing to run.

There's also the question what the most beneficial subset actually
is. "Run make check-tcg when you are doing tcg changes" is an unfortunately
rare obvious case.

>
>> - define a 'ci-subsystem' set that covers usual pain points for a
>>   subsystem
>> - have subsystem maintainers run a pull req test in the qemu-project
>>   context (i.e. using extra CI minutes that the project may have), or
>>   put them on a special list on subsystem maintainers so they can use
>>   more minutes
>
> I think ultimately we should be looking to take email out of the loop
> for merging pull requests from subsys maintainers.
>
> If we expect subsys maintainers to use gitlab CI before sending, then
> we have a crazy situation where subsys maintainers pushes to gitlab,
> has CI run, then send email PULL to list, then Peter downloads and
> applies the mails, and pushes back to gitlab and runs CI again and
> then pushes to gitlab again for master.

Hm, I thought Peter pulled the subsys maintainers' signed tags?

[As a side remark, I recommend using b4 for applying patches. The only
advantage for me when pulling vs applying with b4 is the usage of signed
tags. Not that I needed to pull anything recently :)]

>
> It would be much simpler if the subsys maintainer pushes to gitlab
> and opens a merge request. This would trigger the CI gating runs,
> consuming minutes from the QEMU project, not the maintainer. Peter
> can ignore merge request until CI passes. The subsys maintainer
> is responsible for getting CI to pass. Once CI passes, the merge
> request could be apporved and merged. The person approving it for
> merge should really only need to validate the the non-automatable
> stuff. eg it meets the current freeze rules; the maintainer is
> the owner of the subsystem being modified, etc.

I hate gitlab MR tooling. (Same for any forge. I'm contemplating using
magit/forge to at least avoid a medium break; anyone been using it
sucessfully?) At the very least, we should have notifications about pull
requests and when they are merged on the mailing list, just to keep
information about development in one page.

>
> Of course this relies on us being able to use GitLab for 100% of
> merge time gating. Cleber's custom runners recently enabled get
> us closer, but I think Peter still uses some other hardware
> outside of GitLab for some testing.

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Daniel P. Berrangé]

On Mon, Aug 16, 2021 at 01:47:08PM +0200, Cornelia Huck wrote:
> On Mon, Aug 16 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> > On Mon, Aug 16, 2021 at 12:44:02PM +0200, Cornelia Huck wrote:
> >> On Thu, Aug 12 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >> > The minimal job set covers:
> >> >
> >> >   * Fedora, CentOS, Ubuntu & Debian system emulator builds
> >> >     to cover all arch targets.
> >> >   * Linux user static build
> >> >   * Cross build i386 (to identify any 32-bit build issues)
> >> >   * Cross build s390x (to identify any big endian build issues)
> >> >   * Containers needed for the above
> >> >   * Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
> >> >   * Simple checks - python unittests, DCO check, checkpatch.pl, etc
> >> >
> >> > This gives about 30 jobs instead of 120 from the "Full" group. It
> >> > is likely reasonable to cut the minimal set down even more, as IMHO
> >> > there are too many system emulator jobs there.
> >> 
> >> Where do you think we should start to cut them down? Limit the set of
> >> tested arch targets to the most common ones?
> >
> > Some of our targets are obviously much more important and
> > frequently changed than others.  For contributors our goal is
> > to mimimize breakage after patches are submitted. Most of our
> > contributors changes will be well covered by x86-64 + aarch64
> > alone. Other targets give varying degrees of extra benefit.
> 
> I'd probably add s390x to that list, not just because of personal bias
> :), but also because it has a unique set of devices, is big endian, and
> has been broken in the past.

The desire to have something covering big endian is why I included
the s390x cross-build in the "minimal" set. Even if we don't run
the tests, building it is still worthwhile.

> > When I'm working on changing gitlab CI rules, then I burn loads of
> > minutes which is especially troubling - limited CI minutes will make
> > it very hard for me to debug future CI rule changes :-(
> 
> I hope that will not make gitlab CI a complete non-starter -- if you
> cannot easily debug a test case that is failing, it's mostly
> useless. We've seen too many cases where a failure could not be
> reproduced when the test case was running locally.

One of the best things about GitLab compared to what we had with
Travis is that the build environment is 100% container based (until
Cleber's custom runners arrived).  This meant that when something
does fail in GitLab, you can pull the container image down from
the gitlab registry and run the build locally. You still have
differences due to hardware or CPUs resources, but its a hell of
alot easier to reproduce than before. This is good enough for most
contributors in general, but not for the CI maintainers, who need
to debug especially the CI system as it exists on GitLab.


> >> (c) a subsystem maintainer is preparing a pull request
> >> 
> >> Ideally, that should run the 'gating' set, to eliminate needless bounces
> >> of the pull request; plus some subsystem-specific manual testing on
> >> top. In practice, the 'full' set might be good enough.
> >
> > Yeah, the full/gating set is what I would thing subsys maintainers
> > would want to use, to minimize risk that Peter's tests throw back
> > the merge due to failure. The only difference of gating vs full
> > is whether the acceptance tests run.
> 
> I can at least run a subset of the acceptance tests locally, but I think
> I may be missing some platforms? Still, better than nothing.

As ever the big problem for most people is non-x86_64 platforms. The
custom runners solve this for gitlab, and in theory people can deploy
a VM using QEMU TCG to do this locally, but massively slower


> >> - define a 'ci-subsystem' set that covers usual pain points for a
> >>   subsystem
> >> - have subsystem maintainers run a pull req test in the qemu-project
> >>   context (i.e. using extra CI minutes that the project may have), or
> >>   put them on a special list on subsystem maintainers so they can use
> >>   more minutes
> >
> > I think ultimately we should be looking to take email out of the loop
> > for merging pull requests from subsys maintainers.
> >
> > If we expect subsys maintainers to use gitlab CI before sending, then
> > we have a crazy situation where subsys maintainers pushes to gitlab,
> > has CI run, then send email PULL to list, then Peter downloads and
> > applies the mails, and pushes back to gitlab and runs CI again and
> > then pushes to gitlab again for master.
> 
> Hm, I thought Peter pulled the subsys maintainers' signed tags?

We could have a CI job that validates the merge request is associated
with a signed tag whose key is on a designated keyring containing the
known maintainers.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Cornelia Huck]

On Mon, Aug 16 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Mon, Aug 16, 2021 at 01:47:08PM +0200, Cornelia Huck wrote:
>> On Mon, Aug 16 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:
>> 
>> > When I'm working on changing gitlab CI rules, then I burn loads of
>> > minutes which is especially troubling - limited CI minutes will make
>> > it very hard for me to debug future CI rule changes :-(
>> 
>> I hope that will not make gitlab CI a complete non-starter -- if you
>> cannot easily debug a test case that is failing, it's mostly
>> useless. We've seen too many cases where a failure could not be
>> reproduced when the test case was running locally.
>
> One of the best things about GitLab compared to what we had with
> Travis is that the build environment is 100% container based (until
> Cleber's custom runners arrived).  This meant that when something
> does fail in GitLab, you can pull the container image down from
> the gitlab registry and run the build locally. You still have
> differences due to hardware or CPUs resources, but its a hell of
> alot easier to reproduce than before. This is good enough for most
> contributors in general, but not for the CI maintainers, who need
> to debug especially the CI system as it exists on GitLab.

Correct me if I'm wrong, but I remember that some of the more
aggravating failures in particular could not be reproduced outside of
the gitlab environment, even while using the same image.

>
>
>> >> (c) a subsystem maintainer is preparing a pull request
>> >> 
>> >> Ideally, that should run the 'gating' set, to eliminate needless bounces
>> >> of the pull request; plus some subsystem-specific manual testing on
>> >> top. In practice, the 'full' set might be good enough.
>> >
>> > Yeah, the full/gating set is what I would thing subsys maintainers
>> > would want to use, to minimize risk that Peter's tests throw back
>> > the merge due to failure. The only difference of gating vs full
>> > is whether the acceptance tests run.
>> 
>> I can at least run a subset of the acceptance tests locally, but I think
>> I may be missing some platforms? Still, better than nothing.
>
> As ever the big problem for most people is non-x86_64 platforms. The
> custom runners solve this for gitlab, and in theory people can deploy
> a VM using QEMU TCG to do this locally, but massively slower

Still, the acceptance tests are usually small enough that running under
tcg is bearable.

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Daniel P. Berrangé]

On Mon, Aug 16, 2021 at 03:19:31PM +0200, Cornelia Huck wrote:
> On Mon, Aug 16 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> > On Mon, Aug 16, 2021 at 01:47:08PM +0200, Cornelia Huck wrote:
> >> On Mon, Aug 16 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >> 
> >> > When I'm working on changing gitlab CI rules, then I burn loads of
> >> > minutes which is especially troubling - limited CI minutes will make
> >> > it very hard for me to debug future CI rule changes :-(
> >> 
> >> I hope that will not make gitlab CI a complete non-starter -- if you
> >> cannot easily debug a test case that is failing, it's mostly
> >> useless. We've seen too many cases where a failure could not be
> >> reproduced when the test case was running locally.
> >
> > One of the best things about GitLab compared to what we had with
> > Travis is that the build environment is 100% container based (until
> > Cleber's custom runners arrived).  This meant that when something
> > does fail in GitLab, you can pull the container image down from
> > the gitlab registry and run the build locally. You still have
> > differences due to hardware or CPUs resources, but its a hell of
> > alot easier to reproduce than before. This is good enough for most
> > contributors in general, but not for the CI maintainers, who need
> > to debug especially the CI system as it exists on GitLab.
> 
> Correct me if I'm wrong, but I remember that some of the more
> aggravating failures in particular could not be reproduced outside of
> the gitlab environment, even while using the same image.

There will always be some like that for sure. Some problems are very
sensitive to timing issues that are affected by load or CPUs parallism,
etc.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Philippe Mathieu-Daudé]

On 8/16/21 2:01 PM, Daniel P. Berrangé wrote:
> On Mon, Aug 16, 2021 at 01:47:08PM +0200, Cornelia Huck wrote:
>> On Mon, Aug 16 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>> On Thu, Aug 12 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:

>>> When I'm working on changing gitlab CI rules, then I burn loads of
>>> minutes which is especially troubling - limited CI minutes will make
>>> it very hard for me to debug future CI rule changes :-(
>>
>> I hope that will not make gitlab CI a complete non-starter -- if you
>> cannot easily debug a test case that is failing, it's mostly
>> useless. We've seen too many cases where a failure could not be
>> reproduced when the test case was running locally.
> 
> One of the best things about GitLab compared to what we had with
> Travis is that the build environment is 100% container based (until
> Cleber's custom runners arrived).  This meant that when something
> does fail in GitLab, you can pull the container image down from
> the gitlab registry and run the build locally. You still have
> differences due to hardware or CPUs resources, but its a hell of
> alot easier to reproduce than before. This is good enough for most
> contributors in general, but not for the CI maintainers, who need
> to debug especially the CI system as it exists on GitLab.

FWIW we could do that with Travis-CI too using:

 $ make docker-test-build@travis

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Thomas Huth]

(meta note: patch does not apply anymore and needs to be refreshed)

On 12/08/2021 20.04, Daniel P. Berrangé wrote:
[...]
> diff --git a/.gitlab-ci.d/rules.yml b/.gitlab-ci.d/rules.yml
> new file mode 100644
> index 0000000000..3399722ca9
> --- /dev/null
> +++ b/.gitlab-ci.d/rules.yml
> @@ -0,0 +1,116 @@
> +
> +# This defines rules used to control individual job execution
> +# See docs/devel/ci-jobs.rst for an explanation of the various
> +# variables and branch naming conventions applied here.
> +#
> +.job_rules:
> +  rules:
> +    # ======================================================================
> +    # Rules that apply regardless of whether the primary qemu repo or a fork
> +    # ======================================================================
> +
> +    # Skip any cirrus job if either repo or api token are missing
> +    # as we won't be able to talk to cirrus
> +    - if: '$CIRRUS_VM_INSTANCE_TYPE && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
> +      when: never
> +
> +    # Any jobs marked as manual, never get automatically run in any scenario
> +    # and don't contribute to pipeline status
> +    - if: '$QEMU_JOB_MANUAL'
> +      when: manual
> +      allow_failure: true
> +
> +    # For the main repo, tags represent official releases.
> +    # The branch associated with the release will have passed
> +    # a CI pipeline already
> +    #
> +    # For user forks, tags are tyically added by tools like

s/tyically/typically/

> +    # git-publish at the same time as pushing the branch prior
> +    # to sending out for review
> +    #
> +    # In neither case do we wish to run CI pipelines for tags
> +    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_TAG'
> +      when: never

Not sure whether I like this rule ... First, tags are very seldom compared 
to normal pushes to branches, so this should not affect us that much. 
Second, I think it might be good for "documentation" purposes to be able to 
say that the CI ran properly for a certain tag. Ok, you could still look it 
up in the push to a branch that might have happened before, but that's 
cumbersome. Just my 0.02 €.

> +
> +    # ====================================
> +    # Rules for running jobs in user forks
> +    # ====================================
> +
> +    # Part 1: gating jobs
> +    # -------------------
> +
> +    # If on a branch with name prefix 'ci-acceptance-', then run
> +    # everything, just as a gating job on 'staging' branch would
> +    - if: '$CI_COMMIT_BRANCH =~ /^ci-gating-/'
> +      when: on_success
> +
> +    # If user set QEMU_CI_GATING=1, then run everything just as
> +    # a gating job on 'staging' branch would
> +    - if: '$QEMU_CI_GATING'
> +      when: on_success
> +
> +    # Otherwise never run jobs marked as gating, but allow manual trigger
> +    # without affecting pipeline status
> +    - if: '$QEMU_JOB_GATING'
> +      when: manual
> +      allow_failure: true

IMHO if it's "gating", then we should not use "allow_failure: true", no 
matter whether the job is manual or not. Otherwise this is very confusing.

I guess the problem is rather whether the "acceptance" tests should be 
called "gating" or not. Currently they are in a rather gray area in the 
gitlab-CI...

> +
> +    # =============================================
> +    # Part 2: opt-in for all CI, except gating jobs
> +    # =============================================
> +
> +    # If pushing to a branch with name prefix 'ci-all', then run all jobs
> +    - if: '$CI_COMMIT_BRANCH =~ /^ci-all/'
> +      when: on_success
> +
> +    # If user QEMU_CI_ALL=1, then run all jobs
> +    - if: '$QEMU_CI_ALL'
> +      when: on_success

Uh, so "all" is not running all jobs? ... that's confusing, too. Could you 
come up with some better naming? QEMU_CI_CORE maybe?

[...]
> +Set variable manually when pushing to git
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>   
>   Variables can be set manually when pushing a branch or tag, using
> -git-push command line arguments.
> +git-push command line arguments. This is useful for fine tuning the
> +jobs on an adhoc basis.

s/adhoc/ad hoc/ (I think)

  Thomas

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Daniel P. Berrangé]

On Wed, Aug 25, 2021 at 12:42:32PM +0200, Thomas Huth wrote:
> 
> (meta note: patch does not apply anymore and needs to be refreshed)
> 
> On 12/08/2021 20.04, Daniel P. Berrangé wrote:
> [...]
> > diff --git a/.gitlab-ci.d/rules.yml b/.gitlab-ci.d/rules.yml
> > new file mode 100644
> > index 0000000000..3399722ca9
> > --- /dev/null
> > +++ b/.gitlab-ci.d/rules.yml
> > @@ -0,0 +1,116 @@
> > +
> > +# This defines rules used to control individual job execution
> > +# See docs/devel/ci-jobs.rst for an explanation of the various
> > +# variables and branch naming conventions applied here.
> > +#
> > +.job_rules:
> > +  rules:
> > +    # ======================================================================
> > +    # Rules that apply regardless of whether the primary qemu repo or a fork
> > +    # ======================================================================
> > +
> > +    # Skip any cirrus job if either repo or api token are missing
> > +    # as we won't be able to talk to cirrus
> > +    - if: '$CIRRUS_VM_INSTANCE_TYPE && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
> > +      when: never
> > +
> > +    # Any jobs marked as manual, never get automatically run in any scenario
> > +    # and don't contribute to pipeline status
> > +    - if: '$QEMU_JOB_MANUAL'
> > +      when: manual
> > +      allow_failure: true
> > +
> > +    # For the main repo, tags represent official releases.
> > +    # The branch associated with the release will have passed
> > +    # a CI pipeline already
> > +    #
> > +    # For user forks, tags are tyically added by tools like
> 
> s/tyically/typically/
> 
> > +    # git-publish at the same time as pushing the branch prior
> > +    # to sending out for review
> > +    #
> > +    # In neither case do we wish to run CI pipelines for tags
> > +    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_TAG'
> > +      when: never
> 
> Not sure whether I like this rule ... First, tags are very seldom compared
> to normal pushes to branches, so this should not affect us that much.
> Second, I think it might be good for "documentation" purposes to be able to
> say that the CI ran properly for a certain tag. Ok, you could still look it
> up in the push to a branch that might have happened before, but that's
> cumbersome. Just my 0.02 €.

"git-publish" creates tags for every version and pushes them to your
repo when you use --pull arg. I don't want jobs triggered when
git-publish pushes tags, because I will have already tested the code
before I ask git-publish to send the pull.  So IMHO skipping CI pipelines
on forks is important to avoid git-publish burning everyone's CI minutes.

In terms of the upstream repo, a tag is only pushed when the changes for
the release have already been pushed. Those changes would have undergone
CI already. There's no point running CI again for the tag, especially if
no one is going to do anything with failures it might report. 

> > +    # ====================================
> > +    # Rules for running jobs in user forks
> > +    # ====================================
> > +
> > +    # Part 1: gating jobs
> > +    # -------------------
> > +
> > +    # If on a branch with name prefix 'ci-acceptance-', then run
> > +    # everything, just as a gating job on 'staging' branch would
> > +    - if: '$CI_COMMIT_BRANCH =~ /^ci-gating-/'
> > +      when: on_success
> > +
> > +    # If user set QEMU_CI_GATING=1, then run everything just as
> > +    # a gating job on 'staging' branch would
> > +    - if: '$QEMU_CI_GATING'
> > +      when: on_success
> > +
> > +    # Otherwise never run jobs marked as gating, but allow manual trigger
> > +    # without affecting pipeline status
> > +    - if: '$QEMU_JOB_GATING'
> > +      when: manual
> > +      allow_failure: true
> 
> IMHO if it's "gating", then we should not use "allow_failure: true", no
> matter whether the job is manual or not. Otherwise this is very confusing.

I think you're misinterpreting this.

Gating jobs run on the upstream pushes to 'staging' (not shown in this
quoted context - its higher in rules.yml), or if you have pushed to a
branch 'ci-gating-' (the first rule here), or if you set QEMU_CI_GATING
env var when pushing (the second rule here).

This 3rd rule is about ensuring gating jobs do NOT get launched in any
other scenario. We could use 'when: never' but that's a big hammer that
prevents users to opt-ing in to running a single job from the web UI,
so we use 'when: manual'.  If you use 'when: manual' on its own, the
pipeline will never complete, as it'll be waiting for someone to
trigger the manual job which is not what we need. 'allow_failure: true'
means that the pipeline will complete without waiting for the manual
job.

Ideally there would be a way to say "Let this job be manually run,
and make its failure affect the pipeline status, but ignore the job
if not ru nmanually".  GitLab doesn't support that AFAIK though, so
this is the next best option, that isn't 'when: never'.

> > +
> > +    # =============================================
> > +    # Part 2: opt-in for all CI, except gating jobs
> > +    # =============================================
> > +
> > +    # If pushing to a branch with name prefix 'ci-all', then run all jobs
> > +    - if: '$CI_COMMIT_BRANCH =~ /^ci-all/'
> > +      when: on_success
> > +
> > +    # If user QEMU_CI_ALL=1, then run all jobs
> > +    - if: '$QEMU_CI_ALL'
> > +      when: on_success
> 
> Uh, so "all" is not running all jobs? ... that's confusing, too. Could you
> come up with some better naming? QEMU_CI_CORE maybe?

"all" is running everything that currently runs today when you do a
git push in a fork.  I admit it is odd that 'gating' runs more than
'all'.

We could change it to "default" instead.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Thomas Huth]

On 12/08/2021 20.04, Daniel P. Berrangé wrote:
> Currently pushes to user forks run the same set of build / test jobs as
> pushes to the main repo. This results in creation of 120+ individual
> jobs. While it is useful for subsystem maintainers, and even regular
> contributors to be able to run the full set of jobs, it is wasteful to
> run all of them all the time.

  Hi Daniel!

Now that gitlab introduces limits for the CI for all users, I think we 
should revisit this topic ... could you maybe respin your patch to the 
current git master branch?

  Thomas


> In addition, with the expected change in GitLab to set a finite CI
> minute allowance on all users we need to be much more conservative.
> 
> This patch thus sets up rules so that no CI jobs will run by default
> on push, except
> 
>   - Pushes to branch name prefix 'staging' on 'qemu-project' repo
>     run full set of jobs
>   - Pushes to 'master' branch, only run jobs needed to publish
>     the website.
> 
> In any pushes to users forks, CI is now strictly opt-in. The pipeline
> will always be created by every singe job will be marked manual. You
> can then use the web UI to start individual jobs.
> 
> For more convenience there are three levels of increasing comprehensive
> CI job sets that can be chosen between
> 
>   - Gating - run exactly the same as pushes to 'staging' branch
>              (except for jobs needing custom runners)
> 
>     Push to 'ci-gating-xxx' branch, or set env QEMU_CI_GATING=1
> 
>   - Full - same as 'Gating', except that acceptance tests don't
>            get run. This is equivalent to historical CI behaviour
> 	  for pushes to user forks.
> 
>     Push to 'ci-full-xxx' branch, or set env QEMU_CI_FULL=1
> 
>   - Minimal - a significantly cut down set of jobs to get a
>               decent sanity check of builds without burning
> 	     massive amounts of CI time.
> 
>     Push to 'ci-min-xxx' branch, or set env QEMU_CI=1
> 
> The minimal job set covers:
> 
>    * Fedora, CentOS, Ubuntu & Debian system emulator builds
>      to cover all arch targets.
>    * Linux user static build
>    * Cross build i386 (to identify any 32-bit build issues)
>    * Cross build s390x (to identify any big endian build issues)
>    * Containers needed for the above
>    * Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
>    * Simple checks - python unittests, DCO check, checkpatch.pl, etc
> 
> This gives about 30 jobs instead of 120 from the "Full" group. It
> is likely reasonable to cut the minimal set down even more, as IMHO
> there are too many system emulator jobs there.
> 
> This is all controlled by defining a set of 'rules' entries that
> apply to (almost) all jobs in the CI pipeline. The exceptions are
> those using custom runners, and the firmware build jobs.
> 
> Using inherited templates together with rules, doesn't work in
> the way you might expect. Any rules defined in the child completely
> replace those from the parent, rather than augmenting them. Thus we
> need to avoid having rules defined in any jobs at all, by inheriting
> from a parent job template that contains the rules. The characteristics
> of individual jobs are then defined by variables set against the job.
> This is described in the updated content for the docs/devel/ci-jobs.rst
> file.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/buildtest-template.yml  |  17 +--
>   .gitlab-ci.d/buildtest.yml           |  43 ++++----
>   .gitlab-ci.d/cirrus.yml              |   6 +-
>   .gitlab-ci.d/container-core.yml      |   4 +
>   .gitlab-ci.d/container-cross.yml     |   9 +-
>   .gitlab-ci.d/container-template.yml  |   1 +
>   .gitlab-ci.d/containers.yml          |   4 +
>   .gitlab-ci.d/crossbuild-template.yml |   3 +
>   .gitlab-ci.d/crossbuilds.yml         |   3 +
>   .gitlab-ci.d/qemu-project.yml        |   1 +
>   .gitlab-ci.d/rules.yml               | 116 ++++++++++++++++++++
>   .gitlab-ci.d/static_checks.yml       |  18 ++--
>   .gitlab-ci.yml                       |   4 -
>   docs/devel/ci-jobs.rst               | 155 ++++++++++++++++++++++++---
>   14 files changed, 317 insertions(+), 67 deletions(-)
>   create mode 100644 .gitlab-ci.d/rules.yml
> 
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index fcbcc4e627..8578b3ed11 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -1,4 +1,6 @@
> +
>   .native_build_job_template:
> +  extends: .job_rules
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     before_script:
> @@ -27,6 +29,7 @@
>         fi
>   
>   .native_test_job_template:
> +  extends: .job_rules
>     stage: test
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     script:
> @@ -67,15 +70,5 @@
>     after_script:
>       - cd build
>       - du -chs ${CI_PROJECT_DIR}/avocado-cache
> -  rules:
> -    # Only run these jobs if running on the mainstream namespace,
> -    # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either
> -    # in its namespace setting or via git-push option, see documentation
> -    # in /.gitlab-ci.yml of this repository).
> -    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
> -      when: on_success
> -    - if: '$QEMU_CI_AVOCADO_TESTING'
> -      when: on_success
> -    # Otherwise, set to manual (the jobs are created but not run).
> -    - when: manual
> -      allow_failure: true
> +  variables:
> +    QEMU_JOB_GATING: 1
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 903ee65f32..c8e03373c5 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -40,6 +40,7 @@ build-system-ubuntu:
>     needs:
>       job: amd64-ubuntu2004-container
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: ubuntu2004
>       CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-slirp=system
>       TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
> @@ -56,6 +57,7 @@ check-system-ubuntu:
>       - job: build-system-ubuntu
>         artifacts: true
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: ubuntu2004
>       MAKE_CHECK_ARGS: check
>   
> @@ -73,6 +75,7 @@ build-system-debian:
>     needs:
>       job: amd64-debian-container
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: debian-amd64
>       CONFIGURE_ARGS: --enable-fdt=system
>       TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
> @@ -89,6 +92,7 @@ check-system-debian:
>       - job: build-system-debian
>         artifacts: true
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: debian-amd64
>       MAKE_CHECK_ARGS: check
>   
> @@ -106,6 +110,7 @@ build-system-fedora:
>     needs:
>       job: amd64-fedora-container
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: fedora
>       CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs
>                --enable-fdt=system --enable-slirp=system --enable-capstone=system
> @@ -123,6 +128,7 @@ check-system-fedora:
>       - job: build-system-fedora
>         artifacts: true
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: fedora
>       MAKE_CHECK_ARGS: check
>   
> @@ -140,6 +146,7 @@ build-system-centos:
>     needs:
>       job: amd64-centos8-container
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: centos8
>       CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
>                       --enable-modules --enable-trace-backends=dtrace
> @@ -157,6 +164,7 @@ check-system-centos:
>       - job: build-system-centos
>         artifacts: true
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: centos8
>       MAKE_CHECK_ARGS: check
>   
> @@ -325,6 +333,7 @@ build-user-static:
>     needs:
>       job: amd64-debian-user-cross-container
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: debian-all-test-cross
>       CONFIGURE_ARGS: --disable-tools --disable-system --static
>       MAKE_CHECK_ARGS: check-tcg
> @@ -411,17 +420,15 @@ build-cfi-aarch64:
>         --enable-safe-stack --enable-slirp=git
>       TARGETS: aarch64-softmmu
>       MAKE_CHECK_ARGS: check-build
> +    # FIXME: This job is often failing, likely due to out-of-memory problems in
> +    # the constrained containers of the shared runners. Thus this is marked as
> +    # manual until the situation has been solved.
> +    QEMU_JOB_MANUAL: 1
>     timeout: 70m
>     artifacts:
>       expire_in: 2 days
>       paths:
>         - build
> -  rules:
> -    # FIXME: This job is often failing, likely due to out-of-memory problems in
> -    # the constrained containers of the shared runners. Thus this is marked as
> -    # manual until the situation has been solved.
> -    - when: manual
> -      allow_failure: true
>   
>   check-cfi-aarch64:
>     extends: .native_test_job_template
> @@ -453,17 +460,15 @@ build-cfi-ppc64-s390x:
>         --enable-safe-stack --enable-slirp=git
>       TARGETS: ppc64-softmmu s390x-softmmu
>       MAKE_CHECK_ARGS: check-build
> +    # FIXME: This job is often failing, likely due to out-of-memory problems in
> +    # the constrained containers of the shared runners. Thus this is marked as
> +    # manual until the situation has been solved.
> +    QEMU_JOB_MANUAL: 1
>     timeout: 70m
>     artifacts:
>       expire_in: 2 days
>       paths:
>         - build
> -  rules:
> -    # FIXME: This job is often failing, likely due to out-of-memory problems in
> -    # the constrained containers of the shared runners. Thus this is marked as
> -    # manual until the situation has been solved.
> -    - when: manual
> -      allow_failure: true
>   
>   check-cfi-ppc64-s390x:
>     extends: .native_test_job_template
> @@ -657,6 +662,7 @@ build-without-default-features:
>       MAKE_CHECK_ARGS: check-unit
>   
>   build-libvhost-user:
> +  extends: .job_rules
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
>     needs:
> @@ -676,6 +682,8 @@ build-tools-and-docs-debian:
>     needs:
>       job: amd64-debian-container
>     variables:
> +    QEMU_JOB_MINIMAL: 1
> +    QEMU_JOB_PUBLISH: 1
>       IMAGE: debian-amd64
>       MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
>       CONFIGURE_ARGS: --disable-system --disable-user --enable-docs --enable-tools
> @@ -698,10 +706,14 @@ build-tools-and-docs-debian:
>   # that users can see the results of their commits, regardless
>   # of what topic branch they're currently using
>   pages:
> +  extends: .native_build_job_template
>     image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
>     stage: test
>     needs:
>       - job: build-tools-and-docs-debian
> +  variables:
> +    QEMU_JOB_MINIMAL: 1
> +    QEMU_JOB_PUBLISH: 1
>     script:
>       - mkdir -p public
>       # HTML-ised source tree
> @@ -715,10 +727,3 @@ pages:
>     artifacts:
>       paths:
>         - public
> -  rules:
> -    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
> -      when: on_success
> -    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
> -      when: never
> -    - if: '$CI_PROJECT_NAMESPACE != "qemu-project"'
> -      when: on_success
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> index 675db69622..4334736796 100644
> --- a/.gitlab-ci.d/cirrus.yml
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -11,6 +11,7 @@
>   # special care, because we can't just override it at the GitLab CI job
>   # definition level or we risk breaking it completely.
>   .cirrus_build_job:
> +  extends: .job_rules
>     stage: build
>     image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
>     needs: []
> @@ -39,8 +40,6 @@
>         <.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
>       - cat .gitlab-ci.d/cirrus/$NAME.yml
>       - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> -  rules:
> -    - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
>   
>   x64-freebsd-12-build:
>     extends: .cirrus_build_job
> @@ -61,6 +60,8 @@ x64-freebsd-12-build:
>   x64-freebsd-13-build:
>     extends: .cirrus_build_job
>     variables:
> +    QEMU_JOB_SOFT_FAIL: 1
> +    QEMU_JOB_MINIMAL: 1
>       NAME: freebsd-13
>       CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
>       CIRRUS_VM_IMAGE_SELECTOR: image_family
> @@ -74,6 +75,7 @@ x64-freebsd-13-build:
>   x64-macos-11-base-build:
>     extends: .cirrus_build_job
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       NAME: macos-11
>       CIRRUS_VM_INSTANCE_TYPE: osx_instance
>       CIRRUS_VM_IMAGE_SELECTOR: image
> diff --git a/.gitlab-ci.d/container-core.yml b/.gitlab-ci.d/container-core.yml
> index e8dd1f476a..8983c32b3b 100644
> --- a/.gitlab-ci.d/container-core.yml
> +++ b/.gitlab-ci.d/container-core.yml
> @@ -4,14 +4,18 @@ include:
>   amd64-centos8-container:
>     extends: .container_job_template
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       NAME: centos8
>   
>   amd64-fedora-container:
>     extends: .container_job_template
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       NAME: fedora
>   
>   amd64-debian10-container:
>     extends: .container_job_template
>     variables:
> +    QEMU_JOB_MINIMAL: 1
> +    QEMU_JOB_PUBLISH: 1
>       NAME: debian10
> diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
> index 0fcebe363a..ef316668eb 100644
> --- a/.gitlab-ci.d/container-cross.yml
> +++ b/.gitlab-ci.d/container-cross.yml
> @@ -50,13 +50,11 @@ armhf-debian-cross-container:
>   # We never want to build hexagon in the CI system and by default we
>   # always want to refer to the master registry where it lives.
>   hexagon-cross-container:
> +  extends: .job_rules
>     image: docker:stable
>     stage: containers
> -  rules:
> -    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
> -      when: never
> -    - when: always
>     variables:
> +    QEMU_JOB_MANUAL: 1
>       NAME: debian-hexagon-cross
>       GIT_DEPTH: 1
>     services:
> @@ -143,6 +141,7 @@ s390x-debian-cross-container:
>     stage: containers-layer2
>     needs: ['amd64-debian10-container']
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       NAME: debian-s390x-cross
>   
>   sh4-debian-cross-container:
> @@ -179,11 +178,13 @@ cris-fedora-cross-container:
>   i386-fedora-cross-container:
>     extends: .container_job_template
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       NAME: fedora-i386-cross
>   
>   win32-fedora-cross-container:
>     extends: .container_job_template
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       NAME: fedora-win32-cross
>   
>   win64-fedora-cross-container:
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index 1baecd9460..e2954e4ed3 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -1,4 +1,5 @@
>   .container_job_template:
> +  extends: .job_rules
>     image: docker:stable
>     stage: containers
>     services:
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> index cd06d3f5f4..6948923892 100644
> --- a/.gitlab-ci.d/containers.yml
> +++ b/.gitlab-ci.d/containers.yml
> @@ -17,6 +17,8 @@ amd64-debian-container:
>     stage: containers-layer2
>     needs: ['amd64-debian10-container']
>     variables:
> +    QEMU_JOB_MINIMAL: 1
> +    QEMU_JOB_PUBLISH: 1
>       NAME: debian-amd64
>   
>   amd64-ubuntu1804-container:
> @@ -27,6 +29,7 @@ amd64-ubuntu1804-container:
>   amd64-ubuntu2004-container:
>     extends: .container_job_template
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       NAME: ubuntu2004
>   
>   amd64-ubuntu-container:
> @@ -42,4 +45,5 @@ amd64-opensuse-leap-container:
>   python-container:
>     extends: .container_job_template
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       NAME: python
> diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
> index 7d3ad00a1e..997ec02f79 100644
> --- a/.gitlab-ci.d/crossbuild-template.yml
> +++ b/.gitlab-ci.d/crossbuild-template.yml
> @@ -1,4 +1,5 @@
>   .cross_system_build_job:
> +  extends: .job_rules
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     timeout: 80m
> @@ -23,6 +24,7 @@
>   # KVM), and set extra options (such disabling other accelerators) via the
>   # $EXTRA_CONFIGURE_OPTS variable.
>   .cross_accel_build_job:
> +  extends: .job_rules
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     timeout: 30m
> @@ -35,6 +37,7 @@
>       - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
>   
>   .cross_user_build_job:
> +  extends: .job_rules
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     script:
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> index 4ff3aa3cfc..9fcd2ac567 100644
> --- a/.gitlab-ci.d/crossbuilds.yml
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -48,6 +48,7 @@ cross-i386-system:
>     needs:
>       job: i386-fedora-cross-container
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: fedora-i386-cross
>       MAKE_CHECK_ARGS: check-qtest
>   
> @@ -129,6 +130,7 @@ cross-s390x-system:
>     needs:
>       job: s390x-debian-cross-container
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: debian-s390x-cross
>   
>   cross-s390x-user:
> @@ -159,6 +161,7 @@ cross-win32-system:
>     needs:
>       job: win32-fedora-cross-container
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       IMAGE: fedora-win32-cross
>     artifacts:
>       paths:
> diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
> index b3d79bc429..7007193fc0 100644
> --- a/.gitlab-ci.d/qemu-project.yml
> +++ b/.gitlab-ci.d/qemu-project.yml
> @@ -2,6 +2,7 @@
>   # https://gitlab.com/qemu-project/qemu/-/pipelines
>   
>   include:
> +  - local: '/.gitlab-ci.d/rules.yml'
>     - local: '/.gitlab-ci.d/stages.yml'
>     - local: '/.gitlab-ci.d/edk2.yml'
>     - local: '/.gitlab-ci.d/opensbi.yml'
> diff --git a/.gitlab-ci.d/rules.yml b/.gitlab-ci.d/rules.yml
> new file mode 100644
> index 0000000000..3399722ca9
> --- /dev/null
> +++ b/.gitlab-ci.d/rules.yml
> @@ -0,0 +1,116 @@
> +
> +# This defines rules used to control individual job execution
> +# See docs/devel/ci-jobs.rst for an explanation of the various
> +# variables and branch naming conventions applied here.
> +#
> +.job_rules:
> +  rules:
> +    # ======================================================================
> +    # Rules that apply regardless of whether the primary qemu repo or a fork
> +    # ======================================================================
> +
> +    # Skip any cirrus job if either repo or api token are missing
> +    # as we won't be able to talk to cirrus
> +    - if: '$CIRRUS_VM_INSTANCE_TYPE && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
> +      when: never
> +
> +    # Any jobs marked as manual, never get automatically run in any scenario
> +    # and don't contribute to pipeline status
> +    - if: '$QEMU_JOB_MANUAL'
> +      when: manual
> +      allow_failure: true
> +
> +    # For the main repo, tags represent official releases.
> +    # The branch associated with the release will have passed
> +    # a CI pipeline already
> +    #
> +    # For user forks, tags are tyically added by tools like
> +    # git-publish at the same time as pushing the branch prior
> +    # to sending out for review
> +    #
> +    # In neither case do we wish to run CI pipelines for tags
> +    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_TAG'
> +      when: never
> +
> +
> +    # ==================================================
> +    # Rules to control what happens in primary qemu repo
> +    # ==================================================
> +
> +    # The 'pages' job must never run on a non-default branch as we don't
> +    # want to publish the website from pre-merged content
> +    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_JOB_NAME == "pages"'
> +      when: never
> +
> +    # Run all jobs for main repo staging branch
> +    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == "staging"'
> +      when: on_success
> +
> +    # Run jobs needed to support website publishing on the main branch
> +    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $QEMU_JOB_PUBLISH'
> +      when: on_success
> +
> +    # Prevent jobs from running on pushes to any other branch / tag
> +    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != "staging"'
> +      when: never
> +
> +
> +    # Remainining pipeline sources follow the same rules as for fork repos
> +
> +
> +
> +    # ====================================
> +    # Rules for running jobs in user forks
> +    # ====================================
> +
> +    # Part 1: gating jobs
> +    # -------------------
> +
> +    # If on a branch with name prefix 'ci-acceptance-', then run
> +    # everything, just as a gating job on 'staging' branch would
> +    - if: '$CI_COMMIT_BRANCH =~ /^ci-gating-/'
> +      when: on_success
> +
> +    # If user set QEMU_CI_GATING=1, then run everything just as
> +    # a gating job on 'staging' branch would
> +    - if: '$QEMU_CI_GATING'
> +      when: on_success
> +
> +    # Otherwise never run jobs marked as gating, but allow manual trigger
> +    # without affecting pipeline status
> +    - if: '$QEMU_JOB_GATING'
> +      when: manual
> +      allow_failure: true
> +
> +
> +    # =============================================
> +    # Part 2: opt-in for all CI, except gating jobs
> +    # =============================================
> +
> +    # If pushing to a branch with name prefix 'ci-all', then run all jobs
> +    - if: '$CI_COMMIT_BRANCH =~ /^ci-all/'
> +      when: on_success
> +
> +    # If user QEMU_CI_ALL=1, then run all jobs
> +    - if: '$QEMU_CI_ALL'
> +      when: on_success
> +
> +
> +    # ===============================
> +    # Part 3: the minimal set of jobs
> +    # ===============================
> +
> +    # If pushing to a branch with name prefix 'ci-min', then run
> +    # only a minimal set of jobs
> +    - if: '$CI_COMMIT_BRANCH =~ /^ci-min/ && $QEMU_JOB_MINIMAL'
> +      when: on_success
> +
> +    # If user set QEMU_CI=1, then run only a minimal set of jobs
> +    - if: '$QEMU_CI && $QEMU_JOB_MINIMAL'
> +      when: on_success
> +
> +    # Any other scenario, let the jobs be manually triggered only.
> +    # Set to be non-fatal, so that pending manual jobs don't
> +    # affect overall pipeline status
> +    - when: manual
> +      allow_failure: true
> diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
> index 96dbd9e310..4ba3e038dd 100644
> --- a/.gitlab-ci.d/static_checks.yml
> +++ b/.gitlab-ci.d/static_checks.yml
> @@ -1,4 +1,5 @@
>   check-patch:
> +  extends: .job_rules
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
>     needs:
> @@ -6,42 +7,41 @@ check-patch:
>     script:
>       - .gitlab-ci.d/check-patch.py
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       GIT_DEPTH: 1000
> -  rules:
> -    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
> -      when: never
> -    - when: on_success
> -      allow_failure: true
> +  allow_failure: true
>   
>   check-dco:
> +  extends: .job_rules
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
>     needs:
>       job: amd64-centos8-container
>     script: .gitlab-ci.d/check-dco.py
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       GIT_DEPTH: 1000
> -  rules:
> -    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
> -      when: never
> -    - when: on_success
>   
>   check-python-pipenv:
> +  extends: .job_rules
>     stage: test
>     image: $CI_REGISTRY_IMAGE/qemu/python:latest
>     script:
>       - make -C python check-pipenv
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       GIT_DEPTH: 1
>     needs:
>       job: python-container
>   
>   check-python-tox:
> +  extends: .job_rules
>     stage: test
>     image: $CI_REGISTRY_IMAGE/qemu/python:latest
>     script:
>       - make -C python check-tox
>     variables:
> +    QEMU_JOB_MINIMAL: 1
>       GIT_DEPTH: 1
>       QEMU_TOX_EXTRA_ARGS: --skip-missing-interpreters=false
>     needs:
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 9762dda2ee..d5bd31f294 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -15,10 +15,6 @@
>   #
>   # QEMU CI jobs are based on templates. Some templates provide
>   # user-configurable options, modifiable via configuration variables.
> -#
> -# See https://qemu-project.gitlab.io/qemu/devel/ci.html#custom-ci-cd-variables
> -# for more information.
> -#
>   
>   include:
>     - local: '/.gitlab-ci.d/qemu-project.yml'
> diff --git a/docs/devel/ci-jobs.rst b/docs/devel/ci-jobs.rst
> index 9cd9819786..6d3f13d495 100644
> --- a/docs/devel/ci-jobs.rst
> +++ b/docs/devel/ci-jobs.rst
> @@ -1,22 +1,49 @@
> -Custom CI/CD variables
> -======================
> +CI Job Rules
> +============
>   
> -QEMU CI pipelines can be tuned by setting some CI environment variables.
> +The following documents how CI rules control execution of jobs in a pipeline
>   
> -Set variable globally in the user's CI namespace
> -------------------------------------------------
> +Job variable naming
> +-------------------
>   
> -Variables can be set globally in the user's CI namespace setting.
> +The rules for controlling jobs in a pipeline will be driven by a number of
> +variables:
> +
> + - ``CI_XXX``, ``GITLAB_XXX`` - variables defined by GitLab:
> +
> +   https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
> +
> + - ``QEMU_JOB_XXX`` - variables defined in the YAML files that express
> +   characteristics of the job used to control default behaviour
> +
> + - ``QEMU_CI_XXX`` - variables defined by the user that are used to fine
> +   tune which jobs are run dynamically
> +
> +and in some cases based on the branch name prefixes.
> +
> +Job fine tuning strategies
> +--------------------------
> +
> +Using a combination of the ``QEMU_CI_XXX`` variables and ``ci-``
> +branch name prefix, users can fine tune what jobs are run.
> +
> +Set variable globally in the gitlab repository CI config
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Variables can be set globally in the user's gitlab repository CI config.
> +These variables will apply to all pipelines associated with the repository
> +thereafter. This is useful for fine tuning the jobs indefinitely.
>   
>   For further information about how to set these variables, please refer to::
>   
>     https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
>   
> -Set variable manually when pushing a branch or tag to the user's repository
> ----------------------------------------------------------------------------
> +Set variable manually when pushing to git
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>   
>   Variables can be set manually when pushing a branch or tag, using
> -git-push command line arguments.
> +git-push command line arguments. This is useful for fine tuning the
> +jobs on an adhoc basis.
>   
>   Example setting the QEMU_CI_EXAMPLE_VAR variable:
>   
> @@ -28,13 +55,107 @@ For further information about how to set these variables, please refer to::
>   
>     https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
>   
> -Here is a list of the most used variables:
> +Pushing to specially named branches
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Branch names starting with a 'ci-' prefix can be used as an alternative
> +to setting variables. Details of supported branch names are detailed
> +later in this document.
> +
> +Provide a custom gitlab CI configuration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The above strategies all provide a way to fine tune the jobs defined by the
> +standard QEMU gitlab CI configuration file. If this is not sufficient it is
> +possible to completely replace the CI configuration file with a custom
> +version. This allows the contributor to achieve essentially anything they
> +desire, within the scope of what GitLab supports.
> +
> +Replacing the ``.gitlab-ci.yml`` file is done in the repository settings:
> +
> +  https://docs.gitlab.com/ee/ci/pipelines/settings.html#specify-a-custom-cicd-configuration-file
> +
> +Note that it is possible to reference an arbitrary URL. This could point
> +to a replacement .gitlab-ci.yml on a specific branch in the same repo,
> +or can point to any external site.
> +
> +
> +Job grouping sets
> +-----------------
> +
> +There are many different jobs defined in the GitLab CI pipeline used by QEMU.
> +It is not practical to run all jobs in every scenario, so there are a set of
> +rules defined that control which jobs are executed on each pipeline run. At
> +a high level the jobs can be grouped into a number of mutually exclusive
> +sets.
> +
> + - Manual jobs
> +
> +   This is a set of jobs that will never be run automatically in any scenario.
> +   The reason a job will be marked manual is if it is known to exhibit some
> +   non-deterministic failures, or liable to trigger timeouts. Ideally jobs are
> +   only present in this set for a short period of time while the problems are
> +   investigated and resolved. Users can manually trigger these jobs from the
> +   the pipelines web UI if desired, but they will never contribute to the
> +   overall pipeline status.
> +
> +   Identified by the variable ``QEMU_JOB_MANUAL: 1``
> +
> +
> + - Minimal jobs
> +
> +   This is a minimal set of jobs that give a reasonable build and test sanity
> +   check, which will be run by default in all possible scenarios. This set of
> +   jobs is intended to be controlled fairly strictly to avoid wasting CI
> +   minutes of maintainers/contributors. The intent is to get 80-90% coverage
> +   of build and unit tests across the most important host platforms and
> +   architectures.
> +
> +   Identified by the variable ``QEMU_JOB_MINIMAL: 1``
> +
> +   Run by setting the variable ``QEMU_CI: 1`` or pushing to a branch
> +   named ``ci-min-XXX``.
> +
> +
> + - Gating jobs
> +
> +   This is a set of jobs will will always run as a gating test prior to code
> +   merging into the default branch of the primary QEMU git repository. In user
> +   forks the jobs will not run by default, but the user can opt-in to trigger
> +   their execution. These jobs may run a particularly thorough set of scenarios
> +   that maintainers are not normally going to exercise before sending series.
> +
> +   Identified by the variable ``QEMU_JOB_GATING: 1``
> +
> +   Run by setting the variable ``QEMU_CI_GATING: 1`` or pushing to a branch
> +   named ``ci-gating-XXX``
> +
> +
> + - Optional jobs
> +
> +   This is all remaining jobs that don't fall into one of the above buckets.
> +   These jobs will always be run as a gating test prior to code merging into
> +   the default branch of the primary QEMU git repository. In user forks the
> +   jobs will not run by default, but the user can opt-in to trigger their
> +   execution. These jobs will aim to expand the host/platform build coverage
> +   gaps left by the default jobs.
> +
> +   Identified by not having any of the above variables set.
> +
> +   Run by setting the variable ``QEMU_CI_ALL: 1`` or pushing to a branch
> +   named ``ci-all-XXXX``
> +
> +Since the sets are mutually exclusive, if more than one of the variables above
> +is set, then only one of them will be honoured. The precedence matches the
> +ordering of the above docs.
> +
> +In addition to the above, there are some special extra sets that can augment
> +any of the above sets
> +
> + - Publishing jobs
>   
> -QEMU_CI_AVOCADO_TESTING
> -~~~~~~~~~~~~~~~~~~~~~~~
> -By default, tests using the Avocado framework are not run automatically in
> -the pipelines (because multiple artifacts have to be downloaded, and if
> -these artifacts are not already cached, downloading them make the jobs
> -reach the timeout limit). Set this variable to have the tests using the
> -Avocado framework run automatically.
> +   These are jobs that are needed to publish the QEMU website. For user fork
> +   repos these follow the normal rules set out above. For the primary QEMU
> +   repo, however, the jobs will also run on the default branch.
>   
> +   Identified by the variable ``QEMU_JOB_PUBLISH: 1``

Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Daniel P. Berrangé]

On Tue, May 10, 2022 at 10:51:09AM +0200, Thomas Huth wrote:
> On 12/08/2021 20.04, Daniel P. Berrangé wrote:
> > Currently pushes to user forks run the same set of build / test jobs as
> > pushes to the main repo. This results in creation of 120+ individual
> > jobs. While it is useful for subsystem maintainers, and even regular
> > contributors to be able to run the full set of jobs, it is wasteful to
> > run all of them all the time.
> 
>  Hi Daniel!
> 
> Now that gitlab introduces limits for the CI for all users, I think we
> should revisit this topic ... could you maybe respin your patch to the
> current git master branch?

Yes, I really do need to revisit this patch.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH 0/2] docs: add definitions of terms for CI/testing

[Willian Rampazzo]

To understand the current state of QEMU CI/testing and have a base to
discuss the plans for the future, it is important to define some usual
terms. This patch defines the terms for "Automated tests", "Unit
testing", "Functional testing", "System testing", "Flaky tests",
"Gating", and "Continuous Integration".

The first patch was borrowed from
20210812180403.4129067-1-berrange@redhat.com.

Signed-off-by: Willian Rampazzo <willianr@redhat.com>

Daniel P. Berrangé (1):
  docs: split the CI docs into two files

Willian Rampazzo (1):
  docs: add definitions of terms for CI/testing

 docs/devel/ci-definitions.rst | 121 +++++++++++++++++++++++++
 docs/devel/ci-jobs.rst        |  40 +++++++++
 docs/devel/ci-runners.rst     | 117 +++++++++++++++++++++++++
 docs/devel/ci.rst             | 160 +---------------------------------
 4 files changed, 281 insertions(+), 157 deletions(-)
 create mode 100644 docs/devel/ci-definitions.rst
 create mode 100644 docs/devel/ci-jobs.rst
 create mode 100644 docs/devel/ci-runners.rst

-- 
2.31.1

[PATCH 1/2] docs: split the CI docs into two files

[Willian Rampazzo]

From: Daniel P. Berrangé <berrange@redhat.com>

This splits the CI docs into one file talking about job setup and usage
and another file describing provisioning of custom runners.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210812180403.4129067-2-berrange@redhat.com>
---
 docs/devel/ci-jobs.rst    |  40 ++++++++++
 docs/devel/ci-runners.rst | 117 ++++++++++++++++++++++++++++
 docs/devel/ci.rst         | 159 +-------------------------------------
 3 files changed, 159 insertions(+), 157 deletions(-)
 create mode 100644 docs/devel/ci-jobs.rst
 create mode 100644 docs/devel/ci-runners.rst

diff --git a/docs/devel/ci-jobs.rst b/docs/devel/ci-jobs.rst
new file mode 100644
index 0000000000..9cd9819786
--- /dev/null
+++ b/docs/devel/ci-jobs.rst
@@ -0,0 +1,40 @@
+Custom CI/CD variables
+======================
+
+QEMU CI pipelines can be tuned by setting some CI environment variables.
+
+Set variable globally in the user's CI namespace
+------------------------------------------------
+
+Variables can be set globally in the user's CI namespace setting.
+
+For further information about how to set these variables, please refer to::
+
+  https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
+
+Set variable manually when pushing a branch or tag to the user's repository
+---------------------------------------------------------------------------
+
+Variables can be set manually when pushing a branch or tag, using
+git-push command line arguments.
+
+Example setting the QEMU_CI_EXAMPLE_VAR variable:
+
+.. code::
+
+   git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch
+
+For further information about how to set these variables, please refer to::
+
+  https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
+
+Here is a list of the most used variables:
+
+QEMU_CI_AVOCADO_TESTING
+~~~~~~~~~~~~~~~~~~~~~~~
+By default, tests using the Avocado framework are not run automatically in
+the pipelines (because multiple artifacts have to be downloaded, and if
+these artifacts are not already cached, downloading them make the jobs
+reach the timeout limit). Set this variable to have the tests using the
+Avocado framework run automatically.
+
diff --git a/docs/devel/ci-runners.rst b/docs/devel/ci-runners.rst
new file mode 100644
index 0000000000..7817001fb2
--- /dev/null
+++ b/docs/devel/ci-runners.rst
@@ -0,0 +1,117 @@
+Jobs on Custom Runners
+======================
+
+Besides the jobs run under the various CI systems listed before, there
+are a number additional jobs that will run before an actual merge.
+These use the same GitLab CI's service/framework already used for all
+other GitLab based CI jobs, but rely on additional systems, not the
+ones provided by GitLab as "shared runners".
+
+The architecture of GitLab's CI service allows different machines to
+be set up with GitLab's "agent", called gitlab-runner, which will take
+care of running jobs created by events such as a push to a branch.
+Here, the combination of a machine, properly configured with GitLab's
+gitlab-runner, is called a "custom runner".
+
+The GitLab CI jobs definition for the custom runners are located under::
+
+  .gitlab-ci.d/custom-runners.yml
+
+Custom runners entail custom machines.  To see a list of the machines
+currently deployed in the QEMU GitLab CI and their maintainers, please
+refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
+
+Machine Setup Howto
+-------------------
+
+For all Linux based systems, the setup can be mostly automated by the
+execution of two Ansible playbooks.  Create an ``inventory`` file
+under ``scripts/ci/setup``, such as this::
+
+  fully.qualified.domain
+  other.machine.hostname
+
+You may need to set some variables in the inventory file itself.  One
+very common need is to tell Ansible to use a Python 3 interpreter on
+those hosts.  This would look like::
+
+  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
+  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
+
+Build environment
+~~~~~~~~~~~~~~~~~
+
+The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
+set up machines with the environment needed to perform builds and run
+QEMU tests.  This playbook consists on the installation of various
+required packages (and a general package update while at it).  It
+currently covers a number of different Linux distributions, but it can
+be expanded to cover other systems.
+
+The minimum required version of Ansible successfully tested in this
+playbook is 2.8.0 (a version check is embedded within the playbook
+itself).  To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory build-environment.yml
+
+Please note that most of the tasks in the playbook require superuser
+privileges, such as those from the ``root`` account or those obtained
+by ``sudo``.  If necessary, please refer to ``ansible-playbook``
+options such as ``--become``, ``--become-method``, ``--become-user``
+and ``--ask-become-pass``.
+
+gitlab-runner setup and registration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gitlab-runner agent needs to be installed on each machine that
+will run jobs.  The association between a machine and a GitLab project
+happens with a registration token.  To find the registration token for
+your repository/project, navigate on GitLab's web UI to:
+
+ * Settings (the gears-like icon at the bottom of the left hand side
+   vertical toolbar), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * Under "Set up a specific Runner manually", look for the value under
+   "And this registration token:"
+
+Copy the ``scripts/ci/setup/vars.yml.template`` file to
+``scripts/ci/setup/vars.yml``.  Then, set the
+``gitlab_runner_registration_token`` variable to the value obtained
+earlier.
+
+To run the playbook, execute::
+
+  cd scripts/ci/setup
+  ansible-playbook -i inventory gitlab-runner.yml
+
+Following the registration, it's necessary to configure the runner tags,
+and optionally other configurations on the GitLab UI.  Navigate to:
+
+ * Settings (the gears like icon), then
+ * CI/CD, then
+ * Runners, and click on the "Expand" button, then
+ * "Runners activated for this project", then
+ * Click on the "Edit" icon (next to the "Lock" Icon)
+
+Tags are very important as they are used to route specific jobs to
+specific types of runners, so it's a good idea to double check that
+the automatically created tags are consistent with the OS and
+architecture.  For instance, an Ubuntu 20.04 aarch64 system should
+have tags set as::
+
+  ubuntu_20.04,aarch64
+
+Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
+would contain::
+
+  ubuntu-20.04-aarch64-all:
+   tags:
+   - ubuntu_20.04
+   - aarch64
+
+It's also recommended to:
+
+ * increase the "Maximum job timeout" to something like ``2h``
+ * give it a better Description
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst
index 205572510c..a6a650968b 100644
--- a/docs/devel/ci.rst
+++ b/docs/devel/ci.rst
@@ -8,160 +8,5 @@ found at::
 
    https://wiki.qemu.org/Testing/CI
 
-Custom CI/CD variables
-======================
-
-QEMU CI pipelines can be tuned by setting some CI environment variables.
-
-Set variable globally in the user's CI namespace
-------------------------------------------------
-
-Variables can be set globally in the user's CI namespace setting.
-
-For further information about how to set these variables, please refer to::
-
-  https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
-
-Set variable manually when pushing a branch or tag to the user's repository
----------------------------------------------------------------------------
-
-Variables can be set manually when pushing a branch or tag, using
-git-push command line arguments.
-
-Example setting the QEMU_CI_EXAMPLE_VAR variable:
-
-.. code::
-
-   git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch
-
-For further information about how to set these variables, please refer to::
-
-  https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
-
-Here is a list of the most used variables:
-
-QEMU_CI_AVOCADO_TESTING
-~~~~~~~~~~~~~~~~~~~~~~~
-By default, tests using the Avocado framework are not run automatically in
-the pipelines (because multiple artifacts have to be downloaded, and if
-these artifacts are not already cached, downloading them make the jobs
-reach the timeout limit). Set this variable to have the tests using the
-Avocado framework run automatically.
-
-Jobs on Custom Runners
-======================
-
-Besides the jobs run under the various CI systems listed before, there
-are a number additional jobs that will run before an actual merge.
-These use the same GitLab CI's service/framework already used for all
-other GitLab based CI jobs, but rely on additional systems, not the
-ones provided by GitLab as "shared runners".
-
-The architecture of GitLab's CI service allows different machines to
-be set up with GitLab's "agent", called gitlab-runner, which will take
-care of running jobs created by events such as a push to a branch.
-Here, the combination of a machine, properly configured with GitLab's
-gitlab-runner, is called a "custom runner".
-
-The GitLab CI jobs definition for the custom runners are located under::
-
-  .gitlab-ci.d/custom-runners.yml
-
-Custom runners entail custom machines.  To see a list of the machines
-currently deployed in the QEMU GitLab CI and their maintainers, please
-refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
-
-Machine Setup Howto
--------------------
-
-For all Linux based systems, the setup can be mostly automated by the
-execution of two Ansible playbooks.  Create an ``inventory`` file
-under ``scripts/ci/setup``, such as this::
-
-  fully.qualified.domain
-  other.machine.hostname
-
-You may need to set some variables in the inventory file itself.  One
-very common need is to tell Ansible to use a Python 3 interpreter on
-those hosts.  This would look like::
-
-  fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
-  other.machine.hostname ansible_python_interpreter=/usr/bin/python3
-
-Build environment
-~~~~~~~~~~~~~~~~~
-
-The ``scripts/ci/setup/build-environment.yml`` Ansible playbook will
-set up machines with the environment needed to perform builds and run
-QEMU tests.  This playbook consists on the installation of various
-required packages (and a general package update while at it).  It
-currently covers a number of different Linux distributions, but it can
-be expanded to cover other systems.
-
-The minimum required version of Ansible successfully tested in this
-playbook is 2.8.0 (a version check is embedded within the playbook
-itself).  To run the playbook, execute::
-
-  cd scripts/ci/setup
-  ansible-playbook -i inventory build-environment.yml
-
-Please note that most of the tasks in the playbook require superuser
-privileges, such as those from the ``root`` account or those obtained
-by ``sudo``.  If necessary, please refer to ``ansible-playbook``
-options such as ``--become``, ``--become-method``, ``--become-user``
-and ``--ask-become-pass``.
-
-gitlab-runner setup and registration
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The gitlab-runner agent needs to be installed on each machine that
-will run jobs.  The association between a machine and a GitLab project
-happens with a registration token.  To find the registration token for
-your repository/project, navigate on GitLab's web UI to:
-
- * Settings (the gears-like icon at the bottom of the left hand side
-   vertical toolbar), then
- * CI/CD, then
- * Runners, and click on the "Expand" button, then
- * Under "Set up a specific Runner manually", look for the value under
-   "And this registration token:"
-
-Copy the ``scripts/ci/setup/vars.yml.template`` file to
-``scripts/ci/setup/vars.yml``.  Then, set the
-``gitlab_runner_registration_token`` variable to the value obtained
-earlier.
-
-To run the playbook, execute::
-
-  cd scripts/ci/setup
-  ansible-playbook -i inventory gitlab-runner.yml
-
-Following the registration, it's necessary to configure the runner tags,
-and optionally other configurations on the GitLab UI.  Navigate to:
-
- * Settings (the gears like icon), then
- * CI/CD, then
- * Runners, and click on the "Expand" button, then
- * "Runners activated for this project", then
- * Click on the "Edit" icon (next to the "Lock" Icon)
-
-Tags are very important as they are used to route specific jobs to
-specific types of runners, so it's a good idea to double check that
-the automatically created tags are consistent with the OS and
-architecture.  For instance, an Ubuntu 20.04 aarch64 system should
-have tags set as::
-
-  ubuntu_20.04,aarch64
-
-Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
-would contain::
-
-  ubuntu-20.04-aarch64-all:
-   tags:
-   - ubuntu_20.04
-   - aarch64
-
-It's also recommended to:
-
- * increase the "Maximum job timeout" to something like ``2h``
- * give it a better Description
+.. include:: ci-jobs.rst
+.. include:: ci-runners.rst
-- 
2.31.1