[PATCH 0/5] gitlab: improvements to handling of stable staging branches

[PATCH 0/5] gitlab: improvements to handling of stable staging branches

[Daniel P. Berrangé]

We just (re)discovered that our gitlab rules don't work nicely with
pipelines running from stable staging branches. Every pipeline gets
published with the 'latest' tag, whether its the main staging branch
or one of the stable staging branches. If pipelines for multiple
staging branches run concurrently they'll get very confused and end
up using the wrong container content. eg a 8.0 stable job will get
run with a container from the development branch, or vica-verca.

With this series we dynamically change the tag so that the 'staging'
branch still uses 'latest', but the stable 'staging-X.Y' branaches
use a 'staging-X-Y' container tag.

We also let the container tag be set explicitly via the new variable

  QEMU_CI_CONTAINER_TAG

to facilitate CI testing, the new variable

  QEMU_CI_UPSTREAM

can be set to the fork namespace, to allow contributors to run a
pipeline as if their fork were upstream.

Finally 'QEMU_CI=1' is also made to work in upstream, so that it is
possible to re-run selective jobs on staging branches.

Daniel P. Berrangé (5):
  gitlab: centralize the container tag name
  gitlab: allow overriding name of the upstream repository
  gitlab: stable staging branches publish containers in a separate tag
  gitlab: avoid extra pipelines for tags and stable branches
  gitlab: support disabling job auto-run in upstream

 .gitlab-ci.d/base.yml                | 63 ++++++++++++++++++++++++----
 .gitlab-ci.d/buildtest-template.yml  |  4 +-
 .gitlab-ci.d/buildtest.yml           |  4 +-
 .gitlab-ci.d/container-template.yml  |  3 +-
 .gitlab-ci.d/crossbuild-template.yml |  6 +--
 .gitlab-ci.d/static_checks.yml       |  4 +-
 docs/devel/ci-jobs.rst.inc           | 11 +++++
 7 files changed, 78 insertions(+), 17 deletions(-)

-- 
2.40.1

[PATCH 1/5] gitlab: centralize the container tag name

[Daniel P. Berrangé]

We use a fixed container tag of 'latest' so that contributors' forks
don't end up with an ever growing number of containers as they work
on throwaway feature branches.

This fixed tag causes problems running CI upstream in stable staging
branches, however, because the stable staging branch will publish old
container content that clashes with that needed by primary staging
branch. This makes it impossible to reliably run CI pipelines in
parallel in upstream for different staging branches.

This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to
change which tag container publishing uses. Initially it can be set
by contributors as a git push option if they want to override the
default use of 'latest' eg

  git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish

this is useful if contributors need to run pipelines for different
branches concurrently in their forks.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml                | 6 ++++++
 .gitlab-ci.d/buildtest-template.yml  | 4 ++--
 .gitlab-ci.d/buildtest.yml           | 4 ++--
 .gitlab-ci.d/container-template.yml  | 3 ++-
 .gitlab-ci.d/crossbuild-template.yml | 6 +++---
 .gitlab-ci.d/static_checks.yml       | 4 ++--
 docs/devel/ci-jobs.rst.inc           | 5 +++++
 7 files changed, 22 insertions(+), 10 deletions(-)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 2fbb58d2a3..fba9d31cc6 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -1,4 +1,10 @@
 
+variables:
+  # On stable branches this needs changing. Should also be
+  # overridden per pipeline if running pipelines concurrently
+  # for different branches in contributor forks.
+  QEMU_CI_CONTAINER_TAG: latest
+
 # The order of rules defined here is critically important.
 # They are evaluated in order and first match wins.
 #
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index a6cfe9be97..094ae485b8 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,7 +1,7 @@
 .native_build_job_template:
   extends: .base_job_template
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   before_script:
     - JOBS=$(expr $(nproc) + 1)
   script:
@@ -44,7 +44,7 @@
 .common_test_job_template:
   extends: .base_job_template
   stage: test
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   script:
     - scripts/git-submodule.sh update
         $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index bb3650a51c..35867dfe77 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -532,7 +532,7 @@ build-without-defaults:
 build-libvhost-user:
   extends: .base_job_template
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
+  image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG
   needs:
     job: amd64-fedora-container
   script:
@@ -572,7 +572,7 @@ build-tools-and-docs-debian:
 # of what topic branch they're currently using
 pages:
   extends: .base_job_template
-  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
+  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:$QEMU_CI_CONTAINER_TAG
   stage: test
   needs:
     - job: build-tools-and-docs-debian
diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
index 519b8a9482..83c693c807 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -5,7 +5,8 @@
   services:
     - docker:dind
   before_script:
-    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
+    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:$QEMU_CI_CONTAINER_TAG"
+    # Always ':latest' because we always use upstream as a common cache source
     - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
     - apk add python3
     - docker info
diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index 4f93b9e4e5..6efb0d2a54 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -1,7 +1,7 @@
 .cross_system_build_job:
   extends: .base_job_template
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   timeout: 80m
   script:
     - mkdir build
@@ -27,7 +27,7 @@
 .cross_accel_build_job:
   extends: .base_job_template
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   timeout: 30m
   script:
     - mkdir build
@@ -39,7 +39,7 @@
 .cross_user_build_job:
   extends: .base_job_template
   stage: build
-  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   script:
     - mkdir build
     - cd build
diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
index b4cbdbce2a..ad9f426a52 100644
--- a/.gitlab-ci.d/static_checks.yml
+++ b/.gitlab-ci.d/static_checks.yml
@@ -26,7 +26,7 @@ check-dco:
 check-python-minreqs:
   extends: .base_job_template
   stage: test
-  image: $CI_REGISTRY_IMAGE/qemu/python:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:$QEMU_CI_CONTAINER_TAG
   script:
     - make -C python check-minreqs
   variables:
@@ -37,7 +37,7 @@ check-python-minreqs:
 check-python-tox:
   extends: .base_job_template
   stage: test
-  image: $CI_REGISTRY_IMAGE/qemu/python:latest
+  image: $CI_REGISTRY_IMAGE/qemu/python:$QEMU_CI_CONTAINER_TAG
   script:
     - make -C python check-tox
   variables:
diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index 1f28fec0d0..f72537853b 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -70,6 +70,11 @@ in a handful of namespaces
    repository CI settings, or as git push variables, to influence
    which jobs get run in a pipeline
 
+ * QEMU_CI_CONTAINER_TAG - the tag used to publish containers
+   in stage 1, for use by build jobs in stage 2. Defaults to
+   'latest', but if running pipelines for different branches
+   concurrently, it should be overridden per pipeline.
+
  * nnn - other misc variables not falling into the above
    categories, or using different names for historical reasons
    and not yet converted.
-- 
2.40.1

[PATCH 2/5] gitlab: allow overriding name of the upstream repository

[Daniel P. Berrangé]

The CI rules have special logic for what happens in upstream. To enable
contributors who modify CI rules to test this logic, however, they need
to be able to override which repo is considered upstream. This
introduces the 'QEMU_CI_UPSTREAM' variable

  git push gitlab <branch> -o ci.variable=QEMU_CI_UPSTREAM=berrange

to make it look as if my namespace is the actual upstream. Namespace in
this context refers to the path fragement in gitlab URLs that is above
the repository. Typically this will be the contributor's gitlab login
name.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml      | 19 ++++++++++++-------
 docs/devel/ci-jobs.rst.inc |  6 ++++++
 2 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index fba9d31cc6..a1d734267a 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -5,6 +5,11 @@ variables:
   # for different branches in contributor forks.
   QEMU_CI_CONTAINER_TAG: latest
 
+  # For purposes of CI rules, upstream is the gitlab.com/qemu-project
+  # namespace. When testing CI, it might be usefult to override this
+  # to point to a fork repo
+  QEMU_CI_UPSTREAM: qemu-project
+
 # The order of rules defined here is critically important.
 # They are evaluated in order and first match wins.
 #
@@ -30,23 +35,23 @@ variables:
       when: never
 
     # Publishing jobs should only run on the default branch in upstream
-    - if: '$QEMU_JOB_PUBLISH == "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH'
+    - if: '$QEMU_JOB_PUBLISH == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH'
       when: never
 
     # Non-publishing jobs should only run on staging branches in upstream
-    - if: '$QEMU_JOB_PUBLISH != "1" && $CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH !~ /staging/'
+    - if: '$QEMU_JOB_PUBLISH != "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH !~ /staging/'
       when: never
 
     # Jobs only intended for forks should always be skipped on upstream
-    - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == "qemu-project"'
+    - if: '$QEMU_JOB_ONLY_FORKS == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
       when: never
 
     # Forks don't get pipelines unless QEMU_CI=1 or QEMU_CI=2 is set
-    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+    - if: '$QEMU_CI != "1" && $QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: never
 
     # Avocado jobs don't run in forks unless $QEMU_CI_AVOCADO_TESTING is set
-    - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+    - if: '$QEMU_JOB_AVOCADO && $QEMU_CI_AVOCADO_TESTING != "1" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: never
 
 
@@ -66,7 +71,7 @@ variables:
       allow_failure: true
 
     # Avocado jobs can be manually start in forks if $QEMU_CI_AVOCADO_TESTING is unset
-    - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != "qemu-project"'
+    - if: '$QEMU_JOB_AVOCADO && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: manual
       allow_failure: true
 
@@ -78,7 +83,7 @@ variables:
 
     # Forks pipeline jobs don't start automatically unless
     # QEMU_CI=2 is set
-    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != "qemu-project"'
+    - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: manual
 
     # Jobs can run if any jobs they depend on were successful
diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc
index f72537853b..f9ef14f2eb 100644
--- a/docs/devel/ci-jobs.rst.inc
+++ b/docs/devel/ci-jobs.rst.inc
@@ -75,6 +75,12 @@ in a handful of namespaces
    'latest', but if running pipelines for different branches
    concurrently, it should be overridden per pipeline.
 
+ * QEMU_CI_UPSTREAM - gitlab namespace that is considerd to be
+   the 'upstream'. This defaults to 'qemu-project'. Contributors
+   may choose to override this if they are modifying rules in
+   base.yml and need to validate how they will operate when in
+   an upstream context, as opposed to their fork context.
+
  * nnn - other misc variables not falling into the above
    categories, or using different names for historical reasons
    and not yet converted.
-- 
2.40.1

[PATCH 3/5] gitlab: stable staging branches publish containers in a separate tag

[Daniel P. Berrangé]

If the stable staging branches publish containers under the 'latest' tag
they will clash with containers published on the primary staging branch,
as well  as with each other. This introduces logic that overrides the
container tag when jobs run against the stable staging branches.

The CI_COMMIT_REF_SLUG variable we use expands to the git branch name,
but with most special characters removed, such that it is valid as a
docker tag name. eg 'staging-8.0' will get a slug of 'staging-8-0'

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index a1d734267a..f379c182a7 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -1,7 +1,7 @@
 
 variables:
-  # On stable branches this needs changing. Should also be
-  # overridden per pipeline if running pipelines concurrently
+  # On stable branches this is changed by later rules. Should also
+  # be overridden per pipeline if running pipelines concurrently
   # for different branches in contributor forks.
   QEMU_CI_CONTAINER_TAG: latest
 
@@ -16,6 +16,9 @@ variables:
 # Thus we group them into a number of stages, ordered from
 # most restrictive to least restrictive
 #
+# For pipelines running for stable "staging-X.Y" branches
+# we must override QEMU_CI_CONTAINER_TAG
+#
 .base_job_template:
   variables:
     # Each script line from will be in a collapsible section in the job output
@@ -61,11 +64,23 @@ variables:
     #############################################################
 
     # Optional jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_OPTIONAL && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
+      when: manual
+      allow_failure: true
+      variables:
+        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG
+
     - if: '$QEMU_JOB_OPTIONAL'
       when: manual
       allow_failure: true
 
     # Skipped jobs should not be run unless manually triggered
+    - if: '$QEMU_JOB_SKIPPED && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
+      when: manual
+      allow_failure: true
+      variables:
+        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG
+
     - if: '$QEMU_JOB_SKIPPED'
       when: manual
       allow_failure: true
@@ -87,4 +102,9 @@ variables:
       when: manual
 
     # Jobs can run if any jobs they depend on were successful
+    - if: '$QEMU_JOB_SKIPPED && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
+      when: on_success
+      variables:
+        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG
+
     - when: on_success
-- 
2.40.1

[PATCH 4/5] gitlab: avoid extra pipelines for tags and stable branches

[Daniel P. Berrangé]

In upstream context we only run pipelines on staging branches, and
limited publishing jobs on the default branch.

We don't want to run pipelines on stable branches, or tags, because
the content will have already been tested on a staging branch before
getting pushed.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index f379c182a7..999149852e 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -33,6 +33,14 @@ variables:
     # want jobs to run
     #############################################################
 
+    # Never run jobs upstream on stable branch, staging branch jobs already ran
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /^stable-/'
+      when: never
+
+    # Never run jobs upstream on tags, staging branch jobs already ran
+    - if: '$CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_TAG'
+      when: never
+
     # Cirrus jobs can't run unless the creds / target repo are set
     - if: '$QEMU_JOB_CIRRUS && ($CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null)'
       when: never
-- 
2.40.1

[PATCH 5/5] gitlab: support disabling job auto-run in upstream

[Daniel P. Berrangé]

In forks QEMU_CI=1 can be used to create a pipeline but not auto-run any
jobs. In upstream jobs always auto-run, which is equiv of QEMU_CI=2.

This supports setting QEMU_CI=1 in upstream, to disable job auto-run.
This can be used to preserve CI minutes if repushing a branch to staging
with a specific fix that only needs testing in limited scenarios.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/base.yml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index 999149852e..188a770799 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -109,6 +109,16 @@ variables:
     - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
       when: manual
 
+    # Upstream pipeline jobs start automatically unless told not to
+    # by setting QEMU_CI=1
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
+      when: manual
+      variables:
+        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG
+
+    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM'
+      when: manual
+
     # Jobs can run if any jobs they depend on were successful
     - if: '$QEMU_JOB_SKIPPED && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
       when: on_success
-- 
2.40.1

Re: [PATCH 5/5] gitlab: support disabling job auto-run in upstream

[Michael Tokarev]

17.05.2023 16:54, Daniel P. Berrangé wrote:
> In forks QEMU_CI=1 can be used to create a pipeline but not auto-run any
> jobs. In upstream jobs always auto-run, which is equiv of QEMU_CI=2.
> 
> This supports setting QEMU_CI=1 in upstream, to disable job auto-run.
> This can be used to preserve CI minutes if repushing a branch to staging
> with a specific fix that only needs testing in limited scenarios.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/base.yml | 10 ++++++++++
>   1 file changed, 10 insertions(+)
> 
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index 999149852e..188a770799 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -109,6 +109,16 @@ variables:
>       - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
>         when: manual
>   
> +    # Upstream pipeline jobs start automatically unless told not to
> +    # by setting QEMU_CI=1
> +    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'

Should this be enabled for main staging branch too?
I dunno how it is useful though.

/mjt

Re: [PATCH 1/5] gitlab: centralize the container tag name

[Michael Tokarev]

17.05.2023 16:54, Daniel P. Berrangé wrote:
> We use a fixed container tag of 'latest' so that contributors' forks
> don't end up with an ever growing number of containers as they work
> on throwaway feature branches.
> 
> This fixed tag causes problems running CI upstream in stable staging
> branches, however, because the stable staging branch will publish old
> container content that clashes with that needed by primary staging
> branch. This makes it impossible to reliably run CI pipelines in
> parallel in upstream for different staging branches.
> 
> This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to
> change which tag container publishing uses. Initially it can be set
> by contributors as a git push option if they want to override the
> default use of 'latest' eg
> 
>    git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish
> 
> this is useful if contributors need to run pipelines for different
> branches concurrently in their forks.

Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>

Re: [PATCH 2/5] gitlab: allow overriding name of the upstream repository

[Michael Tokarev]

17.05.2023 16:54, Daniel P. Berrangé wrote:
> The CI rules have special logic for what happens in upstream. To enable
> contributors who modify CI rules to test this logic, however, they need
> to be able to override which repo is considered upstream. This
> introduces the 'QEMU_CI_UPSTREAM' variable
> 
>    git push gitlab <branch> -o ci.variable=QEMU_CI_UPSTREAM=berrange
> 
> to make it look as if my namespace is the actual upstream. Namespace in
> this context refers to the path fragement in gitlab URLs that is above
> the repository. Typically this will be the contributor's gitlab login
> name.

Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>

/mjt

Re: [PATCH 3/5] gitlab: stable staging branches publish containers in a separate tag

[Michael Tokarev]

17.05.2023 16:54, Daniel P. Berrangé wrote:
> If the stable staging branches publish containers under the 'latest' tag
> they will clash with containers published on the primary staging branch,
> as well  as with each other. This introduces logic that overrides the
> container tag when jobs run against the stable staging branches.
> 
> The CI_COMMIT_REF_SLUG variable we use expands to the git branch name,
> but with most special characters removed, such that it is valid as a
> docker tag name. eg 'staging-8.0' will get a slug of 'staging-8-0'
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/base.yml | 24 ++++++++++++++++++++++--
>   1 file changed, 22 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index a1d734267a..f379c182a7 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -1,7 +1,7 @@
>   
>   variables:
> -  # On stable branches this needs changing. Should also be
> -  # overridden per pipeline if running pipelines concurrently
> +  # On stable branches this is changed by later rules. Should also
> +  # be overridden per pipeline if running pipelines concurrently
>     # for different branches in contributor forks.
>     QEMU_CI_CONTAINER_TAG: latest
>   
> @@ -16,6 +16,9 @@ variables:
>   # Thus we group them into a number of stages, ordered from
>   # most restrictive to least restrictive
>   #
> +# For pipelines running for stable "staging-X.Y" branches
> +# we must override QEMU_CI_CONTAINER_TAG
> +#
>   .base_job_template:
>     variables:
>       # Each script line from will be in a collapsible section in the job output
> @@ -61,11 +64,23 @@ variables:
>       #############################################################
>   
>       # Optional jobs should not be run unless manually triggered
> +    - if: '$QEMU_JOB_OPTIONAL && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
> +      when: manual
> +      allow_failure: true
> +      variables:
> +        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG

Here, it somehow feels better to use $CI_COMMIT_BRANCH instead of $CI_COMMIT_REF_SLUG.
I know little about gitlab CI. It is REF_SLUG like a hashed value of COMMIT_BRANCH?

Other than that (here and below),

Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>

/mjt

Re: [PATCH 4/5] gitlab: avoid extra pipelines for tags and stable branches

[Michael Tokarev]

17.05.2023 16:54, Daniel P. Berrangé wrote:
> In upstream context we only run pipelines on staging branches, and
> limited publishing jobs on the default branch.
> 
> We don't want to run pipelines on stable branches, or tags, because
> the content will have already been tested on a staging branch before
> getting pushed.

Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>

/mjt

Re: [PATCH 3/5] gitlab: stable staging branches publish containers in a separate tag

[Daniel P. Berrangé]

On Wed, May 17, 2023 at 06:26:56PM +0300, Michael Tokarev wrote:
> 17.05.2023 16:54, Daniel P. Berrangé wrote:
> > If the stable staging branches publish containers under the 'latest' tag
> > they will clash with containers published on the primary staging branch,
> > as well  as with each other. This introduces logic that overrides the
> > container tag when jobs run against the stable staging branches.
> > 
> > The CI_COMMIT_REF_SLUG variable we use expands to the git branch name,
> > but with most special characters removed, such that it is valid as a
> > docker tag name. eg 'staging-8.0' will get a slug of 'staging-8-0'
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/base.yml | 24 ++++++++++++++++++++++--
> >   1 file changed, 22 insertions(+), 2 deletions(-)
> > 
> > diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> > index a1d734267a..f379c182a7 100644
> > --- a/.gitlab-ci.d/base.yml
> > +++ b/.gitlab-ci.d/base.yml
> > @@ -1,7 +1,7 @@
> >   variables:
> > -  # On stable branches this needs changing. Should also be
> > -  # overridden per pipeline if running pipelines concurrently
> > +  # On stable branches this is changed by later rules. Should also
> > +  # be overridden per pipeline if running pipelines concurrently
> >     # for different branches in contributor forks.
> >     QEMU_CI_CONTAINER_TAG: latest
> > @@ -16,6 +16,9 @@ variables:
> >   # Thus we group them into a number of stages, ordered from
> >   # most restrictive to least restrictive
> >   #
> > +# For pipelines running for stable "staging-X.Y" branches
> > +# we must override QEMU_CI_CONTAINER_TAG
> > +#
> >   .base_job_template:
> >     variables:
> >       # Each script line from will be in a collapsible section in the job output
> > @@ -61,11 +64,23 @@ variables:
> >       #############################################################
> >       # Optional jobs should not be run unless manually triggered
> > +    - if: '$QEMU_JOB_OPTIONAL && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
> > +      when: manual
> > +      allow_failure: true
> > +      variables:
> > +        QEMU_CI_CONTAINER_TAG: $CI_COMMIT_REF_SLUG
> 
> Here, it somehow feels better to use $CI_COMMIT_BRANCH instead of $CI_COMMIT_REF_SLUG.
> I know little about gitlab CI. It is REF_SLUG like a hashed value of COMMIT_BRANCH?

The git branch / tag name can contain characters that are not permitted
for docker tag names. The CI_COMMIT_REF_SLUG is CI_COMMIT_BRANCH but with
everything except 0-9 and a-z replaced with -, making it safe for use as
a docker tag [1]. 


With regards,
Daniel

[1] https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 5/5] gitlab: support disabling job auto-run in upstream

[Daniel P. Berrangé]

On Wed, May 17, 2023 at 06:13:44PM +0300, Michael Tokarev wrote:
> 17.05.2023 16:54, Daniel P. Berrangé wrote:
> > In forks QEMU_CI=1 can be used to create a pipeline but not auto-run any
> > jobs. In upstream jobs always auto-run, which is equiv of QEMU_CI=2.
> > 
> > This supports setting QEMU_CI=1 in upstream, to disable job auto-run.
> > This can be used to preserve CI minutes if repushing a branch to staging
> > with a specific fix that only needs testing in limited scenarios.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/base.yml | 10 ++++++++++
> >   1 file changed, 10 insertions(+)
> > 
> > diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> > index 999149852e..188a770799 100644
> > --- a/.gitlab-ci.d/base.yml
> > +++ b/.gitlab-ci.d/base.yml
> > @@ -109,6 +109,16 @@ variables:
> >       - if: '$QEMU_CI != "2" && $CI_PROJECT_NAMESPACE != $QEMU_CI_UPSTREAM'
> >         when: manual
> > +    # Upstream pipeline jobs start automatically unless told not to
> > +    # by setting QEMU_CI=1
> > +    - if: '$QEMU_CI == "1" && $CI_PROJECT_NAMESPACE == $QEMU_CI_UPSTREAM && $CI_COMMIT_BRANCH =~ /staging-[[:digit:]]+\.[[:digit:]]/'
> 
> Should this be enabled for main staging branch too?
> I dunno how it is useful though.

Yes, actually it should have been. There's no downside to enabling it for
'staging' too, since it is opt-in.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 1/5] gitlab: centralize the container tag name

[Richard Henderson]

On 5/17/23 06:54, Daniel P. Berrangé wrote:
> We use a fixed container tag of 'latest' so that contributors' forks
> don't end up with an ever growing number of containers as they work
> on throwaway feature branches.
> 
> This fixed tag causes problems running CI upstream in stable staging
> branches, however, because the stable staging branch will publish old
> container content that clashes with that needed by primary staging
> branch. This makes it impossible to reliably run CI pipelines in
> parallel in upstream for different staging branches.
> 
> This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to
> change which tag container publishing uses. Initially it can be set
> by contributors as a git push option if they want to override the
> default use of 'latest' eg
> 
>    git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish
> 
> this is useful if contributors need to run pipelines for different
> branches concurrently in their forks.
> 
> Signed-off-by: Daniel P. Berrangé<berrange@redhat.com>
> ---
>   .gitlab-ci.d/base.yml                | 6 ++++++
>   .gitlab-ci.d/buildtest-template.yml  | 4 ++--
>   .gitlab-ci.d/buildtest.yml           | 4 ++--
>   .gitlab-ci.d/container-template.yml  | 3 ++-
>   .gitlab-ci.d/crossbuild-template.yml | 6 +++---
>   .gitlab-ci.d/static_checks.yml       | 4 ++--
>   docs/devel/ci-jobs.rst.inc           | 5 +++++
>   7 files changed, 22 insertions(+), 10 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [PATCH 1/5] gitlab: centralize the container tag name

[Thomas Huth]

On 17/05/2023 15.54, Daniel P. Berrangé wrote:
> We use a fixed container tag of 'latest' so that contributors' forks
> don't end up with an ever growing number of containers as they work
> on throwaway feature branches.
> 
> This fixed tag causes problems running CI upstream in stable staging
> branches, however, because the stable staging branch will publish old
> container content that clashes with that needed by primary staging
> branch. This makes it impossible to reliably run CI pipelines in
> parallel in upstream for different staging branches.
> 
> This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to
> change which tag container publishing uses. Initially it can be set
> by contributors as a git push option if they want to override the
> default use of 'latest' eg
> 
>    git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish
> 
> this is useful if contributors need to run pipelines for different
> branches concurrently in their forks.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/base.yml                | 6 ++++++
>   .gitlab-ci.d/buildtest-template.yml  | 4 ++--
>   .gitlab-ci.d/buildtest.yml           | 4 ++--
>   .gitlab-ci.d/container-template.yml  | 3 ++-
>   .gitlab-ci.d/crossbuild-template.yml | 6 +++---
>   .gitlab-ci.d/static_checks.yml       | 4 ++--
>   docs/devel/ci-jobs.rst.inc           | 5 +++++
>   7 files changed, 22 insertions(+), 10 deletions(-)
> 
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index 2fbb58d2a3..fba9d31cc6 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -1,4 +1,10 @@
>   
> +variables:
> +  # On stable branches this needs changing. Should also be
> +  # overridden per pipeline if running pipelines concurrently
> +  # for different branches in contributor forks.
> +  QEMU_CI_CONTAINER_TAG: latest
> +
>   # The order of rules defined here is critically important.
>   # They are evaluated in order and first match wins.
>   #
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index a6cfe9be97..094ae485b8 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -1,7 +1,7 @@
>   .native_build_job_template:
>     extends: .base_job_template
>     stage: build
> -  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
>     before_script:
>       - JOBS=$(expr $(nproc) + 1)
>     script:
> @@ -44,7 +44,7 @@
>   .common_test_job_template:
>     extends: .base_job_template
>     stage: test
> -  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
>     script:
>       - scripts/git-submodule.sh update
>           $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index bb3650a51c..35867dfe77 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -532,7 +532,7 @@ build-without-defaults:
>   build-libvhost-user:
>     extends: .base_job_template
>     stage: build
> -  image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
> +  image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG
>     needs:
>       job: amd64-fedora-container
>     script:
> @@ -572,7 +572,7 @@ build-tools-and-docs-debian:
>   # of what topic branch they're currently using
>   pages:
>     extends: .base_job_template
> -  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
> +  image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:$QEMU_CI_CONTAINER_TAG
>     stage: test
>     needs:
>       - job: build-tools-and-docs-debian
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index 519b8a9482..83c693c807 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -5,7 +5,8 @@
>     services:
>       - docker:dind
>     before_script:
> -    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> +    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:$QEMU_CI_CONTAINER_TAG"
> +    # Always ':latest' because we always use upstream as a common cache source
>       - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
>       - apk add python3
>       - docker info

This patch no longer applies ... could you please rebase and resend a v2? 
Thanks!

  Thomas

Re: [PATCH 1/5] gitlab: centralize the container tag name

[Daniel P. Berrangé]

On Fri, May 26, 2023 at 09:25:39AM +0200, Thomas Huth wrote:
> On 17/05/2023 15.54, Daniel P. Berrangé wrote:
> > We use a fixed container tag of 'latest' so that contributors' forks
> > don't end up with an ever growing number of containers as they work
> > on throwaway feature branches.
> > 
> > This fixed tag causes problems running CI upstream in stable staging
> > branches, however, because the stable staging branch will publish old
> > container content that clashes with that needed by primary staging
> > branch. This makes it impossible to reliably run CI pipelines in
> > parallel in upstream for different staging branches.
> > 
> > This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to
> > change which tag container publishing uses. Initially it can be set
> > by contributors as a git push option if they want to override the
> > default use of 'latest' eg
> > 
> >    git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish
> > 
> > this is useful if contributors need to run pipelines for different
> > branches concurrently in their forks.

> 
> This patch no longer applies ... could you please rebase and resend a v2?
> Thanks!

I've rebased and sent a v2, but I didn't get any conflits when
rebasing, so v1 should have applied OK.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 1/5] gitlab: centralize the container tag name

[Thomas Huth]

On 26/05/2023 12.20, Daniel P. Berrangé wrote:
> On Fri, May 26, 2023 at 09:25:39AM +0200, Thomas Huth wrote:
>> On 17/05/2023 15.54, Daniel P. Berrangé wrote:
>>> We use a fixed container tag of 'latest' so that contributors' forks
>>> don't end up with an ever growing number of containers as they work
>>> on throwaway feature branches.
>>>
>>> This fixed tag causes problems running CI upstream in stable staging
>>> branches, however, because the stable staging branch will publish old
>>> container content that clashes with that needed by primary staging
>>> branch. This makes it impossible to reliably run CI pipelines in
>>> parallel in upstream for different staging branches.
>>>
>>> This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to
>>> change which tag container publishing uses. Initially it can be set
>>> by contributors as a git push option if they want to override the
>>> default use of 'latest' eg
>>>
>>>     git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish
>>>
>>> this is useful if contributors need to run pipelines for different
>>> branches concurrently in their forks.
> 
>>
>> This patch no longer applies ... could you please rebase and resend a v2?
>> Thanks!
> 
> I've rebased and sent a v2, but I didn't get any conflits when
> rebasing, so v1 should have applied OK.

git-am is sometimes more picky than git-rebase ... I think there were some 
contextual conflicts with the patches from Camilla (commit 5f63a67adb5847 
and b105ce60ca8bdee3) ... I should have maybe tried with "--3way" first 
before complaining, sorry.

  Thomas

Re: [PATCH 1/5] gitlab: centralize the container tag name

[Daniel P. Berrangé]

On Fri, May 26, 2023 at 12:31:21PM +0200, Thomas Huth wrote:
> On 26/05/2023 12.20, Daniel P. Berrangé wrote:
> > On Fri, May 26, 2023 at 09:25:39AM +0200, Thomas Huth wrote:
> > > On 17/05/2023 15.54, Daniel P. Berrangé wrote:
> > > > We use a fixed container tag of 'latest' so that contributors' forks
> > > > don't end up with an ever growing number of containers as they work
> > > > on throwaway feature branches.
> > > > 
> > > > This fixed tag causes problems running CI upstream in stable staging
> > > > branches, however, because the stable staging branch will publish old
> > > > container content that clashes with that needed by primary staging
> > > > branch. This makes it impossible to reliably run CI pipelines in
> > > > parallel in upstream for different staging branches.
> > > > 
> > > > This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to
> > > > change which tag container publishing uses. Initially it can be set
> > > > by contributors as a git push option if they want to override the
> > > > default use of 'latest' eg
> > > > 
> > > >     git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish
> > > > 
> > > > this is useful if contributors need to run pipelines for different
> > > > branches concurrently in their forks.
> > 
> > > 
> > > This patch no longer applies ... could you please rebase and resend a v2?
> > > Thanks!
> > 
> > I've rebased and sent a v2, but I didn't get any conflits when
> > rebasing, so v1 should have applied OK.
> 
> git-am is sometimes more picky than git-rebase ... I think there were some
> contextual conflicts with the patches from Camilla (commit 5f63a67adb5847
> and b105ce60ca8bdee3) ... I should have maybe tried with "--3way" first
> before complaining, sorry.

No worries, it is annoying that git am doesn't default to enabling -3,
as that's basically always what you want :-(


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|