[RFC PATCH 0/2] gitlab-ci: Maintain a public runner, allow manual pipeline

[RFC PATCH 0/2] gitlab-ci: Maintain a public runner, allow manual pipeline

[Philippe Mathieu-Daudé]

Yet another attempt to make GitLab-CI inclusive to the
all users from the QEMU community.

We allow manual pipeline for some users, letting default
to all jobs started.

Also I step in to maintain in my work time the public
runner based on Fedora that we use for the X86 32-bit
builds, because I don't want it to bitrot, as it is
important for my daily work.

Regards,

Phil.

Philippe Mathieu-Daudé (2):
  gitlab-ci: Step in to maintain the fedora-i386-cross runner
  gitlab-ci: Introduce allow_skipping_job_template

 .gitlab-ci.d/crossbuilds.yml | 19 +++++++++++++++++++
 .gitlab-ci.yml               | 18 ++++++++++++++++++
 2 files changed, 37 insertions(+)

-- 
2.26.2

[RFC PATCH 1/2] gitlab-ci: Step in to maintain the fedora-i386-cross runner

[Philippe Mathieu-Daudé]

As I am interested in using the runner based on the fedora-i386-cross
docker image, add a smoke test job to be sure this image is usable,
and set the JOB_MAINTAINER_NAME/JOB_MAINTAINER_EMAIL variables so
other developers can contact me in case there is a problem with the
public runner based on this image.

We will eventually send a patch to make script/checkpatch.pl parse
the Gitlab YAML jobs to understand the JOB_MAINTAINER_NAME and
JOB_MAINTAINER_EMAIL variables.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci.d/crossbuilds.yml | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index bd6473a75a7..864cad7cec5 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -39,6 +39,13 @@
       ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system
     - make -j$(expr $(nproc) + 1) all check-build
 
+.cross_sanity_check_job:
+  stage: build
+  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  timeout: 3m
+  script:
+    - /bin/true
+
 cross-armel-system:
   extends: .cross_system_build_job
   variables:
@@ -69,6 +76,14 @@ cross-arm64-user:
   variables:
     IMAGE: debian-arm64-cross
 
+cross-i386-sanity-check:
+  extends: .cross_sanity_check_job
+  variables:
+    JOB_MAINTAINER_NAME: "Philippe Mathieu-Daudé"
+    JOB_MAINTAINER_EMAIL: philmd@redhat.com
+    GIT_SUBMODULE_STRATEGY: none
+    IMAGE: fedora-i386-cross
+
 cross-mips-system:
   extends: .cross_system_build_job
   variables:
-- 
2.26.2

Re: [RFC PATCH 1/2] gitlab-ci: Step in to maintain the fedora-i386-cross runner

[Thomas Huth]

On 11/12/2020 18.17, Philippe Mathieu-Daudé wrote:
> As I am interested in using the runner based on the fedora-i386-cross
> docker image, add a smoke test job to be sure this image is usable,
> and set the JOB_MAINTAINER_NAME/JOB_MAINTAINER_EMAIL variables so
> other developers can contact me in case there is a problem with the
> public runner based on this image.
> 
> We will eventually send a patch to make script/checkpatch.pl parse
> the Gitlab YAML jobs to understand the JOB_MAINTAINER_NAME and
> JOB_MAINTAINER_EMAIL variables.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  .gitlab-ci.d/crossbuilds.yml | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> index bd6473a75a7..864cad7cec5 100644
> --- a/.gitlab-ci.d/crossbuilds.yml
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -39,6 +39,13 @@
>        ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system
>      - make -j$(expr $(nproc) + 1) all check-build
>  
> +.cross_sanity_check_job:
> +  stage: build
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  timeout: 3m
> +  script:
> +    - /bin/true

I guess it makes more sense if I respin my patch here instead:

https://www.mail-archive.com/qemu-devel@nongnu.org/msg762919.html

?

 Thomas

Re: [RFC PATCH 1/2] gitlab-ci: Step in to maintain the fedora-i386-cross runner

[Philippe Mathieu-Daudé]

On 12/14/20 1:35 PM, Thomas Huth wrote:
> On 11/12/2020 18.17, Philippe Mathieu-Daudé wrote:
>> As I am interested in using the runner based on the fedora-i386-cross
>> docker image, add a smoke test job to be sure this image is usable,
>> and set the JOB_MAINTAINER_NAME/JOB_MAINTAINER_EMAIL variables so
>> other developers can contact me in case there is a problem with the
>> public runner based on this image.
>>
>> We will eventually send a patch to make script/checkpatch.pl parse
>> the Gitlab YAML jobs to understand the JOB_MAINTAINER_NAME and
>> JOB_MAINTAINER_EMAIL variables.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  .gitlab-ci.d/crossbuilds.yml | 15 +++++++++++++++
>>  1 file changed, 15 insertions(+)
>>
>> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
>> index bd6473a75a7..864cad7cec5 100644
>> --- a/.gitlab-ci.d/crossbuilds.yml
>> +++ b/.gitlab-ci.d/crossbuilds.yml
>> @@ -39,6 +39,13 @@
>>        ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system
>>      - make -j$(expr $(nproc) + 1) all check-build
>>  
>> +.cross_sanity_check_job:
>> +  stage: build
>> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>> +  timeout: 3m
>> +  script:
>> +    - /bin/true
> 
> I guess it makes more sense if I respin my patch here instead:
> 
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg762919.html
> 
> ?

I think these are different. The idea of testing /bin/true is to be
able to make the distinction between a broken runner and a broken
test.

> 
>  Thomas
> 

Re: [RFC PATCH 1/2] gitlab-ci: Step in to maintain the fedora-i386-cross runner

[Daniel P. Berrangé]

On Mon, Dec 14, 2020 at 06:52:31PM +0100, Philippe Mathieu-Daudé wrote:
> On 12/14/20 1:35 PM, Thomas Huth wrote:
> > On 11/12/2020 18.17, Philippe Mathieu-Daudé wrote:
> >> As I am interested in using the runner based on the fedora-i386-cross
> >> docker image, add a smoke test job to be sure this image is usable,
> >> and set the JOB_MAINTAINER_NAME/JOB_MAINTAINER_EMAIL variables so
> >> other developers can contact me in case there is a problem with the
> >> public runner based on this image.
> >>
> >> We will eventually send a patch to make script/checkpatch.pl parse
> >> the Gitlab YAML jobs to understand the JOB_MAINTAINER_NAME and
> >> JOB_MAINTAINER_EMAIL variables.
> >>
> >> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> >> ---
> >>  .gitlab-ci.d/crossbuilds.yml | 15 +++++++++++++++
> >>  1 file changed, 15 insertions(+)
> >>
> >> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> >> index bd6473a75a7..864cad7cec5 100644
> >> --- a/.gitlab-ci.d/crossbuilds.yml
> >> +++ b/.gitlab-ci.d/crossbuilds.yml
> >> @@ -39,6 +39,13 @@
> >>        ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system
> >>      - make -j$(expr $(nproc) + 1) all check-build
> >>  
> >> +.cross_sanity_check_job:
> >> +  stage: build
> >> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> >> +  timeout: 3m
> >> +  script:
> >> +    - /bin/true
> > 
> > I guess it makes more sense if I respin my patch here instead:
> > 
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg762919.html
> > 
> > ?
> 
> I think these are different. The idea of testing /bin/true is to be
> able to make the distinction between a broken runner and a broken
> test.

I don't see any point in this job.  /bin/true will always succeeed
if we've successfully created the container image if the container
build job.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[RFC PATCH 2/2] gitlab-ci: Introduce allow_skipping_job_template

[Philippe Mathieu-Daudé]

It might be overkill (and frustrating) to rerun all the jobs when
iterating over a single code unit. As some users are interested
in only rerunning a specific set of jobs, allow them to use a
"manual" profile, when jobs are created in the pipeline but not
started. It is up to the user to manually start the jobs of
interest.

To switch a repository to manual, set QEMU_DEFAULT_WORKFLOW_MANUAL
in the project UI, following [1].

Once a repository is using the manual workflow, the user can still
automatically start the jobs by explicitly set the
QEMU_IGNORE_DEFAULT_WORKFLOW which disable the manual mode.

Example while pushing a branch/tag from command line (see [2]):

  $ git push -o ci.variable="QEMU_IGNORE_DEFAULT_WORKFLOW=1" \
      myrepo mybranch

[1] https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui
[2] https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci.d/crossbuilds.yml |  4 ++++
 .gitlab-ci.yml               | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 864cad7cec5..56b053cbb52 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -1,4 +1,5 @@
 .cross_system_build_job:
+  extends: .allow_skipping_job
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -18,6 +19,7 @@
 # KVM), and set extra options (such disabling other accelerators) via the
 # $ACCEL_CONFIGURE_OPTS variable.
 .cross_accel_build_job:
+  extends: .allow_skipping_job
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 30m
@@ -30,6 +32,7 @@
     - make -j$(expr $(nproc) + 1) all check-build
 
 .cross_user_build_job:
+  extends: .allow_skipping_job
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -40,6 +43,7 @@
     - make -j$(expr $(nproc) + 1) all check-build
 
 .cross_sanity_check_job:
+  extends: .allow_skipping_job
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 3m
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 98bff03b47b..26b85f08f35 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -13,7 +13,21 @@ include:
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
 
+.allow_skipping_job_template: &allow_skipping_job_definition
+  rules:
+    # Skip if the user set the QEMU_DEFAULT_WORKFLOW_MANUAL variable for its
+    # project. Can be overridden by setting the QEMU_IGNORE_DEFAULT_WORKFLOW
+    # variable.
+    - if: $CI_PIPELINE_SOURCE == "push" && $QEMU_DEFAULT_WORKFLOW_MANUAL && $QEMU_IGNORE_DEFAULT_WORKFLOW == null
+      when: manual
+      allow_failure: true
+    - when: always
+
+.allow_skipping_job:
+  <<: *allow_skipping_job_definition
+
 .native_build_job_template: &native_build_job_definition
+  <<: *allow_skipping_job_definition
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -35,6 +49,7 @@ include:
       fi
 
 .native_test_job_template: &native_test_job_definition
+  <<: *allow_skipping_job_definition
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -43,6 +58,7 @@ include:
     - make $MAKE_CHECK_ARGS
 
 .acceptance_template: &acceptance_definition
+  <<: *allow_skipping_job_definition
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
@@ -536,6 +552,7 @@ check-dco:
     GIT_DEPTH: 1000
 
 build-libvhost-user:
+  <<: *allow_skipping_job_definition
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
   before_script:
@@ -547,6 +564,7 @@ build-libvhost-user:
     - ninja
 
 pages:
+  <<: *allow_skipping_job_definition
   image: $CI_REGISTRY_IMAGE/qemu/ubuntu2004:latest
   stage: test
   needs:
-- 
2.26.2

Re: [RFC PATCH 2/2] gitlab-ci: Introduce allow_skipping_job_template

[Thomas Huth]

On 11/12/2020 18.17, Philippe Mathieu-Daudé wrote:
> It might be overkill (and frustrating) to rerun all the jobs when
> iterating over a single code unit. As some users are interested
> in only rerunning a specific set of jobs, allow them to use a
> "manual" profile, when jobs are created in the pipeline but not
> started. It is up to the user to manually start the jobs of
> interest.
> 
> To switch a repository to manual, set QEMU_DEFAULT_WORKFLOW_MANUAL
> in the project UI, following [1].
> 
> Once a repository is using the manual workflow, the user can still
> automatically start the jobs by explicitly set the
> QEMU_IGNORE_DEFAULT_WORKFLOW which disable the manual mode.
> 
> Example while pushing a branch/tag from command line (see [2]):
> 
>   $ git push -o ci.variable="QEMU_IGNORE_DEFAULT_WORKFLOW=1" \
>       myrepo mybranch
> 
> [1] https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui
> [2] https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  .gitlab-ci.d/crossbuilds.yml |  4 ++++
>  .gitlab-ci.yml               | 18 ++++++++++++++++++
>  2 files changed, 22 insertions(+)
> 
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> index 864cad7cec5..56b053cbb52 100644
> --- a/.gitlab-ci.d/crossbuilds.yml
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -1,4 +1,5 @@
>  .cross_system_build_job:
> +  extends: .allow_skipping_job
>    stage: build
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    timeout: 80m
> @@ -18,6 +19,7 @@
>  # KVM), and set extra options (such disabling other accelerators) via the
>  # $ACCEL_CONFIGURE_OPTS variable.
>  .cross_accel_build_job:
> +  extends: .allow_skipping_job
>    stage: build
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    timeout: 30m
> @@ -30,6 +32,7 @@
>      - make -j$(expr $(nproc) + 1) all check-build
>  
>  .cross_user_build_job:
> +  extends: .allow_skipping_job
>    stage: build
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    script:
> @@ -40,6 +43,7 @@
>      - make -j$(expr $(nproc) + 1) all check-build
>  
>  .cross_sanity_check_job:
> +  extends: .allow_skipping_job

So "extends" is possible ....

>    stage: build
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    timeout: 3m
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 98bff03b47b..26b85f08f35 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -13,7 +13,21 @@ include:
>    - local: '/.gitlab-ci.d/containers.yml'
>    - local: '/.gitlab-ci.d/crossbuilds.yml'
>  
> +.allow_skipping_job_template: &allow_skipping_job_definition
> +  rules:
> +    # Skip if the user set the QEMU_DEFAULT_WORKFLOW_MANUAL variable for its
> +    # project. Can be overridden by setting the QEMU_IGNORE_DEFAULT_WORKFLOW
> +    # variable.
> +    - if: $CI_PIPELINE_SOURCE == "push" && $QEMU_DEFAULT_WORKFLOW_MANUAL && $QEMU_IGNORE_DEFAULT_WORKFLOW == null
> +      when: manual
> +      allow_failure: true
> +    - when: always
> +
> +.allow_skipping_job:
> +  <<: *allow_skipping_job_definition

... before the anchestor has been defined? That's nice!

I'm basically fine with this patch, but can you please also add some
sentence how to use this to docs/devel/testing.rst ?

 Thanks,
  Thomas

Re: [RFC PATCH 2/2] gitlab-ci: Introduce allow_skipping_job_template

[Philippe Mathieu-Daudé]

On 12/14/20 2:32 PM, Thomas Huth wrote:
> On 11/12/2020 18.17, Philippe Mathieu-Daudé wrote:
>> It might be overkill (and frustrating) to rerun all the jobs when
>> iterating over a single code unit. As some users are interested
>> in only rerunning a specific set of jobs, allow them to use a
>> "manual" profile, when jobs are created in the pipeline but not
>> started. It is up to the user to manually start the jobs of
>> interest.
>>
>> To switch a repository to manual, set QEMU_DEFAULT_WORKFLOW_MANUAL
>> in the project UI, following [1].
>>
>> Once a repository is using the manual workflow, the user can still
>> automatically start the jobs by explicitly set the
>> QEMU_IGNORE_DEFAULT_WORKFLOW which disable the manual mode.
>>
>> Example while pushing a branch/tag from command line (see [2]):
>>
>>   $ git push -o ci.variable="QEMU_IGNORE_DEFAULT_WORKFLOW=1" \
>>       myrepo mybranch
>>
>> [1] https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui
>> [2] https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  .gitlab-ci.d/crossbuilds.yml |  4 ++++
>>  .gitlab-ci.yml               | 18 ++++++++++++++++++
>>  2 files changed, 22 insertions(+)
>>
>> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
>> index 864cad7cec5..56b053cbb52 100644
>> --- a/.gitlab-ci.d/crossbuilds.yml
>> +++ b/.gitlab-ci.d/crossbuilds.yml
>> @@ -1,4 +1,5 @@
>>  .cross_system_build_job:
>> +  extends: .allow_skipping_job
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    timeout: 80m
>> @@ -18,6 +19,7 @@
>>  # KVM), and set extra options (such disabling other accelerators) via the
>>  # $ACCEL_CONFIGURE_OPTS variable.
>>  .cross_accel_build_job:
>> +  extends: .allow_skipping_job
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    timeout: 30m
>> @@ -30,6 +32,7 @@
>>      - make -j$(expr $(nproc) + 1) all check-build
>>  
>>  .cross_user_build_job:
>> +  extends: .allow_skipping_job
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    script:
>> @@ -40,6 +43,7 @@
>>      - make -j$(expr $(nproc) + 1) all check-build
>>  
>>  .cross_sanity_check_job:
>> +  extends: .allow_skipping_job
> 
> So "extends" is possible ....
> 
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    timeout: 3m
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 98bff03b47b..26b85f08f35 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -13,7 +13,21 @@ include:
>>    - local: '/.gitlab-ci.d/containers.yml'
>>    - local: '/.gitlab-ci.d/crossbuilds.yml'
>>  
>> +.allow_skipping_job_template: &allow_skipping_job_definition
>> +  rules:
>> +    # Skip if the user set the QEMU_DEFAULT_WORKFLOW_MANUAL variable for its
>> +    # project. Can be overridden by setting the QEMU_IGNORE_DEFAULT_WORKFLOW
>> +    # variable.
>> +    - if: $CI_PIPELINE_SOURCE == "push" && $QEMU_DEFAULT_WORKFLOW_MANUAL && $QEMU_IGNORE_DEFAULT_WORKFLOW == null
>> +      when: manual
>> +      allow_failure: true
>> +    - when: always
>> +
>> +.allow_skipping_job:
>> +  <<: *allow_skipping_job_definition
> 
> ... before the anchestor has been defined? That's nice!

'extends' is indeed more powerful and less hackish.

> I'm basically fine with this patch, but can you please also add some
> sentence how to use this to docs/devel/testing.rst ?

Thanks, I later discovered this isn't complete, because if a children
job is in automatic state and a parent (dependency) is in manual, there
is some "dead lock": the pipeline waits until manual job is started.

To be complete, I have to ensure if a job is in manual state, all its
dependencies also are.

Then we are back to 'extends' VS 'anchors', as this doesn't work with
anchors, I have to convert all jobs to 'extends'. Good news is I already
did it, I simply need to rebase this series:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg758331.html

Thanks for reviewing,

Phil.