[PATCH v1 1/1] gitlab-ci: Let project define runner using environment variables

[PATCH v1 1/1] gitlab-ci: Let project define runner using environment variables

[Leonardo Bras]

Currently it's not possible to select which runner will handle a pipeline
without changing the codebase.

Add CI_TAGS environment variable, which can be used to select a runner
either from a commit message, or directly from Gitlab interface.

Also add Documentation for this variable.

Signed-off-by: Leonardo Bras <leobras@redhat.com>
---

This patch goes on top on a previous patchset sent by Helen:
https://lore.kernel.org/all/20240228225527.1052240-1-helen.koike@collabora.com/

With this patch I could run CI with gitlab.com runners, by setting
CI_TAGS=saas-linux-medium-amd64 

The result of this pipeline can be seen in:
https://gitlab.com/linux-kernel/linux/-/pipelines/1228999646

 Documentation/ci/gitlab-ci/gitlab-ci.rst | 5 +++++
 ci/gitlab-ci/yml/gitlab-ci.yml           | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/Documentation/ci/gitlab-ci/gitlab-ci.rst b/Documentation/ci/gitlab-ci/gitlab-ci.rst
index 4f7ef03cca95..18360da835bd 100644
--- a/Documentation/ci/gitlab-ci/gitlab-ci.rst
+++ b/Documentation/ci/gitlab-ci/gitlab-ci.rst
@@ -304,20 +304,25 @@ Description of Each Variable
 **KCI_CHECKPATCH_OPTIONS**
     Used in `checkpatch.pl "$KCI_CHECKPATCH_OPTIONS"` (see checkpatch
     documentation). It is commonly used with the --ignore flag to suppress
     specific warnings generated by checkpatch.pl. It can also be defined in the
     commit message, since it is evaluated in run time.
 
 **KCI_PATCH_SERIES_SIZE**
     Used to define the size of the patch series, see `job: checkpatch` section
     above. It is evaluated in run time, and can be set in the commit message.
 
+**CI_TAGS**
+    Used to help choose which runner will deal with the current pipeline.
+    If using Gitlab.com runners, set saas-linux-medium-amd64 or a better runner
+    so there is enough resources to build & commit the base image.
+
 .. _triggering-pipelines-from-command-line:
 
 Triggering Pipelines from Command Line
 --------------------------------------
 
 Pipelines can be triggered from the command line with custom variables using the
 `GitLab CLI tool <https://docs.gitlab.com/ee/editor_extensions/gitlab_cli>`_.
 
 Example:
 
diff --git a/ci/gitlab-ci/yml/gitlab-ci.yml b/ci/gitlab-ci/yml/gitlab-ci.yml
index 57b9c0290471..359b7715e3ab 100644
--- a/ci/gitlab-ci/yml/gitlab-ci.yml
+++ b/ci/gitlab-ci/yml/gitlab-ci.yml
@@ -33,20 +33,22 @@ workflow:
     - if: $FORCE_CI == 'true'
 
 variables:
   FDO_UPSTREAM_REPO: helen.fornazier/linux   # The repo where to look for cached images
     # ccache builds in gitlab-runner to speed up builds
   SMATCH_DB_DIR: /smatch/smatch_data
   # exit code of bash script on `script` will be the exit code of the job
   FF_USE_NEW_BASH_EVAL_STRATEGY: "true"
 
 default:
+  tags:
+    - $CI_TAGS
   artifacts:
     paths:
       - artifacts/
     when: always
 
 include:
   - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/ci-fairy.yml'
   - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/debian.yml'
 
   - ci/gitlab-ci/yml/kernel-combinations.yml
-- 
2.44.0

Re: [PATCH v1 1/1] gitlab-ci: Let project define runner using environment variables

[Helen Koike]

Hi Leonardo,

Thanks for the patch.

On 26/03/2024 22:30, Leonardo Bras wrote:
> Currently it's not possible to select which runner will handle a pipeline
> without changing the codebase.
> 
> Add CI_TAGS environment variable, which can be used to select a runner
> either from a commit message, or directly from Gitlab interface.
> 
> Also add Documentation for this variable.
> 
> Signed-off-by: Leonardo Bras <leobras@redhat.com>
> ---
> 
> This patch goes on top on a previous patchset sent by Helen:
> https://lore.kernel.org/all/20240228225527.1052240-1-helen.koike@collabora.com/
> 
> With this patch I could run CI with gitlab.com runners, by setting
> CI_TAGS=saas-linux-medium-amd64
> 
> The result of this pipeline can be seen in:
> https://gitlab.com/linux-kernel/linux/-/pipelines/1228999646
> 
>   Documentation/ci/gitlab-ci/gitlab-ci.rst | 5 +++++
>   ci/gitlab-ci/yml/gitlab-ci.yml           | 2 ++
>   2 files changed, 7 insertions(+)
> 
> diff --git a/Documentation/ci/gitlab-ci/gitlab-ci.rst b/Documentation/ci/gitlab-ci/gitlab-ci.rst
> index 4f7ef03cca95..18360da835bd 100644
> --- a/Documentation/ci/gitlab-ci/gitlab-ci.rst
> +++ b/Documentation/ci/gitlab-ci/gitlab-ci.rst
> @@ -304,20 +304,25 @@ Description of Each Variable
>   **KCI_CHECKPATCH_OPTIONS**
>       Used in `checkpatch.pl "$KCI_CHECKPATCH_OPTIONS"` (see checkpatch
>       documentation). It is commonly used with the --ignore flag to suppress
>       specific warnings generated by checkpatch.pl. It can also be defined in the
>       commit message, since it is evaluated in run time.
>   
>   **KCI_PATCH_SERIES_SIZE**
>       Used to define the size of the patch series, see `job: checkpatch` section
>       above. It is evaluated in run time, and can be set in the commit message.
>   
> +**CI_TAGS**

I would just s/CI/KCI , I'm using KCI_prefix for those that are defined 
by us, since gitlab already uses CI_ prefix.


We will also need a way to specify tags per job at some point 
KCI_TAGS_$JOB_NAME or something, but this can be added later.

Acked-by: Helen Koike <helen.koike@collabora.com>

I'll add on top of my tree for now, so we can prepare v2 with this change.

Thanks
Helen

> +    Used to help choose which runner will deal with the current pipeline.
> +    If using Gitlab.com runners, set saas-linux-medium-amd64 or a better runner
> +    so there is enough resources to build & commit the base image.
> +
>   .. _triggering-pipelines-from-command-line:
>   
>   Triggering Pipelines from Command Line
>   --------------------------------------
>   
>   Pipelines can be triggered from the command line with custom variables using the
>   `GitLab CLI tool <https://docs.gitlab.com/ee/editor_extensions/gitlab_cli>`_.
>   
>   Example:
>   
> diff --git a/ci/gitlab-ci/yml/gitlab-ci.yml b/ci/gitlab-ci/yml/gitlab-ci.yml
> index 57b9c0290471..359b7715e3ab 100644
> --- a/ci/gitlab-ci/yml/gitlab-ci.yml
> +++ b/ci/gitlab-ci/yml/gitlab-ci.yml
> @@ -33,20 +33,22 @@ workflow:
>       - if: $FORCE_CI == 'true'
>   
>   variables:
>     FDO_UPSTREAM_REPO: helen.fornazier/linux   # The repo where to look for cached images
>       # ccache builds in gitlab-runner to speed up builds
>     SMATCH_DB_DIR: /smatch/smatch_data
>     # exit code of bash script on `script` will be the exit code of the job
>     FF_USE_NEW_BASH_EVAL_STRATEGY: "true"
>   
>   default:
> +  tags:
> +    - $CI_TAGS
>     artifacts:
>       paths:
>         - artifacts/
>       when: always
>   
>   include:
>     - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/ci-fairy.yml'
>     - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/debian.yml'
>   
>     - ci/gitlab-ci/yml/kernel-combinations.yml

Re: [PATCH v1 1/1] gitlab-ci: Let project define runner using environment variables

[Leonardo Bras]

On Thu, Mar 28, 2024 at 10:17:00AM -0300, Helen Koike wrote:
> Hi Leonardo,
> 
> Thanks for the patch.
> 
> On 26/03/2024 22:30, Leonardo Bras wrote:
> > Currently it's not possible to select which runner will handle a pipeline
> > without changing the codebase.
> > 
> > Add CI_TAGS environment variable, which can be used to select a runner
> > either from a commit message, or directly from Gitlab interface.
> > 
> > Also add Documentation for this variable.
> > 
> > Signed-off-by: Leonardo Bras <leobras@redhat.com>
> > ---
> > 
> > This patch goes on top on a previous patchset sent by Helen:
> > https://lore.kernel.org/all/20240228225527.1052240-1-helen.koike@collabora.com/
> > 
> > With this patch I could run CI with gitlab.com runners, by setting
> > CI_TAGS=saas-linux-medium-amd64
> > 
> > The result of this pipeline can be seen in:
> > https://gitlab.com/linux-kernel/linux/-/pipelines/1228999646
> > 
> >   Documentation/ci/gitlab-ci/gitlab-ci.rst | 5 +++++
> >   ci/gitlab-ci/yml/gitlab-ci.yml           | 2 ++
> >   2 files changed, 7 insertions(+)
> > 
> > diff --git a/Documentation/ci/gitlab-ci/gitlab-ci.rst b/Documentation/ci/gitlab-ci/gitlab-ci.rst
> > index 4f7ef03cca95..18360da835bd 100644
> > --- a/Documentation/ci/gitlab-ci/gitlab-ci.rst
> > +++ b/Documentation/ci/gitlab-ci/gitlab-ci.rst
> > @@ -304,20 +304,25 @@ Description of Each Variable
> >   **KCI_CHECKPATCH_OPTIONS**
> >       Used in `checkpatch.pl "$KCI_CHECKPATCH_OPTIONS"` (see checkpatch
> >       documentation). It is commonly used with the --ignore flag to suppress
> >       specific warnings generated by checkpatch.pl. It can also be defined in the
> >       commit message, since it is evaluated in run time.
> >   **KCI_PATCH_SERIES_SIZE**
> >       Used to define the size of the patch series, see `job: checkpatch` section
> >       above. It is evaluated in run time, and can be set in the commit message.
> > +**CI_TAGS**
> 
> I would just s/CI/KCI , I'm using KCI_prefix for those that are defined by
> us, since gitlab already uses CI_ prefix.

I was aware of CI_ vs KCI_ prefix, but for some reason at the time I 
decided that since this was about setting the gitlab tags, and not 
something internal to Kernel CI,  it would make more sense to use the CI_ 
prefix. 

But I have no strong opinion on this, so if you think it's better to 
call it KCI_TAGS, let's do that instead.

> 
> We will also need a way to specify tags per job at some point
> KCI_TAGS_$JOB_NAME or something, but this can be added later.

Yeah, agree.

I wish we could specify tags with running requirements for each task in 
some way, and let the gitlab instance to pick the better runner.


> 
> Acked-by: Helen Koike <helen.koike@collabora.com>
> 
> I'll add on top of my tree for now, so we can prepare v2 with this change.
> 
> Thanks
> Helen

Thank you!
Leo

> 
> > +    Used to help choose which runner will deal with the current pipeline.
> > +    If using Gitlab.com runners, set saas-linux-medium-amd64 or a better runner
> > +    so there is enough resources to build & commit the base image.
> > +
> >   .. _triggering-pipelines-from-command-line:
> >   Triggering Pipelines from Command Line
> >   --------------------------------------
> >   Pipelines can be triggered from the command line with custom variables using the
> >   `GitLab CLI tool <https://docs.gitlab.com/ee/editor_extensions/gitlab_cli>`_.
> >   Example:
> > diff --git a/ci/gitlab-ci/yml/gitlab-ci.yml b/ci/gitlab-ci/yml/gitlab-ci.yml
> > index 57b9c0290471..359b7715e3ab 100644
> > --- a/ci/gitlab-ci/yml/gitlab-ci.yml
> > +++ b/ci/gitlab-ci/yml/gitlab-ci.yml
> > @@ -33,20 +33,22 @@ workflow:
> >       - if: $FORCE_CI == 'true'
> >   variables:
> >     FDO_UPSTREAM_REPO: helen.fornazier/linux   # The repo where to look for cached images
> >       # ccache builds in gitlab-runner to speed up builds
> >     SMATCH_DB_DIR: /smatch/smatch_data
> >     # exit code of bash script on `script` will be the exit code of the job
> >     FF_USE_NEW_BASH_EVAL_STRATEGY: "true"
> >   default:
> > +  tags:
> > +    - $CI_TAGS
> >     artifacts:
> >       paths:
> >         - artifacts/
> >       when: always
> >   include:
> >     - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/ci-fairy.yml'
> >     - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/debian.yml'
> >     - ci/gitlab-ci/yml/kernel-combinations.yml
>