qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] gitlab-ci: Restrict jobs using Docker to runners having 'docker' tag
@ 2021-03-19  0:43 Philippe Mathieu-Daudé
  2021-03-19  5:40 ` Thomas Huth
  0 siblings, 1 reply; 5+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-03-19  0:43 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Daniel P . Berrange, Alex Bennée,
	Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Bin Meng, Laszlo Ersek

When a job is based on a Docker image [1], or is using a Docker
service, it requires a runner with Docker installed.

Gitlab shared runners provide the 'docker' tag when they have it
installed.

Are Gitlab shared runners are limited resources, we'd like to
add more runners to QEMU repositories hosted on Gitlab. If a
runner doesn't provide Docker, our jobs requiring it will fail.

Use the standard 'docker' tag to mark the jobs requiring Docker
on the runner.

[1] https://docs.gitlab.com/ee/ci/yaml/#image
[2] https://docs.gitlab.com/ee/ci/yaml/#services

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
If someone is interested in testing or filling the documentation
gap, what I ran is:

$ sudo usermod -aG docker,kvm gitlab-runner
$ sudo gitlab-runner --log-format text --log-level debug \
    register \
 --non-interactive \
 --url https://gitlab.com --registration-token MYTOKEN --description myrunner \
 --tag-list 'docker,linux,x86_64,kvm' --run-untagged --limit 2 \
 --executor docker --docker-image docker:dind --docker-cpus 4 \
 --docker-volumes /var/run/docker.sock:/var/run/docker.sock \
 --docker-dns 8.8.8.8

--docker-volumes is for docker:dind else it was not working
This comes from this 3 year old thread:
https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1986

We can not use the 'docker:dind' tag for a runner having docker:dind
and /var/run/docker.sock volume because this is not a tag used by
the shared runners, so we can't use them anymore.
---
 .gitlab-ci.d/containers.yml  | 2 ++
 .gitlab-ci.d/crossbuilds.yml | 4 ++++
 .gitlab-ci.d/edk2.yml        | 4 ++++
 .gitlab-ci.d/opensbi.yml     | 4 ++++
 .gitlab-ci.yml               | 4 ++++
 5 files changed, 18 insertions(+)

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 33e4046e233..8e2a6a99889 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -1,4 +1,6 @@
 .container_job_template: &container_job_definition
+  tags:
+  - docker
   image: docker:stable
   stage: containers
   services:
diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index d5098c986b8..e59fbfdc73f 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -1,4 +1,6 @@
 .cross_system_build_job:
+  tags:
+  - docker
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -18,6 +20,8 @@
 # KVM), and set extra options (such disabling other accelerators) via the
 # $ACCEL_CONFIGURE_OPTS variable.
 .cross_accel_build_job:
+  tags:
+  - docker
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 30m
diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
index ba7280605c4..afbd8e4d915 100644
--- a/.gitlab-ci.d/edk2.yml
+++ b/.gitlab-ci.d/edk2.yml
@@ -5,6 +5,8 @@ docker-edk2:
    - .gitlab-ci.d/edk2.yml
    - .gitlab-ci.d/edk2/Dockerfile
    when: always
+ tags:
+ - docker
  image: docker:19.03.1
  services:
  - docker:19.03.1-dind
@@ -24,6 +26,8 @@ docker-edk2:
  - docker push $IMAGE_TAG
 
 build-edk2:
+ tags:
+ - docker
  stage: build
  needs: ['docker-edk2']
  rules: # Only run this job when ...
diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
index f66cd1d9089..a4a93222c2d 100644
--- a/.gitlab-ci.d/opensbi.yml
+++ b/.gitlab-ci.d/opensbi.yml
@@ -5,6 +5,8 @@ docker-opensbi:
    - .gitlab-ci.d/opensbi.yml
    - .gitlab-ci.d/opensbi/Dockerfile
    when: always
+ tags:
+ - docker
  image: docker:19.03.1
  services:
  - docker:19.03.1-dind
@@ -24,6 +26,8 @@ docker-opensbi:
  - docker push $IMAGE_TAG
 
 build-opensbi:
+ tags:
+ - docker
  stage: build
  needs: ['docker-opensbi']
  rules: # Only run this job when ...
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index f65cb11c4d3..d4511cf7dea 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -14,6 +14,8 @@ include:
   - local: '/.gitlab-ci.d/crossbuilds.yml'
 
 .native_build_job_template: &native_build_job_definition
+  tags:
+  - docker
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
@@ -38,6 +40,8 @@ include:
       fi
 
 .native_test_job_template: &native_test_job_definition
+  tags:
+  - docker
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
-- 
2.26.2



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] gitlab-ci: Restrict jobs using Docker to runners having 'docker' tag
  2021-03-19  0:43 [PATCH] gitlab-ci: Restrict jobs using Docker to runners having 'docker' tag Philippe Mathieu-Daudé
@ 2021-03-19  5:40 ` Thomas Huth
  2021-03-19 14:51   ` Laszlo Ersek
  2021-04-14 10:10   ` Philippe Mathieu-Daudé
  0 siblings, 2 replies; 5+ messages in thread
From: Thomas Huth @ 2021-03-19  5:40 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Daniel P . Berrange, Alex Bennée,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Bin Meng, Laszlo Ersek

On 19/03/2021 01.43, Philippe Mathieu-Daudé wrote:
> When a job is based on a Docker image [1], or is using a Docker
> service, it requires a runner with Docker installed.
> 
> Gitlab shared runners provide the 'docker' tag when they have it
> installed.
> 
> Are Gitlab shared runners are limited resources, we'd like to

s/Are/As/

> add more runners to QEMU repositories hosted on Gitlab. If a
> runner doesn't provide Docker, our jobs requiring it will fail.
> 
> Use the standard 'docker' tag to mark the jobs requiring Docker
> on the runner.
> 
> [1] https://docs.gitlab.com/ee/ci/yaml/#image
> [2] https://docs.gitlab.com/ee/ci/yaml/#services
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
[...]
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index f65cb11c4d3..d4511cf7dea 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -14,6 +14,8 @@ include:
>     - local: '/.gitlab-ci.d/crossbuilds.yml'
>   
>   .native_build_job_template: &native_build_job_definition
> +  tags:
> +  - docker
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     before_script:
> @@ -38,6 +40,8 @@ include:
>         fi
>   
>   .native_test_job_template: &native_test_job_definition
> +  tags:
> +  - docker
>     stage: test
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     script:

If you add it to the templates ... won't this disable most of the jobs on 
the dedicated runners that don't have docker? Wouldn't it be better to add 
the tag only to the jobs that run "make check-tcg" ?

  Thomas



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] gitlab-ci: Restrict jobs using Docker to runners having 'docker' tag
  2021-03-19  5:40 ` Thomas Huth
@ 2021-03-19 14:51   ` Laszlo Ersek
  2021-04-14 10:10   ` Philippe Mathieu-Daudé
  1 sibling, 0 replies; 5+ messages in thread
From: Laszlo Ersek @ 2021-03-19 14:51 UTC (permalink / raw)
  To: Thomas Huth, Philippe Mathieu-Daudé, qemu-devel
  Cc: Daniel P . Berrange, Wainer dos Santos Moschetta,
	Willian Rampazzo, Cleber Rosa, Bin Meng, Alex Bennée

On 03/19/21 06:40, Thomas Huth wrote:
> On 19/03/2021 01.43, Philippe Mathieu-Daudé wrote:
>> When a job is based on a Docker image [1], or is using a Docker
>> service, it requires a runner with Docker installed.
>>
>> Gitlab shared runners provide the 'docker' tag when they have it
>> installed.
>>
>> Are Gitlab shared runners are limited resources, we'd like to
> 
> s/Are/As/
> 
>> add more runners to QEMU repositories hosted on Gitlab. If a
>> runner doesn't provide Docker, our jobs requiring it will fail.
>>
>> Use the standard 'docker' tag to mark the jobs requiring Docker
>> on the runner.
>>
>> [1] https://docs.gitlab.com/ee/ci/yaml/#image
>> [2] https://docs.gitlab.com/ee/ci/yaml/#services
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> [...]
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index f65cb11c4d3..d4511cf7dea 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -14,6 +14,8 @@ include:
>>     - local: '/.gitlab-ci.d/crossbuilds.yml'
>>     .native_build_job_template: &native_build_job_definition
>> +  tags:
>> +  - docker
>>     stage: build
>>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>     before_script:
>> @@ -38,6 +40,8 @@ include:
>>         fi
>>     .native_test_job_template: &native_test_job_definition
>> +  tags:
>> +  - docker
>>     stage: test
>>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>     script:
> 
> If you add it to the templates ... won't this disable most of the jobs
> on the dedicated runners that don't have docker? Wouldn't it be better
> to add the tag only to the jobs that run "make check-tcg" ?

(I don't know if the docker dependency is presently expressed with the
exact granularity that we need, but I'm willing to ACK the edk2 part, on
principle. We should be explicit about dependencies.)

Thanks
Laszlo



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] gitlab-ci: Restrict jobs using Docker to runners having 'docker' tag
  2021-03-19  5:40 ` Thomas Huth
  2021-03-19 14:51   ` Laszlo Ersek
@ 2021-04-14 10:10   ` Philippe Mathieu-Daudé
  2021-05-10 14:53     ` Philippe Mathieu-Daudé
  1 sibling, 1 reply; 5+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-04-14 10:10 UTC (permalink / raw)
  To: Thomas Huth, qemu-devel
  Cc: Daniel P . Berrange, Laszlo Ersek, Wainer dos Santos Moschetta,
	Willian Rampazzo, Cleber Rosa, Bin Meng, Alex Bennée

On 3/19/21 6:40 AM, Thomas Huth wrote:
> On 19/03/2021 01.43, Philippe Mathieu-Daudé wrote:
>> When a job is based on a Docker image [1], or is using a Docker
>> service, it requires a runner with Docker installed.
>>
>> Gitlab shared runners provide the 'docker' tag when they have it
>> installed.
>>
>> Are Gitlab shared runners are limited resources, we'd like to
> 
> s/Are/As/
> 
>> add more runners to QEMU repositories hosted on Gitlab. If a
>> runner doesn't provide Docker, our jobs requiring it will fail.
>>
>> Use the standard 'docker' tag to mark the jobs requiring Docker
>> on the runner.
>>
>> [1] https://docs.gitlab.com/ee/ci/yaml/#image
>> [2] https://docs.gitlab.com/ee/ci/yaml/#services
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> [...]
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index f65cb11c4d3..d4511cf7dea 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -14,6 +14,8 @@ include:
>>     - local: '/.gitlab-ci.d/crossbuilds.yml'
>>     .native_build_job_template: &native_build_job_definition
>> +  tags:
>> +  - docker
>>     stage: build
>>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>     before_script:
>> @@ -38,6 +40,8 @@ include:
>>         fi
>>     .native_test_job_template: &native_test_job_definition
>> +  tags:
>> +  - docker
>>     stage: test
>>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>     script:
> 
> If you add it to the templates ... won't this disable most of the jobs
> on the dedicated runners that don't have docker? Wouldn't it be better
> to add the tag only to the jobs that run "make check-tcg" ?

But this is the point, if a runner doesn't have Docker, it can not
run the job...


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] gitlab-ci: Restrict jobs using Docker to runners having 'docker' tag
  2021-04-14 10:10   ` Philippe Mathieu-Daudé
@ 2021-05-10 14:53     ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 5+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-10 14:53 UTC (permalink / raw)
  To: Thomas Huth, qemu-devel
  Cc: Daniel P . Berrange, Alex Bennée,
	Wainer dos Santos Moschetta, Willian Rampazzo, Cleber Rosa,
	Bin Meng, Laszlo Ersek

On 4/14/21 12:10 PM, Philippe Mathieu-Daudé wrote:
> On 3/19/21 6:40 AM, Thomas Huth wrote:
>> On 19/03/2021 01.43, Philippe Mathieu-Daudé wrote:
>>> When a job is based on a Docker image [1], or is using a Docker
>>> service, it requires a runner with Docker installed.
>>>
>>> Gitlab shared runners provide the 'docker' tag when they have it
>>> installed.
>>>
>>> Are Gitlab shared runners are limited resources, we'd like to
>>
>> s/Are/As/
>>
>>> add more runners to QEMU repositories hosted on Gitlab. If a
>>> runner doesn't provide Docker, our jobs requiring it will fail.
>>>
>>> Use the standard 'docker' tag to mark the jobs requiring Docker
>>> on the runner.
>>>
>>> [1] https://docs.gitlab.com/ee/ci/yaml/#image
>>> [2] https://docs.gitlab.com/ee/ci/yaml/#services
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> [...]
>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>> index f65cb11c4d3..d4511cf7dea 100644
>>> --- a/.gitlab-ci.yml
>>> +++ b/.gitlab-ci.yml
>>> @@ -14,6 +14,8 @@ include:
>>>     - local: '/.gitlab-ci.d/crossbuilds.yml'
>>>     .native_build_job_template: &native_build_job_definition
>>> +  tags:
>>> +  - docker
>>>     stage: build
>>>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>>     before_script:
>>> @@ -38,6 +40,8 @@ include:
>>>         fi
>>>     .native_test_job_template: &native_test_job_definition
>>> +  tags:
>>> +  - docker
>>>     stage: test
>>>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>>     script:
>>
>> If you add it to the templates ... won't this disable most of the jobs
>> on the dedicated runners that don't have docker? Wouldn't it be better
>> to add the tag only to the jobs that run "make check-tcg" ?
> 
> But this is the point, if a runner doesn't have Docker, it can not
> run the job...

Apparently gitlab isn't clever enough to figure the 'image:' tag implies
we are expecting Docker... I suppose they wanted to keep it simple and
filter with runner tags.

Now the public runners are named 'gitlab-org-docker', see:

https://gitlab.com/gitlab-org/gitlab-ui/-/merge_requests/1267/diffs
https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/800/diffs
https://gitlab.com/gitlab-org/gitlab-docs/blob/master/.gitlab-ci.yml#L483
https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9685

Long term it would be simpler if we use gitlab recommended templates,
so we don't have to update ours when they change.

Meanwhile I'll simpy respin using 'gitlab-org-docker'.

Regards,

Phil.


^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-05-10 14:55 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-19  0:43 [PATCH] gitlab-ci: Restrict jobs using Docker to runners having 'docker' tag Philippe Mathieu-Daudé
2021-03-19  5:40 ` Thomas Huth
2021-03-19 14:51   ` Laszlo Ersek
2021-04-14 10:10   ` Philippe Mathieu-Daudé
2021-05-10 14:53     ` Philippe Mathieu-Daudé

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).