* [RFC PATCH] gitlab-ci.yml: Speed up CI by using "meson test --no-rebuild"
@ 2021-01-24 7:58 Thomas Huth
2021-01-24 11:32 ` Paolo Bonzini
0 siblings, 1 reply; 3+ messages in thread
From: Thomas Huth @ 2021-01-24 7:58 UTC (permalink / raw)
To: qemu-devel, Paolo Bonzini
Cc: Philippe Mathieu-Daudé, Daniel P . Berrangé,
Wainer dos Santos Moschetta
Currently, our check-system-* jobs are recompiling the whole sources
again. This happens due to the fact that the jobs are checking out
the whole source tree and required submodules again, and only try
to use the "build" directory with the binaries and object files
as an artifact from the previous stage - which simply does not work
anymore (with the current version of meson). Due to some changed
time stamps, meson is always trying to rebuild the whole tree.
To fix this problem, use "meson test --no-rebuild" instead of
"make check" to avoid rebuilding all binaries every time. This
saves ca. 15 - 20 minutes of precious CI cycles in each run.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
Marked as "RFC" since I'm not quite sure whether "meson test" has
the same test coverage as "make check"... Paolo?
.gitlab-ci.yml | 41 ++++++++++++++++++++++-------------------
1 file changed, 22 insertions(+), 19 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index de3a3d25b5..c9fb11c325 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -34,6 +34,19 @@ include:
make -j"$JOBS" $MAKE_CHECK_ARGS ;
fi
+.native_meson_test_job:
+ stage: test
+ image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+ script:
+ - cd build
+ - touch *
+ - make git-submodule-update
+ - if [ -x ../meson/meson.py ]; then
+ ../meson/meson.py test --no-rebuild -t 5 $MESON_TEST_ARGS ;
+ else
+ meson test --no-rebuild -t 5 $MESON_TEST_ARGS ;
+ fi
+
.native_test_job_template: &native_test_job_definition
stage: test
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
@@ -83,17 +96,15 @@ build-system-alpine:
artifacts:
expire_in: 2 days
paths:
- - .git-submodule-status
- build
check-system-alpine:
- <<: *native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-alpine
artifacts: true
variables:
IMAGE: alpine
- MAKE_CHECK_ARGS: check
acceptance-system-alpine:
<<: *native_test_job_definition
@@ -118,13 +129,12 @@ build-system-ubuntu:
- build
check-system-ubuntu:
- <<: *native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-ubuntu
artifacts: true
variables:
IMAGE: ubuntu2004
- MAKE_CHECK_ARGS: check
acceptance-system-ubuntu:
<<: *native_test_job_definition
@@ -149,13 +159,12 @@ build-system-debian:
- build
check-system-debian:
- <<: *native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-debian
artifacts: true
variables:
IMAGE: debian-amd64
- MAKE_CHECK_ARGS: check
# No targets are built here, just tools, docs, and unit tests. This
# also feeds into the eventual documentation deployment steps later
@@ -194,13 +203,12 @@ build-system-fedora:
- build
check-system-fedora:
- <<: *native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-fedora
artifacts: true
variables:
IMAGE: fedora
- MAKE_CHECK_ARGS: check
acceptance-system-fedora:
<<: *native_test_job_definition
@@ -226,13 +234,12 @@ build-system-centos:
- build
check-system-centos:
- <<: *native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-centos
artifacts: true
variables:
IMAGE: centos8
- MAKE_CHECK_ARGS: check
acceptance-system-centos:
<<: *native_test_job_definition
@@ -256,13 +263,12 @@ build-system-opensuse:
- build
check-system-opensuse:
- <<: *native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-system-opensuse
artifacts: true
variables:
IMAGE: opensuse-leap
- MAKE_CHECK_ARGS: check
acceptance-system-opensuse:
<<: *native_test_job_definition
@@ -525,13 +531,12 @@ build-crypto-old-nettle:
- build
check-crypto-old-nettle:
- <<: *native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-crypto-old-nettle
artifacts: true
variables:
IMAGE: centos7
- MAKE_CHECK_ARGS: check
build-crypto-old-gcrypt:
@@ -546,13 +551,12 @@ build-crypto-old-gcrypt:
- build
check-crypto-old-gcrypt:
- <<: *native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-crypto-old-gcrypt
artifacts: true
variables:
IMAGE: centos7
- MAKE_CHECK_ARGS: check
build-crypto-only-gnutls:
@@ -567,13 +571,12 @@ build-crypto-only-gnutls:
- build
check-crypto-only-gnutls:
- <<: *native_test_job_definition
+ extends: .native_meson_test_job
needs:
- job: build-crypto-only-gnutls
artifacts: true
variables:
IMAGE: centos7
- MAKE_CHECK_ARGS: check
# We don't need to exercise every backend with every front-end
build-trace-multi-user:
--
2.27.0
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [RFC PATCH] gitlab-ci.yml: Speed up CI by using "meson test --no-rebuild"
2021-01-24 7:58 [RFC PATCH] gitlab-ci.yml: Speed up CI by using "meson test --no-rebuild" Thomas Huth
@ 2021-01-24 11:32 ` Paolo Bonzini
2021-01-25 9:09 ` Thomas Huth
0 siblings, 1 reply; 3+ messages in thread
From: Paolo Bonzini @ 2021-01-24 11:32 UTC (permalink / raw)
To: Thomas Huth
Cc: Philippe Mathieu-Daudé, Daniel P . Berrangé,
qemu-devel, Wainer dos Santos Moschetta
[-- Attachment #1: Type: text/plain, Size: 5333 bytes --]
check-block is not run by "meson test".
Paolo
Il dom 24 gen 2021, 08:58 Thomas Huth <thuth@redhat.com> ha scritto:
> Currently, our check-system-* jobs are recompiling the whole sources
> again. This happens due to the fact that the jobs are checking out
> the whole source tree and required submodules again, and only try
> to use the "build" directory with the binaries and object files
> as an artifact from the previous stage - which simply does not work
> anymore (with the current version of meson). Due to some changed
> time stamps, meson is always trying to rebuild the whole tree.
>
> To fix this problem, use "meson test --no-rebuild" instead of
> "make check" to avoid rebuilding all binaries every time. This
> saves ca. 15 - 20 minutes of precious CI cycles in each run.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> Marked as "RFC" since I'm not quite sure whether "meson test" has
> the same test coverage as "make check"... Paolo?
>
> .gitlab-ci.yml | 41 ++++++++++++++++++++++-------------------
> 1 file changed, 22 insertions(+), 19 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index de3a3d25b5..c9fb11c325 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -34,6 +34,19 @@ include:
> make -j"$JOBS" $MAKE_CHECK_ARGS ;
> fi
>
> +.native_meson_test_job:
> + stage: test
> + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> + script:
> + - cd build
> + - touch *
> + - make git-submodule-update
> + - if [ -x ../meson/meson.py ]; then
> + ../meson/meson.py test --no-rebuild -t 5 $MESON_TEST_ARGS ;
> + else
> + meson test --no-rebuild -t 5 $MESON_TEST_ARGS ;
> + fi
> +
> .native_test_job_template: &native_test_job_definition
> stage: test
> image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> @@ -83,17 +96,15 @@ build-system-alpine:
> artifacts:
> expire_in: 2 days
> paths:
> - - .git-submodule-status
> - build
>
> check-system-alpine:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-alpine
> artifacts: true
> variables:
> IMAGE: alpine
> - MAKE_CHECK_ARGS: check
>
> acceptance-system-alpine:
> <<: *native_test_job_definition
> @@ -118,13 +129,12 @@ build-system-ubuntu:
> - build
>
> check-system-ubuntu:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-ubuntu
> artifacts: true
> variables:
> IMAGE: ubuntu2004
> - MAKE_CHECK_ARGS: check
>
> acceptance-system-ubuntu:
> <<: *native_test_job_definition
> @@ -149,13 +159,12 @@ build-system-debian:
> - build
>
> check-system-debian:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-debian
> artifacts: true
> variables:
> IMAGE: debian-amd64
> - MAKE_CHECK_ARGS: check
>
> # No targets are built here, just tools, docs, and unit tests. This
> # also feeds into the eventual documentation deployment steps later
> @@ -194,13 +203,12 @@ build-system-fedora:
> - build
>
> check-system-fedora:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-fedora
> artifacts: true
> variables:
> IMAGE: fedora
> - MAKE_CHECK_ARGS: check
>
> acceptance-system-fedora:
> <<: *native_test_job_definition
> @@ -226,13 +234,12 @@ build-system-centos:
> - build
>
> check-system-centos:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-centos
> artifacts: true
> variables:
> IMAGE: centos8
> - MAKE_CHECK_ARGS: check
>
> acceptance-system-centos:
> <<: *native_test_job_definition
> @@ -256,13 +263,12 @@ build-system-opensuse:
> - build
>
> check-system-opensuse:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-opensuse
> artifacts: true
> variables:
> IMAGE: opensuse-leap
> - MAKE_CHECK_ARGS: check
>
> acceptance-system-opensuse:
> <<: *native_test_job_definition
> @@ -525,13 +531,12 @@ build-crypto-old-nettle:
> - build
>
> check-crypto-old-nettle:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-crypto-old-nettle
> artifacts: true
> variables:
> IMAGE: centos7
> - MAKE_CHECK_ARGS: check
>
>
> build-crypto-old-gcrypt:
> @@ -546,13 +551,12 @@ build-crypto-old-gcrypt:
> - build
>
> check-crypto-old-gcrypt:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-crypto-old-gcrypt
> artifacts: true
> variables:
> IMAGE: centos7
> - MAKE_CHECK_ARGS: check
>
>
> build-crypto-only-gnutls:
> @@ -567,13 +571,12 @@ build-crypto-only-gnutls:
> - build
>
> check-crypto-only-gnutls:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-crypto-only-gnutls
> artifacts: true
> variables:
> IMAGE: centos7
> - MAKE_CHECK_ARGS: check
>
> # We don't need to exercise every backend with every front-end
> build-trace-multi-user:
> --
> 2.27.0
>
>
[-- Attachment #2: Type: text/html, Size: 6757 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [RFC PATCH] gitlab-ci.yml: Speed up CI by using "meson test --no-rebuild"
2021-01-24 11:32 ` Paolo Bonzini
@ 2021-01-25 9:09 ` Thomas Huth
0 siblings, 0 replies; 3+ messages in thread
From: Thomas Huth @ 2021-01-25 9:09 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Daniel P . Berrangé, Philippe Mathieu-Daudé,
qemu-devel, Wainer dos Santos Moschetta
On 24/01/2021 12.32, Paolo Bonzini wrote:
> check-block is not run by "meson test".
Thanks! I've just send a v1 now which also runs the iotests again.
Thomas
>
> Il dom 24 gen 2021, 08:58 Thomas Huth <thuth@redhat.com
> <mailto:thuth@redhat.com>> ha scritto:
>
> Currently, our check-system-* jobs are recompiling the whole sources
> again. This happens due to the fact that the jobs are checking out
> the whole source tree and required submodules again, and only try
> to use the "build" directory with the binaries and object files
> as an artifact from the previous stage - which simply does not work
> anymore (with the current version of meson). Due to some changed
> time stamps, meson is always trying to rebuild the whole tree.
>
> To fix this problem, use "meson test --no-rebuild" instead of
> "make check" to avoid rebuilding all binaries every time. This
> saves ca. 15 - 20 minutes of precious CI cycles in each run.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com <mailto:thuth@redhat.com>>
> ---
> Marked as "RFC" since I'm not quite sure whether "meson test" has
> the same test coverage as "make check"... Paolo?
>
> .gitlab-ci.yml | 41 ++++++++++++++++++++++-------------------
> 1 file changed, 22 insertions(+), 19 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index de3a3d25b5..c9fb11c325 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -34,6 +34,19 @@ include:
> make -j"$JOBS" $MAKE_CHECK_ARGS ;
> fi
>
> +.native_meson_test_job:
> + stage: test
> + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> + script:
> + - cd build
> + - touch *
> + - make git-submodule-update
> + - if [ -x ../meson/meson.py ]; then
> + ../meson/meson.py test --no-rebuild -t 5 $MESON_TEST_ARGS ;
> + else
> + meson test --no-rebuild -t 5 $MESON_TEST_ARGS ;
> + fi
> +
> .native_test_job_template: &native_test_job_definition
> stage: test
> image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> @@ -83,17 +96,15 @@ build-system-alpine:
> artifacts:
> expire_in: 2 days
> paths:
> - - .git-submodule-status
> - build
>
> check-system-alpine:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-alpine
> artifacts: true
> variables:
> IMAGE: alpine
> - MAKE_CHECK_ARGS: check
>
> acceptance-system-alpine:
> <<: *native_test_job_definition
> @@ -118,13 +129,12 @@ build-system-ubuntu:
> - build
>
> check-system-ubuntu:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-ubuntu
> artifacts: true
> variables:
> IMAGE: ubuntu2004
> - MAKE_CHECK_ARGS: check
>
> acceptance-system-ubuntu:
> <<: *native_test_job_definition
> @@ -149,13 +159,12 @@ build-system-debian:
> - build
>
> check-system-debian:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-debian
> artifacts: true
> variables:
> IMAGE: debian-amd64
> - MAKE_CHECK_ARGS: check
>
> # No targets are built here, just tools, docs, and unit tests. This
> # also feeds into the eventual documentation deployment steps later
> @@ -194,13 +203,12 @@ build-system-fedora:
> - build
>
> check-system-fedora:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-fedora
> artifacts: true
> variables:
> IMAGE: fedora
> - MAKE_CHECK_ARGS: check
>
> acceptance-system-fedora:
> <<: *native_test_job_definition
> @@ -226,13 +234,12 @@ build-system-centos:
> - build
>
> check-system-centos:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-centos
> artifacts: true
> variables:
> IMAGE: centos8
> - MAKE_CHECK_ARGS: check
>
> acceptance-system-centos:
> <<: *native_test_job_definition
> @@ -256,13 +263,12 @@ build-system-opensuse:
> - build
>
> check-system-opensuse:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-system-opensuse
> artifacts: true
> variables:
> IMAGE: opensuse-leap
> - MAKE_CHECK_ARGS: check
>
> acceptance-system-opensuse:
> <<: *native_test_job_definition
> @@ -525,13 +531,12 @@ build-crypto-old-nettle:
> - build
>
> check-crypto-old-nettle:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-crypto-old-nettle
> artifacts: true
> variables:
> IMAGE: centos7
> - MAKE_CHECK_ARGS: check
>
>
> build-crypto-old-gcrypt:
> @@ -546,13 +551,12 @@ build-crypto-old-gcrypt:
> - build
>
> check-crypto-old-gcrypt:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-crypto-old-gcrypt
> artifacts: true
> variables:
> IMAGE: centos7
> - MAKE_CHECK_ARGS: check
>
>
> build-crypto-only-gnutls:
> @@ -567,13 +571,12 @@ build-crypto-only-gnutls:
> - build
>
> check-crypto-only-gnutls:
> - <<: *native_test_job_definition
> + extends: .native_meson_test_job
> needs:
> - job: build-crypto-only-gnutls
> artifacts: true
> variables:
> IMAGE: centos7
> - MAKE_CHECK_ARGS: check
>
> # We don't need to exercise every backend with every front-end
> build-trace-multi-user:
> --
> 2.27.0
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-01-25 9:10 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-24 7:58 [RFC PATCH] gitlab-ci.yml: Speed up CI by using "meson test --no-rebuild" Thomas Huth
2021-01-24 11:32 ` Paolo Bonzini
2021-01-25 9:09 ` Thomas Huth
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.