From: "Daniel P. Berrangé" <berrange@redhat.com>
To: Thomas Huth <thuth@redhat.com>
Cc: "Willian Rampazzo" <willianr@redhat.com>,
"Philippe Mathieu-Daudé" <philmd@redhat.com>,
"Alex Bennée" <alex.bennee@linaro.org>,
qemu-devel@nongnu.org,
"Wainer dos Santos Moschetta" <wainersm@redhat.com>
Subject: Re: [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI
Date: Fri, 22 Jan 2021 10:18:33 +0000 [thread overview]
Message-ID: <20210122101833.GC3150238@redhat.com> (raw)
In-Reply-To: <20210122100722.705375-1-thuth@redhat.com>
On Fri, Jan 22, 2021 at 11:07:22AM +0100, Thomas Huth wrote:
> Currently, our check-system-* jobs are recompiling the whole sources
> again. This happens due to the fact that the jobs are checking out
> the whole source tree and required submodules again, and only try
> to use the "build" directory with the binaries and object files
> as an artifact from the previous stage - which simply does not work
> anymore (with the current version of meson). Due to some changed
> time stamps, meson is always trying to rebuild the whole tree.
This used to work in the past didn't it ? Did something change in
meson to break this, or have we just not noticed before.
> So instead of trying to marry a freshly checked out source tree
> with the pre-built binaries in these jobs, let's simply pass the
> whole source including the submodules and the build tree as artifact
> to the test jobs. That way timestamps get preserved and there is
> no rebuild of the sources anymore. This saves ca. 15 - 20 minutes
> of precious CI cycles in each run.
I'm a little worried we might end up hitting the artifact size
limit which is supposedly 1GB on gitlab.com. Im guessing this
must be measuring the compressed size though, as a src checkout
with build dir and .git dir is already way over 1GB.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> This is how a job looked like before my patch, running for 42 minutes:
> https://gitlab.com/huth/qemu/-/jobs/978432757
>
> And this is how it looks like afterwards - it just took 18 minutes:
> https://gitlab.com/huth/qemu/-/jobs/979500316
>
> .gitlab-ci.d/containers.yml | 1 +
> .gitlab-ci.yml | 40 +++++++++++++++++++++++++------------
> 2 files changed, 28 insertions(+), 13 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index e2f9c99e27..d55280661f 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -39,7 +39,6 @@ include:
> image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> script:
> - cd build
> - - find . -type f -exec touch {} +
> - make $MAKE_CHECK_ARGS
>
> .acceptance_template: &acceptance_definition
> @@ -83,8 +82,7 @@ build-system-alpine:
> artifacts:
> expire_in: 2 days
> paths:
> - - .git-submodule-status
> - - build
> + - "*"
>
> check-system-alpine:
> <<: *native_test_job_definition
> @@ -92,6 +90,7 @@ check-system-alpine:
> - job: build-system-alpine
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: alpine
> MAKE_CHECK_ARGS: check
>
> @@ -101,6 +100,7 @@ acceptance-system-alpine:
> - job: build-system-alpine
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: alpine
> MAKE_CHECK_ARGS: check-acceptance
> <<: *acceptance_definition
> @@ -116,7 +116,7 @@ build-system-ubuntu:
> artifacts:
> expire_in: 2 days
> paths:
> - - build
> + - "*"
>
> check-system-ubuntu:
> <<: *native_test_job_definition
> @@ -124,6 +124,7 @@ check-system-ubuntu:
> - job: build-system-ubuntu
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: ubuntu2004
> MAKE_CHECK_ARGS: check
>
> @@ -133,6 +134,7 @@ acceptance-system-ubuntu:
> - job: build-system-ubuntu
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: ubuntu2004
> MAKE_CHECK_ARGS: check-acceptance
> <<: *acceptance_definition
> @@ -148,7 +150,7 @@ build-system-debian:
> artifacts:
> expire_in: 2 days
> paths:
> - - build
> + - "*"
>
> check-system-debian:
> <<: *native_test_job_definition
> @@ -156,6 +158,7 @@ check-system-debian:
> - job: build-system-debian
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: debian-amd64
> MAKE_CHECK_ARGS: check
>
> @@ -170,7 +173,7 @@ build-tools-and-docs-debian:
> artifacts:
> expire_in: 2 days
> paths:
> - - build
> + - "*"
>
> acceptance-system-debian:
> <<: *native_test_job_definition
> @@ -178,6 +181,7 @@ acceptance-system-debian:
> - job: build-system-debian
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: debian-amd64
> MAKE_CHECK_ARGS: check-acceptance
> <<: *acceptance_definition
> @@ -194,7 +198,7 @@ build-system-fedora:
> artifacts:
> expire_in: 2 days
> paths:
> - - build
> + - "*"
>
> check-system-fedora:
> <<: *native_test_job_definition
> @@ -202,6 +206,7 @@ check-system-fedora:
> - job: build-system-fedora
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: fedora
> MAKE_CHECK_ARGS: check
>
> @@ -211,6 +216,7 @@ acceptance-system-fedora:
> - job: build-system-fedora
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: fedora
> MAKE_CHECK_ARGS: check-acceptance
> <<: *acceptance_definition
> @@ -226,7 +232,7 @@ build-system-centos:
> artifacts:
> expire_in: 2 days
> paths:
> - - build
> + - "*"
>
> check-system-centos:
> <<: *native_test_job_definition
> @@ -234,6 +240,7 @@ check-system-centos:
> - job: build-system-centos
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: centos8
> MAKE_CHECK_ARGS: check
>
> @@ -243,6 +250,7 @@ acceptance-system-centos:
> - job: build-system-centos
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: centos8
> MAKE_CHECK_ARGS: check-acceptance
> <<: *acceptance_definition
> @@ -257,7 +265,7 @@ build-system-opensuse:
> artifacts:
> expire_in: 2 days
> paths:
> - - build
> + - "*"
>
> check-system-opensuse:
> <<: *native_test_job_definition
> @@ -265,6 +273,7 @@ check-system-opensuse:
> - job: build-system-opensuse
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: opensuse-leap
> MAKE_CHECK_ARGS: check
>
> @@ -274,6 +283,7 @@ acceptance-system-opensuse:
> - job: build-system-opensuse
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: opensuse-leap
> MAKE_CHECK_ARGS: check-acceptance
> <<: *acceptance_definition
> @@ -444,7 +454,7 @@ build-deprecated:
> artifacts:
> expire_in: 2 days
> paths:
> - - build
> + - "*"
>
> # We split the check-tcg step as test failures are expected but we still
> # want to catch the build breaking.
> @@ -454,6 +464,7 @@ check-deprecated:
> - job: build-deprecated
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: debian-all-test-cross
> MAKE_CHECK_ARGS: check-tcg
> allow_failure: true
> @@ -526,7 +537,7 @@ build-crypto-old-nettle:
> MAKE_CHECK_ARGS: check-build
> artifacts:
> paths:
> - - build
> + - "*"
>
> check-crypto-old-nettle:
> <<: *native_test_job_definition
> @@ -534,6 +545,7 @@ check-crypto-old-nettle:
> - job: build-crypto-old-nettle
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: centos7
> MAKE_CHECK_ARGS: check
>
> @@ -547,7 +559,7 @@ build-crypto-old-gcrypt:
> MAKE_CHECK_ARGS: check-build
> artifacts:
> paths:
> - - build
> + - "*"
>
> check-crypto-old-gcrypt:
> <<: *native_test_job_definition
> @@ -555,6 +567,7 @@ check-crypto-old-gcrypt:
> - job: build-crypto-old-gcrypt
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: centos7
> MAKE_CHECK_ARGS: check
>
> @@ -568,7 +581,7 @@ build-crypto-only-gnutls:
> MAKE_CHECK_ARGS: check-build
> artifacts:
> paths:
> - - build
> + - "*"
>
> check-crypto-only-gnutls:
> <<: *native_test_job_definition
> @@ -576,6 +589,7 @@ check-crypto-only-gnutls:
> - job: build-crypto-only-gnutls
> artifacts: true
> variables:
> + GIT_CHECKOUT: "false"
> IMAGE: centos7
> MAKE_CHECK_ARGS: check
>
> --
> 2.27.0
>
>
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
next prev parent reply other threads:[~2021-01-22 10:20 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-22 10:07 [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI Thomas Huth
2021-01-22 10:18 ` Daniel P. Berrangé [this message]
2021-01-22 10:20 ` Daniel P. Berrangé
2021-01-22 10:30 ` Philippe Mathieu-Daudé
2021-01-22 16:25 ` Thomas Huth
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210122101833.GC3150238@redhat.com \
--to=berrange@redhat.com \
--cc=alex.bennee@linaro.org \
--cc=philmd@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=thuth@redhat.com \
--cc=wainersm@redhat.com \
--cc=willianr@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.