All of lore.kernel.org
 help / color / mirror / Atom feed
From: Thomas Huth <thuth@redhat.com>
To: qemu-devel@nongnu.org, "Alex Bennée" <alex.bennee@linaro.org>
Cc: "Willian Rampazzo" <willianr@redhat.com>,
	"Philippe Mathieu-Daudé" <philmd@redhat.com>,
	"Wainer dos Santos Moschetta" <wainersm@redhat.com>
Subject: [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI
Date: Fri, 22 Jan 2021 11:07:22 +0100	[thread overview]
Message-ID: <20210122100722.705375-1-thuth@redhat.com> (raw)

Currently, our check-system-* jobs are recompiling the whole sources
again. This happens due to the fact that the jobs are checking out
the whole source tree and required submodules again, and only try
to use the "build" directory with the binaries and object files
as an artifact from the previous stage - which simply does not work
anymore (with the current version of meson). Due to some changed
time stamps, meson is always trying to rebuild the whole tree.

So instead of trying to marry a freshly checked out source tree
with the pre-built binaries in these jobs, let's simply pass the
whole source including the submodules and the build tree as artifact
to the test jobs. That way timestamps get preserved and there is
no rebuild of the sources anymore. This saves ca. 15 - 20 minutes
of precious CI cycles in each run.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 This is how a job looked like before my patch, running for 42 minutes:
 https://gitlab.com/huth/qemu/-/jobs/978432757

 And this is how it looks like afterwards - it just took 18 minutes:
 https://gitlab.com/huth/qemu/-/jobs/979500316

 .gitlab-ci.d/containers.yml |  1 +
 .gitlab-ci.yml              | 40 +++++++++++++++++++++++++------------
 2 files changed, 28 insertions(+), 13 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index e2f9c99e27..d55280661f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -39,7 +39,6 @@ include:
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
     - cd build
-    - find . -type f -exec touch {} +
     - make $MAKE_CHECK_ARGS
 
 .acceptance_template: &acceptance_definition
@@ -83,8 +82,7 @@ build-system-alpine:
   artifacts:
     expire_in: 2 days
     paths:
-      - .git-submodule-status
-      - build
+      - "*"
 
 check-system-alpine:
   <<: *native_test_job_definition
@@ -92,6 +90,7 @@ check-system-alpine:
     - job: build-system-alpine
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: alpine
     MAKE_CHECK_ARGS: check
 
@@ -101,6 +100,7 @@ acceptance-system-alpine:
     - job: build-system-alpine
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: alpine
     MAKE_CHECK_ARGS: check-acceptance
   <<: *acceptance_definition
@@ -116,7 +116,7 @@ build-system-ubuntu:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 check-system-ubuntu:
   <<: *native_test_job_definition
@@ -124,6 +124,7 @@ check-system-ubuntu:
     - job: build-system-ubuntu
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check
 
@@ -133,6 +134,7 @@ acceptance-system-ubuntu:
     - job: build-system-ubuntu
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check-acceptance
   <<: *acceptance_definition
@@ -148,7 +150,7 @@ build-system-debian:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 check-system-debian:
   <<: *native_test_job_definition
@@ -156,6 +158,7 @@ check-system-debian:
     - job: build-system-debian
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check
 
@@ -170,7 +173,7 @@ build-tools-and-docs-debian:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 acceptance-system-debian:
   <<: *native_test_job_definition
@@ -178,6 +181,7 @@ acceptance-system-debian:
     - job: build-system-debian
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-acceptance
   <<: *acceptance_definition
@@ -194,7 +198,7 @@ build-system-fedora:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 check-system-fedora:
   <<: *native_test_job_definition
@@ -202,6 +206,7 @@ check-system-fedora:
     - job: build-system-fedora
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
@@ -211,6 +216,7 @@ acceptance-system-fedora:
     - job: build-system-fedora
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
   <<: *acceptance_definition
@@ -226,7 +232,7 @@ build-system-centos:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 check-system-centos:
   <<: *native_test_job_definition
@@ -234,6 +240,7 @@ check-system-centos:
     - job: build-system-centos
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: centos8
     MAKE_CHECK_ARGS: check
 
@@ -243,6 +250,7 @@ acceptance-system-centos:
     - job: build-system-centos
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: centos8
     MAKE_CHECK_ARGS: check-acceptance
   <<: *acceptance_definition
@@ -257,7 +265,7 @@ build-system-opensuse:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 check-system-opensuse:
   <<: *native_test_job_definition
@@ -265,6 +273,7 @@ check-system-opensuse:
     - job: build-system-opensuse
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: opensuse-leap
     MAKE_CHECK_ARGS: check
 
@@ -274,6 +283,7 @@ acceptance-system-opensuse:
      - job: build-system-opensuse
        artifacts: true
    variables:
+     GIT_CHECKOUT: "false"
      IMAGE: opensuse-leap
      MAKE_CHECK_ARGS: check-acceptance
    <<: *acceptance_definition
@@ -444,7 +454,7 @@ build-deprecated:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 # We split the check-tcg step as test failures are expected but we still
 # want to catch the build breaking.
@@ -454,6 +464,7 @@ check-deprecated:
     - job: build-deprecated
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: debian-all-test-cross
     MAKE_CHECK_ARGS: check-tcg
   allow_failure: true
@@ -526,7 +537,7 @@ build-crypto-old-nettle:
     MAKE_CHECK_ARGS: check-build
   artifacts:
     paths:
-      - build
+      - "*"
 
 check-crypto-old-nettle:
   <<: *native_test_job_definition
@@ -534,6 +545,7 @@ check-crypto-old-nettle:
     - job: build-crypto-old-nettle
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: centos7
     MAKE_CHECK_ARGS: check
 
@@ -547,7 +559,7 @@ build-crypto-old-gcrypt:
     MAKE_CHECK_ARGS: check-build
   artifacts:
     paths:
-      - build
+      - "*"
 
 check-crypto-old-gcrypt:
   <<: *native_test_job_definition
@@ -555,6 +567,7 @@ check-crypto-old-gcrypt:
     - job: build-crypto-old-gcrypt
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: centos7
     MAKE_CHECK_ARGS: check
 
@@ -568,7 +581,7 @@ build-crypto-only-gnutls:
     MAKE_CHECK_ARGS: check-build
   artifacts:
     paths:
-      - build
+      - "*"
 
 check-crypto-only-gnutls:
   <<: *native_test_job_definition
@@ -576,6 +589,7 @@ check-crypto-only-gnutls:
     - job: build-crypto-only-gnutls
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: centos7
     MAKE_CHECK_ARGS: check
 
-- 
2.27.0



             reply	other threads:[~2021-01-22 10:08 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-22 10:07 Thomas Huth [this message]
2021-01-22 10:18 ` [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI Daniel P. Berrangé
2021-01-22 10:20   ` Daniel P. Berrangé
2021-01-22 10:30   ` Philippe Mathieu-Daudé
2021-01-22 16:25   ` Thomas Huth

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210122100722.705375-1-thuth@redhat.com \
    --to=thuth@redhat.com \
    --cc=alex.bennee@linaro.org \
    --cc=philmd@redhat.com \
    --cc=qemu-devel@nongnu.org \
    --cc=wainersm@redhat.com \
    --cc=willianr@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.