qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI
@ 2021-01-22 10:07 Thomas Huth
  2021-01-22 10:18 ` Daniel P. Berrangé
  0 siblings, 1 reply; 5+ messages in thread
From: Thomas Huth @ 2021-01-22 10:07 UTC (permalink / raw)
  To: qemu-devel, Alex Bennée
  Cc: Willian Rampazzo, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta

Currently, our check-system-* jobs are recompiling the whole sources
again. This happens due to the fact that the jobs are checking out
the whole source tree and required submodules again, and only try
to use the "build" directory with the binaries and object files
as an artifact from the previous stage - which simply does not work
anymore (with the current version of meson). Due to some changed
time stamps, meson is always trying to rebuild the whole tree.

So instead of trying to marry a freshly checked out source tree
with the pre-built binaries in these jobs, let's simply pass the
whole source including the submodules and the build tree as artifact
to the test jobs. That way timestamps get preserved and there is
no rebuild of the sources anymore. This saves ca. 15 - 20 minutes
of precious CI cycles in each run.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 This is how a job looked like before my patch, running for 42 minutes:
 https://gitlab.com/huth/qemu/-/jobs/978432757

 And this is how it looks like afterwards - it just took 18 minutes:
 https://gitlab.com/huth/qemu/-/jobs/979500316

 .gitlab-ci.d/containers.yml |  1 +
 .gitlab-ci.yml              | 40 +++++++++++++++++++++++++------------
 2 files changed, 28 insertions(+), 13 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index e2f9c99e27..d55280661f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -39,7 +39,6 @@ include:
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
     - cd build
-    - find . -type f -exec touch {} +
     - make $MAKE_CHECK_ARGS
 
 .acceptance_template: &acceptance_definition
@@ -83,8 +82,7 @@ build-system-alpine:
   artifacts:
     expire_in: 2 days
     paths:
-      - .git-submodule-status
-      - build
+      - "*"
 
 check-system-alpine:
   <<: *native_test_job_definition
@@ -92,6 +90,7 @@ check-system-alpine:
     - job: build-system-alpine
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: alpine
     MAKE_CHECK_ARGS: check
 
@@ -101,6 +100,7 @@ acceptance-system-alpine:
     - job: build-system-alpine
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: alpine
     MAKE_CHECK_ARGS: check-acceptance
   <<: *acceptance_definition
@@ -116,7 +116,7 @@ build-system-ubuntu:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 check-system-ubuntu:
   <<: *native_test_job_definition
@@ -124,6 +124,7 @@ check-system-ubuntu:
     - job: build-system-ubuntu
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check
 
@@ -133,6 +134,7 @@ acceptance-system-ubuntu:
     - job: build-system-ubuntu
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check-acceptance
   <<: *acceptance_definition
@@ -148,7 +150,7 @@ build-system-debian:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 check-system-debian:
   <<: *native_test_job_definition
@@ -156,6 +158,7 @@ check-system-debian:
     - job: build-system-debian
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check
 
@@ -170,7 +173,7 @@ build-tools-and-docs-debian:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 acceptance-system-debian:
   <<: *native_test_job_definition
@@ -178,6 +181,7 @@ acceptance-system-debian:
     - job: build-system-debian
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-acceptance
   <<: *acceptance_definition
@@ -194,7 +198,7 @@ build-system-fedora:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 check-system-fedora:
   <<: *native_test_job_definition
@@ -202,6 +206,7 @@ check-system-fedora:
     - job: build-system-fedora
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
@@ -211,6 +216,7 @@ acceptance-system-fedora:
     - job: build-system-fedora
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
   <<: *acceptance_definition
@@ -226,7 +232,7 @@ build-system-centos:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 check-system-centos:
   <<: *native_test_job_definition
@@ -234,6 +240,7 @@ check-system-centos:
     - job: build-system-centos
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: centos8
     MAKE_CHECK_ARGS: check
 
@@ -243,6 +250,7 @@ acceptance-system-centos:
     - job: build-system-centos
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: centos8
     MAKE_CHECK_ARGS: check-acceptance
   <<: *acceptance_definition
@@ -257,7 +265,7 @@ build-system-opensuse:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 check-system-opensuse:
   <<: *native_test_job_definition
@@ -265,6 +273,7 @@ check-system-opensuse:
     - job: build-system-opensuse
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: opensuse-leap
     MAKE_CHECK_ARGS: check
 
@@ -274,6 +283,7 @@ acceptance-system-opensuse:
      - job: build-system-opensuse
        artifacts: true
    variables:
+     GIT_CHECKOUT: "false"
      IMAGE: opensuse-leap
      MAKE_CHECK_ARGS: check-acceptance
    <<: *acceptance_definition
@@ -444,7 +454,7 @@ build-deprecated:
   artifacts:
     expire_in: 2 days
     paths:
-      - build
+      - "*"
 
 # We split the check-tcg step as test failures are expected but we still
 # want to catch the build breaking.
@@ -454,6 +464,7 @@ check-deprecated:
     - job: build-deprecated
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: debian-all-test-cross
     MAKE_CHECK_ARGS: check-tcg
   allow_failure: true
@@ -526,7 +537,7 @@ build-crypto-old-nettle:
     MAKE_CHECK_ARGS: check-build
   artifacts:
     paths:
-      - build
+      - "*"
 
 check-crypto-old-nettle:
   <<: *native_test_job_definition
@@ -534,6 +545,7 @@ check-crypto-old-nettle:
     - job: build-crypto-old-nettle
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: centos7
     MAKE_CHECK_ARGS: check
 
@@ -547,7 +559,7 @@ build-crypto-old-gcrypt:
     MAKE_CHECK_ARGS: check-build
   artifacts:
     paths:
-      - build
+      - "*"
 
 check-crypto-old-gcrypt:
   <<: *native_test_job_definition
@@ -555,6 +567,7 @@ check-crypto-old-gcrypt:
     - job: build-crypto-old-gcrypt
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: centos7
     MAKE_CHECK_ARGS: check
 
@@ -568,7 +581,7 @@ build-crypto-only-gnutls:
     MAKE_CHECK_ARGS: check-build
   artifacts:
     paths:
-      - build
+      - "*"
 
 check-crypto-only-gnutls:
   <<: *native_test_job_definition
@@ -576,6 +589,7 @@ check-crypto-only-gnutls:
     - job: build-crypto-only-gnutls
       artifacts: true
   variables:
+    GIT_CHECKOUT: "false"
     IMAGE: centos7
     MAKE_CHECK_ARGS: check
 
-- 
2.27.0



^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI
  2021-01-22 10:07 [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI Thomas Huth
@ 2021-01-22 10:18 ` Daniel P. Berrangé
  2021-01-22 10:20   ` Daniel P. Berrangé
                     ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: Daniel P. Berrangé @ 2021-01-22 10:18 UTC (permalink / raw)
  To: Thomas Huth
  Cc: Willian Rampazzo, Philippe Mathieu-Daudé,
	Alex Bennée, qemu-devel, Wainer dos Santos Moschetta

On Fri, Jan 22, 2021 at 11:07:22AM +0100, Thomas Huth wrote:
> Currently, our check-system-* jobs are recompiling the whole sources
> again. This happens due to the fact that the jobs are checking out
> the whole source tree and required submodules again, and only try
> to use the "build" directory with the binaries and object files
> as an artifact from the previous stage - which simply does not work
> anymore (with the current version of meson). Due to some changed
> time stamps, meson is always trying to rebuild the whole tree.

This used to work in the past didn't it ? Did something change in
meson to break this, or have we just not noticed before.

> So instead of trying to marry a freshly checked out source tree
> with the pre-built binaries in these jobs, let's simply pass the
> whole source including the submodules and the build tree as artifact
> to the test jobs. That way timestamps get preserved and there is
> no rebuild of the sources anymore. This saves ca. 15 - 20 minutes
> of precious CI cycles in each run.

I'm a little worried we might end up hitting the artifact size
limit which is supposedly 1GB on gitlab.com.  Im guessing this
must be measuring the compressed size though, as a src checkout
with build dir  and .git dir is already way over 1GB.

> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  This is how a job looked like before my patch, running for 42 minutes:
>  https://gitlab.com/huth/qemu/-/jobs/978432757
> 
>  And this is how it looks like afterwards - it just took 18 minutes:
>  https://gitlab.com/huth/qemu/-/jobs/979500316
> 
>  .gitlab-ci.d/containers.yml |  1 +
>  .gitlab-ci.yml              | 40 +++++++++++++++++++++++++------------
>  2 files changed, 28 insertions(+), 13 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index e2f9c99e27..d55280661f 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -39,7 +39,6 @@ include:
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    script:
>      - cd build
> -    - find . -type f -exec touch {} +
>      - make $MAKE_CHECK_ARGS
>  
>  .acceptance_template: &acceptance_definition
> @@ -83,8 +82,7 @@ build-system-alpine:
>    artifacts:
>      expire_in: 2 days
>      paths:
> -      - .git-submodule-status
> -      - build
> +      - "*"
>  
>  check-system-alpine:
>    <<: *native_test_job_definition
> @@ -92,6 +90,7 @@ check-system-alpine:
>      - job: build-system-alpine
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: alpine
>      MAKE_CHECK_ARGS: check
>  
> @@ -101,6 +100,7 @@ acceptance-system-alpine:
>      - job: build-system-alpine
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: alpine
>      MAKE_CHECK_ARGS: check-acceptance
>    <<: *acceptance_definition
> @@ -116,7 +116,7 @@ build-system-ubuntu:
>    artifacts:
>      expire_in: 2 days
>      paths:
> -      - build
> +      - "*"
>  
>  check-system-ubuntu:
>    <<: *native_test_job_definition
> @@ -124,6 +124,7 @@ check-system-ubuntu:
>      - job: build-system-ubuntu
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: ubuntu2004
>      MAKE_CHECK_ARGS: check
>  
> @@ -133,6 +134,7 @@ acceptance-system-ubuntu:
>      - job: build-system-ubuntu
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: ubuntu2004
>      MAKE_CHECK_ARGS: check-acceptance
>    <<: *acceptance_definition
> @@ -148,7 +150,7 @@ build-system-debian:
>    artifacts:
>      expire_in: 2 days
>      paths:
> -      - build
> +      - "*"
>  
>  check-system-debian:
>    <<: *native_test_job_definition
> @@ -156,6 +158,7 @@ check-system-debian:
>      - job: build-system-debian
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: debian-amd64
>      MAKE_CHECK_ARGS: check
>  
> @@ -170,7 +173,7 @@ build-tools-and-docs-debian:
>    artifacts:
>      expire_in: 2 days
>      paths:
> -      - build
> +      - "*"
>  
>  acceptance-system-debian:
>    <<: *native_test_job_definition
> @@ -178,6 +181,7 @@ acceptance-system-debian:
>      - job: build-system-debian
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: debian-amd64
>      MAKE_CHECK_ARGS: check-acceptance
>    <<: *acceptance_definition
> @@ -194,7 +198,7 @@ build-system-fedora:
>    artifacts:
>      expire_in: 2 days
>      paths:
> -      - build
> +      - "*"
>  
>  check-system-fedora:
>    <<: *native_test_job_definition
> @@ -202,6 +206,7 @@ check-system-fedora:
>      - job: build-system-fedora
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: fedora
>      MAKE_CHECK_ARGS: check
>  
> @@ -211,6 +216,7 @@ acceptance-system-fedora:
>      - job: build-system-fedora
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: fedora
>      MAKE_CHECK_ARGS: check-acceptance
>    <<: *acceptance_definition
> @@ -226,7 +232,7 @@ build-system-centos:
>    artifacts:
>      expire_in: 2 days
>      paths:
> -      - build
> +      - "*"
>  
>  check-system-centos:
>    <<: *native_test_job_definition
> @@ -234,6 +240,7 @@ check-system-centos:
>      - job: build-system-centos
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: centos8
>      MAKE_CHECK_ARGS: check
>  
> @@ -243,6 +250,7 @@ acceptance-system-centos:
>      - job: build-system-centos
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: centos8
>      MAKE_CHECK_ARGS: check-acceptance
>    <<: *acceptance_definition
> @@ -257,7 +265,7 @@ build-system-opensuse:
>    artifacts:
>      expire_in: 2 days
>      paths:
> -      - build
> +      - "*"
>  
>  check-system-opensuse:
>    <<: *native_test_job_definition
> @@ -265,6 +273,7 @@ check-system-opensuse:
>      - job: build-system-opensuse
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: opensuse-leap
>      MAKE_CHECK_ARGS: check
>  
> @@ -274,6 +283,7 @@ acceptance-system-opensuse:
>       - job: build-system-opensuse
>         artifacts: true
>     variables:
> +     GIT_CHECKOUT: "false"
>       IMAGE: opensuse-leap
>       MAKE_CHECK_ARGS: check-acceptance
>     <<: *acceptance_definition
> @@ -444,7 +454,7 @@ build-deprecated:
>    artifacts:
>      expire_in: 2 days
>      paths:
> -      - build
> +      - "*"
>  
>  # We split the check-tcg step as test failures are expected but we still
>  # want to catch the build breaking.
> @@ -454,6 +464,7 @@ check-deprecated:
>      - job: build-deprecated
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: debian-all-test-cross
>      MAKE_CHECK_ARGS: check-tcg
>    allow_failure: true
> @@ -526,7 +537,7 @@ build-crypto-old-nettle:
>      MAKE_CHECK_ARGS: check-build
>    artifacts:
>      paths:
> -      - build
> +      - "*"
>  
>  check-crypto-old-nettle:
>    <<: *native_test_job_definition
> @@ -534,6 +545,7 @@ check-crypto-old-nettle:
>      - job: build-crypto-old-nettle
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: centos7
>      MAKE_CHECK_ARGS: check
>  
> @@ -547,7 +559,7 @@ build-crypto-old-gcrypt:
>      MAKE_CHECK_ARGS: check-build
>    artifacts:
>      paths:
> -      - build
> +      - "*"
>  
>  check-crypto-old-gcrypt:
>    <<: *native_test_job_definition
> @@ -555,6 +567,7 @@ check-crypto-old-gcrypt:
>      - job: build-crypto-old-gcrypt
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: centos7
>      MAKE_CHECK_ARGS: check
>  
> @@ -568,7 +581,7 @@ build-crypto-only-gnutls:
>      MAKE_CHECK_ARGS: check-build
>    artifacts:
>      paths:
> -      - build
> +      - "*"
>  
>  check-crypto-only-gnutls:
>    <<: *native_test_job_definition
> @@ -576,6 +589,7 @@ check-crypto-only-gnutls:
>      - job: build-crypto-only-gnutls
>        artifacts: true
>    variables:
> +    GIT_CHECKOUT: "false"
>      IMAGE: centos7
>      MAKE_CHECK_ARGS: check
>  
> -- 
> 2.27.0
> 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI
  2021-01-22 10:18 ` Daniel P. Berrangé
@ 2021-01-22 10:20   ` Daniel P. Berrangé
  2021-01-22 10:30   ` Philippe Mathieu-Daudé
  2021-01-22 16:25   ` Thomas Huth
  2 siblings, 0 replies; 5+ messages in thread
From: Daniel P. Berrangé @ 2021-01-22 10:20 UTC (permalink / raw)
  To: Thomas Huth
  Cc: Willian Rampazzo, Alex Bennée, Philippe Mathieu-Daudé,
	qemu-devel, Wainer dos Santos Moschetta

On Fri, Jan 22, 2021 at 10:18:33AM +0000, Daniel P. Berrangé wrote:
> On Fri, Jan 22, 2021 at 11:07:22AM +0100, Thomas Huth wrote:
> > Currently, our check-system-* jobs are recompiling the whole sources
> > again. This happens due to the fact that the jobs are checking out
> > the whole source tree and required submodules again, and only try
> > to use the "build" directory with the binaries and object files
> > as an artifact from the previous stage - which simply does not work
> > anymore (with the current version of meson). Due to some changed
> > time stamps, meson is always trying to rebuild the whole tree.
> 
> This used to work in the past didn't it ? Did something change in
> meson to break this, or have we just not noticed before.

For to ask, could we address it by using  'meson test --no-rebuild'
perhaps ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI
  2021-01-22 10:18 ` Daniel P. Berrangé
  2021-01-22 10:20   ` Daniel P. Berrangé
@ 2021-01-22 10:30   ` Philippe Mathieu-Daudé
  2021-01-22 16:25   ` Thomas Huth
  2 siblings, 0 replies; 5+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-01-22 10:30 UTC (permalink / raw)
  To: Daniel P. Berrangé, Thomas Huth
  Cc: Willian Rampazzo, Alex Bennée, qemu-devel,
	Wainer dos Santos Moschetta

On 1/22/21 11:18 AM, Daniel P. Berrangé wrote:
> On Fri, Jan 22, 2021 at 11:07:22AM +0100, Thomas Huth wrote:
>> Currently, our check-system-* jobs are recompiling the whole sources
>> again. This happens due to the fact that the jobs are checking out
>> the whole source tree and required submodules again, and only try
>> to use the "build" directory with the binaries and object files
>> as an artifact from the previous stage - which simply does not work
>> anymore (with the current version of meson). Due to some changed
>> time stamps, meson is always trying to rebuild the whole tree.
> 
> This used to work in the past didn't it ? Did something change in
> meson to break this, or have we just not noticed before.

Likely https://github.com/mesonbuild/meson/pull/7900/

Kludge:
https://lists.gnu.org/archive/html/qemu-devel/2021-01/msg05491.html



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI
  2021-01-22 10:18 ` Daniel P. Berrangé
  2021-01-22 10:20   ` Daniel P. Berrangé
  2021-01-22 10:30   ` Philippe Mathieu-Daudé
@ 2021-01-22 16:25   ` Thomas Huth
  2 siblings, 0 replies; 5+ messages in thread
From: Thomas Huth @ 2021-01-22 16:25 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Willian Rampazzo, Philippe Mathieu-Daudé,
	Alex Bennée, qemu-devel, Wainer dos Santos Moschetta

On 22/01/2021 11.18, Daniel P. Berrangé wrote:
> On Fri, Jan 22, 2021 at 11:07:22AM +0100, Thomas Huth wrote:
>> Currently, our check-system-* jobs are recompiling the whole sources
>> again. This happens due to the fact that the jobs are checking out
>> the whole source tree and required submodules again, and only try
>> to use the "build" directory with the binaries and object files
>> as an artifact from the previous stage - which simply does not work
>> anymore (with the current version of meson). Due to some changed
>> time stamps, meson is always trying to rebuild the whole tree.
> 
> This used to work in the past didn't it ? Did something change in
> meson to break this, or have we just not noticed before.

As mentioned by Philippe already, there seems to be a problem in current 
meson which triggers a rebuild of the whole tree in some cases. See this 
thread here:

https://lists.gnu.org/archive/html/qemu-devel/2021-01/msg05409.html

>> So instead of trying to marry a freshly checked out source tree
>> with the pre-built binaries in these jobs, let's simply pass the
>> whole source including the submodules and the build tree as artifact
>> to the test jobs. That way timestamps get preserved and there is
>> no rebuild of the sources anymore. This saves ca. 15 - 20 minutes
>> of precious CI cycles in each run.
> 
> I'm a little worried we might end up hitting the artifact size
> limit which is supposedly 1GB on gitlab.com.  Im guessing this
> must be measuring the compressed size though, as a src checkout
> with build dir  and .git dir is already way over 1GB.

I've added a "du -sh" in one of the jobs and the size of the directory was 
indeed 2 GB already. But I did not see any problems due to that size. So 
either the limit must be bigger, or it's about the compressed size. Either 
way, it's working at least right now.

We could maybe also simply exclude the .git directory of the source 
checkout, I think that's the biggest chunk there and likely not needed for 
the tests.

 > For to ask, could we address it by using  'meson test
 > --no-rebuild' perhaps ?

Interesting idea! I'll give it a try, and if it works, I'll send a patch for 
that instead.

  Thomas



^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-01-22 16:26 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-22 10:07 [PATCH] gitlab-ci.yml: Use the whole tree as artifacts to speed up the CI Thomas Huth
2021-01-22 10:18 ` Daniel P. Berrangé
2021-01-22 10:20   ` Daniel P. Berrangé
2021-01-22 10:30   ` Philippe Mathieu-Daudé
2021-01-22 16:25   ` Thomas Huth

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).