[RFC PATCH 0/5] Use ccache in the gitlab-CI

[RFC PATCH 0/5] Use ccache in the gitlab-CI

[Thomas Huth]

Here are some patches to enable ccache in the gitlab-CI. Using the
compiler cache can speed up some of the jobs in the CI by avoiding
to recompile the same sources again and again. With this patch series,
some jobs indeed get quite a bit faster. For example "build-system-fedora"
goes down from 29 minutes (https://gitlab.com/thuth/qemu/-/jobs/1168149051)
to 12 minutes (https://gitlab.com/thuth/qemu/-/jobs/1175479324) in the
best case.
However, and that's why I've marked this series still as RFC, there
are some jobs which refuse to improve much, for example the *-cfi-* jobs
are still running for a long time, see e.g.:

 https://gitlab.com/thuth/qemu/-/jobs/1168149079 (50 minutes on master)
 https://gitlab.com/thuth/qemu/-/jobs/1175479342 (43 minutes with ccache)

Thus the overall run time did not improve very much yet.

Additionally, the jobs are sometimes running even slower, e.g. if the
cache has not been populated yet or if there are a lot of cache misses,
and I've seen some jobs hitting the 60 min time limit because of this.
And I've even seen two or three jobs failing intermittendly with a
crashing test, e.g.:

 https://gitlab.com/thuth/qemu/-/jobs/1175479336

Not sure whether that's related to some bad caching of ccache, or
whether it's something that also occurs on the master branch recently?
Has anybody else experienced such a problem?

Thus this series is RFC only - in case somebody else wants to play
with it and help to improve it, you're very welcome!

Thomas Huth (5):
  tests/docker/dockerfiles: Add ccache to containers where it was
    missing
  tests/docker/dockerfiles/ubuntu2004: Add missing symlink for cc
  gitlab-ci.d/crossbuilds: Limit the amount of targets in the
    cross-win64 build
  gitlab-ci.yml: Compile with ccache enabled
  gitlab-ci.d/crossbuilds.yml: Enable compilation with ccache

 .gitlab-ci.d/crossbuilds.yml                  | 34 +++++++++++++++++++
 .gitlab-ci.yml                                | 20 +++++++++++
 tests/docker/dockerfiles/alpine.docker        |  1 +
 .../dockerfiles/fedora-i386-cross.docker      |  1 +
 .../dockerfiles/fedora-win32-cross.docker     |  1 +
 .../dockerfiles/fedora-win64-cross.docker     |  1 +
 tests/docker/dockerfiles/opensuse-leap.docker |  1 +
 tests/docker/dockerfiles/ubuntu2004.docker    |  3 ++
 8 files changed, 62 insertions(+)

-- 
2.27.0

[RFC PATCH 1/5] tests/docker/dockerfiles: Add ccache to containers where it was missing

[Thomas Huth]

We're going to enable ccache in our gitlab-CI. Make sure that it is
available in all containers.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/docker/dockerfiles/alpine.docker             | 1 +
 tests/docker/dockerfiles/fedora-i386-cross.docker  | 1 +
 tests/docker/dockerfiles/fedora-win32-cross.docker | 1 +
 tests/docker/dockerfiles/fedora-win64-cross.docker | 1 +
 tests/docker/dockerfiles/opensuse-leap.docker      | 1 +
 5 files changed, 5 insertions(+)

diff --git a/tests/docker/dockerfiles/alpine.docker b/tests/docker/dockerfiles/alpine.docker
index d63a269aef..a1ef408a6a 100644
--- a/tests/docker/dockerfiles/alpine.docker
+++ b/tests/docker/dockerfiles/alpine.docker
@@ -9,6 +9,7 @@ ENV PACKAGES \
 	alsa-lib-dev \
 	bash \
 	binutils \
+	ccache \
 	coreutils \
 	curl-dev \
 	g++ \
diff --git a/tests/docker/dockerfiles/fedora-i386-cross.docker b/tests/docker/dockerfiles/fedora-i386-cross.docker
index 966072c08e..66cdb06c19 100644
--- a/tests/docker/dockerfiles/fedora-i386-cross.docker
+++ b/tests/docker/dockerfiles/fedora-i386-cross.docker
@@ -1,6 +1,7 @@
 FROM fedora:33
 ENV PACKAGES \
     bzip2 \
+    ccache \
     diffutils \
     findutils \
     gcc \
diff --git a/tests/docker/dockerfiles/fedora-win32-cross.docker b/tests/docker/dockerfiles/fedora-win32-cross.docker
index 81b5659e9c..3733df63e9 100644
--- a/tests/docker/dockerfiles/fedora-win32-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win32-cross.docker
@@ -4,6 +4,7 @@ FROM fedora:33
 ENV PACKAGES \
     bc \
     bzip2 \
+    ccache \
     diffutils \
     findutils \
     gcc \
diff --git a/tests/docker/dockerfiles/fedora-win64-cross.docker b/tests/docker/dockerfiles/fedora-win64-cross.docker
index bcb428e724..2564ce4979 100644
--- a/tests/docker/dockerfiles/fedora-win64-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win64-cross.docker
@@ -4,6 +4,7 @@ FROM fedora:33
 ENV PACKAGES \
     bc \
     bzip2 \
+    ccache \
     diffutils \
     findutils \
     gcc \
diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index 0e64893e4a..f7e1cbfbe6 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -5,6 +5,7 @@ ENV PACKAGES \
     bc \
     brlapi-devel \
     bzip2 \
+    ccache \
     cyrus-sasl-devel \
     gcc \
     gcc-c++ \
-- 
2.27.0

[RFC PATCH 2/5] tests/docker/dockerfiles/ubuntu2004: Add missing symlink for cc

[Thomas Huth]

For some reasons, the "cc" symlink is missing in Ubuntu 20.04.
Add it manually.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/docker/dockerfiles/ubuntu2004.docker | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index 9750016e51..e1b309e313 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -74,3 +74,6 @@ ENV FEATURES clang tsan pyyaml sdl2
 # Apply patch https://reviews.llvm.org/D75820
 # This is required for TSan in clang-10 to compile with QEMU.
 RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h
+
+# The symlink for cc is missing on Ubuntu
+RUN cd /usr/lib/ccache && ln -s ../../bin/ccache cc
-- 
2.27.0

[RFC PATCH 3/5] gitlab-ci.d/crossbuilds: Limit the amount of targets in the cross-win64 build

[Thomas Huth]

The cross-win64-system job is one of the slowest in our gitlab-ci.
With the upcoming changes that introduce ccache, it might even get
slower initially as long as the cache is not populated yet. Thus
let's limit the amount of targets that we build here even further
to avoid that we might hit the 80 minutes timeout here. Most of the
others targets are already covered by the win32 job anyway.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.d/crossbuilds.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 2d95784ed5..017731a894 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -189,6 +189,12 @@ cross-win64-system:
     job: win64-fedora-cross-container
   variables:
     IMAGE: fedora-win64-cross
+  script:
+    - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
+      ./configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
+        --target-list="cris-softmmu x86_64-softmmu microblaze-softmmu
+          mipsel-softmmu mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
+    - make -j$(expr $(nproc) + 1) all check-build
 
 cross-amd64-xen-only:
   extends: .cross_accel_build_job
-- 
2.27.0

[RFC PATCH 4/5] gitlab-ci.yml: Compile with ccache enabled

[Thomas Huth]

Ccache can speed up the compilation in some cases, so enable it for
our CI builds, too.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 52d65d6c04..422608381e 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -18,7 +18,26 @@ include:
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   before_script:
     - JOBS=$(expr $(nproc) + 1)
+  cache:
+    key: "$CI_JOB_NAME"
+    paths:
+      - ccache
   script:
+    - mkdir -p ccache
+    - if command -v ccache ; then
+        export CCACHE_BASEDIR=${PWD} ;
+        export CCACHE_DIR=${PWD}/ccache ;
+        export CCACHE_MAXSIZE="1G" ;
+        export CCACHE_COMPILERCHECK=content ;
+        if [ -e /usr/lib/ccache/bin ]; then
+          export PATH=/usr/lib/ccache/bin:$PATH ;
+        elif [ -e /usr/lib/ccache ]; then
+          export PATH=/usr/lib/ccache:$PATH ;
+        elif [ -e /usr/lib64/ccache ]; then
+          export PATH=/usr/lib64/ccache:$PATH ;
+        fi ;
+        ccache --zero-stats ;
+      fi
     - mkdir build
     - cd build
     - if test -n "$TARGETS";
@@ -36,6 +55,7 @@ include:
       then
         make -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
+    - if command -v ccache ; then ccache --show-stats ; fi
 
 .native_test_job_template: &native_test_job_definition
   stage: test
-- 
2.27.0

[RFC PATCH 5/5] gitlab-ci.d/crossbuilds.yml: Enable compilation with ccache

[Thomas Huth]

Ccache can speed up the compilation in some cases, so enable it for
our cross-compilation CI builds, too.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.d/crossbuilds.yml | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 017731a894..dee327a936 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -1,4 +1,26 @@
+
+.ccache_cross_setup:
+  cache:
+    key: "$CI_JOB_NAME"
+    paths:
+      - ccache
+  before_script:
+    - mkdir -p ccache
+    - if command -v ccache ; then
+        export CCACHE_BASEDIR=${PWD} ;
+        export CCACHE_DIR=${PWD}/ccache ;
+        export CCACHE_MAXSIZE="1G" ;
+        if [ -e /usr/lib/ccache ]; then
+          export PATH=/usr/lib/ccache:$PATH ;
+        fi ;
+        if [ -e /usr/lib64/ccache ]; then
+          export PATH=/usr/lib64/ccache:$PATH ;
+        fi ;
+        ccache --zero-stats ;
+      fi
+
 .cross_system_build_job:
+  extends: .ccache_cross_setup
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 80m
@@ -11,6 +33,7 @@
           i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
           mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
     - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
+    - if command -v ccache ; then ccache --show-stats ; fi
 
 # Job to cross-build specific accelerators.
 #
@@ -18,6 +41,7 @@
 # KVM), and set extra options (such disabling other accelerators) via the
 # $ACCEL_CONFIGURE_OPTS variable.
 .cross_accel_build_job:
+  extends: .ccache_cross_setup
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   timeout: 30m
@@ -28,8 +52,10 @@
       ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
         --disable-tools --enable-${ACCEL:-kvm} $ACCEL_CONFIGURE_OPTS
     - make -j$(expr $(nproc) + 1) all check-build
+    - if command -v ccache ; then ccache --show-stats ; fi
 
 .cross_user_build_job:
+  extends: .ccache_cross_setup
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
@@ -39,6 +65,7 @@
       ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
         --disable-system
     - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
+    - if command -v ccache ; then ccache --show-stats ; fi
 
 cross-armel-system:
   extends: .cross_system_build_job
@@ -195,6 +222,7 @@ cross-win64-system:
         --target-list="cris-softmmu x86_64-softmmu microblaze-softmmu
           mipsel-softmmu mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
     - make -j$(expr $(nproc) + 1) all check-build
+    - if command -v ccache ; then ccache --show-stats ; fi
 
 cross-amd64-xen-only:
   extends: .cross_accel_build_job
-- 
2.27.0

Re: [RFC PATCH 2/5] tests/docker/dockerfiles/ubuntu2004: Add missing symlink for cc

[Philippe Mathieu-Daudé]

On 4/14/21 10:19 AM, Thomas Huth wrote:
> For some reasons, the "cc" symlink is missing in Ubuntu 20.04.
> Add it manually.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/docker/dockerfiles/ubuntu2004.docker | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
> index 9750016e51..e1b309e313 100644
> --- a/tests/docker/dockerfiles/ubuntu2004.docker
> +++ b/tests/docker/dockerfiles/ubuntu2004.docker
> @@ -74,3 +74,6 @@ ENV FEATURES clang tsan pyyaml sdl2
>  # Apply patch https://reviews.llvm.org/D75820
>  # This is required for TSan in clang-10 to compile with QEMU.
>  RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h
> +
> +# The symlink for cc is missing on Ubuntu

Maybe precise the version: "on Ubuntu 20.04" so we can remove
when upgrading to the next release?

> +RUN cd /usr/lib/ccache && ln -s ../../bin/ccache cc

No need to change current directory:

   RUN ln -s ../../bin/ccache /usr/lib/ccache/cc

Or use absolute (but if other use relative, better keep relative,
I haven't check):

   RUN ln -s /usr/bin/ccache /usr/lib/ccache/cc

Re: [RFC PATCH 5/5] gitlab-ci.d/crossbuilds.yml: Enable compilation with ccache

[Philippe Mathieu-Daudé]

On 4/14/21 10:19 AM, Thomas Huth wrote:
> Ccache can speed up the compilation in some cases, so enable it for
> our cross-compilation CI builds, too.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.d/crossbuilds.yml | 28 ++++++++++++++++++++++++++++
>  1 file changed, 28 insertions(+)
> 
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> index 017731a894..dee327a936 100644
> --- a/.gitlab-ci.d/crossbuilds.yml
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -1,4 +1,26 @@
> +
> +.ccache_cross_setup:
> +  cache:
> +    key: "$CI_JOB_NAME"
> +    paths:
> +      - ccache

I discovered last month the cache/artifact passing feature is useful
when all runners are in the same premises, but when you used runner
in different locations it become slower (except if all your runners
are in very fast clouds probably).

Not a short term problem, but can hit us long term. I haven't looked
if GitLab already figured how to manage this globally, like registering
runners with affinity, or assigning them to pools.

If we decide to stick with the free-tier runners, then this is a great
improvement.

For custom runners, a simple kludge is to assert they do *not* use
cache/artifacts (from free-tier runners).

We could restrict this to free-tier runners but these runners tag aren't
standardized, and AFAIK we can not (yet?) use the "ANY" word to "select
runners with any of this tag list". So far I noticed gitlab uses these:
- shared docker
- gce docker
- gitlab-org
- gitlab-org-docker

The last 2 are recent, and seem to help with the problem I reported
that some runners don't have Docker installed:
https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02213.html

Regards,

Phil.

Re: [RFC PATCH 0/5] Use ccache in the gitlab-CI

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2221 bytes --]

On Wed, Apr 14, 2021 at 10:19:02AM +0200, Thomas Huth wrote:
> Here are some patches to enable ccache in the gitlab-CI. Using the
> compiler cache can speed up some of the jobs in the CI by avoiding
> to recompile the same sources again and again. With this patch series,
> some jobs indeed get quite a bit faster. For example "build-system-fedora"
> goes down from 29 minutes (https://gitlab.com/thuth/qemu/-/jobs/1168149051)
> to 12 minutes (https://gitlab.com/thuth/qemu/-/jobs/1175479324) in the
> best case.
> However, and that's why I've marked this series still as RFC, there
> are some jobs which refuse to improve much, for example the *-cfi-* jobs
> are still running for a long time, see e.g.:
> 
>  https://gitlab.com/thuth/qemu/-/jobs/1168149079 (50 minutes on master)
>  https://gitlab.com/thuth/qemu/-/jobs/1175479342 (43 minutes with ccache)
> 
> Thus the overall run time did not improve very much yet.
> 
> Additionally, the jobs are sometimes running even slower, e.g. if the
> cache has not been populated yet or if there are a lot of cache misses,
> and I've seen some jobs hitting the 60 min time limit because of this.
> And I've even seen two or three jobs failing intermittendly with a
> crashing test, e.g.:
> 
>  https://gitlab.com/thuth/qemu/-/jobs/1175479336
> 
> Not sure whether that's related to some bad caching of ccache, or
> whether it's something that also occurs on the master branch recently?
> Has anybody else experienced such a problem?

The GitLab CI cache implementation does not appear to be designed for
ccache because the files are saved in a cache.zip file and then
extracted again each time a job runs.

That is very inefficient for something like a 1 GiB ccache directory.

More details:
https://docs.gitlab.com/ee/ci/caching/#availability-of-the-cache

The GitLab CI cache feature in its current form just doesn't look
suitable for ccache.

I wonder if we can implement ccache differently on dedicated runners.
The ccache directory would really be shared (mounted within job
containers) without the additional cache.zip step to avoid wasting CPU
cycles compressing/decompressing and RAM with duplicated page cache.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH 5/5] gitlab-ci.d/crossbuilds.yml: Enable compilation with ccache

[Philippe Mathieu-Daudé]

On 4/14/21 12:29 PM, Philippe Mathieu-Daudé wrote:
> On 4/14/21 10:19 AM, Thomas Huth wrote:
>> Ccache can speed up the compilation in some cases, so enable it for
>> our cross-compilation CI builds, too.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  .gitlab-ci.d/crossbuilds.yml | 28 ++++++++++++++++++++++++++++
>>  1 file changed, 28 insertions(+)
>>
>> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
>> index 017731a894..dee327a936 100644
>> --- a/.gitlab-ci.d/crossbuilds.yml
>> +++ b/.gitlab-ci.d/crossbuilds.yml
>> @@ -1,4 +1,26 @@
>> +
>> +.ccache_cross_setup:
>> +  cache:
>> +    key: "$CI_JOB_NAME"
>> +    paths:
>> +      - ccache
> 
> I discovered last month the cache/artifact passing feature is useful
> when all runners are in the same premises, but when you used runner
> in different locations it become slower (except if all your runners
> are in very fast clouds probably).
> 
> Not a short term problem, but can hit us long term. I haven't looked
> if GitLab already figured how to manage this globally, like registering
> runners with affinity, or assigning them to pools.
> 
> If we decide to stick with the free-tier runners, then this is a great
> improvement.
> 
> For custom runners, a simple kludge is to assert they do *not* use
> cache/artifacts (from free-tier runners).
> 
> We could restrict this to free-tier runners but these runners tag aren't
> standardized, and AFAIK we can not (yet?) use the "ANY" word to "select
> runners with any of this tag list". So far I noticed gitlab uses these:
> - shared docker
> - gce docker
> - gitlab-org
> - gitlab-org-docker
> 
> The last 2 are recent, and seem to help with the problem I reported
> that some runners don't have Docker installed:
> https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02213.html

Actually found the doc for the last 2:
https://docs.gitlab.com/ee/user/gitlab_com/#linux-shared-runners

- gitlab-org-docker

All your CI/CD jobs run on n1-standard-1 instances with 3.75GB of RAM,
CoreOS and the latest Docker Engine installed. Instances provide 1 vCPU
and 25GB of HDD disk space. The default region of the VMs is US East1.
Each instance is used only for one job, this ensures any sensitive data
left on the system can’t be accessed by other people their CI jobs.

- gitlab-org

The gitlab-shared-runners-manager-X.gitlab.com fleet of runners are
dedicated for GitLab projects as well as community forks of them. They
use a slightly larger machine type (n1-standard-2) and have a bigger SSD
disk size. They don’t run untagged jobs and unlike the general fleet of
shared runners, the instances are re-used up to 40 times.