On Wed, Apr 14, 2021 at 10:19:02AM +0200, Thomas Huth wrote:
> Here are some patches to enable ccache in the gitlab-CI. Using the
> compiler cache can speed up some of the jobs in the CI by avoiding
> to recompile the same sources again and again. With this patch series,
> some jobs indeed get quite a bit faster. For example "build-system-fedora"
> goes down from 29 minutes (https://gitlab.com/thuth/qemu/-/jobs/1168149051)
> to 12 minutes (https://gitlab.com/thuth/qemu/-/jobs/1175479324) in the
> best case.
> However, and that's why I've marked this series still as RFC, there
> are some jobs which refuse to improve much, for example the *-cfi-* jobs
> are still running for a long time, see e.g.:
> 
>  https://gitlab.com/thuth/qemu/-/jobs/1168149079 (50 minutes on master)
>  https://gitlab.com/thuth/qemu/-/jobs/1175479342 (43 minutes with ccache)
> 
> Thus the overall run time did not improve very much yet.
> 
> Additionally, the jobs are sometimes running even slower, e.g. if the
> cache has not been populated yet or if there are a lot of cache misses,
> and I've seen some jobs hitting the 60 min time limit because of this.
> And I've even seen two or three jobs failing intermittendly with a
> crashing test, e.g.:
> 
>  https://gitlab.com/thuth/qemu/-/jobs/1175479336
> 
> Not sure whether that's related to some bad caching of ccache, or
> whether it's something that also occurs on the master branch recently?
> Has anybody else experienced such a problem?

The GitLab CI cache implementation does not appear to be designed for
ccache because the files are saved in a cache.zip file and then
extracted again each time a job runs.

That is very inefficient for something like a 1 GiB ccache directory.

More details:
https://docs.gitlab.com/ee/ci/caching/#availability-of-the-cache

The GitLab CI cache feature in its current form just doesn't look
suitable for ccache.

I wonder if we can implement ccache differently on dedicated runners.
The ccache directory would really be shared (mounted within job
containers) without the additional cache.zip step to avoid wasting CPU
cycles compressing/decompressing and RAM with duplicated page cache.

Stefan