Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Willian Rampazzo <wrampazz@redhat.com>]

On Fri, May 21, 2021 at 8:03 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> > On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> >> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> >>> +Stefan/Daniel
> >>>
> >>> On 5/20/21 10:02 AM, Thomas Huth wrote:
> >>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> >>>>> If a runner has ccache installed, use it and display statistics
> >>>>> at the end of the build.
> >>>>>
> >>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >>>>> ---
> >>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
> >>>>>     1 file changed, 5 insertions(+)
> >>>>>
> >>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
> >>>>> b/.gitlab-ci.d/buildtest-template.yml
> >>>>> index f284d7a0eec..a625c697d3b 100644
> >>>>> --- a/.gitlab-ci.d/buildtest-template.yml
> >>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
> >>>>> @@ -6,13 +6,18 @@
> >>>>>           then
> >>>>>             JOBS=$(sysctl -n hw.ncpu)
> >>>>>             MAKE=gmake
> >>>>> +        PATH=/usr/local/libexec/ccache:$PATH
> >>>>>             ;
> >>>>>           else
> >>>>>             JOBS=$(expr $(nproc) + 1)
> >>>>>             MAKE=make
> >>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> >>>>
> >>>> That does not make sense for the shared runners yet. We first need
> >>>> something to enable the caching there - see my series "Use ccache in the
> >>>> gitlab-CI" from April (which is currently stalled unfortunately).
> >>>
> >>> TL;DR: I don't think we should restrict our templates to shared runners.
> >>
> >> I'm certainly not voting for restricting ourselves to only use shared
> >> runners here - but my concern is that this actually *slows* down the shared
> >> runners even more! (sorry, I should have elaborated on that in my previous
> >> mail already)
> >>
> >> When I was experimenting with ccache in the shared runners, I saw that the
> >> jobs are running even slower with ccache enabled as long as the cache is not
> >> populated yet. You only get a speedup afterwards. So if you add this now
> >> without also adding the possibility to store the cache persistently, the
> >> shared runners will try to populate the cache each time just to throw away
> >> the results afterwards again. Thus all the shared runners only get slower
> >> without any real benefit here.
> >>

I was in favor of adding the changes and benefiting custom runners
until I saw your reply. In this case, I agree we should investigate
how these changes affect shared runners.

> >> Thus we either need to get ccache working properly for the shared runners
> >> first, or you have to think of a different way of enabling ccache for the
> >> non-shared runners, so that it does not affect the shared runners
> >> negatively.
> >
> > Is there anything functional holding up your previous full cccache support
> > series from last month ? Or is it just lack of reviews ?
>
> It's basically the problems mentioned in the cover letter and Stefan's
> comment here:
>
>   https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>
> The series needs more love and more testing, if it's feasible with the
> gitlab-CI architecture at all to use ccache in a good way ... if we don't
> get a real speedup by the patches, it's certainly not worth the effort to
> integrate this...
>
> If someone wants to have a try to improve the patch series, your help is
> certainly welcome - since at least I personally lack the time and motivation
> to improve this any further.

Do you mind if I play with your patch series? I do not promise 100% of
my time working on it, but I was thinking about dedicating some time
to ccache before your patch series.

>
>   Thomas
>