[PATCH v3 0/6] gitlab-ci: Allow using FreeBSD runners

[PATCH v3 0/6] gitlab-ci: Allow using FreeBSD runners

[Philippe Mathieu-Daudé]

Minor shell fixes to use FreeBSD runners.

Since v2:
- fixed typo (Warner)
- added Warner R-b tag

Since v1:
- share the FreeBSD jobs, restricted with QEMU_CUSTOM_RUNNER=freebsd
  so mergeable in mainstream.

Jobs: https://gitlab.com/philmd/qemu/-/pipelines/301361673

Based-on: <20210511072952.2813358-1-f4bug@amsat.org>
gitlab-ci: Ease forks pipeline workflow
which itself is based on:
https://gitlab.com/thuth/qemu.git tags/pull-request-2021-05-03

Philippe Mathieu-Daudé (6):
  gitlab-ci: Extract &environment_variables template
  gitlab-ci: Adapt JOBS variable for FreeBSD runners
  gitlab-ci: Run GNU make via the $MAKE variable
  gitlab-ci: Add ccache in $PATH and display statistics
  gitlab-ci: Simplify before/after script for Avocado based jobs
  gitlab-ci: Add FreeBSD jobs

 .gitlab-ci.d/buildtest-freebsd.yml  | 59 +++++++++++++++++++++++++++++
 .gitlab-ci.d/buildtest-template.yml | 38 +++++++++++++++----
 .gitlab-ci.d/qemu-project.yml       |  1 +
 3 files changed, 91 insertions(+), 7 deletions(-)
 create mode 100644 .gitlab-ci.d/buildtest-freebsd.yml

-- 
2.26.3

[PATCH v3 1/6] gitlab-ci: Extract &environment_variables template

[Philippe Mathieu-Daudé]

To be able to set the same environment variables to multiple jobs,
extract what we currently have as a template.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 32aaef1a213..58b01744751 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,8 +1,11 @@
+.environment_variables_template:
+  before_script:
+    - JOBS=$(expr $(nproc) + 1)
+
 .native_build_job_template:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
-  before_script:
-    - JOBS=$(expr $(nproc) + 1)
+  extends: .environment_variables_template
   script:
     - if test -n "$LD_JOBS";
       then
-- 
2.26.3

[PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners

[Philippe Mathieu-Daudé]

'nproc' is not available on FreeBSD:

  $ JOBS=$(expr $(nproc) + 1)
  bash: line 119: nproc: command not found
  expr: syntax error

Instead, use 'sysctl -n hw.ncpu'.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 58b01744751..fe4f18595ac 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,6 +1,16 @@
 .environment_variables_template:
   before_script:
-    - JOBS=$(expr $(nproc) + 1)
+    - if
+        test $(uname) = "FreeBSD"
+        ;
+      then
+        JOBS=$(sysctl -n hw.ncpu)
+        ;
+      else
+        JOBS=$(expr $(nproc) + 1)
+        ;
+      fi
+    - echo "=== Using $JOBS simultaneous jobs ==="
 
 .native_build_job_template:
   stage: build
-- 
2.26.3

[PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable

[Philippe Mathieu-Daudé]

Add the $MAKE variable to call GNU make, and set it to 'gmake'
on FreeBSD to avoid:

  $ make -j"$JOBS"
  make: Unknown modifier ','
  make: "/builds/dTyar424/0/qemu/build/Makefile" line 3: Need an operator
  make: "/builds/dTyar424/0/qemu/build/Makefile" line 4: Missing dependency operator

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index fe4f18595ac..f284d7a0eec 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -5,9 +5,11 @@
         ;
       then
         JOBS=$(sysctl -n hw.ncpu)
+        MAKE=gmake
         ;
       else
         JOBS=$(expr $(nproc) + 1)
+        MAKE=make
         ;
       fi
     - echo "=== Using $JOBS simultaneous jobs ==="
@@ -33,22 +35,23 @@
       then
         ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
       fi || exit 1;
-    - make -j"$JOBS"
+    - $MAKE -j"$JOBS"
     - if test -n "$MAKE_CHECK_ARGS";
       then
-        make -j"$JOBS" $MAKE_CHECK_ARGS ;
+        $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
 
 .native_test_job_template:
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  extends: .environment_variables_template
   script:
     - scripts/git-submodule.sh update
         $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
     - cd build
     - find . -type f -exec touch {} +
     # Avoid recompiling by hiding ninja with NINJA=":"
-    - make NINJA=":" $MAKE_CHECK_ARGS
+    - $MAKE NINJA=":" $MAKE_CHECK_ARGS
 
 .integration_test_job_template:
   extends: .native_test_job_template
-- 
2.26.3

[PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Philippe Mathieu-Daudé]

If a runner has ccache installed, use it and display statistics
at the end of the build.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index f284d7a0eec..a625c697d3b 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -6,13 +6,18 @@
       then
         JOBS=$(sysctl -n hw.ncpu)
         MAKE=gmake
+        PATH=/usr/local/libexec/ccache:$PATH
         ;
       else
         JOBS=$(expr $(nproc) + 1)
         MAKE=make
+        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
         ;
       fi
     - echo "=== Using $JOBS simultaneous jobs ==="
+    - if command -v ccache > /dev/null ; then ccache --zero-stats ; fi
+  after_script:
+    - if command -v ccache > /dev/null ; then ccache --show-stats ; fi
 
 .native_build_job_template:
   stage: build
-- 
2.26.3

[PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index a625c697d3b..f968fa1ad99 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -74,7 +74,7 @@
       - build/tests/results/latest/test-results
     reports:
       junit: build/tests/results/latest/results.xml
-  before_script:
+  script:
     - mkdir -p ~/.config/avocado
     - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
     - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
@@ -85,6 +85,9 @@
         du -chs ${CI_PROJECT_DIR}/avocado-cache ;
       fi
     - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
-  after_script:
     - cd build
+    - find . -type f -exec touch {} +
+    # Avoid recompiling by hiding ninja with NINJA=":"
+    - $MAKE NINJA=":" $MAKE_CHECK_ARGS
+  after_script:
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
-- 
2.26.3

[PATCH v3 6/6] gitlab-ci: Add FreeBSD jobs

[Philippe Mathieu-Daudé]

Add system/user emulation jobs on FreeBSD host.

To build these jobs, you need to add a FreeBSD runner and
add 'freebsd' to the QEMU_CUSTOM_RUNNER variable in your
GitLab project.

Reviewed by: Warner Losh <imp@bsdimp.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-freebsd.yml | 59 ++++++++++++++++++++++++++++++
 .gitlab-ci.d/qemu-project.yml      |  1 +
 2 files changed, 60 insertions(+)
 create mode 100644 .gitlab-ci.d/buildtest-freebsd.yml

diff --git a/.gitlab-ci.d/buildtest-freebsd.yml b/.gitlab-ci.d/buildtest-freebsd.yml
new file mode 100644
index 00000000000..bfaeff89330
--- /dev/null
+++ b/.gitlab-ci.d/buildtest-freebsd.yml
@@ -0,0 +1,59 @@
+include:
+  - local: '/.gitlab-ci.d/buildtest-template.yml'
+
+.runner_freebsd_template:
+  rules:
+    # To enable this job template, add 'freebsd' to the QEMU_CUSTOM_RUNNER
+    # variable in your project UI via Settings -> CI/CD -> Variables
+    # https://docs.gitlab.com/ee/ci/variables/README.html#project-cicd-variables
+    - if: $QEMU_CUSTOM_RUNNER =~ /freebsd/
+      when: always
+    - when: never
+  tags:
+    - freebsd
+
+build-user-freebsd:
+  extends:
+    - .runner_freebsd_template
+    - .native_build_job_template
+  image:
+  variables:
+    MAKE_CHECK_ARGS: check-build
+    CONFIGURE_ARGS: --disable-system --python=/usr/local/bin/python3.7
+
+build-system-freebsd:
+  extends:
+    - .runner_freebsd_template
+    - .native_build_job_template
+  image:
+  variables:
+    TARGETS: aarch64-softmmu avr-softmmu hppa-softmmu ppc64-softmmu
+      riscv64-softmmu s390x-softmmu x86_64-softmmu
+    MAKE_CHECK_ARGS: check-build
+    CONFIGURE_ARGS: --enable-trace-backends=log,simple,syslog
+      --python=/usr/local/bin/python3.7
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - .git-submodule-status
+      - build
+
+check-system-freebsd:
+  extends:
+    - .runner_freebsd_template
+    - .native_test_job_template
+  needs:
+    - job: build-system-freebsd
+      artifacts: true
+  variables:
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-freebsd:
+  extends:
+    - .runner_freebsd_template
+    - .integration_test_job_template
+  needs:
+    - job: build-system-freebsd
+      artifacts: true
+  variables:
+    MAKE_CHECK_ARGS: check-acceptance
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 64cb2ba1da5..5dcf1d34c5b 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -8,4 +8,5 @@ include:
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
   - local: '/.gitlab-ci.d/buildtest.yml'
+  - local: '/.gitlab-ci.d/buildtest-freebsd.yml'
   - local: '/.gitlab-ci.d/static_checks.yml'
-- 
2.26.3

Re: [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable

[Willian Rampazzo]

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> Add the $MAKE variable to call GNU make, and set it to 'gmake'
> on FreeBSD to avoid:
>
>   $ make -j"$JOBS"
>   make: Unknown modifier ','
>   make: "/builds/dTyar424/0/qemu/build/Makefile" line 3: Need an operator
>   make: "/builds/dTyar424/0/qemu/build/Makefile" line 4: Missing dependency operator
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners

[Willian Rampazzo]

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> 'nproc' is not available on FreeBSD:
>
>   $ JOBS=$(expr $(nproc) + 1)
>   bash: line 119: nproc: command not found
>   expr: syntax error
>
> Instead, use 'sysctl -n hw.ncpu'.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template

[Willian Rampazzo]

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> To be able to set the same environment variables to multiple jobs,
> extract what we currently have as a template.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Willian Rampazzo]

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> If a runner has ccache installed, use it and display statistics
> at the end of the build.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 5 +++++
>  1 file changed, 5 insertions(+)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs

[Willian Rampazzo]

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v3 6/6] gitlab-ci: Add FreeBSD jobs

[Willian Rampazzo]

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> Add system/user emulation jobs on FreeBSD host.
>
> To build these jobs, you need to add a FreeBSD runner and
> add 'freebsd' to the QEMU_CUSTOM_RUNNER variable in your
> GitLab project.
>
> Reviewed by: Warner Losh <imp@bsdimp.com>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-freebsd.yml | 59 ++++++++++++++++++++++++++++++
>  .gitlab-ci.d/qemu-project.yml      |  1 +
>  2 files changed, 60 insertions(+)
>  create mode 100644 .gitlab-ci.d/buildtest-freebsd.yml
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners

[Thomas Huth]

On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> 'nproc' is not available on FreeBSD:
> 
>    $ JOBS=$(expr $(nproc) + 1)
>    bash: line 119: nproc: command not found
>    expr: syntax error
> 
> Instead, use 'sysctl -n hw.ncpu'.

Could we maybe use "getconf _NPROCESSORS_ONLN" instead? ... that might work 
on both, Linux and FreeBSD...?

  Thomas

Re: [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable

[Thomas Huth]

On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> Add the $MAKE variable to call GNU make, and set it to 'gmake'
> on FreeBSD to avoid:
> 
>    $ make -j"$JOBS"
>    make: Unknown modifier ','
>    make: "/builds/dTyar424/0/qemu/build/Makefile" line 3: Need an operator
>    make: "/builds/dTyar424/0/qemu/build/Makefile" line 4: Missing dependency operator
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 9 ++++++---
>   1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index fe4f18595ac..f284d7a0eec 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -5,9 +5,11 @@
>           ;
>         then
>           JOBS=$(sysctl -n hw.ncpu)
> +        MAKE=gmake
>           ;
>         else
>           JOBS=$(expr $(nproc) + 1)
> +        MAKE=make

Maybe we could use "gmake" on Linux, too, so we do not have to use the 
indirection with a variable here? Or are there Linux distros where the 
"gmake" link is not available?

  Thomas

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Thomas Huth]

On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> If a runner has ccache installed, use it and display statistics
> at the end of the build.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 5 +++++
>   1 file changed, 5 insertions(+)
> 
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index f284d7a0eec..a625c697d3b 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -6,13 +6,18 @@
>         then
>           JOBS=$(sysctl -n hw.ncpu)
>           MAKE=gmake
> +        PATH=/usr/local/libexec/ccache:$PATH
>           ;
>         else
>           JOBS=$(expr $(nproc) + 1)
>           MAKE=make
> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH

That does not make sense for the shared runners yet. We first need something 
to enable the caching there - see my series "Use ccache in the gitlab-CI" 
from April (which is currently stalled unfortunately).

  Thomas

Re: [PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs

[Thomas Huth]

Missing patch description. Could you please elaborate why this change is ok?

  Thanks,
   Thomas


On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index a625c697d3b..f968fa1ad99 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -74,7 +74,7 @@
>         - build/tests/results/latest/test-results
>       reports:
>         junit: build/tests/results/latest/results.xml
> -  before_script:
> +  script:
>       - mkdir -p ~/.config/avocado
>       - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
>       - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
> @@ -85,6 +85,9 @@
>           du -chs ${CI_PROJECT_DIR}/avocado-cache ;
>         fi
>       - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
> -  after_script:
>       - cd build
> +    - find . -type f -exec touch {} +
> +    # Avoid recompiling by hiding ninja with NINJA=":"
> +    - $MAKE NINJA=":" $MAKE_CHECK_ARGS
> +  after_script:
>       - du -chs ${CI_PROJECT_DIR}/avocado-cache
> 

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Philippe Mathieu-Daudé]

+Stefan/Daniel

On 5/20/21 10:02 AM, Thomas Huth wrote:
> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>> If a runner has ccache installed, use it and display statistics
>> at the end of the build.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>   .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>   1 file changed, 5 insertions(+)
>>
>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>> b/.gitlab-ci.d/buildtest-template.yml
>> index f284d7a0eec..a625c697d3b 100644
>> --- a/.gitlab-ci.d/buildtest-template.yml
>> +++ b/.gitlab-ci.d/buildtest-template.yml
>> @@ -6,13 +6,18 @@
>>         then
>>           JOBS=$(sysctl -n hw.ncpu)
>>           MAKE=gmake
>> +        PATH=/usr/local/libexec/ccache:$PATH
>>           ;
>>         else
>>           JOBS=$(expr $(nproc) + 1)
>>           MAKE=make
>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> 
> That does not make sense for the shared runners yet. We first need
> something to enable the caching there - see my series "Use ccache in the
> gitlab-CI" from April (which is currently stalled unfortunately).

TL;DR: I don't think we should restrict our templates to shared runners.

I don't think mainstream interest for shared runners use should limit
forks... This is a great improvement for contributors having to use
private/custom runner, see this example:

https://gitlab.com/philmd/qemu/-/jobs/1255197705
cache hit (direct)                  1529
cache hit (preprocessed)             736
cache miss                           855
cache hit rate                     72.60 %

Having been caught 2 times for testing as hard as crypto miner, I
can not rely on shared runners for my workflow.

Beside, the recent announcement:
https://about.gitlab.com/blog/2021/05/17/prevent-crypto-mining-abuse/
"starting May 17, 2021, GitLab will require new free users to provide a
valid credit or debit card number in order to use shared runners on
GitLab.com."

On this report only 17/142 countries are listed as having more than
50% of their population with a credit card:
https://www.theglobaleconomy.com/rankings/people_with_credit_cards/
and 60 with a debit card:
https://www.theglobaleconomy.com/rankings/people_with_debit_cards/

We have some contributors in countries with low percentage. I don't
think we should start a poll to verify, but I do think relying on
shared runners without proposing free alternative is discriminatory
to a part of our community.

I am very open to alternatives to improve the CI/testing situation.

Regards,

Phil.

Re: [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template

[Wainer dos Santos Moschetta]

Hi,

On 5/19/21 3:45 PM, Philippe Mathieu-Daudé wrote:
> To be able to set the same environment variables to multiple jobs,
> extract what we currently have as a template.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index 32aaef1a213..58b01744751 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -1,8 +1,11 @@
> +.environment_variables_template:
> +  before_script:
> +    - JOBS=$(expr $(nproc) + 1)
> +

I wish we could wrap environment variables definitions in `variables` 
(https://docs.gitlab.com/ee/ci/yaml/README.html#variables) instead of 
`before_scripts`, but it wouldn't evaluate the shell expressions.

Unless I am wrong, if the job definition has its own `before_script` 
then including `.environment_variables_template` won't take effect. 
Shouldn't it be documented?

- Wainer

>   .native_build_job_template:
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> -  before_script:
> -    - JOBS=$(expr $(nproc) + 1)
> +  extends: .environment_variables_template
>     script:
>       - if test -n "$LD_JOBS";
>         then

Re: [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners

[Wainer dos Santos Moschetta]

Hi,

On 5/19/21 3:45 PM, Philippe Mathieu-Daudé wrote:
> 'nproc' is not available on FreeBSD:
>
>    $ JOBS=$(expr $(nproc) + 1)
>    bash: line 119: nproc: command not found
>    expr: syntax error
>
> Instead, use 'sysctl -n hw.ncpu'.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 12 +++++++++++-
>   1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index 58b01744751..fe4f18595ac 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -1,6 +1,16 @@
>   .environment_variables_template:
>     before_script:
> -    - JOBS=$(expr $(nproc) + 1)
> +    - if
> +        test $(uname) = "FreeBSD"
> +        ;
> +      then
> +        JOBS=$(sysctl -n hw.ncpu)
> +        ;
> +      else
> +        JOBS=$(expr $(nproc) + 1)
> +        ;
> +      fi
> +    - echo "=== Using $JOBS simultaneous jobs ==="

It has grown to beyond variables definitions. I suggest to rename it to 
something like `.before_script_template` (pure creativity, see?).

- Wainer

>   
>   .native_build_job_template:
>     stage: build

Re: [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable

[Wainer dos Santos Moschetta]

On 5/19/21 3:45 PM, Philippe Mathieu-Daudé wrote:
> Add the $MAKE variable to call GNU make, and set it to 'gmake'
> on FreeBSD to avoid:
>
>    $ make -j"$JOBS"
>    make: Unknown modifier ','
>    make: "/builds/dTyar424/0/qemu/build/Makefile" line 3: Need an operator
>    make: "/builds/dTyar424/0/qemu/build/Makefile" line 4: Missing dependency operator
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 9 ++++++---
>   1 file changed, 6 insertions(+), 3 deletions(-)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index fe4f18595ac..f284d7a0eec 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -5,9 +5,11 @@
>           ;
>         then
>           JOBS=$(sysctl -n hw.ncpu)
> +        MAKE=gmake
>           ;
>         else
>           JOBS=$(expr $(nproc) + 1)
> +        MAKE=make
>           ;
>         fi
>       - echo "=== Using $JOBS simultaneous jobs ==="
> @@ -33,22 +35,23 @@
>         then
>           ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
>         fi || exit 1;
> -    - make -j"$JOBS"
> +    - $MAKE -j"$JOBS"
>       - if test -n "$MAKE_CHECK_ARGS";
>         then
> -        make -j"$JOBS" $MAKE_CHECK_ARGS ;
> +        $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
>         fi
>   
>   .native_test_job_template:
>     stage: test
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  extends: .environment_variables_template
>     script:
>       - scripts/git-submodule.sh update
>           $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
>       - cd build
>       - find . -type f -exec touch {} +
>       # Avoid recompiling by hiding ninja with NINJA=":"
> -    - make NINJA=":" $MAKE_CHECK_ARGS
> +    - $MAKE NINJA=":" $MAKE_CHECK_ARGS
>   
>   .integration_test_job_template:
>     extends: .native_test_job_template

Re: [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

On Thu, May 20, 2021 at 12:18 PM Wainer dos Santos Moschetta <
wainersm@redhat.com> wrote:

> Hi,
>
> On 5/19/21 3:45 PM, Philippe Mathieu-Daudé wrote:
> > 'nproc' is not available on FreeBSD:
> >
> >    $ JOBS=$(expr $(nproc) + 1)
> >    bash: line 119: nproc: command not found
> >    expr: syntax error
> >
> > Instead, use 'sysctl -n hw.ncpu'.
> >
> > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > ---
> >   .gitlab-ci.d/buildtest-template.yml | 12 +++++++++++-
> >   1 file changed, 11 insertions(+), 1 deletion(-)
> >
> > diff --git a/.gitlab-ci.d/buildtest-template.yml
> b/.gitlab-ci.d/buildtest-template.yml
> > index 58b01744751..fe4f18595ac 100644
> > --- a/.gitlab-ci.d/buildtest-template.yml
> > +++ b/.gitlab-ci.d/buildtest-template.yml
> > @@ -1,6 +1,16 @@
> >   .environment_variables_template:
> >     before_script:
> > -    - JOBS=$(expr $(nproc) + 1)
> > +    - if
> > +        test $(uname) = "FreeBSD"
> > +        ;
> > +      then
> > +        JOBS=$(sysctl -n hw.ncpu)
> > +        ;
> > +      else
> > +        JOBS=$(expr $(nproc) + 1)
> > +        ;
> > +      fi
> > +    - echo "=== Using $JOBS simultaneous jobs ==="
>
> It has grown to beyond variables definitions. I suggest to rename it to
> something like `.before_script_template` (pure creativity, see?).
>

Also, would it help if FreeBSD were to add nproc in the future?

Warner

[-- Attachment #2: Type: text/html, Size: 2095 bytes --]

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Thomas Huth]

On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> +Stefan/Daniel
> 
> On 5/20/21 10:02 AM, Thomas Huth wrote:
>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>> If a runner has ccache installed, use it and display statistics
>>> at the end of the build.
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>    .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>    1 file changed, 5 insertions(+)
>>>
>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>> b/.gitlab-ci.d/buildtest-template.yml
>>> index f284d7a0eec..a625c697d3b 100644
>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>> @@ -6,13 +6,18 @@
>>>          then
>>>            JOBS=$(sysctl -n hw.ncpu)
>>>            MAKE=gmake
>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>            ;
>>>          else
>>>            JOBS=$(expr $(nproc) + 1)
>>>            MAKE=make
>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>
>> That does not make sense for the shared runners yet. We first need
>> something to enable the caching there - see my series "Use ccache in the
>> gitlab-CI" from April (which is currently stalled unfortunately).
> 
> TL;DR: I don't think we should restrict our templates to shared runners.

I'm certainly not voting for restricting ourselves to only use shared 
runners here - but my concern is that this actually *slows* down the shared 
runners even more! (sorry, I should have elaborated on that in my previous 
mail already)

When I was experimenting with ccache in the shared runners, I saw that the 
jobs are running even slower with ccache enabled as long as the cache is not 
populated yet. You only get a speedup afterwards. So if you add this now 
without also adding the possibility to store the cache persistently, the 
shared runners will try to populate the cache each time just to throw away 
the results afterwards again. Thus all the shared runners only get slower 
without any real benefit here.

Thus we either need to get ccache working properly for the shared runners 
first, or you have to think of a different way of enabling ccache for the 
non-shared runners, so that it does not affect the shared runners negatively.

  Thomas

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Daniel P. Berrangé]

On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> > +Stefan/Daniel
> > 
> > On 5/20/21 10:02 AM, Thomas Huth wrote:
> > > On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> > > > If a runner has ccache installed, use it and display statistics
> > > > at the end of the build.
> > > > 
> > > > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > > > ---
> > > >    .gitlab-ci.d/buildtest-template.yml | 5 +++++
> > > >    1 file changed, 5 insertions(+)
> > > > 
> > > > diff --git a/.gitlab-ci.d/buildtest-template.yml
> > > > b/.gitlab-ci.d/buildtest-template.yml
> > > > index f284d7a0eec..a625c697d3b 100644
> > > > --- a/.gitlab-ci.d/buildtest-template.yml
> > > > +++ b/.gitlab-ci.d/buildtest-template.yml
> > > > @@ -6,13 +6,18 @@
> > > >          then
> > > >            JOBS=$(sysctl -n hw.ncpu)
> > > >            MAKE=gmake
> > > > +        PATH=/usr/local/libexec/ccache:$PATH
> > > >            ;
> > > >          else
> > > >            JOBS=$(expr $(nproc) + 1)
> > > >            MAKE=make
> > > > +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> > > 
> > > That does not make sense for the shared runners yet. We first need
> > > something to enable the caching there - see my series "Use ccache in the
> > > gitlab-CI" from April (which is currently stalled unfortunately).
> > 
> > TL;DR: I don't think we should restrict our templates to shared runners.
> 
> I'm certainly not voting for restricting ourselves to only use shared
> runners here - but my concern is that this actually *slows* down the shared
> runners even more! (sorry, I should have elaborated on that in my previous
> mail already)
> 
> When I was experimenting with ccache in the shared runners, I saw that the
> jobs are running even slower with ccache enabled as long as the cache is not
> populated yet. You only get a speedup afterwards. So if you add this now
> without also adding the possibility to store the cache persistently, the
> shared runners will try to populate the cache each time just to throw away
> the results afterwards again. Thus all the shared runners only get slower
> without any real benefit here.
> 
> Thus we either need to get ccache working properly for the shared runners
> first, or you have to think of a different way of enabling ccache for the
> non-shared runners, so that it does not affect the shared runners
> negatively.

Is there anything functional holding up your previous full cccache support
series from last month ? Or is it just lack of reviews ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Thomas Huth]

On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
>>> +Stefan/Daniel
>>>
>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>>>> If a runner has ccache installed, use it and display statistics
>>>>> at the end of the build.
>>>>>
>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>> ---
>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>>>     1 file changed, 5 insertions(+)
>>>>>
>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>>>> b/.gitlab-ci.d/buildtest-template.yml
>>>>> index f284d7a0eec..a625c697d3b 100644
>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>>>> @@ -6,13 +6,18 @@
>>>>>           then
>>>>>             JOBS=$(sysctl -n hw.ncpu)
>>>>>             MAKE=gmake
>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>>>             ;
>>>>>           else
>>>>>             JOBS=$(expr $(nproc) + 1)
>>>>>             MAKE=make
>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>>>
>>>> That does not make sense for the shared runners yet. We first need
>>>> something to enable the caching there - see my series "Use ccache in the
>>>> gitlab-CI" from April (which is currently stalled unfortunately).
>>>
>>> TL;DR: I don't think we should restrict our templates to shared runners.
>>
>> I'm certainly not voting for restricting ourselves to only use shared
>> runners here - but my concern is that this actually *slows* down the shared
>> runners even more! (sorry, I should have elaborated on that in my previous
>> mail already)
>>
>> When I was experimenting with ccache in the shared runners, I saw that the
>> jobs are running even slower with ccache enabled as long as the cache is not
>> populated yet. You only get a speedup afterwards. So if you add this now
>> without also adding the possibility to store the cache persistently, the
>> shared runners will try to populate the cache each time just to throw away
>> the results afterwards again. Thus all the shared runners only get slower
>> without any real benefit here.
>>
>> Thus we either need to get ccache working properly for the shared runners
>> first, or you have to think of a different way of enabling ccache for the
>> non-shared runners, so that it does not affect the shared runners
>> negatively.
> 
> Is there anything functional holding up your previous full cccache support
> series from last month ? Or is it just lack of reviews ?

It's basically the problems mentioned in the cover letter and Stefan's 
comment here:

  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html

The series needs more love and more testing, if it's feasible with the 
gitlab-CI architecture at all to use ccache in a good way ... if we don't 
get a real speedup by the patches, it's certainly not worth the effort to 
integrate this...

If someone wants to have a try to improve the patch series, your help is 
certainly welcome - since at least I personally lack the time and motivation 
to improve this any further.

  Thomas

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Daniel P. Berrangé]

On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> > On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> > > On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> > > > +Stefan/Daniel
> > > > 
> > > > On 5/20/21 10:02 AM, Thomas Huth wrote:
> > > > > On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> > > > > > If a runner has ccache installed, use it and display statistics
> > > > > > at the end of the build.
> > > > > > 
> > > > > > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > > > > > ---
> > > > > >     .gitlab-ci.d/buildtest-template.yml | 5 +++++
> > > > > >     1 file changed, 5 insertions(+)
> > > > > > 
> > > > > > diff --git a/.gitlab-ci.d/buildtest-template.yml
> > > > > > b/.gitlab-ci.d/buildtest-template.yml
> > > > > > index f284d7a0eec..a625c697d3b 100644
> > > > > > --- a/.gitlab-ci.d/buildtest-template.yml
> > > > > > +++ b/.gitlab-ci.d/buildtest-template.yml
> > > > > > @@ -6,13 +6,18 @@
> > > > > >           then
> > > > > >             JOBS=$(sysctl -n hw.ncpu)
> > > > > >             MAKE=gmake
> > > > > > +        PATH=/usr/local/libexec/ccache:$PATH
> > > > > >             ;
> > > > > >           else
> > > > > >             JOBS=$(expr $(nproc) + 1)
> > > > > >             MAKE=make
> > > > > > +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> > > > > 
> > > > > That does not make sense for the shared runners yet. We first need
> > > > > something to enable the caching there - see my series "Use ccache in the
> > > > > gitlab-CI" from April (which is currently stalled unfortunately).
> > > > 
> > > > TL;DR: I don't think we should restrict our templates to shared runners.
> > > 
> > > I'm certainly not voting for restricting ourselves to only use shared
> > > runners here - but my concern is that this actually *slows* down the shared
> > > runners even more! (sorry, I should have elaborated on that in my previous
> > > mail already)
> > > 
> > > When I was experimenting with ccache in the shared runners, I saw that the
> > > jobs are running even slower with ccache enabled as long as the cache is not
> > > populated yet. You only get a speedup afterwards. So if you add this now
> > > without also adding the possibility to store the cache persistently, the
> > > shared runners will try to populate the cache each time just to throw away
> > > the results afterwards again. Thus all the shared runners only get slower
> > > without any real benefit here.
> > > 
> > > Thus we either need to get ccache working properly for the shared runners
> > > first, or you have to think of a different way of enabling ccache for the
> > > non-shared runners, so that it does not affect the shared runners
> > > negatively.
> > 
> > Is there anything functional holding up your previous full cccache support
> > series from last month ? Or is it just lack of reviews ?
> 
> It's basically the problems mentioned in the cover letter and Stefan's
> comment here:
> 
>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html

I'm not sure I understand why Stefan thinks gitlab's caching doesn't
benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
sped up our builds significantly.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Philippe Mathieu-Daudé]

On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
> On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
>> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
>>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
>>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
>>>>> +Stefan/Daniel
>>>>>
>>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
>>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>>>>>> If a runner has ccache installed, use it and display statistics
>>>>>>> at the end of the build.
>>>>>>>
>>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>>>> ---
>>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>>>>>     1 file changed, 5 insertions(+)
>>>>>>>
>>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>>>>>> b/.gitlab-ci.d/buildtest-template.yml
>>>>>>> index f284d7a0eec..a625c697d3b 100644
>>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>>>>>> @@ -6,13 +6,18 @@
>>>>>>>           then
>>>>>>>             JOBS=$(sysctl -n hw.ncpu)
>>>>>>>             MAKE=gmake
>>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>>>>>             ;
>>>>>>>           else
>>>>>>>             JOBS=$(expr $(nproc) + 1)
>>>>>>>             MAKE=make
>>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>>>>>
>>>>>> That does not make sense for the shared runners yet. We first need
>>>>>> something to enable the caching there - see my series "Use ccache in the
>>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
>>>>>
>>>>> TL;DR: I don't think we should restrict our templates to shared runners.
>>>>
>>>> I'm certainly not voting for restricting ourselves to only use shared
>>>> runners here - but my concern is that this actually *slows* down the shared
>>>> runners even more! (sorry, I should have elaborated on that in my previous
>>>> mail already)
>>>>
>>>> When I was experimenting with ccache in the shared runners, I saw that the
>>>> jobs are running even slower with ccache enabled as long as the cache is not
>>>> populated yet. You only get a speedup afterwards. So if you add this now
>>>> without also adding the possibility to store the cache persistently, the
>>>> shared runners will try to populate the cache each time just to throw away
>>>> the results afterwards again. Thus all the shared runners only get slower
>>>> without any real benefit here.
>>>>
>>>> Thus we either need to get ccache working properly for the shared runners
>>>> first, or you have to think of a different way of enabling ccache for the
>>>> non-shared runners, so that it does not affect the shared runners
>>>> negatively.
>>>
>>> Is there anything functional holding up your previous full cccache support
>>> series from last month ? Or is it just lack of reviews ?
>>
>> It's basically the problems mentioned in the cover letter and Stefan's
>> comment here:
>>
>>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
> 
> I'm not sure I understand why Stefan thinks gitlab's caching doesn't
> benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
> sped up our builds significantly.

I think Stefan is referring to a comment I made, when using both
shared runners and dedicated runners (what I'm currently testing)
various jobs are stuck transferring artifacts/cache {FROM, TO}
{shared, dedicated} runners at the same time, which is sub-optimal
because it saturate the dedicated runner network link.

If we want to use pool of runners to restrict transfer between
runners from the same physical pool, then it because a maintenance
headache.

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Willian Rampazzo]

On Fri, May 21, 2021 at 8:03 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> > On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> >> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> >>> +Stefan/Daniel
> >>>
> >>> On 5/20/21 10:02 AM, Thomas Huth wrote:
> >>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> >>>>> If a runner has ccache installed, use it and display statistics
> >>>>> at the end of the build.
> >>>>>
> >>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >>>>> ---
> >>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
> >>>>>     1 file changed, 5 insertions(+)
> >>>>>
> >>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
> >>>>> b/.gitlab-ci.d/buildtest-template.yml
> >>>>> index f284d7a0eec..a625c697d3b 100644
> >>>>> --- a/.gitlab-ci.d/buildtest-template.yml
> >>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
> >>>>> @@ -6,13 +6,18 @@
> >>>>>           then
> >>>>>             JOBS=$(sysctl -n hw.ncpu)
> >>>>>             MAKE=gmake
> >>>>> +        PATH=/usr/local/libexec/ccache:$PATH
> >>>>>             ;
> >>>>>           else
> >>>>>             JOBS=$(expr $(nproc) + 1)
> >>>>>             MAKE=make
> >>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> >>>>
> >>>> That does not make sense for the shared runners yet. We first need
> >>>> something to enable the caching there - see my series "Use ccache in the
> >>>> gitlab-CI" from April (which is currently stalled unfortunately).
> >>>
> >>> TL;DR: I don't think we should restrict our templates to shared runners.
> >>
> >> I'm certainly not voting for restricting ourselves to only use shared
> >> runners here - but my concern is that this actually *slows* down the shared
> >> runners even more! (sorry, I should have elaborated on that in my previous
> >> mail already)
> >>
> >> When I was experimenting with ccache in the shared runners, I saw that the
> >> jobs are running even slower with ccache enabled as long as the cache is not
> >> populated yet. You only get a speedup afterwards. So if you add this now
> >> without also adding the possibility to store the cache persistently, the
> >> shared runners will try to populate the cache each time just to throw away
> >> the results afterwards again. Thus all the shared runners only get slower
> >> without any real benefit here.
> >>

I was in favor of adding the changes and benefiting custom runners
until I saw your reply. In this case, I agree we should investigate
how these changes affect shared runners.

> >> Thus we either need to get ccache working properly for the shared runners
> >> first, or you have to think of a different way of enabling ccache for the
> >> non-shared runners, so that it does not affect the shared runners
> >> negatively.
> >
> > Is there anything functional holding up your previous full cccache support
> > series from last month ? Or is it just lack of reviews ?
>
> It's basically the problems mentioned in the cover letter and Stefan's
> comment here:
>
>   https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>
> The series needs more love and more testing, if it's feasible with the
> gitlab-CI architecture at all to use ccache in a good way ... if we don't
> get a real speedup by the patches, it's certainly not worth the effort to
> integrate this...
>
> If someone wants to have a try to improve the patch series, your help is
> certainly welcome - since at least I personally lack the time and motivation
> to improve this any further.

Do you mind if I play with your patch series? I do not promise 100% of
my time working on it, but I was thinking about dedicating some time
to ccache before your patch series.

>
>   Thomas
>

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Daniel P. Berrangé]

On Fri, May 21, 2021 at 02:27:26PM +0200, Philippe Mathieu-Daudé wrote:
> On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
> > On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
> >> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> >>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> >>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> >>>>> +Stefan/Daniel
> >>>>>
> >>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
> >>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> >>>>>>> If a runner has ccache installed, use it and display statistics
> >>>>>>> at the end of the build.
> >>>>>>>
> >>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >>>>>>> ---
> >>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
> >>>>>>>     1 file changed, 5 insertions(+)
> >>>>>>>
> >>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
> >>>>>>> b/.gitlab-ci.d/buildtest-template.yml
> >>>>>>> index f284d7a0eec..a625c697d3b 100644
> >>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
> >>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
> >>>>>>> @@ -6,13 +6,18 @@
> >>>>>>>           then
> >>>>>>>             JOBS=$(sysctl -n hw.ncpu)
> >>>>>>>             MAKE=gmake
> >>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
> >>>>>>>             ;
> >>>>>>>           else
> >>>>>>>             JOBS=$(expr $(nproc) + 1)
> >>>>>>>             MAKE=make
> >>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> >>>>>>
> >>>>>> That does not make sense for the shared runners yet. We first need
> >>>>>> something to enable the caching there - see my series "Use ccache in the
> >>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
> >>>>>
> >>>>> TL;DR: I don't think we should restrict our templates to shared runners.
> >>>>
> >>>> I'm certainly not voting for restricting ourselves to only use shared
> >>>> runners here - but my concern is that this actually *slows* down the shared
> >>>> runners even more! (sorry, I should have elaborated on that in my previous
> >>>> mail already)
> >>>>
> >>>> When I was experimenting with ccache in the shared runners, I saw that the
> >>>> jobs are running even slower with ccache enabled as long as the cache is not
> >>>> populated yet. You only get a speedup afterwards. So if you add this now
> >>>> without also adding the possibility to store the cache persistently, the
> >>>> shared runners will try to populate the cache each time just to throw away
> >>>> the results afterwards again. Thus all the shared runners only get slower
> >>>> without any real benefit here.
> >>>>
> >>>> Thus we either need to get ccache working properly for the shared runners
> >>>> first, or you have to think of a different way of enabling ccache for the
> >>>> non-shared runners, so that it does not affect the shared runners
> >>>> negatively.
> >>>
> >>> Is there anything functional holding up your previous full cccache support
> >>> series from last month ? Or is it just lack of reviews ?
> >>
> >> It's basically the problems mentioned in the cover letter and Stefan's
> >> comment here:
> >>
> >>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
> > 
> > I'm not sure I understand why Stefan thinks gitlab's caching doesn't
> > benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
> > sped up our builds significantly.
> 
> I think Stefan is referring to a comment I made, when using both
> shared runners and dedicated runners (what I'm currently testing)
> various jobs are stuck transferring artifacts/cache {FROM, TO}
> {shared, dedicated} runners at the same time, which is sub-optimal
> because it saturate the dedicated runner network link.

I think we're over thinking things a bit too much and worrying about
scenarios that we're not actually hitting that frequently today, and
delaying the benefit for everyone.

Our common case is that most contributors are simply using shared
runners exclusively, as is the main qemu repo staging branch. AFAIK
these should benefit from a simple ccache enablement today.

Since there are questions about other setups though, we can just
provide an easy way to turn it off. eg:

  if test -z "$QEMU_CI_SKIP_CCACHE"
  then
     PATH=/usr/local/libexec/ccache:$PATH
  fi

anyone who wishes to disable it, can just set that variable in their
git repo fork. If there are specific jobs we want to disable cccache
for, those jobs can set that too.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Thomas Huth]

On 21/05/2021 14.36, Willian Rampazzo wrote:
> On Fri, May 21, 2021 at 8:03 AM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
[...]
>>> Is there anything functional holding up your previous full cccache support
>>> series from last month ? Or is it just lack of reviews ?
>>
>> It's basically the problems mentioned in the cover letter and Stefan's
>> comment here:
>>
>>    https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>>
>> The series needs more love and more testing, if it's feasible with the
>> gitlab-CI architecture at all to use ccache in a good way ... if we don't
>> get a real speedup by the patches, it's certainly not worth the effort to
>> integrate this...
>>
>> If someone wants to have a try to improve the patch series, your help is
>> certainly welcome - since at least I personally lack the time and motivation
>> to improve this any further.
> 
> Do you mind if I play with your patch series? I do not promise 100% of
> my time working on it, but I was thinking about dedicating some time
> to ccache before your patch series.

I certainly don't mind! If you get it running well enough so that the total 
runtime decreases, that would be really great!

  Thomas

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Philippe Mathieu-Daudé]

On 5/21/21 2:52 PM, Daniel P. Berrangé wrote:
> On Fri, May 21, 2021 at 02:27:26PM +0200, Philippe Mathieu-Daudé wrote:
>> On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
>>> On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
>>>> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
>>>>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
>>>>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
>>>>>>> +Stefan/Daniel
>>>>>>>
>>>>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
>>>>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>>>>>>>> If a runner has ccache installed, use it and display statistics
>>>>>>>>> at the end of the build.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>>>>>> ---
>>>>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>>>>>>>     1 file changed, 5 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>> b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>> index f284d7a0eec..a625c697d3b 100644
>>>>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>> @@ -6,13 +6,18 @@
>>>>>>>>>           then
>>>>>>>>>             JOBS=$(sysctl -n hw.ncpu)
>>>>>>>>>             MAKE=gmake
>>>>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>>>>>>>             ;
>>>>>>>>>           else
>>>>>>>>>             JOBS=$(expr $(nproc) + 1)
>>>>>>>>>             MAKE=make
>>>>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>>>>>>>
>>>>>>>> That does not make sense for the shared runners yet. We first need
>>>>>>>> something to enable the caching there - see my series "Use ccache in the
>>>>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
>>>>>>>
>>>>>>> TL;DR: I don't think we should restrict our templates to shared runners.
>>>>>>
>>>>>> I'm certainly not voting for restricting ourselves to only use shared
>>>>>> runners here - but my concern is that this actually *slows* down the shared
>>>>>> runners even more! (sorry, I should have elaborated on that in my previous
>>>>>> mail already)
>>>>>>
>>>>>> When I was experimenting with ccache in the shared runners, I saw that the
>>>>>> jobs are running even slower with ccache enabled as long as the cache is not
>>>>>> populated yet. You only get a speedup afterwards. So if you add this now
>>>>>> without also adding the possibility to store the cache persistently, the
>>>>>> shared runners will try to populate the cache each time just to throw away
>>>>>> the results afterwards again. Thus all the shared runners only get slower
>>>>>> without any real benefit here.
>>>>>>
>>>>>> Thus we either need to get ccache working properly for the shared runners
>>>>>> first, or you have to think of a different way of enabling ccache for the
>>>>>> non-shared runners, so that it does not affect the shared runners
>>>>>> negatively.
>>>>>
>>>>> Is there anything functional holding up your previous full cccache support
>>>>> series from last month ? Or is it just lack of reviews ?
>>>>
>>>> It's basically the problems mentioned in the cover letter and Stefan's
>>>> comment here:
>>>>
>>>>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>>>
>>> I'm not sure I understand why Stefan thinks gitlab's caching doesn't
>>> benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
>>> sped up our builds significantly.
>>
>> I think Stefan is referring to a comment I made, when using both
>> shared runners and dedicated runners (what I'm currently testing)
>> various jobs are stuck transferring artifacts/cache {FROM, TO}
>> {shared, dedicated} runners at the same time, which is sub-optimal
>> because it saturate the dedicated runner network link.
> 
> I think we're over thinking things a bit too much and worrying about
> scenarios that we're not actually hitting that frequently today, and
> delaying the benefit for everyone.
> 
> Our common case is that most contributors are simply using shared
> runners exclusively, as is the main qemu repo staging branch. AFAIK
> these should benefit from a simple ccache enablement today.
> 
> Since there are questions about other setups though, we can just
> provide an easy way to turn it off. eg:
> 
>   if test -z "$QEMU_CI_SKIP_CCACHE"
>   then
>      PATH=/usr/local/libexec/ccache:$PATH
>   fi
> 
> anyone who wishes to disable it, can just set that variable in their
> git repo fork. If there are specific jobs we want to disable cccache
> for, those jobs can set that too.

OK, understood. I'll see with Willian how to have ccache working.

Thanks for the feedback,

Phil.

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4636 bytes --]

On Fri, May 21, 2021 at 01:52:13PM +0100, Daniel P. Berrangé wrote:
> On Fri, May 21, 2021 at 02:27:26PM +0200, Philippe Mathieu-Daudé wrote:
> > On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
> > > On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
> > >> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> > >>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> > >>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> > >>>>> +Stefan/Daniel
> > >>>>>
> > >>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
> > >>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> > >>>>>>> If a runner has ccache installed, use it and display statistics
> > >>>>>>> at the end of the build.
> > >>>>>>>
> > >>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > >>>>>>> ---
> > >>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
> > >>>>>>>     1 file changed, 5 insertions(+)
> > >>>>>>>
> > >>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
> > >>>>>>> b/.gitlab-ci.d/buildtest-template.yml
> > >>>>>>> index f284d7a0eec..a625c697d3b 100644
> > >>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
> > >>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
> > >>>>>>> @@ -6,13 +6,18 @@
> > >>>>>>>           then
> > >>>>>>>             JOBS=$(sysctl -n hw.ncpu)
> > >>>>>>>             MAKE=gmake
> > >>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
> > >>>>>>>             ;
> > >>>>>>>           else
> > >>>>>>>             JOBS=$(expr $(nproc) + 1)
> > >>>>>>>             MAKE=make
> > >>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> > >>>>>>
> > >>>>>> That does not make sense for the shared runners yet. We first need
> > >>>>>> something to enable the caching there - see my series "Use ccache in the
> > >>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
> > >>>>>
> > >>>>> TL;DR: I don't think we should restrict our templates to shared runners.
> > >>>>
> > >>>> I'm certainly not voting for restricting ourselves to only use shared
> > >>>> runners here - but my concern is that this actually *slows* down the shared
> > >>>> runners even more! (sorry, I should have elaborated on that in my previous
> > >>>> mail already)
> > >>>>
> > >>>> When I was experimenting with ccache in the shared runners, I saw that the
> > >>>> jobs are running even slower with ccache enabled as long as the cache is not
> > >>>> populated yet. You only get a speedup afterwards. So if you add this now
> > >>>> without also adding the possibility to store the cache persistently, the
> > >>>> shared runners will try to populate the cache each time just to throw away
> > >>>> the results afterwards again. Thus all the shared runners only get slower
> > >>>> without any real benefit here.
> > >>>>
> > >>>> Thus we either need to get ccache working properly for the shared runners
> > >>>> first, or you have to think of a different way of enabling ccache for the
> > >>>> non-shared runners, so that it does not affect the shared runners
> > >>>> negatively.
> > >>>
> > >>> Is there anything functional holding up your previous full cccache support
> > >>> series from last month ? Or is it just lack of reviews ?
> > >>
> > >> It's basically the problems mentioned in the cover letter and Stefan's
> > >> comment here:
> > >>
> > >>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
> > > 
> > > I'm not sure I understand why Stefan thinks gitlab's caching doesn't
> > > benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
> > > sped up our builds significantly.
> > 
> > I think Stefan is referring to a comment I made, when using both
> > shared runners and dedicated runners (what I'm currently testing)
> > various jobs are stuck transferring artifacts/cache {FROM, TO}
> > {shared, dedicated} runners at the same time, which is sub-optimal
> > because it saturate the dedicated runner network link.
> 
> I think we're over thinking things a bit too much and worrying about
> scenarios that we're not actually hitting that frequently today, and
> delaying the benefit for everyone.

Thomas' original email indicated using ccache with QEMU isn't
necessarily a win:

  Additionally, the jobs are sometimes running even slower, e.g. if the
  cache has not been populated yet or if there are a lot of cache
  misses,

Let's measure the time taken both on first run and on a subsequent run.
This information can be included in the patch series cover letter so we
know ccache performance has been measured and it works well.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Philippe Mathieu-Daudé]

On 5/21/21 4:21 PM, Philippe Mathieu-Daudé wrote:
> On 5/21/21 2:52 PM, Daniel P. Berrangé wrote:
>> On Fri, May 21, 2021 at 02:27:26PM +0200, Philippe Mathieu-Daudé wrote:
>>> On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
>>>> On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
>>>>> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
>>>>>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
>>>>>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
>>>>>>>> +Stefan/Daniel
>>>>>>>>
>>>>>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
>>>>>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>>>>>>>>> If a runner has ccache installed, use it and display statistics
>>>>>>>>>> at the end of the build.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>>>>>>> ---
>>>>>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>>>>>>>>     1 file changed, 5 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> index f284d7a0eec..a625c697d3b 100644
>>>>>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> @@ -6,13 +6,18 @@
>>>>>>>>>>           then
>>>>>>>>>>             JOBS=$(sysctl -n hw.ncpu)
>>>>>>>>>>             MAKE=gmake
>>>>>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>>>>>>>>             ;
>>>>>>>>>>           else
>>>>>>>>>>             JOBS=$(expr $(nproc) + 1)
>>>>>>>>>>             MAKE=make
>>>>>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>>>>>>>>
>>>>>>>>> That does not make sense for the shared runners yet. We first need
>>>>>>>>> something to enable the caching there - see my series "Use ccache in the
>>>>>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
>>>>>>>>
>>>>>>>> TL;DR: I don't think we should restrict our templates to shared runners.
>>>>>>>
>>>>>>> I'm certainly not voting for restricting ourselves to only use shared
>>>>>>> runners here - but my concern is that this actually *slows* down the shared
>>>>>>> runners even more! (sorry, I should have elaborated on that in my previous
>>>>>>> mail already)
>>>>>>>
>>>>>>> When I was experimenting with ccache in the shared runners, I saw that the
>>>>>>> jobs are running even slower with ccache enabled as long as the cache is not
>>>>>>> populated yet. You only get a speedup afterwards. So if you add this now
>>>>>>> without also adding the possibility to store the cache persistently, the
>>>>>>> shared runners will try to populate the cache each time just to throw away
>>>>>>> the results afterwards again. Thus all the shared runners only get slower
>>>>>>> without any real benefit here.
>>>>>>>
>>>>>>> Thus we either need to get ccache working properly for the shared runners
>>>>>>> first, or you have to think of a different way of enabling ccache for the
>>>>>>> non-shared runners, so that it does not affect the shared runners
>>>>>>> negatively.
>>>>>>
>>>>>> Is there anything functional holding up your previous full cccache support
>>>>>> series from last month ? Or is it just lack of reviews ?
>>>>>
>>>>> It's basically the problems mentioned in the cover letter and Stefan's
>>>>> comment here:
>>>>>
>>>>>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>>>>
>>>> I'm not sure I understand why Stefan thinks gitlab's caching doesn't
>>>> benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
>>>> sped up our builds significantly.
>>>
>>> I think Stefan is referring to a comment I made, when using both
>>> shared runners and dedicated runners (what I'm currently testing)
>>> various jobs are stuck transferring artifacts/cache {FROM, TO}
>>> {shared, dedicated} runners at the same time, which is sub-optimal
>>> because it saturate the dedicated runner network link.

FYI In case we need to sort this out later, the 'resource_group' might
help us with this:
https://docs.gitlab.com/ee/ci/yaml/#resource_group

>> I think we're over thinking things a bit too much and worrying about
>> scenarios that we're not actually hitting that frequently today, and
>> delaying the benefit for everyone.
>>
>> Our common case is that most contributors are simply using shared
>> runners exclusively, as is the main qemu repo staging branch. AFAIK
>> these should benefit from a simple ccache enablement today.
>>
>> Since there are questions about other setups though, we can just
>> provide an easy way to turn it off. eg:
>>
>>   if test -z "$QEMU_CI_SKIP_CCACHE"
>>   then
>>      PATH=/usr/local/libexec/ccache:$PATH
>>   fi
>>
>> anyone who wishes to disable it, can just set that variable in their
>> git repo fork. If there are specific jobs we want to disable cccache
>> for, those jobs can set that too.
> 
> OK, understood. I'll see with Willian how to have ccache working.
> 
> Thanks for the feedback,
> 
> Phil.
> 

Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics

[Philippe Mathieu-Daudé]

On 5/24/21 3:51 PM, Stefan Hajnoczi wrote:
> On Fri, May 21, 2021 at 01:52:13PM +0100, Daniel P. Berrangé wrote:
>> On Fri, May 21, 2021 at 02:27:26PM +0200, Philippe Mathieu-Daudé wrote:
>>> On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
>>>> On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
>>>>> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
>>>>>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
>>>>>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
>>>>>>>> +Stefan/Daniel
>>>>>>>>
>>>>>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
>>>>>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>>>>>>>>> If a runner has ccache installed, use it and display statistics
>>>>>>>>>> at the end of the build.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>>>>>>> ---
>>>>>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>>>>>>>>     1 file changed, 5 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> index f284d7a0eec..a625c697d3b 100644
>>>>>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> @@ -6,13 +6,18 @@
>>>>>>>>>>           then
>>>>>>>>>>             JOBS=$(sysctl -n hw.ncpu)
>>>>>>>>>>             MAKE=gmake
>>>>>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>>>>>>>>             ;
>>>>>>>>>>           else
>>>>>>>>>>             JOBS=$(expr $(nproc) + 1)
>>>>>>>>>>             MAKE=make
>>>>>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>>>>>>>>
>>>>>>>>> That does not make sense for the shared runners yet. We first need
>>>>>>>>> something to enable the caching there - see my series "Use ccache in the
>>>>>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
>>>>>>>>
>>>>>>>> TL;DR: I don't think we should restrict our templates to shared runners.
>>>>>>>
>>>>>>> I'm certainly not voting for restricting ourselves to only use shared
>>>>>>> runners here - but my concern is that this actually *slows* down the shared
>>>>>>> runners even more! (sorry, I should have elaborated on that in my previous
>>>>>>> mail already)
>>>>>>>
>>>>>>> When I was experimenting with ccache in the shared runners, I saw that the
>>>>>>> jobs are running even slower with ccache enabled as long as the cache is not
>>>>>>> populated yet. You only get a speedup afterwards. So if you add this now
>>>>>>> without also adding the possibility to store the cache persistently, the
>>>>>>> shared runners will try to populate the cache each time just to throw away
>>>>>>> the results afterwards again. Thus all the shared runners only get slower
>>>>>>> without any real benefit here.
>>>>>>>
>>>>>>> Thus we either need to get ccache working properly for the shared runners
>>>>>>> first, or you have to think of a different way of enabling ccache for the
>>>>>>> non-shared runners, so that it does not affect the shared runners
>>>>>>> negatively.
>>>>>>
>>>>>> Is there anything functional holding up your previous full cccache support
>>>>>> series from last month ? Or is it just lack of reviews ?
>>>>>
>>>>> It's basically the problems mentioned in the cover letter and Stefan's
>>>>> comment here:
>>>>>
>>>>>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>>>>
>>>> I'm not sure I understand why Stefan thinks gitlab's caching doesn't
>>>> benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
>>>> sped up our builds significantly.
>>>
>>> I think Stefan is referring to a comment I made, when using both
>>> shared runners and dedicated runners (what I'm currently testing)
>>> various jobs are stuck transferring artifacts/cache {FROM, TO}
>>> {shared, dedicated} runners at the same time, which is sub-optimal
>>> because it saturate the dedicated runner network link.
>>
>> I think we're over thinking things a bit too much and worrying about
>> scenarios that we're not actually hitting that frequently today, and
>> delaying the benefit for everyone.
> 
> Thomas' original email indicated using ccache with QEMU isn't
> necessarily a win:
> 
>   Additionally, the jobs are sometimes running even slower, e.g. if the
>   cache has not been populated yet or if there are a lot of cache
>   misses,
> 
> Let's measure the time taken both on first run and on a subsequent run.
> This information can be included in the patch series cover letter so we
> know ccache performance has been measured and it works well.

To whoever is interested in looking at this problem, GitLab provides
the TRANSFER_METER_FREQUENCY variable to debug this problem; and the
compression ratio can be adjusted using ARTIFACT_COMPRESSION_LEVEL
and CACHE_COMPRESSION_LEVEL:
https://docs.gitlab.com/ee/ci/runners/README.html#artifact-and-cache-settings