All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v3 0/6] gitlab-ci: Allow using FreeBSD runners
@ 2021-05-19 18:45 Philippe Mathieu-Daudé
  2021-05-19 18:45 ` [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template Philippe Mathieu-Daudé
                   ` (5 more replies)
  0 siblings, 6 replies; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-19 18:45 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée,
	Warner Losh

Minor shell fixes to use FreeBSD runners.\r
\r
Since v2:\r
- fixed typo (Warner)\r
- added Warner R-b tag\r
\r
Since v1:\r
- share the FreeBSD jobs, restricted with QEMU_CUSTOM_RUNNER=freebsd\r
  so mergeable in mainstream.\r
\r
Jobs: https://gitlab.com/philmd/qemu/-/pipelines/301361673\r
\r
Based-on: <20210511072952.2813358-1-f4bug@amsat.org>\r
gitlab-ci: Ease forks pipeline workflow\r
which itself is based on:\r
https://gitlab.com/thuth/qemu.git tags/pull-request-2021-05-03\r
\r
Philippe Mathieu-Daudé (6):\r
  gitlab-ci: Extract &environment_variables template\r
  gitlab-ci: Adapt JOBS variable for FreeBSD runners\r
  gitlab-ci: Run GNU make via the $MAKE variable\r
  gitlab-ci: Add ccache in $PATH and display statistics\r
  gitlab-ci: Simplify before/after script for Avocado based jobs\r
  gitlab-ci: Add FreeBSD jobs\r
\r
 .gitlab-ci.d/buildtest-freebsd.yml  | 59 +++++++++++++++++++++++++++++\r
 .gitlab-ci.d/buildtest-template.yml | 38 +++++++++++++++----\r
 .gitlab-ci.d/qemu-project.yml       |  1 +\r
 3 files changed, 91 insertions(+), 7 deletions(-)\r
 create mode 100644 .gitlab-ci.d/buildtest-freebsd.yml\r
\r
-- \r
2.26.3\r
\r


^ permalink raw reply	[flat|nested] 34+ messages in thread

* [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template
  2021-05-19 18:45 [PATCH v3 0/6] gitlab-ci: Allow using FreeBSD runners Philippe Mathieu-Daudé
@ 2021-05-19 18:45 ` Philippe Mathieu-Daudé
  2021-05-19 19:12   ` Willian Rampazzo
  2021-05-20 18:00   ` Wainer dos Santos Moschetta
  2021-05-19 18:45 ` [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners Philippe Mathieu-Daudé
                   ` (4 subsequent siblings)
  5 siblings, 2 replies; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-19 18:45 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée,
	Warner Losh

To be able to set the same environment variables to multiple jobs,
extract what we currently have as a template.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 32aaef1a213..58b01744751 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,8 +1,11 @@
+.environment_variables_template:
+  before_script:
+    - JOBS=$(expr $(nproc) + 1)
+
 .native_build_job_template:
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
-  before_script:
-    - JOBS=$(expr $(nproc) + 1)
+  extends: .environment_variables_template
   script:
     - if test -n "$LD_JOBS";
       then
-- 
2.26.3



^ permalink raw reply	[flat|nested] 34+ messages in thread

* [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners
  2021-05-19 18:45 [PATCH v3 0/6] gitlab-ci: Allow using FreeBSD runners Philippe Mathieu-Daudé
  2021-05-19 18:45 ` [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template Philippe Mathieu-Daudé
@ 2021-05-19 18:45 ` Philippe Mathieu-Daudé
  2021-05-19 19:11   ` Willian Rampazzo
                     ` (2 more replies)
  2021-05-19 18:45 ` [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable Philippe Mathieu-Daudé
                   ` (3 subsequent siblings)
  5 siblings, 3 replies; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-19 18:45 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée,
	Warner Losh

'nproc' is not available on FreeBSD:

  $ JOBS=$(expr $(nproc) + 1)
  bash: line 119: nproc: command not found
  expr: syntax error

Instead, use 'sysctl -n hw.ncpu'.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 58b01744751..fe4f18595ac 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,6 +1,16 @@
 .environment_variables_template:
   before_script:
-    - JOBS=$(expr $(nproc) + 1)
+    - if
+        test $(uname) = "FreeBSD"
+        ;
+      then
+        JOBS=$(sysctl -n hw.ncpu)
+        ;
+      else
+        JOBS=$(expr $(nproc) + 1)
+        ;
+      fi
+    - echo "=== Using $JOBS simultaneous jobs ==="
 
 .native_build_job_template:
   stage: build
-- 
2.26.3



^ permalink raw reply	[flat|nested] 34+ messages in thread

* [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable
  2021-05-19 18:45 [PATCH v3 0/6] gitlab-ci: Allow using FreeBSD runners Philippe Mathieu-Daudé
  2021-05-19 18:45 ` [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template Philippe Mathieu-Daudé
  2021-05-19 18:45 ` [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners Philippe Mathieu-Daudé
@ 2021-05-19 18:45 ` Philippe Mathieu-Daudé
  2021-05-19 19:10   ` Willian Rampazzo
                     ` (2 more replies)
  2021-05-19 18:45 ` [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics Philippe Mathieu-Daudé
                   ` (2 subsequent siblings)
  5 siblings, 3 replies; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-19 18:45 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée,
	Warner Losh

Add the $MAKE variable to call GNU make, and set it to 'gmake'
on FreeBSD to avoid:

  $ make -j"$JOBS"
  make: Unknown modifier ','
  make: "/builds/dTyar424/0/qemu/build/Makefile" line 3: Need an operator
  make: "/builds/dTyar424/0/qemu/build/Makefile" line 4: Missing dependency operator

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index fe4f18595ac..f284d7a0eec 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -5,9 +5,11 @@
         ;
       then
         JOBS=$(sysctl -n hw.ncpu)
+        MAKE=gmake
         ;
       else
         JOBS=$(expr $(nproc) + 1)
+        MAKE=make
         ;
       fi
     - echo "=== Using $JOBS simultaneous jobs ==="
@@ -33,22 +35,23 @@
       then
         ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
       fi || exit 1;
-    - make -j"$JOBS"
+    - $MAKE -j"$JOBS"
     - if test -n "$MAKE_CHECK_ARGS";
       then
-        make -j"$JOBS" $MAKE_CHECK_ARGS ;
+        $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
 
 .native_test_job_template:
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
+  extends: .environment_variables_template
   script:
     - scripts/git-submodule.sh update
         $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
     - cd build
     - find . -type f -exec touch {} +
     # Avoid recompiling by hiding ninja with NINJA=":"
-    - make NINJA=":" $MAKE_CHECK_ARGS
+    - $MAKE NINJA=":" $MAKE_CHECK_ARGS
 
 .integration_test_job_template:
   extends: .native_test_job_template
-- 
2.26.3



^ permalink raw reply	[flat|nested] 34+ messages in thread

* [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-19 18:45 [PATCH v3 0/6] gitlab-ci: Allow using FreeBSD runners Philippe Mathieu-Daudé
                   ` (2 preceding siblings ...)
  2021-05-19 18:45 ` [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable Philippe Mathieu-Daudé
@ 2021-05-19 18:45 ` Philippe Mathieu-Daudé
  2021-05-19 19:14   ` Willian Rampazzo
  2021-05-20  8:02   ` Thomas Huth
  2021-05-19 18:45 ` [PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs Philippe Mathieu-Daudé
  2021-05-19 18:45 ` [PATCH v3 6/6] gitlab-ci: Add FreeBSD jobs Philippe Mathieu-Daudé
  5 siblings, 2 replies; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-19 18:45 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée,
	Warner Losh

If a runner has ccache installed, use it and display statistics
at the end of the build.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index f284d7a0eec..a625c697d3b 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -6,13 +6,18 @@
       then
         JOBS=$(sysctl -n hw.ncpu)
         MAKE=gmake
+        PATH=/usr/local/libexec/ccache:$PATH
         ;
       else
         JOBS=$(expr $(nproc) + 1)
         MAKE=make
+        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
         ;
       fi
     - echo "=== Using $JOBS simultaneous jobs ==="
+    - if command -v ccache > /dev/null ; then ccache --zero-stats ; fi
+  after_script:
+    - if command -v ccache > /dev/null ; then ccache --show-stats ; fi
 
 .native_build_job_template:
   stage: build
-- 
2.26.3



^ permalink raw reply	[flat|nested] 34+ messages in thread

* [PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs
  2021-05-19 18:45 [PATCH v3 0/6] gitlab-ci: Allow using FreeBSD runners Philippe Mathieu-Daudé
                   ` (3 preceding siblings ...)
  2021-05-19 18:45 ` [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics Philippe Mathieu-Daudé
@ 2021-05-19 18:45 ` Philippe Mathieu-Daudé
  2021-05-19 19:42   ` Willian Rampazzo
  2021-05-20  8:04   ` Thomas Huth
  2021-05-19 18:45 ` [PATCH v3 6/6] gitlab-ci: Add FreeBSD jobs Philippe Mathieu-Daudé
  5 siblings, 2 replies; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-19 18:45 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée,
	Warner Losh

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-template.yml | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index a625c697d3b..f968fa1ad99 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -74,7 +74,7 @@
       - build/tests/results/latest/test-results
     reports:
       junit: build/tests/results/latest/results.xml
-  before_script:
+  script:
     - mkdir -p ~/.config/avocado
     - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
     - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
@@ -85,6 +85,9 @@
         du -chs ${CI_PROJECT_DIR}/avocado-cache ;
       fi
     - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
-  after_script:
     - cd build
+    - find . -type f -exec touch {} +
+    # Avoid recompiling by hiding ninja with NINJA=":"
+    - $MAKE NINJA=":" $MAKE_CHECK_ARGS
+  after_script:
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
-- 
2.26.3



^ permalink raw reply	[flat|nested] 34+ messages in thread

* [PATCH v3 6/6] gitlab-ci: Add FreeBSD jobs
  2021-05-19 18:45 [PATCH v3 0/6] gitlab-ci: Allow using FreeBSD runners Philippe Mathieu-Daudé
                   ` (4 preceding siblings ...)
  2021-05-19 18:45 ` [PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs Philippe Mathieu-Daudé
@ 2021-05-19 18:45 ` Philippe Mathieu-Daudé
  2021-05-19 19:43   ` Willian Rampazzo
  5 siblings, 1 reply; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-19 18:45 UTC (permalink / raw)
  To: qemu-devel
  Cc: Thomas Huth, Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée,
	Warner Losh

Add system/user emulation jobs on FreeBSD host.

To build these jobs, you need to add a FreeBSD runner and
add 'freebsd' to the QEMU_CUSTOM_RUNNER variable in your
GitLab project.

Reviewed by: Warner Losh <imp@bsdimp.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitlab-ci.d/buildtest-freebsd.yml | 59 ++++++++++++++++++++++++++++++
 .gitlab-ci.d/qemu-project.yml      |  1 +
 2 files changed, 60 insertions(+)
 create mode 100644 .gitlab-ci.d/buildtest-freebsd.yml

diff --git a/.gitlab-ci.d/buildtest-freebsd.yml b/.gitlab-ci.d/buildtest-freebsd.yml
new file mode 100644
index 00000000000..bfaeff89330
--- /dev/null
+++ b/.gitlab-ci.d/buildtest-freebsd.yml
@@ -0,0 +1,59 @@
+include:
+  - local: '/.gitlab-ci.d/buildtest-template.yml'
+
+.runner_freebsd_template:
+  rules:
+    # To enable this job template, add 'freebsd' to the QEMU_CUSTOM_RUNNER
+    # variable in your project UI via Settings -> CI/CD -> Variables
+    # https://docs.gitlab.com/ee/ci/variables/README.html#project-cicd-variables
+    - if: $QEMU_CUSTOM_RUNNER =~ /freebsd/
+      when: always
+    - when: never
+  tags:
+    - freebsd
+
+build-user-freebsd:
+  extends:
+    - .runner_freebsd_template
+    - .native_build_job_template
+  image:
+  variables:
+    MAKE_CHECK_ARGS: check-build
+    CONFIGURE_ARGS: --disable-system --python=/usr/local/bin/python3.7
+
+build-system-freebsd:
+  extends:
+    - .runner_freebsd_template
+    - .native_build_job_template
+  image:
+  variables:
+    TARGETS: aarch64-softmmu avr-softmmu hppa-softmmu ppc64-softmmu
+      riscv64-softmmu s390x-softmmu x86_64-softmmu
+    MAKE_CHECK_ARGS: check-build
+    CONFIGURE_ARGS: --enable-trace-backends=log,simple,syslog
+      --python=/usr/local/bin/python3.7
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - .git-submodule-status
+      - build
+
+check-system-freebsd:
+  extends:
+    - .runner_freebsd_template
+    - .native_test_job_template
+  needs:
+    - job: build-system-freebsd
+      artifacts: true
+  variables:
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-freebsd:
+  extends:
+    - .runner_freebsd_template
+    - .integration_test_job_template
+  needs:
+    - job: build-system-freebsd
+      artifacts: true
+  variables:
+    MAKE_CHECK_ARGS: check-acceptance
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 64cb2ba1da5..5dcf1d34c5b 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -8,4 +8,5 @@ include:
   - local: '/.gitlab-ci.d/containers.yml'
   - local: '/.gitlab-ci.d/crossbuilds.yml'
   - local: '/.gitlab-ci.d/buildtest.yml'
+  - local: '/.gitlab-ci.d/buildtest-freebsd.yml'
   - local: '/.gitlab-ci.d/static_checks.yml'
-- 
2.26.3



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable
  2021-05-19 18:45 ` [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable Philippe Mathieu-Daudé
@ 2021-05-19 19:10   ` Willian Rampazzo
  2021-05-20  7:56   ` Thomas Huth
  2021-05-20 18:26   ` Wainer dos Santos Moschetta
  2 siblings, 0 replies; 34+ messages in thread
From: Willian Rampazzo @ 2021-05-19 19:10 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Alex Bennée, Warner Losh

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> Add the $MAKE variable to call GNU make, and set it to 'gmake'
> on FreeBSD to avoid:
>
>   $ make -j"$JOBS"
>   make: Unknown modifier ','
>   make: "/builds/dTyar424/0/qemu/build/Makefile" line 3: Need an operator
>   make: "/builds/dTyar424/0/qemu/build/Makefile" line 4: Missing dependency operator
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners
  2021-05-19 18:45 ` [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners Philippe Mathieu-Daudé
@ 2021-05-19 19:11   ` Willian Rampazzo
  2021-05-20  5:50   ` Thomas Huth
  2021-05-20 18:18   ` Wainer dos Santos Moschetta
  2 siblings, 0 replies; 34+ messages in thread
From: Willian Rampazzo @ 2021-05-19 19:11 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Alex Bennée, Warner Losh

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> 'nproc' is not available on FreeBSD:
>
>   $ JOBS=$(expr $(nproc) + 1)
>   bash: line 119: nproc: command not found
>   expr: syntax error
>
> Instead, use 'sysctl -n hw.ncpu'.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template
  2021-05-19 18:45 ` [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template Philippe Mathieu-Daudé
@ 2021-05-19 19:12   ` Willian Rampazzo
  2021-05-20 18:00   ` Wainer dos Santos Moschetta
  1 sibling, 0 replies; 34+ messages in thread
From: Willian Rampazzo @ 2021-05-19 19:12 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Alex Bennée, Warner Losh

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> To be able to set the same environment variables to multiple jobs,
> extract what we currently have as a template.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-19 18:45 ` [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics Philippe Mathieu-Daudé
@ 2021-05-19 19:14   ` Willian Rampazzo
  2021-05-20  8:02   ` Thomas Huth
  1 sibling, 0 replies; 34+ messages in thread
From: Willian Rampazzo @ 2021-05-19 19:14 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Alex Bennée, Warner Losh

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> If a runner has ccache installed, use it and display statistics
> at the end of the build.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 5 +++++
>  1 file changed, 5 insertions(+)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs
  2021-05-19 18:45 ` [PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs Philippe Mathieu-Daudé
@ 2021-05-19 19:42   ` Willian Rampazzo
  2021-05-20  8:04   ` Thomas Huth
  1 sibling, 0 replies; 34+ messages in thread
From: Willian Rampazzo @ 2021-05-19 19:42 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Alex Bennée, Warner Losh

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 6/6] gitlab-ci: Add FreeBSD jobs
  2021-05-19 18:45 ` [PATCH v3 6/6] gitlab-ci: Add FreeBSD jobs Philippe Mathieu-Daudé
@ 2021-05-19 19:43   ` Willian Rampazzo
  0 siblings, 0 replies; 34+ messages in thread
From: Willian Rampazzo @ 2021-05-19 19:43 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Alex Bennée, Warner Losh

On Wed, May 19, 2021 at 3:46 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> Add system/user emulation jobs on FreeBSD host.
>
> To build these jobs, you need to add a FreeBSD runner and
> add 'freebsd' to the QEMU_CUSTOM_RUNNER variable in your
> GitLab project.
>
> Reviewed by: Warner Losh <imp@bsdimp.com>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .gitlab-ci.d/buildtest-freebsd.yml | 59 ++++++++++++++++++++++++++++++
>  .gitlab-ci.d/qemu-project.yml      |  1 +
>  2 files changed, 60 insertions(+)
>  create mode 100644 .gitlab-ci.d/buildtest-freebsd.yml
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners
  2021-05-19 18:45 ` [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners Philippe Mathieu-Daudé
  2021-05-19 19:11   ` Willian Rampazzo
@ 2021-05-20  5:50   ` Thomas Huth
  2021-05-20 18:18   ` Wainer dos Santos Moschetta
  2 siblings, 0 replies; 34+ messages in thread
From: Thomas Huth @ 2021-05-20  5:50 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Kyle Evans, Wainer dos Santos Moschetta, Willian Rampazzo,
	Alex Bennée, Warner Losh

On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> 'nproc' is not available on FreeBSD:
> 
>    $ JOBS=$(expr $(nproc) + 1)
>    bash: line 119: nproc: command not found
>    expr: syntax error
> 
> Instead, use 'sysctl -n hw.ncpu'.

Could we maybe use "getconf _NPROCESSORS_ONLN" instead? ... that might work 
on both, Linux and FreeBSD...?

  Thomas



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable
  2021-05-19 18:45 ` [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable Philippe Mathieu-Daudé
  2021-05-19 19:10   ` Willian Rampazzo
@ 2021-05-20  7:56   ` Thomas Huth
  2021-05-20 18:26   ` Wainer dos Santos Moschetta
  2 siblings, 0 replies; 34+ messages in thread
From: Thomas Huth @ 2021-05-20  7:56 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Willian Rampazzo, Kyle Evans, Alex Bennée, Warner Losh,
	Wainer dos Santos Moschetta

On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> Add the $MAKE variable to call GNU make, and set it to 'gmake'
> on FreeBSD to avoid:
> 
>    $ make -j"$JOBS"
>    make: Unknown modifier ','
>    make: "/builds/dTyar424/0/qemu/build/Makefile" line 3: Need an operator
>    make: "/builds/dTyar424/0/qemu/build/Makefile" line 4: Missing dependency operator
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 9 ++++++---
>   1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index fe4f18595ac..f284d7a0eec 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -5,9 +5,11 @@
>           ;
>         then
>           JOBS=$(sysctl -n hw.ncpu)
> +        MAKE=gmake
>           ;
>         else
>           JOBS=$(expr $(nproc) + 1)
> +        MAKE=make

Maybe we could use "gmake" on Linux, too, so we do not have to use the 
indirection with a variable here? Or are there Linux distros where the 
"gmake" link is not available?

  Thomas



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-19 18:45 ` [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics Philippe Mathieu-Daudé
  2021-05-19 19:14   ` Willian Rampazzo
@ 2021-05-20  8:02   ` Thomas Huth
  2021-05-20 11:27     ` Philippe Mathieu-Daudé
  1 sibling, 1 reply; 34+ messages in thread
From: Thomas Huth @ 2021-05-20  8:02 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Willian Rampazzo, Kyle Evans, Alex Bennée, Warner Losh,
	Wainer dos Santos Moschetta

On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> If a runner has ccache installed, use it and display statistics
> at the end of the build.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 5 +++++
>   1 file changed, 5 insertions(+)
> 
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index f284d7a0eec..a625c697d3b 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -6,13 +6,18 @@
>         then
>           JOBS=$(sysctl -n hw.ncpu)
>           MAKE=gmake
> +        PATH=/usr/local/libexec/ccache:$PATH
>           ;
>         else
>           JOBS=$(expr $(nproc) + 1)
>           MAKE=make
> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH

That does not make sense for the shared runners yet. We first need something 
to enable the caching there - see my series "Use ccache in the gitlab-CI" 
from April (which is currently stalled unfortunately).

  Thomas



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs
  2021-05-19 18:45 ` [PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs Philippe Mathieu-Daudé
  2021-05-19 19:42   ` Willian Rampazzo
@ 2021-05-20  8:04   ` Thomas Huth
  1 sibling, 0 replies; 34+ messages in thread
From: Thomas Huth @ 2021-05-20  8:04 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Willian Rampazzo, Kyle Evans, Alex Bennée, Warner Losh,
	Wainer dos Santos Moschetta


Missing patch description. Could you please elaborate why this change is ok?

  Thanks,
   Thomas


On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index a625c697d3b..f968fa1ad99 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -74,7 +74,7 @@
>         - build/tests/results/latest/test-results
>       reports:
>         junit: build/tests/results/latest/results.xml
> -  before_script:
> +  script:
>       - mkdir -p ~/.config/avocado
>       - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
>       - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
> @@ -85,6 +85,9 @@
>           du -chs ${CI_PROJECT_DIR}/avocado-cache ;
>         fi
>       - export AVOCADO_ALLOW_UNTRUSTED_CODE=1
> -  after_script:
>       - cd build
> +    - find . -type f -exec touch {} +
> +    # Avoid recompiling by hiding ninja with NINJA=":"
> +    - $MAKE NINJA=":" $MAKE_CHECK_ARGS
> +  after_script:
>       - du -chs ${CI_PROJECT_DIR}/avocado-cache
> 



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-20  8:02   ` Thomas Huth
@ 2021-05-20 11:27     ` Philippe Mathieu-Daudé
  2021-05-21 10:48       ` Thomas Huth
  0 siblings, 1 reply; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-20 11:27 UTC (permalink / raw)
  To: Thomas Huth, qemu-devel, Alex Bennée
  Cc: Daniel P . Berrange, Kyle Evans, Wainer dos Santos Moschetta,
	Willian Rampazzo, Stefan Hajnoczi, Warner Losh

+Stefan/Daniel

On 5/20/21 10:02 AM, Thomas Huth wrote:
> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>> If a runner has ccache installed, use it and display statistics
>> at the end of the build.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>   .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>   1 file changed, 5 insertions(+)
>>
>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>> b/.gitlab-ci.d/buildtest-template.yml
>> index f284d7a0eec..a625c697d3b 100644
>> --- a/.gitlab-ci.d/buildtest-template.yml
>> +++ b/.gitlab-ci.d/buildtest-template.yml
>> @@ -6,13 +6,18 @@
>>         then
>>           JOBS=$(sysctl -n hw.ncpu)
>>           MAKE=gmake
>> +        PATH=/usr/local/libexec/ccache:$PATH
>>           ;
>>         else
>>           JOBS=$(expr $(nproc) + 1)
>>           MAKE=make
>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> 
> That does not make sense for the shared runners yet. We first need
> something to enable the caching there - see my series "Use ccache in the
> gitlab-CI" from April (which is currently stalled unfortunately).

TL;DR: I don't think we should restrict our templates to shared runners.

I don't think mainstream interest for shared runners use should limit
forks... This is a great improvement for contributors having to use
private/custom runner, see this example:

https://gitlab.com/philmd/qemu/-/jobs/1255197705
cache hit (direct)                  1529
cache hit (preprocessed)             736
cache miss                           855
cache hit rate                     72.60 %

Having been caught 2 times for testing as hard as crypto miner, I
can not rely on shared runners for my workflow.

Beside, the recent announcement:
https://about.gitlab.com/blog/2021/05/17/prevent-crypto-mining-abuse/
"starting May 17, 2021, GitLab will require new free users to provide a
valid credit or debit card number in order to use shared runners on
GitLab.com."

On this report only 17/142 countries are listed as having more than
50% of their population with a credit card:
https://www.theglobaleconomy.com/rankings/people_with_credit_cards/
and 60 with a debit card:
https://www.theglobaleconomy.com/rankings/people_with_debit_cards/

We have some contributors in countries with low percentage. I don't
think we should start a poll to verify, but I do think relying on
shared runners without proposing free alternative is discriminatory
to a part of our community.

I am very open to alternatives to improve the CI/testing situation.

Regards,

Phil.


^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template
  2021-05-19 18:45 ` [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template Philippe Mathieu-Daudé
  2021-05-19 19:12   ` Willian Rampazzo
@ 2021-05-20 18:00   ` Wainer dos Santos Moschetta
  1 sibling, 0 replies; 34+ messages in thread
From: Wainer dos Santos Moschetta @ 2021-05-20 18:00 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Willian Rampazzo, Kyle Evans, Thomas Huth, Alex Bennée, Warner Losh

Hi,

On 5/19/21 3:45 PM, Philippe Mathieu-Daudé wrote:
> To be able to set the same environment variables to multiple jobs,
> extract what we currently have as a template.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index 32aaef1a213..58b01744751 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -1,8 +1,11 @@
> +.environment_variables_template:
> +  before_script:
> +    - JOBS=$(expr $(nproc) + 1)
> +

I wish we could wrap environment variables definitions in `variables` 
(https://docs.gitlab.com/ee/ci/yaml/README.html#variables) instead of 
`before_scripts`, but it wouldn't evaluate the shell expressions.

Unless I am wrong, if the job definition has its own `before_script` 
then including `.environment_variables_template` won't take effect. 
Shouldn't it be documented?

- Wainer

>   .native_build_job_template:
>     stage: build
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> -  before_script:
> -    - JOBS=$(expr $(nproc) + 1)
> +  extends: .environment_variables_template
>     script:
>       - if test -n "$LD_JOBS";
>         then



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners
  2021-05-19 18:45 ` [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners Philippe Mathieu-Daudé
  2021-05-19 19:11   ` Willian Rampazzo
  2021-05-20  5:50   ` Thomas Huth
@ 2021-05-20 18:18   ` Wainer dos Santos Moschetta
  2021-05-20 20:02     ` Warner Losh
  2 siblings, 1 reply; 34+ messages in thread
From: Wainer dos Santos Moschetta @ 2021-05-20 18:18 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Willian Rampazzo, Kyle Evans, Thomas Huth, Alex Bennée, Warner Losh

Hi,

On 5/19/21 3:45 PM, Philippe Mathieu-Daudé wrote:
> 'nproc' is not available on FreeBSD:
>
>    $ JOBS=$(expr $(nproc) + 1)
>    bash: line 119: nproc: command not found
>    expr: syntax error
>
> Instead, use 'sysctl -n hw.ncpu'.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 12 +++++++++++-
>   1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index 58b01744751..fe4f18595ac 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -1,6 +1,16 @@
>   .environment_variables_template:
>     before_script:
> -    - JOBS=$(expr $(nproc) + 1)
> +    - if
> +        test $(uname) = "FreeBSD"
> +        ;
> +      then
> +        JOBS=$(sysctl -n hw.ncpu)
> +        ;
> +      else
> +        JOBS=$(expr $(nproc) + 1)
> +        ;
> +      fi
> +    - echo "=== Using $JOBS simultaneous jobs ==="

It has grown to beyond variables definitions. I suggest to rename it to 
something like `.before_script_template` (pure creativity, see?).

- Wainer

>   
>   .native_build_job_template:
>     stage: build



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable
  2021-05-19 18:45 ` [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable Philippe Mathieu-Daudé
  2021-05-19 19:10   ` Willian Rampazzo
  2021-05-20  7:56   ` Thomas Huth
@ 2021-05-20 18:26   ` Wainer dos Santos Moschetta
  2 siblings, 0 replies; 34+ messages in thread
From: Wainer dos Santos Moschetta @ 2021-05-20 18:26 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel
  Cc: Willian Rampazzo, Kyle Evans, Thomas Huth, Alex Bennée, Warner Losh


On 5/19/21 3:45 PM, Philippe Mathieu-Daudé wrote:
> Add the $MAKE variable to call GNU make, and set it to 'gmake'
> on FreeBSD to avoid:
>
>    $ make -j"$JOBS"
>    make: Unknown modifier ','
>    make: "/builds/dTyar424/0/qemu/build/Makefile" line 3: Need an operator
>    make: "/builds/dTyar424/0/qemu/build/Makefile" line 4: Missing dependency operator
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 9 ++++++---
>   1 file changed, 6 insertions(+), 3 deletions(-)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index fe4f18595ac..f284d7a0eec 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -5,9 +5,11 @@
>           ;
>         then
>           JOBS=$(sysctl -n hw.ncpu)
> +        MAKE=gmake
>           ;
>         else
>           JOBS=$(expr $(nproc) + 1)
> +        MAKE=make
>           ;
>         fi
>       - echo "=== Using $JOBS simultaneous jobs ==="
> @@ -33,22 +35,23 @@
>         then
>           ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
>         fi || exit 1;
> -    - make -j"$JOBS"
> +    - $MAKE -j"$JOBS"
>       - if test -n "$MAKE_CHECK_ARGS";
>         then
> -        make -j"$JOBS" $MAKE_CHECK_ARGS ;
> +        $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
>         fi
>   
>   .native_test_job_template:
>     stage: test
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  extends: .environment_variables_template
>     script:
>       - scripts/git-submodule.sh update
>           $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak)
>       - cd build
>       - find . -type f -exec touch {} +
>       # Avoid recompiling by hiding ninja with NINJA=":"
> -    - make NINJA=":" $MAKE_CHECK_ARGS
> +    - $MAKE NINJA=":" $MAKE_CHECK_ARGS
>   
>   .integration_test_job_template:
>     extends: .native_test_job_template



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners
  2021-05-20 18:18   ` Wainer dos Santos Moschetta
@ 2021-05-20 20:02     ` Warner Losh
  0 siblings, 0 replies; 34+ messages in thread
From: Warner Losh @ 2021-05-20 20:02 UTC (permalink / raw)
  To: Wainer dos Santos Moschetta
  Cc: Thomas Huth, Kyle Evans, Philippe Mathieu-Daudé,
	QEMU Developers, Willian Rampazzo, Alex Bennée

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

On Thu, May 20, 2021 at 12:18 PM Wainer dos Santos Moschetta <
wainersm@redhat.com> wrote:

> Hi,
>
> On 5/19/21 3:45 PM, Philippe Mathieu-Daudé wrote:
> > 'nproc' is not available on FreeBSD:
> >
> >    $ JOBS=$(expr $(nproc) + 1)
> >    bash: line 119: nproc: command not found
> >    expr: syntax error
> >
> > Instead, use 'sysctl -n hw.ncpu'.
> >
> > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > ---
> >   .gitlab-ci.d/buildtest-template.yml | 12 +++++++++++-
> >   1 file changed, 11 insertions(+), 1 deletion(-)
> >
> > diff --git a/.gitlab-ci.d/buildtest-template.yml
> b/.gitlab-ci.d/buildtest-template.yml
> > index 58b01744751..fe4f18595ac 100644
> > --- a/.gitlab-ci.d/buildtest-template.yml
> > +++ b/.gitlab-ci.d/buildtest-template.yml
> > @@ -1,6 +1,16 @@
> >   .environment_variables_template:
> >     before_script:
> > -    - JOBS=$(expr $(nproc) + 1)
> > +    - if
> > +        test $(uname) = "FreeBSD"
> > +        ;
> > +      then
> > +        JOBS=$(sysctl -n hw.ncpu)
> > +        ;
> > +      else
> > +        JOBS=$(expr $(nproc) + 1)
> > +        ;
> > +      fi
> > +    - echo "=== Using $JOBS simultaneous jobs ==="
>
> It has grown to beyond variables definitions. I suggest to rename it to
> something like `.before_script_template` (pure creativity, see?).
>

Also, would it help if FreeBSD were to add nproc in the future?

Warner

[-- Attachment #2: Type: text/html, Size: 2095 bytes --]

^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-20 11:27     ` Philippe Mathieu-Daudé
@ 2021-05-21 10:48       ` Thomas Huth
  2021-05-21 10:50         ` Daniel P. Berrangé
  0 siblings, 1 reply; 34+ messages in thread
From: Thomas Huth @ 2021-05-21 10:48 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé, qemu-devel, Alex Bennée
  Cc: Daniel P . Berrange, Kyle Evans, Wainer dos Santos Moschetta,
	Willian Rampazzo, Stefan Hajnoczi, Warner Losh

On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> +Stefan/Daniel
> 
> On 5/20/21 10:02 AM, Thomas Huth wrote:
>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>> If a runner has ccache installed, use it and display statistics
>>> at the end of the build.
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> ---
>>>    .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>    1 file changed, 5 insertions(+)
>>>
>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>> b/.gitlab-ci.d/buildtest-template.yml
>>> index f284d7a0eec..a625c697d3b 100644
>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>> @@ -6,13 +6,18 @@
>>>          then
>>>            JOBS=$(sysctl -n hw.ncpu)
>>>            MAKE=gmake
>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>            ;
>>>          else
>>>            JOBS=$(expr $(nproc) + 1)
>>>            MAKE=make
>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>
>> That does not make sense for the shared runners yet. We first need
>> something to enable the caching there - see my series "Use ccache in the
>> gitlab-CI" from April (which is currently stalled unfortunately).
> 
> TL;DR: I don't think we should restrict our templates to shared runners.

I'm certainly not voting for restricting ourselves to only use shared 
runners here - but my concern is that this actually *slows* down the shared 
runners even more! (sorry, I should have elaborated on that in my previous 
mail already)

When I was experimenting with ccache in the shared runners, I saw that the 
jobs are running even slower with ccache enabled as long as the cache is not 
populated yet. You only get a speedup afterwards. So if you add this now 
without also adding the possibility to store the cache persistently, the 
shared runners will try to populate the cache each time just to throw away 
the results afterwards again. Thus all the shared runners only get slower 
without any real benefit here.

Thus we either need to get ccache working properly for the shared runners 
first, or you have to think of a different way of enabling ccache for the 
non-shared runners, so that it does not affect the shared runners negatively.

  Thomas



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-21 10:48       ` Thomas Huth
@ 2021-05-21 10:50         ` Daniel P. Berrangé
  2021-05-21 11:02           ` Thomas Huth
  0 siblings, 1 reply; 34+ messages in thread
From: Daniel P. Berrangé @ 2021-05-21 10:50 UTC (permalink / raw)
  To: Thomas Huth
  Cc: Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, qemu-devel, Willian Rampazzo,
	Stefan Hajnoczi, Alex Bennée, Warner Losh

On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> > +Stefan/Daniel
> > 
> > On 5/20/21 10:02 AM, Thomas Huth wrote:
> > > On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> > > > If a runner has ccache installed, use it and display statistics
> > > > at the end of the build.
> > > > 
> > > > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > > > ---
> > > >    .gitlab-ci.d/buildtest-template.yml | 5 +++++
> > > >    1 file changed, 5 insertions(+)
> > > > 
> > > > diff --git a/.gitlab-ci.d/buildtest-template.yml
> > > > b/.gitlab-ci.d/buildtest-template.yml
> > > > index f284d7a0eec..a625c697d3b 100644
> > > > --- a/.gitlab-ci.d/buildtest-template.yml
> > > > +++ b/.gitlab-ci.d/buildtest-template.yml
> > > > @@ -6,13 +6,18 @@
> > > >          then
> > > >            JOBS=$(sysctl -n hw.ncpu)
> > > >            MAKE=gmake
> > > > +        PATH=/usr/local/libexec/ccache:$PATH
> > > >            ;
> > > >          else
> > > >            JOBS=$(expr $(nproc) + 1)
> > > >            MAKE=make
> > > > +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> > > 
> > > That does not make sense for the shared runners yet. We first need
> > > something to enable the caching there - see my series "Use ccache in the
> > > gitlab-CI" from April (which is currently stalled unfortunately).
> > 
> > TL;DR: I don't think we should restrict our templates to shared runners.
> 
> I'm certainly not voting for restricting ourselves to only use shared
> runners here - but my concern is that this actually *slows* down the shared
> runners even more! (sorry, I should have elaborated on that in my previous
> mail already)
> 
> When I was experimenting with ccache in the shared runners, I saw that the
> jobs are running even slower with ccache enabled as long as the cache is not
> populated yet. You only get a speedup afterwards. So if you add this now
> without also adding the possibility to store the cache persistently, the
> shared runners will try to populate the cache each time just to throw away
> the results afterwards again. Thus all the shared runners only get slower
> without any real benefit here.
> 
> Thus we either need to get ccache working properly for the shared runners
> first, or you have to think of a different way of enabling ccache for the
> non-shared runners, so that it does not affect the shared runners
> negatively.

Is there anything functional holding up your previous full cccache support
series from last month ? Or is it just lack of reviews ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-21 10:50         ` Daniel P. Berrangé
@ 2021-05-21 11:02           ` Thomas Huth
  2021-05-21 11:53             ` Daniel P. Berrangé
  2021-05-21 12:36             ` Willian Rampazzo
  0 siblings, 2 replies; 34+ messages in thread
From: Thomas Huth @ 2021-05-21 11:02 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, qemu-devel, Willian Rampazzo,
	Stefan Hajnoczi, Alex Bennée, Warner Losh

On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
>>> +Stefan/Daniel
>>>
>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>>>> If a runner has ccache installed, use it and display statistics
>>>>> at the end of the build.
>>>>>
>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>> ---
>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>>>     1 file changed, 5 insertions(+)
>>>>>
>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>>>> b/.gitlab-ci.d/buildtest-template.yml
>>>>> index f284d7a0eec..a625c697d3b 100644
>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>>>> @@ -6,13 +6,18 @@
>>>>>           then
>>>>>             JOBS=$(sysctl -n hw.ncpu)
>>>>>             MAKE=gmake
>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>>>             ;
>>>>>           else
>>>>>             JOBS=$(expr $(nproc) + 1)
>>>>>             MAKE=make
>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>>>
>>>> That does not make sense for the shared runners yet. We first need
>>>> something to enable the caching there - see my series "Use ccache in the
>>>> gitlab-CI" from April (which is currently stalled unfortunately).
>>>
>>> TL;DR: I don't think we should restrict our templates to shared runners.
>>
>> I'm certainly not voting for restricting ourselves to only use shared
>> runners here - but my concern is that this actually *slows* down the shared
>> runners even more! (sorry, I should have elaborated on that in my previous
>> mail already)
>>
>> When I was experimenting with ccache in the shared runners, I saw that the
>> jobs are running even slower with ccache enabled as long as the cache is not
>> populated yet. You only get a speedup afterwards. So if you add this now
>> without also adding the possibility to store the cache persistently, the
>> shared runners will try to populate the cache each time just to throw away
>> the results afterwards again. Thus all the shared runners only get slower
>> without any real benefit here.
>>
>> Thus we either need to get ccache working properly for the shared runners
>> first, or you have to think of a different way of enabling ccache for the
>> non-shared runners, so that it does not affect the shared runners
>> negatively.
> 
> Is there anything functional holding up your previous full cccache support
> series from last month ? Or is it just lack of reviews ?

It's basically the problems mentioned in the cover letter and Stefan's 
comment here:

  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html

The series needs more love and more testing, if it's feasible with the 
gitlab-CI architecture at all to use ccache in a good way ... if we don't 
get a real speedup by the patches, it's certainly not worth the effort to 
integrate this...

If someone wants to have a try to improve the patch series, your help is 
certainly welcome - since at least I personally lack the time and motivation 
to improve this any further.

  Thomas



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-21 11:02           ` Thomas Huth
@ 2021-05-21 11:53             ` Daniel P. Berrangé
  2021-05-21 12:27               ` Philippe Mathieu-Daudé
  2021-05-21 12:36             ` Willian Rampazzo
  1 sibling, 1 reply; 34+ messages in thread
From: Daniel P. Berrangé @ 2021-05-21 11:53 UTC (permalink / raw)
  To: Thomas Huth
  Cc: Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, qemu-devel, Willian Rampazzo,
	Stefan Hajnoczi, Alex Bennée, Warner Losh

On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> > On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> > > On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> > > > +Stefan/Daniel
> > > > 
> > > > On 5/20/21 10:02 AM, Thomas Huth wrote:
> > > > > On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> > > > > > If a runner has ccache installed, use it and display statistics
> > > > > > at the end of the build.
> > > > > > 
> > > > > > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > > > > > ---
> > > > > >     .gitlab-ci.d/buildtest-template.yml | 5 +++++
> > > > > >     1 file changed, 5 insertions(+)
> > > > > > 
> > > > > > diff --git a/.gitlab-ci.d/buildtest-template.yml
> > > > > > b/.gitlab-ci.d/buildtest-template.yml
> > > > > > index f284d7a0eec..a625c697d3b 100644
> > > > > > --- a/.gitlab-ci.d/buildtest-template.yml
> > > > > > +++ b/.gitlab-ci.d/buildtest-template.yml
> > > > > > @@ -6,13 +6,18 @@
> > > > > >           then
> > > > > >             JOBS=$(sysctl -n hw.ncpu)
> > > > > >             MAKE=gmake
> > > > > > +        PATH=/usr/local/libexec/ccache:$PATH
> > > > > >             ;
> > > > > >           else
> > > > > >             JOBS=$(expr $(nproc) + 1)
> > > > > >             MAKE=make
> > > > > > +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> > > > > 
> > > > > That does not make sense for the shared runners yet. We first need
> > > > > something to enable the caching there - see my series "Use ccache in the
> > > > > gitlab-CI" from April (which is currently stalled unfortunately).
> > > > 
> > > > TL;DR: I don't think we should restrict our templates to shared runners.
> > > 
> > > I'm certainly not voting for restricting ourselves to only use shared
> > > runners here - but my concern is that this actually *slows* down the shared
> > > runners even more! (sorry, I should have elaborated on that in my previous
> > > mail already)
> > > 
> > > When I was experimenting with ccache in the shared runners, I saw that the
> > > jobs are running even slower with ccache enabled as long as the cache is not
> > > populated yet. You only get a speedup afterwards. So if you add this now
> > > without also adding the possibility to store the cache persistently, the
> > > shared runners will try to populate the cache each time just to throw away
> > > the results afterwards again. Thus all the shared runners only get slower
> > > without any real benefit here.
> > > 
> > > Thus we either need to get ccache working properly for the shared runners
> > > first, or you have to think of a different way of enabling ccache for the
> > > non-shared runners, so that it does not affect the shared runners
> > > negatively.
> > 
> > Is there anything functional holding up your previous full cccache support
> > series from last month ? Or is it just lack of reviews ?
> 
> It's basically the problems mentioned in the cover letter and Stefan's
> comment here:
> 
>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html

I'm not sure I understand why Stefan thinks gitlab's caching doesn't
benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
sped up our builds significantly.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-21 11:53             ` Daniel P. Berrangé
@ 2021-05-21 12:27               ` Philippe Mathieu-Daudé
  2021-05-21 12:52                 ` Daniel P. Berrangé
  0 siblings, 1 reply; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-21 12:27 UTC (permalink / raw)
  To: Daniel P. Berrangé, Thomas Huth
  Cc: Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Stefan Hajnoczi, Alex Bennée, Warner Losh

On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
> On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
>> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
>>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
>>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
>>>>> +Stefan/Daniel
>>>>>
>>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
>>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>>>>>> If a runner has ccache installed, use it and display statistics
>>>>>>> at the end of the build.
>>>>>>>
>>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>>>> ---
>>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>>>>>     1 file changed, 5 insertions(+)
>>>>>>>
>>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>>>>>> b/.gitlab-ci.d/buildtest-template.yml
>>>>>>> index f284d7a0eec..a625c697d3b 100644
>>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>>>>>> @@ -6,13 +6,18 @@
>>>>>>>           then
>>>>>>>             JOBS=$(sysctl -n hw.ncpu)
>>>>>>>             MAKE=gmake
>>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>>>>>             ;
>>>>>>>           else
>>>>>>>             JOBS=$(expr $(nproc) + 1)
>>>>>>>             MAKE=make
>>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>>>>>
>>>>>> That does not make sense for the shared runners yet. We first need
>>>>>> something to enable the caching there - see my series "Use ccache in the
>>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
>>>>>
>>>>> TL;DR: I don't think we should restrict our templates to shared runners.
>>>>
>>>> I'm certainly not voting for restricting ourselves to only use shared
>>>> runners here - but my concern is that this actually *slows* down the shared
>>>> runners even more! (sorry, I should have elaborated on that in my previous
>>>> mail already)
>>>>
>>>> When I was experimenting with ccache in the shared runners, I saw that the
>>>> jobs are running even slower with ccache enabled as long as the cache is not
>>>> populated yet. You only get a speedup afterwards. So if you add this now
>>>> without also adding the possibility to store the cache persistently, the
>>>> shared runners will try to populate the cache each time just to throw away
>>>> the results afterwards again. Thus all the shared runners only get slower
>>>> without any real benefit here.
>>>>
>>>> Thus we either need to get ccache working properly for the shared runners
>>>> first, or you have to think of a different way of enabling ccache for the
>>>> non-shared runners, so that it does not affect the shared runners
>>>> negatively.
>>>
>>> Is there anything functional holding up your previous full cccache support
>>> series from last month ? Or is it just lack of reviews ?
>>
>> It's basically the problems mentioned in the cover letter and Stefan's
>> comment here:
>>
>>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
> 
> I'm not sure I understand why Stefan thinks gitlab's caching doesn't
> benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
> sped up our builds significantly.

I think Stefan is referring to a comment I made, when using both
shared runners and dedicated runners (what I'm currently testing)
various jobs are stuck transferring artifacts/cache {FROM, TO}
{shared, dedicated} runners at the same time, which is sub-optimal
because it saturate the dedicated runner network link.

If we want to use pool of runners to restrict transfer between
runners from the same physical pool, then it because a maintenance
headache.


^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-21 11:02           ` Thomas Huth
  2021-05-21 11:53             ` Daniel P. Berrangé
@ 2021-05-21 12:36             ` Willian Rampazzo
  2021-05-21 13:02               ` Thomas Huth
  1 sibling, 1 reply; 34+ messages in thread
From: Willian Rampazzo @ 2021-05-21 12:36 UTC (permalink / raw)
  To: Thomas Huth
  Cc: Daniel P. Berrangé, Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, qemu-devel, Stefan Hajnoczi,
	Alex Bennée, Warner Losh

On Fri, May 21, 2021 at 8:03 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> > On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> >> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> >>> +Stefan/Daniel
> >>>
> >>> On 5/20/21 10:02 AM, Thomas Huth wrote:
> >>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> >>>>> If a runner has ccache installed, use it and display statistics
> >>>>> at the end of the build.
> >>>>>
> >>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >>>>> ---
> >>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
> >>>>>     1 file changed, 5 insertions(+)
> >>>>>
> >>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
> >>>>> b/.gitlab-ci.d/buildtest-template.yml
> >>>>> index f284d7a0eec..a625c697d3b 100644
> >>>>> --- a/.gitlab-ci.d/buildtest-template.yml
> >>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
> >>>>> @@ -6,13 +6,18 @@
> >>>>>           then
> >>>>>             JOBS=$(sysctl -n hw.ncpu)
> >>>>>             MAKE=gmake
> >>>>> +        PATH=/usr/local/libexec/ccache:$PATH
> >>>>>             ;
> >>>>>           else
> >>>>>             JOBS=$(expr $(nproc) + 1)
> >>>>>             MAKE=make
> >>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> >>>>
> >>>> That does not make sense for the shared runners yet. We first need
> >>>> something to enable the caching there - see my series "Use ccache in the
> >>>> gitlab-CI" from April (which is currently stalled unfortunately).
> >>>
> >>> TL;DR: I don't think we should restrict our templates to shared runners.
> >>
> >> I'm certainly not voting for restricting ourselves to only use shared
> >> runners here - but my concern is that this actually *slows* down the shared
> >> runners even more! (sorry, I should have elaborated on that in my previous
> >> mail already)
> >>
> >> When I was experimenting with ccache in the shared runners, I saw that the
> >> jobs are running even slower with ccache enabled as long as the cache is not
> >> populated yet. You only get a speedup afterwards. So if you add this now
> >> without also adding the possibility to store the cache persistently, the
> >> shared runners will try to populate the cache each time just to throw away
> >> the results afterwards again. Thus all the shared runners only get slower
> >> without any real benefit here.
> >>

I was in favor of adding the changes and benefiting custom runners
until I saw your reply. In this case, I agree we should investigate
how these changes affect shared runners.

> >> Thus we either need to get ccache working properly for the shared runners
> >> first, or you have to think of a different way of enabling ccache for the
> >> non-shared runners, so that it does not affect the shared runners
> >> negatively.
> >
> > Is there anything functional holding up your previous full cccache support
> > series from last month ? Or is it just lack of reviews ?
>
> It's basically the problems mentioned in the cover letter and Stefan's
> comment here:
>
>   https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>
> The series needs more love and more testing, if it's feasible with the
> gitlab-CI architecture at all to use ccache in a good way ... if we don't
> get a real speedup by the patches, it's certainly not worth the effort to
> integrate this...
>
> If someone wants to have a try to improve the patch series, your help is
> certainly welcome - since at least I personally lack the time and motivation
> to improve this any further.

Do you mind if I play with your patch series? I do not promise 100% of
my time working on it, but I was thinking about dedicating some time
to ccache before your patch series.

>
>   Thomas
>



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-21 12:27               ` Philippe Mathieu-Daudé
@ 2021-05-21 12:52                 ` Daniel P. Berrangé
  2021-05-21 14:21                   ` Philippe Mathieu-Daudé
  2021-05-24 13:51                   ` Stefan Hajnoczi
  0 siblings, 2 replies; 34+ messages in thread
From: Daniel P. Berrangé @ 2021-05-21 12:52 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Thomas Huth, Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Stefan Hajnoczi, Alex Bennée, Warner Losh

On Fri, May 21, 2021 at 02:27:26PM +0200, Philippe Mathieu-Daudé wrote:
> On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
> > On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
> >> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> >>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> >>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> >>>>> +Stefan/Daniel
> >>>>>
> >>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
> >>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> >>>>>>> If a runner has ccache installed, use it and display statistics
> >>>>>>> at the end of the build.
> >>>>>>>
> >>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >>>>>>> ---
> >>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
> >>>>>>>     1 file changed, 5 insertions(+)
> >>>>>>>
> >>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
> >>>>>>> b/.gitlab-ci.d/buildtest-template.yml
> >>>>>>> index f284d7a0eec..a625c697d3b 100644
> >>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
> >>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
> >>>>>>> @@ -6,13 +6,18 @@
> >>>>>>>           then
> >>>>>>>             JOBS=$(sysctl -n hw.ncpu)
> >>>>>>>             MAKE=gmake
> >>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
> >>>>>>>             ;
> >>>>>>>           else
> >>>>>>>             JOBS=$(expr $(nproc) + 1)
> >>>>>>>             MAKE=make
> >>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> >>>>>>
> >>>>>> That does not make sense for the shared runners yet. We first need
> >>>>>> something to enable the caching there - see my series "Use ccache in the
> >>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
> >>>>>
> >>>>> TL;DR: I don't think we should restrict our templates to shared runners.
> >>>>
> >>>> I'm certainly not voting for restricting ourselves to only use shared
> >>>> runners here - but my concern is that this actually *slows* down the shared
> >>>> runners even more! (sorry, I should have elaborated on that in my previous
> >>>> mail already)
> >>>>
> >>>> When I was experimenting with ccache in the shared runners, I saw that the
> >>>> jobs are running even slower with ccache enabled as long as the cache is not
> >>>> populated yet. You only get a speedup afterwards. So if you add this now
> >>>> without also adding the possibility to store the cache persistently, the
> >>>> shared runners will try to populate the cache each time just to throw away
> >>>> the results afterwards again. Thus all the shared runners only get slower
> >>>> without any real benefit here.
> >>>>
> >>>> Thus we either need to get ccache working properly for the shared runners
> >>>> first, or you have to think of a different way of enabling ccache for the
> >>>> non-shared runners, so that it does not affect the shared runners
> >>>> negatively.
> >>>
> >>> Is there anything functional holding up your previous full cccache support
> >>> series from last month ? Or is it just lack of reviews ?
> >>
> >> It's basically the problems mentioned in the cover letter and Stefan's
> >> comment here:
> >>
> >>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
> > 
> > I'm not sure I understand why Stefan thinks gitlab's caching doesn't
> > benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
> > sped up our builds significantly.
> 
> I think Stefan is referring to a comment I made, when using both
> shared runners and dedicated runners (what I'm currently testing)
> various jobs are stuck transferring artifacts/cache {FROM, TO}
> {shared, dedicated} runners at the same time, which is sub-optimal
> because it saturate the dedicated runner network link.

I think we're over thinking things a bit too much and worrying about
scenarios that we're not actually hitting that frequently today, and
delaying the benefit for everyone.

Our common case is that most contributors are simply using shared
runners exclusively, as is the main qemu repo staging branch. AFAIK
these should benefit from a simple ccache enablement today.

Since there are questions about other setups though, we can just
provide an easy way to turn it off. eg:

  if test -z "$QEMU_CI_SKIP_CCACHE"
  then
     PATH=/usr/local/libexec/ccache:$PATH
  fi

anyone who wishes to disable it, can just set that variable in their
git repo fork. If there are specific jobs we want to disable cccache
for, those jobs can set that too.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-21 12:36             ` Willian Rampazzo
@ 2021-05-21 13:02               ` Thomas Huth
  0 siblings, 0 replies; 34+ messages in thread
From: Thomas Huth @ 2021-05-21 13:02 UTC (permalink / raw)
  To: Willian Rampazzo
  Cc: Daniel P. Berrangé, Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, qemu-devel, Stefan Hajnoczi,
	Alex Bennée, Warner Losh

On 21/05/2021 14.36, Willian Rampazzo wrote:
> On Fri, May 21, 2021 at 8:03 AM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
[...]
>>> Is there anything functional holding up your previous full cccache support
>>> series from last month ? Or is it just lack of reviews ?
>>
>> It's basically the problems mentioned in the cover letter and Stefan's
>> comment here:
>>
>>    https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>>
>> The series needs more love and more testing, if it's feasible with the
>> gitlab-CI architecture at all to use ccache in a good way ... if we don't
>> get a real speedup by the patches, it's certainly not worth the effort to
>> integrate this...
>>
>> If someone wants to have a try to improve the patch series, your help is
>> certainly welcome - since at least I personally lack the time and motivation
>> to improve this any further.
> 
> Do you mind if I play with your patch series? I do not promise 100% of
> my time working on it, but I was thinking about dedicating some time
> to ccache before your patch series.

I certainly don't mind! If you get it running well enough so that the total 
runtime decreases, that would be really great!

  Thomas



^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-21 12:52                 ` Daniel P. Berrangé
@ 2021-05-21 14:21                   ` Philippe Mathieu-Daudé
  2021-05-25  5:49                     ` Philippe Mathieu-Daudé
  2021-05-24 13:51                   ` Stefan Hajnoczi
  1 sibling, 1 reply; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-21 14:21 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Thomas Huth, Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Stefan Hajnoczi, Alex Bennée, Warner Losh

On 5/21/21 2:52 PM, Daniel P. Berrangé wrote:
> On Fri, May 21, 2021 at 02:27:26PM +0200, Philippe Mathieu-Daudé wrote:
>> On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
>>> On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
>>>> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
>>>>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
>>>>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
>>>>>>> +Stefan/Daniel
>>>>>>>
>>>>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
>>>>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>>>>>>>> If a runner has ccache installed, use it and display statistics
>>>>>>>>> at the end of the build.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>>>>>> ---
>>>>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>>>>>>>     1 file changed, 5 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>> b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>> index f284d7a0eec..a625c697d3b 100644
>>>>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>> @@ -6,13 +6,18 @@
>>>>>>>>>           then
>>>>>>>>>             JOBS=$(sysctl -n hw.ncpu)
>>>>>>>>>             MAKE=gmake
>>>>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>>>>>>>             ;
>>>>>>>>>           else
>>>>>>>>>             JOBS=$(expr $(nproc) + 1)
>>>>>>>>>             MAKE=make
>>>>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>>>>>>>
>>>>>>>> That does not make sense for the shared runners yet. We first need
>>>>>>>> something to enable the caching there - see my series "Use ccache in the
>>>>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
>>>>>>>
>>>>>>> TL;DR: I don't think we should restrict our templates to shared runners.
>>>>>>
>>>>>> I'm certainly not voting for restricting ourselves to only use shared
>>>>>> runners here - but my concern is that this actually *slows* down the shared
>>>>>> runners even more! (sorry, I should have elaborated on that in my previous
>>>>>> mail already)
>>>>>>
>>>>>> When I was experimenting with ccache in the shared runners, I saw that the
>>>>>> jobs are running even slower with ccache enabled as long as the cache is not
>>>>>> populated yet. You only get a speedup afterwards. So if you add this now
>>>>>> without also adding the possibility to store the cache persistently, the
>>>>>> shared runners will try to populate the cache each time just to throw away
>>>>>> the results afterwards again. Thus all the shared runners only get slower
>>>>>> without any real benefit here.
>>>>>>
>>>>>> Thus we either need to get ccache working properly for the shared runners
>>>>>> first, or you have to think of a different way of enabling ccache for the
>>>>>> non-shared runners, so that it does not affect the shared runners
>>>>>> negatively.
>>>>>
>>>>> Is there anything functional holding up your previous full cccache support
>>>>> series from last month ? Or is it just lack of reviews ?
>>>>
>>>> It's basically the problems mentioned in the cover letter and Stefan's
>>>> comment here:
>>>>
>>>>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>>>
>>> I'm not sure I understand why Stefan thinks gitlab's caching doesn't
>>> benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
>>> sped up our builds significantly.
>>
>> I think Stefan is referring to a comment I made, when using both
>> shared runners and dedicated runners (what I'm currently testing)
>> various jobs are stuck transferring artifacts/cache {FROM, TO}
>> {shared, dedicated} runners at the same time, which is sub-optimal
>> because it saturate the dedicated runner network link.
> 
> I think we're over thinking things a bit too much and worrying about
> scenarios that we're not actually hitting that frequently today, and
> delaying the benefit for everyone.
> 
> Our common case is that most contributors are simply using shared
> runners exclusively, as is the main qemu repo staging branch. AFAIK
> these should benefit from a simple ccache enablement today.
> 
> Since there are questions about other setups though, we can just
> provide an easy way to turn it off. eg:
> 
>   if test -z "$QEMU_CI_SKIP_CCACHE"
>   then
>      PATH=/usr/local/libexec/ccache:$PATH
>   fi
> 
> anyone who wishes to disable it, can just set that variable in their
> git repo fork. If there are specific jobs we want to disable cccache
> for, those jobs can set that too.

OK, understood. I'll see with Willian how to have ccache working.

Thanks for the feedback,

Phil.


^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-21 12:52                 ` Daniel P. Berrangé
  2021-05-21 14:21                   ` Philippe Mathieu-Daudé
@ 2021-05-24 13:51                   ` Stefan Hajnoczi
  2021-05-25  5:50                     ` Philippe Mathieu-Daudé
  1 sibling, 1 reply; 34+ messages in thread
From: Stefan Hajnoczi @ 2021-05-24 13:51 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Thomas Huth, Kyle Evans, Philippe Mathieu-Daudé,
	Wainer dos Santos Moschetta, qemu-devel, Willian Rampazzo,
	Alex Bennée, Warner Losh

[-- Attachment #1: Type: text/plain, Size: 4636 bytes --]

On Fri, May 21, 2021 at 01:52:13PM +0100, Daniel P. Berrangé wrote:
> On Fri, May 21, 2021 at 02:27:26PM +0200, Philippe Mathieu-Daudé wrote:
> > On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
> > > On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
> > >> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
> > >>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
> > >>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
> > >>>>> +Stefan/Daniel
> > >>>>>
> > >>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
> > >>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
> > >>>>>>> If a runner has ccache installed, use it and display statistics
> > >>>>>>> at the end of the build.
> > >>>>>>>
> > >>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > >>>>>>> ---
> > >>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
> > >>>>>>>     1 file changed, 5 insertions(+)
> > >>>>>>>
> > >>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
> > >>>>>>> b/.gitlab-ci.d/buildtest-template.yml
> > >>>>>>> index f284d7a0eec..a625c697d3b 100644
> > >>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
> > >>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
> > >>>>>>> @@ -6,13 +6,18 @@
> > >>>>>>>           then
> > >>>>>>>             JOBS=$(sysctl -n hw.ncpu)
> > >>>>>>>             MAKE=gmake
> > >>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
> > >>>>>>>             ;
> > >>>>>>>           else
> > >>>>>>>             JOBS=$(expr $(nproc) + 1)
> > >>>>>>>             MAKE=make
> > >>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
> > >>>>>>
> > >>>>>> That does not make sense for the shared runners yet. We first need
> > >>>>>> something to enable the caching there - see my series "Use ccache in the
> > >>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
> > >>>>>
> > >>>>> TL;DR: I don't think we should restrict our templates to shared runners.
> > >>>>
> > >>>> I'm certainly not voting for restricting ourselves to only use shared
> > >>>> runners here - but my concern is that this actually *slows* down the shared
> > >>>> runners even more! (sorry, I should have elaborated on that in my previous
> > >>>> mail already)
> > >>>>
> > >>>> When I was experimenting with ccache in the shared runners, I saw that the
> > >>>> jobs are running even slower with ccache enabled as long as the cache is not
> > >>>> populated yet. You only get a speedup afterwards. So if you add this now
> > >>>> without also adding the possibility to store the cache persistently, the
> > >>>> shared runners will try to populate the cache each time just to throw away
> > >>>> the results afterwards again. Thus all the shared runners only get slower
> > >>>> without any real benefit here.
> > >>>>
> > >>>> Thus we either need to get ccache working properly for the shared runners
> > >>>> first, or you have to think of a different way of enabling ccache for the
> > >>>> non-shared runners, so that it does not affect the shared runners
> > >>>> negatively.
> > >>>
> > >>> Is there anything functional holding up your previous full cccache support
> > >>> series from last month ? Or is it just lack of reviews ?
> > >>
> > >> It's basically the problems mentioned in the cover letter and Stefan's
> > >> comment here:
> > >>
> > >>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
> > > 
> > > I'm not sure I understand why Stefan thinks gitlab's caching doesn't
> > > benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
> > > sped up our builds significantly.
> > 
> > I think Stefan is referring to a comment I made, when using both
> > shared runners and dedicated runners (what I'm currently testing)
> > various jobs are stuck transferring artifacts/cache {FROM, TO}
> > {shared, dedicated} runners at the same time, which is sub-optimal
> > because it saturate the dedicated runner network link.
> 
> I think we're over thinking things a bit too much and worrying about
> scenarios that we're not actually hitting that frequently today, and
> delaying the benefit for everyone.

Thomas' original email indicated using ccache with QEMU isn't
necessarily a win:

  Additionally, the jobs are sometimes running even slower, e.g. if the
  cache has not been populated yet or if there are a lot of cache
  misses,

Let's measure the time taken both on first run and on a subsequent run.
This information can be included in the patch series cover letter so we
know ccache performance has been measured and it works well.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-21 14:21                   ` Philippe Mathieu-Daudé
@ 2021-05-25  5:49                     ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-25  5:49 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Thomas Huth, Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Stefan Hajnoczi, Alex Bennée, Warner Losh

On 5/21/21 4:21 PM, Philippe Mathieu-Daudé wrote:
> On 5/21/21 2:52 PM, Daniel P. Berrangé wrote:
>> On Fri, May 21, 2021 at 02:27:26PM +0200, Philippe Mathieu-Daudé wrote:
>>> On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
>>>> On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
>>>>> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
>>>>>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
>>>>>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
>>>>>>>> +Stefan/Daniel
>>>>>>>>
>>>>>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
>>>>>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>>>>>>>>> If a runner has ccache installed, use it and display statistics
>>>>>>>>>> at the end of the build.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>>>>>>> ---
>>>>>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>>>>>>>>     1 file changed, 5 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> index f284d7a0eec..a625c697d3b 100644
>>>>>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> @@ -6,13 +6,18 @@
>>>>>>>>>>           then
>>>>>>>>>>             JOBS=$(sysctl -n hw.ncpu)
>>>>>>>>>>             MAKE=gmake
>>>>>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>>>>>>>>             ;
>>>>>>>>>>           else
>>>>>>>>>>             JOBS=$(expr $(nproc) + 1)
>>>>>>>>>>             MAKE=make
>>>>>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>>>>>>>>
>>>>>>>>> That does not make sense for the shared runners yet. We first need
>>>>>>>>> something to enable the caching there - see my series "Use ccache in the
>>>>>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
>>>>>>>>
>>>>>>>> TL;DR: I don't think we should restrict our templates to shared runners.
>>>>>>>
>>>>>>> I'm certainly not voting for restricting ourselves to only use shared
>>>>>>> runners here - but my concern is that this actually *slows* down the shared
>>>>>>> runners even more! (sorry, I should have elaborated on that in my previous
>>>>>>> mail already)
>>>>>>>
>>>>>>> When I was experimenting with ccache in the shared runners, I saw that the
>>>>>>> jobs are running even slower with ccache enabled as long as the cache is not
>>>>>>> populated yet. You only get a speedup afterwards. So if you add this now
>>>>>>> without also adding the possibility to store the cache persistently, the
>>>>>>> shared runners will try to populate the cache each time just to throw away
>>>>>>> the results afterwards again. Thus all the shared runners only get slower
>>>>>>> without any real benefit here.
>>>>>>>
>>>>>>> Thus we either need to get ccache working properly for the shared runners
>>>>>>> first, or you have to think of a different way of enabling ccache for the
>>>>>>> non-shared runners, so that it does not affect the shared runners
>>>>>>> negatively.
>>>>>>
>>>>>> Is there anything functional holding up your previous full cccache support
>>>>>> series from last month ? Or is it just lack of reviews ?
>>>>>
>>>>> It's basically the problems mentioned in the cover letter and Stefan's
>>>>> comment here:
>>>>>
>>>>>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>>>>
>>>> I'm not sure I understand why Stefan thinks gitlab's caching doesn't
>>>> benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
>>>> sped up our builds significantly.
>>>
>>> I think Stefan is referring to a comment I made, when using both
>>> shared runners and dedicated runners (what I'm currently testing)
>>> various jobs are stuck transferring artifacts/cache {FROM, TO}
>>> {shared, dedicated} runners at the same time, which is sub-optimal
>>> because it saturate the dedicated runner network link.

FYI In case we need to sort this out later, the 'resource_group' might
help us with this:
https://docs.gitlab.com/ee/ci/yaml/#resource_group

>> I think we're over thinking things a bit too much and worrying about
>> scenarios that we're not actually hitting that frequently today, and
>> delaying the benefit for everyone.
>>
>> Our common case is that most contributors are simply using shared
>> runners exclusively, as is the main qemu repo staging branch. AFAIK
>> these should benefit from a simple ccache enablement today.
>>
>> Since there are questions about other setups though, we can just
>> provide an easy way to turn it off. eg:
>>
>>   if test -z "$QEMU_CI_SKIP_CCACHE"
>>   then
>>      PATH=/usr/local/libexec/ccache:$PATH
>>   fi
>>
>> anyone who wishes to disable it, can just set that variable in their
>> git repo fork. If there are specific jobs we want to disable cccache
>> for, those jobs can set that too.
> 
> OK, understood. I'll see with Willian how to have ccache working.
> 
> Thanks for the feedback,
> 
> Phil.
> 


^ permalink raw reply	[flat|nested] 34+ messages in thread

* Re: [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics
  2021-05-24 13:51                   ` Stefan Hajnoczi
@ 2021-05-25  5:50                     ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 34+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-05-25  5:50 UTC (permalink / raw)
  To: Stefan Hajnoczi, Daniel P. Berrangé
  Cc: Thomas Huth, Kyle Evans, qemu-devel, Wainer dos Santos Moschetta,
	Willian Rampazzo, Alex Bennée, Warner Losh

On 5/24/21 3:51 PM, Stefan Hajnoczi wrote:
> On Fri, May 21, 2021 at 01:52:13PM +0100, Daniel P. Berrangé wrote:
>> On Fri, May 21, 2021 at 02:27:26PM +0200, Philippe Mathieu-Daudé wrote:
>>> On 5/21/21 1:53 PM, Daniel P. Berrangé wrote:
>>>> On Fri, May 21, 2021 at 01:02:51PM +0200, Thomas Huth wrote:
>>>>> On 21/05/2021 12.50, Daniel P. Berrangé wrote:
>>>>>> On Fri, May 21, 2021 at 12:48:21PM +0200, Thomas Huth wrote:
>>>>>>> On 20/05/2021 13.27, Philippe Mathieu-Daudé wrote:
>>>>>>>> +Stefan/Daniel
>>>>>>>>
>>>>>>>> On 5/20/21 10:02 AM, Thomas Huth wrote:
>>>>>>>>> On 19/05/2021 20.45, Philippe Mathieu-Daudé wrote:
>>>>>>>>>> If a runner has ccache installed, use it and display statistics
>>>>>>>>>> at the end of the build.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>>>>>>>> ---
>>>>>>>>>>     .gitlab-ci.d/buildtest-template.yml | 5 +++++
>>>>>>>>>>     1 file changed, 5 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> index f284d7a0eec..a625c697d3b 100644
>>>>>>>>>> --- a/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> +++ b/.gitlab-ci.d/buildtest-template.yml
>>>>>>>>>> @@ -6,13 +6,18 @@
>>>>>>>>>>           then
>>>>>>>>>>             JOBS=$(sysctl -n hw.ncpu)
>>>>>>>>>>             MAKE=gmake
>>>>>>>>>> +        PATH=/usr/local/libexec/ccache:$PATH
>>>>>>>>>>             ;
>>>>>>>>>>           else
>>>>>>>>>>             JOBS=$(expr $(nproc) + 1)
>>>>>>>>>>             MAKE=make
>>>>>>>>>> +        PATH=/usr/lib/ccache:/usr/lib64/ccache:$PATH
>>>>>>>>>
>>>>>>>>> That does not make sense for the shared runners yet. We first need
>>>>>>>>> something to enable the caching there - see my series "Use ccache in the
>>>>>>>>> gitlab-CI" from April (which is currently stalled unfortunately).
>>>>>>>>
>>>>>>>> TL;DR: I don't think we should restrict our templates to shared runners.
>>>>>>>
>>>>>>> I'm certainly not voting for restricting ourselves to only use shared
>>>>>>> runners here - but my concern is that this actually *slows* down the shared
>>>>>>> runners even more! (sorry, I should have elaborated on that in my previous
>>>>>>> mail already)
>>>>>>>
>>>>>>> When I was experimenting with ccache in the shared runners, I saw that the
>>>>>>> jobs are running even slower with ccache enabled as long as the cache is not
>>>>>>> populated yet. You only get a speedup afterwards. So if you add this now
>>>>>>> without also adding the possibility to store the cache persistently, the
>>>>>>> shared runners will try to populate the cache each time just to throw away
>>>>>>> the results afterwards again. Thus all the shared runners only get slower
>>>>>>> without any real benefit here.
>>>>>>>
>>>>>>> Thus we either need to get ccache working properly for the shared runners
>>>>>>> first, or you have to think of a different way of enabling ccache for the
>>>>>>> non-shared runners, so that it does not affect the shared runners
>>>>>>> negatively.
>>>>>>
>>>>>> Is there anything functional holding up your previous full cccache support
>>>>>> series from last month ? Or is it just lack of reviews ?
>>>>>
>>>>> It's basically the problems mentioned in the cover letter and Stefan's
>>>>> comment here:
>>>>>
>>>>>  https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02219.html
>>>>
>>>> I'm not sure I understand why Stefan thinks gitlab's caching doesn't
>>>> benefit ccache. We add ccache for libvirt GitLab CI, and AFAIR it
>>>> sped up our builds significantly.
>>>
>>> I think Stefan is referring to a comment I made, when using both
>>> shared runners and dedicated runners (what I'm currently testing)
>>> various jobs are stuck transferring artifacts/cache {FROM, TO}
>>> {shared, dedicated} runners at the same time, which is sub-optimal
>>> because it saturate the dedicated runner network link.
>>
>> I think we're over thinking things a bit too much and worrying about
>> scenarios that we're not actually hitting that frequently today, and
>> delaying the benefit for everyone.
> 
> Thomas' original email indicated using ccache with QEMU isn't
> necessarily a win:
> 
>   Additionally, the jobs are sometimes running even slower, e.g. if the
>   cache has not been populated yet or if there are a lot of cache
>   misses,
> 
> Let's measure the time taken both on first run and on a subsequent run.
> This information can be included in the patch series cover letter so we
> know ccache performance has been measured and it works well.

To whoever is interested in looking at this problem, GitLab provides
the TRANSFER_METER_FREQUENCY variable to debug this problem; and the
compression ratio can be adjusted using ARTIFACT_COMPRESSION_LEVEL
and CACHE_COMPRESSION_LEVEL:
https://docs.gitlab.com/ee/ci/runners/README.html#artifact-and-cache-settings


^ permalink raw reply	[flat|nested] 34+ messages in thread

end of thread, other threads:[~2021-05-25  5:52 UTC | newest]

Thread overview: 34+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-19 18:45 [PATCH v3 0/6] gitlab-ci: Allow using FreeBSD runners Philippe Mathieu-Daudé
2021-05-19 18:45 ` [PATCH v3 1/6] gitlab-ci: Extract &environment_variables template Philippe Mathieu-Daudé
2021-05-19 19:12   ` Willian Rampazzo
2021-05-20 18:00   ` Wainer dos Santos Moschetta
2021-05-19 18:45 ` [PATCH v3 2/6] gitlab-ci: Adapt JOBS variable for FreeBSD runners Philippe Mathieu-Daudé
2021-05-19 19:11   ` Willian Rampazzo
2021-05-20  5:50   ` Thomas Huth
2021-05-20 18:18   ` Wainer dos Santos Moschetta
2021-05-20 20:02     ` Warner Losh
2021-05-19 18:45 ` [PATCH v3 3/6] gitlab-ci: Run GNU make via the $MAKE variable Philippe Mathieu-Daudé
2021-05-19 19:10   ` Willian Rampazzo
2021-05-20  7:56   ` Thomas Huth
2021-05-20 18:26   ` Wainer dos Santos Moschetta
2021-05-19 18:45 ` [PATCH v3 4/6] gitlab-ci: Add ccache in $PATH and display statistics Philippe Mathieu-Daudé
2021-05-19 19:14   ` Willian Rampazzo
2021-05-20  8:02   ` Thomas Huth
2021-05-20 11:27     ` Philippe Mathieu-Daudé
2021-05-21 10:48       ` Thomas Huth
2021-05-21 10:50         ` Daniel P. Berrangé
2021-05-21 11:02           ` Thomas Huth
2021-05-21 11:53             ` Daniel P. Berrangé
2021-05-21 12:27               ` Philippe Mathieu-Daudé
2021-05-21 12:52                 ` Daniel P. Berrangé
2021-05-21 14:21                   ` Philippe Mathieu-Daudé
2021-05-25  5:49                     ` Philippe Mathieu-Daudé
2021-05-24 13:51                   ` Stefan Hajnoczi
2021-05-25  5:50                     ` Philippe Mathieu-Daudé
2021-05-21 12:36             ` Willian Rampazzo
2021-05-21 13:02               ` Thomas Huth
2021-05-19 18:45 ` [PATCH v3 5/6] gitlab-ci: Simplify before/after script for Avocado based jobs Philippe Mathieu-Daudé
2021-05-19 19:42   ` Willian Rampazzo
2021-05-20  8:04   ` Thomas Huth
2021-05-19 18:45 ` [PATCH v3 6/6] gitlab-ci: Add FreeBSD jobs Philippe Mathieu-Daudé
2021-05-19 19:43   ` Willian Rampazzo

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.