[Qemu-devel] [PATCH 00/10] travis: Use caching when possible

[Qemu-devel] [PATCH 00/10] travis: Use caching when possible

[Philippe Mathieu-Daudé]

This series tries to reduce Travis CI failures due to network
issues, and tries to speed builds a bit, reducing failures due
to job timeouting.

Travis feature: https://docs.travis-ci.com/user/caching/

To see improvement over time, I recommend to apply this series
on top of different merge commit on master, to see changes in
cache use.

Few patches come from:
https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg04935.html

Regards,

Phil.

Philippe Mathieu-Daudé (10):
  .travis.yml: Enable multiple caching features
  .travis.yml: Increase cache timeout from 3min to 20min
  .travis.yml: Cache Python PIP packages
  .travis.yml: Cache Avocado cache
  .travis.yml: Improve ccache use
  .travis.yml: Enable ccache on OSX
  .travis.yml: Document how the build matrix use caches
  .travis.yml: Cache Linux/GCC 'debug profile' jobs together
  .travis.yml: Cache Linux/GCC 'non-debug profile' jobs together
  .travis.yml: Cache Linux/Clang jobs together

 .travis.yml | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)

-- 
2.20.1

[Qemu-devel] [PATCH 01/10] .travis.yml: Enable multiple caching features

[Philippe Mathieu-Daudé]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Using the 'multiple caching features' means explode the YAML array,
thus it eases the git workflow (it is easier to move patches around).

See https://docs.travis-ci.com/user/caching#enabling-multiple-caching-features

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20170809202712.6951-4-f4bug@amsat.org>
---
 .travis.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index caf0a1f8fa..7478d758e1 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -5,7 +5,8 @@ dist: xenial
 language: c
 compiler:
   - gcc
-cache: ccache
+cache:
+  ccache: true
 
 
 addons:
-- 
2.20.1

[Qemu-devel] [PATCH 02/10] .travis.yml: Increase cache timeout from 3min to 20min

[Philippe Mathieu-Daudé]

We are going to cache few gigabytes, increase the cache timeout
to avoid build failures when uploading our cache.

See https://docs.travis-ci.com/user/caching/#setting-the-timeout

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.travis.yml b/.travis.yml
index 7478d758e1..a65d13a592 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -6,6 +6,7 @@ language: c
 compiler:
   - gcc
 cache:
+  timeout: 1200
   ccache: true
 
 
-- 
2.20.1

[Qemu-devel] [PATCH 03/10] .travis.yml: Cache Python PIP packages

[Philippe Mathieu-Daudé]

We always install the same packages ever and ever, cache them.

See https://docs.travis-ci.com/user/caching/#pip-cache

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.travis.yml b/.travis.yml
index a65d13a592..3211667adb 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -8,6 +8,7 @@ compiler:
 cache:
   timeout: 1200
   ccache: true
+  pip: true
 
 
 addons:
-- 
2.20.1

[Qemu-devel] [PATCH 04/10] .travis.yml: Cache Avocado cache

[Philippe Mathieu-Daudé]

Avocado tests download artifacts from various sources.
These sources sometime have network issues resulting in build
failures. Cache Avocado cache to reduce build failure.

See https://docs.travis-ci.com/user/caching/#arbitrary-directories

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index 3211667adb..a3db5b3f63 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -9,6 +9,8 @@ cache:
   timeout: 1200
   ccache: true
   pip: true
+  directories:
+  - $HOME/avocado/data/cache
 
 
 addons:
-- 
2.20.1

[Qemu-devel] [PATCH 05/10] .travis.yml: Improve ccache use

[Philippe Mathieu-Daudé]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Per https://ccache.dev/manual/latest.html:

  By default, ccache tries to give as few false cache hits as
  possible. However, in certain situations it’s possible that
  you know things that ccache can’t take for granted.

  [The CCACHE_SLOPINESS environment variable] makes it possible
  to tell ccache to relax some checks in order to increase the
  hit rate.

We can relax the ctime/mtime header checks:

  - include_file_ctime

    By default, ccache also will not cache a file if it
    includes a header whose ctime is too new. This option
    disables that check.

  - include_file_mtime

    By default, ccache will not cache a file if it includes
    a header whose mtime is too new. This option disables
    that check.

We also add a call to clear the cache statistics before running
the build, and display them when the build finishes.

See https://docs.travis-ci.com/user/caching/#ccache-cache

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .travis.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index a3db5b3f63..fccc1e091d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -70,6 +70,9 @@ env:
     - TEST_CMD="make check -j3 V=1"
     # This is broadly a list of "mainline" softmmu targets which have support across the major distros
     - MAIN_SOFTMMU_TARGETS="aarch64-softmmu,arm-softmmu,i386-softmmu,mips-softmmu,mips64-softmmu,ppc64-softmmu,riscv64-softmmu,s390x-softmmu,x86_64-softmmu"
+    - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime"
+    - CCACHE_MAXSIZE=1G
+
 
 git:
   # we want to do this ourselves
@@ -77,10 +80,13 @@ git:
 
 
 before_script:
+  - command -v ccache && ccache --zero-stats
   - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
   - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
 script:
   - make -j3 && travis_retry ${TEST_CMD}
+after_script:
+  - command -v ccache && ccache --show-stats
 
 
 matrix:
-- 
2.20.1

[Qemu-devel] [PATCH 06/10] .travis.yml: Enable ccache on OSX

[Philippe Mathieu-Daudé]

By default, ccache is not installed on macOS environments.

See https://docs.travis-ci.com/user/caching/#ccache-on-macos

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index fccc1e091d..640d84f201 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -45,6 +45,7 @@ addons:
       - gcovr
   homebrew:
     packages:
+      - ccache
       - glib
       - pixman
       - gnu-sed
@@ -80,6 +81,7 @@ git:
 
 
 before_script:
+  - if [ "$TRAVIS_OS_NAME" == "osx" ] ; then export PATH="/usr/local/opt/ccache/libexec:$PATH" ; fi
   - command -v ccache && ccache --zero-stats
   - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
   - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
-- 
2.20.1

[Qemu-devel] [PATCH 07/10] .travis.yml: Document how the build matrix use caches

[Philippe Mathieu-Daudé]

We will set the CACHE_NAME variable to improve the caching
of various jobs using the same characteristics. Document it
first.

See https://docs.travis-ci.com/user/caching/#caches-and-build-matrices

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index 640d84f201..c64a551829 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -6,6 +6,12 @@ language: c
 compiler:
   - gcc
 cache:
+  # There is one cache per branch and compiler version.
+  # characteristics of each job are used to identify the cache:
+  # - OS name (currently, linux, osx, or windows)
+  # - OS distribution (for Linux, xenial, trusty, or precise)
+  # - macOS image name (e.g., xcode7.2)
+  # - Names and values of visible environment variables set in .travis.yml or Settings panel
   timeout: 1200
   ccache: true
   pip: true
-- 
2.20.1

[Qemu-devel] [PATCH 08/10] .travis.yml: Cache Linux/GCC 'debug profile' jobs together

[Philippe Mathieu-Daudé]

These jobs build different components but use the same
host features. Put them in the same cache bucket.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index c64a551829..e362c3fe02 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -120,11 +120,13 @@ matrix:
 
     - env:
         - CONFIG="--enable-debug --enable-debug-tcg --disable-user"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
 
 
     # TCG debug can be run just on it's own and is mostly agnostic to user/softmmu distinctions
     - env:
         - CONFIG="--enable-debug-tcg --disable-system"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
 
 
     - env:
-- 
2.20.1

CACHE_NAME in .travis.yml (was Re: [Qemu-devel] [PATCH 08/10] .travis.yml: Cache Linux/GCC 'debug profile' jobs together)

[Eduardo Habkost]

I just stumbled upon this when editing .travis.yml, and I'm
trying to understand how the $CACHE_NAME trick is supposed to
work.

If I understood the Travis documentation correctly, all
environment variables will affect the cache id (including
$CONFIG, in the case below).  This would mean the two entries
would still use a different cache even if $CACHE_NAME is the
same.

See, for example:
https://travis-ci.org/github/qemu/qemu/jobs/735738551#L965
https://travis-ci.org/github/qemu/qemu/jobs/735738553#L967

Both build tasks have the same value for $CACHE_NAME, but they
are fetching two different cache URLs:

master/cache--linux-focal-50ba7438064455bdb04bb41dcde2c91c34bb797265fe435b84cee4bae7d9b61d--compiler-gcc.tgz
master/cache--linux-focal-8991cfc9b7f3f268e7e50aff4218daa4c206597ae796c60643a84ed58bba0cfb--compiler-gcc.tgz


On Tue, Sep 03, 2019 at 03:26:47PM +0200, Philippe Mathieu-Daudé wrote:
> These jobs build different components but use the same
> host features. Put them in the same cache bucket.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  .travis.yml | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/.travis.yml b/.travis.yml
> index c64a551829..e362c3fe02 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -120,11 +120,13 @@ matrix:
>  
>      - env:
>          - CONFIG="--enable-debug --enable-debug-tcg --disable-user"
> +        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
>  
>  
>      # TCG debug can be run just on it's own and is mostly agnostic to user/softmmu distinctions
>      - env:
>          - CONFIG="--enable-debug-tcg --disable-system"
> +        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
>  
>  
>      - env:
> -- 
> 2.20.1
> 
> 

-- 
Eduardo

[Qemu-devel] [PATCH 09/10] .travis.yml: Cache Linux/GCC 'non-debug profile' jobs together

[Philippe Mathieu-Daudé]

These jobs build different components but use the same
host features. Put them in the same cache bucket.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index e362c3fe02..7132cf8fba 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -101,15 +101,18 @@ matrix:
   include:
     - env:
         - CONFIG="--disable-system --static"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
 
 
     # we split the system builds as it takes a while to build them all
     - env:
         - CONFIG="--disable-user --target-list=${MAIN_SOFTMMU_TARGETS}"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
 
 
     - env:
         - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
 
 
     # Just build tools and run minimal unit and softfloat checks
@@ -117,6 +120,8 @@ matrix:
         - BASE_CONFIG="--enable-tools"
         - CONFIG="--disable-user --disable-system"
         - TEST_CMD="make check-unit check-softfloat -j3"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
+
 
     - env:
         - CONFIG="--enable-debug --enable-debug-tcg --disable-user"
@@ -136,6 +141,7 @@ matrix:
     # Module builds are mostly of interest to major distros
     - env:
         - CONFIG="--enable-modules --target-list=${MAIN_SOFTMMU_TARGETS}"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
 
 
     # Alternate coroutines implementations are only really of interest to KVM users
@@ -155,6 +161,7 @@ matrix:
         - BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.."
         - BASE_CONFIG="--enable-tools --enable-docs"
         - CONFIG="--target-list=x86_64-softmmu,aarch64-linux-user"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
       addons:
         apt:
           packages:
@@ -196,6 +203,7 @@ matrix:
     # We manually include builds which we disable "make check" for
     - env:
         - CONFIG="--without-default-devices --disable-user"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
         - TEST_CMD=""
 
 
@@ -239,6 +247,7 @@ matrix:
     # Python builds
     - env:
         - CONFIG="--target-list=x86_64-softmmu"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
       language: python
       python:
         - "3.4"
@@ -246,6 +255,7 @@ matrix:
 
     - env:
         - CONFIG="--target-list=x86_64-softmmu"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
       language: python
       python:
         - "3.6"
@@ -312,8 +322,11 @@ matrix:
     - env:
         - CONFIG="--disable-system"
         - TEST_CMD="make -j3 check-tcg V=1"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
+
 
     # Run check-tcg against softmmu targets
     - env:
         - CONFIG="--target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu"
         - TEST_CMD="make -j3 check-tcg V=1"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
-- 
2.20.1

[Qemu-devel] [PATCH 10/10] .travis.yml: Cache Linux/Clang jobs together

[Philippe Mathieu-Daudé]

These jobs build different components but use the same
host features. Put them in the same cache bucket.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index 7132cf8fba..bd891fe284 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -173,11 +173,13 @@ matrix:
     # Test with Clang for compile portability (Travis uses clang-5.0)
     - env:
         - CONFIG="--disable-system"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
       compiler: clang
 
 
     - env:
         - CONFIG="--disable-user --target-list=${MAIN_SOFTMMU_TARGETS}"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
       compiler: clang
 
 
@@ -190,6 +192,7 @@ matrix:
 
     - env:
         - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
+        - CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
       compiler: clang
 
 
-- 
2.20.1

Re: [Qemu-devel] [PATCH 00/10] travis: Use caching when possible

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> This series tries to reduce Travis CI failures due to network
> issues, and tries to speed builds a bit, reducing failures due
> to job timeouting.

Looking good.

Queued to testing/next, thanks.

>
> Travis feature: https://docs.travis-ci.com/user/caching/
>
> To see improvement over time, I recommend to apply this series
> on top of different merge commit on master, to see changes in
> cache use.
>
> Few patches come from:
> https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg04935.html
>
> Regards,
>
> Phil.
>
> Philippe Mathieu-Daudé (10):
>   .travis.yml: Enable multiple caching features
>   .travis.yml: Increase cache timeout from 3min to 20min
>   .travis.yml: Cache Python PIP packages
>   .travis.yml: Cache Avocado cache
>   .travis.yml: Improve ccache use
>   .travis.yml: Enable ccache on OSX
>   .travis.yml: Document how the build matrix use caches
>   .travis.yml: Cache Linux/GCC 'debug profile' jobs together
>   .travis.yml: Cache Linux/GCC 'non-debug profile' jobs together
>   .travis.yml: Cache Linux/Clang jobs together
>
>  .travis.yml | 39 ++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 38 insertions(+), 1 deletion(-)


--
Alex Bennée