* [PATCH v2 00/12] testing/next (with build fixes!)
@ 2020-01-30 11:32 Alex Bennée
2020-01-30 11:32 ` [PATCH v2 01/12] tests/docker: move most cross compilers to buster base Alex Bennée
` (11 more replies)
0 siblings, 12 replies; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Alex Bennée, richard.henderson,
f4bug, cota, stefanha, marcandre.lureau, pbonzini, aurelien
Hi,
It seemed the CI degraded while I wasn't looking so I've rolled up a
bunch of fixes to try and get things working again.
There is some messing around for documentation builds (disabling on
shippable, moving to bionic on Travis), a build fix for hppa +
no-default-configs and some attempts to squash timeouts on our new
multiple architecture builds.
I shall build the forthcoming PR directly from this posted series.
The following patches need review:
01 - tests docker move most cross compilers to buster
02 - tests docker better handle symlinked libs
03 - gitlab ci Refresh the list of iotests
04 - travis.yml Install genisoimage package
05 - .shippable disable docs for cross compile tests
07 - .travis.yml Drop superfluous use of python python
08 - .travis.yml Add description to each job
09 - .travis.yml build documents under bionic
10 - .travis.yml move cache flushing to early common p
11 - .travis.yml probe for number of available process
12 - .travis.yml limit the arm64 target list
Alex Bennée (7):
tests/docker: move most cross compilers to buster base
tests/docker: better handle symlinked libs
.shippable: --disable-docs for cross-compile tests
.travis.yml: build documents under bionic
.travis.yml: move cache flushing to early common phase
.travis.yml: probe for number of available processors
.travis.yml: limit the arm64 target list
Philippe Mathieu-Daudé (3):
hw/hppa/Kconfig: LASI chipset requires PARALLEL port
.travis.yml: Drop superfluous use of --python=python3 parameter
.travis.yml: Add description to each job
Thomas Huth (1):
gitlab-ci: Refresh the list of iotests
Wainer dos Santos Moschetta (1):
travis.yml: Install genisoimage package
.gitlab-ci.yml | 12 +-
.shippable.yml | 2 +-
.travis.yml | 145 ++++++++++++------
hw/hppa/Kconfig | 1 +
tests/docker/Makefile.include | 16 +-
tests/docker/docker.py | 5 +-
tests/docker/dockerfiles/debian-amd64.docker | 2 +-
.../dockerfiles/debian-armel-cross.docker | 2 +-
.../dockerfiles/debian-armhf-cross.docker | 2 +-
.../dockerfiles/debian-mips64el-cross.docker | 2 +-
.../dockerfiles/debian-mipsel-cross.docker | 2 +-
.../dockerfiles/debian-ppc64el-cross.docker | 2 +-
.../dockerfiles/debian-s390x-cross.docker | 2 +-
13 files changed, 121 insertions(+), 74 deletions(-)
--
2.20.1
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH v2 01/12] tests/docker: move most cross compilers to buster base
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-31 15:22 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 02/12] tests/docker: better handle symlinked libs Alex Bennée
` (10 subsequent siblings)
11 siblings, 1 reply; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Alex Bennée, richard.henderson,
f4bug, Philippe Mathieu-Daudé,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
This includes fixing up the dependencies (Which were already wrong for
one of the mips variants).
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/Makefile.include | 16 ++++++++--------
tests/docker/dockerfiles/debian-amd64.docker | 2 +-
.../docker/dockerfiles/debian-armel-cross.docker | 2 +-
.../docker/dockerfiles/debian-armhf-cross.docker | 2 +-
.../dockerfiles/debian-mips64el-cross.docker | 2 +-
.../dockerfiles/debian-mipsel-cross.docker | 2 +-
.../dockerfiles/debian-ppc64el-cross.docker | 2 +-
.../docker/dockerfiles/debian-s390x-cross.docker | 2 +-
8 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 19dbe26169..43a8678688 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -91,19 +91,12 @@ endif
# Enforce dependencies for composite images
docker-image-debian9-mxe: docker-image-debian9
ifeq ($(HOST_ARCH),x86_64)
-docker-image-debian-amd64: docker-image-debian9
+docker-image-debian-amd64: docker-image-debian10
DOCKER_PARTIAL_IMAGES += debian-amd64-cross
else
docker-image-debian-amd64-cross: docker-image-debian10
DOCKER_PARTIAL_IMAGES += debian-amd64
endif
-docker-image-debian-armel-cross: docker-image-debian9
-docker-image-debian-armhf-cross: docker-image-debian9
-docker-image-debian-mips-cross: docker-image-debian9
-docker-image-debian-mipsel-cross: docker-image-debian9
-docker-image-debian-mips64el-cross: docker-image-debian9
-docker-image-debian-ppc64el-cross: docker-image-debian9
-docker-image-debian-s390x-cross: docker-image-debian9
docker-image-debian-win32-cross: docker-image-debian9-mxe
docker-image-debian-win64-cross: docker-image-debian9-mxe
@@ -118,12 +111,19 @@ endif
docker-image-debian-alpha-cross: docker-image-debian10
docker-image-debian-arm64-cross: docker-image-debian10
+docker-image-debian-armel-cross: docker-image-debian10
+docker-image-debian-armhf-cross: docker-image-debian10
docker-image-debian-hppa-cross: docker-image-debian10
docker-image-debian-m68k-cross: docker-image-debian10
+docker-image-debian-mips-cross: docker-image-debian10
docker-image-debian-mips64-cross: docker-image-debian10
+docker-image-debian-mips64el-cross: docker-image-debian10
+docker-image-debian-mipsel-cross: docker-image-debian10
docker-image-debian-powerpc-cross: docker-image-debian10
docker-image-debian-ppc64-cross: docker-image-debian10
+docker-image-debian-ppc64el-cross: docker-image-debian10
docker-image-debian-riscv64-cross: docker-image-debian10
+docker-image-debian-s390x-cross: docker-image-debian10
docker-image-debian-sh4-cross: docker-image-debian10
docker-image-debian-sparc64-cross: docker-image-debian10
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index 431e947ebd..3b860af106 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -4,7 +4,7 @@
# This docker target builds on the debian Stretch base image. Further
# libraries which are not widely available are installed by hand.
#
-FROM qemu:debian9
+FROM qemu:debian10
MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
RUN apt update && \
diff --git a/tests/docker/dockerfiles/debian-armel-cross.docker b/tests/docker/dockerfiles/debian-armel-cross.docker
index 15378f8ea2..e3794a61c9 100644
--- a/tests/docker/dockerfiles/debian-armel-cross.docker
+++ b/tests/docker/dockerfiles/debian-armel-cross.docker
@@ -3,7 +3,7 @@
#
# This docker target builds on the debian Stretch base image.
#
-FROM qemu:debian9
+FROM qemu:debian10
MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
# Add the foreign architecture we want and install dependencies
diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 4a20af6fe1..e163b8b956 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -3,7 +3,7 @@
#
# This docker target builds on the debian Stretch base image.
#
-FROM qemu:debian9
+FROM qemu:debian10
# Add the foreign architecture we want and install dependencies
RUN dpkg --add-architecture armhf
diff --git a/tests/docker/dockerfiles/debian-mips64el-cross.docker b/tests/docker/dockerfiles/debian-mips64el-cross.docker
index 2fca112405..453b53ef72 100644
--- a/tests/docker/dockerfiles/debian-mips64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-mips64el-cross.docker
@@ -4,7 +4,7 @@
# This docker target builds on the debian Stretch base image.
#
-FROM qemu:debian9
+FROM qemu:debian10
MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
index 4abf7832ac..3b6e975c68 100644
--- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -3,7 +3,7 @@
#
# This docker target builds on the debian Stretch base image.
#
-FROM qemu:debian9
+FROM qemu:debian10
MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
diff --git a/tests/docker/dockerfiles/debian-ppc64el-cross.docker b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
index 9973df9ff7..cd386f01d9 100644
--- a/tests/docker/dockerfiles/debian-ppc64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
@@ -3,7 +3,7 @@
#
# This docker target builds on the debian Stretch base image.
#
-FROM qemu:debian9
+FROM qemu:debian10
# Add the foreign architecture we want and install dependencies
RUN dpkg --add-architecture ppc64el && \
diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index eb73c98855..43fe59836f 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -3,7 +3,7 @@
#
# This docker target builds on the debian Stretch base image.
#
-FROM qemu:debian9
+FROM qemu:debian10
# Add the s390x architecture
RUN dpkg --add-architecture s390x
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 02/12] tests/docker: better handle symlinked libs
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
2020-01-30 11:32 ` [PATCH v2 01/12] tests/docker: move most cross compilers to buster base Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-31 15:58 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 03/12] gitlab-ci: Refresh the list of iotests Alex Bennée
` (9 subsequent siblings)
11 siblings, 1 reply; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Alex Bennée, richard.henderson,
f4bug, Philippe Mathieu-Daudé,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
When we are copying we want to ensure we grab the first
resolution (the found in path section). However even that binary might
be a symlink so lets make sure we chase the symlinks to copy the right
binary to where it can be found.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
tests/docker/docker.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 31d8adf836..7dfca63fe4 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -109,7 +109,7 @@ def _get_so_libs(executable):
ensure theright data is copied."""
libs = []
- ldd_re = re.compile(r"(/.*/)(\S*)")
+ ldd_re = re.compile(r"(?:\S+ => )?(\S*) \(:?0x[0-9a-f]+\)")
try:
ldd_output = subprocess.check_output(["ldd", executable]).decode('utf-8')
for line in ldd_output.split("\n"):
@@ -145,7 +145,8 @@ def _copy_binary_with_libs(src, bin_dest, dest_dir):
if libs:
for l in libs:
so_path = os.path.dirname(l)
- _copy_with_mkdir(l, dest_dir, so_path)
+ real_l = os.path.realpath(l)
+ _copy_with_mkdir(real_l, dest_dir, so_path)
def _check_binfmt_misc(executable):
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 03/12] gitlab-ci: Refresh the list of iotests
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
2020-01-30 11:32 ` [PATCH v2 01/12] tests/docker: move most cross compilers to buster base Alex Bennée
2020-01-30 11:32 ` [PATCH v2 02/12] tests/docker: better handle symlinked libs Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-30 11:32 ` [PATCH v2 04/12] travis.yml: Install genisoimage package Alex Bennée
` (8 subsequent siblings)
11 siblings, 0 replies; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, berrange, stefanb, richard.henderson, f4bug,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
From: Thomas Huth <thuth@redhat.com>
iotest 147 and 205 have recently been marked as "NBD-only", so they
are currently simply skipped and thus can be removed.
iotest 129 occasionally fails in the gitlab-CI, and according to Max,
there are some known issues with this test (see for example this URL:
https://lists.nongnu.org/archive/html/qemu-block/2019-06/msg00499.html ),
so for the time being, let's disable it until the problems are fixed.
The iotests 040, 127, 203 and 256 are scheduled to become part of "make
check-block", so we also do not have to test them seperately here anymore.
On the other side, new iotests have been added to the QEMU repository
in the past months, so we can now add some new test > 256 instead.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200121131936.8214-1-thuth@redhat.com>
---
.gitlab-ci.yml | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 228783993e..c15e394f09 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -54,12 +54,12 @@ build-tcg-disabled:
- make check-qapi-schema
- cd tests/qemu-iotests/
- ./check -raw 001 002 003 004 005 008 009 010 011 012 021 025 032 033 048
- 052 063 077 086 101 104 106 113 147 148 150 151 152 157 159 160
- 163 170 171 183 184 192 194 197 205 208 215 221 222 226 227 236
- - ./check -qcow2 028 040 051 056 057 058 065 067 068 082 085 091 095 096 102
- 122 124 127 129 132 139 142 144 145 147 151 152 155 157 165 194
- 196 197 200 202 203 205 208 209 215 216 218 222 227 234 246 247
- 248 250 254 255 256
+ 052 063 077 086 101 104 106 113 148 150 151 152 157 159 160 163
+ 170 171 183 184 192 194 197 208 215 221 222 226 227 236 253 277
+ - ./check -qcow2 028 051 056 057 058 065 067 068 082 085 091 095 096 102 122
+ 124 132 139 142 144 145 151 152 155 157 165 194 196 197 200 202
+ 208 209 215 216 218 222 227 234 246 247 248 250 254 255 257 258
+ 260 261 262 263 264 270 272 273 277 279
build-user:
script:
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 04/12] travis.yml: Install genisoimage package
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
` (2 preceding siblings ...)
2020-01-30 11:32 ` [PATCH v2 03/12] gitlab-ci: Refresh the list of iotests Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-30 19:04 ` Thomas Huth
2020-01-31 15:56 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 05/12] .shippable: --disable-docs for cross-compile tests Alex Bennée
` (7 subsequent siblings)
11 siblings, 2 replies; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Alex Bennée, richard.henderson,
f4bug, Wainer dos Santos Moschetta, Philippe Mathieu-Daudé,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
From: Wainer dos Santos Moschetta <wainersm@redhat.com>
The genisoimage program is required for tests/cdrom-test
tests, otherwise they are skipped. The current Travis
environments do not provide it by default, so let's
explicitly require the genisoimage package.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20200110191254.11303-2-wainersm@redhat.com>
---
.travis.yml | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/.travis.yml b/.travis.yml
index 1ae645e9fc..e75c979b6c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -51,6 +51,8 @@ addons:
- sparse
- uuid-dev
- gcovr
+ # Tests dependencies
+ - genisoimage
# The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu
@@ -383,6 +385,8 @@ matrix:
- libusb-1.0-0-dev
- libvdeplug-dev
- libvte-2.91-dev
+ # Tests dependencies
+ - genisoimage
env:
- TEST_CMD="make check check-tcg V=1"
- CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS}"
@@ -412,6 +416,8 @@ matrix:
- libusb-1.0-0-dev
- libvdeplug-dev
- libvte-2.91-dev
+ # Tests dependencies
+ - genisoimage
env:
- TEST_CMD="make check check-tcg V=1"
- CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},ppc64le-linux-user"
@@ -441,6 +447,8 @@ matrix:
- libusb-1.0-0-dev
- libvdeplug-dev
- libvte-2.91-dev
+ # Tests dependencies
+ - genisoimage
env:
- TEST_CMD="make check check-tcg V=1"
- CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},s390x-linux-user"
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 05/12] .shippable: --disable-docs for cross-compile tests
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
` (3 preceding siblings ...)
2020-01-30 11:32 ` [PATCH v2 04/12] travis.yml: Install genisoimage package Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-31 15:23 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 06/12] hw/hppa/Kconfig: LASI chipset requires PARALLEL port Alex Bennée
` (6 subsequent siblings)
11 siblings, 1 reply; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Alex Bennée, richard.henderson,
f4bug, Philippe Mathieu-Daudé,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
The sphinx support is fairly new and we don't seem to have all the
bugs worked out for cross development environments right now.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.shippable.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.shippable.yml b/.shippable.yml
index 83aae08bb4..2cce7b5689 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -37,5 +37,5 @@ build:
- unset CC
- mkdir build
- cd build
- - ../configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
+ - ../configure --disable-docs ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
- make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 06/12] hw/hppa/Kconfig: LASI chipset requires PARALLEL port
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
` (4 preceding siblings ...)
2020-01-30 11:32 ` [PATCH v2 05/12] .shippable: --disable-docs for cross-compile tests Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-30 11:32 ` [PATCH v2 07/12] .travis.yml: Drop superfluous use of --python=python3 parameter Alex Bennée
` (5 subsequent siblings)
11 siblings, 0 replies; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Philippe Mathieu-Daudé,
Helge Deller, richard.henderson, f4bug, Alex Bennée, cota,
stefanha, marcandre.lureau, pbonzini, aurelien,
Richard Henderson
From: Philippe Mathieu-Daudé <philmd@redhat.com>
The PARISC Lasi chipset emulation requires some of the common parallel
support and fails to build on a --without-default-devices:
LINK hppa-softmmu/qemu-system-hppa
/usr/bin/ld: hw/hppa/lasi.o: in function `lasi_init':
hw/hppa/lasi.c:324: undefined reference to `parallel_mm_init'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:206: qemu-system-hppa] Error 1
Fixes: 376b851909d
Reported-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Helge Deller <deller@gmx.de>
Message-Id: <20200129192350.27143-1-philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
hw/hppa/Kconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/hw/hppa/Kconfig b/hw/hppa/Kconfig
index 82178c7dcb..22948db025 100644
--- a/hw/hppa/Kconfig
+++ b/hw/hppa/Kconfig
@@ -12,4 +12,5 @@ config DINO
select LSI_SCSI_PCI
select LASI_82596
select LASIPS2
+ select PARALLEL
select ARTIST
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 07/12] .travis.yml: Drop superfluous use of --python=python3 parameter
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
` (5 preceding siblings ...)
2020-01-30 11:32 ` [PATCH v2 06/12] hw/hppa/Kconfig: LASI chipset requires PARALLEL port Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-30 11:32 ` [PATCH v2 08/12] .travis.yml: Add description to each job Alex Bennée
` (4 subsequent siblings)
11 siblings, 0 replies; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, Thomas Huth, berrange, stefanb, Alex Bennée,
richard.henderson, f4bug, Philippe Mathieu-Daudé,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
From: Philippe Mathieu-Daudé <f4bug@amsat.org>
As we require Python3 since commit ddf9069963, we don't need to
explicit it with the --python=/usr/bin/python3 configure option.
Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200125184217.30034-1-f4bug@amsat.org>
---
.travis.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.travis.yml b/.travis.yml
index e75c979b6c..7526967847 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -270,7 +270,7 @@ matrix:
# Acceptance (Functional) tests
- env:
- - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
+ - CONFIG="--target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
- TEST_CMD="make check-acceptance"
after_script:
- python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 08/12] .travis.yml: Add description to each job
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
` (6 preceding siblings ...)
2020-01-30 11:32 ` [PATCH v2 07/12] .travis.yml: Drop superfluous use of --python=python3 parameter Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-30 19:06 ` Thomas Huth
2020-01-30 11:32 ` [PATCH v2 09/12] .travis.yml: build documents under bionic Alex Bennée
` (3 subsequent siblings)
11 siblings, 1 reply; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Alex Bennée, richard.henderson,
f4bug, Philippe Mathieu-Daudé,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
From: Philippe Mathieu-Daudé <f4bug@amsat.org>
The NAME variable can be used to describe nicely a job (see [*]).
As we currently have 32 jobs, use it. This helps for quickly
finding a particular job.
before: https://travis-ci.org/qemu/qemu/builds/639887646
after: https://travis-ci.org/philmd/qemu/builds/641795043
[*] https://docs.travis-ci.com/user/customizing-the-build/#naming-jobs-within-matrices
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200125183135.28317-1-f4bug@amsat.org>
---
[ajb]
- drop [x86]
---
.travis.yml | 101 ++++++++++++++++++++++++++++++++++------------------
1 file changed, 67 insertions(+), 34 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 7526967847..056db1adac 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -96,24 +96,28 @@ after_script:
matrix:
include:
- - env:
+ - name: "GCC static (user)"
+ env:
- CONFIG="--disable-system --static"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
# we split the system builds as it takes a while to build them all
- - env:
+ - name: "GCC (main-softmmu)"
+ env:
- CONFIG="--disable-user --target-list=${MAIN_SOFTMMU_TARGETS}"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
- - env:
- - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
+ - name: "GCC (other-softmmu)"
+ env:
+ - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
# Just build tools and run minimal unit and softfloat checks
- - env:
+ - name: "GCC check-softfloat (user)"
+ env:
- BASE_CONFIG="--enable-tools"
- CONFIG="--disable-user --disable-system"
- TEST_CMD="make check-unit check-softfloat -j3"
@@ -121,41 +125,48 @@ matrix:
# --enable-debug implies --enable-debug-tcg, also runs quite a bit slower
- - env:
+ - name: "GCC debug (main-softmmu)"
+ env:
- CONFIG="--enable-debug --target-list=${MAIN_SOFTMMU_TARGETS}"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
# TCG debug can be run just on its own and is mostly agnostic to user/softmmu distinctions
- - env:
+ - name: "GCC debug (user)"
+ env:
- CONFIG="--enable-debug-tcg --disable-system"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
- - env:
+ - name: "GCC some libs disabled (main-softmmu)"
+ env:
- CONFIG="--disable-linux-aio --disable-cap-ng --disable-attr --disable-brlapi --disable-libusb --disable-replication --target-list=${MAIN_SOFTMMU_TARGETS}"
# Module builds are mostly of interest to major distros
- - env:
+ - name: "GCC modules (main-softmmu)"
+ env:
- CONFIG="--enable-modules --target-list=${MAIN_SOFTMMU_TARGETS}"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
# Alternate coroutines implementations are only really of interest to KVM users
# However we can't test against KVM on Travis so we can only run unit tests
- - env:
+ - name: "check-unit coroutine=ucontext"
+ env:
- CONFIG="--with-coroutine=ucontext --disable-tcg"
- TEST_CMD="make check-unit -j3 V=1"
- - env:
+ - name: "check-unit coroutine=sigaltstack"
+ env:
- CONFIG="--with-coroutine=sigaltstack --disable-tcg"
- TEST_CMD="make check-unit -j3 V=1"
# Check we can build docs and tools (out of tree)
- - env:
+ - name: "[x86] tools and docs"
+ env:
- BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.."
- BASE_CONFIG="--enable-tools --enable-docs"
- CONFIG="--target-list=x86_64-softmmu,aarch64-linux-user"
@@ -169,13 +180,15 @@ matrix:
# Test with Clang for compile portability (Travis uses clang-5.0)
- - env:
+ - name: "Clang (user)"
+ env:
- CONFIG="--disable-system"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
compiler: clang
- - env:
+ - name: "Clang (main-softmmu)"
+ env:
- CONFIG="--target-list=${MAIN_SOFTMMU_TARGETS} "
- CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-sanitize"
compiler: clang
@@ -184,52 +197,60 @@ matrix:
- ${SRC_DIR}/configure ${CONFIG} --extra-cflags="-fsanitize=undefined -Werror" || { cat config.log && exit 1; }
- - env:
+ - name: "Clang (other-softmmu)"
+ env:
- CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-clang-default"
compiler: clang
# gprof/gcov are GCC features
- - env:
+ - name: "GCC gprof/gcov"
+ env:
- CONFIG="--enable-gprof --enable-gcov --disable-pie --target-list=${MAIN_SOFTMMU_TARGETS}"
after_success:
- ${SRC_DIR}/scripts/travis/coverage-summary.sh
# We manually include builds which we disable "make check" for
- - env:
+ - name: "GCC without-default-devices (softmmu)"
+ env:
- CONFIG="--without-default-devices --disable-user"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
- TEST_CMD=""
# Check the TCG interpreter (TCI)
- - env:
+ - name: "GCC TCI"
+ env:
- CONFIG="--enable-debug-tcg --enable-tcg-interpreter --disable-kvm --disable-containers
--target-list=alpha-softmmu,arm-softmmu,hppa-softmmu,m68k-softmmu,microblaze-softmmu,moxie-softmmu,ppc-softmmu,s390x-softmmu,x86_64-softmmu"
- TEST_CMD="make check-qtest check-tcg V=1"
# We don't need to exercise every backend with every front-end
- - env:
+ - name: "GCC trace log,simple,syslog (user)"
+ env:
- CONFIG="--enable-trace-backends=log,simple,syslog --disable-system"
- TEST_CMD=""
- - env:
+ - name: "GCC trace ftrace (x86_64-softmmu)"
+ env:
- CONFIG="--enable-trace-backends=ftrace --target-list=x86_64-softmmu"
- TEST_CMD=""
- - env:
+ - name: "GCC trace ust (x86_64-softmmu)"
+ env:
- CONFIG="--enable-trace-backends=ust --target-list=x86_64-softmmu"
- TEST_CMD=""
# MacOSX builds - cirrus.yml also tests some MacOS builds including latest Xcode
- - env:
+ - name: "OSX Xcode 10.3"
+ env:
- CONFIG="--target-list=i386-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,x86_64-softmmu"
os: osx
osx_image: xcode10.3
@@ -252,7 +273,8 @@ matrix:
# Python builds
- - env:
+ - name: "GCC Python 3.5 (x86_64-softmmu)"
+ env:
- CONFIG="--target-list=x86_64-softmmu"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
language: python
@@ -260,7 +282,8 @@ matrix:
- "3.5"
- - env:
+ - name: "GCC Python 3.6 (x86_64-softmmu)"
+ env:
- CONFIG="--target-list=x86_64-softmmu"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
language: python
@@ -269,7 +292,8 @@ matrix:
# Acceptance (Functional) tests
- - env:
+ - name: "GCC check-acceptance"
+ env:
- CONFIG="--target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
- TEST_CMD="make check-acceptance"
after_script:
@@ -285,7 +309,8 @@ matrix:
# Using newer GCC with sanitizers
- - addons:
+ - name: "GCC9 with sanitizers (softmmu)"
+ addons:
apt:
update: true
sources:
@@ -333,7 +358,8 @@ matrix:
# Run check-tcg against linux-user
- - env:
+ - name: "GCC check-tcg (user)"
+ env:
- CONFIG="--disable-system --enable-debug-tcg"
- TEST_CMD="make -j3 check-tcg V=1"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
@@ -341,26 +367,30 @@ matrix:
# Run check-tcg against linux-user (with plugins)
# we skip sparc64-linux-user until it has been fixed somewhat
- - env:
+ - name: "GCC plugins check-tcg (user)"
+ env:
- CONFIG="--disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user"
- TEST_CMD="make -j3 check-tcg V=1"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
# Run check-tcg against softmmu targets
- - env:
+ - name: "GCC check-tcg (some-softmmu)"
+ env:
- CONFIG="--enable-debug-tcg --target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu"
- TEST_CMD="make -j3 check-tcg V=1"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
# Run check-tcg against softmmu targets (with plugins)
- - env:
+ - name: "GCC plugins check-tcg (some-softmmu)"
+ env:
- CONFIG="--enable-plugins --enable-debug-tcg --target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu"
- TEST_CMD="make -j3 check-tcg V=1"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
- - arch: arm64
+ - name: "[aarch64] GCC check-tcg"
+ arch: arm64
dist: xenial
addons:
apt_packages:
@@ -391,7 +421,8 @@ matrix:
- TEST_CMD="make check check-tcg V=1"
- CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS}"
- - arch: ppc64le
+ - name: "[ppc64] GCC check-tcg"
+ arch: ppc64le
dist: xenial
addons:
apt_packages:
@@ -422,7 +453,8 @@ matrix:
- TEST_CMD="make check check-tcg V=1"
- CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},ppc64le-linux-user"
- - arch: s390x
+ - name: "[s390x] GCC check-tcg"
+ arch: s390x
dist: bionic
addons:
apt_packages:
@@ -456,7 +488,8 @@ matrix:
# Release builds
# The make-release script expect a QEMU version, so our tag must start with a 'v'.
# This is the case when release candidate tags are created.
- - if: tag IS present AND tag =~ /^v\d+\.\d+(\.\d+)?(-\S*)?$/
+ - name: "Release tarball"
+ if: tag IS present AND tag =~ /^v\d+\.\d+(\.\d+)?(-\S*)?$/
env:
# We want to build from the release tarball
- BUILD_DIR="release/build/dir" SRC_DIR="../../.."
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 09/12] .travis.yml: build documents under bionic
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
` (7 preceding siblings ...)
2020-01-30 11:32 ` [PATCH v2 08/12] .travis.yml: Add description to each job Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-31 15:27 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 10/12] .travis.yml: move cache flushing to early common phase Alex Bennée
` (2 subsequent siblings)
11 siblings, 1 reply; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Alex Bennée, richard.henderson,
f4bug, Philippe Mathieu-Daudé,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
It looks like the xenial tooling doesn't like something in our setup.
We should probably be moving to bionic for everything soon
anyway (libssh aside).
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.travis.yml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/.travis.yml b/.travis.yml
index 056db1adac..f245765cfe 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -165,7 +165,8 @@ matrix:
# Check we can build docs and tools (out of tree)
- - name: "[x86] tools and docs"
+ - name: "tools and docs (bionic)"
+ dist: bionic
env:
- BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.."
- BASE_CONFIG="--enable-tools --enable-docs"
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 10/12] .travis.yml: move cache flushing to early common phase
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
` (8 preceding siblings ...)
2020-01-30 11:32 ` [PATCH v2 09/12] .travis.yml: build documents under bionic Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-31 15:31 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 11/12] .travis.yml: probe for number of available processors Alex Bennée
2020-01-30 11:32 ` [PATCH v2 12/12] .travis.yml: limit the arm64 target list Alex Bennée
11 siblings, 1 reply; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Alex Bennée, richard.henderson,
f4bug, Philippe Mathieu-Daudé,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
We shall be adding more common early setup in a future commit.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.travis.yml | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index f245765cfe..1b92f40eab 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -82,11 +82,16 @@ git:
# we want to do this ourselves
submodules: false
+# Common first phase for all steps
+before_install:
+ - if command -v ccache ; then ccache --zero-stats ; fi
+# Configure step - may be overridden
before_script:
- - if command -v ccache ; then ccache --zero-stats ; fi
- mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
- ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
+
+# Main build & test - rarely overridden - controlled by TEST_CMD
script:
- BUILD_RC=0 && make -j3 || BUILD_RC=$?
- if [ "$BUILD_RC" -eq 0 ] ; then travis_retry ${TEST_CMD} ; else $(exit $BUILD_RC); fi
@@ -268,7 +273,6 @@ matrix:
before_script:
- brew link --overwrite python
- export PATH="/usr/local/opt/ccache/libexec:$PATH"
- - if command -v ccache ; then ccache --zero-stats ; fi
- mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
- ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
@@ -499,9 +503,6 @@ matrix:
- TEST_CMD="make install -j3"
- QEMU_VERSION="${TRAVIS_TAG:1}"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
- before_script:
- - command -v ccache && ccache --zero-stats
- - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
script:
- make -C ${SRC_DIR} qemu-${QEMU_VERSION}.tar.bz2
- ls -l ${SRC_DIR}/qemu-${QEMU_VERSION}.tar.bz2
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 11/12] .travis.yml: probe for number of available processors
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
` (9 preceding siblings ...)
2020-01-30 11:32 ` [PATCH v2 10/12] .travis.yml: move cache flushing to early common phase Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-30 19:09 ` Thomas Huth
2020-01-31 15:51 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 12/12] .travis.yml: limit the arm64 target list Alex Bennée
11 siblings, 2 replies; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Alex Bennée, richard.henderson,
f4bug, Philippe Mathieu-Daudé,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
The arm64 hardware was especially hit by only building on 3 of the 32
available cores. Introduce a JOBS environment variable which we use
for all parallel builds. We still run the main checks single threaded
though so to make it easier to spot hangs.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.travis.yml | 20 +++++++++++---------
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 1b92f40eab..a600f508b0 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -85,6 +85,8 @@ git:
# Common first phase for all steps
before_install:
- if command -v ccache ; then ccache --zero-stats ; fi
+ - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
+ - echo "=== Using ${JOBS} simultaneous jobs ==="
# Configure step - may be overridden
before_script:
@@ -93,7 +95,7 @@ before_script:
# Main build & test - rarely overridden - controlled by TEST_CMD
script:
- - BUILD_RC=0 && make -j3 || BUILD_RC=$?
+ - BUILD_RC=0 && make -j${JOBS} || BUILD_RC=$?
- if [ "$BUILD_RC" -eq 0 ] ; then travis_retry ${TEST_CMD} ; else $(exit $BUILD_RC); fi
after_script:
- if command -v ccache ; then ccache --show-stats ; fi
@@ -125,7 +127,7 @@ matrix:
env:
- BASE_CONFIG="--enable-tools"
- CONFIG="--disable-user --disable-system"
- - TEST_CMD="make check-unit check-softfloat -j3"
+ - TEST_CMD="make check-unit check-softfloat -j${JOBS}"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
@@ -160,13 +162,13 @@ matrix:
- name: "check-unit coroutine=ucontext"
env:
- CONFIG="--with-coroutine=ucontext --disable-tcg"
- - TEST_CMD="make check-unit -j3 V=1"
+ - TEST_CMD="make check-unit -j${JOBS} V=1"
- name: "check-unit coroutine=sigaltstack"
env:
- CONFIG="--with-coroutine=sigaltstack --disable-tcg"
- - TEST_CMD="make check-unit -j3 V=1"
+ - TEST_CMD="make check-unit -j${JOBS} V=1"
# Check we can build docs and tools (out of tree)
@@ -366,7 +368,7 @@ matrix:
- name: "GCC check-tcg (user)"
env:
- CONFIG="--disable-system --enable-debug-tcg"
- - TEST_CMD="make -j3 check-tcg V=1"
+ - TEST_CMD="make -j${JOBS} check-tcg V=1"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
@@ -375,7 +377,7 @@ matrix:
- name: "GCC plugins check-tcg (user)"
env:
- CONFIG="--disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user"
- - TEST_CMD="make -j3 check-tcg V=1"
+ - TEST_CMD="make -j${JOBS} check-tcg V=1"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
@@ -383,7 +385,7 @@ matrix:
- name: "GCC check-tcg (some-softmmu)"
env:
- CONFIG="--enable-debug-tcg --target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu"
- - TEST_CMD="make -j3 check-tcg V=1"
+ - TEST_CMD="make -j${JOBS} check-tcg V=1"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
@@ -391,7 +393,7 @@ matrix:
- name: "GCC plugins check-tcg (some-softmmu)"
env:
- CONFIG="--enable-plugins --enable-debug-tcg --target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu"
- - TEST_CMD="make -j3 check-tcg V=1"
+ - TEST_CMD="make -j${JOBS} check-tcg V=1"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
- name: "[aarch64] GCC check-tcg"
@@ -500,7 +502,7 @@ matrix:
- BUILD_DIR="release/build/dir" SRC_DIR="../../.."
- BASE_CONFIG="--prefix=$PWD/dist"
- CONFIG="--target-list=x86_64-softmmu,aarch64-softmmu,armeb-linux-user,ppc-linux-user"
- - TEST_CMD="make install -j3"
+ - TEST_CMD="make install -j${JOBS}"
- QEMU_VERSION="${TRAVIS_TAG:1}"
- CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
script:
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [PATCH v2 12/12] .travis.yml: limit the arm64 target list
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
` (10 preceding siblings ...)
2020-01-30 11:32 ` [PATCH v2 11/12] .travis.yml: probe for number of available processors Alex Bennée
@ 2020-01-30 11:32 ` Alex Bennée
2020-01-30 19:43 ` Thomas Huth
2020-01-31 15:55 ` Philippe Mathieu-Daudé
11 siblings, 2 replies; 30+ messages in thread
From: Alex Bennée @ 2020-01-30 11:32 UTC (permalink / raw)
To: qemu-devel
Cc: fam, berrange, stefanb, Alex Bennée, richard.henderson,
f4bug, Philippe Mathieu-Daudé,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
Even though we can do a good multi-threaded build we are still
exceeding the 50 minute build time while running the tests.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
.travis.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.travis.yml b/.travis.yml
index a600f508b0..71bf650b1d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -426,7 +426,7 @@ matrix:
- genisoimage
env:
- TEST_CMD="make check check-tcg V=1"
- - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS}"
+ - CONFIG="--disable-containers --target-list=aarch64-softmmu,aarch64-linux-user,arm-softmmu,arm-linux-user"
- name: "[ppc64] GCC check-tcg"
arch: ppc64le
--
2.20.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [PATCH v2 04/12] travis.yml: Install genisoimage package
2020-01-30 11:32 ` [PATCH v2 04/12] travis.yml: Install genisoimage package Alex Bennée
@ 2020-01-30 19:04 ` Thomas Huth
2020-01-31 15:56 ` Philippe Mathieu-Daudé
1 sibling, 0 replies; 30+ messages in thread
From: Thomas Huth @ 2020-01-30 19:04 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug,
Wainer dos Santos Moschetta, cota, stefanha, pbonzini,
marcandre.lureau, Philippe Mathieu-Daudé,
aurelien
On 30/01/2020 12.32, Alex Bennée wrote:
> From: Wainer dos Santos Moschetta <wainersm@redhat.com>
>
> The genisoimage program is required for tests/cdrom-test
> tests, otherwise they are skipped. The current Travis
> environments do not provide it by default, so let's
> explicitly require the genisoimage package.
>
> Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Message-Id: <20200110191254.11303-2-wainersm@redhat.com>
> ---
> .travis.yml | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/.travis.yml b/.travis.yml
> index 1ae645e9fc..e75c979b6c 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -51,6 +51,8 @@ addons:
> - sparse
> - uuid-dev
> - gcovr
> + # Tests dependencies
> + - genisoimage
>
>
> # The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu
> @@ -383,6 +385,8 @@ matrix:
> - libusb-1.0-0-dev
> - libvdeplug-dev
> - libvte-2.91-dev
> + # Tests dependencies
> + - genisoimage
> env:
> - TEST_CMD="make check check-tcg V=1"
> - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS}"
> @@ -412,6 +416,8 @@ matrix:
> - libusb-1.0-0-dev
> - libvdeplug-dev
> - libvte-2.91-dev
> + # Tests dependencies
> + - genisoimage
> env:
> - TEST_CMD="make check check-tcg V=1"
> - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},ppc64le-linux-user"
> @@ -441,6 +447,8 @@ matrix:
> - libusb-1.0-0-dev
> - libvdeplug-dev
> - libvte-2.91-dev
> + # Tests dependencies
> + - genisoimage
> env:
> - TEST_CMD="make check check-tcg V=1"
> - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},s390x-linux-user"
>
Reviewed-by: Thomas Huth <thuth@redhat.com>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 08/12] .travis.yml: Add description to each job
2020-01-30 11:32 ` [PATCH v2 08/12] .travis.yml: Add description to each job Alex Bennée
@ 2020-01-30 19:06 ` Thomas Huth
2020-01-31 15:24 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 30+ messages in thread
From: Thomas Huth @ 2020-01-30 19:06 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, cota, stefanha,
pbonzini, marcandre.lureau, Philippe Mathieu-Daudé,
aurelien
On 30/01/2020 12.32, Alex Bennée wrote:
> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
>
> The NAME variable can be used to describe nicely a job (see [*]).
> As we currently have 32 jobs, use it. This helps for quickly
> finding a particular job.
>
> before: https://travis-ci.org/qemu/qemu/builds/639887646
> after: https://travis-ci.org/philmd/qemu/builds/641795043
>
> [*] https://docs.travis-ci.com/user/customizing-the-build/#naming-jobs-within-matrices
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Message-Id: <20200125183135.28317-1-f4bug@amsat.org>
>
> ---
> [ajb]
> - drop [x86]
> ---
> .travis.yml | 101 ++++++++++++++++++++++++++++++++++------------------
> 1 file changed, 67 insertions(+), 34 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 7526967847..056db1adac 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -96,24 +96,28 @@ after_script:
>
> matrix:
> include:
> - - env:
> + - name: "GCC static (user)"
> + env:
> - CONFIG="--disable-system --static"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>
>
> # we split the system builds as it takes a while to build them all
> - - env:
> + - name: "GCC (main-softmmu)"
> + env:
> - CONFIG="--disable-user --target-list=${MAIN_SOFTMMU_TARGETS}"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>
>
> - - env:
> - - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
> + - name: "GCC (other-softmmu)"
> + env:
> + - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>
>
> # Just build tools and run minimal unit and softfloat checks
> - - env:
> + - name: "GCC check-softfloat (user)"
> + env:
> - BASE_CONFIG="--enable-tools"
> - CONFIG="--disable-user --disable-system"
> - TEST_CMD="make check-unit check-softfloat -j3"
> @@ -121,41 +125,48 @@ matrix:
>
>
> # --enable-debug implies --enable-debug-tcg, also runs quite a bit slower
> - - env:
> + - name: "GCC debug (main-softmmu)"
> + env:
> - CONFIG="--enable-debug --target-list=${MAIN_SOFTMMU_TARGETS}"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
>
>
> # TCG debug can be run just on its own and is mostly agnostic to user/softmmu distinctions
> - - env:
> + - name: "GCC debug (user)"
> + env:
> - CONFIG="--enable-debug-tcg --disable-system"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
>
>
> - - env:
> + - name: "GCC some libs disabled (main-softmmu)"
> + env:
> - CONFIG="--disable-linux-aio --disable-cap-ng --disable-attr --disable-brlapi --disable-libusb --disable-replication --target-list=${MAIN_SOFTMMU_TARGETS}"
>
>
> # Module builds are mostly of interest to major distros
> - - env:
> + - name: "GCC modules (main-softmmu)"
> + env:
> - CONFIG="--enable-modules --target-list=${MAIN_SOFTMMU_TARGETS}"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>
>
> # Alternate coroutines implementations are only really of interest to KVM users
> # However we can't test against KVM on Travis so we can only run unit tests
> - - env:
> + - name: "check-unit coroutine=ucontext"
> + env:
> - CONFIG="--with-coroutine=ucontext --disable-tcg"
> - TEST_CMD="make check-unit -j3 V=1"
>
>
> - - env:
> + - name: "check-unit coroutine=sigaltstack"
> + env:
> - CONFIG="--with-coroutine=sigaltstack --disable-tcg"
> - TEST_CMD="make check-unit -j3 V=1"
>
>
> # Check we can build docs and tools (out of tree)
> - - env:
> + - name: "[x86] tools and docs"
Could you please also drop this "[x86]" here?
Thanks,
Thomas
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 11/12] .travis.yml: probe for number of available processors
2020-01-30 11:32 ` [PATCH v2 11/12] .travis.yml: probe for number of available processors Alex Bennée
@ 2020-01-30 19:09 ` Thomas Huth
2020-01-31 15:51 ` Philippe Mathieu-Daudé
1 sibling, 0 replies; 30+ messages in thread
From: Thomas Huth @ 2020-01-30 19:09 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, cota, stefanha,
pbonzini, marcandre.lureau, Philippe Mathieu-Daudé,
aurelien
On 30/01/2020 12.32, Alex Bennée wrote:
> The arm64 hardware was especially hit by only building on 3 of the 32
> available cores. Introduce a JOBS environment variable which we use
> for all parallel builds. We still run the main checks single threaded
> though so to make it easier to spot hangs.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> .travis.yml | 20 +++++++++++---------
> 1 file changed, 11 insertions(+), 9 deletions(-)
Reviewed-by: Thomas Huth <thuth@redhat.com>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 12/12] .travis.yml: limit the arm64 target list
2020-01-30 11:32 ` [PATCH v2 12/12] .travis.yml: limit the arm64 target list Alex Bennée
@ 2020-01-30 19:43 ` Thomas Huth
2020-01-31 15:55 ` Philippe Mathieu-Daudé
1 sibling, 0 replies; 30+ messages in thread
From: Thomas Huth @ 2020-01-30 19:43 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, cota, stefanha,
pbonzini, marcandre.lureau, Philippe Mathieu-Daudé,
aurelien
On 30/01/2020 12.32, Alex Bennée wrote:
> Even though we can do a good multi-threaded build we are still
> exceeding the 50 minute build time while running the tests.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> .travis.yml | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index a600f508b0..71bf650b1d 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -426,7 +426,7 @@ matrix:
> - genisoimage
> env:
> - TEST_CMD="make check check-tcg V=1"
> - - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS}"
> + - CONFIG="--disable-containers --target-list=aarch64-softmmu,aarch64-linux-user,arm-softmmu,arm-linux-user"
Maybe still include x86_64-softmmu? (I assume arm folks might be
interested in that?)
Thomas
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 01/12] tests/docker: move most cross compilers to buster base
2020-01-30 11:32 ` [PATCH v2 01/12] tests/docker: move most cross compilers to buster base Alex Bennée
@ 2020-01-31 15:22 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-01-31 15:22 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, cota, stefanha,
marcandre.lureau, pbonzini, aurelien
On 1/30/20 12:32 PM, Alex Bennée wrote:
> This includes fixing up the dependencies (Which were already wrong for
> one of the mips variants).
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> tests/docker/Makefile.include | 16 ++++++++--------
> tests/docker/dockerfiles/debian-amd64.docker | 2 +-
> .../docker/dockerfiles/debian-armel-cross.docker | 2 +-
> .../docker/dockerfiles/debian-armhf-cross.docker | 2 +-
> .../dockerfiles/debian-mips64el-cross.docker | 2 +-
> .../dockerfiles/debian-mipsel-cross.docker | 2 +-
> .../dockerfiles/debian-ppc64el-cross.docker | 2 +-
> .../docker/dockerfiles/debian-s390x-cross.docker | 2 +-
> 8 files changed, 15 insertions(+), 15 deletions(-)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 19dbe26169..43a8678688 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -91,19 +91,12 @@ endif
> # Enforce dependencies for composite images
> docker-image-debian9-mxe: docker-image-debian9
> ifeq ($(HOST_ARCH),x86_64)
> -docker-image-debian-amd64: docker-image-debian9
> +docker-image-debian-amd64: docker-image-debian10
> DOCKER_PARTIAL_IMAGES += debian-amd64-cross
> else
> docker-image-debian-amd64-cross: docker-image-debian10
> DOCKER_PARTIAL_IMAGES += debian-amd64
> endif
> -docker-image-debian-armel-cross: docker-image-debian9
> -docker-image-debian-armhf-cross: docker-image-debian9
> -docker-image-debian-mips-cross: docker-image-debian9
> -docker-image-debian-mipsel-cross: docker-image-debian9
> -docker-image-debian-mips64el-cross: docker-image-debian9
> -docker-image-debian-ppc64el-cross: docker-image-debian9
> -docker-image-debian-s390x-cross: docker-image-debian9
> docker-image-debian-win32-cross: docker-image-debian9-mxe
> docker-image-debian-win64-cross: docker-image-debian9-mxe
>
> @@ -118,12 +111,19 @@ endif
>
> docker-image-debian-alpha-cross: docker-image-debian10
> docker-image-debian-arm64-cross: docker-image-debian10
> +docker-image-debian-armel-cross: docker-image-debian10
> +docker-image-debian-armhf-cross: docker-image-debian10
> docker-image-debian-hppa-cross: docker-image-debian10
> docker-image-debian-m68k-cross: docker-image-debian10
> +docker-image-debian-mips-cross: docker-image-debian10
> docker-image-debian-mips64-cross: docker-image-debian10
> +docker-image-debian-mips64el-cross: docker-image-debian10
> +docker-image-debian-mipsel-cross: docker-image-debian10
> docker-image-debian-powerpc-cross: docker-image-debian10
> docker-image-debian-ppc64-cross: docker-image-debian10
> +docker-image-debian-ppc64el-cross: docker-image-debian10
> docker-image-debian-riscv64-cross: docker-image-debian10
> +docker-image-debian-s390x-cross: docker-image-debian10
> docker-image-debian-sh4-cross: docker-image-debian10
> docker-image-debian-sparc64-cross: docker-image-debian10
>
> diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
> index 431e947ebd..3b860af106 100644
> --- a/tests/docker/dockerfiles/debian-amd64.docker
> +++ b/tests/docker/dockerfiles/debian-amd64.docker
> @@ -4,7 +4,7 @@
> # This docker target builds on the debian Stretch base image. Further
> # libraries which are not widely available are installed by hand.
> #
> -FROM qemu:debian9
> +FROM qemu:debian10
> MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
>
> RUN apt update && \
> diff --git a/tests/docker/dockerfiles/debian-armel-cross.docker b/tests/docker/dockerfiles/debian-armel-cross.docker
> index 15378f8ea2..e3794a61c9 100644
> --- a/tests/docker/dockerfiles/debian-armel-cross.docker
> +++ b/tests/docker/dockerfiles/debian-armel-cross.docker
> @@ -3,7 +3,7 @@
> #
> # This docker target builds on the debian Stretch base image.
> #
> -FROM qemu:debian9
> +FROM qemu:debian10
> MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
>
> # Add the foreign architecture we want and install dependencies
> diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
> index 4a20af6fe1..e163b8b956 100644
> --- a/tests/docker/dockerfiles/debian-armhf-cross.docker
> +++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
> @@ -3,7 +3,7 @@
> #
> # This docker target builds on the debian Stretch base image.
> #
> -FROM qemu:debian9
> +FROM qemu:debian10
>
> # Add the foreign architecture we want and install dependencies
> RUN dpkg --add-architecture armhf
> diff --git a/tests/docker/dockerfiles/debian-mips64el-cross.docker b/tests/docker/dockerfiles/debian-mips64el-cross.docker
> index 2fca112405..453b53ef72 100644
> --- a/tests/docker/dockerfiles/debian-mips64el-cross.docker
> +++ b/tests/docker/dockerfiles/debian-mips64el-cross.docker
> @@ -4,7 +4,7 @@
> # This docker target builds on the debian Stretch base image.
> #
>
> -FROM qemu:debian9
> +FROM qemu:debian10
>
> MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
>
> diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
> index 4abf7832ac..3b6e975c68 100644
> --- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
> +++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
> @@ -3,7 +3,7 @@
> #
> # This docker target builds on the debian Stretch base image.
> #
> -FROM qemu:debian9
> +FROM qemu:debian10
>
> MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
>
> diff --git a/tests/docker/dockerfiles/debian-ppc64el-cross.docker b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
> index 9973df9ff7..cd386f01d9 100644
> --- a/tests/docker/dockerfiles/debian-ppc64el-cross.docker
> +++ b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
> @@ -3,7 +3,7 @@
> #
> # This docker target builds on the debian Stretch base image.
> #
> -FROM qemu:debian9
> +FROM qemu:debian10
>
> # Add the foreign architecture we want and install dependencies
> RUN dpkg --add-architecture ppc64el && \
> diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
> index eb73c98855..43fe59836f 100644
> --- a/tests/docker/dockerfiles/debian-s390x-cross.docker
> +++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
> @@ -3,7 +3,7 @@
> #
> # This docker target builds on the debian Stretch base image.
> #
> -FROM qemu:debian9
> +FROM qemu:debian10
>
> # Add the s390x architecture
> RUN dpkg --add-architecture s390x
>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 05/12] .shippable: --disable-docs for cross-compile tests
2020-01-30 11:32 ` [PATCH v2 05/12] .shippable: --disable-docs for cross-compile tests Alex Bennée
@ 2020-01-31 15:23 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-01-31 15:23 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, cota, stefanha,
marcandre.lureau, pbonzini, aurelien
On 1/30/20 12:32 PM, Alex Bennée wrote:
> The sphinx support is fairly new and we don't seem to have all the
> bugs worked out for cross development environments right now.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> .shippable.yml | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 83aae08bb4..2cce7b5689 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -37,5 +37,5 @@ build:
> - unset CC
> - mkdir build
> - cd build
> - - ../configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
> + - ../configure --disable-docs ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
> - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 08/12] .travis.yml: Add description to each job
2020-01-30 19:06 ` Thomas Huth
@ 2020-01-31 15:24 ` Philippe Mathieu-Daudé
2020-01-31 15:37 ` Thomas Huth
0 siblings, 1 reply; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-01-31 15:24 UTC (permalink / raw)
To: Thomas Huth, Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, cota, stefanha,
pbonzini, marcandre.lureau, aurelien
On 1/30/20 8:06 PM, Thomas Huth wrote:
> On 30/01/2020 12.32, Alex Bennée wrote:
>> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>
>> The NAME variable can be used to describe nicely a job (see [*]).
>> As we currently have 32 jobs, use it. This helps for quickly
>> finding a particular job.
>>
>> before: https://travis-ci.org/qemu/qemu/builds/639887646
>> after: https://travis-ci.org/philmd/qemu/builds/641795043
>>
>> [*] https://docs.travis-ci.com/user/customizing-the-build/#naming-jobs-within-matrices
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> Message-Id: <20200125183135.28317-1-f4bug@amsat.org>
>>
>> ---
>> [ajb]
>> - drop [x86]
>> ---
>> .travis.yml | 101 ++++++++++++++++++++++++++++++++++------------------
>> 1 file changed, 67 insertions(+), 34 deletions(-)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 7526967847..056db1adac 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -96,24 +96,28 @@ after_script:
>>
>> matrix:
>> include:
>> - - env:
>> + - name: "GCC static (user)"
>> + env:
>> - CONFIG="--disable-system --static"
>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>>
>>
>> # we split the system builds as it takes a while to build them all
>> - - env:
>> + - name: "GCC (main-softmmu)"
>> + env:
>> - CONFIG="--disable-user --target-list=${MAIN_SOFTMMU_TARGETS}"
>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>>
>>
>> - - env:
>> - - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
>> + - name: "GCC (other-softmmu)"
>> + env:
>> + - CONFIG="--disable-user --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>>
>>
>> # Just build tools and run minimal unit and softfloat checks
>> - - env:
>> + - name: "GCC check-softfloat (user)"
>> + env:
>> - BASE_CONFIG="--enable-tools"
>> - CONFIG="--disable-user --disable-system"
>> - TEST_CMD="make check-unit check-softfloat -j3"
>> @@ -121,41 +125,48 @@ matrix:
>>
>>
>> # --enable-debug implies --enable-debug-tcg, also runs quite a bit slower
>> - - env:
>> + - name: "GCC debug (main-softmmu)"
>> + env:
>> - CONFIG="--enable-debug --target-list=${MAIN_SOFTMMU_TARGETS}"
>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
>>
>>
>> # TCG debug can be run just on its own and is mostly agnostic to user/softmmu distinctions
>> - - env:
>> + - name: "GCC debug (user)"
>> + env:
>> - CONFIG="--enable-debug-tcg --disable-system"
>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
>>
>>
>> - - env:
>> + - name: "GCC some libs disabled (main-softmmu)"
>> + env:
>> - CONFIG="--disable-linux-aio --disable-cap-ng --disable-attr --disable-brlapi --disable-libusb --disable-replication --target-list=${MAIN_SOFTMMU_TARGETS}"
>>
>>
>> # Module builds are mostly of interest to major distros
>> - - env:
>> + - name: "GCC modules (main-softmmu)"
>> + env:
>> - CONFIG="--enable-modules --target-list=${MAIN_SOFTMMU_TARGETS}"
>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>>
>>
>> # Alternate coroutines implementations are only really of interest to KVM users
>> # However we can't test against KVM on Travis so we can only run unit tests
>> - - env:
>> + - name: "check-unit coroutine=ucontext"
>> + env:
>> - CONFIG="--with-coroutine=ucontext --disable-tcg"
>> - TEST_CMD="make check-unit -j3 V=1"
>>
>>
>> - - env:
>> + - name: "check-unit coroutine=sigaltstack"
>> + env:
>> - CONFIG="--with-coroutine=sigaltstack --disable-tcg"
>> - TEST_CMD="make check-unit -j3 V=1"
>>
>>
>> # Check we can build docs and tools (out of tree)
>> - - env:
>> + - name: "[x86] tools and docs"
>
> Could you please also drop this "[x86]" here?
The hunk got applied into the next patch:
# Check we can build docs and tools (out of tree)
- - name: "[x86] tools and docs"
+ - name: "tools and docs (bionic)"
+ dist: bionic
env:
- BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.."
- BASE_CONFIG="--enable-tools --enable-docs"
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 09/12] .travis.yml: build documents under bionic
2020-01-30 11:32 ` [PATCH v2 09/12] .travis.yml: build documents under bionic Alex Bennée
@ 2020-01-31 15:27 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-01-31 15:27 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, cota, stefanha,
marcandre.lureau, pbonzini, aurelien
On 1/30/20 12:32 PM, Alex Bennée wrote:
> It looks like the xenial tooling doesn't like something in our setup.
> We should probably be moving to bionic for everything soon
> anyway (libssh aside).
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> .travis.yml | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 056db1adac..f245765cfe 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -165,7 +165,8 @@ matrix:
>
>
> # Check we can build docs and tools (out of tree)
> - - name: "[x86] tools and docs"
> + - name: "tools and docs (bionic)"
As noticed Thomas, it would be clearer to remove '[x86]' in the previous
patch, and add '(bionic)' description in this one.
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> + dist: bionic
> env:
> - BUILD_DIR="out-of-tree/build/dir" SRC_DIR="../../.."
> - BASE_CONFIG="--enable-tools --enable-docs"
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 10/12] .travis.yml: move cache flushing to early common phase
2020-01-30 11:32 ` [PATCH v2 10/12] .travis.yml: move cache flushing to early common phase Alex Bennée
@ 2020-01-31 15:31 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-01-31 15:31 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, cota, stefanha,
marcandre.lureau, pbonzini, aurelien
On 1/30/20 12:32 PM, Alex Bennée wrote:
> We shall be adding more common early setup in a future commit.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> .travis.yml | 11 ++++++-----
> 1 file changed, 6 insertions(+), 5 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index f245765cfe..1b92f40eab 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -82,11 +82,16 @@ git:
> # we want to do this ourselves
> submodules: false
>
> +# Common first phase for all steps
> +before_install:
> + - if command -v ccache ; then ccache --zero-stats ; fi
Odd, this rule is called 'before_install' but is run *after* the
addons:apt:packages list is installed.
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>
> +# Configure step - may be overridden
> before_script:
> - - if command -v ccache ; then ccache --zero-stats ; fi
> - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
> - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
> +
> +# Main build & test - rarely overridden - controlled by TEST_CMD
> script:
> - BUILD_RC=0 && make -j3 || BUILD_RC=$?
> - if [ "$BUILD_RC" -eq 0 ] ; then travis_retry ${TEST_CMD} ; else $(exit $BUILD_RC); fi
> @@ -268,7 +273,6 @@ matrix:
> before_script:
> - brew link --overwrite python
> - export PATH="/usr/local/opt/ccache/libexec:$PATH"
> - - if command -v ccache ; then ccache --zero-stats ; fi
> - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
> - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
>
> @@ -499,9 +503,6 @@ matrix:
> - TEST_CMD="make install -j3"
> - QEMU_VERSION="${TRAVIS_TAG:1}"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
> - before_script:
> - - command -v ccache && ccache --zero-stats
> - - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
> script:
> - make -C ${SRC_DIR} qemu-${QEMU_VERSION}.tar.bz2
> - ls -l ${SRC_DIR}/qemu-${QEMU_VERSION}.tar.bz2
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 08/12] .travis.yml: Add description to each job
2020-01-31 15:24 ` Philippe Mathieu-Daudé
@ 2020-01-31 15:37 ` Thomas Huth
0 siblings, 0 replies; 30+ messages in thread
From: Thomas Huth @ 2020-01-31 15:37 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, cota, stefanha,
pbonzini, marcandre.lureau, aurelien
On 31/01/2020 16.24, Philippe Mathieu-Daudé wrote:
> On 1/30/20 8:06 PM, Thomas Huth wrote:
>> On 30/01/2020 12.32, Alex Bennée wrote:
>>> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>
>>> The NAME variable can be used to describe nicely a job (see [*]).
>>> As we currently have 32 jobs, use it. This helps for quickly
>>> finding a particular job.
>>>
>>> before: https://travis-ci.org/qemu/qemu/builds/639887646
>>> after: https://travis-ci.org/philmd/qemu/builds/641795043
>>>
>>> [*]
>>> https://docs.travis-ci.com/user/customizing-the-build/#naming-jobs-within-matrices
>>>
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>> Message-Id: <20200125183135.28317-1-f4bug@amsat.org>
>>>
>>> ---
>>> [ajb]
>>> - drop [x86]
>>> ---
>>> .travis.yml | 101 ++++++++++++++++++++++++++++++++++------------------
>>> 1 file changed, 67 insertions(+), 34 deletions(-)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index 7526967847..056db1adac 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -96,24 +96,28 @@ after_script:
>>> matrix:
>>> include:
>>> - - env:
>>> + - name: "GCC static (user)"
>>> + env:
>>> - CONFIG="--disable-system --static"
>>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>>> # we split the system builds as it takes a while to build
>>> them all
>>> - - env:
>>> + - name: "GCC (main-softmmu)"
>>> + env:
>>> - CONFIG="--disable-user
>>> --target-list=${MAIN_SOFTMMU_TARGETS}"
>>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>>> - - env:
>>> - - CONFIG="--disable-user
>>> --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
>>> + - name: "GCC (other-softmmu)"
>>> + env:
>>> + - CONFIG="--disable-user
>>> --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
>>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>>> # Just build tools and run minimal unit and softfloat checks
>>> - - env:
>>> + - name: "GCC check-softfloat (user)"
>>> + env:
>>> - BASE_CONFIG="--enable-tools"
>>> - CONFIG="--disable-user --disable-system"
>>> - TEST_CMD="make check-unit check-softfloat -j3"
>>> @@ -121,41 +125,48 @@ matrix:
>>> # --enable-debug implies --enable-debug-tcg, also runs
>>> quite a bit slower
>>> - - env:
>>> + - name: "GCC debug (main-softmmu)"
>>> + env:
>>> - CONFIG="--enable-debug
>>> --target-list=${MAIN_SOFTMMU_TARGETS}"
>>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug"
>>> # TCG debug can be run just on its own and is mostly
>>> agnostic to user/softmmu distinctions
>>> - - env:
>>> + - name: "GCC debug (user)"
>>> + env:
>>> - CONFIG="--enable-debug-tcg --disable-system"
>>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
>>> - - env:
>>> + - name: "GCC some libs disabled (main-softmmu)"
>>> + env:
>>> - CONFIG="--disable-linux-aio --disable-cap-ng
>>> --disable-attr --disable-brlapi --disable-libusb
>>> --disable-replication --target-list=${MAIN_SOFTMMU_TARGETS}"
>>> # Module builds are mostly of interest to major distros
>>> - - env:
>>> + - name: "GCC modules (main-softmmu)"
>>> + env:
>>> - CONFIG="--enable-modules
>>> --target-list=${MAIN_SOFTMMU_TARGETS}"
>>> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>>> # Alternate coroutines implementations are only really of
>>> interest to KVM users
>>> # However we can't test against KVM on Travis so we can only
>>> run unit tests
>>> - - env:
>>> + - name: "check-unit coroutine=ucontext"
>>> + env:
>>> - CONFIG="--with-coroutine=ucontext --disable-tcg"
>>> - TEST_CMD="make check-unit -j3 V=1"
>>> - - env:
>>> + - name: "check-unit coroutine=sigaltstack"
>>> + env:
>>> - CONFIG="--with-coroutine=sigaltstack --disable-tcg"
>>> - TEST_CMD="make check-unit -j3 V=1"
>>> # Check we can build docs and tools (out of tree)
>>> - - env:
>>> + - name: "[x86] tools and docs"
>>
>> Could you please also drop this "[x86]" here?
>
> The hunk got applied into the next patch:
>
> # Check we can build docs and tools (out of tree)
> - - name: "[x86] tools and docs"
> + - name: "tools and docs (bionic)"
> + dist: bionic
Ok, fine for me, too.
Thomas
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 11/12] .travis.yml: probe for number of available processors
2020-01-30 11:32 ` [PATCH v2 11/12] .travis.yml: probe for number of available processors Alex Bennée
2020-01-30 19:09 ` Thomas Huth
@ 2020-01-31 15:51 ` Philippe Mathieu-Daudé
2020-02-01 22:13 ` Ed Vielmetti
1 sibling, 1 reply; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-01-31 15:51 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, Ed Vielmetti,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
(Cc'ing Ed Vielmetti)
On 1/30/20 12:32 PM, Alex Bennée wrote:
> The arm64 hardware was especially hit by only building on 3 of the 32
> available cores. Introduce a JOBS environment variable which we use
> for all parallel builds. We still run the main checks single threaded
> though so to make it easier to spot hangs.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> .travis.yml | 20 +++++++++++---------
> 1 file changed, 11 insertions(+), 9 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index 1b92f40eab..a600f508b0 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -85,6 +85,8 @@ git:
> # Common first phase for all steps
> before_install:
> - if command -v ccache ; then ccache --zero-stats ; fi
> + - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
Yeah finally!
Note, on the Cavium ThunderX CN88XX provided by Packet, Ed Vielmetti
once suggested to use the --load-average make option due to Amdahl's
law, and I noticed a minor speedup using -j96 -l47.5 (-l48 already
starts to decrease).
On https://docs.travis-ci.com/user/reference/overview/#linux I read
"LXD compliant OS images for arm64 are run in Packet."
Per
https://travis-ci.community/t/what-machine-s-does-travis-use-for-arm64/5579/2
the CPU seems to be a Ampere eMAG Skylark:
https://en.wikichip.org/wiki/apm/microarchitectures/skylark
Probably the eMAG 8180:
https://en.wikichip.org/wiki/ampere_computing/emag/8180
I don't know what would be the best limit for this CPU.
Back to this patch, it indeed reduced the build time by 2+, so:
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> + - echo "=== Using ${JOBS} simultaneous jobs ==="
>
> # Configure step - may be overridden
> before_script:
> @@ -93,7 +95,7 @@ before_script:
>
> # Main build & test - rarely overridden - controlled by TEST_CMD
> script:
> - - BUILD_RC=0 && make -j3 || BUILD_RC=$?
> + - BUILD_RC=0 && make -j${JOBS} || BUILD_RC=$?
> - if [ "$BUILD_RC" -eq 0 ] ; then travis_retry ${TEST_CMD} ; else $(exit $BUILD_RC); fi
> after_script:
> - if command -v ccache ; then ccache --show-stats ; fi
> @@ -125,7 +127,7 @@ matrix:
> env:
> - BASE_CONFIG="--enable-tools"
> - CONFIG="--disable-user --disable-system"
> - - TEST_CMD="make check-unit check-softfloat -j3"
> + - TEST_CMD="make check-unit check-softfloat -j${JOBS}"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
>
>
> @@ -160,13 +162,13 @@ matrix:
> - name: "check-unit coroutine=ucontext"
> env:
> - CONFIG="--with-coroutine=ucontext --disable-tcg"
> - - TEST_CMD="make check-unit -j3 V=1"
> + - TEST_CMD="make check-unit -j${JOBS} V=1"
>
>
> - name: "check-unit coroutine=sigaltstack"
> env:
> - CONFIG="--with-coroutine=sigaltstack --disable-tcg"
> - - TEST_CMD="make check-unit -j3 V=1"
> + - TEST_CMD="make check-unit -j${JOBS} V=1"
>
>
> # Check we can build docs and tools (out of tree)
> @@ -366,7 +368,7 @@ matrix:
> - name: "GCC check-tcg (user)"
> env:
> - CONFIG="--disable-system --enable-debug-tcg"
> - - TEST_CMD="make -j3 check-tcg V=1"
> + - TEST_CMD="make -j${JOBS} check-tcg V=1"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
>
>
> @@ -375,7 +377,7 @@ matrix:
> - name: "GCC plugins check-tcg (user)"
> env:
> - CONFIG="--disable-system --enable-plugins --enable-debug-tcg --target-list-exclude=sparc64-linux-user"
> - - TEST_CMD="make -j3 check-tcg V=1"
> + - TEST_CMD="make -j${JOBS} check-tcg V=1"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
>
>
> @@ -383,7 +385,7 @@ matrix:
> - name: "GCC check-tcg (some-softmmu)"
> env:
> - CONFIG="--enable-debug-tcg --target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu"
> - - TEST_CMD="make -j3 check-tcg V=1"
> + - TEST_CMD="make -j${JOBS} check-tcg V=1"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
>
>
> @@ -391,7 +393,7 @@ matrix:
> - name: "GCC plugins check-tcg (some-softmmu)"
> env:
> - CONFIG="--enable-plugins --enable-debug-tcg --target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu"
> - - TEST_CMD="make -j3 check-tcg V=1"
> + - TEST_CMD="make -j${JOBS} check-tcg V=1"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
>
> - name: "[aarch64] GCC check-tcg"
> @@ -500,7 +502,7 @@ matrix:
> - BUILD_DIR="release/build/dir" SRC_DIR="../../.."
> - BASE_CONFIG="--prefix=$PWD/dist"
> - CONFIG="--target-list=x86_64-softmmu,aarch64-softmmu,armeb-linux-user,ppc-linux-user"
> - - TEST_CMD="make install -j3"
> + - TEST_CMD="make install -j${JOBS}"
> - QEMU_VERSION="${TRAVIS_TAG:1}"
> - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
> script:
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 12/12] .travis.yml: limit the arm64 target list
2020-01-30 11:32 ` [PATCH v2 12/12] .travis.yml: limit the arm64 target list Alex Bennée
2020-01-30 19:43 ` Thomas Huth
@ 2020-01-31 15:55 ` Philippe Mathieu-Daudé
2020-01-31 16:14 ` Philippe Mathieu-Daudé
1 sibling, 1 reply; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-01-31 15:55 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug, cota, stefanha,
marcandre.lureau, pbonzini, aurelien
On 1/30/20 12:32 PM, Alex Bennée wrote:
> Even though we can do a good multi-threaded build we are still
> exceeding the 50 minute build time while running the tests.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> .travis.yml | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index a600f508b0..71bf650b1d 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -426,7 +426,7 @@ matrix:
> - genisoimage
> env:
> - TEST_CMD="make check check-tcg V=1"
> - - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS}"
> + - CONFIG="--disable-containers --target-list=aarch64-softmmu,aarch64-linux-user,arm-softmmu,arm-linux-user"
We should split this job in 2 using --target-list-exclude, like on
X86/debug:
env:
- CONFIG="--disable-containers
--target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
>
> - name: "[ppc64] GCC check-tcg"
> arch: ppc64le
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 04/12] travis.yml: Install genisoimage package
2020-01-30 11:32 ` [PATCH v2 04/12] travis.yml: Install genisoimage package Alex Bennée
2020-01-30 19:04 ` Thomas Huth
@ 2020-01-31 15:56 ` Philippe Mathieu-Daudé
1 sibling, 0 replies; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-01-31 15:56 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, stefanb, richard.henderson, f4bug,
Wainer dos Santos Moschetta, cota, stefanha, marcandre.lureau,
pbonzini, aurelien
On 1/30/20 12:32 PM, Alex Bennée wrote:
> From: Wainer dos Santos Moschetta <wainersm@redhat.com>
>
> The genisoimage program is required for tests/cdrom-test
> tests, otherwise they are skipped. The current Travis
> environments do not provide it by default, so let's
> explicitly require the genisoimage package.
>
> Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Message-Id: <20200110191254.11303-2-wainersm@redhat.com>
> ---
> .travis.yml | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/.travis.yml b/.travis.yml
> index 1ae645e9fc..e75c979b6c 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -51,6 +51,8 @@ addons:
> - sparse
> - uuid-dev
> - gcovr
> + # Tests dependencies
> + - genisoimage
>
>
> # The channel name "irc.oftc.net#qemu" is encrypted against qemu/qemu
> @@ -383,6 +385,8 @@ matrix:
> - libusb-1.0-0-dev
> - libvdeplug-dev
> - libvte-2.91-dev
> + # Tests dependencies
> + - genisoimage
> env:
> - TEST_CMD="make check check-tcg V=1"
> - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS}"
> @@ -412,6 +416,8 @@ matrix:
> - libusb-1.0-0-dev
> - libvdeplug-dev
> - libvte-2.91-dev
> + # Tests dependencies
> + - genisoimage
> env:
> - TEST_CMD="make check check-tcg V=1"
> - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},ppc64le-linux-user"
> @@ -441,6 +447,8 @@ matrix:
> - libusb-1.0-0-dev
> - libvdeplug-dev
> - libvte-2.91-dev
> + # Tests dependencies
> + - genisoimage
> env:
> - TEST_CMD="make check check-tcg V=1"
> - CONFIG="--disable-containers --target-list=${MAIN_SOFTMMU_TARGETS},s390x-linux-user"
>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 02/12] tests/docker: better handle symlinked libs
2020-01-30 11:32 ` [PATCH v2 02/12] tests/docker: better handle symlinked libs Alex Bennée
@ 2020-01-31 15:58 ` Philippe Mathieu-Daudé
2020-01-31 16:48 ` Alex Bennée
0 siblings, 1 reply; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-01-31 15:58 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, berrange, Robert Foley, stefanb, richard.henderson, f4bug,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
On 1/30/20 12:32 PM, Alex Bennée wrote:
> When we are copying we want to ensure we grab the first
> resolution (the found in path section). However even that binary might
> be a symlink so lets make sure we chase the symlinks to copy the right
> binary to where it can be found.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
> tests/docker/docker.py | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 31d8adf836..7dfca63fe4 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -109,7 +109,7 @@ def _get_so_libs(executable):
> ensure theright data is copied."""
>
> libs = []
> - ldd_re = re.compile(r"(/.*/)(\S*)")
> + ldd_re = re.compile(r"(?:\S+ => )?(\S*) \(:?0x[0-9a-f]+\)")
> try:
> ldd_output = subprocess.check_output(["ldd", executable]).decode('utf-8')
> for line in ldd_output.split("\n"):
> @@ -145,7 +145,8 @@ def _copy_binary_with_libs(src, bin_dest, dest_dir):
> if libs:
> for l in libs:
> so_path = os.path.dirname(l)
> - _copy_with_mkdir(l, dest_dir, so_path)
> + real_l = os.path.realpath(l)
> + _copy_with_mkdir(real_l, dest_dir, so_path)
>
>
> def _check_binfmt_misc(executable):
>
Rob raised an issue in this patch, it appears in a separated thread:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg675307.html
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 12/12] .travis.yml: limit the arm64 target list
2020-01-31 15:55 ` Philippe Mathieu-Daudé
@ 2020-01-31 16:14 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 30+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-01-31 16:14 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: fam, Thomas Huth, berrange, stefanb, richard.henderson, f4bug,
cota, stefanha, marcandre.lureau, pbonzini, aurelien
On 1/31/20 4:55 PM, Philippe Mathieu-Daudé wrote:
> On 1/30/20 12:32 PM, Alex Bennée wrote:
>> Even though we can do a good multi-threaded build we are still
>> exceeding the 50 minute build time while running the tests.
Oh now I remember, we are not using ccache on Aarch64 because Thomas
reported it was not working:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg650940.html
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>> .travis.yml | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index a600f508b0..71bf650b1d 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -426,7 +426,7 @@ matrix:
>> - genisoimage
>> env:
>> - TEST_CMD="make check check-tcg V=1"
>> - - CONFIG="--disable-containers
>> --target-list=${MAIN_SOFTMMU_TARGETS}"
>> + - CONFIG="--disable-containers
>> --target-list=aarch64-softmmu,aarch64-linux-user,arm-softmmu,arm-linux-user"
>>
>
> We should split this job in 2 using --target-list-exclude, like on
> X86/debug:
>
> env:
> - CONFIG="--disable-containers
> --target-list-exclude=${MAIN_SOFTMMU_TARGETS}"
>
>> - name: "[ppc64] GCC check-tcg"
>> arch: ppc64le
>>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 02/12] tests/docker: better handle symlinked libs
2020-01-31 15:58 ` Philippe Mathieu-Daudé
@ 2020-01-31 16:48 ` Alex Bennée
0 siblings, 0 replies; 30+ messages in thread
From: Alex Bennée @ 2020-01-31 16:48 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: fam, berrange, Robert Foley, stefanb, richard.henderson,
qemu-devel, f4bug, cota, stefanha, marcandre.lureau, pbonzini,
aurelien
Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> On 1/30/20 12:32 PM, Alex Bennée wrote:
>> When we are copying we want to ensure we grab the first
>> resolution (the found in path section). However even that binary might
>> be a symlink so lets make sure we chase the symlinks to copy the right
>> binary to where it can be found.
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>> tests/docker/docker.py | 5 +++--
>> 1 file changed, 3 insertions(+), 2 deletions(-)
>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>> index 31d8adf836..7dfca63fe4 100755
>> --- a/tests/docker/docker.py
>> +++ b/tests/docker/docker.py
>> @@ -109,7 +109,7 @@ def _get_so_libs(executable):
>> ensure theright data is copied."""
>> libs = []
>> - ldd_re = re.compile(r"(/.*/)(\S*)")
>> + ldd_re = re.compile(r"(?:\S+ => )?(\S*) \(:?0x[0-9a-f]+\)")
>> try:
>> ldd_output = subprocess.check_output(["ldd", executable]).decode('utf-8')
>> for line in ldd_output.split("\n"):
>> @@ -145,7 +145,8 @@ def _copy_binary_with_libs(src, bin_dest, dest_dir):
>> if libs:
>> for l in libs:
>> so_path = os.path.dirname(l)
>> - _copy_with_mkdir(l, dest_dir, so_path)
>> + real_l = os.path.realpath(l)
>> + _copy_with_mkdir(real_l, dest_dir, so_path)
>>
>> def _check_binfmt_misc(executable):
>>
>
> Rob raised an issue in this patch, it appears in a separated thread:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg675307.html
If fixed it up thusly:
def _get_so_libs(executable):
"""Return a list of libraries associated with an executable.
The paths may be symbolic links which would need to be resolved to
ensure the right data is copied."""
libs = []
ldd_re = re.compile(r"(?:\S+ => )?(\S*) \(:?0x[0-9a-f]+\)")
try:
ldd_output = subprocess.check_output(["ldd", executable]).decode('utf-8')
for line in ldd_output.split("\n"):
search = ldd_re.search(line)
if search:
try:
libs.append(s.group(1))
except IndexError:
pass
except subprocess.CalledProcessError:
print("%s had no associated libraries (static build?)" % (executable))
return libs
--
Alex Bennée
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH v2 11/12] .travis.yml: probe for number of available processors
2020-01-31 15:51 ` Philippe Mathieu-Daudé
@ 2020-02-01 22:13 ` Ed Vielmetti
0 siblings, 0 replies; 30+ messages in thread
From: Ed Vielmetti @ 2020-02-01 22:13 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Fam Zheng, berrange, stefanb, richard.henderson, qemu-devel,
f4bug, cota, stefanha, marcandre.lureau, Paolo Bonzini,
Alex Bennée, aurelien
[-- Attachment #1: Type: text/plain, Size: 7669 bytes --]
Thanks Philippe.
For reference, the two machine types in typical use in the Packet system
can be referenced as follows:
c1.large.arm - 96-core Cavium (Marvell) ThunderX
c2.large.arm - 32-core Ampere eMag
The Ampere data sheet from their OEM (Lenovo) is below.
https://amperecomputing.com/wp-content/uploads/2019/01/Lenovo_ThinkSystem_HR330A_20190118.pdf
Memory configuration of both systems is 128G so that should not be a need
to adjust for.
The Travis systems are indeed using the c2.large.arm builds within an LXD
container. There may be other limits provided to each container there, but
certainly you want to use more than 3 cores. The Travis machines are
specially configured with NVMe instead of SSD disk, which helps IO quite a
bit.
Configuring optimal builds across a variety of multi-core systems can be
hard to tune for optimal performance. Depending on the system - and they
vary a lot - you might or might not get more performance from adding more
cores, as sometimes you start to max out memory bandwidth. As a concrete
example, note that the Marvell ThunderX2 configuration will report 4
hardware threads per core to the system, making you think that you can do
`make -j 224`, but if you do timings on actual throughput often you'll be
better off with `make -j 56`, apparently due to scheduling leading to
contention between threads.
I concur with your decision to run the test suite single-threaded. Unless
the test environment has been designed from the start to use lots of cores,
my observation is that lots of real-world test suites have unavoidable
dependencies in test run order, and generally make optimistic assumptions
about machine state between tests.
Happy to be a resource for any other performance issues, and if you run
into anything fun I'm also happy to relate it to my friends at Ampere who
have been tracking perf and regressions across a wide set of operating and
open source packages. Also, I am always and forever looking for people who
have interest and tools and good intuition about how to make the best use
out of machines with many cores.
thanks
Ed
On Fri, Jan 31, 2020 at 10:52 AM Philippe Mathieu-Daudé <philmd@redhat.com>
wrote:
> (Cc'ing Ed Vielmetti)
>
> On 1/30/20 12:32 PM, Alex Bennée wrote:
> > The arm64 hardware was especially hit by only building on 3 of the 32
> > available cores. Introduce a JOBS environment variable which we use
> > for all parallel builds. We still run the main checks single threaded
> > though so to make it easier to spot hangs.
> >
> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > ---
> > .travis.yml | 20 +++++++++++---------
> > 1 file changed, 11 insertions(+), 9 deletions(-)
> >
> > diff --git a/.travis.yml b/.travis.yml
> > index 1b92f40eab..a600f508b0 100644
> > --- a/.travis.yml
> > +++ b/.travis.yml
> > @@ -85,6 +85,8 @@ git:
> > # Common first phase for all steps
> > before_install:
> > - if command -v ccache ; then ccache --zero-stats ; fi
> > + - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
>
> Yeah finally!
>
> Note, on the Cavium ThunderX CN88XX provided by Packet, Ed Vielmetti
> once suggested to use the --load-average make option due to Amdahl's
> law, and I noticed a minor speedup using -j96 -l47.5 (-l48 already
> starts to decrease).
>
> On https://docs.travis-ci.com/user/reference/overview/#linux I read
> "LXD compliant OS images for arm64 are run in Packet."
>
> Per
>
> https://travis-ci.community/t/what-machine-s-does-travis-use-for-arm64/5579/2
> the CPU seems to be a Ampere eMAG Skylark:
> https://en.wikichip.org/wiki/apm/microarchitectures/skylark
> Probably the eMAG 8180:
> https://en.wikichip.org/wiki/ampere_computing/emag/8180
>
> I don't know what would be the best limit for this CPU.
>
> Back to this patch, it indeed reduced the build time by 2+, so:
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>
> > + - echo "=== Using ${JOBS} simultaneous jobs ==="
> >
> > # Configure step - may be overridden
> > before_script:
> > @@ -93,7 +95,7 @@ before_script:
> >
> > # Main build & test - rarely overridden - controlled by TEST_CMD
> > script:
> > - - BUILD_RC=0 && make -j3 || BUILD_RC=$?
> > + - BUILD_RC=0 && make -j${JOBS} || BUILD_RC=$?
> > - if [ "$BUILD_RC" -eq 0 ] ; then travis_retry ${TEST_CMD} ; else
> $(exit $BUILD_RC); fi
> > after_script:
> > - if command -v ccache ; then ccache --show-stats ; fi
> > @@ -125,7 +127,7 @@ matrix:
> > env:
> > - BASE_CONFIG="--enable-tools"
> > - CONFIG="--disable-user --disable-system"
> > - - TEST_CMD="make check-unit check-softfloat -j3"
> > + - TEST_CMD="make check-unit check-softfloat -j${JOBS}"
> > - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
> >
> >
> > @@ -160,13 +162,13 @@ matrix:
> > - name: "check-unit coroutine=ucontext"
> > env:
> > - CONFIG="--with-coroutine=ucontext --disable-tcg"
> > - - TEST_CMD="make check-unit -j3 V=1"
> > + - TEST_CMD="make check-unit -j${JOBS} V=1"
> >
> >
> > - name: "check-unit coroutine=sigaltstack"
> > env:
> > - CONFIG="--with-coroutine=sigaltstack --disable-tcg"
> > - - TEST_CMD="make check-unit -j3 V=1"
> > + - TEST_CMD="make check-unit -j${JOBS} V=1"
> >
> >
> > # Check we can build docs and tools (out of tree)
> > @@ -366,7 +368,7 @@ matrix:
> > - name: "GCC check-tcg (user)"
> > env:
> > - CONFIG="--disable-system --enable-debug-tcg"
> > - - TEST_CMD="make -j3 check-tcg V=1"
> > + - TEST_CMD="make -j${JOBS} check-tcg V=1"
> > - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
> >
> >
> > @@ -375,7 +377,7 @@ matrix:
> > - name: "GCC plugins check-tcg (user)"
> > env:
> > - CONFIG="--disable-system --enable-plugins --enable-debug-tcg
> --target-list-exclude=sparc64-linux-user"
> > - - TEST_CMD="make -j3 check-tcg V=1"
> > + - TEST_CMD="make -j${JOBS} check-tcg V=1"
> > - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
> >
> >
> > @@ -383,7 +385,7 @@ matrix:
> > - name: "GCC check-tcg (some-softmmu)"
> > env:
> > - CONFIG="--enable-debug-tcg
> --target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu"
> > - - TEST_CMD="make -j3 check-tcg V=1"
> > + - TEST_CMD="make -j${JOBS} check-tcg V=1"
> > - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
> >
> >
> > @@ -391,7 +393,7 @@ matrix:
> > - name: "GCC plugins check-tcg (some-softmmu)"
> > env:
> > - CONFIG="--enable-plugins --enable-debug-tcg
> --target-list=xtensa-softmmu,arm-softmmu,aarch64-softmmu,alpha-softmmu"
> > - - TEST_CMD="make -j3 check-tcg V=1"
> > + - TEST_CMD="make -j${JOBS} check-tcg V=1"
> > - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-debug-tcg"
> >
> > - name: "[aarch64] GCC check-tcg"
> > @@ -500,7 +502,7 @@ matrix:
> > - BUILD_DIR="release/build/dir" SRC_DIR="../../.."
> > - BASE_CONFIG="--prefix=$PWD/dist"
> > -
> CONFIG="--target-list=x86_64-softmmu,aarch64-softmmu,armeb-linux-user,ppc-linux-user"
> > - - TEST_CMD="make install -j3"
> > + - TEST_CMD="make install -j${JOBS}"
> > - QEMU_VERSION="${TRAVIS_TAG:1}"
> > - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-default"
> > script:
> >
>
>
[-- Attachment #2: Type: text/html, Size: 10342 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2020-02-01 22:15 UTC | newest]
Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-30 11:32 [PATCH v2 00/12] testing/next (with build fixes!) Alex Bennée
2020-01-30 11:32 ` [PATCH v2 01/12] tests/docker: move most cross compilers to buster base Alex Bennée
2020-01-31 15:22 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 02/12] tests/docker: better handle symlinked libs Alex Bennée
2020-01-31 15:58 ` Philippe Mathieu-Daudé
2020-01-31 16:48 ` Alex Bennée
2020-01-30 11:32 ` [PATCH v2 03/12] gitlab-ci: Refresh the list of iotests Alex Bennée
2020-01-30 11:32 ` [PATCH v2 04/12] travis.yml: Install genisoimage package Alex Bennée
2020-01-30 19:04 ` Thomas Huth
2020-01-31 15:56 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 05/12] .shippable: --disable-docs for cross-compile tests Alex Bennée
2020-01-31 15:23 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 06/12] hw/hppa/Kconfig: LASI chipset requires PARALLEL port Alex Bennée
2020-01-30 11:32 ` [PATCH v2 07/12] .travis.yml: Drop superfluous use of --python=python3 parameter Alex Bennée
2020-01-30 11:32 ` [PATCH v2 08/12] .travis.yml: Add description to each job Alex Bennée
2020-01-30 19:06 ` Thomas Huth
2020-01-31 15:24 ` Philippe Mathieu-Daudé
2020-01-31 15:37 ` Thomas Huth
2020-01-30 11:32 ` [PATCH v2 09/12] .travis.yml: build documents under bionic Alex Bennée
2020-01-31 15:27 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 10/12] .travis.yml: move cache flushing to early common phase Alex Bennée
2020-01-31 15:31 ` Philippe Mathieu-Daudé
2020-01-30 11:32 ` [PATCH v2 11/12] .travis.yml: probe for number of available processors Alex Bennée
2020-01-30 19:09 ` Thomas Huth
2020-01-31 15:51 ` Philippe Mathieu-Daudé
2020-02-01 22:13 ` Ed Vielmetti
2020-01-30 11:32 ` [PATCH v2 12/12] .travis.yml: limit the arm64 target list Alex Bennée
2020-01-30 19:43 ` Thomas Huth
2020-01-31 15:55 ` Philippe Mathieu-Daudé
2020-01-31 16:14 ` Philippe Mathieu-Daudé
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.