* [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets
@ 2017-05-08 4:56 Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 01/23] shippable: use C locale to simplify console output Philippe Mathieu-Daudé
` (22 more replies)
0 siblings, 23 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:56 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng; +Cc: Philippe Mathieu-Daudé
This patchset add 2 more architectures to the cross-build farm.
- Reorganize Dockerfiles to use less layers, resulting in smaller images.
This also reduce time of image transfer, for example while using:
`docker save qemu:debian | ssh remote docker load`
- Install more debian packages so the configure script enable more features and
more code can be compiled/covered.
- There are still some incorrect multiarch packages on debian/jessie used in
the docker images, add a script to generate fake packages and avoid
dependencies issues.
- Modify the docker.py script to include an extra file while building images.
There is still some issue trying to cross-build mips64el-softmmu, it seems the
cross tools use the system outdated libfdt instead of the one checkouted in the
dtc submodule. I disabled this target for now.
I had some issues in some jobs, git reporting "Problem with the SSL CA cert"
trying to clone from github.com. I enabled GIT_CURL_VERBOSE but didn't want to
spend more time on this right now so I disabled the verification by setting
GIT_SSL_NO_VERIFY. Since we use Shippable services to compile the codebase and
not to release binaries it seemed Ok to me.
The branch https://github.com/philmd/qemu/tree/docker_shippable can be checked
here at Shippable:
https://app.shippable.com/github/philmd/qemu/status/dashboard
With the free open source projects service, each arch builds in around ~9min,
the 5 jobs take ~44min in total. It seems each docker image is rebuilt from
scratch, maybe we could save a lot pushing/pulling to a registry.
Regards,
Phil.
Philippe Mathieu-Daudé (23):
shippable: use C locale to simplify console output
shippable: build using all available cpus
shippable: ignore SSL verification
docker: compact debian base
docker: compact debian armhf
docker: compact debian arm64
docker: add extra libs to armhf target to extend codebase coverage
docker: add extra libs to arm64 target to extend codebase coverage
docker: add extra libs to s390x target to extend codebase coverage
docker: add mipsel build target
shippable: add mipsel target
docker: let _copy_with_mkdir() sub_path argument be optional
docker: add --include-file argument to 'build' command
docker: add 'apt-fake' script which generate fake debian packages
docker: fix mipsel debian/stable dependencies problem
shippable: do not initialize submodules automatically
shippable: use dtc submodule if distrib packages are too old
shippable: temporary disable mips64el-softmmu build
docker: add powerpc build target
shippable: add powerpc target
docker: fix powerpc debian/stable dependencies problem
MAINTAINERS: add Shippable automation platform URL
MAINTAINERS: self-appoint me as reviewer in build/test automation
.shippable.yml | 23 ++++++++++-
MAINTAINERS | 2 +
tests/docker/Makefile.include | 7 +++-
tests/docker/docker.py | 7 +++-
tests/docker/dockerfiles/debian-apt-fake.sh | 46 ++++++++++++++++++++++
tests/docker/dockerfiles/debian-arm64-cross.docker | 16 ++++++--
tests/docker/dockerfiles/debian-armhf-cross.docker | 16 ++++++--
.../docker/dockerfiles/debian-mipsel-cross.docker | 30 ++++++++++++++
.../docker/dockerfiles/debian-powerpc-cross.docker | 30 ++++++++++++++
tests/docker/dockerfiles/debian-s390x-cross.docker | 6 +++
tests/docker/dockerfiles/debian.docker | 24 ++++++-----
11 files changed, 186 insertions(+), 21 deletions(-)
create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker
--
2.11.0
^ permalink raw reply [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 01/23] shippable: use C locale to simplify console output
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
@ 2017-05-08 4:56 ` Philippe Mathieu-Daudé
2017-05-08 10:54 ` Alex Bennée
2017-05-08 4:56 ` [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus Philippe Mathieu-Daudé
` (21 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:56 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng; +Cc: Philippe Mathieu-Daudé
remove this noise:
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = "en_US.UTF-8",
LC_CTYPE = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
.shippable.yml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.shippable.yml b/.shippable.yml
index 653bd750fe..231c29b620 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,5 +1,7 @@
language: c
env:
+ global:
+ - LC_ALL=C
matrix:
- IMAGE=debian-armhf-cross
TARGET_LIST=arm-softmmu,arm-linux-user
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 01/23] shippable: use C locale to simplify console output Philippe Mathieu-Daudé
@ 2017-05-08 4:56 ` Philippe Mathieu-Daudé
2017-05-08 10:55 ` Alex Bennée
2017-05-08 4:56 ` [Qemu-devel] [RFC PATCH 03/23] shippable: ignore SSL verification Philippe Mathieu-Daudé
` (20 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:56 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng; +Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
.shippable.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.shippable.yml b/.shippable.yml
index 231c29b620..5170486ff9 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -20,4 +20,4 @@ build:
ci:
- unset CC
- ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
- - make -j2
+ - make -j$(getconf _NPROCESSORS_ONLN)
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [RFC PATCH 03/23] shippable: ignore SSL verification
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 01/23] shippable: use C locale to simplify console output Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus Philippe Mathieu-Daudé
@ 2017-05-08 4:56 ` Philippe Mathieu-Daudé
2017-05-08 10:58 ` Alex Bennée
2017-05-08 4:56 ` [Qemu-devel] [PATCH 04/23] docker: compact debian base Philippe Mathieu-Daudé
` (19 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:56 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Daniel P . Berrange
Cc: Philippe Mathieu-Daudé
shippable logs:
--------------
git_sync
- ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
Cloning into '/root/src/github.com/philmd/qemu'...
fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
retrying 1 of 3 times...
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
.shippable.yml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/.shippable.yml b/.shippable.yml
index 5170486ff9..b661e667b3 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -2,6 +2,10 @@ language: c
env:
global:
- LC_ALL=C
+ # sometimes Shippable fails to clone from github (git_sync stage):
+ # "Problem with the SSL CA cert (path? access rights?)"
+ # for now disable SSL verification.
+ - GIT_SSL_NO_VERIFY=1
matrix:
- IMAGE=debian-armhf-cross
TARGET_LIST=arm-softmmu,arm-linux-user
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 04/23] docker: compact debian base
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (2 preceding siblings ...)
2017-05-08 4:56 ` [Qemu-devel] [RFC PATCH 03/23] shippable: ignore SSL verification Philippe Mathieu-Daudé
@ 2017-05-08 4:56 ` Philippe Mathieu-Daudé
2017-05-08 11:04 ` Alex Bennée
2017-05-08 4:56 ` [Qemu-devel] [PATCH 05/23] docker: compact debian armhf Philippe Mathieu-Daudé
` (18 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:56 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng; +Cc: Philippe Mathieu-Daudé
- install common/basic tools at once
- one-line Emdebian setup
- use eatmydata and remove apt cache to save space
- add bison and flex and git
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/dockerfiles/debian.docker | 22 ++++++++++++----------
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 52bd79938e..694f8d3b63 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -9,17 +9,19 @@
#
FROM debian:stable-slim
-# Setup some basic tools we need
-RUN apt update
-RUN apt install -yy curl aptitude
-
-# Setup Emdebian
-RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
-RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
+# Install some basic tools and common build utilities
+RUN apt-get update && \
+ DEBIAN_FRONTEND=noninteractive apt-get install -yy \
+ eatmydata && \
+ DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+ curl aptitude \
+ build-essential clang git \
+ bison flex && \
+ rm -rf /var/lib/apt
# Duplicate deb line as deb-src
RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
-# Install common build utilities
-RUN apt update
-RUN apt install -yy build-essential clang
+# Setup Emdebian
+RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
+ curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 05/23] docker: compact debian armhf
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (3 preceding siblings ...)
2017-05-08 4:56 ` [Qemu-devel] [PATCH 04/23] docker: compact debian base Philippe Mathieu-Daudé
@ 2017-05-08 4:56 ` Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 06/23] docker: compact debian arm64 Philippe Mathieu-Daudé
` (17 subsequent siblings)
22 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:56 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng; +Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/dockerfiles/debian-armhf-cross.docker | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 668d60aeb3..533e525971 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -6,10 +6,12 @@
FROM qemu:debian
# Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture armhf
-RUN apt update
-RUN apt install -yy crossbuild-essential-armhf
-RUN apt-get build-dep -yy -a armhf qemu
+RUN dpkg --add-architecture armhf && \
+ apt-get update && \
+ DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+ crossbuild-essential-armhf
# Specify the cross prefix for this image (see tests/docker/common.rc)
ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a armhf qemu
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 06/23] docker: compact debian arm64
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (4 preceding siblings ...)
2017-05-08 4:56 ` [Qemu-devel] [PATCH 05/23] docker: compact debian armhf Philippe Mathieu-Daudé
@ 2017-05-08 4:56 ` Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 07/23] docker: add extra libs to armhf target to extend codebase coverage Philippe Mathieu-Daudé
` (16 subsequent siblings)
22 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:56 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng; +Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/dockerfiles/debian-arm64-cross.docker | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 592b5d7055..12b30aae54 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -6,10 +6,12 @@
FROM qemu:debian
# Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture arm64
-RUN apt update
-RUN apt install -yy crossbuild-essential-arm64
-RUN apt-get build-dep -yy -a arm64 qemu
+RUN dpkg --add-architecture arm64 && \
+ apt-get update && \
+ DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+ crossbuild-essential-arm64
# Specify the cross prefix for this image (see tests/docker/common.rc)
ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a arm64 qemu
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 07/23] docker: add extra libs to armhf target to extend codebase coverage
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (5 preceding siblings ...)
2017-05-08 4:56 ` [Qemu-devel] [PATCH 06/23] docker: compact debian arm64 Philippe Mathieu-Daudé
@ 2017-05-08 4:56 ` Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 08/23] docker: add extra libs to arm64 " Philippe Mathieu-Daudé
` (15 subsequent siblings)
22 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:56 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Peter Maydell, qemu-arm
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/dockerfiles/debian-armhf-cross.docker | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 533e525971..01d9fcb7f2 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -14,4 +14,10 @@ RUN dpkg --add-architecture armhf && \
# Specify the cross prefix for this image (see tests/docker/common.rc)
ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
-RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a armhf qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a armhf qemu && \
+ DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+ glusterfs-common:armhf \
+ libncursesw5-dev:armhf libnfs-dev:armhf \
+ libbz2-dev:armhf liblzo2-dev:armhf \
+ libsnappy-dev:armhf librdmacm-dev:armhf \
+ libxen-dev:armhf
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 08/23] docker: add extra libs to arm64 target to extend codebase coverage
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (6 preceding siblings ...)
2017-05-08 4:56 ` [Qemu-devel] [PATCH 07/23] docker: add extra libs to armhf target to extend codebase coverage Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 09/23] docker: add extra libs to s390x " Philippe Mathieu-Daudé
` (14 subsequent siblings)
22 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Peter Maydell, qemu-arm
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/dockerfiles/debian-arm64-cross.docker | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 12b30aae54..5e0036917b 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -14,4 +14,10 @@ RUN dpkg --add-architecture arm64 && \
# Specify the cross prefix for this image (see tests/docker/common.rc)
ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
-RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a arm64 qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a arm64 qemu && \
+ DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+ glusterfs-common:arm64 \
+ libncursesw5-dev:arm64 libnfs-dev:arm64 \
+ libbz2-dev:arm64 liblzo2-dev:arm64 \
+ libsnappy-dev:arm64 librdmacm-dev:arm64 \
+ libxen-dev:arm64
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 09/23] docker: add extra libs to s390x target to extend codebase coverage
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (7 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 08/23] docker: add extra libs to arm64 " Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 10/23] docker: add mipsel build target Philippe Mathieu-Daudé
` (13 subsequent siblings)
22 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Richard Henderson,
Alexander Graf
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/dockerfiles/debian-s390x-cross.docker | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index 3a687feda0..119b152971 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -20,3 +20,9 @@ RUN apt install -yy gcc-multilib-s390x-linux-gnu binutils-multiarch
# Specify the cross prefix for this image (see tests/docker/common.rc)
ENV QEMU_CONFIGURE_OPTS --cross-prefix=s390x-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+ glusterfs-common:s390x \
+ libncursesw5-dev:s390x libnfs-dev:s390x \
+ libbz2-dev:s390x liblzo2-dev:s390x \
+ libsnappy-dev:s390x librdmacm-dev:s390x
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 10/23] docker: add mipsel build target
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (8 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 09/23] docker: add extra libs to s390x " Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 11/23] shippable: add mipsel target Philippe Mathieu-Daudé
` (12 subsequent siblings)
22 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Aurelien Jarno, Yongbok Kim
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/Makefile.include | 1 +
.../docker/dockerfiles/debian-mipsel-cross.docker | 23 ++++++++++++++++++++++
2 files changed, 24 insertions(+)
create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 03eda37bf4..ceff2c1654 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -57,6 +57,7 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
# Enforce dependancies for composite images
docker-image-debian-armhf-cross: docker-image-debian
docker-image-debian-arm64-cross: docker-image-debian
+docker-image-debian-mipsel-cross: docker-image-debian
# Expand all the pre-requistes for each docker image and test combination
$(foreach i,$(DOCKER_IMAGES), \
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
new file mode 100644
index 0000000000..c569a2e1b4
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -0,0 +1,23 @@
+#
+# Docker mipsel cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture mipsel && \
+ apt-get update && \
+ DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+ crossbuild-essential-mipsel
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a mipsel qemu && \
+ DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+ glusterfs-common:mipsel \
+ libncursesw5-dev:mipsel libnfs-dev:mipsel \
+ libbz2-dev:mipsel liblzo2-dev:mipsel \
+ libsnappy-dev:mipsel librdmacm-dev:mipsel
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 11/23] shippable: add mipsel target
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (9 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 10/23] docker: add mipsel build target Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 11:06 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 12/23] docker: let _copy_with_mkdir() sub_path argument be optional Philippe Mathieu-Daudé
` (11 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Aurelien Jarno, Yongbok Kim
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
.shippable.yml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.shippable.yml b/.shippable.yml
index b661e667b3..fa2d6a4e9b 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -13,6 +13,8 @@ env:
TARGET_LIST=aarch64-softmmu,aarch64-linux-user
- IMAGE=debian-s390x-cross
TARGET_LIST=s390x-softmmu,s390x-linux-user
+ - IMAGE=debian-mipsel-cross
+ TARGET_LIST=mipsel-softmmu,mips64el-softmmu,mipsel-linux-user,mips64el-linux-user
build:
pre_ci:
- make docker-image-${IMAGE}
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 12/23] docker: let _copy_with_mkdir() sub_path argument be optional
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (10 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 11/23] shippable: add mipsel target Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [RFC PATCH 13/23] docker: add --include-file argument to 'build' command Philippe Mathieu-Daudé
` (10 subsequent siblings)
22 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng; +Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/docker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 8747f6a440..6ddc6e4c2a 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -52,7 +52,7 @@ def _guess_docker_command():
raise Exception("Cannot find working docker command. Tried:\n%s" % \
commands_txt)
-def _copy_with_mkdir(src, root_dir, sub_path):
+def _copy_with_mkdir(src, root_dir, sub_path='.'):
"""Copy src into root_dir, creating sub_path as needed."""
dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
try:
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [RFC PATCH 13/23] docker: add --include-file argument to 'build' command
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (11 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 12/23] docker: let _copy_with_mkdir() sub_path argument be optional Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 11:10 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [RFC PATCH 14/23] docker: add 'apt-fake' script which generate fake debian packages Philippe Mathieu-Daudé
` (9 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng; +Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/Makefile.include | 3 ++-
tests/docker/docker.py | 5 +++++
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index ceff2c1654..47978fb56c 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -51,7 +51,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
$(if $(NOUSER),,--add-current-user) \
- $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
+ $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE))\
+ $(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
"BUILD","$*")
# Enforce dependancies for composite images
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 6ddc6e4c2a..4c096a8178 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -237,6 +237,9 @@ class BuildCommand(SubCommand):
help="""Specify a binary that will be copied to the
container together with all its dependent
libraries""")
+ parser.add_argument("--include-file", "-f",
+ help="""Specify a binary that will be copied to the
+ container""")
parser.add_argument("--add-current-user", "-u", dest="user",
action="store_true",
help="Add the current user to image's passwd")
@@ -274,6 +277,8 @@ class BuildCommand(SubCommand):
if args.include_executable:
_copy_binary_with_libs(args.include_executable,
docker_dir)
+ if args.include_file:
+ _copy_with_mkdir(args.include_file, docker_dir)
argv += ["--build-arg=" + k.lower() + "=" + v
for k, v in os.environ.iteritems()
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [RFC PATCH 14/23] docker: add 'apt-fake' script which generate fake debian packages
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (12 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [RFC PATCH 13/23] docker: add --include-file argument to 'build' command Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 11:12 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 15/23] docker: fix mipsel debian/stable dependencies problem Philippe Mathieu-Daudé
` (8 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Aurelien Jarno,
Riku Voipio, Vagrant Cascadian, Michael Tokarev
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/Makefile.include | 2 ++
tests/docker/dockerfiles/debian-apt-fake.sh | 46 +++++++++++++++++++++++++++++
tests/docker/dockerfiles/debian.docker | 2 ++
3 files changed, 50 insertions(+)
create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 47978fb56c..ca6f57a292 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -55,6 +55,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
$(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
"BUILD","$*")
+docker-image-debian: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
+
# Enforce dependancies for composite images
docker-image-debian-armhf-cross: docker-image-debian
docker-image-debian-arm64-cross: docker-image-debian
diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
new file mode 100755
index 0000000000..387522c174
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-apt-fake.sh
@@ -0,0 +1,46 @@
+#! /bin/sh
+#
+# Generate fake debian package to resolve unimportant unmet dependencies held
+# by upstream multiarch broken packages.
+#
+# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+test $1 = "install" && shift 1
+
+fake_install()
+{
+ echo "Generating fake $2 $1 $3 ..."
+ (cd /var/cache/apt/archives
+ (cat << 'EOF'
+Section: misc
+Priority: optional
+Standards-Version: 3.9.2
+
+Package: NAME
+Version: VERSION
+Maintainer: qemu-devel@nongnu.org
+Architecture: any
+Multi-Arch: same
+Description: fake NAME
+EOF
+ ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
+ equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
+ dpkg -i $2_$3_$1.deb
+ )
+}
+
+try_install()
+{
+ name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
+ arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
+ vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
+ apt-get install -q -yy $1 || fake_install $arch $name $vers
+}
+
+for package in $*; do
+ try_install $package
+done
diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 694f8d3b63..a99396ae8f 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -22,6 +22,8 @@ RUN apt-get update && \
# Duplicate deb line as deb-src
RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
+ADD debian-apt-fake.sh /usr/local/bin/apt-fake
+
# Setup Emdebian
RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 15/23] docker: fix mipsel debian/stable dependencies problem
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (13 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [RFC PATCH 14/23] docker: add 'apt-fake' script which generate fake debian packages Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 11:14 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 16/23] shippable: do not initialize submodules automatically Philippe Mathieu-Daudé
` (7 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Aurelien Jarno,
Yongbok Kim, Riku Voipio, Vagrant Cascadian, Michael Tokarev
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/dockerfiles/debian-mipsel-cross.docker | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
index c569a2e1b4..3c4945d78e 100644
--- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -12,6 +12,13 @@ RUN dpkg --add-architecture mipsel && \
DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
crossbuild-essential-mipsel
+# <kludge> to fix "following packages have unmet dependencies" ...
+RUN apt-get install -y --no-install-recommends equivs pkg-config && \
+ apt-fake install pkgconf:mipsel=0.9.7-fake pkg-config:mipsel=0.28-1.1-fake && \
+ ln -s pkg-config /usr/bin/mipsel-linux-gnu-pkg-config
+ENV PKG_CONFIG_PATH /usr/lib/mipsel-linux-gnu/pkgconfig
+# </kludge>
+
# Specify the cross prefix for this image (see tests/docker/common.rc)
ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 16/23] shippable: do not initialize submodules automatically
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (14 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 15/23] docker: fix mipsel debian/stable dependencies problem Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 11:15 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 17/23] shippable: use dtc submodule if distrib packages are too old Philippe Mathieu-Daudé
` (6 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng; +Cc: Philippe Mathieu-Daudé
do it in the 'ci' target when needed.
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
.shippable.yml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.shippable.yml b/.shippable.yml
index fa2d6a4e9b..b3cbca458c 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,4 +1,6 @@
language: c
+git:
+ submodules: false
env:
global:
- LC_ALL=C
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 17/23] shippable: use dtc submodule if distrib packages are too old
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (15 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 16/23] shippable: do not initialize submodules automatically Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 11:15 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 19/23] docker: add powerpc build target Philippe Mathieu-Daudé
` (5 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Aurelien Jarno,
Riku Voipio, Vagrant Cascadian, Michael Tokarev
Cc: Philippe Mathieu-Daudé
shippable output:
----------------
LINK mips64el-softmmu/qemu-system-mips64el
../hw/core/loader-fit.o: In function `load_fit':
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:278: undefined reference to `fdt_first_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:286: undefined reference to `fdt_next_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:277: undefined reference to `fdt_first_subnode'
collect2: error: ld returned 1 exit status
Makefile:201: recipe for target 'qemu-system-mips64el' failed
make[1]: *** [qemu-system-mips64el] Error 1
Makefile:327: recipe for target 'subdir-mips64el-softmmu' failed
make: *** [subdir-mips64el-softmmu] Error 2
having this too old libfdt version (required version >= 1.4.2):
# dpkg-query --showformat='${Version}\n' --show libfdt-dev
1.4.0+dfsg-1
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
.shippable.yml | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/.shippable.yml b/.shippable.yml
index b3cbca458c..6601243060 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -27,5 +27,13 @@ build:
options: "-e HOME=/root"
ci:
- unset CC
+ # some targets require newer up to date packages, for example TARGET_LIST matching
+ # aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu)
+ # see the configure script:
+ # error_exit "DTC (libfdt) version >= 1.4.2 not present. Your options:"
+ # " (1) Preferred: Install the DTC (libfdt) devel package"
+ # " (2) Fetch the DTC submodule, using:"
+ # " git submodule update --init dtc"
+ - dpkg --compare-versions `dpkg-query --showformat='${Version}' --show libfdt-dev` ge 1.4.2 || git submodule update --init dtc
- ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
- make -j$(getconf _NPROCESSORS_ONLN)
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 19/23] docker: add powerpc build target
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (16 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 17/23] shippable: use dtc submodule if distrib packages are too old Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 20/23] shippable: add powerpc target Philippe Mathieu-Daudé
` (4 subsequent siblings)
22 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, David Gibson,
Alexander Graf, qemu-ppc
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/Makefile.include | 1 +
.../docker/dockerfiles/debian-powerpc-cross.docker | 23 ++++++++++++++++++++++
2 files changed, 24 insertions(+)
create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index ca6f57a292..704b74bfbd 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -61,6 +61,7 @@ docker-image-debian: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
docker-image-debian-armhf-cross: docker-image-debian
docker-image-debian-arm64-cross: docker-image-debian
docker-image-debian-mipsel-cross: docker-image-debian
+docker-image-debian-powerpc-cross: docker-image-debian
# Expand all the pre-requistes for each docker image and test combination
$(foreach i,$(DOCKER_IMAGES), \
diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
new file mode 100644
index 0000000000..9107ebc2d3
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
@@ -0,0 +1,23 @@
+#
+# Docker powerpc cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture powerpc && \
+ apt-get update && \
+ DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+ crossbuild-essential-powerpc
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a powerpc qemu && \
+ DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+ glusterfs-common:powerpc \
+ libncursesw5-dev:powerpc libnfs-dev:powerpc \
+ libbz2-dev:powerpc liblzo2-dev:powerpc \
+ libsnappy-dev:powerpc librdmacm-dev:powerpc
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 20/23] shippable: add powerpc target
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (17 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 19/23] docker: add powerpc build target Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 21/23] docker: fix powerpc debian/stable dependencies problem Philippe Mathieu-Daudé
` (3 subsequent siblings)
22 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, David Gibson,
Alexander Graf, qemu-ppc
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
.shippable.yml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.shippable.yml b/.shippable.yml
index 7e714991f9..3a80165b6a 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -18,6 +18,8 @@ env:
# mips64el-softmmu disabled due to libfdt problem
- IMAGE=debian-mipsel-cross
TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
+ - IMAGE=debian-powerpc-cross
+ TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
build:
pre_ci:
- make docker-image-${IMAGE}
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 21/23] docker: fix powerpc debian/stable dependencies problem
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (18 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 20/23] shippable: add powerpc target Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 11:19 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 22/23] MAINTAINERS: add Shippable automation platform URL Philippe Mathieu-Daudé
` (2 subsequent siblings)
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, David Gibson,
Alexander Graf, qemu-ppc, Aurelien Jarno, Riku Voipio,
Vagrant Cascadian, Michael Tokarev
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
tests/docker/dockerfiles/debian-powerpc-cross.docker | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
index 9107ebc2d3..d6e4d1e977 100644
--- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
+++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
@@ -12,6 +12,13 @@ RUN dpkg --add-architecture powerpc && \
DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
crossbuild-essential-powerpc
+# <kludge> to fix "following packages have unmet dependencies" ...
+RUN apt-get install -y --no-install-recommends equivs pkg-config && \
+ apt-fake install pkg-config:powerpc=0.28-1.1-fake && \
+ ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
+ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
+# </kludge>
+
# Specify the cross prefix for this image (see tests/docker/common.rc)
ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 22/23] MAINTAINERS: add Shippable automation platform URL
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (19 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 21/23] docker: fix powerpc debian/stable dependencies problem Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 11:21 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 23/23] MAINTAINERS: self-appoint me as reviewer in build/test automation Philippe Mathieu-Daudé
2017-05-08 5:02 ` [Qemu-devel] [PATCH 18/23] shippable: temporary disable mips64el-softmmu build Philippe Mathieu-Daudé
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Thomas Huth,
Stefan Hajnoczi, Michael S. Tsirkin, Markus Armbruster
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
MAINTAINERS | 1 +
1 file changed, 1 insertion(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index a1d2b3a4d3..9251f575d8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1833,6 +1833,7 @@ F: .travis.yml
F: .shippable.yml
F: tests/docker/
W: https://travis-ci.org/qemu/qemu
+W: https://app.shippable.com/github/qemu/qemu
W: http://patchew.org/QEMU/
Documentation
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 23/23] MAINTAINERS: self-appoint me as reviewer in build/test automation
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (20 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 22/23] MAINTAINERS: add Shippable automation platform URL Philippe Mathieu-Daudé
@ 2017-05-08 4:57 ` Philippe Mathieu-Daudé
2017-05-08 11:22 ` Alex Bennée
2017-05-08 5:02 ` [Qemu-devel] [PATCH 18/23] shippable: temporary disable mips64el-softmmu build Philippe Mathieu-Daudé
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 4:57 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Paolo Bonzini,
Thomas Huth, Stefan Hajnoczi, Michael S. Tsirkin,
Markus Armbruster
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
MAINTAINERS | 1 +
1 file changed, 1 insertion(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 9251f575d8..c7520b76db 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1827,6 +1827,7 @@ Build and test automation
-------------------------
M: Alex Bennée <alex.bennee@linaro.org>
M: Fam Zheng <famz@redhat.com>
+R: Philippe Mathieu-Daudé <f4bug@amsat.org>
L: qemu-devel@nongnu.org
S: Maintained
F: .travis.yml
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* [Qemu-devel] [PATCH 18/23] shippable: temporary disable mips64el-softmmu build
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
` (21 preceding siblings ...)
2017-05-08 4:57 ` [Qemu-devel] [PATCH 23/23] MAINTAINERS: self-appoint me as reviewer in build/test automation Philippe Mathieu-Daudé
@ 2017-05-08 5:02 ` Philippe Mathieu-Daudé
2017-05-08 11:22 ` Alex Bennée
22 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 5:02 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Fam Zheng, Aurelien Jarno, Yongbok Kim
Cc: Philippe Mathieu-Daudé
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
.shippable.yml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/.shippable.yml b/.shippable.yml
index 6601243060..7e714991f9 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -15,8 +15,9 @@ env:
TARGET_LIST=aarch64-softmmu,aarch64-linux-user
- IMAGE=debian-s390x-cross
TARGET_LIST=s390x-softmmu,s390x-linux-user
+ # mips64el-softmmu disabled due to libfdt problem
- IMAGE=debian-mipsel-cross
- TARGET_LIST=mipsel-softmmu,mips64el-softmmu,mipsel-linux-user,mips64el-linux-user
+ TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
build:
pre_ci:
- make docker-image-${IMAGE}
--
2.11.0
^ permalink raw reply related [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 01/23] shippable: use C locale to simplify console output
2017-05-08 4:56 ` [Qemu-devel] [PATCH 01/23] shippable: use C locale to simplify console output Philippe Mathieu-Daudé
@ 2017-05-08 10:54 ` Alex Bennée
0 siblings, 0 replies; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 10:54 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: qemu-devel, Fam Zheng
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> remove this noise:
>
> perl: warning: Setting locale failed.
> perl: warning: Please check that your locale settings:
> LANGUAGE = (unset),
> LC_ALL = "en_US.UTF-8",
> LC_CTYPE = "en_US.UTF-8",
> LANG = "en_US.UTF-8"
> are supported and installed on your system.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> .shippable.yml | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 653bd750fe..231c29b620 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -1,5 +1,7 @@
> language: c
> env:
> + global:
> + - LC_ALL=C
> matrix:
> - IMAGE=debian-armhf-cross
> TARGET_LIST=arm-softmmu,arm-linux-user
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus
2017-05-08 4:56 ` [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus Philippe Mathieu-Daudé
@ 2017-05-08 10:55 ` Alex Bennée
2017-05-08 13:21 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 10:55 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: qemu-devel, Fam Zheng
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> .shippable.yml | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 231c29b620..5170486ff9 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -20,4 +20,4 @@ build:
> ci:
> - unset CC
> - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
> - - make -j2
> + - make -j$(getconf _NPROCESSORS_ONLN)
Did you ever check what this generates on Shippable? AFAICT they only
have one core for each docker image. I following the core+1 pattern
although that may not make as much sense for low core counts.
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [RFC PATCH 03/23] shippable: ignore SSL verification
2017-05-08 4:56 ` [Qemu-devel] [RFC PATCH 03/23] shippable: ignore SSL verification Philippe Mathieu-Daudé
@ 2017-05-08 10:58 ` Alex Bennée
2017-05-08 11:56 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 10:58 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: qemu-devel, Fam Zheng, Daniel P . Berrange
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> shippable logs:
> --------------
> git_sync
> - ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
> Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
> Cloning into '/root/src/github.com/philmd/qemu'...
> fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
> retrying 1 of 3 times...
I've not seen this before but which git_sync stage is it. The host or
the container? If it is the container that probably just means we are
missing the CA Cert package.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> .shippable.yml | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 5170486ff9..b661e667b3 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -2,6 +2,10 @@ language: c
> env:
> global:
> - LC_ALL=C
> + # sometimes Shippable fails to clone from github (git_sync stage):
> + # "Problem with the SSL CA cert (path? access rights?)"
> + # for now disable SSL verification.
> + - GIT_SSL_NO_VERIFY=1
> matrix:
> - IMAGE=debian-armhf-cross
> TARGET_LIST=arm-softmmu,arm-linux-user
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 04/23] docker: compact debian base
2017-05-08 4:56 ` [Qemu-devel] [PATCH 04/23] docker: compact debian base Philippe Mathieu-Daudé
@ 2017-05-08 11:04 ` Alex Bennée
2017-05-08 18:19 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:04 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: qemu-devel, Fam Zheng
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> - install common/basic tools at once
> - one-line Emdebian setup
> - use eatmydata and remove apt cache to save space
> - add bison and flex and git
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> tests/docker/dockerfiles/debian.docker | 22 ++++++++++++----------
> 1 file changed, 12 insertions(+), 10 deletions(-)
>
> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> index 52bd79938e..694f8d3b63 100644
> --- a/tests/docker/dockerfiles/debian.docker
> +++ b/tests/docker/dockerfiles/debian.docker
> @@ -9,17 +9,19 @@
> #
> FROM debian:stable-slim
>
> -# Setup some basic tools we need
> -RUN apt update
> -RUN apt install -yy curl aptitude
> -
> -# Setup Emdebian
> -RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
> -RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
> +# Install some basic tools and common build utilities
> +RUN apt-get update && \
> + DEBIAN_FRONTEND=noninteractive apt-get install -yy \
> + eatmydata && \
I wonder if we should just use ENV instead to set DEBIAN_FRONTEND?
> + DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
> + curl aptitude \
> + build-essential clang git \
> + bison flex && \
> + rm -rf /var/lib/apt
Use apt clean to do this.
>
> # Duplicate deb line as deb-src
> RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
>
> -# Install common build utilities
> -RUN apt update
> -RUN apt install -yy build-essential clang
> +# Setup Emdebian
> +RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
> + curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
Why move this to the end? Once we have curl installed it would be nice
to do just one update step and have emdebian synced up in the base
image.
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 11/23] shippable: add mipsel target
2017-05-08 4:57 ` [Qemu-devel] [PATCH 11/23] shippable: add mipsel target Philippe Mathieu-Daudé
@ 2017-05-08 11:06 ` Alex Bennée
0 siblings, 0 replies; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:06 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: qemu-devel, Fam Zheng, Aurelien Jarno, Yongbok Kim
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> .shippable.yml | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index b661e667b3..fa2d6a4e9b 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -13,6 +13,8 @@ env:
> TARGET_LIST=aarch64-softmmu,aarch64-linux-user
> - IMAGE=debian-s390x-cross
> TARGET_LIST=s390x-softmmu,s390x-linux-user
> + - IMAGE=debian-mipsel-cross
> + TARGET_LIST=mipsel-softmmu,mips64el-softmmu,mipsel-linux-user,mips64el-linux-user
> build:
> pre_ci:
> - make docker-image-${IMAGE}
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [RFC PATCH 13/23] docker: add --include-file argument to 'build' command
2017-05-08 4:57 ` [Qemu-devel] [RFC PATCH 13/23] docker: add --include-file argument to 'build' command Philippe Mathieu-Daudé
@ 2017-05-08 11:10 ` Alex Bennée
0 siblings, 0 replies; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:10 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: qemu-devel, Fam Zheng
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> tests/docker/Makefile.include | 3 ++-
> tests/docker/docker.py | 5 +++++
> 2 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index ceff2c1654..47978fb56c 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -51,7 +51,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
> $(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
> $(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
> $(if $(NOUSER),,--add-current-user) \
> - $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
> + $(if $(EXECUTABLE),--include-executable=$(EXECUTABLE))\
> + $(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
> "BUILD","$*")
>
> # Enforce dependancies for composite images
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 6ddc6e4c2a..4c096a8178 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -237,6 +237,9 @@ class BuildCommand(SubCommand):
> help="""Specify a binary that will be copied to the
> container together with all its dependent
> libraries""")
> + parser.add_argument("--include-file", "-f",
> + help="""Specify a binary that will be copied to the
> + container""")
It's not a binary, just an additional file (which might be a binary).
> parser.add_argument("--add-current-user", "-u", dest="user",
> action="store_true",
> help="Add the current user to image's passwd")
> @@ -274,6 +277,8 @@ class BuildCommand(SubCommand):
> if args.include_executable:
> _copy_binary_with_libs(args.include_executable,
> docker_dir)
> + if args.include_file:
> + _copy_with_mkdir(args.include_file, docker_dir)
>
> argv += ["--build-arg=" + k.lower() + "=" + v
> for k, v in os.environ.iteritems()
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [RFC PATCH 14/23] docker: add 'apt-fake' script which generate fake debian packages
2017-05-08 4:57 ` [Qemu-devel] [RFC PATCH 14/23] docker: add 'apt-fake' script which generate fake debian packages Philippe Mathieu-Daudé
@ 2017-05-08 11:12 ` Alex Bennée
2017-05-08 17:48 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:12 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: qemu-devel, Fam Zheng, Aurelien Jarno, Riku Voipio,
Vagrant Cascadian, Michael Tokarev
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> tests/docker/Makefile.include | 2 ++
> tests/docker/dockerfiles/debian-apt-fake.sh | 46 +++++++++++++++++++++++++++++
> tests/docker/dockerfiles/debian.docker | 2 ++
> 3 files changed, 50 insertions(+)
> create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 47978fb56c..ca6f57a292 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -55,6 +55,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
> $(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
> "BUILD","$*")
>
> +docker-image-debian: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
> +
Can we only include debian-apt-fake on the debian images that require it please?
> # Enforce dependancies for composite images
> docker-image-debian-armhf-cross: docker-image-debian
> docker-image-debian-arm64-cross: docker-image-debian
> diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
> new file mode 100755
> index 0000000000..387522c174
> --- /dev/null
> +++ b/tests/docker/dockerfiles/debian-apt-fake.sh
> @@ -0,0 +1,46 @@
> +#! /bin/sh
> +#
> +# Generate fake debian package to resolve unimportant unmet dependencies held
> +# by upstream multiarch broken packages.
> +#
> +# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2
> +# or (at your option) any later version. See the COPYING file in
> +# the top-level directory.
> +
> +test $1 = "install" && shift 1
> +
> +fake_install()
> +{
> + echo "Generating fake $2 $1 $3 ..."
> + (cd /var/cache/apt/archives
> + (cat << 'EOF'
> +Section: misc
> +Priority: optional
> +Standards-Version: 3.9.2
> +
> +Package: NAME
> +Version: VERSION
> +Maintainer: qemu-devel@nongnu.org
> +Architecture: any
> +Multi-Arch: same
> +Description: fake NAME
> +EOF
> + ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
> + equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
> + dpkg -i $2_$3_$1.deb
> + )
> +}
> +
> +try_install()
> +{
> + name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
> + arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
> + vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
> + apt-get install -q -yy $1 || fake_install $arch $name $vers
> +}
> +
> +for package in $*; do
> + try_install $package
> +done
> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> index 694f8d3b63..a99396ae8f 100644
> --- a/tests/docker/dockerfiles/debian.docker
> +++ b/tests/docker/dockerfiles/debian.docker
> @@ -22,6 +22,8 @@ RUN apt-get update && \
> # Duplicate deb line as deb-src
> RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
>
> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
> +
And only add this in the images that need it.
> # Setup Emdebian
> RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
> curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 15/23] docker: fix mipsel debian/stable dependencies problem
2017-05-08 4:57 ` [Qemu-devel] [PATCH 15/23] docker: fix mipsel debian/stable dependencies problem Philippe Mathieu-Daudé
@ 2017-05-08 11:14 ` Alex Bennée
0 siblings, 0 replies; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:14 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: qemu-devel, Fam Zheng, Aurelien Jarno, Yongbok Kim, Riku Voipio,
Vagrant Cascadian, Michael Tokarev
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> tests/docker/dockerfiles/debian-mipsel-cross.docker | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
> index c569a2e1b4..3c4945d78e 100644
> --- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
> +++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
> @@ -12,6 +12,13 @@ RUN dpkg --add-architecture mipsel && \
> DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
> crossbuild-essential-mipsel
>
> +# <kludge> to fix "following packages have unmet dependencies" ...
> +RUN apt-get install -y --no-install-recommends equivs pkg-config && \
> + apt-fake install pkgconf:mipsel=0.9.7-fake
> pkg-config:mipsel=0.28-1.1-fake && \
I'll take your word that this fixes the problem for broken packages.
When I ran into these I just started using the linux-user approach
because it seemed fragile otherwise.
I assume a dot release of any of these packages might break the image
build?
> + ln -s pkg-config /usr/bin/mipsel-linux-gnu-pkg-config
> +ENV PKG_CONFIG_PATH /usr/lib/mipsel-linux-gnu/pkgconfig
> +
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> # Specify the cross prefix for this image (see tests/docker/common.rc)
> ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 16/23] shippable: do not initialize submodules automatically
2017-05-08 4:57 ` [Qemu-devel] [PATCH 16/23] shippable: do not initialize submodules automatically Philippe Mathieu-Daudé
@ 2017-05-08 11:15 ` Alex Bennée
2017-05-08 17:48 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:15 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: qemu-devel, Fam Zheng
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> do it in the 'ci' target when needed.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> .shippable.yml | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index fa2d6a4e9b..b3cbca458c 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -1,4 +1,6 @@
> language: c
> +git:
> + submodules: false
> env:
> global:
> - LC_ALL=C
Maybe this should be merged with the next patch?
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 17/23] shippable: use dtc submodule if distrib packages are too old
2017-05-08 4:57 ` [Qemu-devel] [PATCH 17/23] shippable: use dtc submodule if distrib packages are too old Philippe Mathieu-Daudé
@ 2017-05-08 11:15 ` Alex Bennée
0 siblings, 0 replies; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:15 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: qemu-devel, Fam Zheng, Aurelien Jarno, Riku Voipio,
Vagrant Cascadian, Michael Tokarev
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> shippable output:
> ----------------
> LINK mips64el-softmmu/qemu-system-mips64el
> ../hw/core/loader-fit.o: In function `load_fit':
> /root/src/github.com/philmd/qemu/hw/core/loader-fit.c:278: undefined reference to `fdt_first_subnode'
> /root/src/github.com/philmd/qemu/hw/core/loader-fit.c:286: undefined reference to `fdt_next_subnode'
> /root/src/github.com/philmd/qemu/hw/core/loader-fit.c:277: undefined reference to `fdt_first_subnode'
> collect2: error: ld returned 1 exit status
> Makefile:201: recipe for target 'qemu-system-mips64el' failed
> make[1]: *** [qemu-system-mips64el] Error 1
> Makefile:327: recipe for target 'subdir-mips64el-softmmu' failed
> make: *** [subdir-mips64el-softmmu] Error 2
>
> having this too old libfdt version (required version >= 1.4.2):
> # dpkg-query --showformat='${Version}\n' --show libfdt-dev
> 1.4.0+dfsg-1
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> .shippable.yml | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index b3cbca458c..6601243060 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -27,5 +27,13 @@ build:
> options: "-e HOME=/root"
> ci:
> - unset CC
> + # some targets require newer up to date packages, for example TARGET_LIST matching
> + # aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu)
> + # see the configure script:
> + # error_exit "DTC (libfdt) version >= 1.4.2 not present. Your options:"
> + # " (1) Preferred: Install the DTC (libfdt) devel package"
> + # " (2) Fetch the DTC submodule, using:"
> + # " git submodule update --init dtc"
> + - dpkg --compare-versions `dpkg-query --showformat='${Version}' --show libfdt-dev` ge 1.4.2 || git submodule update --init dtc
> - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
> - make -j$(getconf _NPROCESSORS_ONLN)
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 21/23] docker: fix powerpc debian/stable dependencies problem
2017-05-08 4:57 ` [Qemu-devel] [PATCH 21/23] docker: fix powerpc debian/stable dependencies problem Philippe Mathieu-Daudé
@ 2017-05-08 11:19 ` Alex Bennée
2017-05-08 17:50 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:19 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: qemu-devel, Fam Zheng, David Gibson, Alexander Graf, qemu-ppc,
Aurelien Jarno, Riku Voipio, Vagrant Cascadian, Michael Tokarev
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> tests/docker/dockerfiles/debian-powerpc-cross.docker | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
> index 9107ebc2d3..d6e4d1e977 100644
> --- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
> +++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
> @@ -12,6 +12,13 @@ RUN dpkg --add-architecture powerpc && \
> DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
> crossbuild-essential-powerpc
>
> +# <kludge> to fix "following packages have unmet dependencies" ...
> +RUN apt-get install -y --no-install-recommends equivs pkg-config && \
> + apt-fake install pkg-config:powerpc=0.28-1.1-fake && \
> + ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
> +ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
> +# </kludge>
> +
> # Specify the cross prefix for this image (see tests/docker/common.rc)
> ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
Given the image needs these package hacks it is probably best folded
into the commit adding the target.
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 22/23] MAINTAINERS: add Shippable automation platform URL
2017-05-08 4:57 ` [Qemu-devel] [PATCH 22/23] MAINTAINERS: add Shippable automation platform URL Philippe Mathieu-Daudé
@ 2017-05-08 11:21 ` Alex Bennée
0 siblings, 0 replies; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:21 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: qemu-devel, Fam Zheng, Thomas Huth, Stefan Hajnoczi,
Michael S. Tsirkin, Markus Armbruster
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> MAINTAINERS | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index a1d2b3a4d3..9251f575d8 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1833,6 +1833,7 @@ F: .travis.yml
> F: .shippable.yml
> F: tests/docker/
> W: https://travis-ci.org/qemu/qemu
> +W: https://app.shippable.com/github/qemu/qemu
> W: http://patchew.org/QEMU/
>
> Documentation
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 18/23] shippable: temporary disable mips64el-softmmu build
2017-05-08 5:02 ` [Qemu-devel] [PATCH 18/23] shippable: temporary disable mips64el-softmmu build Philippe Mathieu-Daudé
@ 2017-05-08 11:22 ` Alex Bennée
2017-05-08 17:52 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:22 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: qemu-devel, Fam Zheng, Aurelien Jarno, Yongbok Kim
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> .shippable.yml | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/.shippable.yml b/.shippable.yml
> index 6601243060..7e714991f9 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -15,8 +15,9 @@ env:
> TARGET_LIST=aarch64-softmmu,aarch64-linux-user
> - IMAGE=debian-s390x-cross
> TARGET_LIST=s390x-softmmu,s390x-linux-user
> + # mips64el-softmmu disabled due to libfdt problem
> - IMAGE=debian-mipsel-cross
> - TARGET_LIST=mipsel-softmmu,mips64el-softmmu,mipsel-linux-user,mips64el-linux-user
> + TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
> build:
> pre_ci:
> - make docker-image-${IMAGE}
If you fix the order of your patches you can get rid of this noise.
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 23/23] MAINTAINERS: self-appoint me as reviewer in build/test automation
2017-05-08 4:57 ` [Qemu-devel] [PATCH 23/23] MAINTAINERS: self-appoint me as reviewer in build/test automation Philippe Mathieu-Daudé
@ 2017-05-08 11:22 ` Alex Bennée
0 siblings, 0 replies; 46+ messages in thread
From: Alex Bennée @ 2017-05-08 11:22 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: qemu-devel, Fam Zheng, Paolo Bonzini, Thomas Huth,
Stefan Hajnoczi, Michael S. Tsirkin, Markus Armbruster
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> MAINTAINERS | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 9251f575d8..c7520b76db 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1827,6 +1827,7 @@ Build and test automation
> -------------------------
> M: Alex Bennée <alex.bennee@linaro.org>
> M: Fam Zheng <famz@redhat.com>
> +R: Philippe Mathieu-Daudé <f4bug@amsat.org>
> L: qemu-devel@nongnu.org
> S: Maintained
> F: .travis.yml
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
--
Alex Bennée
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [RFC PATCH 03/23] shippable: ignore SSL verification
2017-05-08 10:58 ` Alex Bennée
@ 2017-05-08 11:56 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 11:56 UTC (permalink / raw)
To: Alex Bennée; +Cc: qemu-devel, Fam Zheng, Daniel P . Berrange
Hi Alex,
On 05/08/2017 07:58 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> shippable logs:
>> --------------
>> git_sync
>> - ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
>> Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
>> Cloning into '/root/src/github.com/philmd/qemu'...
>> fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
>> retrying 1 of 3 times...
>
> I've not seen this before but which git_sync stage is it. The host or
> the container? If it is the container that probably just means we are
> missing the CA Cert package.
You are right! Adding the ca-certificates package in the docker image
solves this issue I had in container's git_sync stage :)
Thank for the review,
Phil.
>
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> .shippable.yml | 4 ++++
>> 1 file changed, 4 insertions(+)
>>
>> diff --git a/.shippable.yml b/.shippable.yml
>> index 5170486ff9..b661e667b3 100644
>> --- a/.shippable.yml
>> +++ b/.shippable.yml
>> @@ -2,6 +2,10 @@ language: c
>> env:
>> global:
>> - LC_ALL=C
>> + # sometimes Shippable fails to clone from github (git_sync stage):
>> + # "Problem with the SSL CA cert (path? access rights?)"
>> + # for now disable SSL verification.
>> + - GIT_SSL_NO_VERIFY=1
>> matrix:
>> - IMAGE=debian-armhf-cross
>> TARGET_LIST=arm-softmmu,arm-linux-user
>
>
> --
> Alex Bennée
>
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus
2017-05-08 10:55 ` Alex Bennée
@ 2017-05-08 13:21 ` Philippe Mathieu-Daudé
2017-05-08 14:42 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 13:21 UTC (permalink / raw)
To: Alex Bennée; +Cc: qemu-devel, Fam Zheng
On 05/08/2017 07:55 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> .shippable.yml | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/.shippable.yml b/.shippable.yml
>> index 231c29b620..5170486ff9 100644
>> --- a/.shippable.yml
>> +++ b/.shippable.yml
>> @@ -20,4 +20,4 @@ build:
>> ci:
>> - unset CC
>> - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
>> - - make -j2
>> + - make -j$(getconf _NPROCESSORS_ONLN)
>
> Did you ever check what this generates on Shippable? AFAICT they only
> have one core for each docker image. I following the core+1 pattern
> although that may not make as much sense for low core counts.
From the 'Job node info':
$ sudo docker info
Architecture: x86_64
CPUs: 2
Checking at 'ci' stage:
$ echo "container proc:" `getconf _NPROCESSORS_ONLN` `getconf
_NPROCESSORS_CONF`
container proc: 2 2
I think the correct cores+1 pattern should instead be:
make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
So we stay building at full speed if Shippable eventually upgrade/allow
to use more cores.
Phil.
> --
> Alex Bennée
>
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus
2017-05-08 13:21 ` Philippe Mathieu-Daudé
@ 2017-05-08 14:42 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 14:42 UTC (permalink / raw)
To: Alex Bennée; +Cc: qemu-devel, Fam Zheng
>>> diff --git a/.shippable.yml b/.shippable.yml
>>> index 231c29b620..5170486ff9 100644
>>> --- a/.shippable.yml
>>> +++ b/.shippable.yml
>>> @@ -20,4 +20,4 @@ build:
>>> ci:
>>> - unset CC
>>> - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
>>> - - make -j2
>>> + - make -j$(getconf _NPROCESSORS_ONLN)
>>
>> Did you ever check what this generates on Shippable? AFAICT they only
>> have one core for each docker image. I following the core+1 pattern
>> although that may not make as much sense for low core counts.
>
> From the 'Job node info':
>
> $ sudo docker info
> Architecture: x86_64
> CPUs: 2
>
> Checking at 'ci' stage:
>
> $ echo "container proc:" `getconf _NPROCESSORS_ONLN` `getconf
> _NPROCESSORS_CONF`
> container proc: 2 2
>
> I think the correct cores+1 pattern should instead be:
>
> make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
>
> So we stay building at full speed if Shippable eventually upgrade/allow
> to use more cores.
For what it's worth using this pattern builds last on average <8min vs
>9min previously :)
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [RFC PATCH 14/23] docker: add 'apt-fake' script which generate fake debian packages
2017-05-08 11:12 ` Alex Bennée
@ 2017-05-08 17:48 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 17:48 UTC (permalink / raw)
To: Alex Bennée
Cc: qemu-devel, Fam Zheng, Aurelien Jarno, Riku Voipio,
Vagrant Cascadian, Michael Tokarev
On 05/08/2017 08:12 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> tests/docker/Makefile.include | 2 ++
>> tests/docker/dockerfiles/debian-apt-fake.sh | 46 +++++++++++++++++++++++++++++
>> tests/docker/dockerfiles/debian.docker | 2 ++
>> 3 files changed, 50 insertions(+)
>> create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index 47978fb56c..ca6f57a292 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -55,6 +55,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>> $(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
>> "BUILD","$*")
>>
>> +docker-image-debian: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
>> +
>
> Can we only include debian-apt-fake on the debian images that require it please?
Sure.
>> # Enforce dependancies for composite images
>> docker-image-debian-armhf-cross: docker-image-debian
>> docker-image-debian-arm64-cross: docker-image-debian
>> diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
>> new file mode 100755
>> index 0000000000..387522c174
>> --- /dev/null
>> +++ b/tests/docker/dockerfiles/debian-apt-fake.sh
>> @@ -0,0 +1,46 @@
>> +#! /bin/sh
>> +#
>> +# Generate fake debian package to resolve unimportant unmet dependencies held
>> +# by upstream multiarch broken packages.
>> +#
>> +# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2
>> +# or (at your option) any later version. See the COPYING file in
>> +# the top-level directory.
>> +
>> +test $1 = "install" && shift 1
>> +
>> +fake_install()
>> +{
>> + echo "Generating fake $2 $1 $3 ..."
>> + (cd /var/cache/apt/archives
>> + (cat << 'EOF'
>> +Section: misc
>> +Priority: optional
>> +Standards-Version: 3.9.2
>> +
>> +Package: NAME
>> +Version: VERSION
>> +Maintainer: qemu-devel@nongnu.org
>> +Architecture: any
>> +Multi-Arch: same
>> +Description: fake NAME
>> +EOF
>> + ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
>> + equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
>> + dpkg -i $2_$3_$1.deb
>> + )
>> +}
>> +
>> +try_install()
>> +{
>> + name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
>> + arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
>> + vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
>> + apt-get install -q -yy $1 || fake_install $arch $name $vers
>> +}
>> +
>> +for package in $*; do
>> + try_install $package
>> +done
>> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
>> index 694f8d3b63..a99396ae8f 100644
>> --- a/tests/docker/dockerfiles/debian.docker
>> +++ b/tests/docker/dockerfiles/debian.docker
>> @@ -22,6 +22,8 @@ RUN apt-get update && \
>> # Duplicate deb line as deb-src
>> RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
>>
>> +ADD debian-apt-fake.sh /usr/local/bin/apt-fake
>> +
>
> And only add this in the images that need it.
>
>> # Setup Emdebian
>> RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
>> curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
>
>
> --
> Alex Bennée
>
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 16/23] shippable: do not initialize submodules automatically
2017-05-08 11:15 ` Alex Bennée
@ 2017-05-08 17:48 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 17:48 UTC (permalink / raw)
To: Alex Bennée; +Cc: qemu-devel, Fam Zheng
On 05/08/2017 08:15 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> do it in the 'ci' target when needed.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> .shippable.yml | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/.shippable.yml b/.shippable.yml
>> index fa2d6a4e9b..b3cbca458c 100644
>> --- a/.shippable.yml
>> +++ b/.shippable.yml
>> @@ -1,4 +1,6 @@
>> language: c
>> +git:
>> + submodules: false
>> env:
>> global:
>> - LC_ALL=C
>
> Maybe this should be merged with the next patch?
Ok.
> --
> Alex Bennée
>
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 21/23] docker: fix powerpc debian/stable dependencies problem
2017-05-08 11:19 ` Alex Bennée
@ 2017-05-08 17:50 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 17:50 UTC (permalink / raw)
To: Alex Bennée
Cc: qemu-devel, Fam Zheng, David Gibson, Alexander Graf, qemu-ppc,
Aurelien Jarno, Riku Voipio, Vagrant Cascadian, Michael Tokarev
On 05/08/2017 08:19 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> tests/docker/dockerfiles/debian-powerpc-cross.docker | 7 +++++++
>> 1 file changed, 7 insertions(+)
>>
>> diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> index 9107ebc2d3..d6e4d1e977 100644
>> --- a/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> +++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
>> @@ -12,6 +12,13 @@ RUN dpkg --add-architecture powerpc && \
>> DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
>> crossbuild-essential-powerpc
>>
>> +# <kludge> to fix "following packages have unmet dependencies" ...
>> +RUN apt-get install -y --no-install-recommends equivs pkg-config && \
>> + apt-fake install pkg-config:powerpc=0.28-1.1-fake && \
>> + ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
>> +ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
>> +# </kludge>
>> +
>> # Specify the cross prefix for this image (see tests/docker/common.rc)
>> ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
>
> Given the image needs these package hacks it is probably best folded
> into the commit adding the target.
Ok, will squash with the "ADD debian-apt-fake.sh
/usr/local/bin/apt-fake" line.
> --
> Alex Bennée
>
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 18/23] shippable: temporary disable mips64el-softmmu build
2017-05-08 11:22 ` Alex Bennée
@ 2017-05-08 17:52 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 17:52 UTC (permalink / raw)
To: Alex Bennée; +Cc: qemu-devel, Fam Zheng, Aurelien Jarno, Yongbok Kim
On 05/08/2017 08:22 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> .shippable.yml | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/.shippable.yml b/.shippable.yml
>> index 6601243060..7e714991f9 100644
>> --- a/.shippable.yml
>> +++ b/.shippable.yml
>> @@ -15,8 +15,9 @@ env:
>> TARGET_LIST=aarch64-softmmu,aarch64-linux-user
>> - IMAGE=debian-s390x-cross
>> TARGET_LIST=s390x-softmmu,s390x-linux-user
>> + # mips64el-softmmu disabled due to libfdt problem
>> - IMAGE=debian-mipsel-cross
>> - TARGET_LIST=mipsel-softmmu,mips64el-softmmu,mipsel-linux-user,mips64el-linux-user
>> + TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
>> build:
>> pre_ci:
>> - make docker-image-${IMAGE}
>
>
> If you fix the order of your patches you can get rid of this noise.
I'll squash with patch 11 (removing your R-B).
> --
> Alex Bennée
>
^ permalink raw reply [flat|nested] 46+ messages in thread
* Re: [Qemu-devel] [PATCH 04/23] docker: compact debian base
2017-05-08 11:04 ` Alex Bennée
@ 2017-05-08 18:19 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 46+ messages in thread
From: Philippe Mathieu-Daudé @ 2017-05-08 18:19 UTC (permalink / raw)
To: Alex Bennée; +Cc: qemu-devel, Fam Zheng
On 05/08/2017 08:04 AM, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
>> - install common/basic tools at once
>> - one-line Emdebian setup
>> - use eatmydata and remove apt cache to save space
>> - add bison and flex and git
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> tests/docker/dockerfiles/debian.docker | 22 ++++++++++++----------
>> 1 file changed, 12 insertions(+), 10 deletions(-)
>>
>> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
>> index 52bd79938e..694f8d3b63 100644
>> --- a/tests/docker/dockerfiles/debian.docker
>> +++ b/tests/docker/dockerfiles/debian.docker
>> @@ -9,17 +9,19 @@
>> #
>> FROM debian:stable-slim
>>
>> -# Setup some basic tools we need
>> -RUN apt update
>> -RUN apt install -yy curl aptitude
>> -
>> -# Setup Emdebian
>> -RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
>> -RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
>> +# Install some basic tools and common build utilities
>> +RUN apt-get update && \
>> + DEBIAN_FRONTEND=noninteractive apt-get install -yy \
>> + eatmydata && \
>
> I wonder if we should just use ENV instead to set DEBIAN_FRONTEND?
In https://github.com/moby/moby/issues/4032#issuecomment-34597177 Tianon
Gravi commented:
[...] setting it via ENV should be actively discouraged. [...]
The reason it's not a good default is that if I docker run -i -t ...
bash, I'm now interactive, and this variable being set is very, very wrong.
>
>> + DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
>> + curl aptitude \
>> + build-essential clang git \
>> + bison flex && \
>> + rm -rf /var/lib/apt
>
> Use apt clean to do this.
Ok.
>>
>> # Duplicate deb line as deb-src
>> RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
>>
>> -# Install common build utilities
>> -RUN apt update
>> -RUN apt install -yy build-essential clang
>> +# Setup Emdebian
>> +RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
>> + curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
>
> Why move this to the end? Once we have curl installed it would be nice
> to do just one update step and have emdebian synced up in the base
> image.
In the first apt-get can install curl + build-essential clang in the
same call, and there is no need to fetch emdebian.org index.
Using 'apt clean' at the end of a layer keeps smaller layer but the next
layer have to call `apt-get update` again. Since this base image is used
to cross-compile, the child image will have to call `dpkg
--add-architecture ...` before fetching emdebian.org arch index again
(apt-get update) to be able to install the `crossbuild-essential-$arch`
package.
So there is no need to update right after adding this repo, the 'child'
will do it (and have to do it).
>
> --
> Alex Bennée
>
^ permalink raw reply [flat|nested] 46+ messages in thread
end of thread, other threads:[~2017-05-08 18:19 UTC | newest]
Thread overview: 46+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-08 4:56 [Qemu-devel] [PATCH 00/23] docker/shippable: cross-build mipsel and powerpc targets Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 01/23] shippable: use C locale to simplify console output Philippe Mathieu-Daudé
2017-05-08 10:54 ` Alex Bennée
2017-05-08 4:56 ` [Qemu-devel] [PATCH 02/23] shippable: build using all available cpus Philippe Mathieu-Daudé
2017-05-08 10:55 ` Alex Bennée
2017-05-08 13:21 ` Philippe Mathieu-Daudé
2017-05-08 14:42 ` Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [RFC PATCH 03/23] shippable: ignore SSL verification Philippe Mathieu-Daudé
2017-05-08 10:58 ` Alex Bennée
2017-05-08 11:56 ` Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 04/23] docker: compact debian base Philippe Mathieu-Daudé
2017-05-08 11:04 ` Alex Bennée
2017-05-08 18:19 ` Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 05/23] docker: compact debian armhf Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 06/23] docker: compact debian arm64 Philippe Mathieu-Daudé
2017-05-08 4:56 ` [Qemu-devel] [PATCH 07/23] docker: add extra libs to armhf target to extend codebase coverage Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 08/23] docker: add extra libs to arm64 " Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 09/23] docker: add extra libs to s390x " Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 10/23] docker: add mipsel build target Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 11/23] shippable: add mipsel target Philippe Mathieu-Daudé
2017-05-08 11:06 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 12/23] docker: let _copy_with_mkdir() sub_path argument be optional Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [RFC PATCH 13/23] docker: add --include-file argument to 'build' command Philippe Mathieu-Daudé
2017-05-08 11:10 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [RFC PATCH 14/23] docker: add 'apt-fake' script which generate fake debian packages Philippe Mathieu-Daudé
2017-05-08 11:12 ` Alex Bennée
2017-05-08 17:48 ` Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 15/23] docker: fix mipsel debian/stable dependencies problem Philippe Mathieu-Daudé
2017-05-08 11:14 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 16/23] shippable: do not initialize submodules automatically Philippe Mathieu-Daudé
2017-05-08 11:15 ` Alex Bennée
2017-05-08 17:48 ` Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 17/23] shippable: use dtc submodule if distrib packages are too old Philippe Mathieu-Daudé
2017-05-08 11:15 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 19/23] docker: add powerpc build target Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 20/23] shippable: add powerpc target Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 21/23] docker: fix powerpc debian/stable dependencies problem Philippe Mathieu-Daudé
2017-05-08 11:19 ` Alex Bennée
2017-05-08 17:50 ` Philippe Mathieu-Daudé
2017-05-08 4:57 ` [Qemu-devel] [PATCH 22/23] MAINTAINERS: add Shippable automation platform URL Philippe Mathieu-Daudé
2017-05-08 11:21 ` Alex Bennée
2017-05-08 4:57 ` [Qemu-devel] [PATCH 23/23] MAINTAINERS: self-appoint me as reviewer in build/test automation Philippe Mathieu-Daudé
2017-05-08 11:22 ` Alex Bennée
2017-05-08 5:02 ` [Qemu-devel] [PATCH 18/23] shippable: temporary disable mips64el-softmmu build Philippe Mathieu-Daudé
2017-05-08 11:22 ` Alex Bennée
2017-05-08 17:52 ` Philippe Mathieu-Daudé
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.