[Qemu-devel] [PATCH v2 00/21] docker/shippable: cross-build mipsel and powerpc targets

[Qemu-devel] [PATCH v2 00/21] docker/shippable: cross-build mipsel and powerpc targets

[Philippe Mathieu-Daudé]

This patchset add 2 more architectures to the cross-build farm.

There is still some issue trying to cross-build mips64el-softmmu, it seems the
cross tools use the system outdated libfdt instead of the one checkouted in the
dtc submodule. I disabled this target for now.

The branch https://github.com/philmd/qemu/tree/docker_shippable_v2 can be
checked at Shippable:
  https://app.shippable.com/github/philmd/qemu/status/dashboard

With the free open source projects service, each arch builds in around ~8min,
the 5 jobs take ~38min in total.

v2:
- Addressed review feedback from Alex, added his R-B
- Fixed the "Problem with the SSL CA cert" problem while cloning from github.com
  installing the ca-certificates package.
- Squashed/split some commits
- use 'apt-get clean' instead of brutal 'rm -rf'
- checked how many cores are available on Shippable and use them fully
  (reduced total time from 44min to 38min)
- build armeb-linux-user target

v1:
- Reorganize Dockerfiles to use less layers, resulting in smaller images.
  This also reduce time of image transfer, for example while using:
  `docker save qemu:debian | ssh remote docker load`
- Install more debian packages so the configure script enable more features and
  more code can be compiled/covered.
- There are still some incorrect multiarch packages on debian/jessie used in
  the docker images, add a script to generate fake packages and avoid
  dependencies issues.
- Modify the docker.py script to include an extra file while building images.

Regards,

Phil.

Philippe Mathieu-Daudé (21):
  docker: let _copy_with_mkdir() sub_path argument be optional
  docker: add --include-file argument to 'build' command
  docker: add 'apt-fake' script which generate fake debian packages
  docker: install ca-certificates package in base image
  docker: compact debian base
  docker: compact debian armhf
  docker: compact debian arm64
  docker: add extra libs to armhf target to extend codebase coverage
  docker: add extra libs to arm64 target to extend codebase coverage
  docker: add extra libs to s390x target to extend codebase coverage
  docker: add mipsel build target
  docker: add powerpc build target
  shippable: use C locale to simplify console output
  shippable: build using all available cpus
  shippable: do not initialize submodules automatically
  shippable: be verbose while building docker images
  shippable: add armeb-linux-user target
  shippable: add powerpc target
  shippable: add mipsel target
  MAINTAINERS: add Shippable automation platform URL
  MAINTAINERS: self-appoint me as reviewer in build/test automation

 .shippable.yml                                     | 23 +++++++++--
 MAINTAINERS                                        |  2 +
 tests/docker/Makefile.include                      |  8 +++-
 tests/docker/docker.py                             |  7 +++-
 tests/docker/dockerfiles/debian-apt-fake.sh        | 46 ++++++++++++++++++++++
 tests/docker/dockerfiles/debian-arm64-cross.docker | 16 ++++++--
 tests/docker/dockerfiles/debian-armhf-cross.docker | 16 ++++++--
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 31 +++++++++++++++
 .../docker/dockerfiles/debian-powerpc-cross.docker | 31 +++++++++++++++
 tests/docker/dockerfiles/debian-s390x-cross.docker |  6 +++
 tests/docker/dockerfiles/debian.docker             | 26 ++++++------
 11 files changed, 186 insertions(+), 26 deletions(-)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh
 create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker

-- 
2.11.0

[Qemu-devel] [PATCH v2 01/21] docker: let _copy_with_mkdir() sub_path argument be optional

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/docker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 8747f6a440..6ddc6e4c2a 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -52,7 +52,7 @@ def _guess_docker_command():
     raise Exception("Cannot find working docker command. Tried:\n%s" % \
                     commands_txt)
 
-def _copy_with_mkdir(src, root_dir, sub_path):
+def _copy_with_mkdir(src, root_dir, sub_path='.'):
     """Copy src into root_dir, creating sub_path as needed."""
     dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path))
     try:
-- 
2.11.0

[Qemu-devel] [PATCH v2 02/21] docker: add --include-file argument to 'build' command

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include | 3 ++-
 tests/docker/docker.py        | 5 +++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 03eda37bf4..c99ce702ea 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -51,7 +51,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
 		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
 		$(if $(NOUSER),,--add-current-user) \
-		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
+		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE))\
+		$(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
 		"BUILD","$*")
 
 # Enforce dependancies for composite images
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 6ddc6e4c2a..90520e4bca 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -237,6 +237,9 @@ class BuildCommand(SubCommand):
                             help="""Specify a binary that will be copied to the
                             container together with all its dependent
                             libraries""")
+        parser.add_argument("--include-file", "-f",
+                            help="""Specify a file that will be copied to the
+                            container""")
         parser.add_argument("--add-current-user", "-u", dest="user",
                             action="store_true",
                             help="Add the current user to image's passwd")
@@ -274,6 +277,8 @@ class BuildCommand(SubCommand):
             if args.include_executable:
                 _copy_binary_with_libs(args.include_executable,
                                        docker_dir)
+            if args.include_file:
+                _copy_with_mkdir(args.include_file, docker_dir)
 
             argv += ["--build-arg=" + k.lower() + "=" + v
                         for k, v in os.environ.iteritems()
-- 
2.11.0

[Qemu-devel] [PATCH v2 03/21] docker: add 'apt-fake' script which generate fake debian packages

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-apt-fake.sh | 46 +++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 tests/docker/dockerfiles/debian-apt-fake.sh

diff --git a/tests/docker/dockerfiles/debian-apt-fake.sh b/tests/docker/dockerfiles/debian-apt-fake.sh
new file mode 100755
index 0000000000..387522c174
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-apt-fake.sh
@@ -0,0 +1,46 @@
+#! /bin/sh
+#
+# Generate fake debian package to resolve unimportant unmet dependencies held
+# by upstream multiarch broken packages.
+#
+# Copyright (c) 2017 Philippe Mathieu-Daudé <f4bug@amsat.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+test $1 = "install" && shift 1
+
+fake_install()
+{
+    echo "Generating fake $2 $1 $3 ..."
+    (cd /var/cache/apt/archives
+        (cat << 'EOF'
+Section: misc
+Priority: optional
+Standards-Version: 3.9.2
+
+Package: NAME
+Version: VERSION
+Maintainer: qemu-devel@nongnu.org
+Architecture: any
+Multi-Arch: same
+Description: fake NAME
+EOF
+        ) | sed s/NAME/$2/g | sed s/VERSION/$3/g > $2.control
+        equivs-build -a $1 $2.control 1>/dev/null 2>/dev/null
+        dpkg -i $2_$3_$1.deb
+    )
+}
+
+try_install()
+{
+    name=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\1/")
+    arch=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\2/")
+    vers=$(echo $1|sed "s/\(.*\):\(.*\)=\(.*\)/\3/")
+    apt-get install -q -yy $1 || fake_install $arch $name $vers
+}
+
+for package in $*; do
+    try_install $package
+done
-- 
2.11.0

[Qemu-devel] [PATCH v2 04/21] docker: install ca-certificates package in base image

[Philippe Mathieu-Daudé]

Resolve SSL verification issue at shippable container's git_sync stage:

shippable logs:
--------------
git_sync
- ssh-agent bash -c 'ssh-add /tmp/ssh/01_deploy; git clone https://github.com/philmd/qemu.git /root/src/github.com/philmd/qemu'
Identity added: /tmp/ssh/01_deploy (rsa w/o comment)
Cloning into '/root/src/github.com/philmd/qemu'...
fatal: unable to access 'https://github.com/philmd/qemu.git/': Problem with the SSL CA cert (path? access rights?)
retrying 1 of 3 times...

Suggested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian.docker | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index 52bd79938e..d08def6a8d 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -11,7 +11,7 @@ FROM debian:stable-slim
 
 # Setup some basic tools we need
 RUN apt update
-RUN apt install -yy curl aptitude
+RUN apt install -yy aptitude ca-certificates curl
 
 # Setup Emdebian
 RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
-- 
2.11.0

[Qemu-devel] [PATCH v2 05/21] docker: compact debian base

[Philippe Mathieu-Daudé]

- install common/basic tools at once
- use eatmydata and remove apt cache to save space
- add bison and flex and git
- create deb-src entry and setup Emdebian in the same layer

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian.docker | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
index d08def6a8d..dcded3ce84 100644
--- a/tests/docker/dockerfiles/debian.docker
+++ b/tests/docker/dockerfiles/debian.docker
@@ -9,17 +9,17 @@
 #
 FROM debian:stable-slim
 
-# Setup some basic tools we need
-RUN apt update
-RUN apt install -yy aptitude ca-certificates curl
+# Install some basic tools and common build utilities
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -yy \
+        eatmydata && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        aptitude ca-certificates curl \
+        build-essential clang git \
+        bison flex && \
+    apt-get clean
 
-# Setup Emdebian
-RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
-RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
-
-# Install common build utilities
-RUN apt update
-RUN apt install -yy build-essential clang
+# Duplicate deb line as deb-src, setup Emdebian
+RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list && \
+    echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
+    curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
-- 
2.11.0

[Qemu-devel] [PATCH v2 06/21] docker: compact debian armhf

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-armhf-cross.docker | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 668d60aeb3..533e525971 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -6,10 +6,12 @@
 FROM qemu:debian
 
 # Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture armhf
-RUN apt update
-RUN apt install -yy crossbuild-essential-armhf
-RUN apt-get build-dep -yy -a armhf qemu
+RUN dpkg --add-architecture armhf && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        crossbuild-essential-armhf
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a armhf qemu
-- 
2.11.0

[Qemu-devel] [PATCH v2 07/21] docker: compact debian arm64

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-arm64-cross.docker | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 592b5d7055..12b30aae54 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -6,10 +6,12 @@
 FROM qemu:debian
 
 # Add the foreign architecture we want and install dependencies
-RUN dpkg --add-architecture arm64
-RUN apt update
-RUN apt install -yy crossbuild-essential-arm64
-RUN apt-get build-dep -yy -a arm64 qemu
+RUN dpkg --add-architecture arm64 && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        crossbuild-essential-arm64
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a arm64 qemu
-- 
2.11.0

[Qemu-devel] [PATCH v2 08/21] docker: add extra libs to armhf target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-armhf-cross.docker | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 533e525971..01d9fcb7f2 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -14,4 +14,10 @@ RUN dpkg --add-architecture armhf && \
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=arm-linux-gnueabihf-
 
-RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a armhf qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a armhf qemu && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        glusterfs-common:armhf \
+        libncursesw5-dev:armhf libnfs-dev:armhf \
+        libbz2-dev:armhf liblzo2-dev:armhf \
+        libsnappy-dev:armhf librdmacm-dev:armhf \
+        libxen-dev:armhf
-- 
2.11.0

[Qemu-devel] [PATCH v2 09/21] docker: add extra libs to arm64 target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-arm64-cross.docker | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 12b30aae54..5e0036917b 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -14,4 +14,10 @@ RUN dpkg --add-architecture arm64 && \
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=aarch64-linux-gnu-
 
-RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a arm64 qemu
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a arm64 qemu && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        glusterfs-common:arm64 \
+        libncursesw5-dev:arm64 libnfs-dev:arm64 \
+        libbz2-dev:arm64 liblzo2-dev:arm64 \
+        libsnappy-dev:arm64 librdmacm-dev:arm64 \
+        libxen-dev:arm64
-- 
2.11.0

[Qemu-devel] [PATCH v2 10/21] docker: add extra libs to s390x target to extend codebase coverage

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/dockerfiles/debian-s390x-cross.docker | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index 3a687feda0..119b152971 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -20,3 +20,9 @@ RUN apt install -yy gcc-multilib-s390x-linux-gnu binutils-multiarch
 
 # Specify the cross prefix for this image (see tests/docker/common.rc)
 ENV QEMU_CONFIGURE_OPTS --cross-prefix=s390x-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    glusterfs-common:s390x \
+    libncursesw5-dev:s390x libnfs-dev:s390x \
+    libbz2-dev:s390x liblzo2-dev:s390x \
+    libsnappy-dev:s390x librdmacm-dev:s390x
-- 
2.11.0

[Qemu-devel] [PATCH v2 11/21] docker: add mipsel build target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include                      |  3 +++
 .../docker/dockerfiles/debian-mipsel-cross.docker  | 31 ++++++++++++++++++++++
 2 files changed, 34 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-mipsel-cross.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index c99ce702ea..f34282a2cb 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -55,9 +55,12 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		$(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
 		"BUILD","$*")
 
+docker-image-debian-mipsel-cross: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
+
 # Enforce dependancies for composite images
 docker-image-debian-armhf-cross: docker-image-debian
 docker-image-debian-arm64-cross: docker-image-debian
+docker-image-debian-mipsel-cross: docker-image-debian
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES), \
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
new file mode 100644
index 0000000000..e4e8d22a7c
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -0,0 +1,31 @@
+#
+# Docker mipsel cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture mipsel && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        crossbuild-essential-mipsel
+
+# <kludge> to fix "following packages have unmet dependencies" ...
+ADD debian-apt-fake.sh /usr/local/bin/apt-fake
+RUN apt-get install -y --no-install-recommends equivs pkg-config && \
+    apt-fake install pkgconf:mipsel=0.9.7-fake pkg-config:mipsel=0.28-1.1-fake && \
+    ln -s pkg-config /usr/bin/mipsel-linux-gnu-pkg-config
+ENV PKG_CONFIG_PATH /usr/lib/mipsel-linux-gnu/pkgconfig
+# </kludge>
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=mipsel-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a mipsel qemu && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        glusterfs-common:mipsel \
+        libncursesw5-dev:mipsel libnfs-dev:mipsel \
+        libbz2-dev:mipsel liblzo2-dev:mipsel \
+        libsnappy-dev:mipsel librdmacm-dev:mipsel
-- 
2.11.0

[Qemu-devel] [PATCH v2 12/21] docker: add powerpc build target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/docker/Makefile.include                      |  2 ++
 .../docker/dockerfiles/debian-powerpc-cross.docker | 31 ++++++++++++++++++++++
 2 files changed, 33 insertions(+)
 create mode 100644 tests/docker/dockerfiles/debian-powerpc-cross.docker

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index f34282a2cb..417d8ea576 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -56,11 +56,13 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 		"BUILD","$*")
 
 docker-image-debian-mipsel-cross: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
+docker-image-debian-powerpc-cross: EXTRA_FILE:=tests/docker/dockerfiles/debian-apt-fake.sh
 
 # Enforce dependancies for composite images
 docker-image-debian-armhf-cross: docker-image-debian
 docker-image-debian-arm64-cross: docker-image-debian
 docker-image-debian-mipsel-cross: docker-image-debian
+docker-image-debian-powerpc-cross: docker-image-debian
 
 # Expand all the pre-requistes for each docker image and test combination
 $(foreach i,$(DOCKER_IMAGES), \
diff --git a/tests/docker/dockerfiles/debian-powerpc-cross.docker b/tests/docker/dockerfiles/debian-powerpc-cross.docker
new file mode 100644
index 0000000000..e0d4c454cc
--- /dev/null
+++ b/tests/docker/dockerfiles/debian-powerpc-cross.docker
@@ -0,0 +1,31 @@
+#
+# Docker powerpc cross-compiler target
+#
+# This docker target builds on the base debian image.
+#
+FROM qemu:debian
+MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
+
+# Add the foreign architecture we want and install dependencies
+RUN dpkg --add-architecture powerpc && \
+    apt-get update && \
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
+        crossbuild-essential-powerpc
+
+# <kludge> to fix "following packages have unmet dependencies" ...
+ADD debian-apt-fake.sh /usr/local/bin/apt-fake
+RUN apt-get install -y --no-install-recommends equivs pkg-config && \
+    apt-fake install pkg-config:powerpc=0.28-1.1-fake && \
+    ln -s pkg-config /usr/bin/powerpc-linux-gnu-pkg-config
+ENV PKG_CONFIG_PATH /usr/lib/powerpc-linux-gnu/pkgconfig
+# </kludge>
+
+# Specify the cross prefix for this image (see tests/docker/common.rc)
+ENV QEMU_CONFIGURE_OPTS --cross-prefix=powerpc-linux-gnu-
+
+RUN DEBIAN_FRONTEND=noninteractive eatmydata apt-get build-dep -yy -a powerpc qemu && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+        glusterfs-common:powerpc \
+        libncursesw5-dev:powerpc libnfs-dev:powerpc \
+        libbz2-dev:powerpc liblzo2-dev:powerpc \
+        libsnappy-dev:powerpc librdmacm-dev:powerpc
-- 
2.11.0

[Qemu-devel] [PATCH v2 13/21] shippable: use C locale to simplify console output

[Philippe Mathieu-Daudé]

remove this noise:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = "en_US.UTF-8",
    LC_CTYPE = "en_US.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 653bd750fe..231c29b620 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,5 +1,7 @@
 language: c
 env:
+  global:
+    - LC_ALL=C
   matrix:
     - IMAGE=debian-armhf-cross
       TARGET_LIST=arm-softmmu,arm-linux-user
-- 
2.11.0

[Qemu-devel] [PATCH v2 14/21] shippable: build using all available cpus

[Philippe Mathieu-Daudé]

As of this commit:

$ echo "container proc:" `getconf _NPROCESSORS_ONLN` `getconf _NPROCESSORS_CONF`
container proc: 2 2

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index 231c29b620..1e3ae35dd9 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -20,4 +20,4 @@ build:
   ci:
     - unset CC
     - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
-    - make -j2
+    - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
-- 
2.11.0

[Qemu-devel] [PATCH v2 15/21] shippable: do not initialize submodules automatically

[Philippe Mathieu-Daudé]

instead do it in the 'ci' target when needed.

for mips64el-softmmu target:
use dtc submodule if distrib packages are too old.

example with outdated libfdt on mips64el-softmmu target (required is >= 1.4.2):
 # dpkg-query --showformat='${Version}\n' --show libfdt-dev
 1.4.0+dfsg-1

shippable output:
----------------
  LINK    mips64el-softmmu/qemu-system-mips64el
../hw/core/loader-fit.o: In function `load_fit':
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:278: undefined reference to `fdt_first_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:286: undefined reference to `fdt_next_subnode'
/root/src/github.com/philmd/qemu/hw/core/loader-fit.c:277: undefined reference to `fdt_first_subnode'
collect2: error: ld returned 1 exit status
Makefile:201: recipe for target 'qemu-system-mips64el' failed
make[1]: *** [qemu-system-mips64el] Error 1
Makefile:327: recipe for target 'subdir-mips64el-softmmu' failed
make: *** [subdir-mips64el-softmmu] Error 2

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 1e3ae35dd9..46adfa030f 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -1,4 +1,6 @@
 language: c
+git:
+   submodules: false
 env:
   global:
     - LC_ALL=C
@@ -19,5 +21,13 @@ build:
     options: "-e HOME=/root"
   ci:
     - unset CC
+    # some targets require newer up to date packages, for example TARGET_LIST matching
+    # aarch64*-softmmu|arm*-softmmu|ppc*-softmmu|microblaze*-softmmu|mips64el-softmmu)
+    # see the configure script:
+    #    error_exit "DTC (libfdt) version >= 1.4.2 not present. Your options:"
+    #    "  (1) Preferred: Install the DTC (libfdt) devel package"
+    #    "  (2) Fetch the DTC submodule, using:"
+    #    "      git submodule update --init dtc"
+    - dpkg --compare-versions `dpkg-query --showformat='${Version}' --show libfdt-dev` ge 1.4.2 || git submodule update --init dtc
     - ./configure ${QEMU_CONFIGURE_OPTS} --target-list=${TARGET_LIST}
     - make -j$(($(getconf _NPROCESSORS_ONLN) + 1))
-- 
2.11.0

[Qemu-devel] [PATCH v2 16/21] shippable: be verbose while building docker images

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index 46adfa030f..fe360f85cb 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -13,7 +13,7 @@ env:
       TARGET_LIST=s390x-softmmu,s390x-linux-user
 build:
   pre_ci:
-    - make docker-image-${IMAGE}
+    - make docker-image-${IMAGE} V=1
   pre_ci_boot:
     image_name: qemu
     image_tag: ${IMAGE}
-- 
2.11.0

[Qemu-devel] [PATCH v2 17/21] shippable: add armeb-linux-user target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.shippable.yml b/.shippable.yml
index fe360f85cb..2070c4d827 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -6,7 +6,7 @@ env:
     - LC_ALL=C
   matrix:
     - IMAGE=debian-armhf-cross
-      TARGET_LIST=arm-softmmu,arm-linux-user
+      TARGET_LIST=arm-softmmu,arm-linux-user,armeb-linux-user
     - IMAGE=debian-arm64-cross
       TARGET_LIST=aarch64-softmmu,aarch64-linux-user
     - IMAGE=debian-s390x-cross
-- 
2.11.0

[Qemu-devel] [PATCH v2 18/21] shippable: add powerpc target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index 2070c4d827..aad66ec5ec 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -11,6 +11,8 @@ env:
       TARGET_LIST=aarch64-softmmu,aarch64-linux-user
     - IMAGE=debian-s390x-cross
       TARGET_LIST=s390x-softmmu,s390x-linux-user
+    - IMAGE=debian-powerpc-cross
+      TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
 build:
   pre_ci:
     - make docker-image-${IMAGE} V=1
-- 
2.11.0

[Qemu-devel] [PATCH v2 19/21] shippable: add mipsel target

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .shippable.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.shippable.yml b/.shippable.yml
index aad66ec5ec..75c2895a21 100644
--- a/.shippable.yml
+++ b/.shippable.yml
@@ -13,6 +13,9 @@ env:
       TARGET_LIST=s390x-softmmu,s390x-linux-user
     - IMAGE=debian-powerpc-cross
       TARGET_LIST=ppc-softmmu,ppcemb-softmmu,ppc-linux-user
+    # mips64el-softmmu disabled due to libfdt problem
+    - IMAGE=debian-mipsel-cross
+      TARGET_LIST=mipsel-softmmu,mipsel-linux-user,mips64el-linux-user
 build:
   pre_ci:
     - make docker-image-${IMAGE} V=1
-- 
2.11.0

[Qemu-devel] [PATCH v2 20/21] MAINTAINERS: add Shippable automation platform URL

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index efdec47319..6d0770fdd0 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1833,6 +1833,7 @@ F: .travis.yml
 F: .shippable.yml
 F: tests/docker/
 W: https://travis-ci.org/qemu/qemu
+W: https://app.shippable.com/github/qemu/qemu
 W: http://patchew.org/QEMU/
 
 Documentation
-- 
2.11.0

[Qemu-devel] [PATCH v2 21/21] MAINTAINERS: self-appoint me as reviewer in build/test automation

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 6d0770fdd0..d61380837a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1827,6 +1827,7 @@ Build and test automation
 -------------------------
 M: Alex Bennée <alex.bennee@linaro.org>
 M: Fam Zheng <famz@redhat.com>
+R: Philippe Mathieu-Daudé <f4bug@amsat.org>
 L: qemu-devel@nongnu.org
 S: Maintained
 F: .travis.yml
-- 
2.11.0

Re: [Qemu-devel] [PATCH v2 02/21] docker: add --include-file argument to 'build' command

[Fam Zheng]

On Mon, 05/08 19:17, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/Makefile.include | 3 ++-
>  tests/docker/docker.py        | 5 +++++
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 03eda37bf4..c99ce702ea 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -51,7 +51,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>  		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
>  		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
>  		$(if $(NOUSER),,--add-current-user) \
> -		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
> +		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE))\
> +		$(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
>  		"BUILD","$*")
>  
>  # Enforce dependancies for composite images
> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> index 6ddc6e4c2a..90520e4bca 100755
> --- a/tests/docker/docker.py
> +++ b/tests/docker/docker.py
> @@ -237,6 +237,9 @@ class BuildCommand(SubCommand):
>                              help="""Specify a binary that will be copied to the
>                              container together with all its dependent
>                              libraries""")
> +        parser.add_argument("--include-file", "-f",
> +                            help="""Specify a file that will be copied to the
> +                            container""")

Is it more useful if it's '--include-files' that accepts multiple files? Or
alternatively allow this option to be use multiple times:

    -f foo -f bar -f ...

Similarly, should EXTRA_FILE be EXTRA_FILES or something?

We are already lacking this with --include-executable's documentation, but
please also document "where" is it copied to in the container's filesystem.

Fam

>          parser.add_argument("--add-current-user", "-u", dest="user",
>                              action="store_true",
>                              help="Add the current user to image's passwd")
> @@ -274,6 +277,8 @@ class BuildCommand(SubCommand):
>              if args.include_executable:
>                  _copy_binary_with_libs(args.include_executable,
>                                         docker_dir)
> +            if args.include_file:
> +                _copy_with_mkdir(args.include_file, docker_dir)
>  
>              argv += ["--build-arg=" + k.lower() + "=" + v
>                          for k, v in os.environ.iteritems()
> -- 
> 2.11.0
> 

Re: [Qemu-devel] [PATCH v2 05/21] docker: compact debian base

[Fam Zheng]

On Mon, 05/08 19:17, Philippe Mathieu-Daudé wrote:
> - install common/basic tools at once
> - use eatmydata and remove apt cache to save space
> - add bison and flex and git
> - create deb-src entry and setup Emdebian in the same layer
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/docker/dockerfiles/debian.docker | 26 +++++++++++++-------------
>  1 file changed, 13 insertions(+), 13 deletions(-)
> 
> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> index d08def6a8d..dcded3ce84 100644
> --- a/tests/docker/dockerfiles/debian.docker
> +++ b/tests/docker/dockerfiles/debian.docker
> @@ -9,17 +9,17 @@
>  #
>  FROM debian:stable-slim
>  
> -# Setup some basic tools we need
> -RUN apt update
> -RUN apt install -yy aptitude ca-certificates curl
> +# Install some basic tools and common build utilities
> +RUN apt-get update && \
> +    DEBIAN_FRONTEND=noninteractive apt-get install -yy \
> +        eatmydata && \
> +    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
> +        aptitude ca-certificates curl \
> +        build-essential clang git \
> +        bison flex && \
> +    apt-get clean

Any particular reason to make multiple "RUN" directives into one?

Fam

>  
> -# Setup Emdebian
> -RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
> -RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
> -
> -# Duplicate deb line as deb-src
> -RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
> -
> -# Install common build utilities
> -RUN apt update
> -RUN apt install -yy build-essential clang
> +# Duplicate deb line as deb-src, setup Emdebian
> +RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list && \
> +    echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
> +    curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
> -- 
> 2.11.0
> 

Re: [Qemu-devel] [PATCH v2 05/21] docker: compact debian base

[Alex Bennée]

Fam Zheng <famz@redhat.com> writes:

> On Mon, 05/08 19:17, Philippe Mathieu-Daudé wrote:
>> - install common/basic tools at once
>> - use eatmydata and remove apt cache to save space
>> - add bison and flex and git
>> - create deb-src entry and setup Emdebian in the same layer
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/docker/dockerfiles/debian.docker | 26 +++++++++++++-------------
>>  1 file changed, 13 insertions(+), 13 deletions(-)
>>
>> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
>> index d08def6a8d..dcded3ce84 100644
>> --- a/tests/docker/dockerfiles/debian.docker
>> +++ b/tests/docker/dockerfiles/debian.docker
>> @@ -9,17 +9,17 @@
>>  #
>>  FROM debian:stable-slim
>>
>> -# Setup some basic tools we need
>> -RUN apt update
>> -RUN apt install -yy aptitude ca-certificates curl
>> +# Install some basic tools and common build utilities
>> +RUN apt-get update && \
>> +    DEBIAN_FRONTEND=noninteractive apt-get install -yy \
>> +        eatmydata && \
>> +    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
>> +        aptitude ca-certificates curl \
>> +        build-essential clang git \
>> +        bison flex && \
>> +    apt-get clean
>
> Any particular reason to make multiple "RUN" directives into one?

It reduces the number of layers you generate during the building
process. I'm not sure how much space that saves in the long run though
(disk space is pretty cheap ;-).

>
> Fam
>
>>
>> -# Setup Emdebian
>> -RUN echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list
>> -RUN curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
>> -
>> -# Duplicate deb line as deb-src
>> -RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list
>> -
>> -# Install common build utilities
>> -RUN apt update
>> -RUN apt install -yy build-essential clang
>> +# Duplicate deb line as deb-src, setup Emdebian
>> +RUN cat /etc/apt/sources.list | sed "s/deb/deb-src/" >> /etc/apt/sources.list && \
>> +    echo "deb http://emdebian.org/tools/debian/ jessie main" >> /etc/apt/sources.list && \
>> +    curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
>> --
>> 2.11.0
>>


--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 05/21] docker: compact debian base

[Fam Zheng]

On Tue, 05/09 08:46, Alex Bennée wrote:
> 
> Fam Zheng <famz@redhat.com> writes:
> 
> > On Mon, 05/08 19:17, Philippe Mathieu-Daudé wrote:
> >> - install common/basic tools at once
> >> - use eatmydata and remove apt cache to save space
> >> - add bison and flex and git
> >> - create deb-src entry and setup Emdebian in the same layer
> >>
> >> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >> ---
> >>  tests/docker/dockerfiles/debian.docker | 26 +++++++++++++-------------
> >>  1 file changed, 13 insertions(+), 13 deletions(-)
> >>
> >> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> >> index d08def6a8d..dcded3ce84 100644
> >> --- a/tests/docker/dockerfiles/debian.docker
> >> +++ b/tests/docker/dockerfiles/debian.docker
> >> @@ -9,17 +9,17 @@
> >>  #
> >>  FROM debian:stable-slim
> >>
> >> -# Setup some basic tools we need
> >> -RUN apt update
> >> -RUN apt install -yy aptitude ca-certificates curl
> >> +# Install some basic tools and common build utilities
> >> +RUN apt-get update && \
> >> +    DEBIAN_FRONTEND=noninteractive apt-get install -yy \
> >> +        eatmydata && \
> >> +    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
> >> +        aptitude ca-certificates curl \
> >> +        build-essential clang git \
> >> +        bison flex && \
> >> +    apt-get clean
> >
> > Any particular reason to make multiple "RUN" directives into one?
> 
> It reduces the number of layers you generate during the building
> process. I'm not sure how much space that saves in the long run though
> (disk space is pretty cheap ;-).

I don't think it matters much, but refreshing an image gets slower due to
coarser cache, for example when you add a package in the big "apt-get install"
command.

I don't see a strong reason why this is better. It clutters the file and is
inconsistent.

Re: [Qemu-devel] [PATCH v2 05/21] docker: compact debian base

[Philippe Mathieu-Daudé]

On 05/09/2017 05:05 AM, Fam Zheng wrote:
> On Tue, 05/09 08:46, Alex Bennée wrote:
>>
>> Fam Zheng <famz@redhat.com> writes:
>>
>>> On Mon, 05/08 19:17, Philippe Mathieu-Daudé wrote:
>>>> - install common/basic tools at once
>>>> - use eatmydata and remove apt cache to save space
>>>> - add bison and flex and git
>>>> - create deb-src entry and setup Emdebian in the same layer
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>> ---
>>>>  tests/docker/dockerfiles/debian.docker | 26 +++++++++++++-------------
>>>>  1 file changed, 13 insertions(+), 13 deletions(-)
>>>>
>>>> diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
>>>> index d08def6a8d..dcded3ce84 100644
>>>> --- a/tests/docker/dockerfiles/debian.docker
>>>> +++ b/tests/docker/dockerfiles/debian.docker
>>>> @@ -9,17 +9,17 @@
>>>>  #
>>>>  FROM debian:stable-slim
>>>>
>>>> -# Setup some basic tools we need
>>>> -RUN apt update
>>>> -RUN apt install -yy aptitude ca-certificates curl
>>>> +# Install some basic tools and common build utilities
>>>> +RUN apt-get update && \
>>>> +    DEBIAN_FRONTEND=noninteractive apt-get install -yy \
>>>> +        eatmydata && \
>>>> +    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
>>>> +        aptitude ca-certificates curl \
>>>> +        build-essential clang git \
>>>> +        bison flex && \
>>>> +    apt-get clean
>>>
>>> Any particular reason to make multiple "RUN" directives into one?
>>
>> It reduces the number of layers you generate during the building
>> process. I'm not sure how much space that saves in the long run though
>> (disk space is pretty cheap ;-).
>
> I don't think it matters much, but refreshing an image gets slower due to
> coarser cache, for example when you add a package in the big "apt-get install"
> command.
>
> I don't see a strong reason why this is better. It clutters the file and is
> inconsistent.

I'm living in a remote place where Internet is pretty bad so it does 
save me some time to have reduced layer while using 'docker save | 
docker load', but no worry :)

I'll revert this patch and just add bison/flex/git packages, and 
eventually keep DEBIAN_FRONTEND=noninteractive to avoid harmless error 
messages.

Re: [Qemu-devel] [PATCH v2 02/21] docker: add --include-file argument to 'build' command

[Philippe Mathieu-Daudé]

On 05/09/2017 01:01 AM, Fam Zheng wrote:
> On Mon, 05/08 19:17, Philippe Mathieu-Daudé wrote:
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  tests/docker/Makefile.include | 3 ++-
>>  tests/docker/docker.py        | 5 +++++
>>  2 files changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index 03eda37bf4..c99ce702ea 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -51,7 +51,8 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
>>  		$(SRC_PATH)/tests/docker/docker.py build qemu:$* $< \
>>  		$(if $V,,--quiet) $(if $(NOCACHE),--no-cache) \
>>  		$(if $(NOUSER),,--add-current-user) \
>> -		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)),\
>> +		$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE))\
>> +		$(if $(EXTRA_FILE),--include-file=$(EXTRA_FILE)),\
>>  		"BUILD","$*")
>>
>>  # Enforce dependancies for composite images
>> diff --git a/tests/docker/docker.py b/tests/docker/docker.py
>> index 6ddc6e4c2a..90520e4bca 100755
>> --- a/tests/docker/docker.py
>> +++ b/tests/docker/docker.py
>> @@ -237,6 +237,9 @@ class BuildCommand(SubCommand):
>>                              help="""Specify a binary that will be copied to the
>>                              container together with all its dependent
>>                              libraries""")
>> +        parser.add_argument("--include-file", "-f",
>> +                            help="""Specify a file that will be copied to the
>> +                            container""")
>
> Is it more useful if it's '--include-files' that accepts multiple files? Or
> alternatively allow this option to be use multiple times:
>
>     -f foo -f bar -f ...

I agree, since I'm not confident with Python so I tried to touch as 
little as possible.

One iter I'm thinking about do a consistent rename with the 
"dockerfiles/%.docker -> docker.d/%/Dockerfile" pattern and adding git 
symlinks to required EXTRA_FILES in each dir, to avoid the call to 
docker.py and be able to build/push images on hub.docker.com directly.

> Similarly, should EXTRA_FILE be EXTRA_FILES or something?

Yes.

> We are already lacking this with --include-executable's documentation, but
> please also document "where" is it copied to in the container's filesystem.

Ok.

Thank for the review,

Phil.

> Fam
>
>>          parser.add_argument("--add-current-user", "-u", dest="user",
>>                              action="store_true",
>>                              help="Add the current user to image's passwd")
>> @@ -274,6 +277,8 @@ class BuildCommand(SubCommand):
>>              if args.include_executable:
>>                  _copy_binary_with_libs(args.include_executable,
>>                                         docker_dir)
>> +            if args.include_file:
>> +                _copy_with_mkdir(args.include_file, docker_dir)
>>
>>              argv += ["--build-arg=" + k.lower() + "=" + v
>>                          for k, v in os.environ.iteritems()
>> --
>> 2.11.0
>>
>

Re: [Qemu-devel] [PATCH v2 05/21] docker: compact debian base

[Fam Zheng]

On Tue, 05/09 09:06, Philippe Mathieu-Daudé wrote:
> On 05/09/2017 05:05 AM, Fam Zheng wrote:
> > On Tue, 05/09 08:46, Alex Bennée wrote:
> > > 
> > > Fam Zheng <famz@redhat.com> writes:
> > > 
> > > > On Mon, 05/08 19:17, Philippe Mathieu-Daudé wrote:
> > > > > - install common/basic tools at once
> > > > > - use eatmydata and remove apt cache to save space
> > > > > - add bison and flex and git
> > > > > - create deb-src entry and setup Emdebian in the same layer
> > > > > 
> > > > > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > > > > ---
> > > > >  tests/docker/dockerfiles/debian.docker | 26 +++++++++++++-------------
> > > > >  1 file changed, 13 insertions(+), 13 deletions(-)
> > > > > 
> > > > > diff --git a/tests/docker/dockerfiles/debian.docker b/tests/docker/dockerfiles/debian.docker
> > > > > index d08def6a8d..dcded3ce84 100644
> > > > > --- a/tests/docker/dockerfiles/debian.docker
> > > > > +++ b/tests/docker/dockerfiles/debian.docker
> > > > > @@ -9,17 +9,17 @@
> > > > >  #
> > > > >  FROM debian:stable-slim
> > > > > 
> > > > > -# Setup some basic tools we need
> > > > > -RUN apt update
> > > > > -RUN apt install -yy aptitude ca-certificates curl
> > > > > +# Install some basic tools and common build utilities
> > > > > +RUN apt-get update && \
> > > > > +    DEBIAN_FRONTEND=noninteractive apt-get install -yy \
> > > > > +        eatmydata && \
> > > > > +    DEBIAN_FRONTEND=noninteractive eatmydata apt-get install -y --no-install-recommends \
> > > > > +        aptitude ca-certificates curl \
> > > > > +        build-essential clang git \
> > > > > +        bison flex && \
> > > > > +    apt-get clean
> > > > 
> > > > Any particular reason to make multiple "RUN" directives into one?
> > > 
> > > It reduces the number of layers you generate during the building
> > > process. I'm not sure how much space that saves in the long run though
> > > (disk space is pretty cheap ;-).
> > 
> > I don't think it matters much, but refreshing an image gets slower due to
> > coarser cache, for example when you add a package in the big "apt-get install"
> > command.
> > 
> > I don't see a strong reason why this is better. It clutters the file and is
> > inconsistent.
> 
> I'm living in a remote place where Internet is pretty bad so it does save me
> some time to have reduced layer while using 'docker save | docker load', but
> no worry :)

Shouldn't 'docker save' collapse the layers anyway and output final data?

On the contrary, Docker build cache saves me time too when I play with the
package list, as 'apt-get update' result is cached in a previous RUN directive,
and doesn't need to run again and again. (Yes, I live in a place where "apt-get
update" is slow :)

Fam

Re: [Qemu-devel] [PATCH v2 02/21] docker: add --include-file argument to 'build' command

[Fam Zheng]

On Tue, 05/09 09:14, Philippe Mathieu-Daudé wrote:
> > 
> > Is it more useful if it's '--include-files' that accepts multiple files? Or
> > alternatively allow this option to be use multiple times:
> > 
> >     -f foo -f bar -f ...
> 
> I agree, since I'm not confident with Python so I tried to touch as little
> as possible.
> 
> One iter I'm thinking about do a consistent rename with the
> "dockerfiles/%.docker -> docker.d/%/Dockerfile" pattern and adding git
> symlinks to required EXTRA_FILES in each dir, to avoid the call to docker.py
> and be able to build/push images on hub.docker.com directly.

Originally I thought about "$QEMU/tests/docker/dockerfiles/$IMAGE/Dockerfile"
but that sounded too fussy.

"$QEMU/tests/docker/docker.d/$IMAGE/Dockerfile" is a bit better.

But I'm not sure doing that can allow us to remove the necessity of docker.py -
it does more such as calling .pre and passing in env vars etc..

Fam

Re: [Qemu-devel] [PATCH v2 00/21] docker/shippable: cross-build mipsel and powerpc targets

[Alex Bennée]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> This patchset add 2 more architectures to the cross-build farm.
>
> There is still some issue trying to cross-build mips64el-softmmu, it seems the
> cross tools use the system outdated libfdt instead of the one checkouted in the
> dtc submodule. I disabled this target for now.
>
> The branch https://github.com/philmd/qemu/tree/docker_shippable_v2 can be
> checked at Shippable:
>   https://app.shippable.com/github/philmd/qemu/status/dashboard
>
> With the free open source projects service, each arch builds in around ~8min,
> the 5 jobs take ~38min in total.
>
> v2:
> - Addressed review feedback from Alex, added his R-B
> - Fixed the "Problem with the SSL CA cert" problem while cloning from github.com
>   installing the ca-certificates package.
> - Squashed/split some commits
> - use 'apt-get clean' instead of brutal 'rm -rf'
> - checked how many cores are available on Shippable and use them fully
>   (reduced total time from 44min to 38min)
> - build armeb-linux-user target

Ping - is there a v3 coming with Fam's comments addressed?

<snip>

--
Alex Bennée

Re: [Qemu-devel] [PATCH v2 00/21] docker/shippable: cross-build mipsel and powerpc targets

[Philippe Mathieu-Daudé]

Hi Alex,

On 05/17/2017 11:28 AM, Alex Bennée wrote:
>> v2:
>> - Addressed review feedback from Alex, added his R-B
>> - Fixed the "Problem with the SSL CA cert" problem while cloning from github.com
>>   installing the ca-certificates package.
>> - Squashed/split some commits
>> - use 'apt-get clean' instead of brutal 'rm -rf'
>> - checked how many cores are available on Shippable and use them fully
>>   (reduced total time from 44min to 38min)
>> - build armeb-linux-user target
>
> Ping - is there a v3 coming with Fam's comments addressed?

I did address Fam's comment, I'm not sure about how Fam wanted to 
document "where" extra files are copied to in the container's filesystem 
but tried to improve what was here.

There is an issue when you add files with the ADD directive, if any file 
is changed the script generates the same SHA1 hash and avoid to rebuild 
the image, I added some lines to also consider per-file checksum, but 
not for the shlibs case (I let a FIXME entry for that).

I then got stuck trying to improve `--include-executable`, I think the 
correct way would be to use ldd to get linked libraries from the host, 
then try to install them in the guest from the distrib packages, and 
finally copy what can't be find in the guest.
But better let it for another serie.

Regards,

Phil.